{
  "id": 74242,
  "title": "Bad Seeds: the sequel",
  "url": "/competitions/elo-merchant-category-recommendation/discussion/74242",
  "author_name": "Konrad Banachewicz",
  "post_date": "2018-12-10T11:59:09.028000",
  "votes": 6,
  "comment_count": 5,
  "views": 0,
  "content": "<p>TL;DR the impact of random seed on the results is off the charts</p>\n\n<p>I decided to investigate the issue of variation across folds in a bit more detail and fed the whole dataset into a pipeline with RepeatedKFolds (the main purpose was actually looking at feature selection, but never mind). The relevant part of the code is:</p>\n\n<pre><code>folds = RepeatedKFold(n_splits= 5, n_repeats= 3, random_state= 15)\nfeature_importance_df = np.zeros((xtrain.shape[1], 1))\n\nparam = {'num_leaves': 120, 'min_data_in_leaf': 30,  'objective':'regression',\n         'max_depth': -1, 'learning_rate': 0.005, \"min_child_samples\": 30,\n         \"boosting\": \"gbdt\", \"feature_fraction\": 0.9, \"bagging_freq\": 1,\n         \"bagging_fraction\": 0.9 , \"bagging_seed\": 11, \"metric\": 'rmse',\n         \"lambda_l1\": 0.1, \"verbosity\": -1}\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(xtrain.values, ytrain.values)):\n    print('----')\n    print(\"fold n°{}\".format(fold_))\n\n    x0,y0 = xtrain.iloc[trn_idx], ytrain[trn_idx]\n    x1,y1 = xtrain.iloc[val_idx], ytrain[val_idx]    \n    trn_data = lgb.Dataset(x0, label= y0)\n    val_data = lgb.Dataset(x1, label= y1)\n\n    num_round = 10000\n\n    clf = lgb.train(param, trn_data, num_round, \n                       valid_sets = [trn_data, val_data], \n                        verbose_eval= 3000, early_stopping_rounds = 100)\n\n    feature_importance_df[:, 0] += clf.feature_importance()\n</code></pre>\n\n<p>And the (partial for brevity) output:</p>\n\n<pre><code>----\nfold n°0\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1192]  training's rmse: 3.01158    valid_1's rmse: 3.6761\n----\nfold n°1\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1051]  training's rmse: 3.06579    valid_1's rmse: 3.62221\n----\nfold n°2\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1060]  training's rmse: 3.0686 valid_1's rmse: 3.60256\n----\nfold n°3\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1152]  training's rmse: 2.9965 valid_1's rmse: 3.78612\n----\nfold n°4\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[901]   training's rmse: 3.11762    valid_1's rmse: 3.62245\n----\nfold n°5\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1124]  training's rmse: 3.0321 valid_1's rmse: 3.70657\n----\nfold n°6\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1036]  training's rmse: 3.05444    valid_1's rmse: 3.67023\n----\nfold n°7\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1312]  training's rmse: 2.9719 valid_1's rmse: 3.69204\n----\nfold n°8\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1008]  training's rmse: 3.09901    valid_1's rmse: 3.54231\n----\nfold n°9\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1736]  training's rmse: 2.86439    valid_1's rmse: 3.6906\n----\nfold n°10\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1608]  training's rmse: 2.89212    valid_1's rmse: 3.6885\n----\nfold n°11\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1286]  training's rmse: 2.98919    valid_1's rmse: 3.6467\n----\nfold n°12\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1229]  training's rmse: 3.0063 valid_1's rmse: 3.66029\n----\nfold n°13\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1478]  training's rmse: 2.92354    valid_1's rmse: 3.68482\n----\nfold n°14\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1056]  training's rmse: 3.06682    valid_1's rmse: 3.61824\n</code></pre>",
  "messages": [
    {
      "id": 436484,
      "postDate": "2018-12-10T11:59:09.027Z",
      "content": "<p>TL;DR the impact of random seed on the results is off the charts</p>\n\n<p>I decided to investigate the issue of variation across folds in a bit more detail and fed the whole dataset into a pipeline with RepeatedKFolds (the main purpose was actually looking at feature selection, but never mind). The relevant part of the code is:</p>\n\n<pre><code>folds = RepeatedKFold(n_splits= 5, n_repeats= 3, random_state= 15)\nfeature_importance_df = np.zeros((xtrain.shape[1], 1))\n\nparam = {'num_leaves': 120, 'min_data_in_leaf': 30,  'objective':'regression',\n         'max_depth': -1, 'learning_rate': 0.005, \"min_child_samples\": 30,\n         \"boosting\": \"gbdt\", \"feature_fraction\": 0.9, \"bagging_freq\": 1,\n         \"bagging_fraction\": 0.9 , \"bagging_seed\": 11, \"metric\": 'rmse',\n         \"lambda_l1\": 0.1, \"verbosity\": -1}\n\nfor fold_, (trn_idx, val_idx) in enumerate(folds.split(xtrain.values, ytrain.values)):\n    print('----')\n    print(\"fold n°{}\".format(fold_))\n\n    x0,y0 = xtrain.iloc[trn_idx], ytrain[trn_idx]\n    x1,y1 = xtrain.iloc[val_idx], ytrain[val_idx]    \n    trn_data = lgb.Dataset(x0, label= y0)\n    val_data = lgb.Dataset(x1, label= y1)\n\n    num_round = 10000\n\n    clf = lgb.train(param, trn_data, num_round, \n                       valid_sets = [trn_data, val_data], \n                        verbose_eval= 3000, early_stopping_rounds = 100)\n\n    feature_importance_df[:, 0] += clf.feature_importance()\n</code></pre>\n\n<p>And the (partial for brevity) output:</p>\n\n<pre><code>----\nfold n°0\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1192]  training's rmse: 3.01158    valid_1's rmse: 3.6761\n----\nfold n°1\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1051]  training's rmse: 3.06579    valid_1's rmse: 3.62221\n----\nfold n°2\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1060]  training's rmse: 3.0686 valid_1's rmse: 3.60256\n----\nfold n°3\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1152]  training's rmse: 2.9965 valid_1's rmse: 3.78612\n----\nfold n°4\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[901]   training's rmse: 3.11762    valid_1's rmse: 3.62245\n----\nfold n°5\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1124]  training's rmse: 3.0321 valid_1's rmse: 3.70657\n----\nfold n°6\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1036]  training's rmse: 3.05444    valid_1's rmse: 3.67023\n----\nfold n°7\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1312]  training's rmse: 2.9719 valid_1's rmse: 3.69204\n----\nfold n°8\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1008]  training's rmse: 3.09901    valid_1's rmse: 3.54231\n----\nfold n°9\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1736]  training's rmse: 2.86439    valid_1's rmse: 3.6906\n----\nfold n°10\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1608]  training's rmse: 2.89212    valid_1's rmse: 3.6885\n----\nfold n°11\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1286]  training's rmse: 2.98919    valid_1's rmse: 3.6467\n----\nfold n°12\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1229]  training's rmse: 3.0063 valid_1's rmse: 3.66029\n----\nfold n°13\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1478]  training's rmse: 2.92354    valid_1's rmse: 3.68482\n----\nfold n°14\nTraining until validation scores don't improve for 100 rounds.\nEarly stopping, best iteration is:\n[1056]  training's rmse: 3.06682    valid_1's rmse: 3.61824\n</code></pre>",
      "rawMarkdown": "TL;DR the impact of random seed on the results is off the charts\n\n\nI decided to investigate the issue of variation across folds in a bit more detail and fed the whole dataset into a pipeline with RepeatedKFolds (the main purpose was actually looking at feature selection, but never mind). The relevant part of the code is:\n\n    folds = RepeatedKFold(n_splits= 5, n_repeats= 3, random_state= 15)\n    feature_importance_df = np.zeros((xtrain.shape[1], 1))\n    \n    param = {'num_leaves': 120, 'min_data_in_leaf': 30,  'objective':'regression',\n             'max_depth': -1, 'learning_rate': 0.005, \"min_child_samples\": 30,\n             \"boosting\": \"gbdt\", \"feature_fraction\": 0.9, \"bagging_freq\": 1,\n             \"bagging_fraction\": 0.9 , \"bagging_seed\": 11, \"metric\": 'rmse',\n             \"lambda_l1\": 0.1, \"verbosity\": -1}\n    \n    for fold_, (trn_idx, val_idx) in enumerate(folds.split(xtrain.values, ytrain.values)):\n        print('----')\n        print(\"fold n°{}\".format(fold_))\n        \n        x0,y0 = xtrain.iloc[trn_idx], ytrain[trn_idx]\n        x1,y1 = xtrain.iloc[val_idx], ytrain[val_idx]    \n        trn_data = lgb.Dataset(x0, label= y0)\n        val_data = lgb.Dataset(x1, label= y1)\n        \n        num_round = 10000\n        \n        clf = lgb.train(param, trn_data, num_round, \n                           valid_sets = [trn_data, val_data], \n                            verbose_eval= 3000, early_stopping_rounds = 100)\n    \n        feature_importance_df[:, 0] += clf.feature_importance()\n\n\nAnd the (partial for brevity) output:\n\n    ----\n    fold n°0\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1192]\ttraining's rmse: 3.01158\tvalid_1's rmse: 3.6761\n    ----\n    fold n°1\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1051]\ttraining's rmse: 3.06579\tvalid_1's rmse: 3.62221\n    ----\n    fold n°2\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1060]\ttraining's rmse: 3.0686\tvalid_1's rmse: 3.60256\n    ----\n    fold n°3\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1152]\ttraining's rmse: 2.9965\tvalid_1's rmse: 3.78612\n    ----\n    fold n°4\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [901]\ttraining's rmse: 3.11762\tvalid_1's rmse: 3.62245\n    ----\n    fold n°5\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1124]\ttraining's rmse: 3.0321\tvalid_1's rmse: 3.70657\n    ----\n    fold n°6\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1036]\ttraining's rmse: 3.05444\tvalid_1's rmse: 3.67023\n    ----\n    fold n°7\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1312]\ttraining's rmse: 2.9719\tvalid_1's rmse: 3.69204\n    ----\n    fold n°8\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1008]\ttraining's rmse: 3.09901\tvalid_1's rmse: 3.54231\n    ----\n    fold n°9\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1736]\ttraining's rmse: 2.86439\tvalid_1's rmse: 3.6906\n    ----\n    fold n°10\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1608]\ttraining's rmse: 2.89212\tvalid_1's rmse: 3.6885\n    ----\n    fold n°11\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1286]\ttraining's rmse: 2.98919\tvalid_1's rmse: 3.6467\n    ----\n    fold n°12\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1229]\ttraining's rmse: 3.0063\tvalid_1's rmse: 3.66029\n    ----\n    fold n°13\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1478]\ttraining's rmse: 2.92354\tvalid_1's rmse: 3.68482\n    ----\n    fold n°14\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1056]\ttraining's rmse: 3.06682\tvalid_1's rmse: 3.61824\n\n",
      "votes": 6
    },
    {
      "id": 437777,
      "postDate": "2018-12-12T13:32:06.180Z",
      "content": "<p>depending on the seed used, I have standard deviations in the range of 0.03x to 0.06x between folds, so yes it is quite a big delta if you look at the top of LB which is mostly separated by 0.00x.</p>\n\n<p>another thing I observe is that this data seems to be quite sensitive to overfitting </p>",
      "rawMarkdown": "depending on the seed used, I have standard deviations in the range of 0.03x to 0.06x between folds, so yes it is quite a big delta if you look at the top of LB which is mostly separated by 0.00x.\n\nanother thing I observe is that this data seems to be quite sensitive to overfitting "
    },
    {
      "id": 436981,
      "postDate": "2018-12-11T07:18:47.620Z",
      "content": "<p>Jesus. This might just mean that the private LB completely upends the standings we see right now. </p>",
      "rawMarkdown": "Jesus. This might just mean that the private LB completely upends the standings we see right now. ",
      "replies": [
        {
          "id": 437067,
          "postDate": "2018-12-11T10:16:37.410Z",
          "content": "<p>Yup, that's precisely what it might end like.</p>",
          "rawMarkdown": "Yup, that's precisely what it might end like."
        }
      ]
    },
    {
      "id": 436502,
      "postDate": "2018-12-10T12:48:24.897Z",
      "rawMarkdown": "",
      "isDeleted": true,
      "replies": [
        {
          "id": 436505,
          "postDate": "2018-12-10T12:52:24.263Z",
          "content": "<p>That's the thing: it's really really worrying - the dependence shouldn't be that strong...</p>",
          "rawMarkdown": "That's the thing: it's really really worrying - the dependence shouldn't be that strong..."
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 437777,
      "author_name": "Yifan Xie",
      "author_url": "",
      "post_date": "2018-12-12T13:32:06.180000",
      "content": "<p>depending on the seed used, I have standard deviations in the range of 0.03x to 0.06x between folds, so yes it is quite a big delta if you look at the top of LB which is mostly separated by 0.00x.</p>\n\n<p>another thing I observe is that this data seems to be quite sensitive to overfitting </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 436981,
      "author_name": "Panchajanya Banerjee (Pancham)",
      "author_url": "",
      "post_date": "2018-12-11T07:18:47.620000",
      "content": "<p>Jesus. This might just mean that the private LB completely upends the standings we see right now. </p>",
      "votes": 0,
      "replies": [
        {
          "id": 437067,
          "author_name": "Konrad Banachewicz",
          "author_url": "",
          "post_date": "2018-12-11T10:16:37.410000",
          "content": "<p>Yup, that's precisely what it might end like.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 436502,
      "author_name": "",
      "author_url": "",
      "post_date": "2018-12-10T12:48:24.897000",
      "content": "",
      "votes": 0,
      "replies": [
        {
          "id": 436505,
          "author_name": "Konrad Banachewicz",
          "author_url": "",
          "post_date": "2018-12-10T12:52:24.263000",
          "content": "<p>That's the thing: it's really really worrying - the dependence shouldn't be that strong...</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "436484": "TL;DR the impact of random seed on the results is off the charts\n\n\nI decided to investigate the issue of variation across folds in a bit more detail and fed the whole dataset into a pipeline with RepeatedKFolds (the main purpose was actually looking at feature selection, but never mind). The relevant part of the code is:\n\n    folds = RepeatedKFold(n_splits= 5, n_repeats= 3, random_state= 15)\n    feature_importance_df = np.zeros((xtrain.shape[1], 1))\n    \n    param = {'num_leaves': 120, 'min_data_in_leaf': 30,  'objective':'regression',\n             'max_depth': -1, 'learning_rate': 0.005, \"min_child_samples\": 30,\n             \"boosting\": \"gbdt\", \"feature_fraction\": 0.9, \"bagging_freq\": 1,\n             \"bagging_fraction\": 0.9 , \"bagging_seed\": 11, \"metric\": 'rmse',\n             \"lambda_l1\": 0.1, \"verbosity\": -1}\n    \n    for fold_, (trn_idx, val_idx) in enumerate(folds.split(xtrain.values, ytrain.values)):\n        print('----')\n        print(\"fold n°{}\".format(fold_))\n        \n        x0,y0 = xtrain.iloc[trn_idx], ytrain[trn_idx]\n        x1,y1 = xtrain.iloc[val_idx], ytrain[val_idx]    \n        trn_data = lgb.Dataset(x0, label= y0)\n        val_data = lgb.Dataset(x1, label= y1)\n        \n        num_round = 10000\n        \n        clf = lgb.train(param, trn_data, num_round, \n                           valid_sets = [trn_data, val_data], \n                            verbose_eval= 3000, early_stopping_rounds = 100)\n    \n        feature_importance_df[:, 0] += clf.feature_importance()\n\n\nAnd the (partial for brevity) output:\n\n    ----\n    fold n°0\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1192]\ttraining's rmse: 3.01158\tvalid_1's rmse: 3.6761\n    ----\n    fold n°1\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1051]\ttraining's rmse: 3.06579\tvalid_1's rmse: 3.62221\n    ----\n    fold n°2\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1060]\ttraining's rmse: 3.0686\tvalid_1's rmse: 3.60256\n    ----\n    fold n°3\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1152]\ttraining's rmse: 2.9965\tvalid_1's rmse: 3.78612\n    ----\n    fold n°4\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [901]\ttraining's rmse: 3.11762\tvalid_1's rmse: 3.62245\n    ----\n    fold n°5\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1124]\ttraining's rmse: 3.0321\tvalid_1's rmse: 3.70657\n    ----\n    fold n°6\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1036]\ttraining's rmse: 3.05444\tvalid_1's rmse: 3.67023\n    ----\n    fold n°7\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1312]\ttraining's rmse: 2.9719\tvalid_1's rmse: 3.69204\n    ----\n    fold n°8\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1008]\ttraining's rmse: 3.09901\tvalid_1's rmse: 3.54231\n    ----\n    fold n°9\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1736]\ttraining's rmse: 2.86439\tvalid_1's rmse: 3.6906\n    ----\n    fold n°10\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1608]\ttraining's rmse: 2.89212\tvalid_1's rmse: 3.6885\n    ----\n    fold n°11\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1286]\ttraining's rmse: 2.98919\tvalid_1's rmse: 3.6467\n    ----\n    fold n°12\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1229]\ttraining's rmse: 3.0063\tvalid_1's rmse: 3.66029\n    ----\n    fold n°13\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1478]\ttraining's rmse: 2.92354\tvalid_1's rmse: 3.68482\n    ----\n    fold n°14\n    Training until validation scores don't improve for 100 rounds.\n    Early stopping, best iteration is:\n    [1056]\ttraining's rmse: 3.06682\tvalid_1's rmse: 3.61824\n\n",
    "437777": "depending on the seed used, I have standard deviations in the range of 0.03x to 0.06x between folds, so yes it is quite a big delta if you look at the top of LB which is mostly separated by 0.00x.\n\nanother thing I observe is that this data seems to be quite sensitive to overfitting ",
    "436981": "Jesus. This might just mean that the private LB completely upends the standings we see right now. ",
    "436502": ""
  }
}