{
  "id": 74845,
  "title": "5-Fold CV RMSE differ from one another hugely  with LB=3.688",
  "url": "/competitions/elo-merchant-category-recommendation/discussion/74845",
  "author_name": "",
  "post_date": "2018-12-16T14:08:44.912514500Z",
  "votes": 1,
  "comment_count": 6,
  "views": 0,
  "content": "<p>5-Fold CV RMSE differ from one another hugely from 3.66911 - 3.62439 with LB=3.688\nfold 0\nTraining until validation scores don't improve for 100 rounds.\n[100]   training's rmse: 3.64962    valid_1's rmse: 3.71598\n[200]   training's rmse: 3.559  valid_1's rmse: 3.68575\n[300]   training's rmse: 3.50339    valid_1's rmse: 3.67455\n[400]   training's rmse: 3.46144    valid_1's rmse: 3.67047\n[500]   training's rmse: 3.42784    valid_1's rmse: 3.66764\n[600]   training's rmse: 3.39949    valid_1's rmse: 3.66577\n[700]   training's rmse: 3.37422    valid_1's rmse: 3.6652\n[800]   training's rmse: 3.35048    valid_1's rmse: 3.66444\n[900]   training's rmse: 3.32788    valid_1's rmse: 3.66384\n[1000]  training's rmse: 3.30667    valid_1's rmse: 3.66349\n[1100]  training's rmse: 3.28579    valid_1's rmse: 3.66314\n[1200]  training's rmse: 3.26698    valid_1's rmse: 3.66295\n[1300]  training's rmse: 3.24828    valid_1's rmse: 3.66263\nEarly stopping, best iteration is:\n[1220]  training's rmse: 3.263  valid_1's rmse: <strong>3.66258</strong>\nfold 1\nTraining until validation scores don't improve for 100 rounds.\n[100]   training's rmse: 3.64795    valid_1's rmse: 3.71574\n[200]   training's rmse: 3.55756    valid_1's rmse: 3.68686\n[300]   training's rmse: 3.50156    valid_1's rmse: 3.67625\n[400]   training's rmse: 3.46048    valid_1's rmse: 3.66943\n[500]   training's rmse: 3.42667    valid_1's rmse: 3.6651\n[600]   training's rmse: 3.39862    valid_1's rmse: 3.66238\n[700]   training's rmse: 3.37295    valid_1's rmse: 3.66078\n[800]   training's rmse: 3.35007    valid_1's rmse: 3.6595\n[900]   training's rmse: 3.32883    valid_1's rmse: 3.65819\n[1000]  training's rmse: 3.30789    valid_1's rmse: 3.65802\n[1100]  training's rmse: 3.28889    valid_1's rmse: 3.65813\n[1200]  training's rmse: 3.27056    valid_1's rmse: 3.65782\n[1300]  training's rmse: 3.25234    valid_1's rmse: 3.65754\n[1400]  training's rmse: 3.23427    valid_1's rmse: 3.65775\nEarly stopping, best iteration is:\n[1346]  training's rmse: 3.24337    valid_1's rmse: <strong>3.65747</strong></p>\n\n<p>fold 2\nTraining until validation scores don't improve for 100 rounds.\n[100]   training's rmse: 3.648  valid_1's rmse: 3.72625\n[200]   training's rmse: 3.55667    valid_1's rmse: 3.69621\n[300]   training's rmse: 3.50009    valid_1's rmse: 3.68529\n[400]   training's rmse: 3.45822    valid_1's rmse: 3.6779\n[500]   training's rmse: 3.42397    valid_1's rmse: 3.67421\n[600]   training's rmse: 3.39508    valid_1's rmse: 3.67265\n[700]   training's rmse: 3.36887    valid_1's rmse: 3.67127\n[800]   training's rmse: 3.34437    valid_1's rmse: 3.67052\n[900]   training's rmse: 3.32241    valid_1's rmse: 3.66988\n[1000]  training's rmse: 3.30081    valid_1's rmse: 3.66913\nEarly stopping, best iteration is:\n[996]   training's rmse: 3.30161    valid_1's rmse: <strong>3.66911</strong>\nfold 3\nTraining until validation scores don't improve for 100 rounds.\n[100]   training's rmse: 3.65365    valid_1's rmse: 3.70779\n[200]   training's rmse: 3.5667 valid_1's rmse: 3.6698\n[300]   training's rmse: 3.51158    valid_1's rmse: 3.65298\n[400]   training's rmse: 3.4712 valid_1's rmse: 3.64289\n[500]   training's rmse: 3.43725    valid_1's rmse: 3.63656\n[600]   training's rmse: 3.40892    valid_1's rmse: 3.6327\n[700]   training's rmse: 3.3842 valid_1's rmse: 3.63006\n[800]   training's rmse: 3.36107    valid_1's rmse: 3.62869\n[900]   training's rmse: 3.33958    valid_1's rmse: 3.62766\n[1000]  training's rmse: 3.31879    valid_1's rmse: 3.6268\n[1100]  training's rmse: 3.29866    valid_1's rmse: 3.62572\n[1200]  training's rmse: 3.27904    valid_1's rmse: 3.625\n[1300]  training's rmse: 3.26061    valid_1's rmse: 3.62452\n[1400]  training's rmse: 3.24266    valid_1's rmse: 3.6248\nEarly stopping, best iteration is:\n[1372]  training's rmse: 3.24788    valid_1's rmse: <strong>3.62439</strong>\nfold 4\nTraining until validation scores don't improve for 100 rounds.\n[100]   training's rmse: 3.65132    valid_1's rmse: 3.71327\n[200]   training's rmse: 3.56074    valid_1's rmse: 3.67946\n[300]   training's rmse: 3.50544    valid_1's rmse: 3.66704\n[400]   training's rmse: 3.46433    valid_1's rmse: 3.65962\n[500]   training's rmse: 3.43087    valid_1's rmse: 3.65497\n[600]   training's rmse: 3.40231    valid_1's rmse: 3.65176\n[700]   training's rmse: 3.37655    valid_1's rmse: 3.64977\n[800]   training's rmse: 3.35433    valid_1's rmse: 3.6484\n[900]   training's rmse: 3.33366    valid_1's rmse: 3.64782\n[1000]  training's rmse: 3.31459    valid_1's rmse: 3.64744\n[1100]  training's rmse: 3.29508    valid_1's rmse: 3.64718\n[1200]  training's rmse: 3.2763 valid_1's rmse: 3.64702\n[1300]  training's rmse: 3.25796    valid_1's rmse: 3.6464\n[1400]  training's rmse: 3.24062    valid_1's rmse: 3.64646\n[1500]  training's rmse: 3.22344    valid_1's rmse: 3.64657\nEarly stopping, best iteration is:\n[1440]  training's rmse: 3.2335 valid_1's rmse: <strong>3.64626</strong>\ndo you have the same issue?\nlooking for  teammates!</p>",
  "messages": [
    {
      "id": "439845",
      "postDate": "12/16/2018 14:08:44",
      "content": "<p>5-Fold CV RMSE differ from one another hugely from 3.66911 - 3.62439 with LB=3.688\nfold 0\nTraining until validation scores don't improve for 100 rounds.\n[100]   training's rmse: 3.64962    valid_1's rmse: 3.71598\n[200]   training's rmse: 3.559  valid_1's rmse: 3.68575\n[300]   training's rmse: 3.50339    valid_1's rmse: 3.67455\n[400]   training's rmse: 3.46144    valid_1's rmse: 3.67047\n[500]   training's rmse: 3.42784    valid_1's rmse: 3.66764\n[600]   training's rmse: 3.39949    valid_1's rmse: 3.66577\n[700]   training's rmse: 3.37422    valid_1's rmse: 3.6652\n[800]   training's rmse: 3.35048    valid_1's rmse: 3.66444\n[900]   training's rmse: 3.32788    valid_1's rmse: 3.66384\n[1000]  training's rmse: 3.30667    valid_1's rmse: 3.66349\n[1100]  training's rmse: 3.28579    valid_1's rmse: 3.66314\n[1200]  training's rmse: 3.26698    valid_1's rmse: 3.66295\n[1300]  training's rmse: 3.24828    valid_1's rmse: 3.66263\nEarly stopping, best iteration is:\n[1220]  training's rmse: 3.263  valid_1's rmse: <strong>3.66258</strong>\nfold 1\nTraining until validation scores don't improve for 100 rounds.\n[100]   training's rmse: 3.64795    valid_1's rmse: 3.71574\n[200]   training's rmse: 3.55756    valid_1's rmse: 3.68686\n[300]   training's rmse: 3.50156    valid_1's rmse: 3.67625\n[400]   training's rmse: 3.46048    valid_1's rmse: 3.66943\n[500]   training's rmse: 3.42667    valid_1's rmse: 3.6651\n[600]   training's rmse: 3.39862    valid_1's rmse: 3.66238\n[700]   training's rmse: 3.37295    valid_1's rmse: 3.66078\n[800]   training's rmse: 3.35007    valid_1's rmse: 3.6595\n[900]   training's rmse: 3.32883    valid_1's rmse: 3.65819\n[1000]  training's rmse: 3.30789    valid_1's rmse: 3.65802\n[1100]  training's rmse: 3.28889    valid_1's rmse: 3.65813\n[1200]  training's rmse: 3.27056    valid_1's rmse: 3.65782\n[1300]  training's rmse: 3.25234    valid_1's rmse: 3.65754\n[1400]  training's rmse: 3.23427    valid_1's rmse: 3.65775\nEarly stopping, best iteration is:\n[1346]  training's rmse: 3.24337    valid_1's rmse: <strong>3.65747</strong></p>\n\n<p>fold 2\nTraining until validation scores don't improve for 100 rounds.\n[100]   training's rmse: 3.648  valid_1's rmse: 3.72625\n[200]   training's rmse: 3.55667    valid_1's rmse: 3.69621\n[300]   training's rmse: 3.50009    valid_1's rmse: 3.68529\n[400]   training's rmse: 3.45822    valid_1's rmse: 3.6779\n[500]   training's rmse: 3.42397    valid_1's rmse: 3.67421\n[600]   training's rmse: 3.39508    valid_1's rmse: 3.67265\n[700]   training's rmse: 3.36887    valid_1's rmse: 3.67127\n[800]   training's rmse: 3.34437    valid_1's rmse: 3.67052\n[900]   training's rmse: 3.32241    valid_1's rmse: 3.66988\n[1000]  training's rmse: 3.30081    valid_1's rmse: 3.66913\nEarly stopping, best iteration is:\n[996]   training's rmse: 3.30161    valid_1's rmse: <strong>3.66911</strong>\nfold 3\nTraining until validation scores don't improve for 100 rounds.\n[100]   training's rmse: 3.65365    valid_1's rmse: 3.70779\n[200]   training's rmse: 3.5667 valid_1's rmse: 3.6698\n[300]   training's rmse: 3.51158    valid_1's rmse: 3.65298\n[400]   training's rmse: 3.4712 valid_1's rmse: 3.64289\n[500]   training's rmse: 3.43725    valid_1's rmse: 3.63656\n[600]   training's rmse: 3.40892    valid_1's rmse: 3.6327\n[700]   training's rmse: 3.3842 valid_1's rmse: 3.63006\n[800]   training's rmse: 3.36107    valid_1's rmse: 3.62869\n[900]   training's rmse: 3.33958    valid_1's rmse: 3.62766\n[1000]  training's rmse: 3.31879    valid_1's rmse: 3.6268\n[1100]  training's rmse: 3.29866    valid_1's rmse: 3.62572\n[1200]  training's rmse: 3.27904    valid_1's rmse: 3.625\n[1300]  training's rmse: 3.26061    valid_1's rmse: 3.62452\n[1400]  training's rmse: 3.24266    valid_1's rmse: 3.6248\nEarly stopping, best iteration is:\n[1372]  training's rmse: 3.24788    valid_1's rmse: <strong>3.62439</strong>\nfold 4\nTraining until validation scores don't improve for 100 rounds.\n[100]   training's rmse: 3.65132    valid_1's rmse: 3.71327\n[200]   training's rmse: 3.56074    valid_1's rmse: 3.67946\n[300]   training's rmse: 3.50544    valid_1's rmse: 3.66704\n[400]   training's rmse: 3.46433    valid_1's rmse: 3.65962\n[500]   training's rmse: 3.43087    valid_1's rmse: 3.65497\n[600]   training's rmse: 3.40231    valid_1's rmse: 3.65176\n[700]   training's rmse: 3.37655    valid_1's rmse: 3.64977\n[800]   training's rmse: 3.35433    valid_1's rmse: 3.6484\n[900]   training's rmse: 3.33366    valid_1's rmse: 3.64782\n[1000]  training's rmse: 3.31459    valid_1's rmse: 3.64744\n[1100]  training's rmse: 3.29508    valid_1's rmse: 3.64718\n[1200]  training's rmse: 3.2763 valid_1's rmse: 3.64702\n[1300]  training's rmse: 3.25796    valid_1's rmse: 3.6464\n[1400]  training's rmse: 3.24062    valid_1's rmse: 3.64646\n[1500]  training's rmse: 3.22344    valid_1's rmse: 3.64657\nEarly stopping, best iteration is:\n[1440]  training's rmse: 3.2335 valid_1's rmse: <strong>3.64626</strong>\ndo you have the same issue?\nlooking for  teammates!</p>",
      "rawMarkdown": "5-Fold CV RMSE differ from one another hugely from 3.66911 - 3.62439 with LB=3.688\nfold 0\nTraining until validation scores don't improve for 100 rounds.\n[100]\ttraining's rmse: 3.64962\tvalid_1's rmse: 3.71598\n[200]\ttraining's rmse: 3.559\tvalid_1's rmse: 3.68575\n[300]\ttraining's rmse: 3.50339\tvalid_1's rmse: 3.67455\n[400]\ttraining's rmse: 3.46144\tvalid_1's rmse: 3.67047\n[500]\ttraining's rmse: 3.42784\tvalid_1's rmse: 3.66764\n[600]\ttraining's rmse: 3.39949\tvalid_1's rmse: 3.66577\n[700]\ttraining's rmse: 3.37422\tvalid_1's rmse: 3.6652\n[800]\ttraining's rmse: 3.35048\tvalid_1's rmse: 3.66444\n[900]\ttraining's rmse: 3.32788\tvalid_1's rmse: 3.66384\n[1000]\ttraining's rmse: 3.30667\tvalid_1's rmse: 3.66349\n[1100]\ttraining's rmse: 3.28579\tvalid_1's rmse: 3.66314\n[1200]\ttraining's rmse: 3.26698\tvalid_1's rmse: 3.66295\n[1300]\ttraining's rmse: 3.24828\tvalid_1's rmse: 3.66263\nEarly stopping, best iteration is:\n[1220]\ttraining's rmse: 3.263\tvalid_1's rmse: **3.66258**\nfold 1\nTraining until validation scores don't improve for 100 rounds.\n[100]\ttraining's rmse: 3.64795\tvalid_1's rmse: 3.71574\n[200]\ttraining's rmse: 3.55756\tvalid_1's rmse: 3.68686\n[300]\ttraining's rmse: 3.50156\tvalid_1's rmse: 3.67625\n[400]\ttraining's rmse: 3.46048\tvalid_1's rmse: 3.66943\n[500]\ttraining's rmse: 3.42667\tvalid_1's rmse: 3.6651\n[600]\ttraining's rmse: 3.39862\tvalid_1's rmse: 3.66238\n[700]\ttraining's rmse: 3.37295\tvalid_1's rmse: 3.66078\n[800]\ttraining's rmse: 3.35007\tvalid_1's rmse: 3.6595\n[900]\ttraining's rmse: 3.32883\tvalid_1's rmse: 3.65819\n[1000]\ttraining's rmse: 3.30789\tvalid_1's rmse: 3.65802\n[1100]\ttraining's rmse: 3.28889\tvalid_1's rmse: 3.65813\n[1200]\ttraining's rmse: 3.27056\tvalid_1's rmse: 3.65782\n[1300]\ttraining's rmse: 3.25234\tvalid_1's rmse: 3.65754\n[1400]\ttraining's rmse: 3.23427\tvalid_1's rmse: 3.65775\nEarly stopping, best iteration is:\n[1346]\ttraining's rmse: 3.24337\tvalid_1's rmse: **3.65747**\n\nfold 2\nTraining until validation scores don't improve for 100 rounds.\n[100]\ttraining's rmse: 3.648\tvalid_1's rmse: 3.72625\n[200]\ttraining's rmse: 3.55667\tvalid_1's rmse: 3.69621\n[300]\ttraining's rmse: 3.50009\tvalid_1's rmse: 3.68529\n[400]\ttraining's rmse: 3.45822\tvalid_1's rmse: 3.6779\n[500]\ttraining's rmse: 3.42397\tvalid_1's rmse: 3.67421\n[600]\ttraining's rmse: 3.39508\tvalid_1's rmse: 3.67265\n[700]\ttraining's rmse: 3.36887\tvalid_1's rmse: 3.67127\n[800]\ttraining's rmse: 3.34437\tvalid_1's rmse: 3.67052\n[900]\ttraining's rmse: 3.32241\tvalid_1's rmse: 3.66988\n[1000]\ttraining's rmse: 3.30081\tvalid_1's rmse: 3.66913\nEarly stopping, best iteration is:\n[996]\ttraining's rmse: 3.30161\tvalid_1's rmse: **3.66911**\nfold 3\nTraining until validation scores don't improve for 100 rounds.\n[100]\ttraining's rmse: 3.65365\tvalid_1's rmse: 3.70779\n[200]\ttraining's rmse: 3.5667\tvalid_1's rmse: 3.6698\n[300]\ttraining's rmse: 3.51158\tvalid_1's rmse: 3.65298\n[400]\ttraining's rmse: 3.4712\tvalid_1's rmse: 3.64289\n[500]\ttraining's rmse: 3.43725\tvalid_1's rmse: 3.63656\n[600]\ttraining's rmse: 3.40892\tvalid_1's rmse: 3.6327\n[700]\ttraining's rmse: 3.3842\tvalid_1's rmse: 3.63006\n[800]\ttraining's rmse: 3.36107\tvalid_1's rmse: 3.62869\n[900]\ttraining's rmse: 3.33958\tvalid_1's rmse: 3.62766\n[1000]\ttraining's rmse: 3.31879\tvalid_1's rmse: 3.6268\n[1100]\ttraining's rmse: 3.29866\tvalid_1's rmse: 3.62572\n[1200]\ttraining's rmse: 3.27904\tvalid_1's rmse: 3.625\n[1300]\ttraining's rmse: 3.26061\tvalid_1's rmse: 3.62452\n[1400]\ttraining's rmse: 3.24266\tvalid_1's rmse: 3.6248\nEarly stopping, best iteration is:\n[1372]\ttraining's rmse: 3.24788\tvalid_1's rmse: **3.62439**\nfold 4\nTraining until validation scores don't improve for 100 rounds.\n[100]\ttraining's rmse: 3.65132\tvalid_1's rmse: 3.71327\n[200]\ttraining's rmse: 3.56074\tvalid_1's rmse: 3.67946\n[300]\ttraining's rmse: 3.50544\tvalid_1's rmse: 3.66704\n[400]\ttraining's rmse: 3.46433\tvalid_1's rmse: 3.65962\n[500]\ttraining's rmse: 3.43087\tvalid_1's rmse: 3.65497\n[600]\ttraining's rmse: 3.40231\tvalid_1's rmse: 3.65176\n[700]\ttraining's rmse: 3.37655\tvalid_1's rmse: 3.64977\n[800]\ttraining's rmse: 3.35433\tvalid_1's rmse: 3.6484\n[900]\ttraining's rmse: 3.33366\tvalid_1's rmse: 3.64782\n[1000]\ttraining's rmse: 3.31459\tvalid_1's rmse: 3.64744\n[1100]\ttraining's rmse: 3.29508\tvalid_1's rmse: 3.64718\n[1200]\ttraining's rmse: 3.2763\tvalid_1's rmse: 3.64702\n[1300]\ttraining's rmse: 3.25796\tvalid_1's rmse: 3.6464\n[1400]\ttraining's rmse: 3.24062\tvalid_1's rmse: 3.64646\n[1500]\ttraining's rmse: 3.22344\tvalid_1's rmse: 3.64657\nEarly stopping, best iteration is:\n[1440]\ttraining's rmse: 3.2335\tvalid_1's rmse: **3.64626**\ndo you have the same issue?\nlooking for  teammates!",
      "votes": null
    },
    {
      "id": "439857",
      "postDate": "12/16/2018 14:27:23",
      "content": "<p>Maybe you should set the \"outliners\" and use it to Kflod</p>",
      "rawMarkdown": "Maybe you should set the \"outliners\" and use it to Kflod",
      "votes": null
    },
    {
      "id": "439878",
      "postDate": "12/16/2018 15:14:22",
      "content": "<p>Yes ,i had did this with StratifiedKFold, and got the result above.</p>",
      "rawMarkdown": "Yes ,i had did this with StratifiedKFold, and got the result above.",
      "votes": null
    },
    {
      "id": "446048",
      "postDate": "12/27/2018 12:10:30",
      "content": "<p>Hi. I'm having the same problem. I divide the dataset using StratifiedKFold by 'is_outlier' but the rmse score for each fold isnt stable, just like yours. I have no clue why. Did you resolve the issue?</p>",
      "rawMarkdown": "Hi. I'm having the same problem. I divide the dataset using StratifiedKFold by 'is_outlier' but the rmse score for each fold isnt stable, just like yours. I have no clue why. Did you resolve the issue?",
      "votes": null
    },
    {
      "id": "446282",
      "postDate": "12/27/2018 19:55:17",
      "content": "<p>What are your hyperparameters? Some of the variation could come from randomness in the bagging (colsample / rowsample) or by the fold stratification not perfectly evenly distributing outliers.</p>",
      "rawMarkdown": "What are your hyperparameters? Some of the variation could come from randomness in the bagging (colsample / rowsample) or by the fold stratification not perfectly evenly distributing outliers.",
      "votes": null
    },
    {
      "id": "446303",
      "postDate": "12/27/2018 20:54:08",
      "content": "<p>I have the same, but I don't think it really is a problem - it is the overall CV score, an the STD between folds that I would look out for</p>",
      "rawMarkdown": "I have the same, but I don't think it really is a problem - it is the overall CV score, an the STD between folds that I would look out for",
      "votes": null
    },
    {
      "id": "446348",
      "postDate": "12/27/2018 23:08:11",
      "content": "<p><a href=\"https://www.kaggle.com/chauhuynh/my-first-kernel-3-699\">https://www.kaggle.com/chauhuynh/my-first-kernel-3-699</a>\nI use the same parameters(well except for random_state) and CV strategy with this kernel, in which the rmse score for each fold seems very stable, but my rmse score for each fold is 1: 3.6604, 2: 3.63968, 3: 3.64905, 4: 3.67192, 5: 3.6209. I dont necessarily think this is the problem but was just wondering why...</p>",
      "rawMarkdown": "https://www.kaggle.com/chauhuynh/my-first-kernel-3-699\nI use the same parameters(well except for random_state) and CV strategy with this kernel, in which the rmse score for each fold seems very stable, but my rmse score for each fold is 1: 3.6604, 2: 3.63968, 3: 3.64905, 4: 3.67192, 5: 3.6209. I dont necessarily think this is the problem but was just wondering why...",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 439857,
      "author_name": "lblhandsome",
      "author_url": "",
      "post_date": "12/16/2018 14:27:23",
      "content": "<p>Maybe you should set the \"outliners\" and use it to Kflod</p>",
      "votes": null,
      "replies": [
        {
          "id": 439878,
          "author_name": "codlife",
          "author_url": "",
          "post_date": "12/16/2018 15:14:22",
          "content": "<p>Yes ,i had did this with StratifiedKFold, and got the result above.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 446048,
          "author_name": "ykskks",
          "author_url": "",
          "post_date": "12/27/2018 12:10:30",
          "content": "<p>Hi. I'm having the same problem. I divide the dataset using StratifiedKFold by 'is_outlier' but the rmse score for each fold isnt stable, just like yours. I have no clue why. Did you resolve the issue?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 446282,
      "author_name": "peterhurford",
      "author_url": "",
      "post_date": "12/27/2018 19:55:17",
      "content": "<p>What are your hyperparameters? Some of the variation could come from randomness in the bagging (colsample / rowsample) or by the fold stratification not perfectly evenly distributing outliers.</p>",
      "votes": null,
      "replies": [
        {
          "id": 446348,
          "author_name": "ykskks",
          "author_url": "",
          "post_date": "12/27/2018 23:08:11",
          "content": "<p><a href=\"https://www.kaggle.com/chauhuynh/my-first-kernel-3-699\">https://www.kaggle.com/chauhuynh/my-first-kernel-3-699</a>\nI use the same parameters(well except for random_state) and CV strategy with this kernel, in which the rmse score for each fold seems very stable, but my rmse score for each fold is 1: 3.6604, 2: 3.63968, 3: 3.64905, 4: 3.67192, 5: 3.6209. I dont necessarily think this is the problem but was just wondering why...</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 446303,
      "author_name": "yifanxie",
      "author_url": "",
      "post_date": "12/27/2018 20:54:08",
      "content": "<p>I have the same, but I don't think it really is a problem - it is the overall CV score, an the STD between folds that I would look out for</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "439845": "5-Fold CV RMSE differ from one another hugely from 3.66911 - 3.62439 with LB=3.688\nfold 0\nTraining until validation scores don't improve for 100 rounds.\n[100]\ttraining's rmse: 3.64962\tvalid_1's rmse: 3.71598\n[200]\ttraining's rmse: 3.559\tvalid_1's rmse: 3.68575\n[300]\ttraining's rmse: 3.50339\tvalid_1's rmse: 3.67455\n[400]\ttraining's rmse: 3.46144\tvalid_1's rmse: 3.67047\n[500]\ttraining's rmse: 3.42784\tvalid_1's rmse: 3.66764\n[600]\ttraining's rmse: 3.39949\tvalid_1's rmse: 3.66577\n[700]\ttraining's rmse: 3.37422\tvalid_1's rmse: 3.6652\n[800]\ttraining's rmse: 3.35048\tvalid_1's rmse: 3.66444\n[900]\ttraining's rmse: 3.32788\tvalid_1's rmse: 3.66384\n[1000]\ttraining's rmse: 3.30667\tvalid_1's rmse: 3.66349\n[1100]\ttraining's rmse: 3.28579\tvalid_1's rmse: 3.66314\n[1200]\ttraining's rmse: 3.26698\tvalid_1's rmse: 3.66295\n[1300]\ttraining's rmse: 3.24828\tvalid_1's rmse: 3.66263\nEarly stopping, best iteration is:\n[1220]\ttraining's rmse: 3.263\tvalid_1's rmse: **3.66258**\nfold 1\nTraining until validation scores don't improve for 100 rounds.\n[100]\ttraining's rmse: 3.64795\tvalid_1's rmse: 3.71574\n[200]\ttraining's rmse: 3.55756\tvalid_1's rmse: 3.68686\n[300]\ttraining's rmse: 3.50156\tvalid_1's rmse: 3.67625\n[400]\ttraining's rmse: 3.46048\tvalid_1's rmse: 3.66943\n[500]\ttraining's rmse: 3.42667\tvalid_1's rmse: 3.6651\n[600]\ttraining's rmse: 3.39862\tvalid_1's rmse: 3.66238\n[700]\ttraining's rmse: 3.37295\tvalid_1's rmse: 3.66078\n[800]\ttraining's rmse: 3.35007\tvalid_1's rmse: 3.6595\n[900]\ttraining's rmse: 3.32883\tvalid_1's rmse: 3.65819\n[1000]\ttraining's rmse: 3.30789\tvalid_1's rmse: 3.65802\n[1100]\ttraining's rmse: 3.28889\tvalid_1's rmse: 3.65813\n[1200]\ttraining's rmse: 3.27056\tvalid_1's rmse: 3.65782\n[1300]\ttraining's rmse: 3.25234\tvalid_1's rmse: 3.65754\n[1400]\ttraining's rmse: 3.23427\tvalid_1's rmse: 3.65775\nEarly stopping, best iteration is:\n[1346]\ttraining's rmse: 3.24337\tvalid_1's rmse: **3.65747**\n\nfold 2\nTraining until validation scores don't improve for 100 rounds.\n[100]\ttraining's rmse: 3.648\tvalid_1's rmse: 3.72625\n[200]\ttraining's rmse: 3.55667\tvalid_1's rmse: 3.69621\n[300]\ttraining's rmse: 3.50009\tvalid_1's rmse: 3.68529\n[400]\ttraining's rmse: 3.45822\tvalid_1's rmse: 3.6779\n[500]\ttraining's rmse: 3.42397\tvalid_1's rmse: 3.67421\n[600]\ttraining's rmse: 3.39508\tvalid_1's rmse: 3.67265\n[700]\ttraining's rmse: 3.36887\tvalid_1's rmse: 3.67127\n[800]\ttraining's rmse: 3.34437\tvalid_1's rmse: 3.67052\n[900]\ttraining's rmse: 3.32241\tvalid_1's rmse: 3.66988\n[1000]\ttraining's rmse: 3.30081\tvalid_1's rmse: 3.66913\nEarly stopping, best iteration is:\n[996]\ttraining's rmse: 3.30161\tvalid_1's rmse: **3.66911**\nfold 3\nTraining until validation scores don't improve for 100 rounds.\n[100]\ttraining's rmse: 3.65365\tvalid_1's rmse: 3.70779\n[200]\ttraining's rmse: 3.5667\tvalid_1's rmse: 3.6698\n[300]\ttraining's rmse: 3.51158\tvalid_1's rmse: 3.65298\n[400]\ttraining's rmse: 3.4712\tvalid_1's rmse: 3.64289\n[500]\ttraining's rmse: 3.43725\tvalid_1's rmse: 3.63656\n[600]\ttraining's rmse: 3.40892\tvalid_1's rmse: 3.6327\n[700]\ttraining's rmse: 3.3842\tvalid_1's rmse: 3.63006\n[800]\ttraining's rmse: 3.36107\tvalid_1's rmse: 3.62869\n[900]\ttraining's rmse: 3.33958\tvalid_1's rmse: 3.62766\n[1000]\ttraining's rmse: 3.31879\tvalid_1's rmse: 3.6268\n[1100]\ttraining's rmse: 3.29866\tvalid_1's rmse: 3.62572\n[1200]\ttraining's rmse: 3.27904\tvalid_1's rmse: 3.625\n[1300]\ttraining's rmse: 3.26061\tvalid_1's rmse: 3.62452\n[1400]\ttraining's rmse: 3.24266\tvalid_1's rmse: 3.6248\nEarly stopping, best iteration is:\n[1372]\ttraining's rmse: 3.24788\tvalid_1's rmse: **3.62439**\nfold 4\nTraining until validation scores don't improve for 100 rounds.\n[100]\ttraining's rmse: 3.65132\tvalid_1's rmse: 3.71327\n[200]\ttraining's rmse: 3.56074\tvalid_1's rmse: 3.67946\n[300]\ttraining's rmse: 3.50544\tvalid_1's rmse: 3.66704\n[400]\ttraining's rmse: 3.46433\tvalid_1's rmse: 3.65962\n[500]\ttraining's rmse: 3.43087\tvalid_1's rmse: 3.65497\n[600]\ttraining's rmse: 3.40231\tvalid_1's rmse: 3.65176\n[700]\ttraining's rmse: 3.37655\tvalid_1's rmse: 3.64977\n[800]\ttraining's rmse: 3.35433\tvalid_1's rmse: 3.6484\n[900]\ttraining's rmse: 3.33366\tvalid_1's rmse: 3.64782\n[1000]\ttraining's rmse: 3.31459\tvalid_1's rmse: 3.64744\n[1100]\ttraining's rmse: 3.29508\tvalid_1's rmse: 3.64718\n[1200]\ttraining's rmse: 3.2763\tvalid_1's rmse: 3.64702\n[1300]\ttraining's rmse: 3.25796\tvalid_1's rmse: 3.6464\n[1400]\ttraining's rmse: 3.24062\tvalid_1's rmse: 3.64646\n[1500]\ttraining's rmse: 3.22344\tvalid_1's rmse: 3.64657\nEarly stopping, best iteration is:\n[1440]\ttraining's rmse: 3.2335\tvalid_1's rmse: **3.64626**\ndo you have the same issue?\nlooking for  teammates!",
    "439857": "Maybe you should set the \"outliners\" and use it to Kflod",
    "439878": "Yes ,i had did this with StratifiedKFold, and got the result above.",
    "446048": "Hi. I'm having the same problem. I divide the dataset using StratifiedKFold by 'is_outlier' but the rmse score for each fold isnt stable, just like yours. I have no clue why. Did you resolve the issue?",
    "446282": "What are your hyperparameters? Some of the variation could come from randomness in the bagging (colsample / rowsample) or by the fold stratification not perfectly evenly distributing outliers.",
    "446303": "I have the same, but I don't think it really is a problem - it is the overall CV score, an the STD between folds that I would look out for",
    "446348": "https://www.kaggle.com/chauhuynh/my-first-kernel-3-699\nI use the same parameters(well except for random_state) and CV strategy with this kernel, in which the rmse score for each fold seems very stable, but my rmse score for each fold is 1: 3.6604, 2: 3.63968, 3: 3.64905, 4: 3.67192, 5: 3.6209. I dont necessarily think this is the problem but was just wondering why..."
  },
  "source": "meta"
}