{
  "id": 80033,
  "title": "single model - 3.52 ( No CV) on local machine - But LB -3.75",
  "url": "/competitions/elo-merchant-category-recommendation/discussion/80033",
  "author_name": "",
  "post_date": "2019-02-09T18:45:44.759486900Z",
  "votes": 2,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Hi Everyone\nBuild my first model with LGB and used stratified sampling to make sure outliers are picked in equal proportion.</p>\n\n<p>I have got this weird situation where my CV score is:\n**</p>\n\n<ul>\n<li>Training until validation scores don't improve for 100 rounds.</li>\n</ul>\n\n<p>**</p>\n\n<p>[100]   valid_0's rmse: 3.54873   </p>\n\n<p>[200]   valid_0's rmse: 3.53051   </p>\n\n<p>[300]   valid_0's rmse: 3.529  </p>\n\n<p>Early stopping, best iteration is:  </p>\n\n<p>[297]   valid_0's rmse: 3.5289  </p>\n\n<p>But LB score is 3.75!!</p>\n\n<p>Total features are around 160.</p>\n\n<p>Did you face similar situation, and what did you do to improve it?\nWhat could be possible reasons?</p>",
  "messages": [
    {
      "id": "468817",
      "postDate": "02/09/2019 18:45:44",
      "content": "<p>Hi Everyone\nBuild my first model with LGB and used stratified sampling to make sure outliers are picked in equal proportion.</p>\n\n<p>I have got this weird situation where my CV score is:\n**</p>\n\n<ul>\n<li>Training until validation scores don't improve for 100 rounds.</li>\n</ul>\n\n<p>**</p>\n\n<p>[100]   valid_0's rmse: 3.54873   </p>\n\n<p>[200]   valid_0's rmse: 3.53051   </p>\n\n<p>[300]   valid_0's rmse: 3.529  </p>\n\n<p>Early stopping, best iteration is:  </p>\n\n<p>[297]   valid_0's rmse: 3.5289  </p>\n\n<p>But LB score is 3.75!!</p>\n\n<p>Total features are around 160.</p>\n\n<p>Did you face similar situation, and what did you do to improve it?\nWhat could be possible reasons?</p>",
      "rawMarkdown": "Hi Everyone\nBuild my first model with LGB and used stratified sampling to make sure outliers are picked in equal proportion.\n\nI have got this weird situation where my CV score is:\n**\n\n - Training until validation scores don't improve for 100 rounds.\n\n**\n\n[100]\tvalid_0's rmse: 3.54873   \n\n[200]\tvalid_0's rmse: 3.53051   \n\n[300]\tvalid_0's rmse: 3.529  \n\nEarly stopping, best iteration is:  \n\n[297]\tvalid_0's rmse: 3.5289  \n\nBut LB score is 3.75!!\n\nTotal features are around 160.\n\nDid you face similar situation, and what did you do to improve it?\nWhat could be possible reasons?",
      "votes": null
    },
    {
      "id": "468823",
      "postDate": "02/09/2019 18:59:54",
      "content": "<p>Everyone has a difference between their cv and lb score due to a greater number of outliers in the leader board set.\nI have a cv score of 3.65 and and lb score of 3.692, which the difference is still apparent but not as much as yours is.  I used 180 features for the model.  If I had to guess, I would say that you are training your model with few variables, which are predictive for the training cv set but not as predictive for the leader board set.</p>\n\n<p>If you want to better your leader board score, look at some kernels and copy some of their features.  Don't worry about closing the gap between the cv and lb score.  Just worry about bettering both of the scores with new features.</p>",
      "rawMarkdown": "Everyone has a difference between their cv and lb score due to a greater number of outliers in the leader board set.\nI have a cv score of 3.65 and and lb score of 3.692, which the difference is still apparent but not as much as yours is.  I used 180 features for the model.  If I had to guess, I would say that you are training your model with few variables, which are predictive for the training cv set but not as predictive for the leader board set.\n\nIf you want to better your leader board score, look at some kernels and copy some of their features.  Don't worry about closing the gap between the cv and lb score.  Just worry about bettering both of the scores with new features.",
      "votes": null
    },
    {
      "id": "468885",
      "postDate": "02/10/2019 00:03:36",
      "content": "<p>what do u mean by no CV? I am not sure I am following. How do you divide your training/validation dataset?</p>",
      "rawMarkdown": "what do u mean by no CV? I am not sure I am following. How do you divide your training/validation dataset?",
      "votes": null
    },
    {
      "id": "468920",
      "postDate": "02/10/2019 03:31:00",
      "content": "<p>Well, I have divided the target variable in to 8 bins based on how far they are from mean, and picked equal proportion from each bin using train_test_split(80%,20%)</p>",
      "rawMarkdown": "Well, I have divided the target variable in to 8 bins based on how far they are from mean, and picked equal proportion from each bin using train_test_split(80%,20%)",
      "votes": null
    },
    {
      "id": "469129",
      "postDate": "02/10/2019 14:59:16",
      "content": "<p>That's only single fold results - are you sure you're doing cross-validation? We say CV talking about mean rmse between all N folds validations. That averaged value may correlate with LB somehow. </p>",
      "rawMarkdown": "That's only single fold results - are you sure you're doing cross-validation? We say CV talking about mean rmse between all N folds validations. That averaged value may correlate with LB somehow.",
      "votes": null
    },
    {
      "id": "469184",
      "postDate": "02/10/2019 16:58:19",
      "content": "<p>this is similar to the fact that among the features (TOP10 features) are strongly correlated with the outlier. </p>",
      "rawMarkdown": "this is similar to the fact that among the features (TOP10 features) are strongly correlated with the outlier.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 468823,
      "author_name": "petersk20",
      "author_url": "",
      "post_date": "02/09/2019 18:59:54",
      "content": "<p>Everyone has a difference between their cv and lb score due to a greater number of outliers in the leader board set.\nI have a cv score of 3.65 and and lb score of 3.692, which the difference is still apparent but not as much as yours is.  I used 180 features for the model.  If I had to guess, I would say that you are training your model with few variables, which are predictive for the training cv set but not as predictive for the leader board set.</p>\n\n<p>If you want to better your leader board score, look at some kernels and copy some of their features.  Don't worry about closing the gap between the cv and lb score.  Just worry about bettering both of the scores with new features.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 468885,
      "author_name": "pukkinming",
      "author_url": "",
      "post_date": "02/10/2019 00:03:36",
      "content": "<p>what do u mean by no CV? I am not sure I am following. How do you divide your training/validation dataset?</p>",
      "votes": null,
      "replies": [
        {
          "id": 468920,
          "author_name": "saikumarallaka",
          "author_url": "",
          "post_date": "02/10/2019 03:31:00",
          "content": "<p>Well, I have divided the target variable in to 8 bins based on how far they are from mean, and picked equal proportion from each bin using train_test_split(80%,20%)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 469129,
      "author_name": "yaroshevskiy",
      "author_url": "",
      "post_date": "02/10/2019 14:59:16",
      "content": "<p>That's only single fold results - are you sure you're doing cross-validation? We say CV talking about mean rmse between all N folds validations. That averaged value may correlate with LB somehow. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 469184,
      "author_name": "jingobox",
      "author_url": "",
      "post_date": "02/10/2019 16:58:19",
      "content": "<p>this is similar to the fact that among the features (TOP10 features) are strongly correlated with the outlier. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "468817": "Hi Everyone\nBuild my first model with LGB and used stratified sampling to make sure outliers are picked in equal proportion.\n\nI have got this weird situation where my CV score is:\n**\n\n - Training until validation scores don't improve for 100 rounds.\n\n**\n\n[100]\tvalid_0's rmse: 3.54873   \n\n[200]\tvalid_0's rmse: 3.53051   \n\n[300]\tvalid_0's rmse: 3.529  \n\nEarly stopping, best iteration is:  \n\n[297]\tvalid_0's rmse: 3.5289  \n\nBut LB score is 3.75!!\n\nTotal features are around 160.\n\nDid you face similar situation, and what did you do to improve it?\nWhat could be possible reasons?",
    "468823": "Everyone has a difference between their cv and lb score due to a greater number of outliers in the leader board set.\nI have a cv score of 3.65 and and lb score of 3.692, which the difference is still apparent but not as much as yours is.  I used 180 features for the model.  If I had to guess, I would say that you are training your model with few variables, which are predictive for the training cv set but not as predictive for the leader board set.\n\nIf you want to better your leader board score, look at some kernels and copy some of their features.  Don't worry about closing the gap between the cv and lb score.  Just worry about bettering both of the scores with new features.",
    "468885": "what do u mean by no CV? I am not sure I am following. How do you divide your training/validation dataset?",
    "468920": "Well, I have divided the target variable in to 8 bins based on how far they are from mean, and picked equal proportion from each bin using train_test_split(80%,20%)",
    "469129": "That's only single fold results - are you sure you're doing cross-validation? We say CV talking about mean rmse between all N folds validations. That averaged value may correlate with LB somehow.",
    "469184": "this is similar to the fact that among the features (TOP10 features) are strongly correlated with the outlier."
  },
  "source": "meta"
}