{
  "id": 75578,
  "title": "Should I trust CV?",
  "url": "/competitions/elo-merchant-category-recommendation/discussion/75578",
  "author_name": "",
  "post_date": "2018-12-23T16:00:40.042647600Z",
  "votes": 2,
  "comment_count": 5,
  "views": 0,
  "content": "<p><a href=\"https://i.loli.net/2018/12/23/5c1fb05973c66.png\"><img src=\"https://i.loli.net/2018/12/23/5c1fb05973c66.png\" alt=\"WX20181223-234814@2x.png\"></a></p>\n\n<p>When I do some feature selection, cv rmse decreased from 3.64357 to 3.64102, but LB rmse increased from 3.691 to 3.695  :( All experiment used same random seed.  Should I trust CV?  </p>\n\n<p>And some other questions: </p>\n\n<ul>\n<li>How to find the distribution gap between the train features and test features? </li>\n<li>How to deal with the gap of cv and LB?</li>\n</ul>\n\n<p>Any advice? Many thanks!</p>",
  "messages": [
    {
      "id": "444235",
      "postDate": "12/23/2018 16:00:40",
      "content": "<p><a href=\"https://i.loli.net/2018/12/23/5c1fb05973c66.png\"><img src=\"https://i.loli.net/2018/12/23/5c1fb05973c66.png\" alt=\"WX20181223-234814@2x.png\"></a></p>\n\n<p>When I do some feature selection, cv rmse decreased from 3.64357 to 3.64102, but LB rmse increased from 3.691 to 3.695  :( All experiment used same random seed.  Should I trust CV?  </p>\n\n<p>And some other questions: </p>\n\n<ul>\n<li>How to find the distribution gap between the train features and test features? </li>\n<li>How to deal with the gap of cv and LB?</li>\n</ul>\n\n<p>Any advice? Many thanks!</p>",
      "rawMarkdown": "[![WX20181223-234814@2x.png](https://i.loli.net/2018/12/23/5c1fb05973c66.png)](https://i.loli.net/2018/12/23/5c1fb05973c66.png)\n\nWhen I do some feature selection, cv rmse decreased from 3.64357 to 3.64102, but LB rmse increased from 3.691 to 3.695  :( All experiment used same random seed.  Should I trust CV?  \n\nAnd some other questions: \n\n- How to find the distribution gap between the train features and test features? \n- How to deal with the gap of cv and LB?\n\nAny advice? Many thanks!",
      "votes": null
    },
    {
      "id": "444240",
      "postDate": "12/23/2018 16:16:39",
      "content": "<p>Turn the question the other way,  I would definitely NOT trust LB score by itself.\nPersonally, I would keep the features set that give you CV boost - as long as you can be confident it is not overfitting - and develop it as a separate branch and see how far you can push CV down.  I would also switch to a different random seed to see if the same feature set would give you the same boost. </p>",
      "rawMarkdown": "Turn the question the other way,  I would definitely NOT trust LB score by itself.\nPersonally, I would keep the features set that give you CV boost - as long as you can be confident it is not overfitting - and develop it as a separate branch and see how far you can push CV down.  I would also switch to a different random seed to see if the same feature set would give you the same boost.",
      "votes": null
    },
    {
      "id": "444411",
      "postDate": "12/24/2018 01:49:09",
      "content": "<p>I'm really struggling with this too.</p>\n\n<p>&gt; Should I trust CV?</p>\n\n<p>Generally, yes. I find it hard to imagine that having <em>fewer</em> features would lead to overfit.</p>\n\n<p>&gt; How to find the distribution gap between the train features and test features?</p>\n\n<p>Maybe adversarial validation?</p>\n\n<p>&gt; How to deal with the gap of cv and LB?</p>\n\n<p>I think this is key. Maybe it is noise? If so, we can ignore it and focus on CV. Maybe it is indicative of some difference in the test set? If so, we must worry. Training an outlier model could be helpful.</p>",
      "rawMarkdown": "I'm really struggling with this too.\n\n&gt; Should I trust CV?\n\nGenerally, yes. I find it hard to imagine that having *fewer* features would lead to overfit.\n\n&gt; How to find the distribution gap between the train features and test features?\n\nMaybe adversarial validation?\n\n&gt; How to deal with the gap of cv and LB?\n\nI think this is key. Maybe it is noise? If so, we can ignore it and focus on CV. Maybe it is indicative of some difference in the test set? If so, we must worry. Training an outlier model could be helpful.",
      "votes": null
    },
    {
      "id": "444422",
      "postDate": "12/24/2018 02:32:05",
      "content": "<p>I selected the features with the same random seed when training the model, maybe I was overfit to the valid set..... <code>switch to a different random seed to see if the same feature set would give you the same boost.</code> is a cool idea, I will try that. Many thanks!</p>",
      "rawMarkdown": "I selected the features with the same random seed when training the model, maybe I was overfit to the valid set..... `switch to a different random seed to see if the same feature set would give you the same boost.` is a cool idea, I will try that. Many thanks!",
      "votes": null
    },
    {
      "id": "444480",
      "postDate": "12/24/2018 05:02:11",
      "content": "<p>Also, it could be awesome to try taking an average of the two models.</p>",
      "rawMarkdown": "Also, it could be awesome to try taking an average of the two models.",
      "votes": null
    },
    {
      "id": "475692",
      "postDate": "02/21/2019 04:31:03",
      "content": "<p>u mean running the same model with different seeds and then get the average? That's brilliant. </p>",
      "rawMarkdown": "u mean running the same model with different seeds and then get the average? That's brilliant.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 444240,
      "author_name": "yifanxie",
      "author_url": "",
      "post_date": "12/23/2018 16:16:39",
      "content": "<p>Turn the question the other way,  I would definitely NOT trust LB score by itself.\nPersonally, I would keep the features set that give you CV boost - as long as you can be confident it is not overfitting - and develop it as a separate branch and see how far you can push CV down.  I would also switch to a different random seed to see if the same feature set would give you the same boost. </p>",
      "votes": null,
      "replies": [
        {
          "id": 444422,
          "author_name": "jacksonler",
          "author_url": "",
          "post_date": "12/24/2018 02:32:05",
          "content": "<p>I selected the features with the same random seed when training the model, maybe I was overfit to the valid set..... <code>switch to a different random seed to see if the same feature set would give you the same boost.</code> is a cool idea, I will try that. Many thanks!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 444411,
      "author_name": "peterhurford",
      "author_url": "",
      "post_date": "12/24/2018 01:49:09",
      "content": "<p>I'm really struggling with this too.</p>\n\n<p>&gt; Should I trust CV?</p>\n\n<p>Generally, yes. I find it hard to imagine that having <em>fewer</em> features would lead to overfit.</p>\n\n<p>&gt; How to find the distribution gap between the train features and test features?</p>\n\n<p>Maybe adversarial validation?</p>\n\n<p>&gt; How to deal with the gap of cv and LB?</p>\n\n<p>I think this is key. Maybe it is noise? If so, we can ignore it and focus on CV. Maybe it is indicative of some difference in the test set? If so, we must worry. Training an outlier model could be helpful.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 444480,
      "author_name": "peterhurford",
      "author_url": "",
      "post_date": "12/24/2018 05:02:11",
      "content": "<p>Also, it could be awesome to try taking an average of the two models.</p>",
      "votes": null,
      "replies": [
        {
          "id": 475692,
          "author_name": "pukkinming",
          "author_url": "",
          "post_date": "02/21/2019 04:31:03",
          "content": "<p>u mean running the same model with different seeds and then get the average? That's brilliant. </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "444235": "[![WX20181223-234814@2x.png](https://i.loli.net/2018/12/23/5c1fb05973c66.png)](https://i.loli.net/2018/12/23/5c1fb05973c66.png)\n\nWhen I do some feature selection, cv rmse decreased from 3.64357 to 3.64102, but LB rmse increased from 3.691 to 3.695  :( All experiment used same random seed.  Should I trust CV?  \n\nAnd some other questions: \n\n- How to find the distribution gap between the train features and test features? \n- How to deal with the gap of cv and LB?\n\nAny advice? Many thanks!",
    "444240": "Turn the question the other way,  I would definitely NOT trust LB score by itself.\nPersonally, I would keep the features set that give you CV boost - as long as you can be confident it is not overfitting - and develop it as a separate branch and see how far you can push CV down.  I would also switch to a different random seed to see if the same feature set would give you the same boost.",
    "444411": "I'm really struggling with this too.\n\n&gt; Should I trust CV?\n\nGenerally, yes. I find it hard to imagine that having *fewer* features would lead to overfit.\n\n&gt; How to find the distribution gap between the train features and test features?\n\nMaybe adversarial validation?\n\n&gt; How to deal with the gap of cv and LB?\n\nI think this is key. Maybe it is noise? If so, we can ignore it and focus on CV. Maybe it is indicative of some difference in the test set? If so, we must worry. Training an outlier model could be helpful.",
    "444422": "I selected the features with the same random seed when training the model, maybe I was overfit to the valid set..... `switch to a different random seed to see if the same feature set would give you the same boost.` is a cool idea, I will try that. Many thanks!",
    "444480": "Also, it could be awesome to try taking an average of the two models.",
    "475692": "u mean running the same model with different seeds and then get the average? That's brilliant."
  },
  "source": "meta"
}