{
  "id": 11272,
  "title": "Coming to terms with overfitting?",
  "url": "/competitions/avazu-ctr-prediction/discussion/11272",
  "author_name": "",
  "post_date": "2014-12-16T19:41:39.477Z",
  "votes": 2,
  "comment_count": 5,
  "views": 1824,
  "content": "<p>Hello everyone,</p>\n<p>I would like to understand how are you guys ensuring you are not overfitting to the leaderboard? I am unable to arrive at a reliable validation framework. If you are resorting to cross-validation, are you making sure you generalize well on the remaining 80% of the data. I doubt that we might have a LB shakeup if we overfit like what we had in African Soil problem.</p>\n<p>Another quick question: Do you think progressive validation loss is a reliable estimate of the error? In an online setting, literature mentions that progressive validation loss is a very good estimate, but none of the Kaggle forums in this competition seem to talk about it.</p>\n<p>Spare my ignorance. Happy to learn from the fantastic peer group here!</p>",
  "messages": [
    {
      "id": "60386",
      "postDate": "12/16/2014 19:41:39",
      "content": "<p>Hello everyone,</p>\n<p>I would like to understand how are you guys ensuring you are not overfitting to the leaderboard? I am unable to arrive at a reliable validation framework. If you are resorting to cross-validation, are you making sure you generalize well on the remaining 80% of the data. I doubt that we might have a LB shakeup if we overfit like what we had in African Soil problem.</p>\n<p>Another quick question: Do you think progressive validation loss is a reliable estimate of the error? In an online setting, literature mentions that progressive validation loss is a very good estimate, but none of the Kaggle forums in this competition seem to talk about it.</p>\n<p>Spare my ignorance. Happy to learn from the fantastic peer group here!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60411",
      "postDate": "12/17/2014 00:08:22",
      "content": "<p>Hi Binga, I may be a few&nbsp;steps behind you in terms of how sophisticated my validation scheme is, but one thing I do is set aside my own holdout set and then compare its logloss to the score on the public leaderboard. &nbsp;I think the difference between those two numbers should some kind of feel for reliability of the score.</p>\n<p>Also, thanks for mentioning progressive validation loss, I think I'll look that up.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60446",
      "postDate": "12/17/2014 08:57:33",
      "content": "<p>As I understand it, progressive loss would be reliable but:</p>\n<p>- it can't be used if you are running more than one pass/epoch</p>\n<p>- it doesn't let you test against a 'brand new day' which should be what we want to test generalisation</p>\n<p>So maybe it's best to test against one, two or more holdout days. I'd still be interested to hear other views on the overfitting question.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60466",
      "postDate": "12/17/2014 11:33:51",
      "content": "<p>I really hope I could overfit to 0.38xx. if so, at least i'm feeling good for now :P</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60471",
      "postDate": "12/17/2014 12:33:31",
      "content": "<p>[quote=lewis ml;60446]</p>\n<p>As I understand it, progressive loss would be reliable but:</p>\n<p>- it can't be used if you are running more than one pass/epoch</p>\n<p>- it doesn't let you test against a 'brand new day' which should be what we want to test generalisation</p>\n<p>So maybe it's best to test against one, two or more holdout days. I'd still be interested to hear other views on the overfitting question.</p>\n<p>[/quote]</p>\n<p>Totally agree on this point. Testing on a brand new day is not trivial by progressive validation loss. We'll have to come up with a new strategy.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "60472",
      "postDate": "12/17/2014 12:36:58",
      "content": "<p>[quote=rcarson;60466]</p>\n<p>I really hope I could overfit to 0.38xx. if so, at least i'm feeling good for now :P</p>\n<p>[/quote]</p>\n<p>Looking at the leaders in the 0.387 zone, I guess you are happy being persuaded to chase 0.38xx. =)</p>\n<p>There's a lot of time in the competition. Let us see where this goes!</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 60411,
      "author_name": "dlsiii",
      "author_url": "",
      "post_date": "12/17/2014 00:08:22",
      "content": "<p>Hi Binga, I may be a few&nbsp;steps behind you in terms of how sophisticated my validation scheme is, but one thing I do is set aside my own holdout set and then compare its logloss to the score on the public leaderboard. &nbsp;I think the difference between those two numbers should some kind of feel for reliability of the score.</p>\n<p>Also, thanks for mentioning progressive validation loss, I think I'll look that up.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60446,
      "author_name": "cartographic",
      "author_url": "",
      "post_date": "12/17/2014 08:57:33",
      "content": "<p>As I understand it, progressive loss would be reliable but:</p>\n<p>- it can't be used if you are running more than one pass/epoch</p>\n<p>- it doesn't let you test against a 'brand new day' which should be what we want to test generalisation</p>\n<p>So maybe it's best to test against one, two or more holdout days. I'd still be interested to hear other views on the overfitting question.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60466,
      "author_name": "jiweiliu",
      "author_url": "",
      "post_date": "12/17/2014 11:33:51",
      "content": "<p>I really hope I could overfit to 0.38xx. if so, at least i'm feeling good for now :P</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60471,
      "author_name": "phanisrikanth",
      "author_url": "",
      "post_date": "12/17/2014 12:33:31",
      "content": "<p>[quote=lewis ml;60446]</p>\n<p>As I understand it, progressive loss would be reliable but:</p>\n<p>- it can't be used if you are running more than one pass/epoch</p>\n<p>- it doesn't let you test against a 'brand new day' which should be what we want to test generalisation</p>\n<p>So maybe it's best to test against one, two or more holdout days. I'd still be interested to hear other views on the overfitting question.</p>\n<p>[/quote]</p>\n<p>Totally agree on this point. Testing on a brand new day is not trivial by progressive validation loss. We'll have to come up with a new strategy.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 60472,
      "author_name": "phanisrikanth",
      "author_url": "",
      "post_date": "12/17/2014 12:36:58",
      "content": "<p>[quote=rcarson;60466]</p>\n<p>I really hope I could overfit to 0.38xx. if so, at least i'm feeling good for now :P</p>\n<p>[/quote]</p>\n<p>Looking at the leaders in the 0.387 zone, I guess you are happy being persuaded to chase 0.38xx. =)</p>\n<p>There's a lot of time in the competition. Let us see where this goes!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "60386": "",
    "60411": "",
    "60446": "",
    "60466": "",
    "60471": "",
    "60472": ""
  },
  "source": "meta"
}