{
  "id": 117684,
  "title": "No Over fitting but high LB Score 1.20",
  "url": "/competitions/ashrae-energy-prediction/discussion/117684",
  "author_name": "",
  "post_date": "2019-11-17T05:08:59.321224400Z",
  "votes": null,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Hello,</p>\n\n<p>I don't see any overfitting but the LB score is 1.20. </p>\n\n<p>Training until validation scores don't improve for 50 rounds\n[25]    training's rmse: 1.04314    valid_1's rmse: 1.04394</p>\n\n<p>[50]    training's rmse: 0.817494   valid_1's rmse: 0.819557</p>\n\n<p>[75]    training's rmse: 0.74556    valid_1's rmse: 0.74897</p>\n\n<p>[100]   training's rmse: 0.706463   valid_1's rmse: 0.711302</p>\n\n<p>[125]   training's rmse: 0.678724   valid_1's rmse: 0.685064</p>\n\n<p>[150]   training's rmse: 0.660915   valid_1's rmse: 0.668598</p>\n\n<p>[175]   training's rmse: 0.646805   valid_1's rmse: 0.655595</p>\n\n<p>[200]   training's rmse: 0.636699   valid_1's rmse: 0.646519</p>\n\n<p>[225]   training's rmse: 0.628841   valid_1's rmse: 0.639589</p>\n\n<p>[250]   training's rmse: 0.622664   valid_1's rmse: 0.634216</p>\n\n<p>[275]   training's rmse: 0.61795    valid_1's rmse: 0.630308</p>\n\n<p>[300]   training's rmse: 0.613978   valid_1's rmse: 0.627036</p>\n\n<p>[325]   training's rmse: 0.609793   valid_1's rmse: 0.623541</p>\n\n<p>[350]   training's rmse: 0.606176   valid_1's rmse: 0.620538</p>\n\n<p>[375]   training's rmse: 0.60238    valid_1's rmse: 0.617376</p>\n\n<p>[400]   training's rmse: 0.599028   valid_1's rmse: 0.614672</p>\n\n<p>[425]   training's rmse: 0.595896   valid_1's rmse: 0.612089</p>\n\n<p>[450]   training's rmse: 0.593072   valid_1's rmse: 0.60985</p>\n\n<p>[475]   training's rmse: 0.590796   valid_1's rmse: 0.608067</p>\n\n<p>[500]   training's rmse: 0.588505   valid_1's rmse: 0.606239</p>\n\n<p>What do you think?</p>\n\n<p>Thanks</p>",
  "messages": [
    {
      "id": "674805",
      "postDate": "11/17/2019 05:08:59",
      "content": "<p>Hello,</p>\n\n<p>I don't see any overfitting but the LB score is 1.20. </p>\n\n<p>Training until validation scores don't improve for 50 rounds\n[25]    training's rmse: 1.04314    valid_1's rmse: 1.04394</p>\n\n<p>[50]    training's rmse: 0.817494   valid_1's rmse: 0.819557</p>\n\n<p>[75]    training's rmse: 0.74556    valid_1's rmse: 0.74897</p>\n\n<p>[100]   training's rmse: 0.706463   valid_1's rmse: 0.711302</p>\n\n<p>[125]   training's rmse: 0.678724   valid_1's rmse: 0.685064</p>\n\n<p>[150]   training's rmse: 0.660915   valid_1's rmse: 0.668598</p>\n\n<p>[175]   training's rmse: 0.646805   valid_1's rmse: 0.655595</p>\n\n<p>[200]   training's rmse: 0.636699   valid_1's rmse: 0.646519</p>\n\n<p>[225]   training's rmse: 0.628841   valid_1's rmse: 0.639589</p>\n\n<p>[250]   training's rmse: 0.622664   valid_1's rmse: 0.634216</p>\n\n<p>[275]   training's rmse: 0.61795    valid_1's rmse: 0.630308</p>\n\n<p>[300]   training's rmse: 0.613978   valid_1's rmse: 0.627036</p>\n\n<p>[325]   training's rmse: 0.609793   valid_1's rmse: 0.623541</p>\n\n<p>[350]   training's rmse: 0.606176   valid_1's rmse: 0.620538</p>\n\n<p>[375]   training's rmse: 0.60238    valid_1's rmse: 0.617376</p>\n\n<p>[400]   training's rmse: 0.599028   valid_1's rmse: 0.614672</p>\n\n<p>[425]   training's rmse: 0.595896   valid_1's rmse: 0.612089</p>\n\n<p>[450]   training's rmse: 0.593072   valid_1's rmse: 0.60985</p>\n\n<p>[475]   training's rmse: 0.590796   valid_1's rmse: 0.608067</p>\n\n<p>[500]   training's rmse: 0.588505   valid_1's rmse: 0.606239</p>\n\n<p>What do you think?</p>\n\n<p>Thanks</p>",
      "rawMarkdown": "Hello,\n\nI don't see any overfitting but the LB score is 1.20. \n\nTraining until validation scores don't improve for 50 rounds\n[25]\ttraining's rmse: 1.04314\tvalid_1's rmse: 1.04394\n\n[50]\ttraining's rmse: 0.817494\tvalid_1's rmse: 0.819557\n\n[75]\ttraining's rmse: 0.74556\tvalid_1's rmse: 0.74897\n\n[100]\ttraining's rmse: 0.706463\tvalid_1's rmse: 0.711302\n\n[125]\ttraining's rmse: 0.678724\tvalid_1's rmse: 0.685064\n\n[150]\ttraining's rmse: 0.660915\tvalid_1's rmse: 0.668598\n\n[175]\ttraining's rmse: 0.646805\tvalid_1's rmse: 0.655595\n\n[200]\ttraining's rmse: 0.636699\tvalid_1's rmse: 0.646519\n\n[225]\ttraining's rmse: 0.628841\tvalid_1's rmse: 0.639589\n\n[250]\ttraining's rmse: 0.622664\tvalid_1's rmse: 0.634216\n\n[275]\ttraining's rmse: 0.61795\tvalid_1's rmse: 0.630308\n\n[300]\ttraining's rmse: 0.613978\tvalid_1's rmse: 0.627036\n\n[325]\ttraining's rmse: 0.609793\tvalid_1's rmse: 0.623541\n\n[350]\ttraining's rmse: 0.606176\tvalid_1's rmse: 0.620538\n\n[375]\ttraining's rmse: 0.60238\tvalid_1's rmse: 0.617376\n\n[400]\ttraining's rmse: 0.599028\tvalid_1's rmse: 0.614672\n\n[425]\ttraining's rmse: 0.595896\tvalid_1's rmse: 0.612089\n\n[450]\ttraining's rmse: 0.593072\tvalid_1's rmse: 0.60985\n\n[475]\ttraining's rmse: 0.590796\tvalid_1's rmse: 0.608067\n\n[500]\ttraining's rmse: 0.588505\tvalid_1's rmse: 0.606239\n\n\n\nWhat do you think?\n\nThanks",
      "votes": null
    },
    {
      "id": "674846",
      "postDate": "11/17/2019 06:26:36",
      "content": "<p>You have a leak</p>",
      "rawMarkdown": "You have a leak",
      "votes": null
    },
    {
      "id": "674853",
      "postDate": "11/17/2019 06:37:48",
      "content": "<p>Thanks. Can you please elaborate on it more? Any link to study will help.</p>",
      "rawMarkdown": "Thanks. Can you please elaborate on it more? Any link to study will help.",
      "votes": null
    },
    {
      "id": "675116",
      "postDate": "11/17/2019 16:08:24",
      "content": "<p>it means what your model learned on the training set is too closely connected to the hold-out and is not generally true, so it does not help you predict on the test set. The half-half kernel is an example of trying to make a cv that isn't \"closely connected\"</p>",
      "rawMarkdown": "it means what your model learned on the training set is too closely connected to the hold-out and is not generally true, so it does not help you predict on the test set. The half-half kernel is an example of trying to make a cv that isn't \"closely connected\"",
      "votes": null
    },
    {
      "id": "675164",
      "postDate": "11/17/2019 17:28:37",
      "content": "<p>I can give you a practical example: \nIf you split your data randomly, then it might occur that your models knows the meter_reading before and after a specific point in time. Thus it will be very easy for the model to predict this point! However, obviously this is no the case for the LB data, since your models does not know the meter reading before and after a specific point in time.</p>",
      "rawMarkdown": "I can give you a practical example: \nIf you split your data randomly, then it might occur that your models knows the meter_reading before and after a specific point in time. Thus it will be very easy for the model to predict this point! However, obviously this is no the case for the LB data, since your models does not know the meter reading before and after a specific point in time.",
      "votes": null
    },
    {
      "id": "675438",
      "postDate": "11/18/2019 04:53:27",
      "content": "<p>Thanks to <a href=\"/gunesevitan\">@gunesevitan</a> <a href=\"/abednadir\">@abednadir</a>  <a href=\"/elysii\">@elysii</a> </p>\n\n<p>I got the point now and it'll help me. </p>",
      "rawMarkdown": "Thanks to @gunesevitan @abednadir  @elysii \n\nI got the point now and it'll help me.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 674846,
      "author_name": "gunesevitan",
      "author_url": "",
      "post_date": "11/17/2019 06:26:36",
      "content": "<p>You have a leak</p>",
      "votes": null,
      "replies": [
        {
          "id": 674853,
          "author_name": "aitude",
          "author_url": "",
          "post_date": "11/17/2019 06:37:48",
          "content": "<p>Thanks. Can you please elaborate on it more? Any link to study will help.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 675116,
          "author_name": "abednadir",
          "author_url": "",
          "post_date": "11/17/2019 16:08:24",
          "content": "<p>it means what your model learned on the training set is too closely connected to the hold-out and is not generally true, so it does not help you predict on the test set. The half-half kernel is an example of trying to make a cv that isn't \"closely connected\"</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 675164,
          "author_name": "elysii",
          "author_url": "",
          "post_date": "11/17/2019 17:28:37",
          "content": "<p>I can give you a practical example: \nIf you split your data randomly, then it might occur that your models knows the meter_reading before and after a specific point in time. Thus it will be very easy for the model to predict this point! However, obviously this is no the case for the LB data, since your models does not know the meter reading before and after a specific point in time.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 675438,
      "author_name": "aitude",
      "author_url": "",
      "post_date": "11/18/2019 04:53:27",
      "content": "<p>Thanks to <a href=\"/gunesevitan\">@gunesevitan</a> <a href=\"/abednadir\">@abednadir</a>  <a href=\"/elysii\">@elysii</a> </p>\n\n<p>I got the point now and it'll help me. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "674805": "Hello,\n\nI don't see any overfitting but the LB score is 1.20. \n\nTraining until validation scores don't improve for 50 rounds\n[25]\ttraining's rmse: 1.04314\tvalid_1's rmse: 1.04394\n\n[50]\ttraining's rmse: 0.817494\tvalid_1's rmse: 0.819557\n\n[75]\ttraining's rmse: 0.74556\tvalid_1's rmse: 0.74897\n\n[100]\ttraining's rmse: 0.706463\tvalid_1's rmse: 0.711302\n\n[125]\ttraining's rmse: 0.678724\tvalid_1's rmse: 0.685064\n\n[150]\ttraining's rmse: 0.660915\tvalid_1's rmse: 0.668598\n\n[175]\ttraining's rmse: 0.646805\tvalid_1's rmse: 0.655595\n\n[200]\ttraining's rmse: 0.636699\tvalid_1's rmse: 0.646519\n\n[225]\ttraining's rmse: 0.628841\tvalid_1's rmse: 0.639589\n\n[250]\ttraining's rmse: 0.622664\tvalid_1's rmse: 0.634216\n\n[275]\ttraining's rmse: 0.61795\tvalid_1's rmse: 0.630308\n\n[300]\ttraining's rmse: 0.613978\tvalid_1's rmse: 0.627036\n\n[325]\ttraining's rmse: 0.609793\tvalid_1's rmse: 0.623541\n\n[350]\ttraining's rmse: 0.606176\tvalid_1's rmse: 0.620538\n\n[375]\ttraining's rmse: 0.60238\tvalid_1's rmse: 0.617376\n\n[400]\ttraining's rmse: 0.599028\tvalid_1's rmse: 0.614672\n\n[425]\ttraining's rmse: 0.595896\tvalid_1's rmse: 0.612089\n\n[450]\ttraining's rmse: 0.593072\tvalid_1's rmse: 0.60985\n\n[475]\ttraining's rmse: 0.590796\tvalid_1's rmse: 0.608067\n\n[500]\ttraining's rmse: 0.588505\tvalid_1's rmse: 0.606239\n\n\n\nWhat do you think?\n\nThanks",
    "674846": "You have a leak",
    "674853": "Thanks. Can you please elaborate on it more? Any link to study will help.",
    "675116": "it means what your model learned on the training set is too closely connected to the hold-out and is not generally true, so it does not help you predict on the test set. The half-half kernel is an example of trying to make a cv that isn't \"closely connected\"",
    "675164": "I can give you a practical example: \nIf you split your data randomly, then it might occur that your models knows the meter_reading before and after a specific point in time. Thus it will be very easy for the model to predict this point! However, obviously this is no the case for the LB data, since your models does not know the meter reading before and after a specific point in time.",
    "675438": "Thanks to @gunesevitan @abednadir  @elysii \n\nI got the point now and it'll help me."
  },
  "source": "meta"
}