{
  "id": 113249,
  "title": "Be careful of overfittting",
  "url": "/competitions/ashrae-energy-prediction/discussion/113249",
  "author_name": "",
  "post_date": "2019-10-18T03:06:45.624479300Z",
  "votes": 61,
  "comment_count": 14,
  "views": 0,
  "content": "<p>Here are a few notes that may help new kagglers to prevent  overffiting.</p>\n\n<ol>\n<li><p>Since it's a timeseries problem,so you would be better to split train and validation set by timestamp , sometimes kfold is also can be used for timeseries problem but it's better to  check first,you can split train and validation set by timestamp, then train a kfold-model using train set to watch the validation set performance.</p></li>\n<li><p>Don't use features only exsited in train but not in test,for example year, time-lagged targetencoding...also don't use features that very different between train and public/private test,like a feature has no/less null values in train but lots of null values  in test, or public test has no/less null but private test has a lot.</p></li>\n<li><p>Since the test time period is longer than train,it's hard to setup a robust validation time period,you will find parameters especially earlystop will have a big impact to leaderboard score,because it will overfit to local validation set but very different from test set. I suggested to try to adjust these parameters <code>max_bin, num_leaves, bagging_fraction, feature_fraction, lambda_l1, lambda_l2, max_depth</code> to deal with overfitting,and try to setup different timesplit validation to watch the change of earlystop round,At last can use a optimized round number to stop the training.</p></li>\n<li><p>Don't trust public leaderboard too much ,because public and private are also splited by time.You can split your train data to simulate train/valid/public/private, make sure your model can improve all of them.</p></li>\n<li><p>Good luck and have fun!</p></li>\n</ol>",
  "messages": [
    {
      "id": "651870",
      "postDate": "10/18/2019 03:06:45",
      "content": "<p>Here are a few notes that may help new kagglers to prevent  overffiting.</p>\n\n<ol>\n<li><p>Since it's a timeseries problem,so you would be better to split train and validation set by timestamp , sometimes kfold is also can be used for timeseries problem but it's better to  check first,you can split train and validation set by timestamp, then train a kfold-model using train set to watch the validation set performance.</p></li>\n<li><p>Don't use features only exsited in train but not in test,for example year, time-lagged targetencoding...also don't use features that very different between train and public/private test,like a feature has no/less null values in train but lots of null values  in test, or public test has no/less null but private test has a lot.</p></li>\n<li><p>Since the test time period is longer than train,it's hard to setup a robust validation time period,you will find parameters especially earlystop will have a big impact to leaderboard score,because it will overfit to local validation set but very different from test set. I suggested to try to adjust these parameters <code>max_bin, num_leaves, bagging_fraction, feature_fraction, lambda_l1, lambda_l2, max_depth</code> to deal with overfitting,and try to setup different timesplit validation to watch the change of earlystop round,At last can use a optimized round number to stop the training.</p></li>\n<li><p>Don't trust public leaderboard too much ,because public and private are also splited by time.You can split your train data to simulate train/valid/public/private, make sure your model can improve all of them.</p></li>\n<li><p>Good luck and have fun!</p></li>\n</ol>",
      "rawMarkdown": "Here are a few notes that may help new kagglers to prevent  overffiting.\n\n1. Since it's a timeseries problem,so you would be better to split train and validation set by timestamp , sometimes kfold is also can be used for timeseries problem but it's better to  check first,you can split train and validation set by timestamp, then train a kfold-model using train set to watch the validation set performance.\n\n2. Don't use features only exsited in train but not in test,for example year, time-lagged targetencoding...also don't use features that very different between train and public/private test,like a feature has no/less null values in train but lots of null values  in test, or public test has no/less null but private test has a lot.\n\n3. Since the test time period is longer than train,it's hard to setup a robust validation time period,you will find parameters especially earlystop will have a big impact to leaderboard score,because it will overfit to local validation set but very different from test set. I suggested to try to adjust these parameters ```max_bin, num_leaves, bagging_fraction, feature_fraction, lambda_l1, lambda_l2, max_depth``` to deal with overfitting,and try to setup different timesplit validation to watch the change of earlystop round,At last can use a optimized round number to stop the training.\n\n4. Don't trust public leaderboard too much ,because public and private are also splited by time.You can split your train data to simulate train/valid/public/private, make sure your model can improve all of them.\n\n5. Good luck and have fun!",
      "votes": null
    },
    {
      "id": "651877",
      "postDate": "10/18/2019 03:17:45",
      "content": "<p>Thanks for sharing! <a href=\"/senkin13\">@senkin13</a> </p>\n\n<p>To anyone that doesn't know how to do a time series split, you can implement the time series split as shown in:\n<a href=\"https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html\">https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html</a></p>\n\n<p><img src=\"https://scikit-learn.org/stable/_images/sphx_glr_plot_cv_indices_010.png\" alt=\"\"></p>",
      "rawMarkdown": "Thanks for sharing! @senkin13 \n\nTo anyone that doesn't know how to do a time series split, you can implement the time series split as shown in:\nhttps://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html\n\n![](https://scikit-learn.org/stable/_images/sphx_glr_plot_cv_indices_010.png)",
      "votes": null
    },
    {
      "id": "651947",
      "postDate": "10/18/2019 06:00:43",
      "content": "<p>Very Informative Share.. Thanks <a href=\"/senkin13\">@senkin13</a>   <a href=\"/tiannnn\">@tiannnn</a> </p>",
      "rawMarkdown": "Very Informative Share.. Thanks @senkin13   @tiannnn",
      "votes": null
    },
    {
      "id": "652033",
      "postDate": "10/18/2019 08:47:20",
      "content": "<p>Hi <a href=\"/senkin13\">@senkin13</a>, I was wondering about that. For this competition the train set is for the year 2016 and the test set for the years 2017 and 2018. If we do a split by timestamp, for example, train on the first 10 months and validate on the last two months, wouldn't this deviate from the \"prediction reality\"? </p>\n\n<p>What I mean is that during validation we can loose the effect of seasonality features. For example, the \"month\" feature will be useless since we will never predict for a month that we trained on, whilst when we predict on the test set this information can be quite important. My experiments with <a href=\"https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html\">KFold</a> and <a href=\"https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html\">TimeSeriesSplit</a> shows that the former underfits and the latter overfits.</p>\n\n<p>Any ideas on how to approach cross validation for this problem?</p>",
      "rawMarkdown": "Hi @senkin13, I was wondering about that. For this competition the train set is for the year 2016 and the test set for the years 2017 and 2018. If we do a split by timestamp, for example, train on the first 10 months and validate on the last two months, wouldn't this deviate from the \"prediction reality\"? \n\nWhat I mean is that during validation we can loose the effect of seasonality features. For example, the \"month\" feature will be useless since we will never predict for a month that we trained on, whilst when we predict on the test set this information can be quite important. My experiments with [KFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html) and [TimeSeriesSplit](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html) shows that the former underfits and the latter overfits.\n\nAny ideas on how to approach cross validation for this problem?",
      "votes": null
    },
    {
      "id": "652102",
      "postDate": "10/18/2019 11:03:12",
      "content": "<blockquote>\n  <p>Don't trust public leaderboard too much ,because public and private are also splited by time.</p>\n</blockquote>\n\n<p><a href=\"/senkin13\">@senkin13</a> do you have any evidence of it? I did split lb probe and seems it's random split. What makes you think it's timebased split? </p>",
      "rawMarkdown": "&gt; Don't trust public leaderboard too much ,because public and private are also splited by time.\n\n@senkin13 do you have any evidence of it? I did split lb probe and seems it's random split. What makes you think it's timebased split?",
      "votes": null
    },
    {
      "id": "652109",
      "postDate": "10/18/2019 11:17:27",
      "content": "<p>wow <a href=\"/kyakovlev\">@kyakovlev</a> I'd like to see that. I think <a href=\"/inversion\">@inversion</a> said:</p>\n\n<blockquote>\n  <p>We always split time-based data by time for Train/Test, and Public/Private. If we didn't do that, the Public leaderboard score would provide feedback (leakage) for future time points.</p>\n</blockquote>\n\n<p>or at least I remember that someone posted that.</p>",
      "rawMarkdown": "wow @kyakovlev I'd like to see that. I think @inversion said:\n&gt; We always split time-based data by time for Train/Test, and Public/Private. If we didn't do that, the Public leaderboard score would provide feedback (leakage) for future time points.\n\nor at least I remember that someone posted that.",
      "votes": null
    },
    {
      "id": "652110",
      "postDate": "10/18/2019 11:20:37",
      "content": "<p>sorry for posting link to other discussion</p>\n\n<p><a href=\"https://www.kaggle.com/c/ashrae-energy-prediction/discussion/112872#651685\">https://www.kaggle.com/c/ashrae-energy-prediction/discussion/112872#651685</a></p>",
      "rawMarkdown": "sorry for posting link to other discussion\n\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/112872#651685",
      "votes": null
    },
    {
      "id": "652123",
      "postDate": "10/18/2019 11:50:15",
      "content": "<p><a href=\"/kyakovlev\">@kyakovlev</a> Thanks for your probing, I think random split will lead to leaderboard leakage,so it's possible as you mentioned that 22% of first lines are private, or public and private are mixed with different month/week...</p>",
      "rawMarkdown": "kyakovlev Thanks for your probing, I think random split will lead to leaderboard leakage,so it's possible as you mentioned that 22% of first lines are private, or public and private are mixed with different month/week...",
      "votes": null
    },
    {
      "id": "652130",
      "postDate": "10/18/2019 11:57:06",
      "content": "<p>Some mix should be. I'm too dumb in stats to prove it knowing lb score. </p>\n\n<p>But I think it's possible to make calculation if you know lb score before and lb after excluding 22% last test data.</p>\n\n<p>What we have:\n* All 0 predictions -&gt; 4.69\n* Baseline - &gt; 1.36\n* Excluding 22% -&gt; 2.6</p>\n\n<p>We need to calculate if 22% of 0s corresponds 1.24 RMSLE difference. If so - &gt; random split, if not possible mix.</p>",
      "rawMarkdown": "Some mix should be. I'm too dumb in stats to prove it knowing lb score. \n\nBut I think it's possible to make calculation if you know lb score before and lb after excluding 22% last test data.\n\nWhat we have:\n* All 0 predictions -&gt; 4.69\n* Baseline - &gt; 1.36\n* Excluding 22% -&gt; 2.6\n\nWe need to calculate if 22% of 0s corresponds 1.24 RMSLE difference. If so - &gt; random split, if not possible mix.",
      "votes": null
    },
    {
      "id": "652140",
      "postDate": "10/18/2019 12:08:55",
      "content": "<p>I agree with your viewpoint,since I have not worked deeply for competition,just talk one simple idea.\n The purpose to find a robust validation is for finding robust features,so you can create many features then use different cross validation to check which are robust,including kfold/timeseriessplit...</p>",
      "rawMarkdown": "I agree with your viewpoint,since I have not worked deeply for competition,just talk one simple idea.\n The purpose to find a robust validation is for finding robust features,so you can create many features then use different cross validation to check which are robust,including kfold/timeseriessplit...",
      "votes": null
    },
    {
      "id": "652493",
      "postDate": "10/18/2019 23:24:54",
      "content": "<p>IF it was I setting up the test split I would do it by building.  If time split like past Kaggle series, than 22% of two years would just be the first 6 months of 2017.</p>\n\n<p>If done by building than the public test could still contain the full two years.</p>\n\n<p>Assume that shortly someone will do the LB probe that lets us know.</p>",
      "rawMarkdown": "IF it was I setting up the test split I would do it by building.  If time split like past Kaggle series, than 22% of two years would just be the first 6 months of 2017.\n\nIf done by building than the public test could still contain the full two years.\n\nAssume that shortly someone will do the LB probe that lets us know.",
      "votes": null
    },
    {
      "id": "652965",
      "postDate": "10/19/2019 18:13:08",
      "content": "<p>4.69^2 = 22.00\n1.36^2 = 1.85\n2.60^2 = 6.76</p>\n\n<p>(6.76 - 1.85)/(22.0 - 1.85) = 0.24</p>\n\n<p>Given that last months are winter, 0.24 being a bit higher than 0.22 makes sense. So the split is probably not 78/22 by time.</p>",
      "rawMarkdown": "4.69^2 = 22.00\n1.36^2 = 1.85\n2.60^2 = 6.76\n\n(6.76 - 1.85)/(22.0 - 1.85) = 0.24\n\nGiven that last months are winter, 0.24 being a bit higher than 0.22 makes sense. So the split is probably not 78/22 by time.",
      "votes": null
    },
    {
      "id": "658145",
      "postDate": "10/25/2019 17:53:36",
      "content": "<p>Thank you for the advice!</p>",
      "rawMarkdown": "Thank you for the advice!",
      "votes": null
    },
    {
      "id": "659683",
      "postDate": "10/28/2019 04:45:11",
      "content": "<p>For anyone from the future, as of 28th Oct 2019, <a href=\"/geoffrey7697\">@geoffrey7697</a> probed the LB and identified that data between August 2018 to Dec 2018 is part of the Private LB. In fact there are evidences that it could be even more (may include July 2018 as well):</p>\n\n<p><a href=\"https://www.kaggle.com/c/ashrae-energy-prediction/discussion/113751#latest-659225\">https://www.kaggle.com/c/ashrae-energy-prediction/discussion/113751#latest-659225</a></p>",
      "rawMarkdown": "For anyone from the future, as of 28th Oct 2019, @geoffrey7697 probed the LB and identified that data between August 2018 to Dec 2018 is part of the Private LB. In fact there are evidences that it could be even more (may include July 2018 as well):\n\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/113751#latest-659225",
      "votes": null
    },
    {
      "id": "659782",
      "postDate": "10/28/2019 08:56:51",
      "content": "<p>Since Public LeaderBoard is much larger (1.5 years) compared to Train (1 year) or Private Leader Board (Possibly 0.5 years), should not we give value to the Public LB score along with Local CV score? Otherwise, it's fairly difficult to understand if the features capturing long term trends are helping or not. I suspect that this is the reason that Kaggle has limited the number of daily submission limit to 2. </p>",
      "rawMarkdown": "Since Public LeaderBoard is much larger (1.5 years) compared to Train (1 year) or Private Leader Board (Possibly 0.5 years), should not we give value to the Public LB score along with Local CV score? Otherwise, it's fairly difficult to understand if the features capturing long term trends are helping or not. I suspect that this is the reason that Kaggle has limited the number of daily submission limit to 2.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 651877,
      "author_name": "tiannnn",
      "author_url": "",
      "post_date": "10/18/2019 03:17:45",
      "content": "<p>Thanks for sharing! <a href=\"/senkin13\">@senkin13</a> </p>\n\n<p>To anyone that doesn't know how to do a time series split, you can implement the time series split as shown in:\n<a href=\"https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html\">https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html</a></p>\n\n<p><img src=\"https://scikit-learn.org/stable/_images/sphx_glr_plot_cv_indices_010.png\" alt=\"\"></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 651947,
      "author_name": "veeralakrishna",
      "author_url": "",
      "post_date": "10/18/2019 06:00:43",
      "content": "<p>Very Informative Share.. Thanks <a href=\"/senkin13\">@senkin13</a>   <a href=\"/tiannnn\">@tiannnn</a> </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 652033,
      "author_name": "pedromb",
      "author_url": "",
      "post_date": "10/18/2019 08:47:20",
      "content": "<p>Hi <a href=\"/senkin13\">@senkin13</a>, I was wondering about that. For this competition the train set is for the year 2016 and the test set for the years 2017 and 2018. If we do a split by timestamp, for example, train on the first 10 months and validate on the last two months, wouldn't this deviate from the \"prediction reality\"? </p>\n\n<p>What I mean is that during validation we can loose the effect of seasonality features. For example, the \"month\" feature will be useless since we will never predict for a month that we trained on, whilst when we predict on the test set this information can be quite important. My experiments with <a href=\"https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html\">KFold</a> and <a href=\"https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html\">TimeSeriesSplit</a> shows that the former underfits and the latter overfits.</p>\n\n<p>Any ideas on how to approach cross validation for this problem?</p>",
      "votes": null,
      "replies": [
        {
          "id": 652140,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "10/18/2019 12:08:55",
          "content": "<p>I agree with your viewpoint,since I have not worked deeply for competition,just talk one simple idea.\n The purpose to find a robust validation is for finding robust features,so you can create many features then use different cross validation to check which are robust,including kfold/timeseriessplit...</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 652102,
      "author_name": "kyakovlev",
      "author_url": "",
      "post_date": "10/18/2019 11:03:12",
      "content": "<blockquote>\n  <p>Don't trust public leaderboard too much ,because public and private are also splited by time.</p>\n</blockquote>\n\n<p><a href=\"/senkin13\">@senkin13</a> do you have any evidence of it? I did split lb probe and seems it's random split. What makes you think it's timebased split? </p>",
      "votes": null,
      "replies": [
        {
          "id": 652109,
          "author_name": "jesucristo",
          "author_url": "",
          "post_date": "10/18/2019 11:17:27",
          "content": "<p>wow <a href=\"/kyakovlev\">@kyakovlev</a> I'd like to see that. I think <a href=\"/inversion\">@inversion</a> said:</p>\n\n<blockquote>\n  <p>We always split time-based data by time for Train/Test, and Public/Private. If we didn't do that, the Public leaderboard score would provide feedback (leakage) for future time points.</p>\n</blockquote>\n\n<p>or at least I remember that someone posted that.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 652110,
          "author_name": "kyakovlev",
          "author_url": "",
          "post_date": "10/18/2019 11:20:37",
          "content": "<p>sorry for posting link to other discussion</p>\n\n<p><a href=\"https://www.kaggle.com/c/ashrae-energy-prediction/discussion/112872#651685\">https://www.kaggle.com/c/ashrae-energy-prediction/discussion/112872#651685</a></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 652123,
          "author_name": "senkin13",
          "author_url": "",
          "post_date": "10/18/2019 11:50:15",
          "content": "<p><a href=\"/kyakovlev\">@kyakovlev</a> Thanks for your probing, I think random split will lead to leaderboard leakage,so it's possible as you mentioned that 22% of first lines are private, or public and private are mixed with different month/week...</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 652130,
          "author_name": "kyakovlev",
          "author_url": "",
          "post_date": "10/18/2019 11:57:06",
          "content": "<p>Some mix should be. I'm too dumb in stats to prove it knowing lb score. </p>\n\n<p>But I think it's possible to make calculation if you know lb score before and lb after excluding 22% last test data.</p>\n\n<p>What we have:\n* All 0 predictions -&gt; 4.69\n* Baseline - &gt; 1.36\n* Excluding 22% -&gt; 2.6</p>\n\n<p>We need to calculate if 22% of 0s corresponds 1.24 RMSLE difference. If so - &gt; random split, if not possible mix.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 652965,
          "author_name": "aerdem4",
          "author_url": "",
          "post_date": "10/19/2019 18:13:08",
          "content": "<p>4.69^2 = 22.00\n1.36^2 = 1.85\n2.60^2 = 6.76</p>\n\n<p>(6.76 - 1.85)/(22.0 - 1.85) = 0.24</p>\n\n<p>Given that last months are winter, 0.24 being a bit higher than 0.22 makes sense. So the split is probably not 78/22 by time.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 659683,
          "author_name": "arnabbiswas1",
          "author_url": "",
          "post_date": "10/28/2019 04:45:11",
          "content": "<p>For anyone from the future, as of 28th Oct 2019, <a href=\"/geoffrey7697\">@geoffrey7697</a> probed the LB and identified that data between August 2018 to Dec 2018 is part of the Private LB. In fact there are evidences that it could be even more (may include July 2018 as well):</p>\n\n<p><a href=\"https://www.kaggle.com/c/ashrae-energy-prediction/discussion/113751#latest-659225\">https://www.kaggle.com/c/ashrae-energy-prediction/discussion/113751#latest-659225</a></p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 652493,
      "author_name": "pcjimmmy",
      "author_url": "",
      "post_date": "10/18/2019 23:24:54",
      "content": "<p>IF it was I setting up the test split I would do it by building.  If time split like past Kaggle series, than 22% of two years would just be the first 6 months of 2017.</p>\n\n<p>If done by building than the public test could still contain the full two years.</p>\n\n<p>Assume that shortly someone will do the LB probe that lets us know.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 658145,
      "author_name": "carlolepelaars",
      "author_url": "",
      "post_date": "10/25/2019 17:53:36",
      "content": "<p>Thank you for the advice!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 659782,
      "author_name": "arnabbiswas1",
      "author_url": "",
      "post_date": "10/28/2019 08:56:51",
      "content": "<p>Since Public LeaderBoard is much larger (1.5 years) compared to Train (1 year) or Private Leader Board (Possibly 0.5 years), should not we give value to the Public LB score along with Local CV score? Otherwise, it's fairly difficult to understand if the features capturing long term trends are helping or not. I suspect that this is the reason that Kaggle has limited the number of daily submission limit to 2. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "651870": "Here are a few notes that may help new kagglers to prevent  overffiting.\n\n1. Since it's a timeseries problem,so you would be better to split train and validation set by timestamp , sometimes kfold is also can be used for timeseries problem but it's better to  check first,you can split train and validation set by timestamp, then train a kfold-model using train set to watch the validation set performance.\n\n2. Don't use features only exsited in train but not in test,for example year, time-lagged targetencoding...also don't use features that very different between train and public/private test,like a feature has no/less null values in train but lots of null values  in test, or public test has no/less null but private test has a lot.\n\n3. Since the test time period is longer than train,it's hard to setup a robust validation time period,you will find parameters especially earlystop will have a big impact to leaderboard score,because it will overfit to local validation set but very different from test set. I suggested to try to adjust these parameters ```max_bin, num_leaves, bagging_fraction, feature_fraction, lambda_l1, lambda_l2, max_depth``` to deal with overfitting,and try to setup different timesplit validation to watch the change of earlystop round,At last can use a optimized round number to stop the training.\n\n4. Don't trust public leaderboard too much ,because public and private are also splited by time.You can split your train data to simulate train/valid/public/private, make sure your model can improve all of them.\n\n5. Good luck and have fun!",
    "651877": "Thanks for sharing! @senkin13 \n\nTo anyone that doesn't know how to do a time series split, you can implement the time series split as shown in:\nhttps://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html\n\n![](https://scikit-learn.org/stable/_images/sphx_glr_plot_cv_indices_010.png)",
    "651947": "Very Informative Share.. Thanks @senkin13   @tiannnn",
    "652033": "Hi @senkin13, I was wondering about that. For this competition the train set is for the year 2016 and the test set for the years 2017 and 2018. If we do a split by timestamp, for example, train on the first 10 months and validate on the last two months, wouldn't this deviate from the \"prediction reality\"? \n\nWhat I mean is that during validation we can loose the effect of seasonality features. For example, the \"month\" feature will be useless since we will never predict for a month that we trained on, whilst when we predict on the test set this information can be quite important. My experiments with [KFold](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html) and [TimeSeriesSplit](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html) shows that the former underfits and the latter overfits.\n\nAny ideas on how to approach cross validation for this problem?",
    "652102": "&gt; Don't trust public leaderboard too much ,because public and private are also splited by time.\n\n@senkin13 do you have any evidence of it? I did split lb probe and seems it's random split. What makes you think it's timebased split?",
    "652109": "wow @kyakovlev I'd like to see that. I think @inversion said:\n&gt; We always split time-based data by time for Train/Test, and Public/Private. If we didn't do that, the Public leaderboard score would provide feedback (leakage) for future time points.\n\nor at least I remember that someone posted that.",
    "652110": "sorry for posting link to other discussion\n\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/112872#651685",
    "652123": "kyakovlev Thanks for your probing, I think random split will lead to leaderboard leakage,so it's possible as you mentioned that 22% of first lines are private, or public and private are mixed with different month/week...",
    "652130": "Some mix should be. I'm too dumb in stats to prove it knowing lb score. \n\nBut I think it's possible to make calculation if you know lb score before and lb after excluding 22% last test data.\n\nWhat we have:\n* All 0 predictions -&gt; 4.69\n* Baseline - &gt; 1.36\n* Excluding 22% -&gt; 2.6\n\nWe need to calculate if 22% of 0s corresponds 1.24 RMSLE difference. If so - &gt; random split, if not possible mix.",
    "652140": "I agree with your viewpoint,since I have not worked deeply for competition,just talk one simple idea.\n The purpose to find a robust validation is for finding robust features,so you can create many features then use different cross validation to check which are robust,including kfold/timeseriessplit...",
    "652493": "IF it was I setting up the test split I would do it by building.  If time split like past Kaggle series, than 22% of two years would just be the first 6 months of 2017.\n\nIf done by building than the public test could still contain the full two years.\n\nAssume that shortly someone will do the LB probe that lets us know.",
    "652965": "4.69^2 = 22.00\n1.36^2 = 1.85\n2.60^2 = 6.76\n\n(6.76 - 1.85)/(22.0 - 1.85) = 0.24\n\nGiven that last months are winter, 0.24 being a bit higher than 0.22 makes sense. So the split is probably not 78/22 by time.",
    "658145": "Thank you for the advice!",
    "659683": "For anyone from the future, as of 28th Oct 2019, @geoffrey7697 probed the LB and identified that data between August 2018 to Dec 2018 is part of the Private LB. In fact there are evidences that it could be even more (may include July 2018 as well):\n\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/113751#latest-659225",
    "659782": "Since Public LeaderBoard is much larger (1.5 years) compared to Train (1 year) or Private Leader Board (Possibly 0.5 years), should not we give value to the Public LB score along with Local CV score? Otherwise, it's fairly difficult to understand if the features capturing long term trends are helping or not. I suspect that this is the reason that Kaggle has limited the number of daily submission limit to 2."
  },
  "source": "meta"
}