Be very careful if you're using time series methods to make predictions. Time series methods often work well for short-range forecasts (like those that apply to the public test data) but not so well for longer-range forecasts (like those that apply to the private test data). Indeed, longer-range forecasts can get really weird. A naive time series fit, based on the data alone without applying any domain knowledge, may end up effectively extrapolating processes that can't reasonably be expected to continue indefinitely. Hopefully you should be able to diagnose and fix such problems by using an appropriate holdout set or rolling window from the training data to validate your model, but you might also want to check your test set predictions to see if they look reasonable.
", "messages": [ { "id": "256906", "postDate": "12/12/2017 22:30:32", "content": "Be very careful if you're using time series methods to make predictions. Time series methods often work well for short-range forecasts (like those that apply to the public test data) but not so well for longer-range forecasts (like those that apply to the private test data). Indeed, longer-range forecasts can get really weird. A naive time series fit, based on the data alone without applying any domain knowledge, may end up effectively extrapolating processes that can't reasonably be expected to continue indefinitely. Hopefully you should be able to diagnose and fix such problems by using an appropriate holdout set or rolling window from the training data to validate your model, but you might also want to check your test set predictions to see if they look reasonable.
", "rawMarkdown": "Be very careful if you're using time series methods to make predictions. Time series methods often work well for short-range forecasts (like those that apply to the public test data) but not so well for longer-range forecasts (like those that apply to the private test data). Indeed, longer-range forecasts can get really weird. A naive time series fit, based on the data alone without applying any domain knowledge, may end up effectively extrapolating processes that can't reasonably be expected to continue indefinitely. Hopefully you should be able to diagnose and fix such problems by using an appropriate holdout set or rolling window from the training data to validate your model, but you might also want to check your test set predictions [to see if they look reasonable][1].\n\n [1]: https://www.kaggle.com/aharless/the-danger-of-far-out-of-sample-ts-forecasts/", "votes": null }, { "id": "257201", "postDate": "12/13/2017 17:28:23", "content": "I'm increasingly realizing that a public LB that consists of the first 5 out of 16 days is not going to be very useful in distinguishing good models from bad ones
", "rawMarkdown": "I'm increasingly realizing that a public LB that consists of the first 5 out of 16 days is not going to be very useful in distinguishing good models from bad ones", "votes": null }, { "id": "258666", "postDate": "12/16/2017 18:47:56", "content": "Also I have a more generic sanity check now.
", "rawMarkdown": "Also I have a more generic [sanity check][1] now.\n\n [1]: https://www.kaggle.com/aharless/quick-sanity-check", "votes": null }, { "id": "259872", "postDate": "12/19/2017 05:24:03", "content": "Hey Andy :
\n\nCame across your inputs when searching for a function createTimeSlices in python\nCan you please comment on discussion topic created on the same
", "rawMarkdown": "Hey Andy :\n\nCame across your inputs when searching for a function createTimeSlices in python\nCan you please comment on [discussion topic][1] created on the same\n\n\n [1]: https://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/45993", "votes": null } ], "comments": [ { "id": 257201, "author_name": "aharless", "author_url": "", "post_date": "12/13/2017 17:28:23", "content": "I'm increasingly realizing that a public LB that consists of the first 5 out of 16 days is not going to be very useful in distinguishing good models from bad ones
", "votes": null, "replies": [] }, { "id": 258666, "author_name": "aharless", "author_url": "", "post_date": "12/16/2017 18:47:56", "content": "Also I have a more generic sanity check now.
", "votes": null, "replies": [] }, { "id": 259872, "author_name": "ydhiman", "author_url": "", "post_date": "12/19/2017 05:24:03", "content": "Hey Andy :
\n\nCame across your inputs when searching for a function createTimeSlices in python\nCan you please comment on discussion topic created on the same
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "256906": "Be very careful if you're using time series methods to make predictions. Time series methods often work well for short-range forecasts (like those that apply to the public test data) but not so well for longer-range forecasts (like those that apply to the private test data). Indeed, longer-range forecasts can get really weird. A naive time series fit, based on the data alone without applying any domain knowledge, may end up effectively extrapolating processes that can't reasonably be expected to continue indefinitely. Hopefully you should be able to diagnose and fix such problems by using an appropriate holdout set or rolling window from the training data to validate your model, but you might also want to check your test set predictions [to see if they look reasonable][1].\n\n [1]: https://www.kaggle.com/aharless/the-danger-of-far-out-of-sample-ts-forecasts/", "257201": "I'm increasingly realizing that a public LB that consists of the first 5 out of 16 days is not going to be very useful in distinguishing good models from bad ones", "258666": "Also I have a more generic [sanity check][1] now.\n\n [1]: https://www.kaggle.com/aharless/quick-sanity-check", "259872": "Hey Andy :\n\nCame across your inputs when searching for a function createTimeSlices in python\nCan you please comment on [discussion topic][1] created on the same\n\n\n [1]: https://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/45993" }, "source": "meta" }