So far I have enjoyed playing with this competition using few simple aggregations. The best public kernel is from Paulo https://www.kaggle.com/paulorzp/log-ma-and-days-of-week-means-lb-0-532
\n\nIt is probably time to slow down and think more about feature engineering and modeling. Of course the validation strategy is also critical. I start to think about the time-based split for the train dataset - Keeping data of the last 16 days for the validation.
\n\nImportant note is the LB is also split by dates. It turns out that the private set starts from 2017-08-21 (see https://www.kaggle.com/tarobxl/how-the-test-set-is-split-lb-0-532?scriptVersionId=1728938)
\n\nAny other suggestions?
", "messages": [ { "id": "241646", "postDate": "11/09/2017 13:47:08", "content": "So far I have enjoyed playing with this competition using few simple aggregations. The best public kernel is from Paulo https://www.kaggle.com/paulorzp/log-ma-and-days-of-week-means-lb-0-532
\n\nIt is probably time to slow down and think more about feature engineering and modeling. Of course the validation strategy is also critical. I start to think about the time-based split for the train dataset - Keeping data of the last 16 days for the validation.
\n\nImportant note is the LB is also split by dates. It turns out that the private set starts from 2017-08-21 (see https://www.kaggle.com/tarobxl/how-the-test-set-is-split-lb-0-532?scriptVersionId=1728938)
\n\nAny other suggestions?
", "rawMarkdown": "So far I have enjoyed playing with this competition using few simple aggregations. The best public kernel is from Paulo https://www.kaggle.com/paulorzp/log-ma-and-days-of-week-means-lb-0-532\n\nIt is probably time to slow down and think more about feature engineering and modeling. Of course the validation strategy is also critical. I start to think about the time-based split for the train dataset - Keeping data of the last 16 days for the validation. \n\nImportant note is the LB is also split by dates. It turns out that the private set starts from 2017-08-21 (see https://www.kaggle.com/tarobxl/how-the-test-set-is-split-lb-0-532?scriptVersionId=1728938)\n\nAny other suggestions?", "votes": null }, { "id": "241660", "postDate": "11/09/2017 14:13:24", "content": "As @Konrad Banachewicz already said you could use a rolling window evaluation. There are however some things you should keep in mind:
\n\nWhat I like to do when it comes to time series prediction (but haven't had the time to do here yet) is use classic cross validation. An excellent point to start from is this blog post as well as the corresponding paper both by Rob J. Hyndman (btw author of the forecast package in R - BIG THANKS FOR THAT!). The paper discusses the mathematical validity of classic cross validation in an autoregressive setting. Think of it as follows:\nConsider a time series y_t. If you assume it is purely autoregressive of a certain order p, meaning that the value of y_t only depends on the values at times (t-1), (t-2), ... , (t-p). Therefore you can set up the following matrix for example of order 3:
\n\nlag_1 | lag_1 | lag_3 | label
\n\ny_1 | y_2 | y_3 | y_4
\n\ny_2 | y_3 | y_4 | y_5
\n\n..............................................
\n\ny_{t-3}|y_{t-2}|y_{t-1}|y_t
\n\nNow you can essentally do cross validation in the classic sense. And you got it mathematically justified (well if you care about that :) )
", "rawMarkdown": "As @Konrad Banachewicz already said you could use a rolling window evaluation. There are however some things you should keep in mind:\n\n1. It's quite expensive to do that. \n2. The period on which you are rolling your window might not be representative for the period you are testing on due to seasonal and holiday effect.\n3. If you want to validate on the period exactly one year before that, you can essentially discard 1 year of information and might not have enough data.... \n\nWhat I like to do when it comes to time series prediction (but haven't had the time to do here yet) is use classic cross validation. An excellent point to start from is [this blog post][1] as well as [the corresponding paper][2] both by [Rob J. Hyndman][3] (btw author of [the forecast package in R][4] - BIG THANKS FOR THAT!). The paper discusses the mathematical validity of classic cross validation in an autoregressive setting. Think of it as follows:\nConsider a time series y_t. If you assume it is purely autoregressive of a certain order p, meaning that the value of y_t only depends on the values at times (t-1), (t-2), ... , (t-p). Therefore you can set up the following matrix for example of order 3:\n\nlag_1 | lag_1 | lag_3 | label \n\n y_1 | y_2 | y_3 | y_4 \n\n y_2 | y_3 | y_4 | y_5 \n\n..............................................\n\ny_{t-3}|y_{t-2}|y_{t-1}|y_t \n\nNow you can essentally do cross validation in the classic sense. And you got it mathematically justified (well if you care about that :) )\n\n [1]: https://robjhyndman.com/publications/cv-time-series/\n [2]: https://robjhyndman.com/papers/cv-wp.pdf\n [3]: https://robjhyndman.com/\n [4]: http://pkg.robjhyndman.com/forecast/", "votes": null }, { "id": "241685", "postDate": "11/09/2017 15:27:49", "content": "That's weird. The score on the 2017-7-26 to 2017-08-10 timespan works fine for me so far. The relation between local score and public LB is almost monotonic. When the relationship broke (has happened a few times) it was usually caused by bugs in my code.
\n\nAveraging models also works for me.
", "rawMarkdown": "That's weird. The score on the 2017-7-26 to 2017-08-10 timespan works fine for me so far. The relation between local score and public LB is almost monotonic. When the relationship broke (has happened a few times) it was usually caused by bugs in my code. \n\nAveraging models also works for me.", "votes": null }, { "id": "241687", "postDate": "11/09/2017 15:30:31", "content": "The test timespan 2017-08-16 to 2017-08-31 starts on Wednesday and ends on Thursday, and 2017-08-10 is the latest Thursday available in training. My intention was to get the same distribution of weekdays.
", "rawMarkdown": "The test timespan 2017-08-16 to 2017-08-31 starts on Wednesday and ends on Thursday, and 2017-08-10 is the latest Thursday available in training. My intention was to get the same distribution of weekdays.", "votes": null }, { "id": "241707", "postDate": "11/09/2017 16:36:56", "content": "Thanks @Konrad, @CenShine and @Max. I am also think of seasoning validation but 1) there might have less data 2) there was a big earthquake in 2016, which might mess up the trend. Well, I will try both if I have time.
", "rawMarkdown": "Thanks @Konrad, @CenShine and @Max. I am also think of seasoning validation but 1) there might have less data 2) there was a big earthquake in 2016, which might mess up the trend. Well, I will try both if I have time.", "votes": null }, { "id": "241708", "postDate": "11/09/2017 16:38:25", "content": "Brilliant, thanks for the tip.
", "rawMarkdown": "Brilliant, thanks for the tip.", "votes": null }, { "id": "248935", "postDate": "11/27/2017 10:08:43", "content": "Hi tarobxl, could you tell me how did you find the LB is split by dates ?
", "rawMarkdown": "Hi tarobxl, could you tell me how did you find the LB is split by dates ?", "votes": null } ], "comments": [ { "id": 241660, "author_name": "konradb", "author_url": "", "post_date": "11/09/2017 14:13:24", "content": "That's weird. The score on the 2017-7-26 to 2017-08-10 timespan works fine for me so far. The relation between local score and public LB is almost monotonic. When the relationship broke (has happened a few times) it was usually caused by bugs in my code.
\n\nAveraging models also works for me.
", "votes": null, "replies": [] }, { "id": 241687, "author_name": "konradb", "author_url": "", "post_date": "11/09/2017 15:30:31", "content": "The test timespan 2017-08-16 to 2017-08-31 starts on Wednesday and ends on Thursday, and 2017-08-10 is the latest Thursday available in training. My intention was to get the same distribution of weekdays.
", "votes": null, "replies": [] }, { "id": 241708, "author_name": "konradb", "author_url": "", "post_date": "11/09/2017 16:38:25", "content": "Brilliant, thanks for the tip.
", "votes": null, "replies": [] } ] }, { "id": 241676, "author_name": "mbernkopf", "author_url": "", "post_date": "11/09/2017 15:01:53", "content": "As @Konrad Banachewicz already said you could use a rolling window evaluation. There are however some things you should keep in mind:
\n\nWhat I like to do when it comes to time series prediction (but haven't had the time to do here yet) is use classic cross validation. An excellent point to start from is this blog post as well as the corresponding paper both by Rob J. Hyndman (btw author of the forecast package in R - BIG THANKS FOR THAT!). The paper discusses the mathematical validity of classic cross validation in an autoregressive setting. Think of it as follows:\nConsider a time series y_t. If you assume it is purely autoregressive of a certain order p, meaning that the value of y_t only depends on the values at times (t-1), (t-2), ... , (t-p). Therefore you can set up the following matrix for example of order 3:
\n\nlag_1 | lag_1 | lag_3 | label
\n\ny_1 | y_2 | y_3 | y_4
\n\ny_2 | y_3 | y_4 | y_5
\n\n..............................................
\n\ny_{t-3}|y_{t-2}|y_{t-1}|y_t
\n\nNow you can essentally do cross validation in the classic sense. And you got it mathematically justified (well if you care about that :) )
", "votes": null, "replies": [] }, { "id": 241707, "author_name": "tarobxl", "author_url": "", "post_date": "11/09/2017 16:36:56", "content": "Thanks @Konrad, @CenShine and @Max. I am also think of seasoning validation but 1) there might have less data 2) there was a big earthquake in 2016, which might mess up the trend. Well, I will try both if I have time.
", "votes": null, "replies": [] }, { "id": 248935, "author_name": "wangziwei02", "author_url": "", "post_date": "11/27/2017 10:08:43", "content": "Hi tarobxl, could you tell me how did you find the LB is split by dates ?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "241646": "So far I have enjoyed playing with this competition using few simple aggregations. The best public kernel is from Paulo https://www.kaggle.com/paulorzp/log-ma-and-days-of-week-means-lb-0-532\n\nIt is probably time to slow down and think more about feature engineering and modeling. Of course the validation strategy is also critical. I start to think about the time-based split for the train dataset - Keeping data of the last 16 days for the validation. \n\nImportant note is the LB is also split by dates. It turns out that the private set starts from 2017-08-21 (see https://www.kaggle.com/tarobxl/how-the-test-set-is-split-lb-0-532?scriptVersionId=1728938)\n\nAny other suggestions?", "241660": "1. the score on the last 16 days of the training set is next to useless - the scores have a kind of directional agreement with public LB at best\n 2. that being said, for ensembling it's worse: I am still scratching my head why a simple average of decent models (by LB performance) is worse on public LB than each individual - first time I've seen sth like that\n 3. I am working on rolling window evaluation, along the lines of Hyndman et al https://robjhyndman.com/hyndsight/tscvexample/", "241676": "As @Konrad Banachewicz already said you could use a rolling window evaluation. There are however some things you should keep in mind:\n\n1. It's quite expensive to do that. \n2. The period on which you are rolling your window might not be representative for the period you are testing on due to seasonal and holiday effect.\n3. If you want to validate on the period exactly one year before that, you can essentially discard 1 year of information and might not have enough data.... \n\nWhat I like to do when it comes to time series prediction (but haven't had the time to do here yet) is use classic cross validation. An excellent point to start from is [this blog post][1] as well as [the corresponding paper][2] both by [Rob J. Hyndman][3] (btw author of [the forecast package in R][4] - BIG THANKS FOR THAT!). The paper discusses the mathematical validity of classic cross validation in an autoregressive setting. Think of it as follows:\nConsider a time series y_t. If you assume it is purely autoregressive of a certain order p, meaning that the value of y_t only depends on the values at times (t-1), (t-2), ... , (t-p). Therefore you can set up the following matrix for example of order 3:\n\nlag_1 | lag_1 | lag_3 | label \n\n y_1 | y_2 | y_3 | y_4 \n\n y_2 | y_3 | y_4 | y_5 \n\n..............................................\n\ny_{t-3}|y_{t-2}|y_{t-1}|y_t \n\nNow you can essentally do cross validation in the classic sense. And you got it mathematically justified (well if you care about that :) )\n\n [1]: https://robjhyndman.com/publications/cv-time-series/\n [2]: https://robjhyndman.com/papers/cv-wp.pdf\n [3]: https://robjhyndman.com/\n [4]: http://pkg.robjhyndman.com/forecast/", "241685": "That's weird. The score on the 2017-7-26 to 2017-08-10 timespan works fine for me so far. The relation between local score and public LB is almost monotonic. When the relationship broke (has happened a few times) it was usually caused by bugs in my code. \n\nAveraging models also works for me.", "241687": "1. Haven't checked the 26.07 - 10.08 period so far, thanks for the hint - out of curiosity, why this particular choice?\n2. I am probably overfitting one of the 'candidate' models - I tried two or three averages of subsets of all my models (using only the really simple ones), and the improvement is indeed visible.", "241702": "The test timespan 2017-08-16 to 2017-08-31 starts on Wednesday and ends on Thursday, and 2017-08-10 is the latest Thursday available in training. My intention was to get the same distribution of weekdays.", "241707": "Thanks @Konrad, @CenShine and @Max. I am also think of seasoning validation but 1) there might have less data 2) there was a big earthquake in 2016, which might mess up the trend. Well, I will try both if I have time.", "241708": "Brilliant, thanks for the tip.", "248935": "Hi tarobxl, could you tell me how did you find the LB is split by dates ?" }, "source": "meta" }