Hey,
\n\nDo I understand it correctly that the predictions for week 10 are only used for the public leaderboard, but the predictions for week 11 are the ones that will be used for the final private scoring?
\n\n\n\n\nThe train and test dataset are split based on time, as well as the public and private leaderboard dataset split.
\n
So why should we try to predict week 10 besides fooling ourselves that we are high on the leaderboard?
\n\nKasper
", "messages": [ { "id": 128385, "postDate": "2016-07-19T20:17:05.620Z", "content": "Just in case:\nI did some tests and it is clear, that the data for the public leaderboard is only from week 10.\nI used a submission and zeroed out the predictions form week 11 and got the same score as before. Also the number of the samples per week suggest such a split.\nI hope it helps
", "rawMarkdown": "Just in case:\r\nI did some tests and it is clear, that the data for the public leaderboard is only from week 10.\r\nI used a submission and zeroed out the predictions form week 11 and got the same score as before. Also the number of the samples per week suggest such a split.\r\nI hope it helps ", "votes": 19 }, { "id": 125256, "postDate": "2016-06-28T06:48:27.020Z", "content": "Hey,
\n\nDo I understand it correctly that the predictions for week 10 are only used for the public leaderboard, but the predictions for week 11 are the ones that will be used for the final private scoring?
\n\n\n\n\nThe train and test dataset are split based on time, as well as the public and private leaderboard dataset split.
\n
So why should we try to predict week 10 besides fooling ourselves that we are high on the leaderboard?
\n\nKasper
", "rawMarkdown": "Hey,\r\n\r\nDo I understand it correctly that the predictions for week 10 are only used for the public leaderboard, but the predictions for week 11 are the ones that will be used for the final private scoring?\r\n\r\n> The train and test dataset are split based on time, as well as the public and private leaderboard dataset split.\r\n\r\nSo why should we try to predict week 10 besides fooling ourselves that we are high on the leaderboard?\r\n\r\nKasper", "votes": 4 }, { "id": 128463, "postDate": "2016-07-20T10:21:29.087Z", "content": "Thank you :)\nDo you think this would work? We wouldn't be able to test this model against the leaderboard... Maybe it's best to just don't trust the leaderboard too much and pick a model stable in validation.
", "rawMarkdown": "Thank you :)\r\nDo you think this would work? We wouldn't be able to test this model against the leaderboard... Maybe it's best to just don't trust the leaderboard too much and pick a model stable in validation. ", "votes": 3 }, { "id": 130107, "postDate": "2016-08-03T21:44:14.590Z", "content": "Here is official Wendy's post to answer the same @bobye's question, it's indeed split by week. I tried it by submitting all zeros demand for week 11.
\n\nThen be careful when using this lag method, it uses what @ihopethiswillfi's wrote.
", "rawMarkdown": "Here is official Wendy's [post][1] to answer the same @bobye's question, it's indeed split by week. I tried it by submitting all zeros demand for week 11. \r\n\r\nThen be careful when using this [lag method][2], it uses what @ihopethiswillfi's wrote. \r\n\r\n\r\n [1]: https://www.kaggle.com/c/grupo-bimbo-inventory-demand/forums/t/21501/welcome/125344#post125344\r\n [2]: https://www.kaggle.com/c/grupo-bimbo-inventory-demand/forums/t/22393/current-method-sharing", "votes": 1 }, { "id": 128529, "postDate": "2016-07-21T01:37:25.287Z", "content": "I modelled weeks 10 and 11 with a decision tree model with default parameter in R, with the training taking a bit longer than expected. I could differentiate weeks 10 or 11 at 60% accuracy, compared to the 0.51 /0.49 split. Main difference are Client ID, with 71252 different entries.
\n\nHowever from other recent post, it seems there is some time series effect in the data, which makes sense, given shops probably purchase in a monthly schedule ,or perhaps bi-weekly. Thus week 10 and week 11 should be quite different imo.
", "rawMarkdown": "I modelled weeks 10 and 11 with a decision tree model with default parameter in R, with the training taking a bit longer than expected. I could differentiate weeks 10 or 11 at 60% accuracy, compared to the 0.51 /0.49 split. Main difference are Client ID, with 71252 different entries. \r\n\r\nHowever from other recent post, it seems there is some time series effect in the data, which makes sense, given shops probably purchase in a monthly schedule ,or perhaps bi-weekly. Thus week 10 and week 11 should be quite different imo. ", "votes": 1 }, { "id": 128451, "postDate": "2016-07-20T08:39:34.580Z", "content": "Try not to overfit ;)
", "rawMarkdown": "Try not to overfit ;)", "votes": 1 }, { "id": 130102, "postDate": "2016-08-03T20:57:24.493Z", "content": "Basically if we're using our own week10's predictions to create a model for predicing week11, and the public leaderboard is showing our performance only on week10, then that basically means it is advisable to overfit the crap out of the public leaderboard in order to get the best baseline for predicting week11? Hmmm there will be a nice leaderboard shakeup when this ends :)
", "rawMarkdown": "Basically if we're using our own week10's predictions to create a model for predicing week11, and the public leaderboard is showing our performance only on week10, then that basically means it is advisable to overfit the crap out of the public leaderboard in order to get the best baseline for predicting week11? Hmmm there will be a nice leaderboard shakeup when this ends :)" }, { "id": 130034, "postDate": "2016-08-03T12:57:54.643Z", "content": "@ yglSmith That's why the top LB does not change much despite people submitting new solutions...
", "rawMarkdown": "@ yglSmith That's why the top LB does not change much despite people submitting new solutions..." }, { "id": 130009, "postDate": "2016-08-03T10:22:15.500Z", "content": "yes,I think Kasper Van Lombeek is right.I optimize week11 twice while my leaderboard has no changing.
", "rawMarkdown": "yes,I think Kasper Van Lombeek is right.I optimize week11 twice while my leaderboard has no changing." }, { "id": 129192, "postDate": "2016-07-27T12:39:15.680Z", "content": "[quote=happycube;129091]
\n\nWhat I wound up doing (and used for the coefficients on my posted script - which was why I said I was confident they'd hold up on the private LB ;) ) was to train on weeks 3-7, and then test on week 9. I did a bit of testing between weeks 8/9 and found 8 had lower error, but I don't have the numbers on me right now...
\n\n[/quote]
\n\nYes, this is plausible. I spent a lot time in the early stage of competition to find a reasonable way to validate my models (by CORRECTLY using LB). I was not quite successful, backtest overfitting was always presented someway. But I was able to discover all the phenomena discussed before. Just note here.
", "rawMarkdown": "[quote=happycube;129091]\r\n\r\nWhat I wound up doing (and used for the coefficients on my posted script - which was why I said I was confident they'd hold up on the private LB ;) ) was to train on weeks 3-7, and then test on week 9. I did a bit of testing between weeks 8/9 and found 8 had lower error, but I don't have the numbers on me right now...\r\n\r\n[/quote]\r\n\r\nYes, this is plausible. I spent a lot time in the early stage of competition to find a reasonable way to validate my models (by CORRECTLY using LB). I was not quite successful, backtest overfitting was always presented someway. But I was able to discover all the phenomena discussed before. Just note here. \r\n" }, { "id": 129091, "postDate": "2016-07-26T16:49:12.050Z", "content": "What I wound up doing (and used for the coefficients on my posted script - which was why I said I was confident they'd hold up on the private LB ;) ) was to train on weeks 3-7, and then test on week 9. I did a bit of testing between weeks 8/9 and found 8 had lower error, but I don't have the numbers on me right now...
", "rawMarkdown": "What I wound up doing (and used for the coefficients on my posted script - which was why I said I was confident they'd hold up on the private LB ;) ) was to train on weeks 3-7, and then test on week 9. I did a bit of testing between weeks 8/9 and found 8 had lower error, but I don't have the numbers on me right now..." }, { "id": 129090, "postDate": "2016-07-26T16:27:01.777Z", "content": "[quote=Sterby;128385]
\n\nJust in case:\nI did some tests and it is clear, that the data for the public leaderboard is only from week 10.\nI used a submission and zeroed out the predictions form week 11 and got the same score as before. Also the number of the samples per week suggest such a split.\nI hope it helps
\n\n[/quote]
\n\nAs a consequence, the private leaderboard is only from week 11 !!! (due to the split 51% public / 49% private)
", "rawMarkdown": "[quote=Sterby;128385]\r\n\r\nJust in case:\r\nI did some tests and it is clear, that the data for the public leaderboard is only from week 10.\r\nI used a submission and zeroed out the predictions form week 11 and got the same score as before. Also the number of the samples per week suggest such a split.\r\nI hope it helps \r\n\r\n[/quote]\r\n\r\nAs a consequence, the private leaderboard is only from week 11 !!! (due to the split 51% public / 49% private)\r\n" }, { "id": 128465, "postDate": "2016-07-20T10:26:50.787Z", "content": "Sterby: No, but you have to build your own test set to forecast two weeks ahead.
", "rawMarkdown": "Sterby: No, but you have to build your own test set to forecast two weeks ahead." }, { "id": 128460, "postDate": "2016-07-20T10:10:18.647Z", "content": "Outstanding job, Sterby!\nTo avoid overfitting, maybe separate models are needed, for week 10 and 11.
", "rawMarkdown": "Outstanding job, Sterby!\r\nTo avoid overfitting, maybe separate models are needed, for week 10 and 11.\r\n" }, { "id": 128455, "postDate": "2016-07-20T09:45:05.630Z", "content": "[quote=Sterby;128452]
\n\nyes, that's always the goal ;)\nBut how to ensure that week 11 is not substancially different from week 10?
\n\n[/quote]
\n\nTry to see if you can make a classifier to differentiate week 10 week 11, and see if you get anything?
", "rawMarkdown": "[quote=Sterby;128452]\r\n\r\nyes, that's always the goal ;)\r\nBut how to ensure that week 11 is not substancially different from week 10?\r\n\r\n[/quote]\r\n\r\nTry to see if you can make a classifier to differentiate week 10 week 11, and see if you get anything? " }, { "id": 128452, "postDate": "2016-07-20T08:41:16.923Z", "content": "yes, that's always the goal ;)\nBut how to ensure that week 11 is not substancially different from week 10?
", "rawMarkdown": "yes, that's always the goal ;)\r\nBut how to ensure that week 11 is not substancially different from week 10?" }, { "id": 128446, "postDate": "2016-07-20T07:44:52.467Z", "content": "Now its the question what to do with this information? ;)
", "rawMarkdown": "Now its the question what to do with this information? ;)" }, { "id": 128442, "postDate": "2016-07-20T07:17:59.280Z", "content": "@Sterby, thanks for confirming this. I was about to do the same test.
", "rawMarkdown": "@Sterby, thanks for confirming this. I was about to do the same test." }, { "id": 125334, "postDate": "2016-06-28T17:55:24.227Z", "content": "[quote=Mattias Fagerlund;125257]
\n\nThe public/private split could be week 10/week 11, it might also be 40%/60%, meaning 40% of week 10 + week 11 is public (sorted by week) and the rest is private. Or 60%/40%. But I can't see a way where that influences the strategies we should use.
\n\nDoing well on public leaderboard doesn't guarantee you'll do well on the private leaderboard, but it's more likely than not. It's easy to overfit on the training data, it's harder to overfit on the public leaderboard (though you can with enough submissions).
\n\n/m
\n\n[/quote]
\n\nI really think it is still easy to overfit public LB by 0.003 (which makes a huge difference in top10). If the data are not split purely random. I also asked Wendy in the top thread, let's wait her response.
", "rawMarkdown": "[quote=Mattias Fagerlund;125257]\r\n\r\nThe public/private split could be week 10/week 11, it might also be 40%/60%, meaning 40% of week 10 + week 11 is public (sorted by week) and the rest is private. Or 60%/40%. But I can't see a way where that influences the strategies we should use.\r\n\r\nDoing well on public leaderboard doesn't guarantee you'll do well on the private leaderboard, but it's more likely than not. It's easy to overfit on the training data, it's harder to overfit on the public leaderboard (though you can with enough submissions).\r\n\r\n/m\r\n\r\n[/quote]\r\n\r\nI really think it is still easy to overfit public LB by 0.003 (which makes a huge difference in top10). If the data are not split purely random. I also asked Wendy in the top thread, let's wait her response. \r\n\r\n \r\n\r\n" }, { "id": 125257, "postDate": "2016-06-28T07:14:54.610Z", "content": "The public/private split could be week 10/week 11, it might also be 40%/60%, meaning 40% of week 10 + week 11 is public (sorted by week) and the rest is private. Or 60%/40%. But I can't see a way where that influences the strategies we should use.
\n\nDoing well on public leaderboard doesn't guarantee you'll do well on the private leaderboard, but it's more likely than not. It's easy to overfit on the training data, it's harder to overfit on the public leaderboard (though you can with enough submissions).
\n\n/m
", "rawMarkdown": "The public/private split could be week 10/week 11, it might also be 40%/60%, meaning 40% of week 10 + week 11 is public (sorted by week) and the rest is private. Or 60%/40%. But I can't see a way where that influences the strategies we should use.\r\n\r\nDoing well on public leaderboard doesn't guarantee you'll do well on the private leaderboard, but it's more likely than not. It's easy to overfit on the training data, it's harder to overfit on the public leaderboard (though you can with enough submissions).\r\n\r\n/m" }, { "id": 128676, "postDate": "2016-07-22T14:06:45.037Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 128385, "author_name": "Sterby", "author_url": "", "post_date": "2016-07-19T20:17:05.620000", "content": "Just in case:\nI did some tests and it is clear, that the data for the public leaderboard is only from week 10.\nI used a submission and zeroed out the predictions form week 11 and got the same score as before. Also the number of the samples per week suggest such a split.\nI hope it helps
", "votes": 19, "replies": [] }, { "id": 128463, "author_name": "Sterby", "author_url": "", "post_date": "2016-07-20T10:21:29.087000", "content": "Thank you :)\nDo you think this would work? We wouldn't be able to test this model against the leaderboard... Maybe it's best to just don't trust the leaderboard too much and pick a model stable in validation.
", "votes": 3, "replies": [] }, { "id": 130107, "author_name": "aldente", "author_url": "", "post_date": "2016-08-03T21:44:14.590000", "content": "Here is official Wendy's post to answer the same @bobye's question, it's indeed split by week. I tried it by submitting all zeros demand for week 11.
\n\nThen be careful when using this lag method, it uses what @ihopethiswillfi's wrote.
", "votes": 1, "replies": [] }, { "id": 128529, "author_name": "leexa", "author_url": "", "post_date": "2016-07-21T01:37:25.287000", "content": "I modelled weeks 10 and 11 with a decision tree model with default parameter in R, with the training taking a bit longer than expected. I could differentiate weeks 10 or 11 at 60% accuracy, compared to the 0.51 /0.49 split. Main difference are Client ID, with 71252 different entries.
\n\nHowever from other recent post, it seems there is some time series effect in the data, which makes sense, given shops probably purchase in a monthly schedule ,or perhaps bi-weekly. Thus week 10 and week 11 should be quite different imo.
", "votes": 1, "replies": [] }, { "id": 128451, "author_name": "CPMP", "author_url": "", "post_date": "2016-07-20T08:39:34.580000", "content": "Try not to overfit ;)
", "votes": 1, "replies": [] }, { "id": 130102, "author_name": "ihopethiswillfi", "author_url": "", "post_date": "2016-08-03T20:57:24.493000", "content": "Basically if we're using our own week10's predictions to create a model for predicing week11, and the public leaderboard is showing our performance only on week10, then that basically means it is advisable to overfit the crap out of the public leaderboard in order to get the best baseline for predicting week11? Hmmm there will be a nice leaderboard shakeup when this ends :)
", "votes": 0, "replies": [] }, { "id": 130034, "author_name": "CPMP", "author_url": "", "post_date": "2016-08-03T12:57:54.643000", "content": "@ yglSmith That's why the top LB does not change much despite people submitting new solutions...
", "votes": 0, "replies": [] }, { "id": 130009, "author_name": "EasyYi", "author_url": "", "post_date": "2016-08-03T10:22:15.500000", "content": "yes,I think Kasper Van Lombeek is right.I optimize week11 twice while my leaderboard has no changing.
", "votes": 0, "replies": [] }, { "id": 129192, "author_name": "Jianbo Ye", "author_url": "", "post_date": "2016-07-27T12:39:15.680000", "content": "[quote=happycube;129091]
\n\nWhat I wound up doing (and used for the coefficients on my posted script - which was why I said I was confident they'd hold up on the private LB ;) ) was to train on weeks 3-7, and then test on week 9. I did a bit of testing between weeks 8/9 and found 8 had lower error, but I don't have the numbers on me right now...
\n\n[/quote]
\n\nYes, this is plausible. I spent a lot time in the early stage of competition to find a reasonable way to validate my models (by CORRECTLY using LB). I was not quite successful, backtest overfitting was always presented someway. But I was able to discover all the phenomena discussed before. Just note here.
", "votes": 0, "replies": [] }, { "id": 129091, "author_name": "happycube", "author_url": "", "post_date": "2016-07-26T16:49:12.050000", "content": "What I wound up doing (and used for the coefficients on my posted script - which was why I said I was confident they'd hold up on the private LB ;) ) was to train on weeks 3-7, and then test on week 9. I did a bit of testing between weeks 8/9 and found 8 had lower error, but I don't have the numbers on me right now...
", "votes": 0, "replies": [] }, { "id": 129090, "author_name": "Toulouse", "author_url": "", "post_date": "2016-07-26T16:27:01.777000", "content": "[quote=Sterby;128385]
\n\nJust in case:\nI did some tests and it is clear, that the data for the public leaderboard is only from week 10.\nI used a submission and zeroed out the predictions form week 11 and got the same score as before. Also the number of the samples per week suggest such a split.\nI hope it helps
\n\n[/quote]
\n\nAs a consequence, the private leaderboard is only from week 11 !!! (due to the split 51% public / 49% private)
", "votes": 0, "replies": [] }, { "id": 128465, "author_name": "gmobaz", "author_url": "", "post_date": "2016-07-20T10:26:50.787000", "content": "Sterby: No, but you have to build your own test set to forecast two weeks ahead.
", "votes": 0, "replies": [] }, { "id": 128460, "author_name": "gmobaz", "author_url": "", "post_date": "2016-07-20T10:10:18.647000", "content": "Outstanding job, Sterby!\nTo avoid overfitting, maybe separate models are needed, for week 10 and 11.
", "votes": 0, "replies": [] }, { "id": 128455, "author_name": "leexa", "author_url": "", "post_date": "2016-07-20T09:45:05.630000", "content": "[quote=Sterby;128452]
\n\nyes, that's always the goal ;)\nBut how to ensure that week 11 is not substancially different from week 10?
\n\n[/quote]
\n\nTry to see if you can make a classifier to differentiate week 10 week 11, and see if you get anything?
", "votes": 0, "replies": [] }, { "id": 128452, "author_name": "Sterby", "author_url": "", "post_date": "2016-07-20T08:41:16.923000", "content": "yes, that's always the goal ;)\nBut how to ensure that week 11 is not substancially different from week 10?
", "votes": 0, "replies": [] }, { "id": 128446, "author_name": "Sterby", "author_url": "", "post_date": "2016-07-20T07:44:52.467000", "content": "Now its the question what to do with this information? ;)
", "votes": 0, "replies": [] }, { "id": 128442, "author_name": "CPMP", "author_url": "", "post_date": "2016-07-20T07:17:59.280000", "content": "@Sterby, thanks for confirming this. I was about to do the same test.
", "votes": 0, "replies": [] }, { "id": 125334, "author_name": "Jianbo Ye", "author_url": "", "post_date": "2016-06-28T17:55:24.227000", "content": "[quote=Mattias Fagerlund;125257]
\n\nThe public/private split could be week 10/week 11, it might also be 40%/60%, meaning 40% of week 10 + week 11 is public (sorted by week) and the rest is private. Or 60%/40%. But I can't see a way where that influences the strategies we should use.
\n\nDoing well on public leaderboard doesn't guarantee you'll do well on the private leaderboard, but it's more likely than not. It's easy to overfit on the training data, it's harder to overfit on the public leaderboard (though you can with enough submissions).
\n\n/m
\n\n[/quote]
\n\nI really think it is still easy to overfit public LB by 0.003 (which makes a huge difference in top10). If the data are not split purely random. I also asked Wendy in the top thread, let's wait her response.
", "votes": 0, "replies": [] }, { "id": 125257, "author_name": "Mattias Fagerlund", "author_url": "", "post_date": "2016-06-28T07:14:54.610000", "content": "The public/private split could be week 10/week 11, it might also be 40%/60%, meaning 40% of week 10 + week 11 is public (sorted by week) and the rest is private. Or 60%/40%. But I can't see a way where that influences the strategies we should use.
\n\nDoing well on public leaderboard doesn't guarantee you'll do well on the private leaderboard, but it's more likely than not. It's easy to overfit on the training data, it's harder to overfit on the public leaderboard (though you can with enough submissions).
\n\n/m
", "votes": 0, "replies": [] }, { "id": 128676, "author_name": "", "author_url": "", "post_date": "2016-07-22T14:06:45.037000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "128385": "Just in case:\r\nI did some tests and it is clear, that the data for the public leaderboard is only from week 10.\r\nI used a submission and zeroed out the predictions form week 11 and got the same score as before. Also the number of the samples per week suggest such a split.\r\nI hope it helps ", "125256": "Hey,\r\n\r\nDo I understand it correctly that the predictions for week 10 are only used for the public leaderboard, but the predictions for week 11 are the ones that will be used for the final private scoring?\r\n\r\n> The train and test dataset are split based on time, as well as the public and private leaderboard dataset split.\r\n\r\nSo why should we try to predict week 10 besides fooling ourselves that we are high on the leaderboard?\r\n\r\nKasper", "128463": "Thank you :)\r\nDo you think this would work? We wouldn't be able to test this model against the leaderboard... Maybe it's best to just don't trust the leaderboard too much and pick a model stable in validation. ", "130107": "Here is official Wendy's [post][1] to answer the same @bobye's question, it's indeed split by week. I tried it by submitting all zeros demand for week 11. \r\n\r\nThen be careful when using this [lag method][2], it uses what @ihopethiswillfi's wrote. \r\n\r\n\r\n [1]: https://www.kaggle.com/c/grupo-bimbo-inventory-demand/forums/t/21501/welcome/125344#post125344\r\n [2]: https://www.kaggle.com/c/grupo-bimbo-inventory-demand/forums/t/22393/current-method-sharing", "128529": "I modelled weeks 10 and 11 with a decision tree model with default parameter in R, with the training taking a bit longer than expected. I could differentiate weeks 10 or 11 at 60% accuracy, compared to the 0.51 /0.49 split. Main difference are Client ID, with 71252 different entries. \r\n\r\nHowever from other recent post, it seems there is some time series effect in the data, which makes sense, given shops probably purchase in a monthly schedule ,or perhaps bi-weekly. Thus week 10 and week 11 should be quite different imo. ", "128451": "Try not to overfit ;)", "130102": "Basically if we're using our own week10's predictions to create a model for predicing week11, and the public leaderboard is showing our performance only on week10, then that basically means it is advisable to overfit the crap out of the public leaderboard in order to get the best baseline for predicting week11? Hmmm there will be a nice leaderboard shakeup when this ends :)", "130034": "@ yglSmith That's why the top LB does not change much despite people submitting new solutions...", "130009": "yes,I think Kasper Van Lombeek is right.I optimize week11 twice while my leaderboard has no changing.", "129192": "[quote=happycube;129091]\r\n\r\nWhat I wound up doing (and used for the coefficients on my posted script - which was why I said I was confident they'd hold up on the private LB ;) ) was to train on weeks 3-7, and then test on week 9. I did a bit of testing between weeks 8/9 and found 8 had lower error, but I don't have the numbers on me right now...\r\n\r\n[/quote]\r\n\r\nYes, this is plausible. I spent a lot time in the early stage of competition to find a reasonable way to validate my models (by CORRECTLY using LB). I was not quite successful, backtest overfitting was always presented someway. But I was able to discover all the phenomena discussed before. Just note here. \r\n", "129091": "What I wound up doing (and used for the coefficients on my posted script - which was why I said I was confident they'd hold up on the private LB ;) ) was to train on weeks 3-7, and then test on week 9. I did a bit of testing between weeks 8/9 and found 8 had lower error, but I don't have the numbers on me right now...", "129090": "[quote=Sterby;128385]\r\n\r\nJust in case:\r\nI did some tests and it is clear, that the data for the public leaderboard is only from week 10.\r\nI used a submission and zeroed out the predictions form week 11 and got the same score as before. Also the number of the samples per week suggest such a split.\r\nI hope it helps \r\n\r\n[/quote]\r\n\r\nAs a consequence, the private leaderboard is only from week 11 !!! (due to the split 51% public / 49% private)\r\n", "128465": "Sterby: No, but you have to build your own test set to forecast two weeks ahead.", "128460": "Outstanding job, Sterby!\r\nTo avoid overfitting, maybe separate models are needed, for week 10 and 11.\r\n", "128455": "[quote=Sterby;128452]\r\n\r\nyes, that's always the goal ;)\r\nBut how to ensure that week 11 is not substancially different from week 10?\r\n\r\n[/quote]\r\n\r\nTry to see if you can make a classifier to differentiate week 10 week 11, and see if you get anything? ", "128452": "yes, that's always the goal ;)\r\nBut how to ensure that week 11 is not substancially different from week 10?", "128446": "Now its the question what to do with this information? ;)", "128442": "@Sterby, thanks for confirming this. I was about to do the same test.", "125334": "[quote=Mattias Fagerlund;125257]\r\n\r\nThe public/private split could be week 10/week 11, it might also be 40%/60%, meaning 40% of week 10 + week 11 is public (sorted by week) and the rest is private. Or 60%/40%. But I can't see a way where that influences the strategies we should use.\r\n\r\nDoing well on public leaderboard doesn't guarantee you'll do well on the private leaderboard, but it's more likely than not. It's easy to overfit on the training data, it's harder to overfit on the public leaderboard (though you can with enough submissions).\r\n\r\n/m\r\n\r\n[/quote]\r\n\r\nI really think it is still easy to overfit public LB by 0.003 (which makes a huge difference in top10). If the data are not split purely random. I also asked Wendy in the top thread, let's wait her response. \r\n\r\n \r\n\r\n", "125257": "The public/private split could be week 10/week 11, it might also be 40%/60%, meaning 40% of week 10 + week 11 is public (sorted by week) and the rest is private. Or 60%/40%. But I can't see a way where that influences the strategies we should use.\r\n\r\nDoing well on public leaderboard doesn't guarantee you'll do well on the private leaderboard, but it's more likely than not. It's easy to overfit on the training data, it's harder to overfit on the public leaderboard (though you can with enough submissions).\r\n\r\n/m", "128676": "" } }