Lots of discussion about how to deal with onpromotion when one expands the train set (I don't want to call it a bias, probably because I'm an ignorant ;) ), but very little discussion of the public LB vs private LB. There is a difference in nature between the two, if one uses all available training data to build features. This is why Giba and I say that CV is more reliable.
\n\nIf one wants to use the public LB in a reliable way to estimate private LB results, as proposed by @sjv for instance, then one must use the public LB with the same kind of lag that the private LB has.
\n\nPublic LB gives feedback on how your model predicts 1 to 5 days in the future. Private LB is about predicting 6 to 16 days in the future.
\n\nIf you build your lag features with data up to 2018-08-10 included for the test dates, then the public LB score gives you feedback on how your model predicts 6 to 10 days in the future. And if you increment the lag, for instance use train data up to 2018-08-05 included, then the public LB gives you feedback on how your model predicts 11 to 16 days in the future. That's how public LB should be used IMHO if one does not want to rely on a local CV.
", "messages": [ { "id": "269747", "postDate": "01/17/2018 07:55:11", "content": "Lots of discussion about how to deal with onpromotion when one expands the train set (I don't want to call it a bias, probably because I'm an ignorant ;) ), but very little discussion of the public LB vs private LB. There is a difference in nature between the two, if one uses all available training data to build features. This is why Giba and I say that CV is more reliable.
\n\nIf one wants to use the public LB in a reliable way to estimate private LB results, as proposed by @sjv for instance, then one must use the public LB with the same kind of lag that the private LB has.
\n\nPublic LB gives feedback on how your model predicts 1 to 5 days in the future. Private LB is about predicting 6 to 16 days in the future.
\n\nIf you build your lag features with data up to 2018-08-10 included for the test dates, then the public LB score gives you feedback on how your model predicts 6 to 10 days in the future. And if you increment the lag, for instance use train data up to 2018-08-05 included, then the public LB gives you feedback on how your model predicts 11 to 16 days in the future. That's how public LB should be used IMHO if one does not want to rely on a local CV.
", "rawMarkdown": "Lots of discussion about how to deal with onpromotion when one expands the train set (I don't want to call it a bias, probably because I'm an ignorant ;) ), but very little discussion of the public LB vs private LB. There is a difference in nature between the two, if one uses all available training data to build features. This is why Giba and I say that CV is more reliable.\n\nIf one wants to use the public LB in a reliable way to estimate private LB results, as proposed by @sjv for instance, then one must use the public LB with the same kind of lag that the private LB has. \n\nPublic LB gives feedback on how your model predicts 1 to 5 days in the future. Private LB is about predicting 6 to 16 days in the future.\n\nIf you build your lag features with data up to 2018-08-10 included for the test dates, then the public LB score gives you feedback on how your model predicts 6 to 10 days in the future. And if you increment the lag, for instance use train data up to 2018-08-05 included, then the public LB gives you feedback on how your model predicts 11 to 16 days in the future. That's how public LB should be used IMHO if one does not want to rely on a local CV.", "votes": null }, { "id": "269757", "postDate": "01/17/2018 08:25:03", "content": "In my individual models, I only kept a feature if it improved my 1-5 days local CV, 6-16 days local CV and public LB. but still there is a high variation between the models I had in private. I think the problem is not lag but the different behavior of promotions between public/private split. You can observe that if you aggregate the promotions by day in the test set. Day 16 has extreme number of promotions which probably have high ratio of zero sale promotions. So if you keep track of the public/private split in your local CV, normally you dont need to lag your submissions I think.
\n\nAn irrelevant question for @CPMP: how come you become a Discussion Grandmaster despite your haters who downvote you every time? :D
", "rawMarkdown": "In my individual models, I only kept a feature if it improved my 1-5 days local CV, 6-16 days local CV and public LB. but still there is a high variation between the models I had in private. I think the problem is not lag but the different behavior of promotions between public/private split. You can observe that if you aggregate the promotions by day in the test set. Day 16 has extreme number of promotions which probably have high ratio of zero sale promotions. So if you keep track of the public/private split in your local CV, normally you dont need to lag your submissions I think.\n\nAn irrelevant question for @CPMP: how come you become a Discussion Grandmaster despite your haters who downvote you every time? :D", "votes": null }, { "id": "269780", "postDate": "01/17/2018 09:44:37", "content": "LOL, being a hater is more tiring than being a fan, I guess they get out of steam faster ;)
\n\nBack to the main topic, our CV/Public LB/Private LB is rather consistent. I cannot prove it is because of the way we set CV, but I believe it is due to it. If people prefer to think we are lucky, fine with me ;)
", "rawMarkdown": "LOL, being a hater is more tiring than being a fan, I guess they get out of steam faster ;)\n\nBack to the main topic, our CV/Public LB/Private LB is rather consistent. I cannot prove it is because of the way we set CV, but I believe it is due to it. If people prefer to think we are lucky, fine with me ;)", "votes": null }, { "id": "269972", "postDate": "01/17/2018 15:48:46", "content": "\n\n\nIn my individual models, I only kept a feature if it improved my 1-5 days local CV, 6-16 days local CV and public LB. but still there is a high variation between the models I had in private. I think the problem is not lag
\n
I concur with this.
", "rawMarkdown": "> In my individual models, I only kept a feature if it improved my 1-5 days local CV, 6-16 days local CV and public LB. but still there is a high variation between the models I had in private. I think the problem is not lag\n\nI concur with this.", "votes": null }, { "id": "269973", "postDate": "01/17/2018 15:50:18", "content": ">Back to the main topic, our CV/Public LB/Private LB is rather consistent. I cannot prove it is because of the way we set CV, but I believe it is due to it.
\n\n@CPMP: May I know which validation period/method you used? I only knew from your previous comment that your validation period ended with a payday, but I don't know exactly your validation period. Thanks in advance!
", "rawMarkdown": ">Back to the main topic, our CV/Public LB/Private LB is rather consistent. I cannot prove it is because of the way we set CV, but I believe it is due to it.\n\n@CPMP: May I know which validation period/method you used? I only knew from your previous comment that your validation period ended with a payday, but I don't know exactly your validation period. Thanks in advance!", "votes": null }, { "id": "269980", "postDate": "01/17/2018 15:59:50", "content": "@Shize, It's in my writeup, 2 periods: 16 days starting on 2017-07-16, and 15 days starting on 2017-08-01. Both periods start right after a payday and end with a payday.
", "rawMarkdown": "Shize, It's in my writeup, 2 periods: 16 days starting on 2017-07-16, and 15 days starting on 2017-08-01. Both periods start right after a payday and end with a payday.", "votes": null }, { "id": "269981", "postDate": "01/17/2018 16:02:40", "content": "@CPMP: Got it, thanks! This payday factor might be what made the difference. Unfortunately, I didn't spend time investigating this payday factor.
", "rawMarkdown": "CPMP: Got it, thanks! This payday factor might be what made the difference. Unfortunately, I didn't spend time investigating this payday factor.", "votes": null }, { "id": "269995", "postDate": "01/17/2018 16:20:34", "content": "@Shize, We tried to capture payday in various ways. For instance using day of month. If you look at sales per day of month, there is clearly a strong pattern, but we could not capture it except, maybe, via the selection of validation periods. I also think that item_nbr leads to overfit as we discussed above. It certainly does for my NN, both CV and LB were worse when using it. Maybe your models are immune to it, but I'd double check if I were you.
", "rawMarkdown": "Shize, We tried to capture payday in various ways. For instance using day of month. If you look at sales per day of month, there is clearly a strong pattern, but we could not capture it except, maybe, via the selection of validation periods. I also think that item_nbr leads to overfit as we discussed above. It certainly does for my NN, both CV and LB were worse when using it. Maybe your models are immune to it, but I'd double check if I were you.", "votes": null }, { "id": "269997", "postDate": "01/17/2018 16:28:48", "content": "> I also think that item_nbr leads to overfit as we discussed above. It certainly does for my NN, both CV and LB were worse when using it. Maybe your models are immune to it, but I'd double check if I were you.
\n\n@CPMP: I didn't do experiments with adding only the item_nbr, instead, I only added all those category variable (store_nbr, item_nbr, family, class, cluster, city, state) together, and this consistently improved both my CV and LB score by about 0.001+. Maybe item_nbr was overfitting, but its negative impacts was compensated by adding other category variables (store_nbr, family, class, cluster, city, state) such that I didn't notice it, or maybe it is due to that we two used different validation periods.
", "rawMarkdown": "> I also think that item_nbr leads to overfit as we discussed above. It certainly does for my NN, both CV and LB were worse when using it. Maybe your models are immune to it, but I'd double check if I were you.\n\n@CPMP: I didn't do experiments with adding only the item_nbr, instead, I only added all those category variable (store_nbr, item_nbr, family, class, cluster, city, state) together, and this consistently improved both my CV and LB score by about 0.001+. Maybe item_nbr was overfitting, but its negative impacts was compensated by adding other category variables (store_nbr, family, class, cluster, city, state) such that I didn't notice it, or maybe it is due to that we two used different validation periods.", "votes": null }, { "id": "270949", "postDate": "01/19/2018 10:03:27", "content": "Good point about using public LB with the same kind of lag that the private LB has. But than you need different feature sets for public and for private part, much more pain=) Did you use this approach in your team?
", "rawMarkdown": "Good point about using public LB with the same kind of lag that the private LB has. But than you need different feature sets for public and for private part, much more pain=) Did you use this approach in your team?", "votes": null }, { "id": "271015", "postDate": "01/19/2018 13:30:45", "content": "No, we did not use public LB feedback in anyway. We created two different datasets with 3 different validation periods to mimic the train/test split. Yes, it is more work and more pain. I'm convinced our CV setting was more robust than what others used, but I cannot prove it of course. And some people disagree so strongly that they downvote me each time I write about it ;)
", "rawMarkdown": "No, we did not use public LB feedback in anyway. We created two different datasets with 3 different validation periods to mimic the train/test split. Yes, it is more work and more pain. I'm convinced our CV setting was more robust than what others used, but I cannot prove it of course. And some people disagree so strongly that they downvote me each time I write about it ;)", "votes": null } ], "comments": [ { "id": 269757, "author_name": "aerdem4", "author_url": "", "post_date": "01/17/2018 08:25:03", "content": "In my individual models, I only kept a feature if it improved my 1-5 days local CV, 6-16 days local CV and public LB. but still there is a high variation between the models I had in private. I think the problem is not lag but the different behavior of promotions between public/private split. You can observe that if you aggregate the promotions by day in the test set. Day 16 has extreme number of promotions which probably have high ratio of zero sale promotions. So if you keep track of the public/private split in your local CV, normally you dont need to lag your submissions I think.
\n\nAn irrelevant question for @CPMP: how come you become a Discussion Grandmaster despite your haters who downvote you every time? :D
", "votes": null, "replies": [ { "id": 269780, "author_name": "cpmpml", "author_url": "", "post_date": "01/17/2018 09:44:37", "content": "LOL, being a hater is more tiring than being a fan, I guess they get out of steam faster ;)
\n\nBack to the main topic, our CV/Public LB/Private LB is rather consistent. I cannot prove it is because of the way we set CV, but I believe it is due to it. If people prefer to think we are lucky, fine with me ;)
", "votes": null, "replies": [] }, { "id": 269972, "author_name": "sushize", "author_url": "", "post_date": "01/17/2018 15:48:46", "content": "\n\n\nIn my individual models, I only kept a feature if it improved my 1-5 days local CV, 6-16 days local CV and public LB. but still there is a high variation between the models I had in private. I think the problem is not lag
\n
I concur with this.
", "votes": null, "replies": [] }, { "id": 269973, "author_name": "sushize", "author_url": "", "post_date": "01/17/2018 15:50:18", "content": ">Back to the main topic, our CV/Public LB/Private LB is rather consistent. I cannot prove it is because of the way we set CV, but I believe it is due to it.
\n\n@CPMP: May I know which validation period/method you used? I only knew from your previous comment that your validation period ended with a payday, but I don't know exactly your validation period. Thanks in advance!
", "votes": null, "replies": [] }, { "id": 269980, "author_name": "cpmpml", "author_url": "", "post_date": "01/17/2018 15:59:50", "content": "@Shize, It's in my writeup, 2 periods: 16 days starting on 2017-07-16, and 15 days starting on 2017-08-01. Both periods start right after a payday and end with a payday.
", "votes": null, "replies": [] }, { "id": 269981, "author_name": "sushize", "author_url": "", "post_date": "01/17/2018 16:02:40", "content": "@CPMP: Got it, thanks! This payday factor might be what made the difference. Unfortunately, I didn't spend time investigating this payday factor.
", "votes": null, "replies": [] }, { "id": 269995, "author_name": "cpmpml", "author_url": "", "post_date": "01/17/2018 16:20:34", "content": "@Shize, We tried to capture payday in various ways. For instance using day of month. If you look at sales per day of month, there is clearly a strong pattern, but we could not capture it except, maybe, via the selection of validation periods. I also think that item_nbr leads to overfit as we discussed above. It certainly does for my NN, both CV and LB were worse when using it. Maybe your models are immune to it, but I'd double check if I were you.
", "votes": null, "replies": [] }, { "id": 269997, "author_name": "sushize", "author_url": "", "post_date": "01/17/2018 16:28:48", "content": "> I also think that item_nbr leads to overfit as we discussed above. It certainly does for my NN, both CV and LB were worse when using it. Maybe your models are immune to it, but I'd double check if I were you.
\n\n@CPMP: I didn't do experiments with adding only the item_nbr, instead, I only added all those category variable (store_nbr, item_nbr, family, class, cluster, city, state) together, and this consistently improved both my CV and LB score by about 0.001+. Maybe item_nbr was overfitting, but its negative impacts was compensated by adding other category variables (store_nbr, family, class, cluster, city, state) such that I didn't notice it, or maybe it is due to that we two used different validation periods.
", "votes": null, "replies": [] } ] }, { "id": 270949, "author_name": "antklen", "author_url": "", "post_date": "01/19/2018 10:03:27", "content": "Good point about using public LB with the same kind of lag that the private LB has. But than you need different feature sets for public and for private part, much more pain=) Did you use this approach in your team?
", "votes": null, "replies": [ { "id": 271015, "author_name": "cpmpml", "author_url": "", "post_date": "01/19/2018 13:30:45", "content": "No, we did not use public LB feedback in anyway. We created two different datasets with 3 different validation periods to mimic the train/test split. Yes, it is more work and more pain. I'm convinced our CV setting was more robust than what others used, but I cannot prove it of course. And some people disagree so strongly that they downvote me each time I write about it ;)
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "269747": "Lots of discussion about how to deal with onpromotion when one expands the train set (I don't want to call it a bias, probably because I'm an ignorant ;) ), but very little discussion of the public LB vs private LB. There is a difference in nature between the two, if one uses all available training data to build features. This is why Giba and I say that CV is more reliable.\n\nIf one wants to use the public LB in a reliable way to estimate private LB results, as proposed by @sjv for instance, then one must use the public LB with the same kind of lag that the private LB has. \n\nPublic LB gives feedback on how your model predicts 1 to 5 days in the future. Private LB is about predicting 6 to 16 days in the future.\n\nIf you build your lag features with data up to 2018-08-10 included for the test dates, then the public LB score gives you feedback on how your model predicts 6 to 10 days in the future. And if you increment the lag, for instance use train data up to 2018-08-05 included, then the public LB gives you feedback on how your model predicts 11 to 16 days in the future. That's how public LB should be used IMHO if one does not want to rely on a local CV.", "269757": "In my individual models, I only kept a feature if it improved my 1-5 days local CV, 6-16 days local CV and public LB. but still there is a high variation between the models I had in private. I think the problem is not lag but the different behavior of promotions between public/private split. You can observe that if you aggregate the promotions by day in the test set. Day 16 has extreme number of promotions which probably have high ratio of zero sale promotions. So if you keep track of the public/private split in your local CV, normally you dont need to lag your submissions I think.\n\nAn irrelevant question for @CPMP: how come you become a Discussion Grandmaster despite your haters who downvote you every time? :D", "269780": "LOL, being a hater is more tiring than being a fan, I guess they get out of steam faster ;)\n\nBack to the main topic, our CV/Public LB/Private LB is rather consistent. I cannot prove it is because of the way we set CV, but I believe it is due to it. If people prefer to think we are lucky, fine with me ;)", "269972": "> In my individual models, I only kept a feature if it improved my 1-5 days local CV, 6-16 days local CV and public LB. but still there is a high variation between the models I had in private. I think the problem is not lag\n\nI concur with this.", "269973": ">Back to the main topic, our CV/Public LB/Private LB is rather consistent. I cannot prove it is because of the way we set CV, but I believe it is due to it.\n\n@CPMP: May I know which validation period/method you used? I only knew from your previous comment that your validation period ended with a payday, but I don't know exactly your validation period. Thanks in advance!", "269980": "Shize, It's in my writeup, 2 periods: 16 days starting on 2017-07-16, and 15 days starting on 2017-08-01. Both periods start right after a payday and end with a payday.", "269981": "CPMP: Got it, thanks! This payday factor might be what made the difference. Unfortunately, I didn't spend time investigating this payday factor.", "269995": "Shize, We tried to capture payday in various ways. For instance using day of month. If you look at sales per day of month, there is clearly a strong pattern, but we could not capture it except, maybe, via the selection of validation periods. I also think that item_nbr leads to overfit as we discussed above. It certainly does for my NN, both CV and LB were worse when using it. Maybe your models are immune to it, but I'd double check if I were you.", "269997": "> I also think that item_nbr leads to overfit as we discussed above. It certainly does for my NN, both CV and LB were worse when using it. Maybe your models are immune to it, but I'd double check if I were you.\n\n@CPMP: I didn't do experiments with adding only the item_nbr, instead, I only added all those category variable (store_nbr, item_nbr, family, class, cluster, city, state) together, and this consistently improved both my CV and LB score by about 0.001+. Maybe item_nbr was overfitting, but its negative impacts was compensated by adding other category variables (store_nbr, family, class, cluster, city, state) such that I didn't notice it, or maybe it is due to that we two used different validation periods.", "270949": "Good point about using public LB with the same kind of lag that the private LB has. But than you need different feature sets for public and for private part, much more pain=) Did you use this approach in your team?", "271015": "No, we did not use public LB feedback in anyway. We created two different datasets with 3 different validation periods to mimic the train/test split. Yes, it is more work and more pain. I'm convinced our CV setting was more robust than what others used, but I cannot prove it of course. And some people disagree so strongly that they downvote me each time I write about it ;)" }, "source": "meta" }