![\"count](\"http://5047-presscdn.pagely.netdna-cdn.com/wp-content/uploads/2016/02/rossmann2_2-e1454538642298.png\")
As promised in the post https://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/47326 , now I am sharing my 10 days journey experience in this competition and my approach^_^
\n\nIn addition to sharing my approach, I am also most interested in discussing why there is such a big shake-up on the private LB, and I would like to hear your thoughts and/or findings as well^_^
\n\n(Warning: This is going to be a long post, hope you have some patience^_^)
\n\nAs I mentioned in that post, I entered this competition quite late (when there was only 10 days left). At that time I just finished the last master only competition and took a short break. Since I had some spare time, I planed to explore this competition a bit, simply for fun. Finally, I am ranked 8th place on the public LB (score 0.503), but unfortunately shaked down a bit on the private LB (score 0.516). But In general, I am quite happy and satisfied with the score I achieved in this competition, given that I entered this competition so late.
\n\nBefore introducing the flowchart of my approach, I want to talk a bit about the CV_LB relationship as well as the final private LB shake-up. I would also love to hear your thoughts and findings on this.
\n\nAs I mentioned in the previous post https://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/47326 , my local CV score and public LB score correlates somewhat reasonably well (especially when comparing same type of models, e.g., comparing different feature engineering ideas for lgb models, or comparing different feature engineering ideas for nn models, for the comparison between nn and lgb models the CV-LB gap was indeed a bit different (nn showed significantly better cv score thank lgb on the similar feature set, while performed slightly worse than lgb on the LB)), which enabled me make progress somewhat smoothly on the public LB and climbed to 8th place on the public LB in the limited 10 days. The final public LB-private LB shake up is much larger than I expected given my somewhat reasonably good CV-LB relationship, and I have't fully figure out why. I did expected some extent of medium shaking up between public and private LB ranking, but definitely not as large as what it turned out to be. I understood that the promotion information is biased between the train and test set, and feature engineering efforts based on such promotion information might be easily over-fitting if not dealt with very carefully, and I guess many competitors relied on feature engineering ideas based on such promotion features to improve the public LB score, which might possibly to some extent explain the big shake-up. But I have no a certain idea on this since I don't have full knowledge about how other competitor did when getting their public LB score.
\n\nOn my side, I did almost zero additional feature engineering efforts based on the promotion variable (except those already used in the shared lgb starter script), and my improvement of public LB score from 0.513 (lgb starter script score) to 0.505 for a single lgb relied on feature engineering efforts that is not related to promotion variable at all, and my local CV-Pub LB relationship is somewhat consistent, though not perfect. I am a bit confused why the CV-LB relationship for my models on the private LB is much less correlated, even if I almost didn't use any additional promotion-based features (compared with the lgb starter script) to achieve my score improvements in CV and public LB. Ironically, it turned out that my 1st entry of this competition, which is a lgb scored 0.511 on public LB and 0.517 on private LB, is my best single model on the private LB. In comparison, my final best single lgb on the public LB (scored 0.505 on public LB) also only gave 0.517 on private LB (and slightly worse than my 1st entry lgb). Namely, all my added feature engineering efforts (which are not related to promotion variable at all) improved both my local CV and public LB score significantly and somewhat consistently, turned out to adding no extra values (and even some small negative effects) to the private LB score, in comparison with my 1st entry lgb in this competition. Bias in promotion variable in train and test set can to some extent explain the shake up on public/private LB for many competitors if they relied on promotion-based features to climb the public LB, but definitely still could not explain many other scenarios, such as my case here that almost no promotion-based engineered features are used to achieve CV/Public LB score improvement, and CV_Public LB patterns are somewhat consistent.
\n\nI would really like to hear the thoughts and experiences from other competitors to understand what actually caused such phenomenon. In addition to the promotion variable bias, are there anything else important we missed here that contributed to the shaking up? Or it is just due to the randomness /unstable time series trends, and the shaking is simply due to luck? If many of us are happy to share our thoughts and findings, maybe we would finally be able to figure out what would be the root cause for such a big shake-up and understand this problem much better, rather than simply jumping to the conclusion that the shaking up is due to lacking of comprehensive cv experiments, or bias for promotion variable.
\n\nNow let me start introducing my approach.
\n\nFirst of all, credits are given to those people who shared the great starter scripts in the kernel, especially to @tunguz, @ceshine. Your starter scripts (see links below) gave me a great starting point in this competition, and enable me to quickly work on key feature engineering and modeling experiments for fast progression in this competition. Actually, these shared great starter scripts is also one major motivation for me to enter this competition when there is only 10 days left; usually I will only enter a competition when there is much more time left.
\n\nhttps://www.kaggle.com/tunguz/lgbm-one-step-ahead-lb-0-513\nhttps://www.kaggle.com/ceshine/lgbm-starter
\n\nSample Data Selection:
\n\nMy final models used only 2017 data to extract features and construct samples. I tried to include more data, such as those 2016-Jul and 2016-Aug data (similar month/week periods as my validation period and LB period) to capture the monthly dynamics, but they only helped marginally or not helpful at all, depending on the models, while take significant longer training time. So I decided to drop those data from my final models.
\n\n*train data：*20170503- 20170719 for cv experiments, and 20170503-20170809 for LB experiments (namely, for LB submissions I retrained the models adding the most recent days data since they are closest to the LB period and might be most valuable)
\n\nvalidation data: 16 days period, 20170726 - 20170810 (and also tried using validation period 20160817-20160901 for some of my nn models at the very end of the competition, but just some simple trial since no much time left for me, and it turned out to perform worse than using 20170726 - 20170810 ). I computed the CV score for the first 5 days, last 11 days, and the full 16 days (simulating the LB split). I observe somewhat consistent CV-LB score improvement trends in all of my experiments, especially when comparing same type of models (e.g., comparing different lgbs, or comparing different nns). For my feature engineering efforts, I even observed even larger cv score improvement for the last 11 days (similar to the private LB) than the cv score improvement for the first 5 days (similar to the public LB), and thus at some time points neard the end of the competition, I once believed that I probably would climbed up a couple of spots on the private LB, due to such CV-LB score findings. Unfortunately, it didn't work as nice as I expected and I was shaked down a little bit. I still don't understand why for this.....
\n\nSince starting date 2017/08/16 for LB period is Wednesday, and thus it is essential to also set the starting date for the validation period as Wednesday to preserve the DayOfWeek pattern.
\n\nOne key trick to make my local CV and public LB score more consistent is as follows:
\n\nStep I: I extract all the store_item combinations that appeared in the period from 2017-07-01 to 2017-08-15.
\n\nStep II: I only keep the training data records (and drop all other records) whose store_item combination appeared in the extracted store_item combinations in Step I. The underlying assumption is that, if a item was not sold (or with identical 0 sales) at all in a store in the period 2017-07-01 to 2017-08-15, it is very likely that it wouldn't be sold (or with identical 0 sales) in the LB period 2017-08-16 to 2017-08-31 either. By looking at the train data, it seems to me that the new store_item combinations are added very slowly in train set, and when considering a short period like 16 days, the amount of newly added \nstore_item combination (with non_zero sales) is almost negiligible.
\n\nStep III: I also only keep the test data records whose store_item combination appeared in the extracted store_item combinations in Step I. I always predict zero target values for the other store_item combination that don't belong to the store_item combinations extracted in Step I.
\n\n(My such validation trick was also partially inspired by the Kaggle Admin Inversion's feedback on the unseen new items in this post , who mentioned that \"I'm jumping in here late, regarding the discussion of new products, in order to clarify a bit. As has been mentioned, there are a very small number of new items that were introduced in the test set. That does not explain the apparent high number of new items seen on the first day of the test set. The reason for this is that the training set does not include records for zero sales. The test set, though, includes all store / item combinations, whether or not that item was seen previously in a store.\" Thus I made an assumption that \"apparent high number of new items seen on the first day of the test set\" are mostly items that have been seen previously in a store but just with zero sales, while the additional \"small number of new items that were introduced in the test set\" are those new store_item combinations if we compare later date of the test set and the first date of the test set. Since this amount is small, I assumed that it is safe to simply always predict zero target values for all those store_item combination that don't belong to the store_item combinations extracted in Step I. And such trick indeed made my CV-public LB relationship reasonably consistent, and I thought that I got it. But I don't know why the same consistency didn't preserve well on the private LB.....)
\n\nAfter this 3-step trick and using the 16 days validation period 20170726 - 20170810, (and train period 20170503-20170719), I observed somewhat reasonably consistent CV_LB relationships when testing my feature engineering ideas, especially when comparing same type models (comparing different lgbs, or compering different nns), and it helped me a lot to make smooth progress on the public LB.
\n\nFeature Engineering:
\n\nJust follow the lgb starter scripts and adding more past sales (and DayOfWeek sales) average/ (weekly average) features . I also add (mean, median, std, sum, max, min) for many of such past sales average/ (weekly average) features. This basically gave my 1st entry submission of this competition, which scored 0.511 on the public LB and 0.517 on the private LB. It seems to me that this part didn't break the CV-Public LB-Private LB relationship yet.
Adding a little bit more promotion features similar to the starter lgb script (similar to those \"promo_14_2017\" , \"promo_60_2017\":, \"promo_140_2017\" type features, but just used more periods like 7, 21, 35, etc). This improved both the CV and public LB score by about 0.0015. But private LB score dropped from 0.517 to 0.518. I guess bias in promotion variable might to some extent explain such broken in CV-LB score consistency, but can't fully explain it, since CV_public LB consistency is still preserved by adding these promotion based features. These several features are the only additional promotion-based features that I used in my final publicLB 0.505 lgb. From now on, all the feature engineering efforts that improved my public LB score from 0.509 to 0.505 (and also improved my CV score in a similar scale) are all not related to promotion variable at all.
Adding the category features from stores.csv and items.csv, and apply label encoding. This improved both the CV and Public LB score by about 0.001+, to get 0.508 public LB score for a single lgb. Private LB score stay the same at 0.518. Not sure why, such category feature should not likely to break CV-LB relationship seems they are no biased. Is that possible this is simply due to that the time series trend pattern in the LB period is not very stable, and luck played a significant role? Or anything important we missed here?
Adding statistical features (or \"target_encoding\" like features for category variables): I use some methods to stat some targets for different keys in different time windows (this is somewhat similar to that one shared the top 1 place solution, with some additional difference being the choice of target and key, as follows:)
key：'store_nbr', 'item_nbr','family', 'class', 'city','state', 'type', 'cluster' (i.e., all the used category variables)
\n\ntarget: I output the lgb feature importance for the above lgb and selecting those most important past sales average/ (weekly average) features & DayOfWeek features, as well as the associated stats features (std, sum, median, etc.). In particular, the complete list of my chosen \"target\" features are as follows: \n'mean_7_2017_01', 'mean_14_2017_01', 'mean_21_2017_01', 'mean_35_2017_01', 'mean_30_2017_01', 'mean_60_2017_01','ahead7_6', 'ahead7_5', 'ahead7_4', 'ahead7_3', 'ahead7_2', 'ahead7_1', 'ahead0_6','ahead0_5', 'ahead0_4', 'ahead0_3', 'ahead0_2', 'ahead0_1', \n 'mean_4_dow0_2017', 'mean_8_dow0_2017', 'mean_16_dow0_2017', 'mean_4_dow1_2017', 'mean_8_dow1_2017', 'mean_16_dow1_2017', 'mean_4_dow2_2017', 'mean_8_dow2_2017', 'mean_16_dow2_2017', 'mean_4_dow3_2017', 'mean_8_dow4_2017', 'mean_16_dow3_2017','mean_4_dow4_2017', 'mean_8_dow4_2017', 'mean_16_dow4_2017','mean_4_dow5_2017', 'mean_8_dow5_2017', 'mean_16_dow5_2017',\n'mean_4_dow6_2017', 'mean_8_dow6_2017', 'mean_16_dow6_2017', 'day_1_2017','day_2_2017','day_3_2017','day_4_2017','day_5_2017','day_6_2017','day_7_2017'
\n\nmethod:\nmean, median, max, min, std
\n\nNote that, in order to avoiding possible future target information leak, such statistical features were generated independent for each week of the training data, validation data, and test data.
\n\nSo in total there are about 1400+ such statistical features, and expand my full feature set size to about 2300+. I then run lgb and output the feature importance, and only select the most important 400 features and use them for my later modeling (for lgb and nn). Such feature selection reduce the training time significantly, and also improved CV and public LB score by about 0.0006 simply by itself.
\n\nSuch statistical features are very helpful for improving my CV and public LB score. In particular, it improved my single lgb score from 0.508 on public LB to 0.505 on the public LB, and similar scale of improvement is observed in local CV. However, the private LB score only improved slightly from 0.518 to 0.517.
\n\nMy nn models were basically trained on the same feature set as my best single lgb, except that I replacing the label-encoding category features by one-hot encoding (i.e., dummie features) category features, since label encoding of category variables doesn't make much sense for models like nn. My best single nn scored 0.506 on public LB & 0.520 on private LB, and an average of two nn with some small difference gave me 0.505 score on the public LB and 0.518 on the private LB. Note that I have only explored nn model when there is only 2 days left (after I am almost done for lgb model and feature engineering) and there is no much time for me to tune the nn though. I just made it somehow work.
\n\nEnsemble of my best lgb and nn gave my final score, 0.503 on public LB and 0.516 on private LB. The ensembling step of nn and lgb does seem to preserve the CV_Public LB_Private LB relationship, namely, they improved the score all by about 0.002.
\n\nTo sum up, based on my experience, what seems to preserve the CV_Public_Private score relationships are:
\n\na) the feature engineering effort 1 described above (adding more past sales weekly average, DayOfWeek sales, as well as the associated mean, std, min, max, sum stats features, this preserve CV-Public-Private relationship nicely)
\n\nb) feature engineering effort 3 described above (\"target_encoding\" like statistical features for category variables, this preserves the CV_Pub_Private relationship somehow ok, but not perfect, CV_Pub score improved by about 0.003, private LB score improved by about 0.001),
\n\nc) ensemble of nn and lgb.
\n\nWhat seems to not preserving the CV_Public_Private score relationships are:
\n\na) the feature engineering effort 2 described above (i.e., adding some more promotion-based features similar as the lgb starter scripts, i.e., similar to \"promo_14_2017\" , \"promo_60_2017\":, \n\"promo_140_2017\" type features, but just used more periods like 7, 21, 35, etc). This might be somehow understood due to the bias in promotion variable.
\n\nb) Adding the category features from stores.csv and items.csv, and apply label encoding. This one didn't preserve the CV_Pub_Private relationship makes me completely confused, since they are not biased between CV_Pub_Private. And note the the CV_Pub score relationship is still nicely preserved when adding such category variables, not sure why it didn't work on private LB.
\n\nc) for the feature engineering effort 4 described above (\"target_encoding\" like statistical features for category variables), they improved both CV and public LB score by about 0.003+, but only improved private LB score by about 0.001 also make me a bit confused. Such features are not related to the promotion variable at all, and why it broke the CV_Private LB relationship (given that the CV_Public LB was still nicely preserved)?
\n\nSingle Model:
\n\npublic LB 0.505/ private LB 0.517, single lgb
\n\npublic LB 0.506/ private LB 0.520, single nn
\n\naverage of 2 nn with small differences gave public LB 0.505/ private LB 0.518.
\n\nFinal Ensemble:
\n\n0.54-0.46 weighted average of lgb and nn, public LB 0.503/ private LB 0.516.
\n\nThat's all for my approach in this 10 days journey of this competition. Overall speaking, I am satisfied with what I achieved in this competition given the limited time, and I have learned a couple of useful things from competitors and my experiments when addressing this time series challenge. To the end, I would really love to hear about your thoughts and findings about what preserves the CV_LB relationships and what didn't preserve the CV_LB relationships, such as sth similar to what I shared above. For now I still couldn't fully understand why the relationship is not preserved on the private LB, given that CV_Pub LB relationship is somewhat consistent for me. By combining our findings, maybe we could have a deep understanding about why there would be such a big shake up on the private LB, rather than simply jumping to the conclusion that it is due to lacking of comprehensive CV experiments or just luck/randomness. Maybe we missed sth important but subtle, which contributed to the big shake-up on the final private LB, and if we figured out that, maybe there is indeed a nice way to preserve the CV_Pub_Private score relationship very well if done appropriately.
\n\nFinally, thanks to Kaggle and the sponsor to host such a great competition, and thanks to all other competitors who made my journey in this competition fun! And, congratulation to the winners!
\n\nBest regards,
\n\nShize
", "messages": [ { "id": "269468", "postDate": "01/16/2018 22:04:41", "content": "As promised in the post https://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/47326 , now I am sharing my 10 days journey experience in this competition and my approach^_^
\n\nIn addition to sharing my approach, I am also most interested in discussing why there is such a big shake-up on the private LB, and I would like to hear your thoughts and/or findings as well^_^
\n\n(Warning: This is going to be a long post, hope you have some patience^_^)
\n\nAs I mentioned in that post, I entered this competition quite late (when there was only 10 days left). At that time I just finished the last master only competition and took a short break. Since I had some spare time, I planed to explore this competition a bit, simply for fun. Finally, I am ranked 8th place on the public LB (score 0.503), but unfortunately shaked down a bit on the private LB (score 0.516). But In general, I am quite happy and satisfied with the score I achieved in this competition, given that I entered this competition so late.
\n\nBefore introducing the flowchart of my approach, I want to talk a bit about the CV_LB relationship as well as the final private LB shake-up. I would also love to hear your thoughts and findings on this.
\n\nAs I mentioned in the previous post https://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/47326 , my local CV score and public LB score correlates somewhat reasonably well (especially when comparing same type of models, e.g., comparing different feature engineering ideas for lgb models, or comparing different feature engineering ideas for nn models, for the comparison between nn and lgb models the CV-LB gap was indeed a bit different (nn showed significantly better cv score thank lgb on the similar feature set, while performed slightly worse than lgb on the LB)), which enabled me make progress somewhat smoothly on the public LB and climbed to 8th place on the public LB in the limited 10 days. The final public LB-private LB shake up is much larger than I expected given my somewhat reasonably good CV-LB relationship, and I have't fully figure out why. I did expected some extent of medium shaking up between public and private LB ranking, but definitely not as large as what it turned out to be. I understood that the promotion information is biased between the train and test set, and feature engineering efforts based on such promotion information might be easily over-fitting if not dealt with very carefully, and I guess many competitors relied on feature engineering ideas based on such promotion features to improve the public LB score, which might possibly to some extent explain the big shake-up. But I have no a certain idea on this since I don't have full knowledge about how other competitor did when getting their public LB score.
\n\nOn my side, I did almost zero additional feature engineering efforts based on the promotion variable (except those already used in the shared lgb starter script), and my improvement of public LB score from 0.513 (lgb starter script score) to 0.505 for a single lgb relied on feature engineering efforts that is not related to promotion variable at all, and my local CV-Pub LB relationship is somewhat consistent, though not perfect. I am a bit confused why the CV-LB relationship for my models on the private LB is much less correlated, even if I almost didn't use any additional promotion-based features (compared with the lgb starter script) to achieve my score improvements in CV and public LB. Ironically, it turned out that my 1st entry of this competition, which is a lgb scored 0.511 on public LB and 0.517 on private LB, is my best single model on the private LB. In comparison, my final best single lgb on the public LB (scored 0.505 on public LB) also only gave 0.517 on private LB (and slightly worse than my 1st entry lgb). Namely, all my added feature engineering efforts (which are not related to promotion variable at all) improved both my local CV and public LB score significantly and somewhat consistently, turned out to adding no extra values (and even some small negative effects) to the private LB score, in comparison with my 1st entry lgb in this competition. Bias in promotion variable in train and test set can to some extent explain the shake up on public/private LB for many competitors if they relied on promotion-based features to climb the public LB, but definitely still could not explain many other scenarios, such as my case here that almost no promotion-based engineered features are used to achieve CV/Public LB score improvement, and CV_Public LB patterns are somewhat consistent.
\n\nI would really like to hear the thoughts and experiences from other competitors to understand what actually caused such phenomenon. In addition to the promotion variable bias, are there anything else important we missed here that contributed to the shaking up? Or it is just due to the randomness /unstable time series trends, and the shaking is simply due to luck? If many of us are happy to share our thoughts and findings, maybe we would finally be able to figure out what would be the root cause for such a big shake-up and understand this problem much better, rather than simply jumping to the conclusion that the shaking up is due to lacking of comprehensive cv experiments, or bias for promotion variable.
\n\nNow let me start introducing my approach.
\n\nFirst of all, credits are given to those people who shared the great starter scripts in the kernel, especially to @tunguz, @ceshine. Your starter scripts (see links below) gave me a great starting point in this competition, and enable me to quickly work on key feature engineering and modeling experiments for fast progression in this competition. Actually, these shared great starter scripts is also one major motivation for me to enter this competition when there is only 10 days left; usually I will only enter a competition when there is much more time left.
\n\nhttps://www.kaggle.com/tunguz/lgbm-one-step-ahead-lb-0-513\nhttps://www.kaggle.com/ceshine/lgbm-starter
\n\nSample Data Selection:
\n\nMy final models used only 2017 data to extract features and construct samples. I tried to include more data, such as those 2016-Jul and 2016-Aug data (similar month/week periods as my validation period and LB period) to capture the monthly dynamics, but they only helped marginally or not helpful at all, depending on the models, while take significant longer training time. So I decided to drop those data from my final models.
\n\n*train data：*20170503- 20170719 for cv experiments, and 20170503-20170809 for LB experiments (namely, for LB submissions I retrained the models adding the most recent days data since they are closest to the LB period and might be most valuable)
\n\nvalidation data: 16 days period, 20170726 - 20170810 (and also tried using validation period 20160817-20160901 for some of my nn models at the very end of the competition, but just some simple trial since no much time left for me, and it turned out to perform worse than using 20170726 - 20170810 ). I computed the CV score for the first 5 days, last 11 days, and the full 16 days (simulating the LB split). I observe somewhat consistent CV-LB score improvement trends in all of my experiments, especially when comparing same type of models (e.g., comparing different lgbs, or comparing different nns). For my feature engineering efforts, I even observed even larger cv score improvement for the last 11 days (similar to the private LB) than the cv score improvement for the first 5 days (similar to the public LB), and thus at some time points neard the end of the competition, I once believed that I probably would climbed up a couple of spots on the private LB, due to such CV-LB score findings. Unfortunately, it didn't work as nice as I expected and I was shaked down a little bit. I still don't understand why for this.....
\n\nSince starting date 2017/08/16 for LB period is Wednesday, and thus it is essential to also set the starting date for the validation period as Wednesday to preserve the DayOfWeek pattern.
\n\nOne key trick to make my local CV and public LB score more consistent is as follows:
\n\nStep I: I extract all the store_item combinations that appeared in the period from 2017-07-01 to 2017-08-15.
\n\nStep II: I only keep the training data records (and drop all other records) whose store_item combination appeared in the extracted store_item combinations in Step I. The underlying assumption is that, if a item was not sold (or with identical 0 sales) at all in a store in the period 2017-07-01 to 2017-08-15, it is very likely that it wouldn't be sold (or with identical 0 sales) in the LB period 2017-08-16 to 2017-08-31 either. By looking at the train data, it seems to me that the new store_item combinations are added very slowly in train set, and when considering a short period like 16 days, the amount of newly added \nstore_item combination (with non_zero sales) is almost negiligible.
\n\nStep III: I also only keep the test data records whose store_item combination appeared in the extracted store_item combinations in Step I. I always predict zero target values for the other store_item combination that don't belong to the store_item combinations extracted in Step I.
\n\n(My such validation trick was also partially inspired by the Kaggle Admin Inversion's feedback on the unseen new items in this post , who mentioned that \"I'm jumping in here late, regarding the discussion of new products, in order to clarify a bit. As has been mentioned, there are a very small number of new items that were introduced in the test set. That does not explain the apparent high number of new items seen on the first day of the test set. The reason for this is that the training set does not include records for zero sales. The test set, though, includes all store / item combinations, whether or not that item was seen previously in a store.\" Thus I made an assumption that \"apparent high number of new items seen on the first day of the test set\" are mostly items that have been seen previously in a store but just with zero sales, while the additional \"small number of new items that were introduced in the test set\" are those new store_item combinations if we compare later date of the test set and the first date of the test set. Since this amount is small, I assumed that it is safe to simply always predict zero target values for all those store_item combination that don't belong to the store_item combinations extracted in Step I. And such trick indeed made my CV-public LB relationship reasonably consistent, and I thought that I got it. But I don't know why the same consistency didn't preserve well on the private LB.....)
\n\nAfter this 3-step trick and using the 16 days validation period 20170726 - 20170810, (and train period 20170503-20170719), I observed somewhat reasonably consistent CV_LB relationships when testing my feature engineering ideas, especially when comparing same type models (comparing different lgbs, or compering different nns), and it helped me a lot to make smooth progress on the public LB.
\n\nFeature Engineering:
\n\nJust follow the lgb starter scripts and adding more past sales (and DayOfWeek sales) average/ (weekly average) features . I also add (mean, median, std, sum, max, min) for many of such past sales average/ (weekly average) features. This basically gave my 1st entry submission of this competition, which scored 0.511 on the public LB and 0.517 on the private LB. It seems to me that this part didn't break the CV-Public LB-Private LB relationship yet.
Adding a little bit more promotion features similar to the starter lgb script (similar to those \"promo_14_2017\" , \"promo_60_2017\":, \"promo_140_2017\" type features, but just used more periods like 7, 21, 35, etc). This improved both the CV and public LB score by about 0.0015. But private LB score dropped from 0.517 to 0.518. I guess bias in promotion variable might to some extent explain such broken in CV-LB score consistency, but can't fully explain it, since CV_public LB consistency is still preserved by adding these promotion based features. These several features are the only additional promotion-based features that I used in my final publicLB 0.505 lgb. From now on, all the feature engineering efforts that improved my public LB score from 0.509 to 0.505 (and also improved my CV score in a similar scale) are all not related to promotion variable at all.
Adding the category features from stores.csv and items.csv, and apply label encoding. This improved both the CV and Public LB score by about 0.001+, to get 0.508 public LB score for a single lgb. Private LB score stay the same at 0.518. Not sure why, such category feature should not likely to break CV-LB relationship seems they are no biased. Is that possible this is simply due to that the time series trend pattern in the LB period is not very stable, and luck played a significant role? Or anything important we missed here?
Adding statistical features (or \"target_encoding\" like features for category variables): I use some methods to stat some targets for different keys in different time windows (this is somewhat similar to that one shared the top 1 place solution, with some additional difference being the choice of target and key, as follows:)
key：'store_nbr', 'item_nbr','family', 'class', 'city','state', 'type', 'cluster' (i.e., all the used category variables)
\n\ntarget: I output the lgb feature importance for the above lgb and selecting those most important past sales average/ (weekly average) features & DayOfWeek features, as well as the associated stats features (std, sum, median, etc.). In particular, the complete list of my chosen \"target\" features are as follows: \n'mean_7_2017_01', 'mean_14_2017_01', 'mean_21_2017_01', 'mean_35_2017_01', 'mean_30_2017_01', 'mean_60_2017_01','ahead7_6', 'ahead7_5', 'ahead7_4', 'ahead7_3', 'ahead7_2', 'ahead7_1', 'ahead0_6','ahead0_5', 'ahead0_4', 'ahead0_3', 'ahead0_2', 'ahead0_1', \n 'mean_4_dow0_2017', 'mean_8_dow0_2017', 'mean_16_dow0_2017', 'mean_4_dow1_2017', 'mean_8_dow1_2017', 'mean_16_dow1_2017', 'mean_4_dow2_2017', 'mean_8_dow2_2017', 'mean_16_dow2_2017', 'mean_4_dow3_2017', 'mean_8_dow4_2017', 'mean_16_dow3_2017','mean_4_dow4_2017', 'mean_8_dow4_2017', 'mean_16_dow4_2017','mean_4_dow5_2017', 'mean_8_dow5_2017', 'mean_16_dow5_2017',\n'mean_4_dow6_2017', 'mean_8_dow6_2017', 'mean_16_dow6_2017', 'day_1_2017','day_2_2017','day_3_2017','day_4_2017','day_5_2017','day_6_2017','day_7_2017'
\n\nmethod:\nmean, median, max, min, std
\n\nNote that, in order to avoiding possible future target information leak, such statistical features were generated independent for each week of the training data, validation data, and test data.
\n\nSo in total there are about 1400+ such statistical features, and expand my full feature set size to about 2300+. I then run lgb and output the feature importance, and only select the most important 400 features and use them for my later modeling (for lgb and nn). Such feature selection reduce the training time significantly, and also improved CV and public LB score by about 0.0006 simply by itself.
\n\nSuch statistical features are very helpful for improving my CV and public LB score. In particular, it improved my single lgb score from 0.508 on public LB to 0.505 on the public LB, and similar scale of improvement is observed in local CV. However, the private LB score only improved slightly from 0.518 to 0.517.
\n\nMy nn models were basically trained on the same feature set as my best single lgb, except that I replacing the label-encoding category features by one-hot encoding (i.e., dummie features) category features, since label encoding of category variables doesn't make much sense for models like nn. My best single nn scored 0.506 on public LB & 0.520 on private LB, and an average of two nn with some small difference gave me 0.505 score on the public LB and 0.518 on the private LB. Note that I have only explored nn model when there is only 2 days left (after I am almost done for lgb model and feature engineering) and there is no much time for me to tune the nn though. I just made it somehow work.
\n\nEnsemble of my best lgb and nn gave my final score, 0.503 on public LB and 0.516 on private LB. The ensembling step of nn and lgb does seem to preserve the CV_Public LB_Private LB relationship, namely, they improved the score all by about 0.002.
\n\nTo sum up, based on my experience, what seems to preserve the CV_Public_Private score relationships are:
\n\na) the feature engineering effort 1 described above (adding more past sales weekly average, DayOfWeek sales, as well as the associated mean, std, min, max, sum stats features, this preserve CV-Public-Private relationship nicely)
\n\nb) feature engineering effort 3 described above (\"target_encoding\" like statistical features for category variables, this preserves the CV_Pub_Private relationship somehow ok, but not perfect, CV_Pub score improved by about 0.003, private LB score improved by about 0.001),
\n\nc) ensemble of nn and lgb.
\n\nWhat seems to not preserving the CV_Public_Private score relationships are:
\n\na) the feature engineering effort 2 described above (i.e., adding some more promotion-based features similar as the lgb starter scripts, i.e., similar to \"promo_14_2017\" , \"promo_60_2017\":, \n\"promo_140_2017\" type features, but just used more periods like 7, 21, 35, etc). This might be somehow understood due to the bias in promotion variable.
\n\nb) Adding the category features from stores.csv and items.csv, and apply label encoding. This one didn't preserve the CV_Pub_Private relationship makes me completely confused, since they are not biased between CV_Pub_Private. And note the the CV_Pub score relationship is still nicely preserved when adding such category variables, not sure why it didn't work on private LB.
\n\nc) for the feature engineering effort 4 described above (\"target_encoding\" like statistical features for category variables), they improved both CV and public LB score by about 0.003+, but only improved private LB score by about 0.001 also make me a bit confused. Such features are not related to the promotion variable at all, and why it broke the CV_Private LB relationship (given that the CV_Public LB was still nicely preserved)?
\n\nSingle Model:
\n\npublic LB 0.505/ private LB 0.517, single lgb
\n\npublic LB 0.506/ private LB 0.520, single nn
\n\naverage of 2 nn with small differences gave public LB 0.505/ private LB 0.518.
\n\nFinal Ensemble:
\n\n0.54-0.46 weighted average of lgb and nn, public LB 0.503/ private LB 0.516.
\n\nThat's all for my approach in this 10 days journey of this competition. Overall speaking, I am satisfied with what I achieved in this competition given the limited time, and I have learned a couple of useful things from competitors and my experiments when addressing this time series challenge. To the end, I would really love to hear about your thoughts and findings about what preserves the CV_LB relationships and what didn't preserve the CV_LB relationships, such as sth similar to what I shared above. For now I still couldn't fully understand why the relationship is not preserved on the private LB, given that CV_Pub LB relationship is somewhat consistent for me. By combining our findings, maybe we could have a deep understanding about why there would be such a big shake up on the private LB, rather than simply jumping to the conclusion that it is due to lacking of comprehensive CV experiments or just luck/randomness. Maybe we missed sth important but subtle, which contributed to the big shake-up on the final private LB, and if we figured out that, maybe there is indeed a nice way to preserve the CV_Pub_Private score relationship very well if done appropriately.
\n\nFinally, thanks to Kaggle and the sponsor to host such a great competition, and thanks to all other competitors who made my journey in this competition fun! And, congratulation to the winners!
\n\nBest regards,
\n\nShize
", "rawMarkdown": "As promised in the post https://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/47326 , now I am sharing my 10 days journey experience in this competition and my approach^_^ \n\nIn addition to sharing my approach, I am also most interested in discussing why there is such a big shake-up on the private LB, and I would like to hear your thoughts and/or findings as well^_^\n\n(Warning: This is going to be a long post, hope you have some patience^_^)\n\nAs I mentioned in that post, I entered this competition quite late (when there was only 10 days left). At that time I just finished the last master only competition and took a short break. Since I had some spare time, I planed to explore this competition a bit, simply for fun. Finally, I am ranked 8th place on the public LB (score 0.503), but unfortunately shaked down a bit on the private LB (score 0.516). But In general, I am quite happy and satisfied with the score I achieved in this competition, given that I entered this competition so late. \n\n____________________________________________________________________________________________________________________\n\n**Before introducing the flowchart of my approach, I want to talk a bit about the CV_LB relationship as well as the final private LB shake-up. I would also love to hear your thoughts and findings on this.**\n\nAs I mentioned in the previous post https://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/47326 , my local CV score and public LB score correlates somewhat reasonably well (especially when comparing same type of models, e.g., comparing different feature engineering ideas for lgb models, or comparing different feature engineering ideas for nn models, for the comparison between nn and lgb models the CV-LB gap was indeed a bit different (nn showed significantly better cv score thank lgb on the similar feature set, while performed slightly worse than lgb on the LB)), which enabled me make progress somewhat smoothly on the public LB and climbed to 8th place on the public LB in the limited 10 days. The final public LB-private LB shake up is much larger than I expected given my somewhat reasonably good CV-LB relationship, and I have't fully figure out why. I did expected some extent of medium shaking up between public and private LB ranking, but definitely not as large as what it turned out to be. I understood that the promotion information is biased between the train and test set, and feature engineering efforts based on such promotion information might be easily over-fitting if not dealt with very carefully, and I guess many competitors relied on feature engineering ideas based on such promotion features to improve the public LB score, which might possibly to some extent explain the big shake-up. But I have no a certain idea on this since I don't have full knowledge about how other competitor did when getting their public LB score. \n\nOn my side, I did almost zero additional feature engineering efforts based on the promotion variable (except those already used in the shared lgb starter script), and my improvement of public LB score from 0.513 (lgb starter script score) to 0.505 for a single lgb relied on feature engineering efforts that is not related to promotion variable at all, and my local CV-Pub LB relationship is somewhat consistent, though not perfect. I am a bit confused why the CV-LB relationship for my models on the private LB is much less correlated, even if I almost didn't use any additional promotion-based features (compared with the lgb starter script) to achieve my score improvements in CV and public LB. Ironically, it turned out that my 1st entry of this competition, which is a lgb scored 0.511 on public LB and 0.517 on private LB, is my best single model on the private LB. In comparison, my final best single lgb on the public LB (scored 0.505 on public LB) also only gave 0.517 on private LB (and slightly worse than my 1st entry lgb). Namely, all my added feature engineering efforts (which are not related to promotion variable at all) improved both my local CV and public LB score significantly and somewhat consistently, turned out to adding no extra values (and even some small negative effects) to the private LB score, in comparison with my 1st entry lgb in this competition. Bias in promotion variable in train and test set can to some extent explain the shake up on public/private LB for many competitors if they relied on promotion-based features to climb the public LB, but definitely still could not explain many other scenarios, such as my case here that almost no promotion-based engineered features are used to achieve CV/Public LB score improvement, and CV_Public LB patterns are somewhat consistent. \n\nI would really like to hear the thoughts and experiences from other competitors to understand what actually caused such phenomenon. In addition to the promotion variable bias, are there anything else important we missed here that contributed to the shaking up? Or it is just due to the randomness /unstable time series trends, and the shaking is simply due to luck? If many of us are happy to share our thoughts and findings, maybe we would finally be able to figure out what would be the root cause for such a big shake-up and understand this problem much better, rather than simply jumping to the conclusion that the shaking up is due to lacking of comprehensive cv experiments, or bias for promotion variable.\n\n___________________________________________________________________________________________________________________\n__________________________________________________________________________________________________________________\n__________________________________________________________________________________________________________________\n__________________________________________________________________________________________________________________\n\n\nNow let me start introducing my approach.\n\n\nFirst of all, credits are given to those people who shared the great starter scripts in the kernel, especially to @tunguz, @ceshine. Your starter scripts (see links below) gave me a great starting point in this competition, and enable me to quickly work on key feature engineering and modeling experiments for fast progression in this competition. Actually, these shared great starter scripts is also one major motivation for me to enter this competition when there is only 10 days left; usually I will only enter a competition when there is much more time left.\n\nhttps://www.kaggle.com/tunguz/lgbm-one-step-ahead-lb-0-513\nhttps://www.kaggle.com/ceshine/lgbm-starter\n\n__________________________________________________________________________________________________________________\n**Sample Data Selection:**\n\nMy final models used only 2017 data to extract features and construct samples. I tried to include more data, such as those 2016-Jul and 2016-Aug data (similar month/week periods as my validation period and LB period) to capture the monthly dynamics, but they only helped marginally or not helpful at all, depending on the models, while take significant longer training time. So I decided to drop those data from my final models.\n\n**train data：**20170503- 20170719 for cv experiments, and 20170503-20170809 for LB experiments (namely, for LB submissions I retrained the models adding the most recent days data since they are closest to the LB period and might be most valuable)\n\n**validation data:** 16 days period, 20170726 - 20170810 (and also tried using validation period 20160817-20160901 for some of my nn models at the very end of the competition, but just some simple trial since no much time left for me, and it turned out to perform worse than using 20170726 - 20170810 ). I computed the CV score for the first 5 days, last 11 days, and the full 16 days (simulating the LB split). I observe somewhat consistent CV-LB score improvement trends in all of my experiments, especially when comparing same type of models (e.g., comparing different lgbs, or comparing different nns). For my feature engineering efforts, I even observed even larger cv score improvement for the last 11 days (similar to the private LB) than the cv score improvement for the first 5 days (similar to the public LB), and thus at some time points neard the end of the competition, I once believed that I probably would climbed up a couple of spots on the private LB, due to such CV-LB score findings. Unfortunately, it didn't work as nice as I expected and I was shaked down a little bit. I still don't understand why for this.....\n\nSince starting date 2017/08/16 for LB period is Wednesday, and thus it is essential to also set the starting date for the validation period as Wednesday to preserve the DayOfWeek pattern.\n\n**One key trick to make my local CV and public LB score more consistent is as follows**:\n\n**Step I:** I extract all the store_item combinations that appeared in the period from 2017-07-01 to 2017-08-15. \n\n**Step II:** I only keep the training data records (and drop all other records) whose store_item combination appeared in the extracted store_item combinations in Step I. The underlying assumption is that, if a item was not sold (or with identical 0 sales) at all in a store in the period 2017-07-01 to 2017-08-15, it is very likely that it wouldn't be sold (or with identical 0 sales) in the LB period 2017-08-16 to 2017-08-31 either. By looking at the train data, it seems to me that the new store_item combinations are added very slowly in train set, and when considering a short period like 16 days, the amount of newly added \nstore_item combination (with non_zero sales) is almost negiligible.\n\n**Step III:** I also only keep the test data records whose store_item combination appeared in the extracted store_item combinations in Step I. I always predict zero target values for the other store_item combination that don't belong to the store_item combinations extracted in Step I.\n\n(My such validation trick was also partially inspired by the Kaggle Admin Inversion's feedback on the unseen new items in this post , who mentioned that \"I'm jumping in here late, regarding the discussion of new products, in order to clarify a bit. As has been mentioned, there are a very small number of new items that were introduced in the test set. That does not explain the apparent high number of new items seen on the first day of the test set. The reason for this is that the training set does not include records for zero sales. The test set, though, includes all store / item combinations, whether or not that item was seen previously in a store.\" Thus I made an assumption that \"apparent high number of new items seen on the first day of the test set\" are mostly items that have been seen previously in a store but just with zero sales, while the additional \"small number of new items that were introduced in the test set\" are those new store_item combinations if we compare later date of the test set and the first date of the test set. Since this amount is small, I assumed that it is safe to simply always predict zero target values for all those store_item combination that don't belong to the store_item combinations extracted in Step I. And such trick indeed made my CV-public LB relationship reasonably consistent, and I thought that I got it. But I don't know why the same consistency didn't preserve well on the private LB.....)\n\nAfter this 3-step trick and using the 16 days validation period 20170726 - 20170810, (and train period 20170503-20170719), I observed somewhat reasonably consistent CV_LB relationships when testing my feature engineering ideas, especially when comparing same type models (comparing different lgbs, or compering different nns), and it helped me a lot to make smooth progress on the public LB.\n\n__________________________________________________________________________________________________________________\n**Feature Engineering:**\n\n1. Just follow the lgb starter scripts and adding more past sales (and DayOfWeek sales) average/ (weekly average) features . I also add (mean, median, std, sum, max, min) for many of such past sales average/ (weekly average) features. This basically gave my 1st entry submission of this competition, which scored 0.511 on the public LB and 0.517 on the private LB. It seems to me that this part didn't break the CV-Public LB-Private LB relationship yet.\n\n2. Adding a little bit more promotion features similar to the starter lgb script (similar to those \"promo_14_2017\" , \"promo_60_2017\":, \"promo_140_2017\" type features, but just used more periods like 7, 21, 35, etc). This improved both the CV and public LB score by about 0.0015. But private LB score dropped from 0.517 to 0.518. I guess bias in promotion variable might to some extent explain such broken in CV-LB score consistency, but can't fully explain it, since CV_public LB consistency is still preserved by adding these promotion based features. These several features are the only additional promotion-based features that I used in my final publicLB 0.505 lgb. From now on, all the feature engineering efforts that improved my public LB score from 0.509 to 0.505 (and also improved my CV score in a similar scale) are all not related to promotion variable at all.\n\n3. Adding the category features from stores.csv and items.csv, and apply label encoding. This improved both the CV and Public LB score by about 0.001+, to get 0.508 public LB score for a single lgb. Private LB score stay the same at 0.518. Not sure why, such category feature should not likely to break CV-LB relationship seems they are no biased. Is that possible this is simply due to that the time series trend pattern in the LB period is not very stable, and luck played a significant role? Or anything important we missed here?\n\n4. Adding statistical features (or \"target_encoding\" like features for category variables): I use some methods to stat some targets for different keys in different time windows (this is somewhat similar to that one shared the top 1 place solution, with some additional difference being the choice of target and key, as follows:)\n\nkey：'store_nbr', 'item_nbr','family', 'class', 'city','state', 'type', 'cluster' (i.e., all the used category variables)\n\ntarget: I output the lgb feature importance for the above lgb and selecting those most important past sales average/ (weekly average) features & DayOfWeek features, as well as the associated stats features (std, sum, median, etc.). In particular, the complete list of my chosen \"target\" features are as follows: \n'mean_7_2017_01', 'mean_14_2017_01', 'mean_21_2017_01', 'mean_35_2017_01', 'mean_30_2017_01', 'mean_60_2017_01','ahead7_6', 'ahead7_5', 'ahead7_4', 'ahead7_3', 'ahead7_2', 'ahead7_1', 'ahead0_6','ahead0_5', 'ahead0_4', 'ahead0_3', 'ahead0_2', 'ahead0_1', \n 'mean_4_dow0_2017', 'mean_8_dow0_2017', 'mean_16_dow0_2017', 'mean_4_dow1_2017', 'mean_8_dow1_2017', 'mean_16_dow1_2017', 'mean_4_dow2_2017', 'mean_8_dow2_2017', 'mean_16_dow2_2017', 'mean_4_dow3_2017', 'mean_8_dow4_2017', 'mean_16_dow3_2017','mean_4_dow4_2017', 'mean_8_dow4_2017', 'mean_16_dow4_2017','mean_4_dow5_2017', 'mean_8_dow5_2017', 'mean_16_dow5_2017',\n'mean_4_dow6_2017', 'mean_8_dow6_2017', 'mean_16_dow6_2017', 'day_1_2017','day_2_2017','day_3_2017','day_4_2017','day_5_2017','day_6_2017','day_7_2017' \n\nmethod:\nmean, median, max, min, std\n\nNote that, in order to avoiding possible future target information leak, such statistical features were generated independent for each week of the training data, validation data, and test data.\n\nSo in total there are about 1400+ such statistical features, and expand my full feature set size to about 2300+. I then run lgb and output the feature importance, and only select the most important 400 features and use them for my later modeling (for lgb and nn). Such feature selection reduce the training time significantly, and also improved CV and public LB score by about 0.0006 simply by itself.\n\n\nSuch statistical features are very helpful for improving my CV and public LB score. In particular, it improved my single lgb score from 0.508 on public LB to 0.505 on the public LB, and similar scale of improvement is observed in local CV. However, the private LB score only improved slightly from 0.518 to 0.517.\n\nMy nn models were basically trained on the same feature set as my best single lgb, except that I replacing the label-encoding category features by one-hot encoding (i.e., dummie features) category features, since label encoding of category variables doesn't make much sense for models like nn. My best single nn scored 0.506 on public LB & 0.520 on private LB, and an average of two nn with some small difference gave me 0.505 score on the public LB and 0.518 on the private LB. Note that I have only explored nn model when there is only 2 days left (after I am almost done for lgb model and feature engineering) and there is no much time for me to tune the nn though. I just made it somehow work. \n\nEnsemble of my best lgb and nn gave my final score, 0.503 on public LB and 0.516 on private LB. The ensembling step of nn and lgb does seem to preserve the CV_Public LB_Private LB relationship, namely, they improved the score all by about 0.002.\n\n**To sum up, based on my experience, what seems to preserve the CV_Public_Private score relationships are**:\n\n a) the feature engineering effort 1 described above (adding more past sales weekly average, DayOfWeek sales, as well as the associated mean, std, min, max, sum stats features, this preserve CV-Public-Private relationship nicely) \n\nb) feature engineering effort 3 described above (\"target_encoding\" like statistical features for category variables, this preserves the CV_Pub_Private relationship somehow ok, but not perfect, CV_Pub score improved by about 0.003, private LB score improved by about 0.001), \n\nc) ensemble of nn and lgb. \n\n**What seems to not preserving the CV_Public_Private score relationships are:**\n\n a) the feature engineering effort 2 described above (i.e., adding some more promotion-based features similar as the lgb starter scripts, i.e., similar to \"promo_14_2017\" , \"promo_60_2017\":, \n\"promo_140_2017\" type features, but just used more periods like 7, 21, 35, etc). This might be somehow understood due to the bias in promotion variable.\n\nb) Adding the category features from stores.csv and items.csv, and apply label encoding. This one didn't preserve the CV_Pub_Private relationship makes me completely confused, since they are not biased between CV_Pub_Private. And note the the CV_Pub score relationship is still nicely preserved when adding such category variables, not sure why it didn't work on private LB.\n\nc) for the feature engineering effort 4 described above (\"target_encoding\" like statistical features for category variables), they improved both CV and public LB score by about 0.003+, but only improved private LB score by about 0.001 also make me a bit confused. Such features are not related to the promotion variable at all, and why it broke the CV_Private LB relationship (given that the CV_Public LB was still nicely preserved)?\n\n____________________________________________________________________________________________________________\n\n**Single Model:** \n\npublic LB 0.505/ private LB 0.517, single lgb\n\npublic LB 0.506/ private LB 0.520, single nn\n\naverage of 2 nn with small differences gave public LB 0.505/ private LB 0.518.\n\n__________________________________________________________________________________________________________\n\n**Final Ensemble:**\n\n0.54-0.46 weighted average of lgb and nn, public LB 0.503/ private LB 0.516.\n\n__________________________________________________________________________________________________________\n\nThat's all for my approach in this 10 days journey of this competition. Overall speaking, I am satisfied with what I achieved in this competition given the limited time, and I have learned a couple of useful things from competitors and my experiments when addressing this time series challenge. To the end, I would really love to hear about your thoughts and findings about what preserves the CV_LB relationships and what didn't preserve the CV_LB relationships, such as sth similar to what I shared above. For now I still couldn't fully understand why the relationship is not preserved on the private LB, given that CV_Pub LB relationship is somewhat consistent for me. By combining our findings, maybe we could have a deep understanding about why there would be such a big shake up on the private LB, rather than simply jumping to the conclusion that it is due to lacking of comprehensive CV experiments or just luck/randomness. Maybe we missed sth important but subtle, which contributed to the big shake-up on the final private LB, and if we figured out that, maybe there is indeed a nice way to preserve the CV_Pub_Private score relationship very well if done appropriately.\n\n__________________________________________________________________________________________________________\n\n Finally, thanks to Kaggle and the sponsor to host such a great competition, and thanks to all other competitors who made my journey in this competition fun! And, congratulation to the winners!\n\nBest regards,\n\nShize", "votes": null }, { "id": "269478", "postDate": "01/16/2018 22:37:23", "content": "Thanks for the write up! Very informative :)
", "rawMarkdown": "Thanks for the write up! Very informative :)", "votes": null }, { "id": "269532", "postDate": "01/17/2018 01:00:12", "content": "Thanks for sharing. I found that using item_nbr was leading to overfit in my local CV. This may explain part of the shakeup.
", "rawMarkdown": "Thanks for sharing. I found that using item_nbr was leading to overfit in my local CV. This may explain part of the shakeup.", "votes": null }, { "id": "269537", "postDate": "01/17/2018 01:10:22", "content": "Thanks for the info. But why? There should not be any bias for item_nbr between public LB and private LB, and thus this factor cannot explain why my CV_Pub LB relationship is consistent, but the consistency broken in private LB..... As I mentioned above, with my CV mechanism, adding item_nbr as well as other category variable improved both my local CV score and Public LB score consistently by about 0.001+, while private LB score stayed the same. This seems indeed very strange to me, and I have no clues why.... Could anyone else confirmed whether they encountered similar situations (i.e., improve both CV and public LB score consistently, but not private LB) when adding the category variables?
", "rawMarkdown": "Thanks for the info. But why? There should not be any bias for item_nbr between public LB and private LB, and thus this factor cannot explain why my CV_Pub LB relationship is consistent, but the consistency broken in private LB..... As I mentioned above, with my CV mechanism, adding item_nbr as well as other category variable improved both my local CV score and Public LB score consistently by about 0.001+, while private LB score stayed the same. This seems indeed very strange to me, and I have no clues why.... Could anyone else confirmed whether they encountered similar situations (i.e., improve both CV and public LB score consistently, but not private LB) when adding the category variables?", "votes": null }, { "id": "269554", "postDate": "01/17/2018 01:35:05", "content": "I'm not sure how to express the gut feeling I have here. Let me reverse the question: why would item_nbr help at all? Why would be the pattern for sales for a given item be significantly different from another one? In the WTF competition I did not use the page either as a feature. It forces the model to generalize on how sales evolve over time across items, instead of learning a pattern for each item, which increases variance and the risk of overfit. But I may be totally misled here, i welcome other feedback too.
\n\nThe other reason may be your choice of validation period: the test period ends with month end, i.e. a payday. We selected our validation periods to end with paydays, with the hunch that this was a significant pattern. I must say this was Giba's idea, I was tempted to do exactly as you did ;)
", "rawMarkdown": "I'm not sure how to express the gut feeling I have here. Let me reverse the question: why would item_nbr help at all? Why would be the pattern for sales for a given item be significantly different from another one? In the WTF competition I did not use the page either as a feature. It forces the model to generalize on how sales evolve over time across items, instead of learning a pattern for each item, which increases variance and the risk of overfit. But I may be totally misled here, i welcome other feedback too.\n\nThe other reason may be your choice of validation period: the test period ends with month end, i.e. a payday. We selected our validation periods to end with paydays, with the hunch that this was a significant pattern. I must say this was Giba's idea, I was tempted to do exactly as you did ;)", "votes": null }, { "id": "269563", "postDate": "01/17/2018 01:57:10", "content": "In my case, adding:
\n- Frequency Encoding (store, item, family, class)
\n- One-Hot_Encoding (clusters, states, types, cities, families)
Did the following:
\n\nThanks for the feedback, CPMP.
\n\n\n\n\nLet me reverse the question: why would item_nbr help at all? Why would be the pattern for sales for a given item be significantly different from another one?
\n
For me the underlying assumption is that, the sales pattern for different items (there are about 10k different items) are possibly somewhat different from each other, though the extent of dissimilarity is hard to tell. At least, it is possibly fair to say that the past sales data for the same item is possibly a bit more helpful than the past sales data for a different item. According to my intuition, adding the item_nbr feature (as well as other category features like store_nbr, family, cluster, etc.) would help the lgb model to more easily identify those most similar records and gave them a bit more weights, while without the item_nbr, since there are about 10k items in total, such info might be just buried in the large number of records and harder for lgb to identify. The same is for other category variables like store_nbr, etc. Anyway, the key for feature engineering efforts is just to make those information more easily captured by the models like lgb. Theoretically speaking, all the information are contained in the given raw data, our feature engineering efforts just convert those information into a form which is easier for the machine learning models to capture, rather than creating new information.
\n\nAlso note that, adding all those category variables (store_nbr, item_nbr, family, class, cluster, etc.) in total gave about 0.001+ improvement in both my CV and Public score, which is not very large but should be nontrivial. It might not necessarily to require \"the pattern for sales for a given item being 'significantly' different from another one\" in order for item_nbr to be helpful in such a extent of 0.001+ score improvements. As long as there are some nontrivial dissimilarity among the sales patterns for different items (and since there are about 10k items in total, I do bet this is true), it is reasonable to accept that adding item_nbr to be helpful (though it is also acceptable if adding item_nbr not helpful, when considering the generalization mentioned by you), namely, adding item_nbr being helpful seems not to be a weird result for me.
\n\nWhat confused me was why adding such category variables (item_nbr, store_nbr, family, cluster, etc.) helps both my CV and public LB, but not private LB. But your raised point about pay day in validation period is indeed a factor that I never considered in my experiments, and it might possibly explain the broken CV_LB relationship on private LB, though I am not sure since I didn't do experiments on this. Also note that the top 1 team also used exactly the same validation period (2017/07/26-2017/08/10) as me, but did not suffer the same issue on private LB, so I am not sure whether the issue is indeed associated with my validation period choice...
", "rawMarkdown": "Thanks for the feedback, CPMP.\n\n> \nLet me reverse the question: why would item_nbr help at all? Why would be the pattern for sales for a given item be significantly different from another one?\n\n\nFor me the underlying assumption is that, the sales pattern for different items (there are about 10k different items) are possibly somewhat different from each other, though the extent of dissimilarity is hard to tell. At least, it is possibly fair to say that the past sales data for the same item is possibly a bit more helpful than the past sales data for a different item. According to my intuition, adding the item_nbr feature (as well as other category features like store_nbr, family, cluster, etc.) would help the lgb model to more easily identify those most similar records and gave them a bit more weights, while without the item_nbr, since there are about 10k items in total, such info might be just buried in the large number of records and harder for lgb to identify. The same is for other category variables like store_nbr, etc. Anyway, the key for feature engineering efforts is just to make those information more easily captured by the models like lgb. Theoretically speaking, all the information are contained in the given raw data, our feature engineering efforts just convert those information into a form which is easier for the machine learning models to capture, rather than creating new information.\n\nAlso note that, adding all those category variables (store_nbr, item_nbr, family, class, cluster, etc.) in total gave about 0.001+ improvement in both my CV and Public score, which is not very large but should be nontrivial. It might not necessarily to require \"the pattern for sales for a given item being 'significantly' different from another one\" in order for item_nbr to be helpful in such a extent of 0.001+ score improvements. As long as there are some nontrivial dissimilarity among the sales patterns for different items (and since there are about 10k items in total, I do bet this is true), it is reasonable to accept that adding item_nbr to be helpful (though it is also acceptable if adding item_nbr not helpful, when considering the generalization mentioned by you), namely, adding item_nbr being helpful seems not to be a weird result for me. \n\nWhat confused me was why adding such category variables (item_nbr, store_nbr, family, cluster, etc.) helps both my CV and public LB, but not private LB. But your raised point about pay day in validation period is indeed a factor that I never considered in my experiments, and it might possibly explain the broken CV_LB relationship on private LB, though I am not sure since I didn't do experiments on this. Also note that the top 1 team also used exactly the same validation period (2017/07/26-2017/08/10) as me, but did not suffer the same issue on private LB, so I am not sure whether the issue is indeed associated with my validation period choice...", "votes": null }, { "id": "269574", "postDate": "01/17/2018 02:20:47", "content": "> Improved private LB score by .002 (or less)
\n\n@Nicolas: So it seems that adding those category variables seems indeed contribute to score improvements in all of your CV, Public LB and Private LB scores, which is different from mine. May I know which validation periods you were using? Also 16 days period starting at 2017/07/26? Or sth else?
", "rawMarkdown": "> Improved private LB score by .002 (or less)\n\n@Nicolas: So it seems that adding those category variables seems indeed contribute to score improvements in all of your CV, Public LB and Private LB scores, which is different from mine. May I know which validation periods you were using? Also 16 days period starting at 2017/07/26? Or sth else?", "votes": null }, { "id": "269578", "postDate": "01/17/2018 02:26:33", "content": "@Shize, you did notice yourself that \"3b) Adding the category features from stores.csv and items.csv, and apply label encoding \" broke the CV LB relationship. I tend to believe that it leads to overfit. Good point re team #1 using the same validation period as you. I haven't studied their code yet, so I cannot comment further.
", "rawMarkdown": "Shize, you did notice yourself that \"3b) Adding the category features from stores.csv and items.csv, and apply label encoding \" broke the CV LB relationship. I tend to believe that it leads to overfit. Good point re team #1 using the same validation period as you. I haven't studied their code yet, so I cannot comment further.", "votes": null }, { "id": "269584", "postDate": "01/17/2018 02:31:14", "content": "@CPMP: Yeah, the thing made me confusing is that adding category features from stores.csv and items.csv only broke my CV_Private LB relationship, and the CV_Public LB relationship was still preserved nicely. That's what made me confused. If it also broke my CV_LB relationship, then I would say maybe it is over-fitting to CV and has worse generalization. But since it improved both CV and Public LB score in a similar scale but not improved the private LB score, this made me confused.
", "rawMarkdown": "CPMP: Yeah, the thing made me confusing is that adding category features from stores.csv and items.csv only broke my CV_Private LB relationship, and the CV_Public LB relationship was still preserved nicely. That's what made me confused. If it also broke my CV_LB relationship, then I would say maybe it is over-fitting to CV and has worse generalization. But since it improved both CV and Public LB score in a similar scale but not improved the private LB score, this made me confused.", "votes": null }, { "id": "269591", "postDate": "01/17/2018 02:37:03", "content": "This dataset was confusing in many ways for me. For instance I tried to use sales from same class as features, it improved CV nicely, and resulted into a public LB of 0.77 !?! I may have done something wrong as I was quite feverish that day, but it made me very cautious about not overfitting by adding too many features. I'm impressed that you and others (eg @Nicolas) managed to add so many features without overfitting.
", "rawMarkdown": "This dataset was confusing in many ways for me. For instance I tried to use sales from same class as features, it improved CV nicely, and resulted into a public LB of 0.77 !?! I may have done something wrong as I was quite feverish that day, but it made me very cautious about not overfitting by adding too many features. I'm impressed that you and others (eg @Nicolas) managed to add so many features without overfitting.", "votes": null }, { "id": "269607", "postDate": "01/17/2018 02:53:46", "content": "@CPMP: Yeah, I just have a feeling that it is probable that we might still miss sth important but sbtle, and if we identify that one, maybe it is possible to achieve consistent CV_Public LB_Private LB score relationship, rather than making the final private LB somehow a lottery. My feeling was due to that : 1) My CV_Public LB relationship is reasonably consistent for my feature engineering ideas; 2) It seems to me that there is no too much difference/bias between public LB and private LB, in terms of those feature engineering efforts I tried. Thus, unless I missed anything, I have such a feeling that achieve consistent CV_Public LB_Private LB score relationship might be possible, there might be just some subtle issue that is hard to figure out that we didn't find it. That's the main motivation why I initiated this discussion about the CV_LB relationship question here.
", "rawMarkdown": "CPMP: Yeah, I just have a feeling that it is probable that we might still miss sth important but sbtle, and if we identify that one, maybe it is possible to achieve consistent CV_Public LB_Private LB score relationship, rather than making the final private LB somehow a lottery. My feeling was due to that : 1) My CV_Public LB relationship is reasonably consistent for my feature engineering ideas; 2) It seems to me that there is no too much difference/bias between public LB and private LB, in terms of those feature engineering efforts I tried. Thus, unless I missed anything, I have such a feeling that achieve consistent CV_Public LB_Private LB score relationship might be possible, there might be just some subtle issue that is hard to figure out that we didn't find it. That's the main motivation why I initiated this discussion about the CV_LB relationship question here.", "votes": null }, { "id": "269617", "postDate": "01/17/2018 03:09:29", "content": "For me, the CV and LB performances were not consistent, and not consistent even between different models, this made me confused for a long time. My Seq2Seq model gave me the best CV result but was stuck at .512 at LB, my CNN model was slightly worse and not stable in CV but got .505 in LB. Finally these two models both achieved .514 in PB. Generally, for me the CV and PB are quite consistent but the LB is problematic.
\n\nActually after noticing the strange pattern of promotion infos in 8/16, which is stated in my solution, I told myself the LB is problematic and cannot be trusted. The CV cannot be trusted either because of the biased promotion info. I have to say there is some luck here to bet on the distribution of the private test data.
", "rawMarkdown": "For me, the CV and LB performances were not consistent, and not consistent even between different models, this made me confused for a long time. My Seq2Seq model gave me the best CV result but was stuck at .512 at LB, my CNN model was slightly worse and not stable in CV but got .505 in LB. Finally these two models both achieved .514 in PB. Generally, for me the CV and PB are quite consistent but the LB is problematic.\n\nActually after noticing the strange pattern of promotion infos in 8/16, which is stated in my solution, I told myself the LB is problematic and cannot be trusted. The CV cannot be trusted either because of the biased promotion info. I have to say there is some luck here to bet on the distribution of the private test data.", "votes": null }, { "id": "269619", "postDate": "01/17/2018 03:12:14", "content": "@Lingzhi: You mentioned that your CV and Private LB is somewhat consistent, but not the public LB. Mine is the other way, namely, CV and Public LB is consistent, but not the private LB. May I know which validation periods/method you were using?
", "rawMarkdown": "Lingzhi: You mentioned that your CV and Private LB is somewhat consistent, but not the public LB. Mine is the other way, namely, CV and Public LB is consistent, but not the private LB. May I know which validation periods/method you were using?", "votes": null }, { "id": "269623", "postDate": "01/17/2018 03:17:09", "content": "I only used 7/26-8/10 for validation for the whole time, and used similar ideas as yours: keeping the store-item combos that are in test data and have values during 2017 only.
", "rawMarkdown": "I only used 7/26-8/10 for validation for the whole time, and used similar ideas as yours: keeping the store-item combos that are in test data and have values during 2017 only.", "votes": null }, { "id": "269627", "postDate": "01/17/2018 03:19:10", "content": "@Linzhi Hmm, then it is quite strange why my models preserve CV_Public LB score consistency, while your models preserve CV_Private LB score consistency, given that we were using similar CV mechanism...
\n\nHmm, after a second thought, a possible explanation is that this might be due to the different type of feature engineering efforts (and/or modeling approaches, I didn't try any sequence models in this competition) between us (at least this is the only thing I can imagine for now). Namely, some type of feature engineering efforts might work for validation period and public LB period, while some other type of feature engineering efforts might work for validation period and private LB, due to some subtle difference between public LB and private LB that we haven't identified yet.
", "rawMarkdown": "Linzhi Hmm, then it is quite strange why my models preserve CV_Public LB score consistency, while your models preserve CV_Private LB score consistency, given that we were using similar CV mechanism... \n\nHmm, after a second thought, a possible explanation is that this might be due to the different type of feature engineering efforts (and/or modeling approaches, I didn't try any sequence models in this competition) between us (at least this is the only thing I can imagine for now). Namely, some type of feature engineering efforts might work for validation period and public LB period, while some other type of feature engineering efforts might work for validation period and private LB, due to some subtle difference between public LB and private LB that we haven't identified yet.", "votes": null }, { "id": "269630", "postDate": "01/17/2018 03:22:13", "content": "This is what I am confused too. Btw I also found item_nbr overfitted my models and I dropped it.
", "rawMarkdown": "This is what I am confused too. Btw I also found item_nbr overfitted my models and I dropped it.", "votes": null }, { "id": "269644", "postDate": "01/17/2018 03:56:07", "content": "awesome! Thanks for sharing your approach.
", "rawMarkdown": "awesome! Thanks for sharing your approach.", "votes": null }, { "id": "269716", "postDate": "01/17/2018 06:44:27", "content": "@Lingzhi I use you shared LGBM starter code, but what confused me is that when i add some new features, the local cv decrease, but the public lb score increase, why? It's overfitting the validation data?
", "rawMarkdown": "Lingzhi I use you shared LGBM starter code, but what confused me is that when i add some new features, the local cv decrease, but the public lb score increase, why? It's overfitting the validation data?", "votes": null }, { "id": "269733", "postDate": "01/17/2018 07:26:10", "content": "@Shize su. From you shared methods, you are using 20170503- 20170719 for training, 20170726 - 20170810 for validation. When you add new features every time, using the validation score to evaluate it, if the new features decrease the validation score for the first 5 days, last 11 days, and the full 16 days then add this features. Is my understand right?
\n\nWhat confused me is that how to debug the overfitting? Can you give some insights of it? Thanks.
", "rawMarkdown": "Shize su. From you shared methods, you are using 20170503- 20170719 for training, 20170726 - 20170810 for validation. When you add new features every time, using the validation score to evaluate it, if the new features decrease the validation score for the first 5 days, last 11 days, and the full 16 days then add this features. Is my understand right?\n\nWhat confused me is that how to debug the overfitting? Can you give some insights of it? Thanks.", "votes": null }, { "id": "269736", "postDate": "01/17/2018 07:35:17", "content": "I am not sure, maybe it was overfitted to the first 5 days but failed to generalize to 6th ~ 16th day, or maybe the increase in LB is just random noise that cannot be relied on. As I said, I don't think the test data in LB is reliable in this competition.
", "rawMarkdown": "I am not sure, maybe it was overfitted to the first 5 days but failed to generalize to 6th ~ 16th day, or maybe the increase in LB is just random noise that cannot be relied on. As I said, I don't think the test data in LB is reliable in this competition.", "votes": null }, { "id": "269740", "postDate": "01/17/2018 07:44:12", "content": "\n\n\nBtw I also found item_nbr overfitted my models and I dropped it.
\n
Thanks for confirming this.
", "rawMarkdown": "> Btw I also found item_nbr overfitted my models and I dropped it.\n\nThanks for confirming this.", "votes": null }, { "id": "269756", "postDate": "01/17/2018 08:22:32", "content": "@Lingzhi The 16 days validation score is decreased after add some new features, maybe as you said \"overfitted to the first 5 days but failed to generalize to 6th ~ 16th day\". How to debug it when it's happening? Or what's you next steps? Thanks for response.
", "rawMarkdown": "Lingzhi The 16 days validation score is decreased after add some new features, maybe as you said \"overfitted to the first 5 days but failed to generalize to 6th ~ 16th day\". How to debug it when it's happening? Or what's you next steps? Thanks for response.", "votes": null }, { "id": "269763", "postDate": "01/17/2018 08:49:31", "content": "had a similar experience, simply adding the item_nbr + store_nbr basically \"broke\" the public LB score; when I check my submissions, I have one with private 0.517 but public 0.908 (and another one 0.517 and 0.699), the CV scores of these two compared to my other submissions were almost the same; there really is something \"weird\" in the public LB
\n\nbtw. I was also validating on 2017-07-26 and I believe that was the key for staying in the \"medal area\"
", "rawMarkdown": "had a similar experience, simply adding the item_nbr + store_nbr basically \"broke\" the public LB score; when I check my submissions, I have one with private 0.517 but public 0.908 (and another one 0.517 and 0.699), the CV scores of these two compared to my other submissions were almost the same; there really is something \"weird\" in the public LB\n\nbtw. I was also validating on 2017-07-26 and I believe that was the key for staying in the \"medal area\"", "votes": null }, { "id": "269855", "postDate": "01/17/2018 12:11:36", "content": "hi, @Su. The \"statistical features were generated independent for each week of the training data\" is still confused to me.
\nFor the training data (20170503- 20170719), we should construct the features and labels. If using the sales 2017/7/4-2017/7/19 as the labels, how to construct the features for it? or what's the labels in you training data? \nThanks for you time.
Very interesting discussion. As I mentioned, adding categorical and frequency encoding improved Public LB the least for me. I kept it because it improved my CV quite a bit.
\n\nI was also using 7/26-8/10 --- using all and only items that appeared between 2017/01/01 and 2017/08/10.
", "rawMarkdown": "Very interesting discussion. As I mentioned, adding categorical and frequency encoding improved Public LB the least for me. I kept it because it improved my CV quite a bit. \n\nI was also using 7/26-8/10 --- using all and only items that appeared between 2017/01/01 and 2017/08/10.", "votes": null }, { "id": "269915", "postDate": "01/17/2018 14:22:30", "content": "@feng: Let me clarify a little bit more. My framework/flowchart is similar to the one shared in the popular lgb starter script, such as this one https://www.kaggle.com/tunguz/lgbm-one-step-ahead-c8de0f. The training data (20170503- 20170719) including 12 weeks in total, similar to the starter script, we just construct the feature set x_temp, y_temp (line 113-121 in the attached starter script code) on each Wednesday, where x_temp including all historical features (e.g., past sales, past sales average, past sales weekly average, DayOfWeek average, and the associated stats features (median, max, std, sum), etc., while the y_temp are the future 16 days periods sales that we need to predict. Then we append/concate each weeks {x_temp, y_temp} together as the final combined train set.
\n\nTaking the week starting as 20170719 (wednesday) for example. The y_temp are the future 16 days (20170719-20170804) period sales, the x_temp are the past sales average/(and weekly average, DayOfWeek sales/average sales, etc.) until 20170718. And when I generating the statistical features, these \"target_encoding\" type features used the most important past_sales features in the x_temp (which only include sales information before or on 20170718) as target, and use those category variables (store_nbr, item_nbr, family, class, cluster, city, state, etc) as the key, and generate the corresponding stats (mean, median, std, sum, ). This is what I called \"target_encoded\" like statistical features. Obviously, since such generated stats feature only used information in x_temp (which only include sales information before or on 20170718), thus they are impossible to contain leaks from the future 16 days period sales labels y_temp.
\n\nNote that, \"target_encoding\" type features for category variable are used quite extensively (and in many case very helpful) in many different kaggle competitions, and you could see a lot of them by looking at previous competition. Usually, in those past competitions, the \"target\" there meant the true label that we need to predict, and we need to be very careful (and a bit tricky) to use techniques like out-of-fold labels, leave-one-out encoding, etc,, to reduce the overfitting to cv when generating such 'target_encoding\" features. Fortunately, in this competition, it is a time series competition, and thus we have the past sales data, which are very predictive (they are top features based on my lgb feature importance), so I just use such past sales as the 'target', and generate the \"target_encoding\" like features with key being those category variable, in this way the leak from future information (i.e., the true label, future 16 days period sales) are avoided completely, and thus if CV score improved by such features, they are not possibly due to leak from future label (y_temp), but just due to that these features and good engineering features that are indeed helpful (i.e., they become normal feature engineering not associated with the true label y_temp), at least for the time period considered in CV. And if such features improve both CV and LB score consistently, we might have some confidence that they are indeed features that works for both the CV and LB periods.
\n\nHope it helps.
", "rawMarkdown": "feng: Let me clarify a little bit more. My framework/flowchart is similar to the one shared in the popular lgb starter script, such as this one https://www.kaggle.com/tunguz/lgbm-one-step-ahead-c8de0f. The training data (20170503- 20170719) including 12 weeks in total, similar to the starter script, we just construct the feature set x_temp, y_temp (line 113-121 in the attached starter script code) on each Wednesday, where x_temp including all historical features (e.g., past sales, past sales average, past sales weekly average, DayOfWeek average, and the associated stats features (median, max, std, sum), etc., while the y_temp are the future 16 days periods sales that we need to predict. Then we append/concate each weeks {x_temp, y_temp} together as the final combined train set.\n\nTaking the week starting as 20170719 (wednesday) for example. The y_temp are the future 16 days (20170719-20170804) period sales, the x_temp are the past sales average/(and weekly average, DayOfWeek sales/average sales, etc.) until 20170718. And when I generating the statistical features, these \"target_encoding\" type features used the most important past_sales features in the x_temp (which only include sales information before or on 20170718) as target, and use those category variables (store_nbr, item_nbr, family, class, cluster, city, state, etc) as the key, and generate the corresponding stats (mean, median, std, sum, ). This is what I called \"target_encoded\" like statistical features. Obviously, since such generated stats feature only used information in x_temp (which only include sales information before or on 20170718), thus they are impossible to contain leaks from the future 16 days period sales labels y_temp.\n\nNote that, \"target_encoding\" type features for category variable are used quite extensively (and in many case very helpful) in many different kaggle competitions, and you could see a lot of them by looking at previous competition. Usually, in those past competitions, the \"target\" there meant the true label that we need to predict, and we need to be very careful (and a bit tricky) to use techniques like out-of-fold labels, leave-one-out encoding, etc,, to reduce the overfitting to cv when generating such 'target_encoding\" features. Fortunately, in this competition, it is a time series competition, and thus we have the past sales data, which are very predictive (they are top features based on my lgb feature importance), so I just use such past sales as the 'target', and generate the \"target_encoding\" like features with key being those category variable, in this way the leak from future information (i.e., the true label, future 16 days period sales) are avoided completely, and thus if CV score improved by such features, they are not possibly due to leak from future label (y_temp), but just due to that these features and good engineering features that are indeed helpful (i.e., they become normal feature engineering not associated with the true label y_temp), at least for the time period considered in CV. And if such features improve both CV and LB score consistently, we might have some confidence that they are indeed features that works for both the CV and LB periods.\n\nHope it helps.", "votes": null }, { "id": "269922", "postDate": "01/17/2018 14:31:20", "content": ">From you shared methods, you are using 20170503- 20170719 for training, 20170726 - 20170810 for validation. When you add new features every time, using the validation score to evaluate it, if the new features decrease the validation score for the first 5 days, last 11 days, and the full 16 days then add this features. Is my understand right?
\n\n>What confused me is that how to debug the overfitting? Can you give some insights of it? Thanks.
\n\n@Feng: Yes, when I add new features, I checked the cv score, and if they improved the cv score in all of the first 5day period, last 11 days period and full 16 days period, then I added the features. Of course, in addition to that, we need to make sure there is no bug in code ,and for feature engineering we didn't use leaks from true target (since such leak won't be available at all for LB period) for such CV score improvement. If everything seems fine, then I made submissions to see the LB score to verify that they indeed works on the LB period. If not, then I doubts: 1) whether there are any subtle bugs (including future label leaks, etc.) in my code that contributed to CV score improvement ----if yes, fix it; 2) whether there are some significant dissimilarity between my CV set and the test set--if yes, then try the best to make CV set as similar to the test set as possible.
\n\nFinally, if your feature engineering improved both your CV score and LB score consistently, and you think your CV set is very similar to the test set, then we should have some confidence that such engineered features are indeed good/helpful features.
\n\nFor this competition, my feature engineering efforts improved my CV and public LB score consistently, but not consistent for the private LB. I bet that there are some key (but subtle) difference between the public LB period and the private LB period that we haven't identified yet, which contributed to such phenomenon.
\n\nHope it helps.
", "rawMarkdown": ">From you shared methods, you are using 20170503- 20170719 for training, 20170726 - 20170810 for validation. When you add new features every time, using the validation score to evaluate it, if the new features decrease the validation score for the first 5 days, last 11 days, and the full 16 days then add this features. Is my understand right?\n\n>What confused me is that how to debug the overfitting? Can you give some insights of it? Thanks.\n\n@Feng: Yes, when I add new features, I checked the cv score, and if they improved the cv score in all of the first 5day period, last 11 days period and full 16 days period, then I added the features. Of course, in addition to that, we need to make sure there is no bug in code ,and for feature engineering we didn't use leaks from true target (since such leak won't be available at all for LB period) for such CV score improvement. If everything seems fine, then I made submissions to see the LB score to verify that they indeed works on the LB period. If not, then I doubts: 1) whether there are any subtle bugs (including future label leaks, etc.) in my code that contributed to CV score improvement ----if yes, fix it; 2) whether there are some significant dissimilarity between my CV set and the test set--if yes, then try the best to make CV set as similar to the test set as possible.\n\nFinally, if your feature engineering improved both your CV score and LB score consistently, and you think your CV set is very similar to the test set, then we should have some confidence that such engineered features are indeed good/helpful features.\n\nFor this competition, my feature engineering efforts improved my CV and public LB score consistently, but not consistent for the private LB. I bet that there are some key (but subtle) difference between the public LB period and the private LB period that we haven't identified yet, which contributed to such phenomenon.\n\nHope it helps.", "votes": null }, { "id": "269947", "postDate": "01/17/2018 15:00:28", "content": "Help a lot!!! Thank you for your enthusiastic answer！
", "rawMarkdown": "Help a lot!!! Thank you for your enthusiastic answer！", "votes": null }, { "id": "269964", "postDate": "01/17/2018 15:30:35", "content": "Thanks! It's very helpful!!!
", "rawMarkdown": "Thanks! It's very helpful!!!", "votes": null }, { "id": "270117", "postDate": "01/17/2018 19:07:09", "content": "I have spent lots of time to train July/Aug data in year of 2014-2016, that doesn't help at all. And it is the major reason that my private LB dropped by 219th. I guess the sales pattern doesn't change much month over month in the summer on grocery. If the test period is December, the seasonality will be much more important, and then we will need train same period previous year.
", "rawMarkdown": "I have spent lots of time to train July/Aug data in year of 2014-2016, that doesn't help at all. And it is the major reason that my private LB dropped by 219th. I guess the sales pattern doesn't change much month over month in the summer on grocery. If the test period is December, the seasonality will be much more important, and then we will need train same period previous year.", "votes": null }, { "id": "270381", "postDate": "01/18/2018 06:48:17", "content": "Hi @Shize Su , \nI found that the rossmann store sales forecasting in kaggle is very similar to this one, and you rank the top 1 in public leaderboard. In that competition, how did you design the validation mechanism? Is there difference with this one?
\n\nThanks in sincerely !
", "rawMarkdown": "Hi @Shize Su , \nI found that the rossmann store sales forecasting in kaggle is very similar to this one, and you rank the top 1 in public leaderboard. In that competition, how did you design the validation mechanism? Is there difference with this one? \n\nThanks in sincerely !", "votes": null }, { "id": "270552", "postDate": "01/18/2018 13:49:07", "content": "@feng: That is a competition long time ago for me, so not sure I still remember the detail exactly. But roughly speaking, at the rossmann competition, we used some normal hold-out period data as validation, together with the public LB. I remembered that the local CV score and public LB score was not very consistent for us (it was somehow correlated but not perfect, especially when comparing different types of models like comparing xgb and ARIMA), but unfortunately we didn't pay too much attentions to adjust/fix the cv mechanism design to make it more consistent. We used the local cv to find improvements for models, and if improvements identified, submitted to the LB. For same type single model (e.g., xgb), local cv score improvements usually results in LB score improvement, but the improvement scale was not always the same (e.g., cv score improved by 0.0002 but LB score improved by 0.0001, or vice versa, etc.). The inconsistency in score improvement is much larger especially when comparing different type of models like xgb and ARIMA. For example, ARIMA worked very well (as a diverse component of our ensemble) on the first couple of days on LB (i.e., public LB) but performed much worse for longer future period on the private LB. Unfortunately, when inconsistency occurred, we trusted a bit more on the public LB rather than our CV, since we thought that the public LB period is closer to the private LB period than our held-out cv period.Thus we decided to assigned a bit larger ensemble weights for the ARIMA models in our final submission, and we were bitten by that quite a bit on final private LB (dropped from #1 on public LB to #7 on private LB). Then in the past two years, I more and more rely on a comprehensive CV mechanism when I want to play a competition seriously, and when the CV_LB score is inconsistent, I paid more attentions /spent much more efforts to design a more sophisticated CV mechanism to achieve the consistency. That's all for now.
\n\nHope it helps.
", "rawMarkdown": "feng: That is a competition long time ago for me, so not sure I still remember the detail exactly. But roughly speaking, at the rossmann competition, we used some normal hold-out period data as validation, together with the public LB. I remembered that the local CV score and public LB score was not very consistent for us (it was somehow correlated but not perfect, especially when comparing different types of models like comparing xgb and ARIMA), but unfortunately we didn't pay too much attentions to adjust/fix the cv mechanism design to make it more consistent. We used the local cv to find improvements for models, and if improvements identified, submitted to the LB. For same type single model (e.g., xgb), local cv score improvements usually results in LB score improvement, but the improvement scale was not always the same (e.g., cv score improved by 0.0002 but LB score improved by 0.0001, or vice versa, etc.). The inconsistency in score improvement is much larger especially when comparing different type of models like xgb and ARIMA. For example, ARIMA worked very well (as a diverse component of our ensemble) on the first couple of days on LB (i.e., public LB) but performed much worse for longer future period on the private LB. Unfortunately, when inconsistency occurred, we trusted a bit more on the public LB rather than our CV, since we thought that the public LB period is closer to the private LB period than our held-out cv period.Thus we decided to assigned a bit larger ensemble weights for the ARIMA models in our final submission, and we were bitten by that quite a bit on final private LB (dropped from #1 on public LB to #7 on private LB). Then in the past two years, I more and more rely on a comprehensive CV mechanism when I want to play a competition seriously, and when the CV_LB score is inconsistent, I paid more attentions /spent much more efforts to design a more sophisticated CV mechanism to achieve the consistency. That's all for now.\n\nHope it helps.", "votes": null }, { "id": "270613", "postDate": "01/18/2018 15:41:48", "content": "Thanks!!! You helped me a lots!!!
", "rawMarkdown": "Thanks!!! You helped me a lots!!!", "votes": null }, { "id": "271287", "postDate": "01/20/2018 03:13:16", "content": "I'm pretty new in these contest but thought this might help. I compared the scores of my models between the public and private leaderboards and I noticed that the submissions where I tried to do something with the promotions data we're the ones were the score improved on the private LB from the public. For example, to fill the missing values in the training set... I determined the average sales lift from promotion by store/item (where we had promo data)... And then I filled na values where the sales of a store/item we're missing and the unit sales was x% above average for that pair. My local CV process was basically nonexistant, but I did notice that when I tried these sorts of things with the promo data my private LB scores were > public LB scores by not trivial amounts.
", "rawMarkdown": "I'm pretty new in these contest but thought this might help. I compared the scores of my models between the public and private leaderboards and I noticed that the submissions where I tried to do something with the promotions data we're the ones were the score improved on the private LB from the public. For example, to fill the missing values in the training set... I determined the average sales lift from promotion by store/item (where we had promo data)... And then I filled na values where the sales of a store/item we're missing and the unit sales was x% above average for that pair. My local CV process was basically nonexistant, but I did notice that when I tried these sorts of things with the promo data my private LB scores were > public LB scores by not trivial amounts.", "votes": null }, { "id": "275233", "postDate": "01/28/2018 14:38:32", "content": "Note that, in order to avoiding possible future target information leak, such statistical features were generated independent for each week of the training data, validation data, and test data.\nHello Shize Su.Does What does it mean? Does it mean separate training data in one week and generate feature?Expect your reply.Thanks
", "rawMarkdown": "Note that, in order to avoiding possible future target information leak, such statistical features were generated independent for each week of the training data, validation data, and test data.\nHello Shize Su.Does What does it mean? Does it mean separate training data in one week and generate feature?Expect your reply.Thanks", "votes": null }, { "id": "275243", "postDate": "01/28/2018 15:02:21", "content": "Hi zhixing,
\n\nAbout generating the statistical features (i.e., 'target-encoding' type features for category variables) , the key is to be careful to avoid utilizing (explicitly or implicitly) any future target information when generating these features.
\n\nBy generating the statistical features for each week independently, we could completely avoid such future target information leak since at that time all the past sales data definitely not including any future sales data information, and the derived statistical features based on these past sales data of course is impossible to contain leak info from future targets.
\n\nTo be more specific, the generation of such statistical features should be put inside a function similar to \"def prepare_dataset(t2017, is_train=True):\" in the lgb startet script https://www.kaggle.com/tunguz/lgbm-one-step-ahead-c8de0f, i.e., it should be generated inside the function for generating the features X_l and y_l for each week, rather than after operation \"X_train = pd.concat(X_l, axis=0)\". If you generating the statistical features after operation \"X_train = pd.concat(X_l, axis=0)\", then features for earlier weeks will implicitly contain future target leak information from later weeks. For example, the past day1 sales for week 2017/07/19 might be future information for the previous week 2017/07/12, and if generating such 'target-encoding' type statistical features for week 2017/07/12 after the operation \"X_train = pd.concat(X_l, axis=0)\", then it contains the future information leak (e.g., past day1 sales for week 2017/07/19, etc.). And such kind of future info leak is sth we need to avoid to make such 'target-encoding' type features works on the LB period, where future info leak is not possible.
\n\nHope it helps.
\n\nBest regards,
\n\nShize
", "rawMarkdown": "Hi zhixing,\n\nAbout generating the statistical features (i.e., 'target-encoding' type features for category variables) , the key is to be careful to avoid utilizing (explicitly or implicitly) any future target information when generating these features. \n\nBy generating the statistical features for each week independently, we could completely avoid such future target information leak since at that time all the past sales data definitely not including any future sales data information, and the derived statistical features based on these past sales data of course is impossible to contain leak info from future targets.\n\nTo be more specific, the generation of such statistical features should be put inside a function similar to \"def prepare_dataset(t2017, is_train=True):\" in the lgb startet script https://www.kaggle.com/tunguz/lgbm-one-step-ahead-c8de0f, i.e., it should be generated inside the function for generating the features X_l and y_l for each week, rather than after operation \"X_train = pd.concat(X_l, axis=0)\". If you generating the statistical features after operation \"X_train = pd.concat(X_l, axis=0)\", then features for earlier weeks will implicitly contain future target leak information from later weeks. For example, the past day1 sales for week 2017/07/19 might be future information for the previous week 2017/07/12, and if generating such 'target-encoding' type statistical features for week 2017/07/12 after the operation \"X_train = pd.concat(X_l, axis=0)\", then it contains the future information leak (e.g., past day1 sales for week 2017/07/19, etc.). And such kind of future info leak is sth we need to avoid to make such 'target-encoding' type features works on the LB period, where future info leak is not possible.\n\nHope it helps.\n\nBest regards,\n\nShize", "votes": null }, { "id": "275582", "postDate": "01/29/2018 14:33:20", "content": "@Shize Su hi. I am back again. Thanks for you valuable answers. I found lots of shared solutions in this competition using similar cross validation settings. And It seem didn't concern about the holiday effects(such as the effect of before or after or in holiday?).
\n\nIn the rossmann competition, count features maybe a good choice, but how to using these features in this cv settings? or count features should using in another cv settings?
\n\nBesides, when I training the datasets with the same cv settings (there is no data leakages) , and look into the train log, I found that in some models the validation error is smaller than training error? Is this normal? or over-fitting? Maybe just the validation distribution is different from training data distribution?
", "rawMarkdown": "Shize Su hi. I am back again. Thanks for you valuable answers. I found lots of shared solutions in this competition using similar cross validation settings. And It seem didn't concern about the holiday effects(such as the effect of before or after or in holiday?). \n\nIn the rossmann competition, [count features][1] maybe a good choice, but how to using these features in this cv settings? or count features should using in another cv settings?\n\nBesides, when I training the datasets with the same cv settings (there is no data leakages) , and look into the train log, I found that in some models the validation error is smaller than training error? Is this normal? or over-fitting? Maybe just the validation distribution is different from training data distribution?\n\n\n [1]: http://blog.kaggle.com/2016/02/03/rossmann-store-sales-winners-interview-2nd-place-nima-shahbazi/", "votes": null }, { "id": "275599", "postDate": "01/29/2018 15:14:22", "content": "@feng:
\n\n1) About holiday, one key observation is that the LB period almost doesn't include any holiday (except a local holiday), and thus the benefits of considering holiday effect might only be marginal even if appropriately addressed. Not to mention that it is very tricky and time consuming to appropriately address the holiday effects to achieve such marginal improvement. This is why most competitors (including me) did not spend much time on dealing with the holiday effects. If the LB period includes many more National holidays, then it would be a totally different story.
\n\n2) About 'count features', I didn't see any clear explanations in your provided link, so i am not sure what specific features you were talking about.
\n\n3) The variance across the time periods is a bit large, thus it is possible (but not very likely) that validation loss is smaller than the training loss, especially when the model is under-fitting to the train set (for example, at the first couple of rounds of my lgb, in quite a couple of times I also observed smaller validation loss than training loss, but by the time early-stopping occurred, the training loss is usually much smaller than validation loss). So my suggestions would be : 1) double check the code to see whether there are any subtle mistakes/bugs; 2) make sure the model is sufficiently trained and not under-fitting to the train set; 3) if everything seems fine, then maybe it is ok-as I said, it is possible (but not very likely) that validation loss is smaller than the training loss for this dataset due to the variance across the time periods.
\n\nHope this helps.
", "rawMarkdown": "feng:\n\n1) About holiday, one key observation is that the LB period almost doesn't include any holiday (except a local holiday), and thus the benefits of considering holiday effect might only be marginal even if appropriately addressed. Not to mention that it is very tricky and time consuming to appropriately address the holiday effects to achieve such marginal improvement. This is why most competitors (including me) did not spend much time on dealing with the holiday effects. If the LB period includes many more National holidays, then it would be a totally different story.\n\n2) About 'count features', I didn't see any clear explanations in your provided link, so i am not sure what specific features you were talking about. \n\n3) The variance across the time periods is a bit large, thus it is possible (but not very likely) that validation loss is smaller than the training loss, especially when the model is under-fitting to the train set (for example, at the first couple of rounds of my lgb, in quite a couple of times I also observed smaller validation loss than training loss, but by the time early-stopping occurred, the training loss is usually much smaller than validation loss). So my suggestions would be : 1) double check the code to see whether there are any subtle mistakes/bugs; 2) make sure the model is sufficiently trained and not under-fitting to the train set; 3) if everything seems fine, then maybe it is ok-as I said, it is possible (but not very likely) that validation loss is smaller than the training loss for this dataset due to the variance across the time periods.\n\nHope this helps.", "votes": null }, { "id": "275765", "postDate": "01/30/2018 00:10:04", "content": "@Shize Su, hi. It's very helpful!
\n\nIf the LB period include the holiday, what's the general methods for this cv settings? Any suggestion?
\n\nAbout the count features in the links, nimashahbazi create the features like below:\n
If the store is closed, the open variable is zero and the sales is zero too. Then he create a dummy variable called MyRowHoliday to capture the information for these zero sales. This dummy variable assigned positive integers before and after the consecutive close and “-1” on other days.
\n\nIn this way, maybe it capture the holiday effects(such as the effect of before or after or in holiday)?
", "rawMarkdown": "Shize Su, hi. It's very helpful!\n\nIf the LB period include the holiday, what's the general methods for this cv settings? Any suggestion?\n\nAbout the count features in the links, nimashahbazi create the features like below:\n![count features][1]\n\nIf the store is closed, the open variable is zero and the sales is zero too. Then he create a dummy variable called MyRowHoliday to capture the information for these zero sales. This dummy variable assigned positive integers before and after the consecutive close and “-1” on other days.\n\nIn this way, maybe it capture the holiday effects(such as the effect of before or after or in holiday)?\n\n [1]: http://5047-presscdn.pagely.netdna-cdn.com/wp-content/uploads/2016/02/rossmann2_2-e1454538642298.png", "votes": null }, { "id": "275768", "postDate": "01/30/2018 00:27:59", "content": "@feng:
\n\n1) If the LB period includes more major holidays, a general guideline is to design the CV data set to be as similar to LB data set (e.g., similar period but in 2016; or period close to LB period but also have similar amount of major holidays). But obviously there is not a simple clear answer about the best cv setting in this case, it might require significant amount of time and efforts (more difficult than the case here) to find a CV setting for consistent CV_LB score trend. You can only figured it out by various of experiments, following the key guideline that CV set should be as similar to LB set as possible.
\n\n2) About the \"count features\" you mentioned above in Nima's post, they are not some common \"count features\" we usually use, and in my opinion those features are somewhat specific to the Rossmann data set. (For the term \"count features\", we (or at least me) usually mean sth like frequency of some category variables, etc. That's what made me confused when you used the term \"count feature\".) Finally, such type of features are probably much less helpful (though I didn't test it) for this competition since there are no that many \"close-open\" switching for stores in this data set (compared with rossmann competition data set), plus that there is no way to know whether a store is closed or not on a date of test data period. But maybe using such so called \"count features\" on the \"promotion\" variable might be somewhat helpful, though needs to be very carefully dealt with due to the large asymmetry in distribution of \"promotion\" variable in train and test data.
\n\nHope this helps.
", "rawMarkdown": "feng:\n\n1) If the LB period includes more major holidays, a general guideline is to design the CV data set to be as similar to LB data set (e.g., similar period but in 2016; or period close to LB period but also have similar amount of major holidays). But obviously there is not a simple clear answer about the best cv setting in this case, it might require significant amount of time and efforts (more difficult than the case here) to find a CV setting for consistent CV_LB score trend. You can only figured it out by various of experiments, following the key guideline that CV set should be as similar to LB set as possible.\n\n2) About the \"count features\" you mentioned above in Nima's post, they are not some common \"count features\" we usually use, and in my opinion those features are somewhat specific to the Rossmann data set. (For the term \"count features\", we (or at least me) usually mean sth like frequency of some category variables, etc. That's what made me confused when you used the term \"count feature\".) Finally, such type of features are probably much less helpful (though I didn't test it) for this competition since there are no that many \"close-open\" switching for stores in this data set (compared with rossmann competition data set), plus that there is no way to know whether a store is closed or not on a date of test data period. But maybe using such so called \"count features\" on the \"promotion\" variable might be somewhat helpful, though needs to be very carefully dealt with due to the large asymmetry in distribution of \"promotion\" variable in train and test data.\n\nHope this helps.", "votes": null }, { "id": "275790", "postDate": "01/30/2018 01:34:09", "content": "Thanks for clarify my doubts. You push me more interest in the competition, there are a lots of modeling skills to learn from you and some people like you. Thanks for your generously!
", "rawMarkdown": "Thanks for clarify my doubts. You push me more interest in the competition, there are a lots of modeling skills to learn from you and some people like you. Thanks for your generously!", "votes": null }, { "id": "276648", "postDate": "02/01/2018 03:02:25", "content": "Thanks your explanation.Your solution and analyse help me learn a lot in this competition.Hope you can stick to it.
", "rawMarkdown": "Thanks your explanation.Your solution and analyse help me learn a lot in this competition.Hope you can stick to it.", "votes": null }, { "id": "285317", "postDate": "02/19/2018 22:55:17", "content": "Thank you for your in-depth post...
", "rawMarkdown": "Thank you for your in-depth post...", "votes": null }, { "id": "574650", "postDate": "07/14/2019 08:58:18", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing.", "votes": null } ], "comments": [ { "id": 269478, "author_name": "gweijie", "author_url": "", "post_date": "01/16/2018 22:37:23", "content": "Thanks for the write up! Very informative :)
", "votes": null, "replies": [] }, { "id": 269532, "author_name": "cpmpml", "author_url": "", "post_date": "01/17/2018 01:00:12", "content": "Thanks for sharing. I found that using item_nbr was leading to overfit in my local CV. This may explain part of the shakeup.
", "votes": null, "replies": [ { "id": 269537, "author_name": "sushize", "author_url": "", "post_date": "01/17/2018 01:10:22", "content": "Thanks for the info. But why? There should not be any bias for item_nbr between public LB and private LB, and thus this factor cannot explain why my CV_Pub LB relationship is consistent, but the consistency broken in private LB..... As I mentioned above, with my CV mechanism, adding item_nbr as well as other category variable improved both my local CV score and Public LB score consistently by about 0.001+, while private LB score stayed the same. This seems indeed very strange to me, and I have no clues why.... Could anyone else confirmed whether they encountered similar situations (i.e., improve both CV and public LB score consistently, but not private LB) when adding the category variables?
", "votes": null, "replies": [] }, { "id": 269554, "author_name": "cpmpml", "author_url": "", "post_date": "01/17/2018 01:35:05", "content": "I'm not sure how to express the gut feeling I have here. Let me reverse the question: why would item_nbr help at all? Why would be the pattern for sales for a given item be significantly different from another one? In the WTF competition I did not use the page either as a feature. It forces the model to generalize on how sales evolve over time across items, instead of learning a pattern for each item, which increases variance and the risk of overfit. But I may be totally misled here, i welcome other feedback too.
\n\nThe other reason may be your choice of validation period: the test period ends with month end, i.e. a payday. We selected our validation periods to end with paydays, with the hunch that this was a significant pattern. I must say this was Giba's idea, I was tempted to do exactly as you did ;)
", "votes": null, "replies": [] }, { "id": 269563, "author_name": "npa02012", "author_url": "", "post_date": "01/17/2018 01:57:10", "content": "In my case, adding:
\n- Frequency Encoding (store, item, family, class)
\n- One-Hot_Encoding (clusters, states, types, cities, families)
Did the following:
\n\nThanks for the feedback, CPMP.
\n\n\n\n\nLet me reverse the question: why would item_nbr help at all? Why would be the pattern for sales for a given item be significantly different from another one?
\n
For me the underlying assumption is that, the sales pattern for different items (there are about 10k different items) are possibly somewhat different from each other, though the extent of dissimilarity is hard to tell. At least, it is possibly fair to say that the past sales data for the same item is possibly a bit more helpful than the past sales data for a different item. According to my intuition, adding the item_nbr feature (as well as other category features like store_nbr, family, cluster, etc.) would help the lgb model to more easily identify those most similar records and gave them a bit more weights, while without the item_nbr, since there are about 10k items in total, such info might be just buried in the large number of records and harder for lgb to identify. The same is for other category variables like store_nbr, etc. Anyway, the key for feature engineering efforts is just to make those information more easily captured by the models like lgb. Theoretically speaking, all the information are contained in the given raw data, our feature engineering efforts just convert those information into a form which is easier for the machine learning models to capture, rather than creating new information.
\n\nAlso note that, adding all those category variables (store_nbr, item_nbr, family, class, cluster, etc.) in total gave about 0.001+ improvement in both my CV and Public score, which is not very large but should be nontrivial. It might not necessarily to require \"the pattern for sales for a given item being 'significantly' different from another one\" in order for item_nbr to be helpful in such a extent of 0.001+ score improvements. As long as there are some nontrivial dissimilarity among the sales patterns for different items (and since there are about 10k items in total, I do bet this is true), it is reasonable to accept that adding item_nbr to be helpful (though it is also acceptable if adding item_nbr not helpful, when considering the generalization mentioned by you), namely, adding item_nbr being helpful seems not to be a weird result for me.
\n\nWhat confused me was why adding such category variables (item_nbr, store_nbr, family, cluster, etc.) helps both my CV and public LB, but not private LB. But your raised point about pay day in validation period is indeed a factor that I never considered in my experiments, and it might possibly explain the broken CV_LB relationship on private LB, though I am not sure since I didn't do experiments on this. Also note that the top 1 team also used exactly the same validation period (2017/07/26-2017/08/10) as me, but did not suffer the same issue on private LB, so I am not sure whether the issue is indeed associated with my validation period choice...
", "votes": null, "replies": [] }, { "id": 269574, "author_name": "sushize", "author_url": "", "post_date": "01/17/2018 02:20:47", "content": "> Improved private LB score by .002 (or less)
\n\n@Nicolas: So it seems that adding those category variables seems indeed contribute to score improvements in all of your CV, Public LB and Private LB scores, which is different from mine. May I know which validation periods you were using? Also 16 days period starting at 2017/07/26? Or sth else?
", "votes": null, "replies": [] }, { "id": 269578, "author_name": "cpmpml", "author_url": "", "post_date": "01/17/2018 02:26:33", "content": "@Shize, you did notice yourself that \"3b) Adding the category features from stores.csv and items.csv, and apply label encoding \" broke the CV LB relationship. I tend to believe that it leads to overfit. Good point re team #1 using the same validation period as you. I haven't studied their code yet, so I cannot comment further.
", "votes": null, "replies": [] }, { "id": 269584, "author_name": "sushize", "author_url": "", "post_date": "01/17/2018 02:31:14", "content": "@CPMP: Yeah, the thing made me confusing is that adding category features from stores.csv and items.csv only broke my CV_Private LB relationship, and the CV_Public LB relationship was still preserved nicely. That's what made me confused. If it also broke my CV_LB relationship, then I would say maybe it is over-fitting to CV and has worse generalization. But since it improved both CV and Public LB score in a similar scale but not improved the private LB score, this made me confused.
", "votes": null, "replies": [] }, { "id": 269591, "author_name": "cpmpml", "author_url": "", "post_date": "01/17/2018 02:37:03", "content": "This dataset was confusing in many ways for me. For instance I tried to use sales from same class as features, it improved CV nicely, and resulted into a public LB of 0.77 !?! I may have done something wrong as I was quite feverish that day, but it made me very cautious about not overfitting by adding too many features. I'm impressed that you and others (eg @Nicolas) managed to add so many features without overfitting.
", "votes": null, "replies": [] }, { "id": 269607, "author_name": "sushize", "author_url": "", "post_date": "01/17/2018 02:53:46", "content": "@CPMP: Yeah, I just have a feeling that it is probable that we might still miss sth important but sbtle, and if we identify that one, maybe it is possible to achieve consistent CV_Public LB_Private LB score relationship, rather than making the final private LB somehow a lottery. My feeling was due to that : 1) My CV_Public LB relationship is reasonably consistent for my feature engineering ideas; 2) It seems to me that there is no too much difference/bias between public LB and private LB, in terms of those feature engineering efforts I tried. Thus, unless I missed anything, I have such a feeling that achieve consistent CV_Public LB_Private LB score relationship might be possible, there might be just some subtle issue that is hard to figure out that we didn't find it. That's the main motivation why I initiated this discussion about the CV_LB relationship question here.
", "votes": null, "replies": [] }, { "id": 269763, "author_name": "steelrose", "author_url": "", "post_date": "01/17/2018 08:49:31", "content": "had a similar experience, simply adding the item_nbr + store_nbr basically \"broke\" the public LB score; when I check my submissions, I have one with private 0.517 but public 0.908 (and another one 0.517 and 0.699), the CV scores of these two compared to my other submissions were almost the same; there really is something \"weird\" in the public LB
\n\nbtw. I was also validating on 2017-07-26 and I believe that was the key for staying in the \"medal area\"
", "votes": null, "replies": [] }, { "id": 269891, "author_name": "npa02012", "author_url": "", "post_date": "01/17/2018 13:40:07", "content": "Very interesting discussion. As I mentioned, adding categorical and frequency encoding improved Public LB the least for me. I kept it because it improved my CV quite a bit.
\n\nI was also using 7/26-8/10 --- using all and only items that appeared between 2017/01/01 and 2017/08/10.
", "votes": null, "replies": [] } ] }, { "id": 269617, "author_name": "vrtjso", "author_url": "", "post_date": "01/17/2018 03:09:29", "content": "For me, the CV and LB performances were not consistent, and not consistent even between different models, this made me confused for a long time. My Seq2Seq model gave me the best CV result but was stuck at .512 at LB, my CNN model was slightly worse and not stable in CV but got .505 in LB. Finally these two models both achieved .514 in PB. Generally, for me the CV and PB are quite consistent but the LB is problematic.
\n\nActually after noticing the strange pattern of promotion infos in 8/16, which is stated in my solution, I told myself the LB is problematic and cannot be trusted. The CV cannot be trusted either because of the biased promotion info. I have to say there is some luck here to bet on the distribution of the private test data.
", "votes": null, "replies": [ { "id": 269619, "author_name": "sushize", "author_url": "", "post_date": "01/17/2018 03:12:14", "content": "@Lingzhi: You mentioned that your CV and Private LB is somewhat consistent, but not the public LB. Mine is the other way, namely, CV and Public LB is consistent, but not the private LB. May I know which validation periods/method you were using?
", "votes": null, "replies": [] }, { "id": 269623, "author_name": "vrtjso", "author_url": "", "post_date": "01/17/2018 03:17:09", "content": "I only used 7/26-8/10 for validation for the whole time, and used similar ideas as yours: keeping the store-item combos that are in test data and have values during 2017 only.
", "votes": null, "replies": [] }, { "id": 269627, "author_name": "sushize", "author_url": "", "post_date": "01/17/2018 03:19:10", "content": "@Linzhi Hmm, then it is quite strange why my models preserve CV_Public LB score consistency, while your models preserve CV_Private LB score consistency, given that we were using similar CV mechanism...
\n\nHmm, after a second thought, a possible explanation is that this might be due to the different type of feature engineering efforts (and/or modeling approaches, I didn't try any sequence models in this competition) between us (at least this is the only thing I can imagine for now). Namely, some type of feature engineering efforts might work for validation period and public LB period, while some other type of feature engineering efforts might work for validation period and private LB, due to some subtle difference between public LB and private LB that we haven't identified yet.
", "votes": null, "replies": [] }, { "id": 269630, "author_name": "vrtjso", "author_url": "", "post_date": "01/17/2018 03:22:13", "content": "This is what I am confused too. Btw I also found item_nbr overfitted my models and I dropped it.
", "votes": null, "replies": [] }, { "id": 269716, "author_name": "hubifeng", "author_url": "", "post_date": "01/17/2018 06:44:27", "content": "@Lingzhi I use you shared LGBM starter code, but what confused me is that when i add some new features, the local cv decrease, but the public lb score increase, why? It's overfitting the validation data?
", "votes": null, "replies": [] }, { "id": 269733, "author_name": "hubifeng", "author_url": "", "post_date": "01/17/2018 07:26:10", "content": "@Shize su. From you shared methods, you are using 20170503- 20170719 for training, 20170726 - 20170810 for validation. When you add new features every time, using the validation score to evaluate it, if the new features decrease the validation score for the first 5 days, last 11 days, and the full 16 days then add this features. Is my understand right?
\n\nWhat confused me is that how to debug the overfitting? Can you give some insights of it? Thanks.
", "votes": null, "replies": [] }, { "id": 269736, "author_name": "vrtjso", "author_url": "", "post_date": "01/17/2018 07:35:17", "content": "I am not sure, maybe it was overfitted to the first 5 days but failed to generalize to 6th ~ 16th day, or maybe the increase in LB is just random noise that cannot be relied on. As I said, I don't think the test data in LB is reliable in this competition.
", "votes": null, "replies": [] }, { "id": 269740, "author_name": "cpmpml", "author_url": "", "post_date": "01/17/2018 07:44:12", "content": "\n\n\nBtw I also found item_nbr overfitted my models and I dropped it.
\n
Thanks for confirming this.
", "votes": null, "replies": [] }, { "id": 269756, "author_name": "hubifeng", "author_url": "", "post_date": "01/17/2018 08:22:32", "content": "@Lingzhi The 16 days validation score is decreased after add some new features, maybe as you said \"overfitted to the first 5 days but failed to generalize to 6th ~ 16th day\". How to debug it when it's happening? Or what's you next steps? Thanks for response.
", "votes": null, "replies": [] }, { "id": 269922, "author_name": "sushize", "author_url": "", "post_date": "01/17/2018 14:31:20", "content": ">From you shared methods, you are using 20170503- 20170719 for training, 20170726 - 20170810 for validation. When you add new features every time, using the validation score to evaluate it, if the new features decrease the validation score for the first 5 days, last 11 days, and the full 16 days then add this features. Is my understand right?
\n\n>What confused me is that how to debug the overfitting? Can you give some insights of it? Thanks.
\n\n@Feng: Yes, when I add new features, I checked the cv score, and if they improved the cv score in all of the first 5day period, last 11 days period and full 16 days period, then I added the features. Of course, in addition to that, we need to make sure there is no bug in code ,and for feature engineering we didn't use leaks from true target (since such leak won't be available at all for LB period) for such CV score improvement. If everything seems fine, then I made submissions to see the LB score to verify that they indeed works on the LB period. If not, then I doubts: 1) whether there are any subtle bugs (including future label leaks, etc.) in my code that contributed to CV score improvement ----if yes, fix it; 2) whether there are some significant dissimilarity between my CV set and the test set--if yes, then try the best to make CV set as similar to the test set as possible.
\n\nFinally, if your feature engineering improved both your CV score and LB score consistently, and you think your CV set is very similar to the test set, then we should have some confidence that such engineered features are indeed good/helpful features.
\n\nFor this competition, my feature engineering efforts improved my CV and public LB score consistently, but not consistent for the private LB. I bet that there are some key (but subtle) difference between the public LB period and the private LB period that we haven't identified yet, which contributed to such phenomenon.
\n\nHope it helps.
", "votes": null, "replies": [] }, { "id": 269947, "author_name": "hubifeng", "author_url": "", "post_date": "01/17/2018 15:00:28", "content": "Help a lot!!! Thank you for your enthusiastic answer！
", "votes": null, "replies": [] } ] }, { "id": 269644, "author_name": "songkevinc", "author_url": "", "post_date": "01/17/2018 03:56:07", "content": "awesome! Thanks for sharing your approach.
", "votes": null, "replies": [] }, { "id": 269855, "author_name": "hubifeng", "author_url": "", "post_date": "01/17/2018 12:11:36", "content": "hi, @Su. The \"statistical features were generated independent for each week of the training data\" is still confused to me.
\nFor the training data (20170503- 20170719), we should construct the features and labels. If using the sales 2017/7/4-2017/7/19 as the labels, how to construct the features for it? or what's the labels in you training data? \nThanks for you time.
@feng: Let me clarify a little bit more. My framework/flowchart is similar to the one shared in the popular lgb starter script, such as this one https://www.kaggle.com/tunguz/lgbm-one-step-ahead-c8de0f. The training data (20170503- 20170719) including 12 weeks in total, similar to the starter script, we just construct the feature set x_temp, y_temp (line 113-121 in the attached starter script code) on each Wednesday, where x_temp including all historical features (e.g., past sales, past sales average, past sales weekly average, DayOfWeek average, and the associated stats features (median, max, std, sum), etc., while the y_temp are the future 16 days periods sales that we need to predict. Then we append/concate each weeks {x_temp, y_temp} together as the final combined train set.
\n\nTaking the week starting as 20170719 (wednesday) for example. The y_temp are the future 16 days (20170719-20170804) period sales, the x_temp are the past sales average/(and weekly average, DayOfWeek sales/average sales, etc.) until 20170718. And when I generating the statistical features, these \"target_encoding\" type features used the most important past_sales features in the x_temp (which only include sales information before or on 20170718) as target, and use those category variables (store_nbr, item_nbr, family, class, cluster, city, state, etc) as the key, and generate the corresponding stats (mean, median, std, sum, ). This is what I called \"target_encoded\" like statistical features. Obviously, since such generated stats feature only used information in x_temp (which only include sales information before or on 20170718), thus they are impossible to contain leaks from the future 16 days period sales labels y_temp.
\n\nNote that, \"target_encoding\" type features for category variable are used quite extensively (and in many case very helpful) in many different kaggle competitions, and you could see a lot of them by looking at previous competition. Usually, in those past competitions, the \"target\" there meant the true label that we need to predict, and we need to be very careful (and a bit tricky) to use techniques like out-of-fold labels, leave-one-out encoding, etc,, to reduce the overfitting to cv when generating such 'target_encoding\" features. Fortunately, in this competition, it is a time series competition, and thus we have the past sales data, which are very predictive (they are top features based on my lgb feature importance), so I just use such past sales as the 'target', and generate the \"target_encoding\" like features with key being those category variable, in this way the leak from future information (i.e., the true label, future 16 days period sales) are avoided completely, and thus if CV score improved by such features, they are not possibly due to leak from future label (y_temp), but just due to that these features and good engineering features that are indeed helpful (i.e., they become normal feature engineering not associated with the true label y_temp), at least for the time period considered in CV. And if such features improve both CV and LB score consistently, we might have some confidence that they are indeed features that works for both the CV and LB periods.
\n\nHope it helps.
", "votes": null, "replies": [] }, { "id": 269964, "author_name": "hubifeng", "author_url": "", "post_date": "01/17/2018 15:30:35", "content": "Thanks! It's very helpful!!!
", "votes": null, "replies": [] }, { "id": 270381, "author_name": "hubifeng", "author_url": "", "post_date": "01/18/2018 06:48:17", "content": "Hi @Shize Su , \nI found that the rossmann store sales forecasting in kaggle is very similar to this one, and you rank the top 1 in public leaderboard. In that competition, how did you design the validation mechanism? Is there difference with this one?
\n\nThanks in sincerely !
", "votes": null, "replies": [] }, { "id": 270552, "author_name": "sushize", "author_url": "", "post_date": "01/18/2018 13:49:07", "content": "@feng: That is a competition long time ago for me, so not sure I still remember the detail exactly. But roughly speaking, at the rossmann competition, we used some normal hold-out period data as validation, together with the public LB. I remembered that the local CV score and public LB score was not very consistent for us (it was somehow correlated but not perfect, especially when comparing different types of models like comparing xgb and ARIMA), but unfortunately we didn't pay too much attentions to adjust/fix the cv mechanism design to make it more consistent. We used the local cv to find improvements for models, and if improvements identified, submitted to the LB. For same type single model (e.g., xgb), local cv score improvements usually results in LB score improvement, but the improvement scale was not always the same (e.g., cv score improved by 0.0002 but LB score improved by 0.0001, or vice versa, etc.). The inconsistency in score improvement is much larger especially when comparing different type of models like xgb and ARIMA. For example, ARIMA worked very well (as a diverse component of our ensemble) on the first couple of days on LB (i.e., public LB) but performed much worse for longer future period on the private LB. Unfortunately, when inconsistency occurred, we trusted a bit more on the public LB rather than our CV, since we thought that the public LB period is closer to the private LB period than our held-out cv period.Thus we decided to assigned a bit larger ensemble weights for the ARIMA models in our final submission, and we were bitten by that quite a bit on final private LB (dropped from #1 on public LB to #7 on private LB). Then in the past two years, I more and more rely on a comprehensive CV mechanism when I want to play a competition seriously, and when the CV_LB score is inconsistent, I paid more attentions /spent much more efforts to design a more sophisticated CV mechanism to achieve the consistency. That's all for now.
\n\nHope it helps.
", "votes": null, "replies": [] }, { "id": 270613, "author_name": "hubifeng", "author_url": "", "post_date": "01/18/2018 15:41:48", "content": "Thanks!!! You helped me a lots!!!
", "votes": null, "replies": [] }, { "id": 275582, "author_name": "hubifeng", "author_url": "", "post_date": "01/29/2018 14:33:20", "content": "@Shize Su hi. I am back again. Thanks for you valuable answers. I found lots of shared solutions in this competition using similar cross validation settings. And It seem didn't concern about the holiday effects(such as the effect of before or after or in holiday?).
\n\nIn the rossmann competition, count features maybe a good choice, but how to using these features in this cv settings? or count features should using in another cv settings?
\n\nBesides, when I training the datasets with the same cv settings (there is no data leakages) , and look into the train log, I found that in some models the validation error is smaller than training error? Is this normal? or over-fitting? Maybe just the validation distribution is different from training data distribution?
", "votes": null, "replies": [] }, { "id": 275599, "author_name": "sushize", "author_url": "", "post_date": "01/29/2018 15:14:22", "content": "@feng:
\n\n1) About holiday, one key observation is that the LB period almost doesn't include any holiday (except a local holiday), and thus the benefits of considering holiday effect might only be marginal even if appropriately addressed. Not to mention that it is very tricky and time consuming to appropriately address the holiday effects to achieve such marginal improvement. This is why most competitors (including me) did not spend much time on dealing with the holiday effects. If the LB period includes many more National holidays, then it would be a totally different story.
\n\n2) About 'count features', I didn't see any clear explanations in your provided link, so i am not sure what specific features you were talking about.
\n\n3) The variance across the time periods is a bit large, thus it is possible (but not very likely) that validation loss is smaller than the training loss, especially when the model is under-fitting to the train set (for example, at the first couple of rounds of my lgb, in quite a couple of times I also observed smaller validation loss than training loss, but by the time early-stopping occurred, the training loss is usually much smaller than validation loss). So my suggestions would be : 1) double check the code to see whether there are any subtle mistakes/bugs; 2) make sure the model is sufficiently trained and not under-fitting to the train set; 3) if everything seems fine, then maybe it is ok-as I said, it is possible (but not very likely) that validation loss is smaller than the training loss for this dataset due to the variance across the time periods.
\n\nHope this helps.
", "votes": null, "replies": [] }, { "id": 275765, "author_name": "hubifeng", "author_url": "", "post_date": "01/30/2018 00:10:04", "content": "@Shize Su, hi. It's very helpful!
\n\nIf the LB period include the holiday, what's the general methods for this cv settings? Any suggestion?
\n\nAbout the count features in the links, nimashahbazi create the features like below:\n
If the store is closed, the open variable is zero and the sales is zero too. Then he create a dummy variable called MyRowHoliday to capture the information for these zero sales. This dummy variable assigned positive integers before and after the consecutive close and “-1” on other days.
\n\nIn this way, maybe it capture the holiday effects(such as the effect of before or after or in holiday)?
", "votes": null, "replies": [] }, { "id": 275768, "author_name": "sushize", "author_url": "", "post_date": "01/30/2018 00:27:59", "content": "@feng:
\n\n1) If the LB period includes more major holidays, a general guideline is to design the CV data set to be as similar to LB data set (e.g., similar period but in 2016; or period close to LB period but also have similar amount of major holidays). But obviously there is not a simple clear answer about the best cv setting in this case, it might require significant amount of time and efforts (more difficult than the case here) to find a CV setting for consistent CV_LB score trend. You can only figured it out by various of experiments, following the key guideline that CV set should be as similar to LB set as possible.
\n\n2) About the \"count features\" you mentioned above in Nima's post, they are not some common \"count features\" we usually use, and in my opinion those features are somewhat specific to the Rossmann data set. (For the term \"count features\", we (or at least me) usually mean sth like frequency of some category variables, etc. That's what made me confused when you used the term \"count feature\".) Finally, such type of features are probably much less helpful (though I didn't test it) for this competition since there are no that many \"close-open\" switching for stores in this data set (compared with rossmann competition data set), plus that there is no way to know whether a store is closed or not on a date of test data period. But maybe using such so called \"count features\" on the \"promotion\" variable might be somewhat helpful, though needs to be very carefully dealt with due to the large asymmetry in distribution of \"promotion\" variable in train and test data.
\n\nHope this helps.
", "votes": null, "replies": [] }, { "id": 275790, "author_name": "hubifeng", "author_url": "", "post_date": "01/30/2018 01:34:09", "content": "Thanks for clarify my doubts. You push me more interest in the competition, there are a lots of modeling skills to learn from you and some people like you. Thanks for your generously!
", "votes": null, "replies": [] } ] }, { "id": 270117, "author_name": "paulz11", "author_url": "", "post_date": "01/17/2018 19:07:09", "content": "I have spent lots of time to train July/Aug data in year of 2014-2016, that doesn't help at all. And it is the major reason that my private LB dropped by 219th. I guess the sales pattern doesn't change much month over month in the summer on grocery. If the test period is December, the seasonality will be much more important, and then we will need train same period previous year.
", "votes": null, "replies": [] }, { "id": 271287, "author_name": "frankcorrigan", "author_url": "", "post_date": "01/20/2018 03:13:16", "content": "I'm pretty new in these contest but thought this might help. I compared the scores of my models between the public and private leaderboards and I noticed that the submissions where I tried to do something with the promotions data we're the ones were the score improved on the private LB from the public. For example, to fill the missing values in the training set... I determined the average sales lift from promotion by store/item (where we had promo data)... And then I filled na values where the sales of a store/item we're missing and the unit sales was x% above average for that pair. My local CV process was basically nonexistant, but I did notice that when I tried these sorts of things with the promo data my private LB scores were > public LB scores by not trivial amounts.
", "votes": null, "replies": [] }, { "id": 275233, "author_name": "zhixing629", "author_url": "", "post_date": "01/28/2018 14:38:32", "content": "Note that, in order to avoiding possible future target information leak, such statistical features were generated independent for each week of the training data, validation data, and test data.\nHello Shize Su.Does What does it mean? Does it mean separate training data in one week and generate feature?Expect your reply.Thanks
", "votes": null, "replies": [ { "id": 275243, "author_name": "sushize", "author_url": "", "post_date": "01/28/2018 15:02:21", "content": "Hi zhixing,
\n\nAbout generating the statistical features (i.e., 'target-encoding' type features for category variables) , the key is to be careful to avoid utilizing (explicitly or implicitly) any future target information when generating these features.
\n\nBy generating the statistical features for each week independently, we could completely avoid such future target information leak since at that time all the past sales data definitely not including any future sales data information, and the derived statistical features based on these past sales data of course is impossible to contain leak info from future targets.
\n\nTo be more specific, the generation of such statistical features should be put inside a function similar to \"def prepare_dataset(t2017, is_train=True):\" in the lgb startet script https://www.kaggle.com/tunguz/lgbm-one-step-ahead-c8de0f, i.e., it should be generated inside the function for generating the features X_l and y_l for each week, rather than after operation \"X_train = pd.concat(X_l, axis=0)\". If you generating the statistical features after operation \"X_train = pd.concat(X_l, axis=0)\", then features for earlier weeks will implicitly contain future target leak information from later weeks. For example, the past day1 sales for week 2017/07/19 might be future information for the previous week 2017/07/12, and if generating such 'target-encoding' type statistical features for week 2017/07/12 after the operation \"X_train = pd.concat(X_l, axis=0)\", then it contains the future information leak (e.g., past day1 sales for week 2017/07/19, etc.). And such kind of future info leak is sth we need to avoid to make such 'target-encoding' type features works on the LB period, where future info leak is not possible.
\n\nHope it helps.
\n\nBest regards,
\n\nShize
", "votes": null, "replies": [] }, { "id": 276648, "author_name": "zhixing629", "author_url": "", "post_date": "02/01/2018 03:02:25", "content": "Thanks your explanation.Your solution and analyse help me learn a lot in this competition.Hope you can stick to it.
", "votes": null, "replies": [] } ] }, { "id": 285317, "author_name": "adityaecdrid", "author_url": "", "post_date": "02/19/2018 22:55:17", "content": "Thank you for your in-depth post...
", "votes": null, "replies": [] }, { "id": 574650, "author_name": "longyin2", "author_url": "", "post_date": "07/14/2019 08:58:18", "content": "Thanks for sharing.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "269468": "As promised in the post https://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/47326 , now I am sharing my 10 days journey experience in this competition and my approach^_^ \n\nIn addition to sharing my approach, I am also most interested in discussing why there is such a big shake-up on the private LB, and I would like to hear your thoughts and/or findings as well^_^\n\n(Warning: This is going to be a long post, hope you have some patience^_^)\n\nAs I mentioned in that post, I entered this competition quite late (when there was only 10 days left). At that time I just finished the last master only competition and took a short break. Since I had some spare time, I planed to explore this competition a bit, simply for fun. Finally, I am ranked 8th place on the public LB (score 0.503), but unfortunately shaked down a bit on the private LB (score 0.516). But In general, I am quite happy and satisfied with the score I achieved in this competition, given that I entered this competition so late. \n\n____________________________________________________________________________________________________________________\n\n**Before introducing the flowchart of my approach, I want to talk a bit about the CV_LB relationship as well as the final private LB shake-up. I would also love to hear your thoughts and findings on this.**\n\nAs I mentioned in the previous post https://www.kaggle.com/c/favorita-grocery-sales-forecasting/discussion/47326 , my local CV score and public LB score correlates somewhat reasonably well (especially when comparing same type of models, e.g., comparing different feature engineering ideas for lgb models, or comparing different feature engineering ideas for nn models, for the comparison between nn and lgb models the CV-LB gap was indeed a bit different (nn showed significantly better cv score thank lgb on the similar feature set, while performed slightly worse than lgb on the LB)), which enabled me make progress somewhat smoothly on the public LB and climbed to 8th place on the public LB in the limited 10 days. The final public LB-private LB shake up is much larger than I expected given my somewhat reasonably good CV-LB relationship, and I have't fully figure out why. I did expected some extent of medium shaking up between public and private LB ranking, but definitely not as large as what it turned out to be. I understood that the promotion information is biased between the train and test set, and feature engineering efforts based on such promotion information might be easily over-fitting if not dealt with very carefully, and I guess many competitors relied on feature engineering ideas based on such promotion features to improve the public LB score, which might possibly to some extent explain the big shake-up. But I have no a certain idea on this since I don't have full knowledge about how other competitor did when getting their public LB score. \n\nOn my side, I did almost zero additional feature engineering efforts based on the promotion variable (except those already used in the shared lgb starter script), and my improvement of public LB score from 0.513 (lgb starter script score) to 0.505 for a single lgb relied on feature engineering efforts that is not related to promotion variable at all, and my local CV-Pub LB relationship is somewhat consistent, though not perfect. I am a bit confused why the CV-LB relationship for my models on the private LB is much less correlated, even if I almost didn't use any additional promotion-based features (compared with the lgb starter script) to achieve my score improvements in CV and public LB. Ironically, it turned out that my 1st entry of this competition, which is a lgb scored 0.511 on public LB and 0.517 on private LB, is my best single model on the private LB. In comparison, my final best single lgb on the public LB (scored 0.505 on public LB) also only gave 0.517 on private LB (and slightly worse than my 1st entry lgb). Namely, all my added feature engineering efforts (which are not related to promotion variable at all) improved both my local CV and public LB score significantly and somewhat consistently, turned out to adding no extra values (and even some small negative effects) to the private LB score, in comparison with my 1st entry lgb in this competition. Bias in promotion variable in train and test set can to some extent explain the shake up on public/private LB for many competitors if they relied on promotion-based features to climb the public LB, but definitely still could not explain many other scenarios, such as my case here that almost no promotion-based engineered features are used to achieve CV/Public LB score improvement, and CV_Public LB patterns are somewhat consistent. \n\nI would really like to hear the thoughts and experiences from other competitors to understand what actually caused such phenomenon. In addition to the promotion variable bias, are there anything else important we missed here that contributed to the shaking up? Or it is just due to the randomness /unstable time series trends, and the shaking is simply due to luck? If many of us are happy to share our thoughts and findings, maybe we would finally be able to figure out what would be the root cause for such a big shake-up and understand this problem much better, rather than simply jumping to the conclusion that the shaking up is due to lacking of comprehensive cv experiments, or bias for promotion variable.\n\n___________________________________________________________________________________________________________________\n__________________________________________________________________________________________________________________\n__________________________________________________________________________________________________________________\n__________________________________________________________________________________________________________________\n\n\nNow let me start introducing my approach.\n\n\nFirst of all, credits are given to those people who shared the great starter scripts in the kernel, especially to @tunguz, @ceshine. Your starter scripts (see links below) gave me a great starting point in this competition, and enable me to quickly work on key feature engineering and modeling experiments for fast progression in this competition. Actually, these shared great starter scripts is also one major motivation for me to enter this competition when there is only 10 days left; usually I will only enter a competition when there is much more time left.\n\nhttps://www.kaggle.com/tunguz/lgbm-one-step-ahead-lb-0-513\nhttps://www.kaggle.com/ceshine/lgbm-starter\n\n__________________________________________________________________________________________________________________\n**Sample Data Selection:**\n\nMy final models used only 2017 data to extract features and construct samples. I tried to include more data, such as those 2016-Jul and 2016-Aug data (similar month/week periods as my validation period and LB period) to capture the monthly dynamics, but they only helped marginally or not helpful at all, depending on the models, while take significant longer training time. So I decided to drop those data from my final models.\n\n**train data：**20170503- 20170719 for cv experiments, and 20170503-20170809 for LB experiments (namely, for LB submissions I retrained the models adding the most recent days data since they are closest to the LB period and might be most valuable)\n\n**validation data:** 16 days period, 20170726 - 20170810 (and also tried using validation period 20160817-20160901 for some of my nn models at the very end of the competition, but just some simple trial since no much time left for me, and it turned out to perform worse than using 20170726 - 20170810 ). I computed the CV score for the first 5 days, last 11 days, and the full 16 days (simulating the LB split). I observe somewhat consistent CV-LB score improvement trends in all of my experiments, especially when comparing same type of models (e.g., comparing different lgbs, or comparing different nns). For my feature engineering efforts, I even observed even larger cv score improvement for the last 11 days (similar to the private LB) than the cv score improvement for the first 5 days (similar to the public LB), and thus at some time points neard the end of the competition, I once believed that I probably would climbed up a couple of spots on the private LB, due to such CV-LB score findings. Unfortunately, it didn't work as nice as I expected and I was shaked down a little bit. I still don't understand why for this.....\n\nSince starting date 2017/08/16 for LB period is Wednesday, and thus it is essential to also set the starting date for the validation period as Wednesday to preserve the DayOfWeek pattern.\n\n**One key trick to make my local CV and public LB score more consistent is as follows**:\n\n**Step I:** I extract all the store_item combinations that appeared in the period from 2017-07-01 to 2017-08-15. \n\n**Step II:** I only keep the training data records (and drop all other records) whose store_item combination appeared in the extracted store_item combinations in Step I. The underlying assumption is that, if a item was not sold (or with identical 0 sales) at all in a store in the period 2017-07-01 to 2017-08-15, it is very likely that it wouldn't be sold (or with identical 0 sales) in the LB period 2017-08-16 to 2017-08-31 either. By looking at the train data, it seems to me that the new store_item combinations are added very slowly in train set, and when considering a short period like 16 days, the amount of newly added \nstore_item combination (with non_zero sales) is almost negiligible.\n\n**Step III:** I also only keep the test data records whose store_item combination appeared in the extracted store_item combinations in Step I. I always predict zero target values for the other store_item combination that don't belong to the store_item combinations extracted in Step I.\n\n(My such validation trick was also partially inspired by the Kaggle Admin Inversion's feedback on the unseen new items in this post , who mentioned that \"I'm jumping in here late, regarding the discussion of new products, in order to clarify a bit. As has been mentioned, there are a very small number of new items that were introduced in the test set. That does not explain the apparent high number of new items seen on the first day of the test set. The reason for this is that the training set does not include records for zero sales. The test set, though, includes all store / item combinations, whether or not that item was seen previously in a store.\" Thus I made an assumption that \"apparent high number of new items seen on the first day of the test set\" are mostly items that have been seen previously in a store but just with zero sales, while the additional \"small number of new items that were introduced in the test set\" are those new store_item combinations if we compare later date of the test set and the first date of the test set. Since this amount is small, I assumed that it is safe to simply always predict zero target values for all those store_item combination that don't belong to the store_item combinations extracted in Step I. And such trick indeed made my CV-public LB relationship reasonably consistent, and I thought that I got it. But I don't know why the same consistency didn't preserve well on the private LB.....)\n\nAfter this 3-step trick and using the 16 days validation period 20170726 - 20170810, (and train period 20170503-20170719), I observed somewhat reasonably consistent CV_LB relationships when testing my feature engineering ideas, especially when comparing same type models (comparing different lgbs, or compering different nns), and it helped me a lot to make smooth progress on the public LB.\n\n__________________________________________________________________________________________________________________\n**Feature Engineering:**\n\n1. Just follow the lgb starter scripts and adding more past sales (and DayOfWeek sales) average/ (weekly average) features . I also add (mean, median, std, sum, max, min) for many of such past sales average/ (weekly average) features. This basically gave my 1st entry submission of this competition, which scored 0.511 on the public LB and 0.517 on the private LB. It seems to me that this part didn't break the CV-Public LB-Private LB relationship yet.\n\n2. Adding a little bit more promotion features similar to the starter lgb script (similar to those \"promo_14_2017\" , \"promo_60_2017\":, \"promo_140_2017\" type features, but just used more periods like 7, 21, 35, etc). This improved both the CV and public LB score by about 0.0015. But private LB score dropped from 0.517 to 0.518. I guess bias in promotion variable might to some extent explain such broken in CV-LB score consistency, but can't fully explain it, since CV_public LB consistency is still preserved by adding these promotion based features. These several features are the only additional promotion-based features that I used in my final publicLB 0.505 lgb. From now on, all the feature engineering efforts that improved my public LB score from 0.509 to 0.505 (and also improved my CV score in a similar scale) are all not related to promotion variable at all.\n\n3. Adding the category features from stores.csv and items.csv, and apply label encoding. This improved both the CV and Public LB score by about 0.001+, to get 0.508 public LB score for a single lgb. Private LB score stay the same at 0.518. Not sure why, such category feature should not likely to break CV-LB relationship seems they are no biased. Is that possible this is simply due to that the time series trend pattern in the LB period is not very stable, and luck played a significant role? Or anything important we missed here?\n\n4. Adding statistical features (or \"target_encoding\" like features for category variables): I use some methods to stat some targets for different keys in different time windows (this is somewhat similar to that one shared the top 1 place solution, with some additional difference being the choice of target and key, as follows:)\n\nkey：'store_nbr', 'item_nbr','family', 'class', 'city','state', 'type', 'cluster' (i.e., all the used category variables)\n\ntarget: I output the lgb feature importance for the above lgb and selecting those most important past sales average/ (weekly average) features & DayOfWeek features, as well as the associated stats features (std, sum, median, etc.). In particular, the complete list of my chosen \"target\" features are as follows: \n'mean_7_2017_01', 'mean_14_2017_01', 'mean_21_2017_01', 'mean_35_2017_01', 'mean_30_2017_01', 'mean_60_2017_01','ahead7_6', 'ahead7_5', 'ahead7_4', 'ahead7_3', 'ahead7_2', 'ahead7_1', 'ahead0_6','ahead0_5', 'ahead0_4', 'ahead0_3', 'ahead0_2', 'ahead0_1', \n 'mean_4_dow0_2017', 'mean_8_dow0_2017', 'mean_16_dow0_2017', 'mean_4_dow1_2017', 'mean_8_dow1_2017', 'mean_16_dow1_2017', 'mean_4_dow2_2017', 'mean_8_dow2_2017', 'mean_16_dow2_2017', 'mean_4_dow3_2017', 'mean_8_dow4_2017', 'mean_16_dow3_2017','mean_4_dow4_2017', 'mean_8_dow4_2017', 'mean_16_dow4_2017','mean_4_dow5_2017', 'mean_8_dow5_2017', 'mean_16_dow5_2017',\n'mean_4_dow6_2017', 'mean_8_dow6_2017', 'mean_16_dow6_2017', 'day_1_2017','day_2_2017','day_3_2017','day_4_2017','day_5_2017','day_6_2017','day_7_2017' \n\nmethod:\nmean, median, max, min, std\n\nNote that, in order to avoiding possible future target information leak, such statistical features were generated independent for each week of the training data, validation data, and test data.\n\nSo in total there are about 1400+ such statistical features, and expand my full feature set size to about 2300+. I then run lgb and output the feature importance, and only select the most important 400 features and use them for my later modeling (for lgb and nn). Such feature selection reduce the training time significantly, and also improved CV and public LB score by about 0.0006 simply by itself.\n\n\nSuch statistical features are very helpful for improving my CV and public LB score. In particular, it improved my single lgb score from 0.508 on public LB to 0.505 on the public LB, and similar scale of improvement is observed in local CV. However, the private LB score only improved slightly from 0.518 to 0.517.\n\nMy nn models were basically trained on the same feature set as my best single lgb, except that I replacing the label-encoding category features by one-hot encoding (i.e., dummie features) category features, since label encoding of category variables doesn't make much sense for models like nn. My best single nn scored 0.506 on public LB & 0.520 on private LB, and an average of two nn with some small difference gave me 0.505 score on the public LB and 0.518 on the private LB. Note that I have only explored nn model when there is only 2 days left (after I am almost done for lgb model and feature engineering) and there is no much time for me to tune the nn though. I just made it somehow work. \n\nEnsemble of my best lgb and nn gave my final score, 0.503 on public LB and 0.516 on private LB. The ensembling step of nn and lgb does seem to preserve the CV_Public LB_Private LB relationship, namely, they improved the score all by about 0.002.\n\n**To sum up, based on my experience, what seems to preserve the CV_Public_Private score relationships are**:\n\n a) the feature engineering effort 1 described above (adding more past sales weekly average, DayOfWeek sales, as well as the associated mean, std, min, max, sum stats features, this preserve CV-Public-Private relationship nicely) \n\nb) feature engineering effort 3 described above (\"target_encoding\" like statistical features for category variables, this preserves the CV_Pub_Private relationship somehow ok, but not perfect, CV_Pub score improved by about 0.003, private LB score improved by about 0.001), \n\nc) ensemble of nn and lgb. \n\n**What seems to not preserving the CV_Public_Private score relationships are:**\n\n a) the feature engineering effort 2 described above (i.e., adding some more promotion-based features similar as the lgb starter scripts, i.e., similar to \"promo_14_2017\" , \"promo_60_2017\":, \n\"promo_140_2017\" type features, but just used more periods like 7, 21, 35, etc). This might be somehow understood due to the bias in promotion variable.\n\nb) Adding the category features from stores.csv and items.csv, and apply label encoding. This one didn't preserve the CV_Pub_Private relationship makes me completely confused, since they are not biased between CV_Pub_Private. And note the the CV_Pub score relationship is still nicely preserved when adding such category variables, not sure why it didn't work on private LB.\n\nc) for the feature engineering effort 4 described above (\"target_encoding\" like statistical features for category variables), they improved both CV and public LB score by about 0.003+, but only improved private LB score by about 0.001 also make me a bit confused. Such features are not related to the promotion variable at all, and why it broke the CV_Private LB relationship (given that the CV_Public LB was still nicely preserved)?\n\n____________________________________________________________________________________________________________\n\n**Single Model:** \n\npublic LB 0.505/ private LB 0.517, single lgb\n\npublic LB 0.506/ private LB 0.520, single nn\n\naverage of 2 nn with small differences gave public LB 0.505/ private LB 0.518.\n\n__________________________________________________________________________________________________________\n\n**Final Ensemble:**\n\n0.54-0.46 weighted average of lgb and nn, public LB 0.503/ private LB 0.516.\n\n__________________________________________________________________________________________________________\n\nThat's all for my approach in this 10 days journey of this competition. Overall speaking, I am satisfied with what I achieved in this competition given the limited time, and I have learned a couple of useful things from competitors and my experiments when addressing this time series challenge. To the end, I would really love to hear about your thoughts and findings about what preserves the CV_LB relationships and what didn't preserve the CV_LB relationships, such as sth similar to what I shared above. For now I still couldn't fully understand why the relationship is not preserved on the private LB, given that CV_Pub LB relationship is somewhat consistent for me. By combining our findings, maybe we could have a deep understanding about why there would be such a big shake up on the private LB, rather than simply jumping to the conclusion that it is due to lacking of comprehensive CV experiments or just luck/randomness. Maybe we missed sth important but subtle, which contributed to the big shake-up on the final private LB, and if we figured out that, maybe there is indeed a nice way to preserve the CV_Pub_Private score relationship very well if done appropriately.\n\n__________________________________________________________________________________________________________\n\n Finally, thanks to Kaggle and the sponsor to host such a great competition, and thanks to all other competitors who made my journey in this competition fun! And, congratulation to the winners!\n\nBest regards,\n\nShize", "269478": "Thanks for the write up! Very informative :)", "269532": "Thanks for sharing. I found that using item_nbr was leading to overfit in my local CV. This may explain part of the shakeup.", "269537": "Thanks for the info. But why? There should not be any bias for item_nbr between public LB and private LB, and thus this factor cannot explain why my CV_Pub LB relationship is consistent, but the consistency broken in private LB..... As I mentioned above, with my CV mechanism, adding item_nbr as well as other category variable improved both my local CV score and Public LB score consistently by about 0.001+, while private LB score stayed the same. This seems indeed very strange to me, and I have no clues why.... Could anyone else confirmed whether they encountered similar situations (i.e., improve both CV and public LB score consistently, but not private LB) when adding the category variables?", "269554": "I'm not sure how to express the gut feeling I have here. Let me reverse the question: why would item_nbr help at all? Why would be the pattern for sales for a given item be significantly different from another one? In the WTF competition I did not use the page either as a feature. It forces the model to generalize on how sales evolve over time across items, instead of learning a pattern for each item, which increases variance and the risk of overfit. But I may be totally misled here, i welcome other feedback too.\n\nThe other reason may be your choice of validation period: the test period ends with month end, i.e. a payday. We selected our validation periods to end with paydays, with the hunch that this was a significant pattern. I must say this was Giba's idea, I was tempted to do exactly as you did ;)", "269563": "In my case, adding: \n- Frequency Encoding (store, item, family, class) \n- One-Hot_Encoding (clusters, states, types, cities, families) \n \nDid the following: \n###Days 1 --> 5 \n- Improvement on public leaderboard of less than .001. \n- Improved CV of first 5 days by .0013 \n\n###Days 6 --> 16 \n- Improved private LB score by .002 (or less)\n- Unfortunately, I do not have the CV total error for days 6-16, but the average 'day_error' improvement was .0018.", "269569": "Thanks for the feedback, CPMP.\n\n> \nLet me reverse the question: why would item_nbr help at all? Why would be the pattern for sales for a given item be significantly different from another one?\n\n\nFor me the underlying assumption is that, the sales pattern for different items (there are about 10k different items) are possibly somewhat different from each other, though the extent of dissimilarity is hard to tell. At least, it is possibly fair to say that the past sales data for the same item is possibly a bit more helpful than the past sales data for a different item. According to my intuition, adding the item_nbr feature (as well as other category features like store_nbr, family, cluster, etc.) would help the lgb model to more easily identify those most similar records and gave them a bit more weights, while without the item_nbr, since there are about 10k items in total, such info might be just buried in the large number of records and harder for lgb to identify. The same is for other category variables like store_nbr, etc. Anyway, the key for feature engineering efforts is just to make those information more easily captured by the models like lgb. Theoretically speaking, all the information are contained in the given raw data, our feature engineering efforts just convert those information into a form which is easier for the machine learning models to capture, rather than creating new information.\n\nAlso note that, adding all those category variables (store_nbr, item_nbr, family, class, cluster, etc.) in total gave about 0.001+ improvement in both my CV and Public score, which is not very large but should be nontrivial. It might not necessarily to require \"the pattern for sales for a given item being 'significantly' different from another one\" in order for item_nbr to be helpful in such a extent of 0.001+ score improvements. As long as there are some nontrivial dissimilarity among the sales patterns for different items (and since there are about 10k items in total, I do bet this is true), it is reasonable to accept that adding item_nbr to be helpful (though it is also acceptable if adding item_nbr not helpful, when considering the generalization mentioned by you), namely, adding item_nbr being helpful seems not to be a weird result for me. \n\nWhat confused me was why adding such category variables (item_nbr, store_nbr, family, cluster, etc.) helps both my CV and public LB, but not private LB. But your raised point about pay day in validation period is indeed a factor that I never considered in my experiments, and it might possibly explain the broken CV_LB relationship on private LB, though I am not sure since I didn't do experiments on this. Also note that the top 1 team also used exactly the same validation period (2017/07/26-2017/08/10) as me, but did not suffer the same issue on private LB, so I am not sure whether the issue is indeed associated with my validation period choice...", "269574": "> Improved private LB score by .002 (or less)\n\n@Nicolas: So it seems that adding those category variables seems indeed contribute to score improvements in all of your CV, Public LB and Private LB scores, which is different from mine. May I know which validation periods you were using? Also 16 days period starting at 2017/07/26? Or sth else?", "269578": "Shize, you did notice yourself that \"3b) Adding the category features from stores.csv and items.csv, and apply label encoding \" broke the CV LB relationship. I tend to believe that it leads to overfit. Good point re team #1 using the same validation period as you. I haven't studied their code yet, so I cannot comment further.", "269584": "CPMP: Yeah, the thing made me confusing is that adding category features from stores.csv and items.csv only broke my CV_Private LB relationship, and the CV_Public LB relationship was still preserved nicely. That's what made me confused. If it also broke my CV_LB relationship, then I would say maybe it is over-fitting to CV and has worse generalization. But since it improved both CV and Public LB score in a similar scale but not improved the private LB score, this made me confused.", "269591": "This dataset was confusing in many ways for me. For instance I tried to use sales from same class as features, it improved CV nicely, and resulted into a public LB of 0.77 !?! I may have done something wrong as I was quite feverish that day, but it made me very cautious about not overfitting by adding too many features. I'm impressed that you and others (eg @Nicolas) managed to add so many features without overfitting.", "269607": "CPMP: Yeah, I just have a feeling that it is probable that we might still miss sth important but sbtle, and if we identify that one, maybe it is possible to achieve consistent CV_Public LB_Private LB score relationship, rather than making the final private LB somehow a lottery. My feeling was due to that : 1) My CV_Public LB relationship is reasonably consistent for my feature engineering ideas; 2) It seems to me that there is no too much difference/bias between public LB and private LB, in terms of those feature engineering efforts I tried. Thus, unless I missed anything, I have such a feeling that achieve consistent CV_Public LB_Private LB score relationship might be possible, there might be just some subtle issue that is hard to figure out that we didn't find it. That's the main motivation why I initiated this discussion about the CV_LB relationship question here.", "269617": "For me, the CV and LB performances were not consistent, and not consistent even between different models, this made me confused for a long time. My Seq2Seq model gave me the best CV result but was stuck at .512 at LB, my CNN model was slightly worse and not stable in CV but got .505 in LB. Finally these two models both achieved .514 in PB. Generally, for me the CV and PB are quite consistent but the LB is problematic.\n\nActually after noticing the strange pattern of promotion infos in 8/16, which is stated in my solution, I told myself the LB is problematic and cannot be trusted. The CV cannot be trusted either because of the biased promotion info. I have to say there is some luck here to bet on the distribution of the private test data.", "269619": "Lingzhi: You mentioned that your CV and Private LB is somewhat consistent, but not the public LB. Mine is the other way, namely, CV and Public LB is consistent, but not the private LB. May I know which validation periods/method you were using?", "269623": "I only used 7/26-8/10 for validation for the whole time, and used similar ideas as yours: keeping the store-item combos that are in test data and have values during 2017 only.", "269627": "Linzhi Hmm, then it is quite strange why my models preserve CV_Public LB score consistency, while your models preserve CV_Private LB score consistency, given that we were using similar CV mechanism... \n\nHmm, after a second thought, a possible explanation is that this might be due to the different type of feature engineering efforts (and/or modeling approaches, I didn't try any sequence models in this competition) between us (at least this is the only thing I can imagine for now). Namely, some type of feature engineering efforts might work for validation period and public LB period, while some other type of feature engineering efforts might work for validation period and private LB, due to some subtle difference between public LB and private LB that we haven't identified yet.", "269630": "This is what I am confused too. Btw I also found item_nbr overfitted my models and I dropped it.", "269644": "awesome! Thanks for sharing your approach.", "269716": "Lingzhi I use you shared LGBM starter code, but what confused me is that when i add some new features, the local cv decrease, but the public lb score increase, why? It's overfitting the validation data?", "269733": "Shize su. From you shared methods, you are using 20170503- 20170719 for training, 20170726 - 20170810 for validation. When you add new features every time, using the validation score to evaluate it, if the new features decrease the validation score for the first 5 days, last 11 days, and the full 16 days then add this features. Is my understand right?\n\nWhat confused me is that how to debug the overfitting? Can you give some insights of it? Thanks.", "269736": "I am not sure, maybe it was overfitted to the first 5 days but failed to generalize to 6th ~ 16th day, or maybe the increase in LB is just random noise that cannot be relied on. As I said, I don't think the test data in LB is reliable in this competition.", "269740": "> Btw I also found item_nbr overfitted my models and I dropped it.\n\nThanks for confirming this.", "269756": "Lingzhi The 16 days validation score is decreased after add some new features, maybe as you said \"overfitted to the first 5 days but failed to generalize to 6th ~ 16th day\". How to debug it when it's happening? Or what's you next steps? Thanks for response.", "269763": "had a similar experience, simply adding the item_nbr + store_nbr basically \"broke\" the public LB score; when I check my submissions, I have one with private 0.517 but public 0.908 (and another one 0.517 and 0.699), the CV scores of these two compared to my other submissions were almost the same; there really is something \"weird\" in the public LB\n\nbtw. I was also validating on 2017-07-26 and I believe that was the key for staying in the \"medal area\"", "269855": "hi, @Su. The \"statistical features were generated independent for each week of the training data\" is still confused to me. \nFor the training data (20170503- 20170719), we should construct the features and labels. If using the sales 2017/7/4-2017/7/19 as the labels, how to construct the features for it? or what's the labels in you training data? \nThanks for you time.", "269891": "Very interesting discussion. As I mentioned, adding categorical and frequency encoding improved Public LB the least for me. I kept it because it improved my CV quite a bit. \n\nI was also using 7/26-8/10 --- using all and only items that appeared between 2017/01/01 and 2017/08/10.", "269915": "feng: Let me clarify a little bit more. My framework/flowchart is similar to the one shared in the popular lgb starter script, such as this one https://www.kaggle.com/tunguz/lgbm-one-step-ahead-c8de0f. The training data (20170503- 20170719) including 12 weeks in total, similar to the starter script, we just construct the feature set x_temp, y_temp (line 113-121 in the attached starter script code) on each Wednesday, where x_temp including all historical features (e.g., past sales, past sales average, past sales weekly average, DayOfWeek average, and the associated stats features (median, max, std, sum), etc., while the y_temp are the future 16 days periods sales that we need to predict. Then we append/concate each weeks {x_temp, y_temp} together as the final combined train set.\n\nTaking the week starting as 20170719 (wednesday) for example. The y_temp are the future 16 days (20170719-20170804) period sales, the x_temp are the past sales average/(and weekly average, DayOfWeek sales/average sales, etc.) until 20170718. And when I generating the statistical features, these \"target_encoding\" type features used the most important past_sales features in the x_temp (which only include sales information before or on 20170718) as target, and use those category variables (store_nbr, item_nbr, family, class, cluster, city, state, etc) as the key, and generate the corresponding stats (mean, median, std, sum, ). This is what I called \"target_encoded\" like statistical features. Obviously, since such generated stats feature only used information in x_temp (which only include sales information before or on 20170718), thus they are impossible to contain leaks from the future 16 days period sales labels y_temp.\n\nNote that, \"target_encoding\" type features for category variable are used quite extensively (and in many case very helpful) in many different kaggle competitions, and you could see a lot of them by looking at previous competition. Usually, in those past competitions, the \"target\" there meant the true label that we need to predict, and we need to be very careful (and a bit tricky) to use techniques like out-of-fold labels, leave-one-out encoding, etc,, to reduce the overfitting to cv when generating such 'target_encoding\" features. Fortunately, in this competition, it is a time series competition, and thus we have the past sales data, which are very predictive (they are top features based on my lgb feature importance), so I just use such past sales as the 'target', and generate the \"target_encoding\" like features with key being those category variable, in this way the leak from future information (i.e., the true label, future 16 days period sales) are avoided completely, and thus if CV score improved by such features, they are not possibly due to leak from future label (y_temp), but just due to that these features and good engineering features that are indeed helpful (i.e., they become normal feature engineering not associated with the true label y_temp), at least for the time period considered in CV. And if such features improve both CV and LB score consistently, we might have some confidence that they are indeed features that works for both the CV and LB periods.\n\nHope it helps.", "269922": ">From you shared methods, you are using 20170503- 20170719 for training, 20170726 - 20170810 for validation. When you add new features every time, using the validation score to evaluate it, if the new features decrease the validation score for the first 5 days, last 11 days, and the full 16 days then add this features. Is my understand right?\n\n>What confused me is that how to debug the overfitting? Can you give some insights of it? Thanks.\n\n@Feng: Yes, when I add new features, I checked the cv score, and if they improved the cv score in all of the first 5day period, last 11 days period and full 16 days period, then I added the features. Of course, in addition to that, we need to make sure there is no bug in code ,and for feature engineering we didn't use leaks from true target (since such leak won't be available at all for LB period) for such CV score improvement. If everything seems fine, then I made submissions to see the LB score to verify that they indeed works on the LB period. If not, then I doubts: 1) whether there are any subtle bugs (including future label leaks, etc.) in my code that contributed to CV score improvement ----if yes, fix it; 2) whether there are some significant dissimilarity between my CV set and the test set--if yes, then try the best to make CV set as similar to the test set as possible.\n\nFinally, if your feature engineering improved both your CV score and LB score consistently, and you think your CV set is very similar to the test set, then we should have some confidence that such engineered features are indeed good/helpful features.\n\nFor this competition, my feature engineering efforts improved my CV and public LB score consistently, but not consistent for the private LB. I bet that there are some key (but subtle) difference between the public LB period and the private LB period that we haven't identified yet, which contributed to such phenomenon.\n\nHope it helps.", "269947": "Help a lot!!! Thank you for your enthusiastic answer！", "269964": "Thanks! It's very helpful!!!", "270117": "I have spent lots of time to train July/Aug data in year of 2014-2016, that doesn't help at all. And it is the major reason that my private LB dropped by 219th. I guess the sales pattern doesn't change much month over month in the summer on grocery. If the test period is December, the seasonality will be much more important, and then we will need train same period previous year.", "270381": "Hi @Shize Su , \nI found that the rossmann store sales forecasting in kaggle is very similar to this one, and you rank the top 1 in public leaderboard. In that competition, how did you design the validation mechanism? Is there difference with this one? \n\nThanks in sincerely !", "270552": "feng: That is a competition long time ago for me, so not sure I still remember the detail exactly. But roughly speaking, at the rossmann competition, we used some normal hold-out period data as validation, together with the public LB. I remembered that the local CV score and public LB score was not very consistent for us (it was somehow correlated but not perfect, especially when comparing different types of models like comparing xgb and ARIMA), but unfortunately we didn't pay too much attentions to adjust/fix the cv mechanism design to make it more consistent. We used the local cv to find improvements for models, and if improvements identified, submitted to the LB. For same type single model (e.g., xgb), local cv score improvements usually results in LB score improvement, but the improvement scale was not always the same (e.g., cv score improved by 0.0002 but LB score improved by 0.0001, or vice versa, etc.). The inconsistency in score improvement is much larger especially when comparing different type of models like xgb and ARIMA. For example, ARIMA worked very well (as a diverse component of our ensemble) on the first couple of days on LB (i.e., public LB) but performed much worse for longer future period on the private LB. Unfortunately, when inconsistency occurred, we trusted a bit more on the public LB rather than our CV, since we thought that the public LB period is closer to the private LB period than our held-out cv period.Thus we decided to assigned a bit larger ensemble weights for the ARIMA models in our final submission, and we were bitten by that quite a bit on final private LB (dropped from #1 on public LB to #7 on private LB). Then in the past two years, I more and more rely on a comprehensive CV mechanism when I want to play a competition seriously, and when the CV_LB score is inconsistent, I paid more attentions /spent much more efforts to design a more sophisticated CV mechanism to achieve the consistency. That's all for now.\n\nHope it helps.", "270613": "Thanks!!! You helped me a lots!!!", "271287": "I'm pretty new in these contest but thought this might help. I compared the scores of my models between the public and private leaderboards and I noticed that the submissions where I tried to do something with the promotions data we're the ones were the score improved on the private LB from the public. For example, to fill the missing values in the training set... I determined the average sales lift from promotion by store/item (where we had promo data)... And then I filled na values where the sales of a store/item we're missing and the unit sales was x% above average for that pair. My local CV process was basically nonexistant, but I did notice that when I tried these sorts of things with the promo data my private LB scores were > public LB scores by not trivial amounts.", "275233": "Note that, in order to avoiding possible future target information leak, such statistical features were generated independent for each week of the training data, validation data, and test data.\nHello Shize Su.Does What does it mean? Does it mean separate training data in one week and generate feature?Expect your reply.Thanks", "275243": "Hi zhixing,\n\nAbout generating the statistical features (i.e., 'target-encoding' type features for category variables) , the key is to be careful to avoid utilizing (explicitly or implicitly) any future target information when generating these features. \n\nBy generating the statistical features for each week independently, we could completely avoid such future target information leak since at that time all the past sales data definitely not including any future sales data information, and the derived statistical features based on these past sales data of course is impossible to contain leak info from future targets.\n\nTo be more specific, the generation of such statistical features should be put inside a function similar to \"def prepare_dataset(t2017, is_train=True):\" in the lgb startet script https://www.kaggle.com/tunguz/lgbm-one-step-ahead-c8de0f, i.e., it should be generated inside the function for generating the features X_l and y_l for each week, rather than after operation \"X_train = pd.concat(X_l, axis=0)\". If you generating the statistical features after operation \"X_train = pd.concat(X_l, axis=0)\", then features for earlier weeks will implicitly contain future target leak information from later weeks. For example, the past day1 sales for week 2017/07/19 might be future information for the previous week 2017/07/12, and if generating such 'target-encoding' type statistical features for week 2017/07/12 after the operation \"X_train = pd.concat(X_l, axis=0)\", then it contains the future information leak (e.g., past day1 sales for week 2017/07/19, etc.). And such kind of future info leak is sth we need to avoid to make such 'target-encoding' type features works on the LB period, where future info leak is not possible.\n\nHope it helps.\n\nBest regards,\n\nShize", "275582": "Shize Su hi. I am back again. Thanks for you valuable answers. I found lots of shared solutions in this competition using similar cross validation settings. And It seem didn't concern about the holiday effects(such as the effect of before or after or in holiday?). \n\nIn the rossmann competition, [count features][1] maybe a good choice, but how to using these features in this cv settings? or count features should using in another cv settings?\n\nBesides, when I training the datasets with the same cv settings (there is no data leakages) , and look into the train log, I found that in some models the validation error is smaller than training error? Is this normal? or over-fitting? Maybe just the validation distribution is different from training data distribution?\n\n\n [1]: http://blog.kaggle.com/2016/02/03/rossmann-store-sales-winners-interview-2nd-place-nima-shahbazi/", "275599": "feng:\n\n1) About holiday, one key observation is that the LB period almost doesn't include any holiday (except a local holiday), and thus the benefits of considering holiday effect might only be marginal even if appropriately addressed. Not to mention that it is very tricky and time consuming to appropriately address the holiday effects to achieve such marginal improvement. This is why most competitors (including me) did not spend much time on dealing with the holiday effects. If the LB period includes many more National holidays, then it would be a totally different story.\n\n2) About 'count features', I didn't see any clear explanations in your provided link, so i am not sure what specific features you were talking about. \n\n3) The variance across the time periods is a bit large, thus it is possible (but not very likely) that validation loss is smaller than the training loss, especially when the model is under-fitting to the train set (for example, at the first couple of rounds of my lgb, in quite a couple of times I also observed smaller validation loss than training loss, but by the time early-stopping occurred, the training loss is usually much smaller than validation loss). So my suggestions would be : 1) double check the code to see whether there are any subtle mistakes/bugs; 2) make sure the model is sufficiently trained and not under-fitting to the train set; 3) if everything seems fine, then maybe it is ok-as I said, it is possible (but not very likely) that validation loss is smaller than the training loss for this dataset due to the variance across the time periods.\n\nHope this helps.", "275765": "Shize Su, hi. It's very helpful!\n\nIf the LB period include the holiday, what's the general methods for this cv settings? Any suggestion?\n\nAbout the count features in the links, nimashahbazi create the features like below:\n![count features][1]\n\nIf the store is closed, the open variable is zero and the sales is zero too. Then he create a dummy variable called MyRowHoliday to capture the information for these zero sales. This dummy variable assigned positive integers before and after the consecutive close and “-1” on other days.\n\nIn this way, maybe it capture the holiday effects(such as the effect of before or after or in holiday)?\n\n [1]: http://5047-presscdn.pagely.netdna-cdn.com/wp-content/uploads/2016/02/rossmann2_2-e1454538642298.png", "275768": "feng:\n\n1) If the LB period includes more major holidays, a general guideline is to design the CV data set to be as similar to LB data set (e.g., similar period but in 2016; or period close to LB period but also have similar amount of major holidays). But obviously there is not a simple clear answer about the best cv setting in this case, it might require significant amount of time and efforts (more difficult than the case here) to find a CV setting for consistent CV_LB score trend. You can only figured it out by various of experiments, following the key guideline that CV set should be as similar to LB set as possible.\n\n2) About the \"count features\" you mentioned above in Nima's post, they are not some common \"count features\" we usually use, and in my opinion those features are somewhat specific to the Rossmann data set. (For the term \"count features\", we (or at least me) usually mean sth like frequency of some category variables, etc. That's what made me confused when you used the term \"count feature\".) Finally, such type of features are probably much less helpful (though I didn't test it) for this competition since there are no that many \"close-open\" switching for stores in this data set (compared with rossmann competition data set), plus that there is no way to know whether a store is closed or not on a date of test data period. But maybe using such so called \"count features\" on the \"promotion\" variable might be somewhat helpful, though needs to be very carefully dealt with due to the large asymmetry in distribution of \"promotion\" variable in train and test data.\n\nHope this helps.", "275790": "Thanks for clarify my doubts. You push me more interest in the competition, there are a lots of modeling skills to learn from you and some people like you. Thanks for your generously!", "276648": "Thanks your explanation.Your solution and analyse help me learn a lot in this competition.Hope you can stick to it.", "285317": "Thank you for your in-depth post...", "574650": "Thanks for sharing." }, "source": "meta" }