First of all, congratulations to the winners. Thanks to @Ceshine Lee, @sjv and others whose codes, kernels and discussions gave me many inspirations. This was my first time to use deep learning and I really learn a lot from previous shared kernels and codes from kagglers.
\n\nModel Overview
\n\nI build 3 models: a lightGBM, a CNN+DNN and a seq2seq RNN model. Final model was a weighted average of these models (where each model is stabilized by training multiple times with different random seeds then take the average), with special dealing with promotion, which I will discuss later. Each model separately can stay in top 1% in the private leaderboard.
\n\nLGBM: It is an upgraded version of the public kernels. More features, data and periods were fed to the model.
\n\nCNN+DNN: This is a traditional NN model, where the CNN part is a dilated causal convolution inspired by WaveNet, and the DNN part is 2 FC layers connected to raw sales sequences. Then the inputs are concatenated together with categorical embeddings and future promotions, and directly output to 16 future days of predictions.
\n\nRNN: This is a seq2seq model with a similar architecture of @Arthur Suilin's solution for the web traffic prediction. Encoder and decoder are both GRUs. The hidden states of the encoder are passed to the decoder through an FC layer connector. This is useful to improve the accuracy significantly.
\n\nFeature Engineering
\n\nFor LGB, for each time periods the mean sales, count of promotions and count of zeros are included. These features are calculated with different ways of splits of the time periods, e.g. with/without promotion, each weekdays, item/store group stat, etc. Categorical features are included with label encoding.
\n\nFor NN, item mean and year-ago/quarter-ago sales are fed as sequences input. Categorical features and time features (weekday, day of month) are fed as embeddings
\n\nTraining and Validation
\n\nFor training and validation, only store-item combinations that are appeared in the test data and have at least one record during 2017 in the training data are included. Validation period is 2017.7.26~2017.8.10 throughout the competition, and training periods are collected randomly during 2017.1.1~2017.7.5.
\n\nFor LGB the number of range is 18, total # of data is ~3M. For NN the data is randomly generated batch by batch. Batch size is 1500 and steps per epoch is 1500 so total # of data trained per epoch is ~2M.
\n\nonPromotion
\n\nAs you guys all noticed, there are some problems with the promotion feature here. Besides the missing promotion information for zero sales in training data, I found that the promotion info in 8/16 is quite abnormal and unreliable (e.g. there are ~6000 unseen store-item combo on promotion on 8/16). So I don't think it reliable to use PB to infer the distribution of promotion in test set.
\n\nThe sales ratio of promo/non-promo items in training set is ~2:1 if I fill missing values with 0. I try to infer this ratio in test data. We know that the proportion of 1s in training promo data is underestimated because I fill all missing values with 0. the proportion of 1s in training set is ~6.3%, while in test set is ~6.9% without 8/16. The proportion of missing value is ~40% in training data. If I assume the true distribution is consistent, the proportion of items that are missing in the training data but actually has promotion is (0.069-0.063)/0.4 = ~1.5%. And as the missing values are all 0s, the true ratio would be (0.063*2 + (0.069-0.063)*0)/0.069 = ~1.83. So I guess our training model is around 10% over-estimating the items on promotion.
\n\nThen I simple re-train all the models, but without the promotion information. These predictions will of course have a lower sales ratio between promo/non-promo items. Then I average the no-promo predictions with the original predictions with promotion info with weights so that the sales ratio of the final predictions approaches 1.83:1 as I inferred.
\n\nThis approach is kind of tricky and based on assumptions I cannot validated, but that's the only way I figure out to deal with the promotion bias. However, it seems like it's not useful to the PB. Ironically, the model without any special dealing with promotion bias gives me .510 on PB. I still have no idea why.
\n\nI have shared my codes on github https://github.com/LenzDu/Kaggle-Competition-Favorita, but it is still quite messy there. I will organize it when I have time.
", "messages": [ { "id": "269107", "postDate": "01/16/2018 08:01:05", "content": "First of all, congratulations to the winners. Thanks to @Ceshine Lee, @sjv and others whose codes, kernels and discussions gave me many inspirations. This was my first time to use deep learning and I really learn a lot from previous shared kernels and codes from kagglers.
\n\nModel Overview
\n\nI build 3 models: a lightGBM, a CNN+DNN and a seq2seq RNN model. Final model was a weighted average of these models (where each model is stabilized by training multiple times with different random seeds then take the average), with special dealing with promotion, which I will discuss later. Each model separately can stay in top 1% in the private leaderboard.
\n\nLGBM: It is an upgraded version of the public kernels. More features, data and periods were fed to the model.
\n\nCNN+DNN: This is a traditional NN model, where the CNN part is a dilated causal convolution inspired by WaveNet, and the DNN part is 2 FC layers connected to raw sales sequences. Then the inputs are concatenated together with categorical embeddings and future promotions, and directly output to 16 future days of predictions.
\n\nRNN: This is a seq2seq model with a similar architecture of @Arthur Suilin's solution for the web traffic prediction. Encoder and decoder are both GRUs. The hidden states of the encoder are passed to the decoder through an FC layer connector. This is useful to improve the accuracy significantly.
\n\nFeature Engineering
\n\nFor LGB, for each time periods the mean sales, count of promotions and count of zeros are included. These features are calculated with different ways of splits of the time periods, e.g. with/without promotion, each weekdays, item/store group stat, etc. Categorical features are included with label encoding.
\n\nFor NN, item mean and year-ago/quarter-ago sales are fed as sequences input. Categorical features and time features (weekday, day of month) are fed as embeddings
\n\nTraining and Validation
\n\nFor training and validation, only store-item combinations that are appeared in the test data and have at least one record during 2017 in the training data are included. Validation period is 2017.7.26~2017.8.10 throughout the competition, and training periods are collected randomly during 2017.1.1~2017.7.5.
\n\nFor LGB the number of range is 18, total # of data is ~3M. For NN the data is randomly generated batch by batch. Batch size is 1500 and steps per epoch is 1500 so total # of data trained per epoch is ~2M.
\n\nonPromotion
\n\nAs you guys all noticed, there are some problems with the promotion feature here. Besides the missing promotion information for zero sales in training data, I found that the promotion info in 8/16 is quite abnormal and unreliable (e.g. there are ~6000 unseen store-item combo on promotion on 8/16). So I don't think it reliable to use PB to infer the distribution of promotion in test set.
\n\nThe sales ratio of promo/non-promo items in training set is ~2:1 if I fill missing values with 0. I try to infer this ratio in test data. We know that the proportion of 1s in training promo data is underestimated because I fill all missing values with 0. the proportion of 1s in training set is ~6.3%, while in test set is ~6.9% without 8/16. The proportion of missing value is ~40% in training data. If I assume the true distribution is consistent, the proportion of items that are missing in the training data but actually has promotion is (0.069-0.063)/0.4 = ~1.5%. And as the missing values are all 0s, the true ratio would be (0.063*2 + (0.069-0.063)*0)/0.069 = ~1.83. So I guess our training model is around 10% over-estimating the items on promotion.
\n\nThen I simple re-train all the models, but without the promotion information. These predictions will of course have a lower sales ratio between promo/non-promo items. Then I average the no-promo predictions with the original predictions with promotion info with weights so that the sales ratio of the final predictions approaches 1.83:1 as I inferred.
\n\nThis approach is kind of tricky and based on assumptions I cannot validated, but that's the only way I figure out to deal with the promotion bias. However, it seems like it's not useful to the PB. Ironically, the model without any special dealing with promotion bias gives me .510 on PB. I still have no idea why.
\n\nI have shared my codes on github https://github.com/LenzDu/Kaggle-Competition-Favorita, but it is still quite messy there. I will organize it when I have time.
", "rawMarkdown": "First of all, congratulations to the winners. Thanks to @Ceshine Lee, @sjv and others whose codes, kernels and discussions gave me many inspirations. This was my first time to use deep learning and I really learn a lot from previous shared kernels and codes from kagglers.\n\n**Model Overview**\n\nI build 3 models: a lightGBM, a CNN+DNN and a seq2seq RNN model. Final model was a weighted average of these models (where each model is stabilized by training multiple times with different random seeds then take the average), with special dealing with promotion, which I will discuss later. Each model separately can stay in top 1% in the private leaderboard.\n\n**LGBM:** It is an upgraded version of the public kernels. More features, data and periods were fed to the model.\n\n**CNN+DNN:** This is a traditional NN model, where the CNN part is a dilated causal convolution inspired by WaveNet, and the DNN part is 2 FC layers connected to raw sales sequences. Then the inputs are concatenated together with categorical embeddings and future promotions, and directly output to 16 future days of predictions.\n\n**RNN:** This is a seq2seq model with a similar architecture of @Arthur Suilin's solution for the web traffic prediction. Encoder and decoder are both GRUs. The hidden states of the encoder are passed to the decoder through an FC layer connector. This is useful to improve the accuracy significantly.\n\n**Feature Engineering**\n\nFor LGB, for each time periods the mean sales, count of promotions and count of zeros are included. These features are calculated with different ways of splits of the time periods, e.g. with/without promotion, each weekdays, item/store group stat, etc. Categorical features are included with label encoding.\n\nFor NN, item mean and year-ago/quarter-ago sales are fed as sequences input. Categorical features and time features (weekday, day of month) are fed as embeddings\n\n**Training and Validation**\n\nFor training and validation, only store-item combinations that are appeared in the test data and have at least one record during 2017 in the training data are included. Validation period is 2017.7.26~2017.8.10 throughout the competition, and training periods are collected randomly during 2017.1.1~2017.7.5.\n\nFor LGB the number of range is 18, total # of data is ~3M. For NN the data is randomly generated batch by batch. Batch size is 1500 and steps per epoch is 1500 so total # of data trained per epoch is ~2M.\n\n**onPromotion**\n\nAs you guys all noticed, there are some problems with the promotion feature here. Besides the missing promotion information for zero sales in training data, I found that the promotion info in 8/16 is quite abnormal and unreliable (e.g. there are ~6000 unseen store-item combo on promotion on 8/16). So I don't think it reliable to use PB to infer the distribution of promotion in test set.\n\nThe sales ratio of promo/non-promo items in training set is ~2:1 if I fill missing values with 0. I try to infer this ratio in test data. We know that the proportion of 1s in training promo data is underestimated because I fill all missing values with 0. the proportion of 1s in training set is ~6.3%, while in test set is ~6.9% without 8/16. The proportion of missing value is ~40% in training data. If I assume the true distribution is consistent, the proportion of items that are missing in the training data but actually has promotion is (0.069-0.063)/0.4 = ~1.5%. And as the missing values are all 0s, the true ratio would be (0.063*2 + (0.069-0.063)*0)/0.069 = ~1.83. So I guess our training model is around 10% over-estimating the items on promotion.\n\nThen I simple re-train all the models, but without the promotion information. These predictions will of course have a lower sales ratio between promo/non-promo items. Then I average the no-promo predictions with the original predictions with promotion info with weights so that the sales ratio of the final predictions approaches 1.83:1 as I inferred.\n\nThis approach is kind of tricky and based on assumptions I cannot validated, but that's the only way I figure out to deal with the promotion bias. However, it seems like it's not useful to the PB. Ironically, the model without any special dealing with promotion bias gives me .510 on PB. I still have no idea why.\n\nI have shared my codes on github https://github.com/LenzDu/Kaggle-Competition-Favorita, but it is still quite messy there. I will organize it when I have time.", "votes": null }, { "id": "269145", "postDate": "01/16/2018 10:05:26", "content": "Excellent, It seems both Arthur and SJV's solution are quite generalize to this kind of problems.
", "rawMarkdown": "Excellent, It seems both Arthur and SJV's solution are quite generalize to this kind of problems.", "votes": null }, { "id": "269290", "postDate": "01/16/2018 15:45:28", "content": "Many thanks for sharing your approach and congrats with gold medal!
", "rawMarkdown": "Many thanks for sharing your approach and congrats with gold medal!", "votes": null }, { "id": "269300", "postDate": "01/16/2018 15:56:08", "content": "Congrats! Very detailed write-up for the solution!
", "rawMarkdown": "Congrats! Very detailed write-up for the solution!", "votes": null }, { "id": "270515", "postDate": "01/18/2018 11:46:32", "content": "Thanks for sharing and congratulations.
", "rawMarkdown": "Thanks for sharing and congratulations.", "votes": null }, { "id": "550725", "postDate": "06/12/2019 01:17:58", "content": "Thanks for sharing and congratulations.
", "rawMarkdown": "Thanks for sharing and congratulations.", "votes": null }, { "id": "563200", "postDate": "06/28/2019 02:50:13", "content": "Thanks for sharing, I see there is some problems in your code , like lgb part(median is also mean here):\n\nX['mean_{}_2017'.format(i)] = get_timespan(df_2017, t2017, i, i).mean(axis=1).values\nX['median_{}_2017'.format(i)] = get_timespan(df_2017, t2017, i, i).mean(axis=1).values\n
Thx for sharing! learn a lot from you.
", "rawMarkdown": "Thx for sharing! learn a lot from you.", "votes": null }, { "id": "1137888", "postDate": "01/04/2021 08:55:22", "content": "Thanks for posting this! This has given me a lot of insight on regression problems with a high amount of categories (products, classes, families, stores).
", "rawMarkdown": "Thanks for posting this! This has given me a lot of insight on regression problems with a high amount of categories (products, classes, families, stores).", "votes": null }, { "id": "1770180", "postDate": "04/28/2022 04:18:54", "content": "good job！
", "rawMarkdown": "*good job！*", "votes": null } ], "comments": [ { "id": 1137888, "author_name": "maximdm", "author_url": "", "post_date": "01/04/2021 08:55:22", "content": "Thanks for posting this! This has given me a lot of insight on regression problems with a high amount of categories (products, classes, families, stores).
", "votes": null, "replies": [] }, { "id": 1770180, "author_name": "djustin", "author_url": "", "post_date": "04/28/2022 04:18:54", "content": "good job！
", "votes": null, "replies": [] }, { "id": 269145, "author_name": "nickycan", "author_url": "", "post_date": "01/16/2018 10:05:26", "content": "Excellent, It seems both Arthur and SJV's solution are quite generalize to this kind of problems.
", "votes": null, "replies": [] }, { "id": 269290, "author_name": "andymasl", "author_url": "", "post_date": "01/16/2018 15:45:28", "content": "Many thanks for sharing your approach and congrats with gold medal!
", "votes": null, "replies": [] }, { "id": 269300, "author_name": "cczaixian", "author_url": "", "post_date": "01/16/2018 15:56:08", "content": "Congrats! Very detailed write-up for the solution!
", "votes": null, "replies": [] }, { "id": 270515, "author_name": "viczyf", "author_url": "", "post_date": "01/18/2018 11:46:32", "content": "Thanks for sharing and congratulations.
", "votes": null, "replies": [] }, { "id": 550725, "author_name": "longyin2", "author_url": "", "post_date": "06/12/2019 01:17:58", "content": "Thanks for sharing and congratulations.
", "votes": null, "replies": [] }, { "id": 563200, "author_name": "longyin2", "author_url": "", "post_date": "06/28/2019 02:50:13", "content": "Thanks for sharing, I see there is some problems in your code , like lgb part(median is also mean here):\n\nX['mean_{}_2017'.format(i)] = get_timespan(df_2017, t2017, i, i).mean(axis=1).values\nX['median_{}_2017'.format(i)] = get_timespan(df_2017, t2017, i, i).mean(axis=1).values\n
Thx for sharing! learn a lot from you.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "269107": "First of all, congratulations to the winners. Thanks to @Ceshine Lee, @sjv and others whose codes, kernels and discussions gave me many inspirations. This was my first time to use deep learning and I really learn a lot from previous shared kernels and codes from kagglers.\n\n**Model Overview**\n\nI build 3 models: a lightGBM, a CNN+DNN and a seq2seq RNN model. Final model was a weighted average of these models (where each model is stabilized by training multiple times with different random seeds then take the average), with special dealing with promotion, which I will discuss later. Each model separately can stay in top 1% in the private leaderboard.\n\n**LGBM:** It is an upgraded version of the public kernels. More features, data and periods were fed to the model.\n\n**CNN+DNN:** This is a traditional NN model, where the CNN part is a dilated causal convolution inspired by WaveNet, and the DNN part is 2 FC layers connected to raw sales sequences. Then the inputs are concatenated together with categorical embeddings and future promotions, and directly output to 16 future days of predictions.\n\n**RNN:** This is a seq2seq model with a similar architecture of @Arthur Suilin's solution for the web traffic prediction. Encoder and decoder are both GRUs. The hidden states of the encoder are passed to the decoder through an FC layer connector. This is useful to improve the accuracy significantly.\n\n**Feature Engineering**\n\nFor LGB, for each time periods the mean sales, count of promotions and count of zeros are included. These features are calculated with different ways of splits of the time periods, e.g. with/without promotion, each weekdays, item/store group stat, etc. Categorical features are included with label encoding.\n\nFor NN, item mean and year-ago/quarter-ago sales are fed as sequences input. Categorical features and time features (weekday, day of month) are fed as embeddings\n\n**Training and Validation**\n\nFor training and validation, only store-item combinations that are appeared in the test data and have at least one record during 2017 in the training data are included. Validation period is 2017.7.26~2017.8.10 throughout the competition, and training periods are collected randomly during 2017.1.1~2017.7.5.\n\nFor LGB the number of range is 18, total # of data is ~3M. For NN the data is randomly generated batch by batch. Batch size is 1500 and steps per epoch is 1500 so total # of data trained per epoch is ~2M.\n\n**onPromotion**\n\nAs you guys all noticed, there are some problems with the promotion feature here. Besides the missing promotion information for zero sales in training data, I found that the promotion info in 8/16 is quite abnormal and unreliable (e.g. there are ~6000 unseen store-item combo on promotion on 8/16). So I don't think it reliable to use PB to infer the distribution of promotion in test set.\n\nThe sales ratio of promo/non-promo items in training set is ~2:1 if I fill missing values with 0. I try to infer this ratio in test data. We know that the proportion of 1s in training promo data is underestimated because I fill all missing values with 0. the proportion of 1s in training set is ~6.3%, while in test set is ~6.9% without 8/16. The proportion of missing value is ~40% in training data. If I assume the true distribution is consistent, the proportion of items that are missing in the training data but actually has promotion is (0.069-0.063)/0.4 = ~1.5%. And as the missing values are all 0s, the true ratio would be (0.063*2 + (0.069-0.063)*0)/0.069 = ~1.83. So I guess our training model is around 10% over-estimating the items on promotion.\n\nThen I simple re-train all the models, but without the promotion information. These predictions will of course have a lower sales ratio between promo/non-promo items. Then I average the no-promo predictions with the original predictions with promotion info with weights so that the sales ratio of the final predictions approaches 1.83:1 as I inferred.\n\nThis approach is kind of tricky and based on assumptions I cannot validated, but that's the only way I figure out to deal with the promotion bias. However, it seems like it's not useful to the PB. Ironically, the model without any special dealing with promotion bias gives me .510 on PB. I still have no idea why.\n\nI have shared my codes on github https://github.com/LenzDu/Kaggle-Competition-Favorita, but it is still quite messy there. I will organize it when I have time.", "269145": "Excellent, It seems both Arthur and SJV's solution are quite generalize to this kind of problems.", "269290": "Many thanks for sharing your approach and congrats with gold medal!", "269300": "Congrats! Very detailed write-up for the solution!", "270515": "Thanks for sharing and congratulations.", "550725": "Thanks for sharing and congratulations.", "563200": "Thanks for sharing, I see there is some problems in your code , like lgb part(median is also mean here):\n``````\nX['mean_{}_2017'.format(i)] = get_timespan(df_2017, t2017, i, i).mean(axis=1).values\nX['median_{}_2017'.format(i)] = get_timespan(df_2017, t2017, i, i).mean(axis=1).values\n``````", "656462": "Thx for sharing! learn a lot from you.", "1137888": "Thanks for posting this! This has given me a lot of insight on regression problems with a high amount of categories (products, classes, families, stores).", "1770180": "*good job！*" }, "source": "meta" }