First off, congrats to the winners and thanks to all those who contributed to the forum discussions. I'm definitely disappointed with the result, but I might as well share my solution anyways. I'll only discuss my contribution to my team's solution, which is a single model which scores .499 on the public leaderboard.
\n\nWe framed forecasting as a sequence to sequence modeling problem, where the encoder reads in a portion of a time series and the decoder sequentially emits estimates of the subsequent 16 values. The encoder and decoder do not share parameters, but they are both similarly parameterized as a stack of ~30 dilated, causal convolutions. Dilated convolutions allowed the receptive field to span the entirety of the 4+ year train period, while still maintaining a small number of parameters (~1 million). To allow the decoder to use future onpromotion values, it was also augmented with a bidirectional LSTM which encoded information from prior and future onpromotion data. The model struggled to accurately predict zero, so we modified the architecture to additionally emit a bernoulli parameter at each timestep (corresponding to the probability that the output was zero) and the final predictions were adjusted accordingly. Weighted RMSE of the log-transformed data was directly minimized.
\n\nThe input to the network consisted of the raw time series values, embeddings of the categorical variables, and some manually engineered features. The manual features included lags, diffs, rolling statistics, date features, and conditioning time series (i.e. average sales for a given product/store/etc.). Promotion data required some special handling, which I'll describe later.
\n\n5% of the time series were held out, and the model was evaluated on random periods from the last 365 training days of these held out series. The purpose of this was to prevent the model from overfitting a validation set which was biased toward any sort of weekly or monthly trends.
\n\nAs discussed in the forums, if you naively impute missing onpromotion values with 0, the sales distribution conditioned on onpromotion values will differ in the train and test sets. I didn't find a way to remedy this, but thankfully my teammates were able to come up with a clever solution which improved my score by about .004. The missing values were imputed randomly: 1 with probability p and 0 otherwise. Determining p was a rather tedious task, since we had no data to model it and we had to resort to trial and error leaderboard submissions. We ended up setting p separately for each day, and it was computed as the mean onpromotion rate for each day, scaled by a learnable factor estimated by stochastic leaderboard descent (leaderboard probing...). All credit goes to my teammates for this, and they can probably explain it better than I can.
\n\nI normally share my source code, but since the core model is nearly identical to one I've already shared, I'll just refer you to that instead: https://github.com/sjvasquez/web-traffic-forecasting
\n\nFinally, I'll admit I'm mostly disappointed with Kaggle. I raised concerns about the unseen items in the test set and also about the onpromotion issue earlier in the competition. Ultimately, the competition was a lottery. For the sake of others, I hope Kaggle will begin to demonstrate a moderate level of care in data preparation and show that they understand that whenever they deviate from the assumption that train set is an unbiased sampling of the test distribution, they induce randomness which turns competitions into lotteries. This will be my last Kaggle competition, as my time will be better spent focusing on research from now on. Best of luck to everyone.
", "messages": [ { "id": 268938, "postDate": "2018-01-16T00:43:13.337Z", "content": "First off, congrats to the winners and thanks to all those who contributed to the forum discussions. I'm definitely disappointed with the result, but I might as well share my solution anyways. I'll only discuss my contribution to my team's solution, which is a single model which scores .499 on the public leaderboard.
\n\nWe framed forecasting as a sequence to sequence modeling problem, where the encoder reads in a portion of a time series and the decoder sequentially emits estimates of the subsequent 16 values. The encoder and decoder do not share parameters, but they are both similarly parameterized as a stack of ~30 dilated, causal convolutions. Dilated convolutions allowed the receptive field to span the entirety of the 4+ year train period, while still maintaining a small number of parameters (~1 million). To allow the decoder to use future onpromotion values, it was also augmented with a bidirectional LSTM which encoded information from prior and future onpromotion data. The model struggled to accurately predict zero, so we modified the architecture to additionally emit a bernoulli parameter at each timestep (corresponding to the probability that the output was zero) and the final predictions were adjusted accordingly. Weighted RMSE of the log-transformed data was directly minimized.
\n\nThe input to the network consisted of the raw time series values, embeddings of the categorical variables, and some manually engineered features. The manual features included lags, diffs, rolling statistics, date features, and conditioning time series (i.e. average sales for a given product/store/etc.). Promotion data required some special handling, which I'll describe later.
\n\n5% of the time series were held out, and the model was evaluated on random periods from the last 365 training days of these held out series. The purpose of this was to prevent the model from overfitting a validation set which was biased toward any sort of weekly or monthly trends.
\n\nAs discussed in the forums, if you naively impute missing onpromotion values with 0, the sales distribution conditioned on onpromotion values will differ in the train and test sets. I didn't find a way to remedy this, but thankfully my teammates were able to come up with a clever solution which improved my score by about .004. The missing values were imputed randomly: 1 with probability p and 0 otherwise. Determining p was a rather tedious task, since we had no data to model it and we had to resort to trial and error leaderboard submissions. We ended up setting p separately for each day, and it was computed as the mean onpromotion rate for each day, scaled by a learnable factor estimated by stochastic leaderboard descent (leaderboard probing...). All credit goes to my teammates for this, and they can probably explain it better than I can.
\n\nI normally share my source code, but since the core model is nearly identical to one I've already shared, I'll just refer you to that instead: https://github.com/sjvasquez/web-traffic-forecasting
\n\nFinally, I'll admit I'm mostly disappointed with Kaggle. I raised concerns about the unseen items in the test set and also about the onpromotion issue earlier in the competition. Ultimately, the competition was a lottery. For the sake of others, I hope Kaggle will begin to demonstrate a moderate level of care in data preparation and show that they understand that whenever they deviate from the assumption that train set is an unbiased sampling of the test distribution, they induce randomness which turns competitions into lotteries. This will be my last Kaggle competition, as my time will be better spent focusing on research from now on. Best of luck to everyone.
", "rawMarkdown": "First off, congrats to the winners and thanks to all those who contributed to the forum discussions. I'm definitely disappointed with the result, but I might as well share my solution anyways. I'll only discuss my contribution to my team's solution, which is a single model which scores .499 on the public leaderboard. \n\n### Model Architecture\nWe framed forecasting as a sequence to sequence modeling problem, where the encoder reads in a portion of a time series and the decoder sequentially emits estimates of the subsequent 16 values. The encoder and decoder do not share parameters, but they are both similarly parameterized as a stack of ~30 dilated, causal convolutions. Dilated convolutions allowed the receptive field to span the entirety of the 4+ year train period, while still maintaining a small number of parameters (~1 million). To allow the decoder to use future onpromotion values, it was also augmented with a bidirectional LSTM which encoded information from prior and future onpromotion data. The model struggled to accurately predict zero, so we modified the architecture to additionally emit a bernoulli parameter at each timestep (corresponding to the probability that the output was zero) and the final predictions were adjusted accordingly. Weighted RMSE of the log-transformed data was directly minimized.\n\n### Input Representation\nThe input to the network consisted of the raw time series values, embeddings of the categorical variables, and some manually engineered features. The manual features included lags, diffs, rolling statistics, date features, and conditioning time series (i.e. average sales for a given product/store/etc.). Promotion data required some special handling, which I'll describe later.\n\n### Validation\n5% of the time series were held out, and the model was evaluated on random periods from the last 365 training days of these held out series. The purpose of this was to prevent the model from overfitting a validation set which was biased toward any sort of weekly or monthly trends.\n\n### Promotion Data\nAs discussed in the forums, if you naively impute missing onpromotion values with 0, the sales distribution conditioned on onpromotion values will differ in the train and test sets. I didn't find a way to remedy this, but thankfully my teammates were able to come up with a clever solution which improved my score by about .004. The missing values were imputed randomly: 1 with probability p and 0 otherwise. Determining p was a rather tedious task, since we had no data to model it and we had to resort to trial and error leaderboard submissions. We ended up setting p separately for each day, and it was computed as the mean onpromotion rate for each day, scaled by a learnable factor estimated by stochastic leaderboard descent (leaderboard probing...). All credit goes to my teammates for this, and they can probably explain it better than I can.\n\n### Source Code\nI normally share my source code, but since the core model is nearly identical to one I've already shared, I'll just refer you to that instead: https://github.com/sjvasquez/web-traffic-forecasting\n\nFinally, I'll admit I'm mostly disappointed with Kaggle. I raised concerns about the unseen items in the test set and also about the onpromotion issue earlier in the competition. Ultimately, the competition was a lottery. For the sake of others, I hope Kaggle will begin to demonstrate a moderate level of care in data preparation and show that they understand that whenever they deviate from the assumption that train set is an unbiased sampling of the test distribution, they induce randomness which turns competitions into lotteries. This will be my last Kaggle competition, as my time will be better spent focusing on research from now on. Best of luck to everyone.", "votes": 112 }, { "id": 269095, "postDate": "2018-01-16T07:29:26.820Z", "content": "I agree about bad data preparation in this competition, but unfortunately it's not the first nor the second time it happens. Kaggle really need to do an octopus review of the datasets before launching competitions.
", "rawMarkdown": "I agree about bad data preparation in this competition, but unfortunately it's not the first nor the second time it happens. Kaggle really need to do an octopus review of the datasets before launching competitions.", "votes": 8 }, { "id": 269421, "postDate": "2018-01-16T20:09:33.783Z", "content": "I agree with CPMP and would add that real-world data starts out very dirty and wrong! I spend most of my time detecting and throwing out mis-labeled data. One of my side research areas is to use unsupervised learning to detect such data. Knn works best so far. ( that is how bad my world is.) I like Kaggle completions as data is so clean. kaggle to me is a game and data leaks and such just add spice. However, SJV I understand your outrage. Also, thank for your WTF code. Before this Categoical embedding was SOA.
", "rawMarkdown": "I agree with CPMP and would add that real-world data starts out very dirty and wrong! I spend most of my time detecting and throwing out mis-labeled data. One of my side research areas is to use unsupervised learning to detect such data. Knn works best so far. ( that is how bad my world is.) I like Kaggle completions as data is so clean. kaggle to me is a game and data leaks and such just add spice. However, SJV I understand your outrage. Also, thank for your WTF code. Before this Categoical embedding was SOA.", "votes": 5 }, { "id": 269042, "postDate": "2018-01-16T05:10:40.917Z", "content": "@sjv, so sad that you are leaving Kaggle. I am always impressed by your solutions that apply DL to time series problems. It gives me a lot of inspiration. All the best to your research.
", "rawMarkdown": "@sjv, so sad that you are leaving Kaggle. I am always impressed by your solutions that apply DL to time series problems. It gives me a lot of inspiration. All the best to your research.", "votes": 6 }, { "id": 269425, "postDate": "2018-01-16T20:20:05.503Z", "content": "I have to add I agree with cpmp on his first post and don’t necessarily agree in the sense that some academics don,t handle very well the dirty real world and some handle it very well.
", "rawMarkdown": "I have to add I agree with cpmp on his first post and don’t necessarily agree in the sense that some academics don,t handle very well the dirty real world and some handle it very well.", "votes": 3 }, { "id": 269311, "postDate": "2018-01-16T16:09:50.730Z", "content": "Thank you for this sharing, and congratulations for your results
", "rawMarkdown": "Thank you for this sharing, and congratulations for your results \n\n", "votes": 3 }, { "id": 269049, "postDate": "2018-01-16T05:25:00.537Z", "content": "Thank you so much and good luck with you. I've learned so much from your previous codes.
", "rawMarkdown": "Thank you so much and good luck with you. I've learned so much from your previous codes.", "votes": 3 }, { "id": 269016, "postDate": "2018-01-16T04:06:25.053Z", "content": "Among many other things, I am most impressed with your clean code. My code always ends up a mess after doing n different kind of experiment. It would be nice if you can share some tips on your workflow, version control and keeping track of experiences.
\n\nSad to see you go, all the best with your research.
", "rawMarkdown": "Among many other things, I am most impressed with your clean code. My code always ends up a mess after doing n different kind of experiment. It would be nice if you can share some tips on your workflow, version control and keeping track of experiences.\n\nSad to see you go, all the best with your research. ", "votes": 3 }, { "id": 269165, "postDate": "2018-01-16T11:19:51.043Z", "content": "Your works on Time Series was referred to me by another Gold medalist here :-D
\n\nAnd congrats on the GM title.
\n\nTBH, on the unseen data, I think it's more a case of Favorita DS team being lazy than Kaggle's fault.
\n\nThey dumped into the Test set all the potential combinations of 'Item_nbr * Store_nbr', while in Real-Life there is no such thing: retailers worldwide use Store Types, Clusters and Assortment Classes to control which 'Item_nbr' will be present in which 'Store_nbr', and not.
\n\nShelf space is a store's most precious asset and is monetized to major Brands (ie. they often pay for guaranteed space in number of \"facings\", including eye or hand level for max exposure, then send sales reps to control it in stores and take pictures if not enforced and ask for refunds=messy).
\n\nEx: a size 5 store (max size +10,000sqm) has room for 10,000 different SKUs (Stock Keeping Units) on shelves, like 10 types of Coca-Cola formats (from 4*33cl cans to 6*2L PET bottles),
\n\nwhile a size 1 (min size 2000sqm) has room for 2,000 SKUS only, like 4 types of Coke (nothing larger than a single 1.5L PET bottle).
\n\nIt's carefully decided at HQ level by Marketing/Merchandising/Purchasing and it triggers down the chain via centralized IT to the store floors:
\n\nEven if that store stole a truck of Coke 6*2L Bottles, it could not sell it via the company's centralized IT system.
\n\nThat 'Item_nbr * Store_nbr', plus probably 1.5M other rows in the Test set could never happen, and Favorita's people at HQ know it very well.
\n\nIt's Retail 101 ^!^
", "rawMarkdown": "Your works on Time Series was referred to me by another Gold medalist here :-D\n\nAnd congrats on the GM title.\n\nTBH, on the unseen data, I think it's more a case of Favorita DS team being lazy than Kaggle's fault.\n\nThey dumped into the Test set all the potential combinations of 'Item_nbr * Store_nbr', while in Real-Life there is no such thing: retailers worldwide use Store Types, Clusters and Assortment Classes to control which 'Item_nbr' will be present in which 'Store_nbr', and not.\n\nShelf space is a store's most precious asset and is monetized to major Brands (ie. they often pay for guaranteed space in number of \"facings\", including eye or hand level for max exposure, then send sales reps to control it in stores and take pictures if not enforced and ask for refunds=messy).\n\nEx: a size 5 store (max size +10,000sqm) has room for 10,000 different SKUs (Stock Keeping Units) on shelves, like 10 types of Coca-Cola formats (from 4*33cl cans to 6*2L PET bottles),\n\nwhile a size 1 (min size 2000sqm) has room for 2,000 SKUS only, like 4 types of Coke (nothing larger than a single 1.5L PET bottle).\n\nIt's carefully decided at HQ level by Marketing/Merchandising/Purchasing and it triggers down the chain via centralized IT to the store floors:\n\n* a size 1 store can NOT order Coke in 6*2L PET Bottles\n- it's NOT listed in his purchasing software client\n- its logistic platform/warehouse can NOT deliver it\n- its employees can NOT print its shelves' labels, btw there's no room for it as shelves facings are full already\n- its cashier checkout can NOT scan it.\n\nEven if that store stole a truck of Coke 6*2L Bottles, it could not sell it via the company's centralized IT system.\n\nThat 'Item_nbr * Store_nbr', plus probably 1.5M other rows in the Test set could never happen, and Favorita's people at HQ know it very well.\n\nIt's Retail 101 ^!^\n\n", "votes": 4 }, { "id": 269535, "postDate": "2018-01-17T01:06:17.290Z", "content": "Sjv May have meant that he (they) could not build a predictive valid set for private but only for public .however I am not sure because I think they gave you private LB so I assume I you could build a linear predictive validation set by probing the private LB. a I did not try myself.
", "rawMarkdown": "Sjv May have meant that he (they) could not build a predictive valid set for private but only for public .however I am not sure because I think they gave you private LB so I assume I you could build a linear predictive validation set by probing the private LB. a I did not try myself. ", "votes": 1 }, { "id": 269041, "postDate": "2018-01-16T05:05:28.277Z", "content": "Hope that Kaggle will fix this issue
", "rawMarkdown": "Hope that Kaggle will fix this issue", "votes": 1 }, { "id": 269025, "postDate": "2018-01-16T04:36:06.973Z", "content": "\n\n\nAs discussed in the forums, if you naively impute missing onpromotion values with 0, the sales distribution conditioned on onpromotion values will differ in the train and test sets. I didn't find a way to remedy this, but thankfully my teammates were able to come up with a clever solution which improved my score by about .004. The missing values were imputed randomly: 1 with probability p and 0 otherwise. Determining p was a rather tedious task, since we had no data to model it and we had to resort to trial and error leaderboard submissions. We ended up setting p separately for each day, and it was computed as the mean onpromotion rate for each day, scaled by a learnable factor estimated by stochastic leaderboard descent (leaderboard probing...). All credit goes to my teammates for this, and they can probably explain it better than I can.
\n
So smart... Does this way also help you improve the private LB score? I'm always impressive with your work and all best with your future research.
", "rawMarkdown": "> \nAs discussed in the forums, if you naively impute missing onpromotion values with 0, the sales distribution conditioned on onpromotion values will differ in the train and test sets. I didn't find a way to remedy this, but thankfully my teammates were able to come up with a clever solution which improved my score by about .004. The missing values were imputed randomly: 1 with probability p and 0 otherwise. Determining p was a rather tedious task, since we had no data to model it and we had to resort to trial and error leaderboard submissions. We ended up setting p separately for each day, and it was computed as the mean onpromotion rate for each day, scaled by a learnable factor estimated by stochastic leaderboard descent (leaderboard probing...). All credit goes to my teammates for this, and they can probably explain it better than I can.\n> \n\nSo smart... Does this way also help you improve the private LB score? I'm always impressive with your work and all best with your future research.", "votes": 1, "replies": [ { "id": 269028, "postDate": "2018-01-16T04:42:52.427Z", "content": "For some models it helped and for some it did not. Our private leaderboard scores varied drastically between runs, even when the public leaderboard score stayed the same. While in the end it did not work out very well, I'm confident that the approach of using the leaderboard as validation was the optimal strategy from a statistical viewpoint. We have no choice but to fit our models to the public leaderboard when the organizers don't provide any training data that resembles the test set...
", "rawMarkdown": "For some models it helped and for some it did not. Our private leaderboard scores varied drastically between runs, even when the public leaderboard score stayed the same. While in the end it did not work out very well, I'm confident that the approach of using the leaderboard as validation was the optimal strategy from a statistical viewpoint. We have no choice but to fit our models to the public leaderboard when the organizers don't provide any training data that resembles the test set...", "votes": 3 } ] }, { "id": 269000, "postDate": "2018-01-16T03:20:12.747Z", "content": "Thanks for sharing. Really clever approach with the promotion data.
", "rawMarkdown": "Thanks for sharing. Really clever approach with the promotion data.", "votes": 1 }, { "id": 268988, "postDate": "2018-01-16T02:58:02.500Z", "content": "Thanks for sharing your excellent solution once again and wish you the best of luck in your future research/study/work and definitely we will miss you !\nAnd one problem I encountered during using your WaveNet framework is that my model can not even converge( I brutely force the model to consume the raw sequence with 210k*1684 and combine it with some embedded features), I guess the problem might be 2/3 of the series are zero and the distribution of the data is not well handled(my dilated rate might not big enough), could you please give me some more advice on how to tackle with your models since handling problems with DL is really exciting.
", "rawMarkdown": "Thanks for sharing your excellent solution once again and wish you the best of luck in your future research/study/work and definitely we will miss you !\nAnd one problem I encountered during using your WaveNet framework is that my model can not even converge( I brutely force the model to consume the raw sequence with 210k*1684 and combine it with some embedded features), I guess the problem might be 2/3 of the series are zero and the distribution of the data is not well handled(my dilated rate might not big enough), could you please give me some more advice on how to tackle with your models since handling problems with DL is really exciting.", "votes": 1 }, { "id": 268974, "postDate": "2018-01-16T02:02:59.103Z", "content": "Your solutions always excite me. Good luck with your research!
", "rawMarkdown": "Your solutions always excite me. Good luck with your research!", "votes": 1 }, { "id": 269141, "postDate": "2018-01-16T09:50:34.693Z", "content": "Thanks for sharing. I had the intent to run your WTF model here, but could not find the energy (because of the flu). Congrats for your GM title, and hope you'll be back some time here.
\n\nI'm not sure I agree with your negative feelings about how this was prepared. In many real life situations I deal with, the data we have now is different from history, for many reasons, mostly because more data is collected now than before. This competition wad a similar flavor, and it does not turn it into a lottery IMHO. It turns it into something more realistic.
", "rawMarkdown": "Thanks for sharing. I had the intent to run your WTF model here, but could not find the energy (because of the flu). Congrats for your GM title, and hope you'll be back some time here.\n\nI'm not sure I agree with your negative feelings about how this was prepared. In many real life situations I deal with, the data we have now is different from history, for many reasons, mostly because more data is collected now than before. This competition wad a similar flavor, and it does not turn it into a lottery IMHO. It turns it into something more realistic." }, { "id": 269475, "postDate": "2018-01-16T22:24:36.527Z", "content": "It is ingenious how you and your team tried to impute the missing promotion value. But did you validated this approach in your local CV as well? I have a feeling this is where you overfitted the public LB.
", "rawMarkdown": "It is ingenious how you and your team tried to impute the missing promotion value. But did you validated this approach in your local CV as well? I have a feeling this is where you overfitted the public LB.", "votes": 2 }, { "id": 269120, "postDate": "2018-01-16T08:35:22.240Z", "content": "Hi sjv, big thanks to you. I adapted your model from WTF to this competition. I think your code is organized well. Solution is clean. I suggest everybody to look at your WTF solution.
\n\nSame as you, I was bothered by onpromotion data for very long time. My approach is when decoding, not only use the onpromotion on that day. I added more onpromotion information as feature by shifting. i.e. when decoding, not only depends on the onpromotion on predicted day. also depends on onpromotion data on previous days and future day.\nSo, the question transferred to if yesterday, today, tomorrow, day after tomorrow etc is onpromotion or not onpromotion, what about today's unit sales. This idea is triggered by WTF top solution which shifted the history yearly, quarterly data to feed into its RNN. I did also, meanwhile, I thought the onpromotion for past and future could also shifted to give more information.
\n\nThanks again for your sharing code on WTF.
", "rawMarkdown": "Hi sjv, big thanks to you. I adapted your model from WTF to this competition. I think your code is organized well. Solution is clean. I suggest everybody to look at your WTF solution.\n\nSame as you, I was bothered by onpromotion data for very long time. My approach is when decoding, not only use the onpromotion on that day. I added more onpromotion information as feature by shifting. i.e. when decoding, not only depends on the onpromotion on predicted day. also depends on onpromotion data on previous days and future day.\nSo, the question transferred to if yesterday, today, tomorrow, day after tomorrow etc is onpromotion or not onpromotion, what about today's unit sales. This idea is triggered by WTF top solution which shifted the history yearly, quarterly data to feed into its RNN. I did also, meanwhile, I thought the onpromotion for past and future could also shifted to give more information.\n\nThanks again for your sharing code on WTF. ", "votes": 2, "replies": [ { "id": 269307, "postDate": "2018-01-16T16:06:09.527Z", "content": "Glad to hear my code was useful. I used a bidirectional LSTM on the 16 future onpromotion values and fed the output states to the decoder. I think this has the same effect as what you are describing. Unfortunately, I think it makes the model more sensitive to onpromotion values, making the effect of the train bias more noticeable.
", "rawMarkdown": "Glad to hear my code was useful. I used a bidirectional LSTM on the 16 future onpromotion values and fed the output states to the decoder. I think this has the same effect as what you are describing. Unfortunately, I think it makes the model more sensitive to onpromotion values, making the effect of the train bias more noticeable." } ] }, { "id": 269040, "postDate": "2018-01-16T04:57:51.630Z", "content": "Similar semi-random shake-ups happens very oftenly last time on Kaggle. +-100 places, and some times +-700 places in average...
", "rawMarkdown": "Similar semi-random shake-ups happens very oftenly last time on Kaggle. +-100 places, and some times +-700 places in average...", "votes": 2, "replies": [ { "id": 269043, "postDate": "2018-01-16T05:12:37.927Z", "content": "Shakeups are expected in certain competitions, and it's generally possible to identify these before entering the competition. If the data was prepared responsibly and the train/test mismatch was handled more carefully, this competition would have had a much more reasonable shakeup. You should not dismiss the shakeup in this competition as an intrinsic part of machine learning competitions -- it was the result of negligence on Kaggle's behalf (though this is not always the case). I don't like to be this critical, but I haven't seen any evidence that Kaggle takes this issue seriously. Frankly, respected academics and AI/ML researchers don't hold my Kaggle performances in high regard at all, particularly because Kaggle results are extremely noisy signals of actual ML competence. Competitions like this one only further this poor reputation.
", "rawMarkdown": "Shakeups are expected in certain competitions, and it's generally possible to identify these before entering the competition. If the data was prepared responsibly and the train/test mismatch was handled more carefully, this competition would have had a much more reasonable shakeup. You should not dismiss the shakeup in this competition as an intrinsic part of machine learning competitions -- it was the result of negligence on Kaggle's behalf (though this is not always the case). I don't like to be this critical, but I haven't seen any evidence that Kaggle takes this issue seriously. Frankly, respected academics and AI/ML researchers don't hold my Kaggle performances in high regard at all, particularly because Kaggle results are extremely noisy signals of actual ML competence. Competitions like this one only further this poor reputation.", "votes": 12 }, { "id": 269356, "postDate": "2018-01-16T17:20:46.480Z", "content": "@sjv, Academics don't value what they are not good at. Academics do not like reality when it does not match their assumptions either. They don't value Kaggle because they either don't really get what it is, or they tried to compete and did poorly. Sure, Kaggle data preparation is not always great, by far, but at least the private score is an independent, not human biased, evaluation. Academics have a poor reputation when it comes to reproducibility of results and unbiased performance evaluation. Look at the number of paper retractations in general. Frankly, I much prefer Kaggle way than peer reviews. And, FYI, I published enough academic papers in top conferences and journals to know what I am discussing here.
", "rawMarkdown": "@sjv, Academics don't value what they are not good at. Academics do not like reality when it does not match their assumptions either. They don't value Kaggle because they either don't really get what it is, or they tried to compete and did poorly. Sure, Kaggle data preparation is not always great, by far, but at least the private score is an independent, not human biased, evaluation. Academics have a poor reputation when it comes to reproducibility of results and unbiased performance evaluation. Look at the number of paper retractations in general. Frankly, I much prefer Kaggle way than peer reviews. And, FYI, I published enough academic papers in top conferences and journals to know what I am discussing here.", "votes": 17 }, { "id": 269976, "postDate": "2018-01-17T15:55:43.363Z", "content": "Hi @CPMP, on an unrelated note, congratulations on becoming grandmaster and discussions #1, I always learnt from you in discussions and hope to see you more in future.
", "rawMarkdown": "Hi @CPMP, on an unrelated note, congratulations on becoming grandmaster and discussions #1, I always learnt from you in discussions and hope to see you more in future.", "votes": 1 }, { "id": 269996, "postDate": "2018-01-17T16:21:24.380Z", "content": "Thanks, glad you find some of my endless posting to be of interest!
", "rawMarkdown": "Thanks, glad you find some of my endless posting to be of interest!", "votes": 1 } ] }, { "id": 268987, "postDate": "2018-01-16T02:57:40.423Z", "content": "Although there is a little regret,congratulate to be a grandmaster.Good luck with your research!
", "rawMarkdown": "Although there is a little regret,congratulate to be a grandmaster.Good luck with your research!", "votes": 2 }, { "id": 270534, "postDate": "2018-01-18T12:49:27.343Z", "content": "Thanks for sharing and Good luck on research
", "rawMarkdown": "Thanks for sharing and Good luck on research" }, { "id": 269592, "postDate": "2018-01-17T02:38:41.203Z", "content": "Thank you for this sharing and good luck with you
", "rawMarkdown": "Thank you for this sharing and good luck with you" }, { "id": 269411, "postDate": "2018-01-16T19:27:19.843Z", "content": "Thanks a lot for the detailed explanation. Your repo is golden for the Kaggle community.
", "rawMarkdown": "Thanks a lot for the detailed explanation. Your repo is golden for the Kaggle community." }, { "id": 269124, "postDate": "2018-01-16T08:49:06.473Z", "content": "Thank you so much, @sjv! I will learn a lot from your approach. \nI would love to study your source code for this competition though! Any chance of sharing this? :) \nPlease :D
", "rawMarkdown": "Thank you so much, @sjv! I will learn a lot from your approach. \nI would **love** to study your source code for this competition though! Any chance of sharing this? :) \nPlease :D" }, { "id": 269114, "postDate": "2018-01-16T08:18:12.830Z", "content": "Would the method described in the following document a potential solution for missing feature data? \nhttp://kennethmarino.weebly.com/uploads/3/7/2/5/37255427/kenneth_marino_goldwater_essay.pdf\nThat is, to add an additional binary input node corresponding to every feature to indicate if it is missing
", "rawMarkdown": "Would the method described in the following document a potential solution for missing feature data? \nhttp://kennethmarino.weebly.com/uploads/3/7/2/5/37255427/kenneth_marino_goldwater_essay.pdf\nThat is, to add an additional binary input node corresponding to every feature to indicate if it is missing" }, { "id": 269109, "postDate": "2018-01-16T08:05:12.723Z", "content": "thanks for sharing
\n\nbtw. I totally agree on the \"unseen items\" issue - I tried a few things but neither worked realy well, the biggest challenge is that you have no chance to know which items are even planned to be sold at a store - I tried item class + store type approach, item class + store cluster, considering only items which are \"onpromotion\" in the test assuming that that's the planned start of the new items etc. but neither really worked; would love to know if some of the top teams managed this challenge better than the rets of us :)
", "rawMarkdown": "thanks for sharing\n\nbtw. I totally agree on the \"unseen items\" issue - I tried a few things but neither worked realy well, the biggest challenge is that you have no chance to know which items are even planned to be sold at a store - I tried item class + store type approach, item class + store cluster, considering only items which are \"onpromotion\" in the test assuming that that's the planned start of the new items etc. but neither really worked; would love to know if some of the top teams managed this challenge better than the rets of us :)" }, { "id": 269069, "postDate": "2018-01-16T06:10:36.823Z", "content": "Congrats. You were quite resourceful to deal with a variety of issues. I enjoyed reading it. Thanks for your sharing.
", "rawMarkdown": "Congrats. You were quite resourceful to deal with a variety of issues. I enjoyed reading it. Thanks for your sharing." }, { "id": 269014, "postDate": "2018-01-16T03:58:58.680Z", "content": "Congratulations! Thanks for sharing the solution.
", "rawMarkdown": "Congratulations! Thanks for sharing the solution." }, { "id": 268946, "postDate": "2018-01-16T01:07:39.377Z", "content": "That's awesome!!!
", "rawMarkdown": "That's awesome!!!" }, { "id": 268945, "postDate": "2018-01-16T01:03:50.337Z", "content": "Thank you for sharing, and congratulations for your result!
", "rawMarkdown": "Thank you for sharing, and congratulations for your result!" }, { "id": 268944, "postDate": "2018-01-16T01:01:14.643Z", "content": "yeah, I think so. It seems that naively put those weired data like having not enough training data to 0 would be better rather than make the model to predict them. The main problem is that this company just simply make the predict dataset from the product of store, item and date. Really disappointed.
", "rawMarkdown": "yeah, I think so. It seems that naively put those weired data like having not enough training data to 0 would be better rather than make the model to predict them. The main problem is that this company just simply make the predict dataset from the product of store, item and date. Really disappointed." }, { "id": 268942, "postDate": "2018-01-16T00:50:19.517Z", "content": "Big cong!
", "rawMarkdown": "Big cong!" }, { "id": 269918, "postDate": "2018-01-17T14:28:03.863Z", "rawMarkdown": "", "isDeleted": true }, { "id": 269063, "postDate": "2018-01-16T05:55:16.733Z", "rawMarkdown": "", "votes": -3, "isDeleted": true }, { "id": 271077, "postDate": "2018-01-19T16:32:27.177Z", "content": "Thank you for sharing
", "rawMarkdown": "Thank you for sharing" }, { "id": 270807, "postDate": "2018-01-19T01:32:13.220Z", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing." }, { "id": 269776, "postDate": "2018-01-17T09:30:32.513Z", "content": "Thanks for sharing.
", "rawMarkdown": "Thanks for sharing." }, { "id": 269102, "postDate": "2018-01-16T07:50:48.953Z", "content": "Thanks for sharing. ALl the best!!
", "rawMarkdown": "Thanks for sharing. ALl the best!!" }, { "id": 269061, "postDate": "2018-01-16T05:51:25.230Z", "content": "Thanks for sharing the solution
", "rawMarkdown": "Thanks for sharing the solution" }, { "id": 268969, "postDate": "2018-01-16T01:57:36.910Z", "content": "Thank you for sharing !
", "rawMarkdown": "Thank you for sharing ! " } ], "comments": [ { "id": 269095, "author_name": "Giba", "author_url": "", "post_date": "2018-01-16T07:29:26.820000", "content": "I agree about bad data preparation in this competition, but unfortunately it's not the first nor the second time it happens. Kaggle really need to do an octopus review of the datasets before launching competitions.
", "votes": 8, "replies": [] }, { "id": 269421, "author_name": "Bcottman", "author_url": "", "post_date": "2018-01-16T20:09:33.783000", "content": "I agree with CPMP and would add that real-world data starts out very dirty and wrong! I spend most of my time detecting and throwing out mis-labeled data. One of my side research areas is to use unsupervised learning to detect such data. Knn works best so far. ( that is how bad my world is.) I like Kaggle completions as data is so clean. kaggle to me is a game and data leaks and such just add spice. However, SJV I understand your outrage. Also, thank for your WTF code. Before this Categoical embedding was SOA.
", "votes": 5, "replies": [] }, { "id": 269042, "author_name": "Rand Xie", "author_url": "", "post_date": "2018-01-16T05:10:40.917000", "content": "@sjv, so sad that you are leaving Kaggle. I am always impressed by your solutions that apply DL to time series problems. It gives me a lot of inspiration. All the best to your research.
", "votes": 6, "replies": [] }, { "id": 269425, "author_name": "Bcottman", "author_url": "", "post_date": "2018-01-16T20:20:05.503000", "content": "I have to add I agree with cpmp on his first post and don’t necessarily agree in the sense that some academics don,t handle very well the dirty real world and some handle it very well.
", "votes": 3, "replies": [] }, { "id": 269311, "author_name": "Laurent Oscard", "author_url": "", "post_date": "2018-01-16T16:09:50.730000", "content": "Thank you for this sharing, and congratulations for your results
", "votes": 3, "replies": [] }, { "id": 269049, "author_name": "Lingzhi", "author_url": "", "post_date": "2018-01-16T05:25:00.537000", "content": "Thank you so much and good luck with you. I've learned so much from your previous codes.
", "votes": 3, "replies": [] }, { "id": 269016, "author_name": "Louis T.", "author_url": "", "post_date": "2018-01-16T04:06:25.053000", "content": "Among many other things, I am most impressed with your clean code. My code always ends up a mess after doing n different kind of experiment. It would be nice if you can share some tips on your workflow, version control and keeping track of experiences.
\n\nSad to see you go, all the best with your research.
", "votes": 3, "replies": [] }, { "id": 269165, "author_name": "Eric Perbos-Brinck", "author_url": "", "post_date": "2018-01-16T11:19:51.043000", "content": "Your works on Time Series was referred to me by another Gold medalist here :-D
\n\nAnd congrats on the GM title.
\n\nTBH, on the unseen data, I think it's more a case of Favorita DS team being lazy than Kaggle's fault.
\n\nThey dumped into the Test set all the potential combinations of 'Item_nbr * Store_nbr', while in Real-Life there is no such thing: retailers worldwide use Store Types, Clusters and Assortment Classes to control which 'Item_nbr' will be present in which 'Store_nbr', and not.
\n\nShelf space is a store's most precious asset and is monetized to major Brands (ie. they often pay for guaranteed space in number of \"facings\", including eye or hand level for max exposure, then send sales reps to control it in stores and take pictures if not enforced and ask for refunds=messy).
\n\nEx: a size 5 store (max size +10,000sqm) has room for 10,000 different SKUs (Stock Keeping Units) on shelves, like 10 types of Coca-Cola formats (from 4*33cl cans to 6*2L PET bottles),
\n\nwhile a size 1 (min size 2000sqm) has room for 2,000 SKUS only, like 4 types of Coke (nothing larger than a single 1.5L PET bottle).
\n\nIt's carefully decided at HQ level by Marketing/Merchandising/Purchasing and it triggers down the chain via centralized IT to the store floors:
\n\nEven if that store stole a truck of Coke 6*2L Bottles, it could not sell it via the company's centralized IT system.
\n\nThat 'Item_nbr * Store_nbr', plus probably 1.5M other rows in the Test set could never happen, and Favorita's people at HQ know it very well.
\n\nIt's Retail 101 ^!^
", "votes": 4, "replies": [] }, { "id": 269535, "author_name": "Bcottman", "author_url": "", "post_date": "2018-01-17T01:06:17.290000", "content": "Sjv May have meant that he (they) could not build a predictive valid set for private but only for public .however I am not sure because I think they gave you private LB so I assume I you could build a linear predictive validation set by probing the private LB. a I did not try myself.
", "votes": 1, "replies": [] }, { "id": 269041, "author_name": "ZavodRobotov", "author_url": "", "post_date": "2018-01-16T05:05:28.277000", "content": "Hope that Kaggle will fix this issue
", "votes": 1, "replies": [] }, { "id": 269025, "author_name": "FengLi", "author_url": "", "post_date": "2018-01-16T04:36:06.973000", "content": "\n\n\nAs discussed in the forums, if you naively impute missing onpromotion values with 0, the sales distribution conditioned on onpromotion values will differ in the train and test sets. I didn't find a way to remedy this, but thankfully my teammates were able to come up with a clever solution which improved my score by about .004. The missing values were imputed randomly: 1 with probability p and 0 otherwise. Determining p was a rather tedious task, since we had no data to model it and we had to resort to trial and error leaderboard submissions. We ended up setting p separately for each day, and it was computed as the mean onpromotion rate for each day, scaled by a learnable factor estimated by stochastic leaderboard descent (leaderboard probing...). All credit goes to my teammates for this, and they can probably explain it better than I can.
\n
So smart... Does this way also help you improve the private LB score? I'm always impressive with your work and all best with your future research.
", "votes": 1, "replies": [ { "id": 269028, "author_name": "sjv", "author_url": "", "post_date": "2018-01-16T04:42:52.427000", "content": "For some models it helped and for some it did not. Our private leaderboard scores varied drastically between runs, even when the public leaderboard score stayed the same. While in the end it did not work out very well, I'm confident that the approach of using the leaderboard as validation was the optimal strategy from a statistical viewpoint. We have no choice but to fit our models to the public leaderboard when the organizers don't provide any training data that resembles the test set...
", "votes": 3, "replies": [] } ] }, { "id": 269000, "author_name": "Arjun Blum", "author_url": "", "post_date": "2018-01-16T03:20:12.747000", "content": "Thanks for sharing. Really clever approach with the promotion data.
", "votes": 1, "replies": [] }, { "id": 268988, "author_name": "Gerryl", "author_url": "", "post_date": "2018-01-16T02:58:02.500000", "content": "Thanks for sharing your excellent solution once again and wish you the best of luck in your future research/study/work and definitely we will miss you !\nAnd one problem I encountered during using your WaveNet framework is that my model can not even converge( I brutely force the model to consume the raw sequence with 210k*1684 and combine it with some embedded features), I guess the problem might be 2/3 of the series are zero and the distribution of the data is not well handled(my dilated rate might not big enough), could you please give me some more advice on how to tackle with your models since handling problems with DL is really exciting.
", "votes": 1, "replies": [] }, { "id": 268974, "author_name": "akiyama", "author_url": "", "post_date": "2018-01-16T02:02:59.103000", "content": "Your solutions always excite me. Good luck with your research!
", "votes": 1, "replies": [] }, { "id": 269141, "author_name": "CPMP", "author_url": "", "post_date": "2018-01-16T09:50:34.693000", "content": "Thanks for sharing. I had the intent to run your WTF model here, but could not find the energy (because of the flu). Congrats for your GM title, and hope you'll be back some time here.
\n\nI'm not sure I agree with your negative feelings about how this was prepared. In many real life situations I deal with, the data we have now is different from history, for many reasons, mostly because more data is collected now than before. This competition wad a similar flavor, and it does not turn it into a lottery IMHO. It turns it into something more realistic.
", "votes": 0, "replies": [] }, { "id": 269475, "author_name": "Louis T.", "author_url": "", "post_date": "2018-01-16T22:24:36.527000", "content": "It is ingenious how you and your team tried to impute the missing promotion value. But did you validated this approach in your local CV as well? I have a feeling this is where you overfitted the public LB.
", "votes": 2, "replies": [] }, { "id": 269120, "author_name": "Lucius Yu", "author_url": "", "post_date": "2018-01-16T08:35:22.240000", "content": "Hi sjv, big thanks to you. I adapted your model from WTF to this competition. I think your code is organized well. Solution is clean. I suggest everybody to look at your WTF solution.
\n\nSame as you, I was bothered by onpromotion data for very long time. My approach is when decoding, not only use the onpromotion on that day. I added more onpromotion information as feature by shifting. i.e. when decoding, not only depends on the onpromotion on predicted day. also depends on onpromotion data on previous days and future day.\nSo, the question transferred to if yesterday, today, tomorrow, day after tomorrow etc is onpromotion or not onpromotion, what about today's unit sales. This idea is triggered by WTF top solution which shifted the history yearly, quarterly data to feed into its RNN. I did also, meanwhile, I thought the onpromotion for past and future could also shifted to give more information.
\n\nThanks again for your sharing code on WTF.
", "votes": 2, "replies": [ { "id": 269307, "author_name": "sjv", "author_url": "", "post_date": "2018-01-16T16:06:09.527000", "content": "Glad to hear my code was useful. I used a bidirectional LSTM on the 16 future onpromotion values and fed the output states to the decoder. I think this has the same effect as what you are describing. Unfortunately, I think it makes the model more sensitive to onpromotion values, making the effect of the train bias more noticeable.
", "votes": 0, "replies": [] } ] }, { "id": 269040, "author_name": "ZavodRobotov", "author_url": "", "post_date": "2018-01-16T04:57:51.630000", "content": "Similar semi-random shake-ups happens very oftenly last time on Kaggle. +-100 places, and some times +-700 places in average...
", "votes": 2, "replies": [ { "id": 269043, "author_name": "sjv", "author_url": "", "post_date": "2018-01-16T05:12:37.927000", "content": "Shakeups are expected in certain competitions, and it's generally possible to identify these before entering the competition. If the data was prepared responsibly and the train/test mismatch was handled more carefully, this competition would have had a much more reasonable shakeup. You should not dismiss the shakeup in this competition as an intrinsic part of machine learning competitions -- it was the result of negligence on Kaggle's behalf (though this is not always the case). I don't like to be this critical, but I haven't seen any evidence that Kaggle takes this issue seriously. Frankly, respected academics and AI/ML researchers don't hold my Kaggle performances in high regard at all, particularly because Kaggle results are extremely noisy signals of actual ML competence. Competitions like this one only further this poor reputation.
", "votes": 12, "replies": [] }, { "id": 269356, "author_name": "CPMP", "author_url": "", "post_date": "2018-01-16T17:20:46.480000", "content": "@sjv, Academics don't value what they are not good at. Academics do not like reality when it does not match their assumptions either. They don't value Kaggle because they either don't really get what it is, or they tried to compete and did poorly. Sure, Kaggle data preparation is not always great, by far, but at least the private score is an independent, not human biased, evaluation. Academics have a poor reputation when it comes to reproducibility of results and unbiased performance evaluation. Look at the number of paper retractations in general. Frankly, I much prefer Kaggle way than peer reviews. And, FYI, I published enough academic papers in top conferences and journals to know what I am discussing here.
", "votes": 17, "replies": [] }, { "id": 269976, "author_name": "Ravi Teja Gutta", "author_url": "", "post_date": "2018-01-17T15:55:43.363000", "content": "Hi @CPMP, on an unrelated note, congratulations on becoming grandmaster and discussions #1, I always learnt from you in discussions and hope to see you more in future.
", "votes": 1, "replies": [] }, { "id": 269996, "author_name": "CPMP", "author_url": "", "post_date": "2018-01-17T16:21:24.380000", "content": "Thanks, glad you find some of my endless posting to be of interest!
", "votes": 1, "replies": [] } ] }, { "id": 268987, "author_name": "plantsgo", "author_url": "", "post_date": "2018-01-16T02:57:40.423000", "content": "Although there is a little regret,congratulate to be a grandmaster.Good luck with your research!
", "votes": 2, "replies": [] }, { "id": 270534, "author_name": "", "author_url": "", "post_date": "2018-01-18T12:49:27.343000", "content": "Thanks for sharing and Good luck on research
", "votes": 0, "replies": [] }, { "id": 269592, "author_name": "Sebastian wanlong", "author_url": "", "post_date": "2018-01-17T02:38:41.203000", "content": "Thank you for this sharing and good luck with you
", "votes": 0, "replies": [] }, { "id": 269411, "author_name": "Samet Marasli", "author_url": "", "post_date": "2018-01-16T19:27:19.843000", "content": "Thanks a lot for the detailed explanation. Your repo is golden for the Kaggle community.
", "votes": 0, "replies": [] }, { "id": 269124, "author_name": "Ricko", "author_url": "", "post_date": "2018-01-16T08:49:06.473000", "content": "Thank you so much, @sjv! I will learn a lot from your approach. \nI would love to study your source code for this competition though! Any chance of sharing this? :) \nPlease :D
", "votes": 0, "replies": [] }, { "id": 269114, "author_name": "Frank Pan", "author_url": "", "post_date": "2018-01-16T08:18:12.830000", "content": "Would the method described in the following document a potential solution for missing feature data? \nhttp://kennethmarino.weebly.com/uploads/3/7/2/5/37255427/kenneth_marino_goldwater_essay.pdf\nThat is, to add an additional binary input node corresponding to every feature to indicate if it is missing
", "votes": 0, "replies": [] }, { "id": 269109, "author_name": "steelrose", "author_url": "", "post_date": "2018-01-16T08:05:12.723000", "content": "thanks for sharing
\n\nbtw. I totally agree on the \"unseen items\" issue - I tried a few things but neither worked realy well, the biggest challenge is that you have no chance to know which items are even planned to be sold at a store - I tried item class + store type approach, item class + store cluster, considering only items which are \"onpromotion\" in the test assuming that that's the planned start of the new items etc. but neither really worked; would love to know if some of the top teams managed this challenge better than the rets of us :)
", "votes": 0, "replies": [] }, { "id": 269069, "author_name": "Ernest Chang", "author_url": "", "post_date": "2018-01-16T06:10:36.823000", "content": "Congrats. You were quite resourceful to deal with a variety of issues. I enjoyed reading it. Thanks for your sharing.
", "votes": 0, "replies": [] }, { "id": 269014, "author_name": "HuangBingchu", "author_url": "", "post_date": "2018-01-16T03:58:58.680000", "content": "Congratulations! Thanks for sharing the solution.
", "votes": 0, "replies": [] }, { "id": 268946, "author_name": "K Lin", "author_url": "", "post_date": "2018-01-16T01:07:39.377000", "content": "That's awesome!!!
", "votes": 0, "replies": [] }, { "id": 268945, "author_name": "Larxel", "author_url": "", "post_date": "2018-01-16T01:03:50.337000", "content": "Thank you for sharing, and congratulations for your result!
", "votes": 0, "replies": [] }, { "id": 268944, "author_name": "", "author_url": "", "post_date": "2018-01-16T01:01:14.643000", "content": "", "votes": 0, "replies": [] }, { "id": 268942, "author_name": "", "author_url": "", "post_date": "2018-01-16T00:50:19.517000", "content": "", "votes": 0, "replies": [] }, { "id": 269918, "author_name": "", "author_url": "", "post_date": "2018-01-17T14:28:03.863000", "content": "", "votes": 0, "replies": [] }, { "id": 269063, "author_name": "", "author_url": "", "post_date": "2018-01-16T05:55:16.733000", "content": "", "votes": -3, "replies": [] }, { "id": 271077, "author_name": "", "author_url": "", "post_date": "2018-01-19T16:32:27.177000", "content": "", "votes": 0, "replies": [] }, { "id": 270807, "author_name": "", "author_url": "", "post_date": "2018-01-19T01:32:13.220000", "content": "", "votes": 0, "replies": [] }, { "id": 269776, "author_name": "", "author_url": "", "post_date": "2018-01-17T09:30:32.513000", "content": "", "votes": 0, "replies": [] }, { "id": 269102, "author_name": "", "author_url": "", "post_date": "2018-01-16T07:50:48.953000", "content": "", "votes": 0, "replies": [] }, { "id": 269061, "author_name": "", "author_url": "", "post_date": "2018-01-16T05:51:25.230000", "content": "", "votes": 0, "replies": [] }, { "id": 268969, "author_name": "", "author_url": "", "post_date": "2018-01-16T01:57:36.910000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "268938": "First off, congrats to the winners and thanks to all those who contributed to the forum discussions. I'm definitely disappointed with the result, but I might as well share my solution anyways. I'll only discuss my contribution to my team's solution, which is a single model which scores .499 on the public leaderboard. \n\n### Model Architecture\nWe framed forecasting as a sequence to sequence modeling problem, where the encoder reads in a portion of a time series and the decoder sequentially emits estimates of the subsequent 16 values. The encoder and decoder do not share parameters, but they are both similarly parameterized as a stack of ~30 dilated, causal convolutions. Dilated convolutions allowed the receptive field to span the entirety of the 4+ year train period, while still maintaining a small number of parameters (~1 million). To allow the decoder to use future onpromotion values, it was also augmented with a bidirectional LSTM which encoded information from prior and future onpromotion data. The model struggled to accurately predict zero, so we modified the architecture to additionally emit a bernoulli parameter at each timestep (corresponding to the probability that the output was zero) and the final predictions were adjusted accordingly. Weighted RMSE of the log-transformed data was directly minimized.\n\n### Input Representation\nThe input to the network consisted of the raw time series values, embeddings of the categorical variables, and some manually engineered features. The manual features included lags, diffs, rolling statistics, date features, and conditioning time series (i.e. average sales for a given product/store/etc.). Promotion data required some special handling, which I'll describe later.\n\n### Validation\n5% of the time series were held out, and the model was evaluated on random periods from the last 365 training days of these held out series. The purpose of this was to prevent the model from overfitting a validation set which was biased toward any sort of weekly or monthly trends.\n\n### Promotion Data\nAs discussed in the forums, if you naively impute missing onpromotion values with 0, the sales distribution conditioned on onpromotion values will differ in the train and test sets. I didn't find a way to remedy this, but thankfully my teammates were able to come up with a clever solution which improved my score by about .004. The missing values were imputed randomly: 1 with probability p and 0 otherwise. Determining p was a rather tedious task, since we had no data to model it and we had to resort to trial and error leaderboard submissions. We ended up setting p separately for each day, and it was computed as the mean onpromotion rate for each day, scaled by a learnable factor estimated by stochastic leaderboard descent (leaderboard probing...). All credit goes to my teammates for this, and they can probably explain it better than I can.\n\n### Source Code\nI normally share my source code, but since the core model is nearly identical to one I've already shared, I'll just refer you to that instead: https://github.com/sjvasquez/web-traffic-forecasting\n\nFinally, I'll admit I'm mostly disappointed with Kaggle. I raised concerns about the unseen items in the test set and also about the onpromotion issue earlier in the competition. Ultimately, the competition was a lottery. For the sake of others, I hope Kaggle will begin to demonstrate a moderate level of care in data preparation and show that they understand that whenever they deviate from the assumption that train set is an unbiased sampling of the test distribution, they induce randomness which turns competitions into lotteries. This will be my last Kaggle competition, as my time will be better spent focusing on research from now on. Best of luck to everyone.", "269095": "I agree about bad data preparation in this competition, but unfortunately it's not the first nor the second time it happens. Kaggle really need to do an octopus review of the datasets before launching competitions.", "269421": "I agree with CPMP and would add that real-world data starts out very dirty and wrong! I spend most of my time detecting and throwing out mis-labeled data. One of my side research areas is to use unsupervised learning to detect such data. Knn works best so far. ( that is how bad my world is.) I like Kaggle completions as data is so clean. kaggle to me is a game and data leaks and such just add spice. However, SJV I understand your outrage. Also, thank for your WTF code. Before this Categoical embedding was SOA.", "269042": "@sjv, so sad that you are leaving Kaggle. I am always impressed by your solutions that apply DL to time series problems. It gives me a lot of inspiration. All the best to your research.", "269425": "I have to add I agree with cpmp on his first post and don’t necessarily agree in the sense that some academics don,t handle very well the dirty real world and some handle it very well.", "269311": "Thank you for this sharing, and congratulations for your results \n\n", "269049": "Thank you so much and good luck with you. I've learned so much from your previous codes.", "269016": "Among many other things, I am most impressed with your clean code. My code always ends up a mess after doing n different kind of experiment. It would be nice if you can share some tips on your workflow, version control and keeping track of experiences.\n\nSad to see you go, all the best with your research. ", "269165": "Your works on Time Series was referred to me by another Gold medalist here :-D\n\nAnd congrats on the GM title.\n\nTBH, on the unseen data, I think it's more a case of Favorita DS team being lazy than Kaggle's fault.\n\nThey dumped into the Test set all the potential combinations of 'Item_nbr * Store_nbr', while in Real-Life there is no such thing: retailers worldwide use Store Types, Clusters and Assortment Classes to control which 'Item_nbr' will be present in which 'Store_nbr', and not.\n\nShelf space is a store's most precious asset and is monetized to major Brands (ie. they often pay for guaranteed space in number of \"facings\", including eye or hand level for max exposure, then send sales reps to control it in stores and take pictures if not enforced and ask for refunds=messy).\n\nEx: a size 5 store (max size +10,000sqm) has room for 10,000 different SKUs (Stock Keeping Units) on shelves, like 10 types of Coca-Cola formats (from 4*33cl cans to 6*2L PET bottles),\n\nwhile a size 1 (min size 2000sqm) has room for 2,000 SKUS only, like 4 types of Coke (nothing larger than a single 1.5L PET bottle).\n\nIt's carefully decided at HQ level by Marketing/Merchandising/Purchasing and it triggers down the chain via centralized IT to the store floors:\n\n* a size 1 store can NOT order Coke in 6*2L PET Bottles\n- it's NOT listed in his purchasing software client\n- its logistic platform/warehouse can NOT deliver it\n- its employees can NOT print its shelves' labels, btw there's no room for it as shelves facings are full already\n- its cashier checkout can NOT scan it.\n\nEven if that store stole a truck of Coke 6*2L Bottles, it could not sell it via the company's centralized IT system.\n\nThat 'Item_nbr * Store_nbr', plus probably 1.5M other rows in the Test set could never happen, and Favorita's people at HQ know it very well.\n\nIt's Retail 101 ^!^\n\n", "269535": "Sjv May have meant that he (they) could not build a predictive valid set for private but only for public .however I am not sure because I think they gave you private LB so I assume I you could build a linear predictive validation set by probing the private LB. a I did not try myself. ", "269041": "Hope that Kaggle will fix this issue", "269025": "> \nAs discussed in the forums, if you naively impute missing onpromotion values with 0, the sales distribution conditioned on onpromotion values will differ in the train and test sets. I didn't find a way to remedy this, but thankfully my teammates were able to come up with a clever solution which improved my score by about .004. The missing values were imputed randomly: 1 with probability p and 0 otherwise. Determining p was a rather tedious task, since we had no data to model it and we had to resort to trial and error leaderboard submissions. We ended up setting p separately for each day, and it was computed as the mean onpromotion rate for each day, scaled by a learnable factor estimated by stochastic leaderboard descent (leaderboard probing...). All credit goes to my teammates for this, and they can probably explain it better than I can.\n> \n\nSo smart... Does this way also help you improve the private LB score? I'm always impressive with your work and all best with your future research.", "269000": "Thanks for sharing. Really clever approach with the promotion data.", "268988": "Thanks for sharing your excellent solution once again and wish you the best of luck in your future research/study/work and definitely we will miss you !\nAnd one problem I encountered during using your WaveNet framework is that my model can not even converge( I brutely force the model to consume the raw sequence with 210k*1684 and combine it with some embedded features), I guess the problem might be 2/3 of the series are zero and the distribution of the data is not well handled(my dilated rate might not big enough), could you please give me some more advice on how to tackle with your models since handling problems with DL is really exciting.", "268974": "Your solutions always excite me. Good luck with your research!", "269141": "Thanks for sharing. I had the intent to run your WTF model here, but could not find the energy (because of the flu). Congrats for your GM title, and hope you'll be back some time here.\n\nI'm not sure I agree with your negative feelings about how this was prepared. In many real life situations I deal with, the data we have now is different from history, for many reasons, mostly because more data is collected now than before. This competition wad a similar flavor, and it does not turn it into a lottery IMHO. It turns it into something more realistic.", "269475": "It is ingenious how you and your team tried to impute the missing promotion value. But did you validated this approach in your local CV as well? I have a feeling this is where you overfitted the public LB.", "269120": "Hi sjv, big thanks to you. I adapted your model from WTF to this competition. I think your code is organized well. Solution is clean. I suggest everybody to look at your WTF solution.\n\nSame as you, I was bothered by onpromotion data for very long time. My approach is when decoding, not only use the onpromotion on that day. I added more onpromotion information as feature by shifting. i.e. when decoding, not only depends on the onpromotion on predicted day. also depends on onpromotion data on previous days and future day.\nSo, the question transferred to if yesterday, today, tomorrow, day after tomorrow etc is onpromotion or not onpromotion, what about today's unit sales. This idea is triggered by WTF top solution which shifted the history yearly, quarterly data to feed into its RNN. I did also, meanwhile, I thought the onpromotion for past and future could also shifted to give more information.\n\nThanks again for your sharing code on WTF. ", "269040": "Similar semi-random shake-ups happens very oftenly last time on Kaggle. +-100 places, and some times +-700 places in average...", "268987": "Although there is a little regret,congratulate to be a grandmaster.Good luck with your research!", "270534": "Thanks for sharing and Good luck on research", "269592": "Thank you for this sharing and good luck with you", "269411": "Thanks a lot for the detailed explanation. Your repo is golden for the Kaggle community.", "269124": "Thank you so much, @sjv! I will learn a lot from your approach. \nI would **love** to study your source code for this competition though! Any chance of sharing this? :) \nPlease :D", "269114": "Would the method described in the following document a potential solution for missing feature data? \nhttp://kennethmarino.weebly.com/uploads/3/7/2/5/37255427/kenneth_marino_goldwater_essay.pdf\nThat is, to add an additional binary input node corresponding to every feature to indicate if it is missing", "269109": "thanks for sharing\n\nbtw. I totally agree on the \"unseen items\" issue - I tried a few things but neither worked realy well, the biggest challenge is that you have no chance to know which items are even planned to be sold at a store - I tried item class + store type approach, item class + store cluster, considering only items which are \"onpromotion\" in the test assuming that that's the planned start of the new items etc. but neither really worked; would love to know if some of the top teams managed this challenge better than the rets of us :)", "269069": "Congrats. You were quite resourceful to deal with a variety of issues. I enjoyed reading it. Thanks for your sharing.", "269014": "Congratulations! Thanks for sharing the solution.", "268946": "That's awesome!!!", "268945": "Thank you for sharing, and congratulations for your result!", "268944": "yeah, I think so. It seems that naively put those weired data like having not enough training data to 0 would be better rather than make the model to predict them. The main problem is that this company just simply make the predict dataset from the product of store, item and date. Really disappointed.", "268942": "Big cong!", "269918": "", "269063": "", "271077": "Thank you for sharing", "270807": "Thanks for sharing.", "269776": "Thanks for sharing.", "269102": "Thanks for sharing. ALl the best!!", "269061": "Thanks for sharing the solution", "268969": "Thank you for sharing ! " } }