Cogratulations to the members of #1 and #2 teams, you did a great job here. And my deepest condolences to the teams suffered in this shake-up: I feel for you, guys.
\n\nMy teammate, Ahmet Erdem, will hopefully describe his approach in more detail later, when the time will permit, but as a short summary : He used gradient boosting (LGBM) with an original set of features, and tried to capture the monthly dynamics by using sales on the same days of different months as the train targets.
\n\nMy part of the final solution consists mostly of neural networks, although I also used a LGBM model with a low weight in the final ensemble.
\n\nWhen it comes to time series, I strongly belive that the two most important things are validation and bagging. By correct validation I mean simulating train/test (train/unknown future) split to avoid any kind of future information leaks. Thus, the validation set should include a time period that is not present in train in any way, even with different timeseries, because these time series could correlate with each other.
\n\nSo I used the following train/validation split:\nfor the training set, I took a history of 80 consequtive sales days for each (item, store pair), and used the next 16 sales days as a train set target. The split between training history and training target was always from Tuesday to Wednesday, as in the original train/test split, in orded to capture the weekly dynamics.\nAfter training my models on this set and estimating an optimal number of epochs to run, I retrained them on the latest data available (i.e. finals days of the original training set) in order to use the most up-to-date information.
\n\nAbout bagging : our final result includes models, trained on 10 runs each. The weights initializations are different each time, so results are different as well, and averaging them helps to improve the solution, particularly when dealing with the uncertain future.
\n\nAnother part of my bagging was a bit unusual. After each training epoch (including the very first) I predicted the targets, and then averaged these predictions. I've read about averaging a few last epochs, but never heard about averaging everything, even the first very underfitted epoch. Yet my validation consistently showed that it does improve the results.
\n\nAs for the model's achitectures, I belive they are relatively less important here. I had an ensemble of 3 NNs : CNN, LSTM and GRU with almost equal weights. As a single model, GRU shows a bit better results than the rest, but an ensemble is even better, as usual.
\n\nAll of them have 3 top (convolutional or recurrent) layers, 2 or 3 bottom fully-connected layers and embedding layers for categorical features. I tried different tweaks like skip-connections, concatenating CNN layers with different convolution kernel sizes and so on, but the results were negligible. The only achitectural tweak that worked better was using dilated convolutions: my CNN uses convolutions with dilation of 2 on the first layer, 8 on the second and 32 on the third (and no pooling). But an original plain CNN without dilated convolutions worked just a bit worse.
\n\nWhat did not work: \nThe main disappointment was the poor results of a version with updated promotion information for the train set. I've used a relatively sophisticated method to fill onpromotion status for the dates of the training period without sales, and it showed very promising results on the public leaderboard on the last day of this competition, boosting our score from .501 to .499. And it failed miserably on the private, so we remained in the prize area only because our other submit was more conservative and used less weight for these adjusted promotions. So next time I'll try to be more careful with the last day \"improvements\".
", "messages": [ { "id": "269132", "postDate": "01/16/2018 09:29:50", "content": "Cogratulations to the members of #1 and #2 teams, you did a great job here. And my deepest condolences to the teams suffered in this shake-up: I feel for you, guys.
\n\nMy teammate, Ahmet Erdem, will hopefully describe his approach in more detail later, when the time will permit, but as a short summary : He used gradient boosting (LGBM) with an original set of features, and tried to capture the monthly dynamics by using sales on the same days of different months as the train targets.
\n\nMy part of the final solution consists mostly of neural networks, although I also used a LGBM model with a low weight in the final ensemble.
\n\nWhen it comes to time series, I strongly belive that the two most important things are validation and bagging. By correct validation I mean simulating train/test (train/unknown future) split to avoid any kind of future information leaks. Thus, the validation set should include a time period that is not present in train in any way, even with different timeseries, because these time series could correlate with each other.
\n\nSo I used the following train/validation split:\nfor the training set, I took a history of 80 consequtive sales days for each (item, store pair), and used the next 16 sales days as a train set target. The split between training history and training target was always from Tuesday to Wednesday, as in the original train/test split, in orded to capture the weekly dynamics.\nAfter training my models on this set and estimating an optimal number of epochs to run, I retrained them on the latest data available (i.e. finals days of the original training set) in order to use the most up-to-date information.
\n\nAbout bagging : our final result includes models, trained on 10 runs each. The weights initializations are different each time, so results are different as well, and averaging them helps to improve the solution, particularly when dealing with the uncertain future.
\n\nAnother part of my bagging was a bit unusual. After each training epoch (including the very first) I predicted the targets, and then averaged these predictions. I've read about averaging a few last epochs, but never heard about averaging everything, even the first very underfitted epoch. Yet my validation consistently showed that it does improve the results.
\n\nAs for the model's achitectures, I belive they are relatively less important here. I had an ensemble of 3 NNs : CNN, LSTM and GRU with almost equal weights. As a single model, GRU shows a bit better results than the rest, but an ensemble is even better, as usual.
\n\nAll of them have 3 top (convolutional or recurrent) layers, 2 or 3 bottom fully-connected layers and embedding layers for categorical features. I tried different tweaks like skip-connections, concatenating CNN layers with different convolution kernel sizes and so on, but the results were negligible. The only achitectural tweak that worked better was using dilated convolutions: my CNN uses convolutions with dilation of 2 on the first layer, 8 on the second and 32 on the third (and no pooling). But an original plain CNN without dilated convolutions worked just a bit worse.
\n\nWhat did not work: \nThe main disappointment was the poor results of a version with updated promotion information for the train set. I've used a relatively sophisticated method to fill onpromotion status for the dates of the training period without sales, and it showed very promising results on the public leaderboard on the last day of this competition, boosting our score from .501 to .499. And it failed miserably on the private, so we remained in the prize area only because our other submit was more conservative and used less weight for these adjusted promotions. So next time I'll try to be more careful with the last day \"improvements\".
", "rawMarkdown": "Cogratulations to the members of #1 and #2 teams, you did a great job here. And my deepest condolences to the teams suffered in this shake-up: I feel for you, guys. \n\nMy teammate, [Ahmet Erdem][1], will hopefully describe his approach in more detail later, when the time will permit, but as a short summary : He used gradient boosting (LGBM) with an original set of features, and tried to capture the monthly dynamics by using sales on the same days of different months as the train targets.\n\nMy part of the final solution consists mostly of neural networks, although I also used a LGBM model with a low weight in the final ensemble.\n\nWhen it comes to time series, I strongly belive that the two most important things are validation and bagging. By correct validation I mean simulating train/test (train/unknown future) split to avoid any kind of future information leaks. Thus, the validation set should include a time period that is not present in train in any way, even with different timeseries, because these time series could correlate with each other. \n\nSo I used the following train/validation split:\nfor the training set, I took a history of 80 consequtive sales days for each (item, store pair), and used the next 16 sales days as a train set target. The split between training history and training target was always from Tuesday to Wednesday, as in the original train/test split, in orded to capture the weekly dynamics.\nAfter training my models on this set and estimating an optimal number of epochs to run, I retrained them on the latest data available (i.e. finals days of the original training set) in order to use the most up-to-date information.\n\nAbout bagging : our final result includes models, trained on 10 runs each. The weights initializations are different each time, so results are different as well, and averaging them helps to improve the solution, particularly when dealing with the uncertain future.\n\nAnother part of my bagging was a bit unusual. After each training epoch (including the very first) I predicted the targets, and then averaged these predictions. I've read about averaging a few last epochs, but never heard about averaging everything, even the first very underfitted epoch. Yet my validation consistently showed that it does improve the results.\n\nAs for the model's achitectures, I belive they are relatively less important here. I had an ensemble of 3 NNs : CNN, LSTM and GRU with almost equal weights. As a single model, GRU shows a bit better results than the rest, but an ensemble is even better, as usual.\n\nAll of them have 3 top (convolutional or recurrent) layers, 2 or 3 bottom fully-connected layers and embedding layers for categorical features. I tried different tweaks like skip-connections, concatenating CNN layers with different convolution kernel sizes and so on, but the results were negligible. The only achitectural tweak that worked better was using dilated convolutions: my CNN uses convolutions with dilation of 2 on the first layer, 8 on the second and 32 on the third (and no pooling). But an original plain CNN without dilated convolutions worked just a bit worse.\n\nWhat did not work: \nThe main disappointment was the poor results of a version with updated promotion information for the train set. I've used a relatively sophisticated method to fill onpromotion status for the dates of the training period without sales, and it showed very promising results on the public leaderboard on the last day of this competition, boosting our score from .501 to .499. And it failed miserably on the private, so we remained in the prize area only because our other submit was more conservative and used less weight for these adjusted promotions. So next time I'll try to be more careful with the last day \"improvements\".\n\n\n\n\n [1]: https://www.kaggle.com/divrikwicky", "votes": null }, { "id": "269137", "postDate": "01/16/2018 09:40:18", "content": "Thanks for sharing.
\n\n> capture the monthly dynamics by using sales on the same days of different months as the train targets.
\n\nInteresting, I'm not sure we managed to capture the monthly dynamics (payday) that well.
\n\n> After each training epoch (including the very first) I predicted the targets, and then averaged these predictions
\n\nAlso something I wanted to try and did not. We definitely do it next time.
\n\nCongrats again on your prize, well deserved.
", "rawMarkdown": "Thanks for sharing.\n\n> capture the monthly dynamics by using sales on the same days of different months as the train targets.\n\nInteresting, I'm not sure we managed to capture the monthly dynamics (payday) that well.\n\n> After each training epoch (including the very first) I predicted the targets, and then averaged these predictions\n\nAlso something I wanted to try and did not. We definitely do it next time.\n\nCongrats again on your prize, well deserved.", "votes": null }, { "id": "270466", "postDate": "01/18/2018 09:25:37", "content": "Creative solutions. Could you share your code? I'm very interesting at your CNN method.
", "rawMarkdown": "Creative solutions. Could you share your code? I'm very interesting at your CNN method.", "votes": null }, { "id": "270510", "postDate": "01/18/2018 11:17:00", "content": "Congratulations and thanks for sharing your ideas. (Tebrikler =))
", "rawMarkdown": "Congratulations and thanks for sharing your ideas. (Tebrikler =))", "votes": null }, { "id": "270669", "postDate": "01/18/2018 17:51:21", "content": "\n\n\nAfter each training epoch (including the very first) I predicted the targets, and then averaged these predictions. I've read about averaging a few last epochs, but never heard about averaging everything, even the first very underfitted epoch. Yet my validation consistently showed that it does improve the results.
\n
I think this is a really big idea, and novel as far as I know.
\n\nI haven't seen this with neural nets, but this is what boosting ensembles are based on. You can bring boosting literature tricks into neural nets when you do this. For example, you benefit from much more epochs, since the overfitting with last epoch models will be counterweighted by the earlier iterations. Even when the final model keeps getting worse, the ensemble over epochs will keep learning. You can also weight each epoch model, and then re-optimize weights for any loss after fitting the model, as is done with Regularized Greedy Forests.
", "rawMarkdown": "> After each training epoch (including the very first) I predicted the targets, and then averaged these predictions. I've read about averaging a few last epochs, but never heard about averaging everything, even the first very underfitted epoch. Yet my validation consistently showed that it does improve the results.\n\nI think this is a really big idea, and novel as far as I know.\n\nI haven't seen this with neural nets, but this is what boosting ensembles are based on. You can bring boosting literature tricks into neural nets when you do this. For example, you benefit from much more epochs, since the overfitting with last epoch models will be counterweighted by the earlier iterations. Even when the final model keeps getting worse, the ensemble over epochs will keep learning. You can also weight each epoch model, and then re-optimize weights for any loss after fitting the model, as is done with Regularized Greedy Forests.", "votes": null }, { "id": "271014", "postDate": "01/19/2018 13:30:25", "content": "Thank you for your interest. Unfortunatelly, at this point I don't know if we will have a legal right to share the code. I'll consult with the organizers about this. In case we will, I will share it and post a link here.
", "rawMarkdown": "Thank you for your interest. Unfortunatelly, at this point I don't know if we will have a legal right to share the code. I'll consult with the organizers about this. In case we will, I will share it and post a link here.", "votes": null }, { "id": "272381", "postDate": "01/23/2018 00:28:47", "content": "Of course.The 1st team has publish their solution in the kernel https://www.kaggle.com/shixw125/1st-place-lgb-model-public-0-506-private-0-511.
", "rawMarkdown": "Of course.The 1st team has publish their solution in the kernel https://www.kaggle.com/shixw125/1st-place-lgb-model-public-0-506-private-0-511.", "votes": null }, { "id": "302253", "postDate": "03/23/2018 20:59:50", "content": "Good job! Thanks for NN architecture idea from Yandex ML training :)
", "rawMarkdown": "Good job! Thanks for NN architecture idea from Yandex ML training :)", "votes": null } ], "comments": [ { "id": 269137, "author_name": "cpmpml", "author_url": "", "post_date": "01/16/2018 09:40:18", "content": "Thanks for sharing.
\n\n> capture the monthly dynamics by using sales on the same days of different months as the train targets.
\n\nInteresting, I'm not sure we managed to capture the monthly dynamics (payday) that well.
\n\n> After each training epoch (including the very first) I predicted the targets, and then averaged these predictions
\n\nAlso something I wanted to try and did not. We definitely do it next time.
\n\nCongrats again on your prize, well deserved.
", "votes": null, "replies": [] }, { "id": 270466, "author_name": "viczyf", "author_url": "", "post_date": "01/18/2018 09:25:37", "content": "Creative solutions. Could you share your code? I'm very interesting at your CNN method.
", "votes": null, "replies": [ { "id": 271014, "author_name": "slonoslon", "author_url": "", "post_date": "01/19/2018 13:30:25", "content": "Thank you for your interest. Unfortunatelly, at this point I don't know if we will have a legal right to share the code. I'll consult with the organizers about this. In case we will, I will share it and post a link here.
", "votes": null, "replies": [] }, { "id": 272381, "author_name": "viczyf", "author_url": "", "post_date": "01/23/2018 00:28:47", "content": "Of course.The 1st team has publish their solution in the kernel https://www.kaggle.com/shixw125/1st-place-lgb-model-public-0-506-private-0-511.
", "votes": null, "replies": [] } ] }, { "id": 270510, "author_name": "sametmarasli", "author_url": "", "post_date": "01/18/2018 11:17:00", "content": "Congratulations and thanks for sharing your ideas. (Tebrikler =))
", "votes": null, "replies": [] }, { "id": 270669, "author_name": "anttip", "author_url": "", "post_date": "01/18/2018 17:51:21", "content": "\n\n\nAfter each training epoch (including the very first) I predicted the targets, and then averaged these predictions. I've read about averaging a few last epochs, but never heard about averaging everything, even the first very underfitted epoch. Yet my validation consistently showed that it does improve the results.
\n
I think this is a really big idea, and novel as far as I know.
\n\nI haven't seen this with neural nets, but this is what boosting ensembles are based on. You can bring boosting literature tricks into neural nets when you do this. For example, you benefit from much more epochs, since the overfitting with last epoch models will be counterweighted by the earlier iterations. Even when the final model keeps getting worse, the ensemble over epochs will keep learning. You can also weight each epoch model, and then re-optimize weights for any loss after fitting the model, as is done with Regularized Greedy Forests.
", "votes": null, "replies": [] }, { "id": 302253, "author_name": "andrrey", "author_url": "", "post_date": "03/23/2018 20:59:50", "content": "Good job! Thanks for NN architecture idea from Yandex ML training :)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "269132": "Cogratulations to the members of #1 and #2 teams, you did a great job here. And my deepest condolences to the teams suffered in this shake-up: I feel for you, guys. \n\nMy teammate, [Ahmet Erdem][1], will hopefully describe his approach in more detail later, when the time will permit, but as a short summary : He used gradient boosting (LGBM) with an original set of features, and tried to capture the monthly dynamics by using sales on the same days of different months as the train targets.\n\nMy part of the final solution consists mostly of neural networks, although I also used a LGBM model with a low weight in the final ensemble.\n\nWhen it comes to time series, I strongly belive that the two most important things are validation and bagging. By correct validation I mean simulating train/test (train/unknown future) split to avoid any kind of future information leaks. Thus, the validation set should include a time period that is not present in train in any way, even with different timeseries, because these time series could correlate with each other. \n\nSo I used the following train/validation split:\nfor the training set, I took a history of 80 consequtive sales days for each (item, store pair), and used the next 16 sales days as a train set target. The split between training history and training target was always from Tuesday to Wednesday, as in the original train/test split, in orded to capture the weekly dynamics.\nAfter training my models on this set and estimating an optimal number of epochs to run, I retrained them on the latest data available (i.e. finals days of the original training set) in order to use the most up-to-date information.\n\nAbout bagging : our final result includes models, trained on 10 runs each. The weights initializations are different each time, so results are different as well, and averaging them helps to improve the solution, particularly when dealing with the uncertain future.\n\nAnother part of my bagging was a bit unusual. After each training epoch (including the very first) I predicted the targets, and then averaged these predictions. I've read about averaging a few last epochs, but never heard about averaging everything, even the first very underfitted epoch. Yet my validation consistently showed that it does improve the results.\n\nAs for the model's achitectures, I belive they are relatively less important here. I had an ensemble of 3 NNs : CNN, LSTM and GRU with almost equal weights. As a single model, GRU shows a bit better results than the rest, but an ensemble is even better, as usual.\n\nAll of them have 3 top (convolutional or recurrent) layers, 2 or 3 bottom fully-connected layers and embedding layers for categorical features. I tried different tweaks like skip-connections, concatenating CNN layers with different convolution kernel sizes and so on, but the results were negligible. The only achitectural tweak that worked better was using dilated convolutions: my CNN uses convolutions with dilation of 2 on the first layer, 8 on the second and 32 on the third (and no pooling). But an original plain CNN without dilated convolutions worked just a bit worse.\n\nWhat did not work: \nThe main disappointment was the poor results of a version with updated promotion information for the train set. I've used a relatively sophisticated method to fill onpromotion status for the dates of the training period without sales, and it showed very promising results on the public leaderboard on the last day of this competition, boosting our score from .501 to .499. And it failed miserably on the private, so we remained in the prize area only because our other submit was more conservative and used less weight for these adjusted promotions. So next time I'll try to be more careful with the last day \"improvements\".\n\n\n\n\n [1]: https://www.kaggle.com/divrikwicky", "269137": "Thanks for sharing.\n\n> capture the monthly dynamics by using sales on the same days of different months as the train targets.\n\nInteresting, I'm not sure we managed to capture the monthly dynamics (payday) that well.\n\n> After each training epoch (including the very first) I predicted the targets, and then averaged these predictions\n\nAlso something I wanted to try and did not. We definitely do it next time.\n\nCongrats again on your prize, well deserved.", "270466": "Creative solutions. Could you share your code? I'm very interesting at your CNN method.", "270510": "Congratulations and thanks for sharing your ideas. (Tebrikler =))", "270669": "> After each training epoch (including the very first) I predicted the targets, and then averaged these predictions. I've read about averaging a few last epochs, but never heard about averaging everything, even the first very underfitted epoch. Yet my validation consistently showed that it does improve the results.\n\nI think this is a really big idea, and novel as far as I know.\n\nI haven't seen this with neural nets, but this is what boosting ensembles are based on. You can bring boosting literature tricks into neural nets when you do this. For example, you benefit from much more epochs, since the overfitting with last epoch models will be counterweighted by the earlier iterations. Even when the final model keeps getting worse, the ensemble over epochs will keep learning. You can also weight each epoch model, and then re-optimize weights for any loss after fitting the model, as is done with Regularized Greedy Forests.", "271014": "Thank you for your interest. Unfortunatelly, at this point I don't know if we will have a legal right to share the code. I'll consult with the organizers about this. In case we will, I will share it and post a link here.", "272381": "Of course.The 1st team has publish their solution in the kernel https://www.kaggle.com/shixw125/1st-place-lgb-model-public-0-506-private-0-511.", "302253": "Good job! Thanks for NN architecture idea from Yandex ML training :)" }, "source": "meta" }