We use data from week 3 - 6 to generate features for week 7, and week 3 - 7 to generate features for week 8 - 9, and then train on week 7 to predict on week 8, and 9. Similarly, we use data from week 3 - 8 to generate features for week 9, and then train xgboost on week 9 to predict week 10 & 11.
\n\nWe have two parts of features:
\n\nShallow features. We calculate max, min, median, mean, and standard deviation of demand on interactions of features. We also include features (Venta_uni_hoy, Venta_hoy, Dev_uni_proxima, Dev_proxima, Demanda_uni_equil) of the last 3 weeks.
Deep features. We train a dozen of linear models to get "deep" interaction signal between features. Vowpal wabbit, FTRL implemented with Pypy, libFM, and libFFM are all used to generate deep features.
Combine these two parts, one single xgboost model is good enough to get the score we have in the public LB.
\n\nPlease comment if you have any questions. And code with detailed doc will be linked here after we get the final approval from kaggle and grupo bimbo. :)
", "messages": [ { "id": "135430", "postDate": "09/14/2016 02:16:10", "content": "We use data from week 3 - 6 to generate features for week 7, and week 3 - 7 to generate features for week 8 - 9, and then train on week 7 to predict on week 8, and 9. Similarly, we use data from week 3 - 8 to generate features for week 9, and then train xgboost on week 9 to predict week 10 & 11.
\n\nWe have two parts of features:
\n\nShallow features. We calculate max, min, median, mean, and standard deviation of demand on interactions of features. We also include features (Venta_uni_hoy, Venta_hoy, Dev_uni_proxima, Dev_proxima, Demanda_uni_equil) of the last 3 weeks.
Deep features. We train a dozen of linear models to get "deep" interaction signal between features. Vowpal wabbit, FTRL implemented with Pypy, libFM, and libFFM are all used to generate deep features.
Combine these two parts, one single xgboost model is good enough to get the score we have in the public LB.
\n\nPlease comment if you have any questions. And code with detailed doc will be linked here after we get the final approval from kaggle and grupo bimbo. :)
", "rawMarkdown": "We use data from week 3 - 6 to generate features for week 7, and week 3 - 7 to generate features for week 8 - 9, and then train on week 7 to predict on week 8, and 9. Similarly, we use data from week 3 - 8 to generate features for week 9, and then train xgboost on week 9 to predict week 10 & 11.\r\n\r\nWe have two parts of features:\r\n\r\n1. Shallow features. We calculate max, min, median, mean, and standard deviation of demand on interactions of features. We also include features (Venta_uni_hoy, Venta_hoy, Dev_uni_proxima, Dev_proxima, Demanda_uni_equil) of the last 3 weeks.\r\n\r\n2. Deep features. We train a dozen of linear models to get \"deep\" interaction signal between features. Vowpal wabbit, FTRL implemented with Pypy, libFM, and libFFM are all used to generate deep features.\r\n\r\nCombine these two parts, one single xgboost model is good enough to get the score we have in the public LB.\r\n\r\nPlease comment if you have any questions. And code with detailed doc will be linked here after we get the final approval from kaggle and grupo bimbo. :)", "votes": null }, { "id": "135445", "postDate": "09/14/2016 06:26:48", "content": "Hi, my training runs took so very long that once I realized that there were gains to be made to make predictions using previous predictions, I gave up on trying because it would take too much time. It would be interesting to know how effective that actually was.
\n\nRe 1) I was using shallow features similar to your implementation, but I didn't use max/min/std dev, that's a nice trick that I'll use in the future.
\n\nRe 2) What features did you use for this?
\n\nThanks for sharing!
\n\n/m
", "rawMarkdown": "Hi, my training runs took so very long that once I realized that there were gains to be made to make predictions using previous predictions, I gave up on trying because it would take too much time. It would be interesting to know how effective that actually was.\r\n\r\nRe 1) I was using shallow features similar to your implementation, but I didn't use max/min/std dev, that's a nice trick that I'll use in the future.\r\n\r\nRe 2) What features did you use for this?\r\n\r\nThanks for sharing!\r\n\r\n/m", "votes": null }, { "id": "135447", "postDate": "09/14/2016 07:11:33", "content": "I tried to combine some of the shallow features you mentioned with libFM using xgboost but my CV scores always got worse than libFM alone. Could you share the parameters of your xgboost model and how important the shallow features are? Thanks!
", "rawMarkdown": "I tried to combine some of the shallow features you mentioned with libFM using xgboost but my CV scores always got worse than libFM alone. Could you share the parameters of your xgboost model and how important the shallow features are? Thanks!", "votes": null }, { "id": "135496", "postDate": "09/14/2016 15:04:59", "content": "[quote=Mattias Fagerlund;135445]
\n\nHi, my training runs took so very long that once I realized that there were gains to be made to make predictions using previous predictions, I gave up on trying because it would take too much time. It would be interesting to know how effective that actually was.
\n\nRe 1) I was using shallow features similar to your implementation, but I didn't use max/min/std dev, that's a nice trick that I'll use in the future.
\n\nRe 2) What features did you use for this?
\n\nThanks for sharing!
\n\n/m
\n\n[/quote]
\n\nJust the predicted result from those linear models. They actually didn't take much time to generate but were the most important features.
", "rawMarkdown": "[quote=Mattias Fagerlund;135445]\r\n\r\nHi, my training runs took so very long that once I realized that there were gains to be made to make predictions using previous predictions, I gave up on trying because it would take too much time. It would be interesting to know how effective that actually was.\r\n\r\nRe 1) I was using shallow features similar to your implementation, but I didn't use max/min/std dev, that's a nice trick that I'll use in the future.\r\n\r\nRe 2) What features did you use for this?\r\n\r\nThanks for sharing!\r\n\r\n/m\r\n\r\n[/quote]\r\n\r\nJust the predicted result from those linear models. They actually didn't take much time to generate but were the most important features.", "votes": null }, { "id": "135497", "postDate": "09/14/2016 15:05:57", "content": "[quote=Sangxia;135447]
\n\nI tried to combine some of the shallow features you mentioned with libFM using xgboost but my CV scores always got worse than libFM alone. Could you share the parameters of your xgboost model and how important the shallow features are? Thanks!
\n\n[/quote]
\n\nWere you doing the same validation as we did? If that is the case, it seemed to improve our score with any libFM models and we didn't have to tune xgboost at all to get better scores.
", "rawMarkdown": "[quote=Sangxia;135447]\r\n\r\nI tried to combine some of the shallow features you mentioned with libFM using xgboost but my CV scores always got worse than libFM alone. Could you share the parameters of your xgboost model and how important the shallow features are? Thanks!\r\n\r\n[/quote]\r\n\r\nWere you doing the same validation as we did? If that is the case, it seemed to improve our score with any libFM models and we didn't have to tune xgboost at all to get better scores.", "votes": null }, { "id": "135506", "postDate": "09/14/2016 16:22:38", "content": "Thank you for an exceptionally concise write-up.
", "rawMarkdown": "Thank you for an exceptionally concise write-up.", "votes": null }, { "id": "135517", "postDate": "09/14/2016 17:02:57", "content": "@Little Boat
\n\nPlease, forgive my "naiveness" but can you please help me understand your point (2), aka, Deep Features?\nI would like to understand the theory behind it and how you did it. Because obviously these deep features was the reason of your success.
\n\nThank you in advanced for the teaching ;)
", "rawMarkdown": "Little Boat\r\n\r\nPlease, forgive my \"naiveness\" but can you please help me understand your point (2), aka, Deep Features?\r\nI would like to understand the theory behind it and how you did it. Because obviously these deep features was the reason of your success.\r\n\r\nThank you in advanced for the teaching ;)", "votes": null }, { "id": "135589", "postDate": "09/15/2016 04:50:16", "content": "[quote=PabloMarin;135517]
\n\n@Little Boat
\n\nPlease, forgive my "naiveness" but can you please help me understand your point (2), aka, Deep Features?\nI would like to understand the theory behind it and how you did it. Because obviously these deep features was the reason of your success.
\n\nThank you in advanced for the teaching ;)
\n\n[/quote]
\n\nBriefly, you can think of the shallow features as simple descriptive statistics while the deep features are linear interaction between all features. Shallow features describe the rough "shape" of those interactions, for example, for the same pair of product and client, they will all have the same values, while deep features model all interactions and give predictions based on all features. You can think shallow feature as "big picture" while deep features are "detail oriented".
", "rawMarkdown": "[quote=PabloMarin;135517]\r\n\r\n@Little Boat\r\n\r\nPlease, forgive my \"naiveness\" but can you please help me understand your point (2), aka, Deep Features?\r\nI would like to understand the theory behind it and how you did it. Because obviously these deep features was the reason of your success.\r\n\r\nThank you in advanced for the teaching ;)\r\n\r\n[/quote]\r\n\r\nBriefly, you can think of the shallow features as simple descriptive statistics while the deep features are linear interaction between all features. Shallow features describe the rough \"shape\" of those interactions, for example, for the same pair of product and client, they will all have the same values, while deep features model all interactions and give predictions based on all features. You can think shallow feature as \"big picture\" while deep features are \"detail oriented\".", "votes": null }, { "id": "136077", "postDate": "09/19/2016 22:49:22", "content": "@Little Boat, thank you very much for sharing, appreciate it! Congratulations.
\n\nI can understand LibFM to be used in here, but for field-aware LibFFM = I'm confused how to build the data set ;)
\n\n\n\n\nAnd code with detailed doc will be linked here after we get the final approval from kaggle and grupo bimbo. :)
\n
Hopefully.. I really want to learn LibFFM. Thank you in advance. (+100 if I can).
", "rawMarkdown": "Little Boat, thank you very much for sharing, appreciate it! Congratulations. \r\n\r\nI can understand LibFM to be used in here, but for field-aware LibFFM = I'm confused how to build the data set ;) \r\n\r\n> *And code with detailed doc will be linked here after we get the final approval from kaggle and grupo bimbo*. :)\r\n\r\nHopefully.. I really want to learn LibFFM. Thank you in advance. (+100 if I can).", "votes": null }, { "id": "136326", "postDate": "09/21/2016 10:07:35", "content": "Thanks a lot for sharing.
\n\nI went somewhat same way as you, but only used one linear model (vw) and the gain wasn't really significant. I'll try harder next time!
", "rawMarkdown": "Thanks a lot for sharing.\r\n\r\nI went somewhat same way as you, but only used one linear model (vw) and the gain wasn't really significant. I'll try harder next time!", "votes": null } ], "comments": [ { "id": 135445, "author_name": "mfagerlund", "author_url": "", "post_date": "09/14/2016 06:26:48", "content": "Hi, my training runs took so very long that once I realized that there were gains to be made to make predictions using previous predictions, I gave up on trying because it would take too much time. It would be interesting to know how effective that actually was.
\n\nRe 1) I was using shallow features similar to your implementation, but I didn't use max/min/std dev, that's a nice trick that I'll use in the future.
\n\nRe 2) What features did you use for this?
\n\nThanks for sharing!
\n\n/m
", "votes": null, "replies": [] }, { "id": 135447, "author_name": "sangxia", "author_url": "", "post_date": "09/14/2016 07:11:33", "content": "I tried to combine some of the shallow features you mentioned with libFM using xgboost but my CV scores always got worse than libFM alone. Could you share the parameters of your xgboost model and how important the shallow features are? Thanks!
", "votes": null, "replies": [] }, { "id": 135496, "author_name": "xiaozhouwang", "author_url": "", "post_date": "09/14/2016 15:04:59", "content": "[quote=Mattias Fagerlund;135445]
\n\nHi, my training runs took so very long that once I realized that there were gains to be made to make predictions using previous predictions, I gave up on trying because it would take too much time. It would be interesting to know how effective that actually was.
\n\nRe 1) I was using shallow features similar to your implementation, but I didn't use max/min/std dev, that's a nice trick that I'll use in the future.
\n\nRe 2) What features did you use for this?
\n\nThanks for sharing!
\n\n/m
\n\n[/quote]
\n\nJust the predicted result from those linear models. They actually didn't take much time to generate but were the most important features.
", "votes": null, "replies": [] }, { "id": 135497, "author_name": "xiaozhouwang", "author_url": "", "post_date": "09/14/2016 15:05:57", "content": "[quote=Sangxia;135447]
\n\nI tried to combine some of the shallow features you mentioned with libFM using xgboost but my CV scores always got worse than libFM alone. Could you share the parameters of your xgboost model and how important the shallow features are? Thanks!
\n\n[/quote]
\n\nWere you doing the same validation as we did? If that is the case, it seemed to improve our score with any libFM models and we didn't have to tune xgboost at all to get better scores.
", "votes": null, "replies": [] }, { "id": 135506, "author_name": "wpanak", "author_url": "", "post_date": "09/14/2016 16:22:38", "content": "Thank you for an exceptionally concise write-up.
", "votes": null, "replies": [] }, { "id": 135517, "author_name": "pablomarin", "author_url": "", "post_date": "09/14/2016 17:02:57", "content": "@Little Boat
\n\nPlease, forgive my "naiveness" but can you please help me understand your point (2), aka, Deep Features?\nI would like to understand the theory behind it and how you did it. Because obviously these deep features was the reason of your success.
\n\nThank you in advanced for the teaching ;)
", "votes": null, "replies": [] }, { "id": 135589, "author_name": "xiaozhouwang", "author_url": "", "post_date": "09/15/2016 04:50:16", "content": "[quote=PabloMarin;135517]
\n\n@Little Boat
\n\nPlease, forgive my "naiveness" but can you please help me understand your point (2), aka, Deep Features?\nI would like to understand the theory behind it and how you did it. Because obviously these deep features was the reason of your success.
\n\nThank you in advanced for the teaching ;)
\n\n[/quote]
\n\nBriefly, you can think of the shallow features as simple descriptive statistics while the deep features are linear interaction between all features. Shallow features describe the rough "shape" of those interactions, for example, for the same pair of product and client, they will all have the same values, while deep features model all interactions and give predictions based on all features. You can think shallow feature as "big picture" while deep features are "detail oriented".
", "votes": null, "replies": [] }, { "id": 136077, "author_name": "aldente", "author_url": "", "post_date": "09/19/2016 22:49:22", "content": "@Little Boat, thank you very much for sharing, appreciate it! Congratulations.
\n\nI can understand LibFM to be used in here, but for field-aware LibFFM = I'm confused how to build the data set ;)
\n\n\n\n\nAnd code with detailed doc will be linked here after we get the final approval from kaggle and grupo bimbo. :)
\n
Hopefully.. I really want to learn LibFFM. Thank you in advance. (+100 if I can).
", "votes": null, "replies": [] }, { "id": 136326, "author_name": "cpmpml", "author_url": "", "post_date": "09/21/2016 10:07:35", "content": "Thanks a lot for sharing.
\n\nI went somewhat same way as you, but only used one linear model (vw) and the gain wasn't really significant. I'll try harder next time!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "135430": "We use data from week 3 - 6 to generate features for week 7, and week 3 - 7 to generate features for week 8 - 9, and then train on week 7 to predict on week 8, and 9. Similarly, we use data from week 3 - 8 to generate features for week 9, and then train xgboost on week 9 to predict week 10 & 11.\r\n\r\nWe have two parts of features:\r\n\r\n1. Shallow features. We calculate max, min, median, mean, and standard deviation of demand on interactions of features. We also include features (Venta_uni_hoy, Venta_hoy, Dev_uni_proxima, Dev_proxima, Demanda_uni_equil) of the last 3 weeks.\r\n\r\n2. Deep features. We train a dozen of linear models to get \"deep\" interaction signal between features. Vowpal wabbit, FTRL implemented with Pypy, libFM, and libFFM are all used to generate deep features.\r\n\r\nCombine these two parts, one single xgboost model is good enough to get the score we have in the public LB.\r\n\r\nPlease comment if you have any questions. And code with detailed doc will be linked here after we get the final approval from kaggle and grupo bimbo. :)", "135445": "Hi, my training runs took so very long that once I realized that there were gains to be made to make predictions using previous predictions, I gave up on trying because it would take too much time. It would be interesting to know how effective that actually was.\r\n\r\nRe 1) I was using shallow features similar to your implementation, but I didn't use max/min/std dev, that's a nice trick that I'll use in the future.\r\n\r\nRe 2) What features did you use for this?\r\n\r\nThanks for sharing!\r\n\r\n/m", "135447": "I tried to combine some of the shallow features you mentioned with libFM using xgboost but my CV scores always got worse than libFM alone. Could you share the parameters of your xgboost model and how important the shallow features are? Thanks!", "135496": "[quote=Mattias Fagerlund;135445]\r\n\r\nHi, my training runs took so very long that once I realized that there were gains to be made to make predictions using previous predictions, I gave up on trying because it would take too much time. It would be interesting to know how effective that actually was.\r\n\r\nRe 1) I was using shallow features similar to your implementation, but I didn't use max/min/std dev, that's a nice trick that I'll use in the future.\r\n\r\nRe 2) What features did you use for this?\r\n\r\nThanks for sharing!\r\n\r\n/m\r\n\r\n[/quote]\r\n\r\nJust the predicted result from those linear models. They actually didn't take much time to generate but were the most important features.", "135497": "[quote=Sangxia;135447]\r\n\r\nI tried to combine some of the shallow features you mentioned with libFM using xgboost but my CV scores always got worse than libFM alone. Could you share the parameters of your xgboost model and how important the shallow features are? Thanks!\r\n\r\n[/quote]\r\n\r\nWere you doing the same validation as we did? If that is the case, it seemed to improve our score with any libFM models and we didn't have to tune xgboost at all to get better scores.", "135506": "Thank you for an exceptionally concise write-up.", "135517": "Little Boat\r\n\r\nPlease, forgive my \"naiveness\" but can you please help me understand your point (2), aka, Deep Features?\r\nI would like to understand the theory behind it and how you did it. Because obviously these deep features was the reason of your success.\r\n\r\nThank you in advanced for the teaching ;)", "135589": "[quote=PabloMarin;135517]\r\n\r\n@Little Boat\r\n\r\nPlease, forgive my \"naiveness\" but can you please help me understand your point (2), aka, Deep Features?\r\nI would like to understand the theory behind it and how you did it. Because obviously these deep features was the reason of your success.\r\n\r\nThank you in advanced for the teaching ;)\r\n\r\n[/quote]\r\n\r\nBriefly, you can think of the shallow features as simple descriptive statistics while the deep features are linear interaction between all features. Shallow features describe the rough \"shape\" of those interactions, for example, for the same pair of product and client, they will all have the same values, while deep features model all interactions and give predictions based on all features. You can think shallow feature as \"big picture\" while deep features are \"detail oriented\".", "136077": "Little Boat, thank you very much for sharing, appreciate it! Congratulations. \r\n\r\nI can understand LibFM to be used in here, but for field-aware LibFFM = I'm confused how to build the data set ;) \r\n\r\n> *And code with detailed doc will be linked here after we get the final approval from kaggle and grupo bimbo*. :)\r\n\r\nHopefully.. I really want to learn LibFFM. Thank you in advance. (+100 if I can).", "136326": "Thanks a lot for sharing.\r\n\r\nI went somewhat same way as you, but only used one linear model (vw) and the gain wasn't really significant. I'll try harder next time!" }, "source": "existing_discussion", "complete": true, "competition_id": 5260, "competition_slug": "grupo-bimbo-inventory-demand", "team_id": 341928, "solution_url": "/competitions/grupo-bimbo-inventory-demand/writeups/team-mystic-a-quick-summary-of-team-mystic-3rd-pla" }