Congratulation to winners! That was a challenging competition given data set size. Like many, I am waiting for some info on the approach used by top teams.
\n\nIn the meantime, here is some info on my solution (18th on final LB).
\n\nIn order to deal with data size I decided to build one model per product id. I knew this would leave money on the table, but it was rather effective with a 16 GB machine (macbook pro). Did anyone else use the same trick?
\n\nCV setting is time based. Use week 9 as validation set, and train models on weeks 6-8. Local CV is well correlated with the public LB, with an almost constant offset.
\n\nFeatures are lag, averages on various groupings. For instance, group by agencia and route, then average sales and returns. In some cases the number of non zero values, and the std is alos computed. Then join this with the training data.
\n\nMore precisely, for the first half of competition, I used previous 4 weeks to build features. I used weeks 3-5 to build features for week 6, weeks 3-6 to build features for week 7, weeks 4-7 to build features for week 8, weeks 5-8 to build features for week 9, and weeks 6-9 to build features for test. Missing features in weeks 6 are treated as NaN.
\n\nFor CV, models were trained on weeks 6-8 with the lag features described above, and validated on week 9 with its lag features. For test predictions, weeks 7-9 with their lag features were used.\nThis led me to 7th place at some point. I wish competition ended then ;)
\n\nI then switched gear, like many, to focus on 2 weeks lag predictions; CV setting is almost the same, but one week is omitted. For instance weeks 4-6 are used to build features for week 8, week 7 is omitted. The rest is unchanged.
\n\nWhat we lose with decomposition is cross product interactions. Two ways were used to cope with that.
\n\nFirst, features based on groupings across products were capturing some interactions.
\n\nSecond, in a last pass, all products with poor local CV values were grouped to form a much smaller data set (about 2.5M entries instead of 75M). Then I could train xgboost on it. This was rather effective when predicting one week ahead, but I could not get it to work well when predicting 2 weeks ahead because of lack of time.
\n\nI did not use ensembling, also because of lack of time.
\n\nTime is the main issue with this approach. Local CV plus submission preparation is taking between 24 and 60 hours as there are almost 1500 models to build. This is the classical time vs memory tradeoff. However, it could scale very well on a cluster as each model can be trained concurrently with others. I will definitely look into a cluster next time!
\n\nThe biggest surprise to me is that the one week ahead model did as well as a the two week ahead model on private LB . Probably because my last pass was effective with the former.
\n\nI did all in Python, with numpy, pandas, scikit-learn and xgboost mainly.
\n\nHope the above is of interest to readers.
", "messages": [ { "id": "133060", "postDate": "08/31/2016 05:10:39", "content": "Congratulation to winners! That was a challenging competition given data set size. Like many, I am waiting for some info on the approach used by top teams.
\n\nIn the meantime, here is some info on my solution (18th on final LB).
\n\nIn order to deal with data size I decided to build one model per product id. I knew this would leave money on the table, but it was rather effective with a 16 GB machine (macbook pro). Did anyone else use the same trick?
\n\nCV setting is time based. Use week 9 as validation set, and train models on weeks 6-8. Local CV is well correlated with the public LB, with an almost constant offset.
\n\nFeatures are lag, averages on various groupings. For instance, group by agencia and route, then average sales and returns. In some cases the number of non zero values, and the std is alos computed. Then join this with the training data.
\n\nMore precisely, for the first half of competition, I used previous 4 weeks to build features. I used weeks 3-5 to build features for week 6, weeks 3-6 to build features for week 7, weeks 4-7 to build features for week 8, weeks 5-8 to build features for week 9, and weeks 6-9 to build features for test. Missing features in weeks 6 are treated as NaN.
\n\nFor CV, models were trained on weeks 6-8 with the lag features described above, and validated on week 9 with its lag features. For test predictions, weeks 7-9 with their lag features were used.\nThis led me to 7th place at some point. I wish competition ended then ;)
\n\nI then switched gear, like many, to focus on 2 weeks lag predictions; CV setting is almost the same, but one week is omitted. For instance weeks 4-6 are used to build features for week 8, week 7 is omitted. The rest is unchanged.
\n\nWhat we lose with decomposition is cross product interactions. Two ways were used to cope with that.
\n\nFirst, features based on groupings across products were capturing some interactions.
\n\nSecond, in a last pass, all products with poor local CV values were grouped to form a much smaller data set (about 2.5M entries instead of 75M). Then I could train xgboost on it. This was rather effective when predicting one week ahead, but I could not get it to work well when predicting 2 weeks ahead because of lack of time.
\n\nI did not use ensembling, also because of lack of time.
\n\nTime is the main issue with this approach. Local CV plus submission preparation is taking between 24 and 60 hours as there are almost 1500 models to build. This is the classical time vs memory tradeoff. However, it could scale very well on a cluster as each model can be trained concurrently with others. I will definitely look into a cluster next time!
\n\nThe biggest surprise to me is that the one week ahead model did as well as a the two week ahead model on private LB . Probably because my last pass was effective with the former.
\n\nI did all in Python, with numpy, pandas, scikit-learn and xgboost mainly.
\n\nHope the above is of interest to readers.
", "rawMarkdown": "Congratulation to winners! That was a challenging competition given data set size. Like many, I am waiting for some info on the approach used by top teams.\r\n\r\nIn the meantime, here is some info on my solution (18th on final LB).\r\n\r\nIn order to deal with data size I decided to build one model per product id. I knew this would leave money on the table, but it was rather effective with a 16 GB machine (macbook pro). Did anyone else use the same trick?\r\n\r\nCV setting is time based. Use week 9 as validation set, and train models on weeks 6-8. Local CV is well correlated with the public LB, with an almost constant offset.\r\n\r\nFeatures are lag, averages on various groupings. For instance, group by agencia and route, then average sales and returns. In some cases the number of non zero values, and the std is alos computed. Then join this with the training data. \r\n\r\nMore precisely, for the first half of competition, I used previous 4 weeks to build features. I used weeks 3-5 to build features for week 6, weeks 3-6 to build features for week 7, weeks 4-7 to build features for week 8, weeks 5-8 to build features for week 9, and weeks 6-9 to build features for test. Missing features in weeks 6 are treated as NaN. \r\n \r\nFor CV, models were trained on weeks 6-8 with the lag features described above, and validated on week 9 with its lag features. For test predictions, weeks 7-9 with their lag features were used.\r\nThis led me to 7th place at some point. I wish competition ended then ;)\r\n\r\nI then switched gear, like many, to focus on 2 weeks lag predictions; CV setting is almost the same, but one week is omitted. For instance weeks 4-6 are used to build features for week 8, week 7 is omitted. The rest is unchanged.\r\n\r\nWhat we lose with decomposition is cross product interactions. Two ways were used to cope with that. \r\n\r\nFirst, features based on groupings across products were capturing some interactions.\r\n\r\nSecond, in a last pass, all products with poor local CV values were grouped to form a much smaller data set (about 2.5M entries instead of 75M). Then I could train xgboost on it. This was rather effective when predicting one week ahead, but I could not get it to work well when predicting 2 weeks ahead because of lack of time.\r\n\r\nI did not use ensembling, also because of lack of time. \r\n\r\nTime is the main issue with this approach. Local CV plus submission preparation is taking between 24 and 60 hours as there are almost 1500 models to build. This is the classical time vs memory tradeoff. However, it could scale very well on a cluster as each model can be trained concurrently with others. I will definitely look into a cluster next time!\r\n\r\nThe biggest surprise to me is that the one week ahead model did as well as a the two week ahead model on private LB . Probably because my last pass was effective with the former.\r\n\r\nI did all in Python, with numpy, pandas, scikit-learn and xgboost mainly. \r\n\r\nHope the above is of interest to readers.", "votes": null }, { "id": "133064", "postDate": "08/31/2016 05:53:35", "content": "That's an elegant solution in terms of memory efficiency! I want to contribute a little to this topic too.
\n\nI took each float feature, multiplied it by 100 and took int8/ int16 of resulting value. Therefore I lowered memory usage by approximately 4 times. I was able to fit models on my macbook with 8GB (and ssd swap ;) ) on whole weeks with 65-70 features each.
\n\nAlso this trick improved my validation and public LB scores. This transformation cut the number of unique values for each feature from tens of thousands to several hundreds. Therefore my xgboost trees was not able to split precisely in attempts to match the noise in data. Overall it led to more generalizing model (70 on public and 56 on private). Obviously, this can be achieved by xgboost parameters optimization, however such optimization will be time consuming.
\n\nI hope some of you will find my observation interesting!
", "rawMarkdown": "That's an elegant solution in terms of memory efficiency! I want to contribute a little to this topic too. \r\n\r\nI took each float feature, multiplied it by 100 and took int8/ int16 of resulting value. Therefore I lowered memory usage by approximately 4 times. I was able to fit models on my macbook with 8GB (and ssd swap ;) ) on whole weeks with 65-70 features each.\r\n\r\nAlso this trick improved my validation and public LB scores. This transformation cut the number of unique values for each feature from tens of thousands to several hundreds. Therefore my xgboost trees was not able to split precisely in attempts to match the noise in data. Overall it led to more generalizing model (70 on public and 56 on private). Obviously, this can be achieved by xgboost parameters optimization, however such optimization will be time consuming. \r\n\r\nI hope some of you will find my observation interesting!", "votes": null }, { "id": "133065", "postDate": "08/31/2016 05:58:08", "content": "Denis, thanks.
\n\nxgboost has an option to use an approximate algorithm that does what you do more or less. If I remember well it bins large cardinality features using 300 bins. But doing so yourself is definitely more memory efficient.
", "rawMarkdown": "Denis, thanks.\r\n\r\nxgboost has an option to use an approximate algorithm that does what you do more or less. If I remember well it bins large cardinality features using 300 bins. But doing so yourself is definitely more memory efficient.", "votes": null }, { "id": "133091", "postDate": "08/31/2016 07:44:23", "content": "Congratulations @CPMP, @Denis! I hope people with low RAM would like to share their approaches as well. Thanks ;)
\n\nMy computer has 2x4GB = 8GB RAM (replaced 2GB with 4GB for this competition, previously 6GB), no SSD, no swap. Data preparation was done in R (data.table), saved all ready-for-use data into harddisk, then XGB fitting in Python. The model has 2 levels. My result:
\n\nPublic : 0.43959 (53rd, after LB cleaning = 41st)
Private: 0.45462 (31st, after LB cleaning = 24th)
Level 1: FTRL with Vowpal Wabbit, to use all weeks 3..9, to be used as meta-feature (vw_ftrl) in level 2. (similar to ML-Bure's 12th feature.)
\n\nLevel 2, ensemble of two groups:
\n\nXGB models, 16 features (similar to ML-Bure's features + _IDs), @2 weeks: [4,9], [5,9], [6,9], [7,9],\n[8,9]
XGB models with less features, @3 weeks: [6,7,9], [6,8,9],\n[7,8,9]
Strongest features are: vw_ftrl, lag1, lag2, lag3.
\n\nI also tried all other week pairs, but public LB scores were not good, so I didn't use them in ensemble.
\n\nImportant: in the usage of lagged demand features, I consider whole test set as week 10. So, week 10's previous week uses week 9 as lag1, the same as week 11's previous week also uses week 9 as lag1. The same as lag2 (to use week 8 for both weeks), lag3 (to use week 7 for both weeks).
\n\nI tried also another version by considering test set to have 2 different weeks (10 and 11), but the result is worse in private LB (I've checked it out by doing re-submission).
\n\nPossible improvement: I believe more XGB models in the ensemble will improve the score. E.g. by combining 50% of each week, etc. And, maybe to use only one week (9) for the fitting as done by ML-Bure.
\n\nThanks to Scirpus for FTRL idea, Kes367 et.al. for lagged demand idea, CPMP for 32 bit idea, all script writers, all forum contributors. Kaggle (Wendy et.al.), Grupo Bimbo, all Kagglers... ;)
\n\nIn the last week, I tried to fit whole weeks [3..9] in Turi GraphLab (academic license, now acquired by Apple), with the boosted_trees_regression (use same features). The fitting took 4+ days, epoch=120, consumed ~6GB RAM, the result is.. worse than single XGB model on 2 weeks [8,9] taking only 2 hours.
\n\nAlso tried to use Apache Spark, successfully installed in Windows machine, but not the XGBoost4J (always getting error).
", "rawMarkdown": "Congratulations @CPMP, @Denis! I hope people with low RAM would like to share their approaches as well. Thanks ;) \r\n\r\nMy computer has 2x4GB = 8GB RAM (replaced 2GB with 4GB for this competition, previously 6GB), no SSD, no swap. Data preparation was done in R (data.table), saved all ready-for-use data into harddisk, then XGB fitting in Python. The model has 2 levels. My result: \r\n\r\n - Public : 0.43959 (53rd, after LB cleaning = 41st)\r\n \r\n - Private: 0.45462 (31st, after LB cleaning = 24th)\r\n\r\nLevel 1: FTRL with Vowpal Wabbit, to use all weeks 3..9, to be used as meta-feature (vw_ftrl) in level 2. (similar to [ML-Bure][1]'s 12th feature.) \r\n\r\nLevel 2, ensemble of two groups: \r\n\r\n 1. XGB models, 16 features (similar to ML-Bure's features + _IDs), @2 weeks: [4,9], [5,9], [6,9], [7,9],\r\n [8,9] \r\n \r\n 2. XGB models with less features, @3 weeks: [6,7,9], [6,8,9],\r\n [7,8,9]\r\n\r\nStrongest features are: vw_ftrl, lag1, lag2, lag3. \r\n\r\nI also tried all other week pairs, but public LB scores were not good, so I didn't use them in ensemble. \r\n\r\nImportant: in the usage of lagged demand features, I consider whole test set as week 10. So, week 10's previous week uses week 9 as lag1, the same as week 11's previous week also uses week 9 as lag1. The same as lag2 (to use week 8 for both weeks), lag3 (to use week 7 for both weeks). \r\n\r\nI tried also another version by considering test set to have 2 different weeks (10 and 11), but the result is worse in private LB (I've checked it out by doing re-submission). \r\n\r\nPossible improvement: I believe more XGB models in the ensemble will improve the score. E.g. by combining 50% of each week, etc. And, maybe to use only one week (9) for the fitting as done by [ML-Bure][2]. \r\n\r\nThanks to Scirpus for FTRL idea, [Kes367][3] et.al. for lagged demand idea, CPMP for 32 bit idea, all script writers, all forum contributors. Kaggle (Wendy et.al.), Grupo Bimbo, all Kagglers... ;) \r\n\r\nIn the last week, I tried to fit whole weeks [3..9] in Turi GraphLab (academic license, now acquired by Apple), with the boosted_trees_regression (use same features). The fitting took 4+ days, epoch=120, consumed ~6GB RAM, the result is.. worse than single XGB model on 2 weeks [8,9] taking only 2 hours. \r\n\r\nAlso tried to use Apache Spark, successfully installed in Windows machine, but not the [XGBoost4J][4] (always getting error). \r\n\r\n\r\n [1]: https://www.kaggle.com/c/grupo-bimbo-inventory-demand/forums/t/23208/19-place-solution\r\n [2]: https://www.kaggle.com/c/grupo-bimbo-inventory-demand/forums/t/23208/19-place-solution\r\n [3]: http://blog.nycdatascience.com/student-works/predicting-demand-historical-sales-data-grupo-bimbo-kaggle-competition/\r\n [4]: http://dmlc.ml/2016/03/14/xgboost4j-portable-distributed-xgboost-in-spark-flink-and-dataflow.html", "votes": null }, { "id": "133153", "postDate": "08/31/2016 16:26:49", "content": "Congrats Aldente!
\n\nI came 89th using just ftlr + isotonic regression based on (week - 1) and (week - 2) predictions - some times old timer techniques can come good ;) This competition finally made me buy some extra ram to go from 6gb to 24 gb. The main reason is that I am spending more and more time coming up with a solution that fits into 6 gig. Truncated mantissas on floats to save 4 bits really helped ;(
", "rawMarkdown": "Congrats Aldente!\r\n\r\nI came 89th using just ftlr + isotonic regression based on (week - 1) and (week - 2) predictions - some times old timer techniques can come good ;) This competition finally made me buy some extra ram to go from 6gb to 24 gb. The main reason is that I am spending more and more time coming up with a solution that fits into 6 gig. Truncated mantissas on floats to save 4 bits really helped ;(", "votes": null }, { "id": "133181", "postDate": "08/31/2016 20:30:00", "content": "@Scirpus, FTRL rocks! I learnt new thing. I also tried your isotonic-regression C++ code, but getting all zero results after modification as I had difficulty to compile it in Windows OS (Visual Studio). Then I tried sklearn's IsotonicRegression (took ~5+ hours to run), but getting slightly worse score. 24GB is big, we'll lose you in the Kernels ;))
", "rawMarkdown": "Scirpus, FTRL rocks! I learnt new thing. I also tried your isotonic-regression C++ code, but getting all zero results after modification as I had difficulty to compile it in Windows OS (Visual Studio). Then I tried sklearn's IsotonicRegression (took ~5+ hours to run), but getting slightly worse score. 24GB is big, we'll lose you in the Kernels ;))", "votes": null }, { "id": "133190", "postDate": "08/31/2016 21:14:42", "content": "Thanks\n24 gig means that I can do 4 times as many kernels ;)
", "rawMarkdown": "Thanks\r\n24 gig means that I can do 4 times as many kernels ;)", "votes": null }, { "id": "133226", "postDate": "09/01/2016 00:38:23", "content": "I hadn't thought of doing integer conversions - and nice pointer CPMP - manually doing that worked well for me on Springleaf but forgot about it on this one.
\n\nAlso under Linux zram is quite useful - it's still a performance hit, but it compresses well allowing 2-3x as much stuff in memory.
", "rawMarkdown": "I hadn't thought of doing integer conversions - and nice pointer CPMP - manually doing that worked well for me on Springleaf but forgot about it on this one.\r\n\r\nAlso under Linux zram is quite useful - it's still a performance hit, but it compresses well allowing 2-3x as much stuff in memory.", "votes": null }, { "id": "133288", "postDate": "09/01/2016 10:48:12", "content": "I also made models by products, but only for top 50 products and these predictions I avereged with full model.
\n\nBefore that I made some processing with products' names - I removed some additional suffixes before brand (I guess that information about producer for Bimbo, not for clients), corrected some name mistakes.\nThis processing reduced number of unique products by half. And after that top 50 products had near 90% share of test set.
\n\nI started this competition too late in August and hadn't enough time to tune features and parameters for each product's model (I'm sure it makes sense). But even with common set of features, products' models led to visible improvement of base model (~ 0.003).
", "rawMarkdown": "I also made models by products, but only for top 50 products and these predictions I avereged with full model. \r\n\r\nBefore that I made some processing with products' names - I removed some additional suffixes before brand (I guess that information about producer for Bimbo, not for clients), corrected some name mistakes.\r\nThis processing reduced number of unique products by half. And after that top 50 products had near 90% share of test set.\r\n\r\nI started this competition too late in August and hadn't enough time to tune features and parameters for each product's model (I'm sure it makes sense). But even with common set of features, products' models led to visible improvement of base model (~ 0.003).", "votes": null }, { "id": "135126", "postDate": "09/11/2016 10:13:38", "content": "@CPMP
\n\nDid you use the same xgboost hyperparameters for all models? or did you do a grid search for each model? approx 1500 grid searches?
", "rawMarkdown": "CPMP\r\n\r\nDid you use the same xgboost hyperparameters for all models? or did you do a grid search for each model? approx 1500 grid searches?", "votes": null } ], "comments": [ { "id": 133064, "author_name": "curlyb", "author_url": "", "post_date": "08/31/2016 05:53:35", "content": "That's an elegant solution in terms of memory efficiency! I want to contribute a little to this topic too.
\n\nI took each float feature, multiplied it by 100 and took int8/ int16 of resulting value. Therefore I lowered memory usage by approximately 4 times. I was able to fit models on my macbook with 8GB (and ssd swap ;) ) on whole weeks with 65-70 features each.
\n\nAlso this trick improved my validation and public LB scores. This transformation cut the number of unique values for each feature from tens of thousands to several hundreds. Therefore my xgboost trees was not able to split precisely in attempts to match the noise in data. Overall it led to more generalizing model (70 on public and 56 on private). Obviously, this can be achieved by xgboost parameters optimization, however such optimization will be time consuming.
\n\nI hope some of you will find my observation interesting!
", "votes": null, "replies": [] }, { "id": 133065, "author_name": "cpmpml", "author_url": "", "post_date": "08/31/2016 05:58:08", "content": "Denis, thanks.
\n\nxgboost has an option to use an approximate algorithm that does what you do more or less. If I remember well it bins large cardinality features using 300 bins. But doing so yourself is definitely more memory efficient.
", "votes": null, "replies": [] }, { "id": 133091, "author_name": "aldente", "author_url": "", "post_date": "08/31/2016 07:44:23", "content": "Congratulations @CPMP, @Denis! I hope people with low RAM would like to share their approaches as well. Thanks ;)
\n\nMy computer has 2x4GB = 8GB RAM (replaced 2GB with 4GB for this competition, previously 6GB), no SSD, no swap. Data preparation was done in R (data.table), saved all ready-for-use data into harddisk, then XGB fitting in Python. The model has 2 levels. My result:
\n\nPublic : 0.43959 (53rd, after LB cleaning = 41st)
Private: 0.45462 (31st, after LB cleaning = 24th)
Level 1: FTRL with Vowpal Wabbit, to use all weeks 3..9, to be used as meta-feature (vw_ftrl) in level 2. (similar to ML-Bure's 12th feature.)
\n\nLevel 2, ensemble of two groups:
\n\nXGB models, 16 features (similar to ML-Bure's features + _IDs), @2 weeks: [4,9], [5,9], [6,9], [7,9],\n[8,9]
XGB models with less features, @3 weeks: [6,7,9], [6,8,9],\n[7,8,9]
Strongest features are: vw_ftrl, lag1, lag2, lag3.
\n\nI also tried all other week pairs, but public LB scores were not good, so I didn't use them in ensemble.
\n\nImportant: in the usage of lagged demand features, I consider whole test set as week 10. So, week 10's previous week uses week 9 as lag1, the same as week 11's previous week also uses week 9 as lag1. The same as lag2 (to use week 8 for both weeks), lag3 (to use week 7 for both weeks).
\n\nI tried also another version by considering test set to have 2 different weeks (10 and 11), but the result is worse in private LB (I've checked it out by doing re-submission).
\n\nPossible improvement: I believe more XGB models in the ensemble will improve the score. E.g. by combining 50% of each week, etc. And, maybe to use only one week (9) for the fitting as done by ML-Bure.
\n\nThanks to Scirpus for FTRL idea, Kes367 et.al. for lagged demand idea, CPMP for 32 bit idea, all script writers, all forum contributors. Kaggle (Wendy et.al.), Grupo Bimbo, all Kagglers... ;)
\n\nIn the last week, I tried to fit whole weeks [3..9] in Turi GraphLab (academic license, now acquired by Apple), with the boosted_trees_regression (use same features). The fitting took 4+ days, epoch=120, consumed ~6GB RAM, the result is.. worse than single XGB model on 2 weeks [8,9] taking only 2 hours.
\n\nAlso tried to use Apache Spark, successfully installed in Windows machine, but not the XGBoost4J (always getting error).
", "votes": null, "replies": [] }, { "id": 133153, "author_name": "scirpus", "author_url": "", "post_date": "08/31/2016 16:26:49", "content": "Congrats Aldente!
\n\nI came 89th using just ftlr + isotonic regression based on (week - 1) and (week - 2) predictions - some times old timer techniques can come good ;) This competition finally made me buy some extra ram to go from 6gb to 24 gb. The main reason is that I am spending more and more time coming up with a solution that fits into 6 gig. Truncated mantissas on floats to save 4 bits really helped ;(
", "votes": null, "replies": [] }, { "id": 133181, "author_name": "aldente", "author_url": "", "post_date": "08/31/2016 20:30:00", "content": "@Scirpus, FTRL rocks! I learnt new thing. I also tried your isotonic-regression C++ code, but getting all zero results after modification as I had difficulty to compile it in Windows OS (Visual Studio). Then I tried sklearn's IsotonicRegression (took ~5+ hours to run), but getting slightly worse score. 24GB is big, we'll lose you in the Kernels ;))
", "votes": null, "replies": [] }, { "id": 133190, "author_name": "scirpus", "author_url": "", "post_date": "08/31/2016 21:14:42", "content": "Thanks\n24 gig means that I can do 4 times as many kernels ;)
", "votes": null, "replies": [] }, { "id": 133226, "author_name": "happycube", "author_url": "", "post_date": "09/01/2016 00:38:23", "content": "I hadn't thought of doing integer conversions - and nice pointer CPMP - manually doing that worked well for me on Springleaf but forgot about it on this one.
\n\nAlso under Linux zram is quite useful - it's still a performance hit, but it compresses well allowing 2-3x as much stuff in memory.
", "votes": null, "replies": [] }, { "id": 133288, "author_name": "johnpateha", "author_url": "", "post_date": "09/01/2016 10:48:12", "content": "I also made models by products, but only for top 50 products and these predictions I avereged with full model.
\n\nBefore that I made some processing with products' names - I removed some additional suffixes before brand (I guess that information about producer for Bimbo, not for clients), corrected some name mistakes.\nThis processing reduced number of unique products by half. And after that top 50 products had near 90% share of test set.
\n\nI started this competition too late in August and hadn't enough time to tune features and parameters for each product's model (I'm sure it makes sense). But even with common set of features, products' models led to visible improvement of base model (~ 0.003).
", "votes": null, "replies": [] }, { "id": 135126, "author_name": "pablomarin", "author_url": "", "post_date": "09/11/2016 10:13:38", "content": "@CPMP
\n\nDid you use the same xgboost hyperparameters for all models? or did you do a grid search for each model? approx 1500 grid searches?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "133060": "Congratulation to winners! That was a challenging competition given data set size. Like many, I am waiting for some info on the approach used by top teams.\r\n\r\nIn the meantime, here is some info on my solution (18th on final LB).\r\n\r\nIn order to deal with data size I decided to build one model per product id. I knew this would leave money on the table, but it was rather effective with a 16 GB machine (macbook pro). Did anyone else use the same trick?\r\n\r\nCV setting is time based. Use week 9 as validation set, and train models on weeks 6-8. Local CV is well correlated with the public LB, with an almost constant offset.\r\n\r\nFeatures are lag, averages on various groupings. For instance, group by agencia and route, then average sales and returns. In some cases the number of non zero values, and the std is alos computed. Then join this with the training data. \r\n\r\nMore precisely, for the first half of competition, I used previous 4 weeks to build features. I used weeks 3-5 to build features for week 6, weeks 3-6 to build features for week 7, weeks 4-7 to build features for week 8, weeks 5-8 to build features for week 9, and weeks 6-9 to build features for test. Missing features in weeks 6 are treated as NaN. \r\n \r\nFor CV, models were trained on weeks 6-8 with the lag features described above, and validated on week 9 with its lag features. For test predictions, weeks 7-9 with their lag features were used.\r\nThis led me to 7th place at some point. I wish competition ended then ;)\r\n\r\nI then switched gear, like many, to focus on 2 weeks lag predictions; CV setting is almost the same, but one week is omitted. For instance weeks 4-6 are used to build features for week 8, week 7 is omitted. The rest is unchanged.\r\n\r\nWhat we lose with decomposition is cross product interactions. Two ways were used to cope with that. \r\n\r\nFirst, features based on groupings across products were capturing some interactions.\r\n\r\nSecond, in a last pass, all products with poor local CV values were grouped to form a much smaller data set (about 2.5M entries instead of 75M). Then I could train xgboost on it. This was rather effective when predicting one week ahead, but I could not get it to work well when predicting 2 weeks ahead because of lack of time.\r\n\r\nI did not use ensembling, also because of lack of time. \r\n\r\nTime is the main issue with this approach. Local CV plus submission preparation is taking between 24 and 60 hours as there are almost 1500 models to build. This is the classical time vs memory tradeoff. However, it could scale very well on a cluster as each model can be trained concurrently with others. I will definitely look into a cluster next time!\r\n\r\nThe biggest surprise to me is that the one week ahead model did as well as a the two week ahead model on private LB . Probably because my last pass was effective with the former.\r\n\r\nI did all in Python, with numpy, pandas, scikit-learn and xgboost mainly. \r\n\r\nHope the above is of interest to readers.", "133064": "That's an elegant solution in terms of memory efficiency! I want to contribute a little to this topic too. \r\n\r\nI took each float feature, multiplied it by 100 and took int8/ int16 of resulting value. Therefore I lowered memory usage by approximately 4 times. I was able to fit models on my macbook with 8GB (and ssd swap ;) ) on whole weeks with 65-70 features each.\r\n\r\nAlso this trick improved my validation and public LB scores. This transformation cut the number of unique values for each feature from tens of thousands to several hundreds. Therefore my xgboost trees was not able to split precisely in attempts to match the noise in data. Overall it led to more generalizing model (70 on public and 56 on private). Obviously, this can be achieved by xgboost parameters optimization, however such optimization will be time consuming. \r\n\r\nI hope some of you will find my observation interesting!", "133065": "Denis, thanks.\r\n\r\nxgboost has an option to use an approximate algorithm that does what you do more or less. If I remember well it bins large cardinality features using 300 bins. But doing so yourself is definitely more memory efficient.", "133091": "Congratulations @CPMP, @Denis! I hope people with low RAM would like to share their approaches as well. Thanks ;) \r\n\r\nMy computer has 2x4GB = 8GB RAM (replaced 2GB with 4GB for this competition, previously 6GB), no SSD, no swap. Data preparation was done in R (data.table), saved all ready-for-use data into harddisk, then XGB fitting in Python. The model has 2 levels. My result: \r\n\r\n - Public : 0.43959 (53rd, after LB cleaning = 41st)\r\n \r\n - Private: 0.45462 (31st, after LB cleaning = 24th)\r\n\r\nLevel 1: FTRL with Vowpal Wabbit, to use all weeks 3..9, to be used as meta-feature (vw_ftrl) in level 2. (similar to [ML-Bure][1]'s 12th feature.) \r\n\r\nLevel 2, ensemble of two groups: \r\n\r\n 1. XGB models, 16 features (similar to ML-Bure's features + _IDs), @2 weeks: [4,9], [5,9], [6,9], [7,9],\r\n [8,9] \r\n \r\n 2. XGB models with less features, @3 weeks: [6,7,9], [6,8,9],\r\n [7,8,9]\r\n\r\nStrongest features are: vw_ftrl, lag1, lag2, lag3. \r\n\r\nI also tried all other week pairs, but public LB scores were not good, so I didn't use them in ensemble. \r\n\r\nImportant: in the usage of lagged demand features, I consider whole test set as week 10. So, week 10's previous week uses week 9 as lag1, the same as week 11's previous week also uses week 9 as lag1. The same as lag2 (to use week 8 for both weeks), lag3 (to use week 7 for both weeks). \r\n\r\nI tried also another version by considering test set to have 2 different weeks (10 and 11), but the result is worse in private LB (I've checked it out by doing re-submission). \r\n\r\nPossible improvement: I believe more XGB models in the ensemble will improve the score. E.g. by combining 50% of each week, etc. And, maybe to use only one week (9) for the fitting as done by [ML-Bure][2]. \r\n\r\nThanks to Scirpus for FTRL idea, [Kes367][3] et.al. for lagged demand idea, CPMP for 32 bit idea, all script writers, all forum contributors. Kaggle (Wendy et.al.), Grupo Bimbo, all Kagglers... ;) \r\n\r\nIn the last week, I tried to fit whole weeks [3..9] in Turi GraphLab (academic license, now acquired by Apple), with the boosted_trees_regression (use same features). The fitting took 4+ days, epoch=120, consumed ~6GB RAM, the result is.. worse than single XGB model on 2 weeks [8,9] taking only 2 hours. \r\n\r\nAlso tried to use Apache Spark, successfully installed in Windows machine, but not the [XGBoost4J][4] (always getting error). \r\n\r\n\r\n [1]: https://www.kaggle.com/c/grupo-bimbo-inventory-demand/forums/t/23208/19-place-solution\r\n [2]: https://www.kaggle.com/c/grupo-bimbo-inventory-demand/forums/t/23208/19-place-solution\r\n [3]: http://blog.nycdatascience.com/student-works/predicting-demand-historical-sales-data-grupo-bimbo-kaggle-competition/\r\n [4]: http://dmlc.ml/2016/03/14/xgboost4j-portable-distributed-xgboost-in-spark-flink-and-dataflow.html", "133153": "Congrats Aldente!\r\n\r\nI came 89th using just ftlr + isotonic regression based on (week - 1) and (week - 2) predictions - some times old timer techniques can come good ;) This competition finally made me buy some extra ram to go from 6gb to 24 gb. The main reason is that I am spending more and more time coming up with a solution that fits into 6 gig. Truncated mantissas on floats to save 4 bits really helped ;(", "133181": "Scirpus, FTRL rocks! I learnt new thing. I also tried your isotonic-regression C++ code, but getting all zero results after modification as I had difficulty to compile it in Windows OS (Visual Studio). Then I tried sklearn's IsotonicRegression (took ~5+ hours to run), but getting slightly worse score. 24GB is big, we'll lose you in the Kernels ;))", "133190": "Thanks\r\n24 gig means that I can do 4 times as many kernels ;)", "133226": "I hadn't thought of doing integer conversions - and nice pointer CPMP - manually doing that worked well for me on Springleaf but forgot about it on this one.\r\n\r\nAlso under Linux zram is quite useful - it's still a performance hit, but it compresses well allowing 2-3x as much stuff in memory.", "133288": "I also made models by products, but only for top 50 products and these predictions I avereged with full model. \r\n\r\nBefore that I made some processing with products' names - I removed some additional suffixes before brand (I guess that information about producer for Bimbo, not for clients), corrected some name mistakes.\r\nThis processing reduced number of unique products by half. And after that top 50 products had near 90% share of test set.\r\n\r\nI started this competition too late in August and hadn't enough time to tune features and parameters for each product's model (I'm sure it makes sense). But even with common set of features, products' models led to visible improvement of base model (~ 0.003).", "135126": "CPMP\r\n\r\nDid you use the same xgboost hyperparameters for all models? or did you do a grid search for each model? approx 1500 grid searches?" }, "source": "existing_discussion", "complete": true, "competition_id": 5260, "competition_slug": "grupo-bimbo-inventory-demand", "team_id": 341795, "solution_url": "/competitions/grupo-bimbo-inventory-demand/writeups/cpmp-low-memory-solution" }