My approach is good old \"feature engineer till you drop\" so I suppose it's not so interesting after all but I hope somebody will find something. :) Here is brief overview of approach.
\n\nSolution follows standard scheme. Train logloss-model for user-product pairs, train logloss-model for None, use probabilities for F1-optimisation.
\n\nLet us denote current user as U, last order of U as LO, current product as P and its aisle and department as A and D. Let us denote order as \"A order\" if it contains at least one product from aisle A (same for P and D).
\n\nProduct only features:
\n\nUser only features
\n\nUser-product features:
\n\nLarge Brown Eggs - Organic Large Brown Grade AA Cage Free Eggs
\n\nLarge Brown Eggs - Organic Large Grade AA Brown Eggs
\n\nGreen Seedless Grapes - Grape White/Green Seedless
\n\nOrganic Large Grade AA Brown Eggs - Large Brown Eggs
\n\nOrganic Broccoli - Organic Broccoli Crowns
\n\nBanana - Organic Banana
\n\nOrganic Banana - Banana
\n\nOrganic Milk Reduced Fat - Organic Reduced Fat Milk
\n\nBanana - Bag of Organic Bananas
\n\nHalf And Half Cream - Organic Half & Half
\n\nUsing these pairs we can generate several features like \"how many products similar to P user U bought since last P order\". If such number is great it could mean user switched from product P to similar product Q.
\n\nI've expanded training set using last prior order for all test users with at least 5 prior orders.
\n\nMain user-product probability model is average of 8 CatBoost models. I've used product ID as categorical feature for CatBoost (yeah, that's why CatBoost has such name :))
\n\nFor None-classifier I've used the same training users. I've used my per-user features from main model and used features based on predictions of main model. It's a little bit not very clean and tidy approach because main model predictions are a little bit overfitted for training users (so features on train distributed in slightly another manner in comparison to features on test) but this approach gave me good results, and I was unable to invent better approach. :) Splitting train users for main/none classifiers gave me considerably worse results. So, None-classifier is a single CatBoost model.
\n\nF1-optimization. At first I used a Monte-Carlo simulation using predicted probabilities to determine solution with max expected F1 for each user. But Faron's approach gave me considerably better results. So I've used it instead of Monte-Carlo. :)
\n\nThanks!
", "comments": [ { "id": 222937, "author_name": "DanilaSavenkov", "author_url": "", "post_date": "2017-09-20T16:33:26.620000", "content": "Thanks for sharing.\nDid you really get boost in performance by using product ID as a categorical feature in CatBoost? If yes, could you please share what method did you use for encoding? I guess here are the methods implemented in CatBoost: https://tech.yandex.com/catboost/doc/dg/concepts/algorithm-main-stages_cat-to-numberic-docpage/ (I'm speaking about the type).
", "votes": 1, "replies": [] }, { "id": 213776, "author_name": "slonoslon", "author_url": "", "post_date": "2017-08-15T08:33:20.183000", "content": "Congratulations! Your feature set seems huge from this description. How many features do you have and how much RAM they use?
", "votes": 1, "replies": [ { "id": 213818, "author_name": "Akulov Yaroslav", "author_url": "", "post_date": "2017-08-15T10:21:30.913000", "content": "Around 1000 features and 100Gb RAM.
", "votes": 3, "replies": [] }, { "id": 213833, "author_name": "Bradley Mitchell", "author_url": "", "post_date": "2017-08-15T11:09:48.563000", "content": "Wow! That is some impressive feature engineering - congrats!!
", "votes": 1, "replies": [] }, { "id": 214143, "author_name": "Mint", "author_url": "", "post_date": "2017-08-16T03:43:55.607000", "content": "Thanks for describing your features, they're interesting and different from the other solutions I've read.
\n\nI'm still fairly beginner at ML, and that many features surprised me because I thought it would cause a lot more overfitting. Is that only the case if
", "votes": 0, "replies": [] }, { "id": 214490, "author_name": "BerkKarahan", "author_url": "", "post_date": "2017-08-17T05:56:06.557000", "content": "@akulov, could you please specify your computer's specifications, your FE seems highly compute intensive, i am curious.
", "votes": 0, "replies": [] } ] }, { "id": 213701, "author_name": "StuartLu", "author_url": "", "post_date": "2017-08-15T04:44:00.673000", "content": "Thanks for the sharing. May I know how you selected your features?
", "votes": 1, "replies": [ { "id": 213704, "author_name": "Akulov Yaroslav", "author_url": "", "post_date": "2017-08-15T04:50:41.590000", "content": "Honestly speaking, I didn't. I had enough RAM and patience :)
", "votes": 8, "replies": [] } ] }, { "id": 214193, "author_name": "Random Hero", "author_url": "", "post_date": "2017-08-16T07:18:56.573000", "content": "Was there any specific reason why you decided to encode 200 P_is_X_product features as binaries instead of just using LabelEncoder?
", "votes": 0, "replies": [ { "id": 214253, "author_name": "Akulov Yaroslav", "author_url": "", "post_date": "2017-08-16T11:51:14.043000", "content": "What is LabelEncoder? :)
", "votes": 0, "replies": [] }, { "id": 214448, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2017-08-16T23:58:34.960000", "content": "@Akulov thanks for sharing.
\n\nTo answer your question, Label Encoder is used to transform non-numerical labels to numerical labels (or nominal categorical variables). Numerical labels are always between 0 and n_classes-1.
", "votes": 0, "replies": [] } ] }, { "id": 213907, "author_name": "Ravi Teja Gutta", "author_url": "", "post_date": "2017-08-15T14:16:16.520000", "content": "If this was a team competition, you and sjv would have made the winning team. Intense feature engineering + pure DL. Congrats and thanks for sharing!
", "votes": 0, "replies": [] }, { "id": 213869, "author_name": "Kirgsn", "author_url": "", "post_date": "2017-08-15T12:48:50.890000", "content": "Congrats! Remarkable feature engineering! :)
", "votes": 0, "replies": [] }, { "id": 213820, "author_name": "Yifan Xie", "author_url": "", "post_date": "2017-08-15T10:31:59.660000", "content": "Wonderful approach in FE, thanks I learnt from you :)
\n\nAlso good to know that Catboost is doing good - yet another tool to investigate :)
\n\nDo you know how much it improve the score by including the last week in prior?
", "votes": 0, "replies": [ { "id": 213875, "author_name": "Akulov Yaroslav", "author_url": "", "post_date": "2017-08-15T13:06:21.637000", "content": "Expanding training set decreased logloss error by ~0.0002-0.0003 but I didn't check how much it improved LB score.
", "votes": 1, "replies": [] } ] }, { "id": 213757, "author_name": "Jing Lu", "author_url": "", "post_date": "2017-08-15T07:32:49.573000", "content": "Congratulations! Why do you choose catboost? Do you think catboost is more appropriate here or better than lightgbm and xgboost in general.
", "votes": 0, "replies": [ { "id": 213767, "author_name": "Akulov Yaroslav", "author_url": "", "post_date": "2017-08-15T08:02:50.330000", "content": "I've tried catboost, xgboost and lightgbm. Catboost gave better results, so I chose it over other two algorithms. :)
", "votes": 0, "replies": [] } ] }, { "id": 213748, "author_name": "CPMP", "author_url": "", "post_date": "2017-08-15T07:06:38.240000", "content": "Thanks for sharing, and congrats!
", "votes": 0, "replies": [] } ], "comment_count": 18, "messages": [ { "id": 213699, "postDate": "2017-08-15T04:34:22.163Z", "content": "My approach is good old \"feature engineer till you drop\" so I suppose it's not so interesting after all but I hope somebody will find something. :) Here is brief overview of approach.
\n\nSolution follows standard scheme. Train logloss-model for user-product pairs, train logloss-model for None, use probabilities for F1-optimisation.
\n\nLet us denote current user as U, last order of U as LO, current product as P and its aisle and department as A and D. Let us denote order as \"A order\" if it contains at least one product from aisle A (same for P and D).
\n\nProduct only features:
\n\nUser only features
\n\nUser-product features:
\n\nLarge Brown Eggs - Organic Large Brown Grade AA Cage Free Eggs
\n\nLarge Brown Eggs - Organic Large Grade AA Brown Eggs
\n\nGreen Seedless Grapes - Grape White/Green Seedless
\n\nOrganic Large Grade AA Brown Eggs - Large Brown Eggs
\n\nOrganic Broccoli - Organic Broccoli Crowns
\n\nBanana - Organic Banana
\n\nOrganic Banana - Banana
\n\nOrganic Milk Reduced Fat - Organic Reduced Fat Milk
\n\nBanana - Bag of Organic Bananas
\n\nHalf And Half Cream - Organic Half & Half
\n\nUsing these pairs we can generate several features like \"how many products similar to P user U bought since last P order\". If such number is great it could mean user switched from product P to similar product Q.
\n\nI've expanded training set using last prior order for all test users with at least 5 prior orders.
\n\nMain user-product probability model is average of 8 CatBoost models. I've used product ID as categorical feature for CatBoost (yeah, that's why CatBoost has such name :))
\n\nFor None-classifier I've used the same training users. I've used my per-user features from main model and used features based on predictions of main model. It's a little bit not very clean and tidy approach because main model predictions are a little bit overfitted for training users (so features on train distributed in slightly another manner in comparison to features on test) but this approach gave me good results, and I was unable to invent better approach. :) Splitting train users for main/none classifiers gave me considerably worse results. So, None-classifier is a single CatBoost model.
\n\nF1-optimization. At first I used a Monte-Carlo simulation using predicted probabilities to determine solution with max expected F1 for each user. But Faron's approach gave me considerably better results. So I've used it instead of Monte-Carlo. :)
\n\nThanks!
", "rawMarkdown": "My approach is good old \"feature engineer till you drop\" so I suppose it's not so interesting after all but I hope somebody will find something. :) Here is brief overview of approach.\n\nSolution follows standard scheme. Train logloss-model for user-product pairs, train logloss-model for None, use probabilities for F1-optimisation.\n\nLet us denote current user as U, last order of U as LO, current product as P and its aisle and department as A and D. Let us denote order as \"A order\" if it contains at least one product from aisle A (same for P and D).\n\n1. Product only features:\n* P is Low Fat/Organic :)\n* 200 binary \"P_is_X_product\" features for top-200 most popular products\n* binary \"A_is_X_aisle\" and \"D_is_X_department\" features for all X aisles and departments\n* average day/count interval between all P orders\n\n2. User only features\n* 7+24 binary day of week and hour of day features for O\n* Ratio of reordered products in last N orders/last M days orders of U\n* Average similarity of product sets for sequential orders (e.g. Jaccard measure of such sets)\n* Average similarity of product sets for same day orders (it could be useful in case LO has days_since == 0)\n\n3. User-product features:\n* Number of days/orders between LO and most recent order of U with P/A/D\n* Number of orders between most recent order with P and most recent order with A (trying to see if user switched products inside A)\n* Binary \"it's been X orders since last order with P\" features where X is from 1 to 10\n* Average day/count interval between all U's orders with P/A/D\n* Share of P products/orders among all/A/D products/orders bought by U (how ofter U buys P)\n* Same share but only for orders since first P order (how ofter U buys P since he began to do so)\n* Same share but for last N orders\n* Same share but for orders during last M days\n* Average position of P in orders of U (raw or normalized by size of order)\n* Ratio of average interval between U's orders with P to average interval between all orders with P\n* Ratio of number of days/orders since most recent U's P order to average interval between all orders with P\n* Let's assume it's been N days/orders since last P order. Using all prior orders for each product P we can calculate probability to reorder P if it's been N days/orders since last P order.\n* Same could be done using only U's prior orders (but it's much more noisy of course). I've tried to smooth one probability with another buy couldn't achieve good results.\n* Product similarity features. Let's pretend that users often switch from product A to product B (for example Bananas and Organic Bananas). How could we try to detect such product pairs. We could assume that buying product B should considerably lower probability of buying product A in the same order. Let's consider ratio = P(A in current order|A in previous orders, B in current order) / P(A in current order|A in previous orders). For unrelated products this ratio should be close to 1. To similar products this ratio should be close to 0. Such probabilities could be easily calculated using prior orders. For example top-10 pairs with lowest ratio (only for pairs with enough statistics) are:\n\nLarge Brown Eggs - Organic Large Brown Grade AA Cage Free Eggs\n\nLarge Brown Eggs - Organic Large Grade AA Brown Eggs\n\nGreen Seedless Grapes - Grape White/Green Seedless\n\nOrganic Large Grade AA Brown Eggs - Large Brown Eggs\n\nOrganic Broccoli - Organic Broccoli Crowns\n\nBanana - Organic Banana\n\nOrganic Banana - Banana\n\nOrganic Milk Reduced Fat - Organic Reduced Fat Milk\n\nBanana - Bag of Organic Bananas\n\nHalf And Half Cream - Organic Half & Half\n\nUsing these pairs we can generate several features like \"how many products similar to P user U bought since last P order\". If such number is great it could mean user switched from product P to similar product Q.\n\n* Average share of P aggregated by months.\n* Linear extrapolation of mentioned above month aggregated share.\n* P orders streak length of user U\n\nI've expanded training set using last prior order for all test users with at least 5 prior orders.\n\nMain user-product probability model is average of 8 CatBoost models. I've used product ID as categorical feature for CatBoost (yeah, that's why CatBoost has such name :))\n\nFor None-classifier I've used the same training users. I've used my per-user features from main model and used features based on predictions of main model. It's a little bit not very clean and tidy approach because main model predictions are a little bit overfitted for training users (so features on train distributed in slightly another manner in comparison to features on test) but this approach gave me good results, and I was unable to invent better approach. :) Splitting train users for main/none classifiers gave me considerably worse results. So, None-classifier is a single CatBoost model.\n\nF1-optimization. At first I used a Monte-Carlo simulation using predicted probabilities to determine solution with max expected F1 for each user. But Faron's approach gave me considerably better results. So I've used it instead of Monte-Carlo. :)\n\nThanks!", "votes": 53 }, { "id": 222937, "postDate": "2017-09-20T16:33:26.620Z", "content": "Thanks for sharing.\nDid you really get boost in performance by using product ID as a categorical feature in CatBoost? If yes, could you please share what method did you use for encoding? I guess here are the methods implemented in CatBoost: https://tech.yandex.com/catboost/doc/dg/concepts/algorithm-main-stages_cat-to-numberic-docpage/ (I'm speaking about the type).
", "rawMarkdown": "Thanks for sharing.\nDid you really get boost in performance by using product ID as a categorical feature in CatBoost? If yes, could you please share what method did you use for encoding? I guess here are the methods implemented in CatBoost: https://tech.yandex.com/catboost/doc/dg/concepts/algorithm-main-stages_cat-to-numberic-docpage/ (I'm speaking about the type).", "votes": 1 }, { "id": 213776, "postDate": "2017-08-15T08:33:20.183Z", "content": "Congratulations! Your feature set seems huge from this description. How many features do you have and how much RAM they use?
", "rawMarkdown": "Congratulations! Your feature set seems huge from this description. How many features do you have and how much RAM they use?", "votes": 1, "replies": [ { "id": 213818, "postDate": "2017-08-15T10:21:30.913Z", "content": "Around 1000 features and 100Gb RAM.
", "rawMarkdown": "Around 1000 features and 100Gb RAM.", "votes": 3 }, { "id": 213833, "postDate": "2017-08-15T11:09:48.563Z", "content": "Wow! That is some impressive feature engineering - congrats!!
", "rawMarkdown": "Wow! That is some impressive feature engineering - congrats!!", "votes": 1 }, { "id": 214143, "postDate": "2017-08-16T03:43:55.607Z", "content": "Thanks for describing your features, they're interesting and different from the other solutions I've read.
\n\nI'm still fairly beginner at ML, and that many features surprised me because I thought it would cause a lot more overfitting. Is that only the case if
", "rawMarkdown": "Thanks for describing your features, they're interesting and different from the other solutions I've read.\n\n I'm still fairly beginner at ML, and that many features surprised me because I thought it would cause a lot more overfitting. Is that only the case if " }, { "id": 214490, "postDate": "2017-08-17T05:56:06.557Z", "content": "@akulov, could you please specify your computer's specifications, your FE seems highly compute intensive, i am curious.
", "rawMarkdown": "@akulov, could you please specify your computer's specifications, your FE seems highly compute intensive, i am curious." } ] }, { "id": 213701, "postDate": "2017-08-15T04:44:00.673Z", "content": "Thanks for the sharing. May I know how you selected your features?
", "rawMarkdown": "Thanks for the sharing. May I know how you selected your features?", "votes": 1, "replies": [ { "id": 213704, "postDate": "2017-08-15T04:50:41.590Z", "content": "Honestly speaking, I didn't. I had enough RAM and patience :)
", "rawMarkdown": "Honestly speaking, I didn't. I had enough RAM and patience :)", "votes": 8 } ] }, { "id": 214193, "postDate": "2017-08-16T07:18:56.573Z", "content": "Was there any specific reason why you decided to encode 200 P_is_X_product features as binaries instead of just using LabelEncoder?
", "rawMarkdown": "Was there any specific reason why you decided to encode 200 P_is_X_product features as binaries instead of just using LabelEncoder?", "replies": [ { "id": 214253, "postDate": "2017-08-16T11:51:14.043Z", "content": "What is LabelEncoder? :)
", "rawMarkdown": "What is LabelEncoder? :)" }, { "id": 214448, "postDate": "2017-08-16T23:58:34.960Z", "content": "@Akulov thanks for sharing.
\n\nTo answer your question, Label Encoder is used to transform non-numerical labels to numerical labels (or nominal categorical variables). Numerical labels are always between 0 and n_classes-1.
", "rawMarkdown": "@Akulov thanks for sharing.\n\nTo answer your question, Label Encoder is used to transform non-numerical labels to numerical labels (or nominal categorical variables). Numerical labels are always between 0 and n_classes-1. " } ] }, { "id": 213907, "postDate": "2017-08-15T14:16:16.520Z", "content": "If this was a team competition, you and sjv would have made the winning team. Intense feature engineering + pure DL. Congrats and thanks for sharing!
", "rawMarkdown": "If this was a team competition, you and sjv would have made the winning team. Intense feature engineering + pure DL. Congrats and thanks for sharing!" }, { "id": 213869, "postDate": "2017-08-15T12:48:50.890Z", "content": "Congrats! Remarkable feature engineering! :)
", "rawMarkdown": "Congrats! Remarkable feature engineering! :)" }, { "id": 213820, "postDate": "2017-08-15T10:31:59.660Z", "content": "Wonderful approach in FE, thanks I learnt from you :)
\n\nAlso good to know that Catboost is doing good - yet another tool to investigate :)
\n\nDo you know how much it improve the score by including the last week in prior?
", "rawMarkdown": "Wonderful approach in FE, thanks I learnt from you :)\n\nAlso good to know that Catboost is doing good - yet another tool to investigate :) \n\nDo you know how much it improve the score by including the last week in prior?", "replies": [ { "id": 213875, "postDate": "2017-08-15T13:06:21.637Z", "content": "Expanding training set decreased logloss error by ~0.0002-0.0003 but I didn't check how much it improved LB score.
", "rawMarkdown": "Expanding training set decreased logloss error by ~0.0002-0.0003 but I didn't check how much it improved LB score.", "votes": 1 } ] }, { "id": 213757, "postDate": "2017-08-15T07:32:49.573Z", "content": "Congratulations! Why do you choose catboost? Do you think catboost is more appropriate here or better than lightgbm and xgboost in general.
", "rawMarkdown": "Congratulations! Why do you choose catboost? Do you think catboost is more appropriate here or better than lightgbm and xgboost in general.", "replies": [ { "id": 213767, "postDate": "2017-08-15T08:02:50.330Z", "content": "I've tried catboost, xgboost and lightgbm. Catboost gave better results, so I chose it over other two algorithms. :)
", "rawMarkdown": "I've tried catboost, xgboost and lightgbm. Catboost gave better results, so I chose it over other two algorithms. :)" } ] }, { "id": 213748, "postDate": "2017-08-15T07:06:38.240Z", "content": "Thanks for sharing, and congrats!
", "rawMarkdown": "Thanks for sharing, and congrats!" } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T02:03:17Z", "competition_id": 6644, "competition_slug": "basket-analysis", "team_id": 739505, "solution_url": "/competitions/basket-analysis/writeups/akulov-yaroslav-6th-place-solution-overview" }