My solution was nothing fancy, a single lightgbm model with Faron's F1 optimization code. I ended up creating more that 170 features in the end, but doing feature selection became very difficult with them in the end. Apart from the common frequency/distribution and interaction based features here are some interesting features that I created:
\n\nApart from these I created many features to capture recency, streak and past order similarity and diversity which helped.
\n\nAnother thing that I tried was a voting ensemble of my final submissions which gave me decent lift on public LB whenever there was enough diversity in submissions (roughly 0.0005-0.001)...but in the end apparently my best ensemble was overfitting and would have dropped me 20 positions had I not selected my single best model (4th best submission on public LB) also.
\n\nFinally I had a lot of fun with this competition especially because of the clean and un-anonymised and learned a lot from all the insightful discussions on the forums.
\n\nThank You
", "comments": [ { "id": 213769, "author_name": "", "author_url": "", "post_date": "2017-08-15T08:08:52.690000", "content": "It is sad that I did not see you to get a gold medal . I get the current position from your help. Really really thanks you.
", "votes": 3, "replies": [] }, { "id": 213777, "author_name": "StuartLu", "author_url": "", "post_date": "2017-08-15T08:34:04.637000", "content": "Thanks for your sharing. Your insights on how to handling none and predict order size help us a lot. Good luck next time.
", "votes": 1, "replies": [] }, { "id": 215148, "author_name": "FuLin", "author_url": "", "post_date": "2017-08-20T04:22:20.813000", "content": "Thank you so much for this sharing. I have learnt a lot from your discussion in the forum.
\n\nWhen you have a chance, can you please explain a little bit more about how you tried to capture the recency?
", "votes": 0, "replies": [ { "id": 215157, "author_name": "SVJ24", "author_url": "", "post_date": "2017-08-20T06:00:00.543000", "content": "I tried several approaches to capture recency based on order number and days_ago variable. I created days_ago feature using days_since last order such that if for a user with four orders of the days_since_last order values were None, 7, 10, 3 then days_ago would look like 20, 13, 3, 0 such the last order (train/test order) is always set at day zero. Next, to capture recency there are various techniques like simply taking sum of these numbers or square of these numbers (squaring will give more weight to older orders with days ago and newer order with order number). Another feature could be sum of days_ago with exponential decay, I tried an exponential decay with half life of 15/30/60 days.
\n\nHope this helps.
\n\nRegards,
", "votes": 2, "replies": [] } ] }, { "id": 215145, "author_name": "yue yu", "author_url": "", "post_date": "2017-08-20T04:01:25.023000", "content": "Thanks for sharing.\nI have learned a lot from your posts before.\nForgive my newbie question: Maybe you treat each order as a sentence and get the product's word2vec，but how did you get the others word2vec?
", "votes": 0, "replies": [ { "id": 215156, "author_name": "SVJ24", "author_url": "", "post_date": "2017-08-20T05:51:54.440000", "content": "You are right to create vectors I treated the orders as sentences and product_ids as words. To get the vectors for aisle_id and department_id as well, I simply added them to the words as well. For example a typical sentence would look like a1, p1, d1, a2,p2,d2....and so on. This also helps bring the vectors belonging to same aisle and department together in the vector space and you get better embedding event when you have less data (not the case in this competition). To get user vectors I used doc2vec and used the user_ids as tags for each order/sentence.
\n\nRegards,
", "votes": 1, "replies": [] }, { "id": 215171, "author_name": "yue yu", "author_url": "", "post_date": "2017-08-20T09:15:21.330000", "content": "It really helps，thanks.
", "votes": 0, "replies": [] } ] }, { "id": 213990, "author_name": "Jing Lu", "author_url": "", "post_date": "2017-08-15T16:47:21.457000", "content": "Congratulations and thanks for sharing! What do you use to do the feature selection?
", "votes": 0, "replies": [ { "id": 214000, "author_name": "SVJ24", "author_url": "", "post_date": "2017-08-15T17:27:07.197000", "content": "Feature selection became a huge problem for me....I didn't know what was working and what wasn't, but here are the things I tried:
\n\nfeature importance in lightgbm using based on split and gain - typically I would remove/modify feature coming in bottom 5 in both the lists
correlation with reordered: not a good technique since only linear relationship would get captured
correlations within independent variables: highly correlated variables (corr > 95%) don't add much to the model may even adversely impact the performance
ratio of variable mean in train and test: it was mostly close to 1 for this competition but I was still able to improve on LB a little bit by removing a few features with higher deviation
for continuous features, segment them into 20-50 buckets and measure diversity in the mean of y. I used kinetic energy to capture diversity but entropy works too. For binary variables something like KL divergence can be used. Higher diversity is better.
with 170 features and after extensive cleaning it became very difficult same variable would come important with one metric but useless with other. In the end I think best way was to remove a few features and see the model performance but that is too time consuming.
\n\nHope this helps.
\n\nRegards,
", "votes": 9, "replies": [] }, { "id": 214016, "author_name": "Jing Lu", "author_url": "", "post_date": "2017-08-15T18:14:28.387000", "content": "@SVJ24, thanks for sharing! That's quite some interesting metrics for measuring the feature importance. Especially the ratio of variable mean in train and test.
", "votes": 0, "replies": [] } ] }, { "id": 213979, "author_name": "plantsgo", "author_url": "", "post_date": "2017-08-15T16:18:44.150000", "content": "Thanks for your sharing.I ignored product name.
", "votes": 0, "replies": [] }, { "id": 213954, "author_name": "Luke Wang", "author_url": "", "post_date": "2017-08-15T15:13:46.623000", "content": "Congratulations. \nLearned a lot from your posts and discussions before. \nThank you!
", "votes": 0, "replies": [] } ], "comment_count": 12, "messages": [ { "id": 213758, "postDate": "2017-08-15T07:33:16.367Z", "content": "My solution was nothing fancy, a single lightgbm model with Faron's F1 optimization code. I ended up creating more that 170 features in the end, but doing feature selection became very difficult with them in the end. Apart from the common frequency/distribution and interaction based features here are some interesting features that I created:
\n\nApart from these I created many features to capture recency, streak and past order similarity and diversity which helped.
\n\nAnother thing that I tried was a voting ensemble of my final submissions which gave me decent lift on public LB whenever there was enough diversity in submissions (roughly 0.0005-0.001)...but in the end apparently my best ensemble was overfitting and would have dropped me 20 positions had I not selected my single best model (4th best submission on public LB) also.
\n\nFinally I had a lot of fun with this competition especially because of the clean and un-anonymised and learned a lot from all the insightful discussions on the forums.
\n\nThank You
", "rawMarkdown": "My solution was nothing fancy, a single lightgbm model with Faron's F1 optimization code. I ended up creating more that 170 features in the end, but doing feature selection became very difficult with them in the end. Apart from the common frequency/distribution and interaction based features here are some interesting features that I created:\n\n - **product name length:** seems weird why this should matter but it did improve the model a little bit\n - **product name rarity:** basically mean of idf of each word in a product (I guess these features help captures people who are connoisseurs of exotic products)\n - **product/user/aisle/department cluster** based on their word2vec\n - **product similarity with none:** captures if the product is usually present in orders with no reorder\n - product vector * user vector\n - product organic/gluten_free\n\nApart from these I created many features to capture recency, streak and past order similarity and diversity which helped.\n \nAnother thing that I tried was a voting ensemble of my final submissions which gave me decent lift on public LB whenever there was enough diversity in submissions (roughly 0.0005-0.001)...but in the end apparently my best ensemble was overfitting and would have dropped me 20 positions had I not selected my single best model (4th best submission on public LB) also.\n\nFinally I had a lot of fun with this competition especially because of the clean and un-anonymised and learned a lot from all the insightful discussions on the forums.\n\nThank You", "votes": 42 }, { "id": 213769, "postDate": "2017-08-15T08:08:52.690Z", "content": "It is sad that I did not see you to get a gold medal . I get the current position from your help. Really really thanks you.
", "rawMarkdown": "It is sad that I did not see you to get a gold medal . I get the current position from your help. Really really thanks you. ", "votes": 3 }, { "id": 213777, "postDate": "2017-08-15T08:34:04.637Z", "content": "Thanks for your sharing. Your insights on how to handling none and predict order size help us a lot. Good luck next time.
", "rawMarkdown": "Thanks for your sharing. Your insights on how to handling none and predict order size help us a lot. Good luck next time. ", "votes": 1 }, { "id": 215148, "postDate": "2017-08-20T04:22:20.813Z", "content": "Thank you so much for this sharing. I have learnt a lot from your discussion in the forum.
\n\nWhen you have a chance, can you please explain a little bit more about how you tried to capture the recency?
", "rawMarkdown": "Thank you so much for this sharing. I have learnt a lot from your discussion in the forum.\n\nWhen you have a chance, can you please explain a little bit more about how you tried to capture the recency?", "replies": [ { "id": 215157, "postDate": "2017-08-20T06:00:00.543Z", "content": "I tried several approaches to capture recency based on order number and days_ago variable. I created days_ago feature using days_since last order such that if for a user with four orders of the days_since_last order values were None, 7, 10, 3 then days_ago would look like 20, 13, 3, 0 such the last order (train/test order) is always set at day zero. Next, to capture recency there are various techniques like simply taking sum of these numbers or square of these numbers (squaring will give more weight to older orders with days ago and newer order with order number). Another feature could be sum of days_ago with exponential decay, I tried an exponential decay with half life of 15/30/60 days.
\n\nHope this helps.
\n\nRegards,
", "rawMarkdown": "I tried several approaches to capture recency based on order number and days_ago variable. I created days_ago feature using days_since last order such that if for a user with four orders of the days_since_last order values were None, 7, 10, 3 then days_ago would look like 20, 13, 3, 0 such the last order (train/test order) is always set at day zero. Next, to capture recency there are various techniques like simply taking sum of these numbers or square of these numbers (squaring will give more weight to older orders with days ago and newer order with order number). Another feature could be sum of days_ago with exponential decay, I tried an exponential decay with half life of 15/30/60 days.\n\nHope this helps.\n\nRegards,", "votes": 2 } ] }, { "id": 215145, "postDate": "2017-08-20T04:01:25.023Z", "content": "Thanks for sharing.\nI have learned a lot from your posts before.\nForgive my newbie question: Maybe you treat each order as a sentence and get the product's word2vec，but how did you get the others word2vec?
", "rawMarkdown": "Thanks for sharing.\nI have learned a lot from your posts before.\nForgive my newbie question: Maybe you treat each order as a sentence and get the product's word2vec，but how did you get the others word2vec?", "replies": [ { "id": 215156, "postDate": "2017-08-20T05:51:54.440Z", "content": "You are right to create vectors I treated the orders as sentences and product_ids as words. To get the vectors for aisle_id and department_id as well, I simply added them to the words as well. For example a typical sentence would look like a1, p1, d1, a2,p2,d2....and so on. This also helps bring the vectors belonging to same aisle and department together in the vector space and you get better embedding event when you have less data (not the case in this competition). To get user vectors I used doc2vec and used the user_ids as tags for each order/sentence.
\n\nRegards,
", "rawMarkdown": "You are right to create vectors I treated the orders as sentences and product_ids as words. To get the vectors for aisle_id and department_id as well, I simply added them to the words as well. For example a typical sentence would look like a1, p1, d1, a2,p2,d2....and so on. This also helps bring the vectors belonging to same aisle and department together in the vector space and you get better embedding event when you have less data (not the case in this competition). To get user vectors I used doc2vec and used the user_ids as tags for each order/sentence. \n\nRegards,", "votes": 1 }, { "id": 215171, "postDate": "2017-08-20T09:15:21.330Z", "content": "It really helps，thanks.
", "rawMarkdown": "It really helps，thanks." } ] }, { "id": 213990, "postDate": "2017-08-15T16:47:21.457Z", "content": "Congratulations and thanks for sharing! What do you use to do the feature selection?
", "rawMarkdown": "Congratulations and thanks for sharing! What do you use to do the feature selection?", "replies": [ { "id": 214000, "postDate": "2017-08-15T17:27:07.197Z", "content": "Feature selection became a huge problem for me....I didn't know what was working and what wasn't, but here are the things I tried:
\n\nfeature importance in lightgbm using based on split and gain - typically I would remove/modify feature coming in bottom 5 in both the lists
correlation with reordered: not a good technique since only linear relationship would get captured
correlations within independent variables: highly correlated variables (corr > 95%) don't add much to the model may even adversely impact the performance
ratio of variable mean in train and test: it was mostly close to 1 for this competition but I was still able to improve on LB a little bit by removing a few features with higher deviation
for continuous features, segment them into 20-50 buckets and measure diversity in the mean of y. I used kinetic energy to capture diversity but entropy works too. For binary variables something like KL divergence can be used. Higher diversity is better.
with 170 features and after extensive cleaning it became very difficult same variable would come important with one metric but useless with other. In the end I think best way was to remove a few features and see the model performance but that is too time consuming.
\n\nHope this helps.
\n\nRegards,
", "rawMarkdown": "Feature selection became a huge problem for me....I didn't know what was working and what wasn't, but here are the things I tried:\n\n - feature importance in lightgbm using based on split and gain - typically I would remove/modify feature coming in bottom 5 in both the lists\n\n - correlation with reordered: not a good technique since only linear relationship would get captured\n\n - correlations within independent variables: highly correlated variables (corr > 95%) don't add much to the model may even adversely impact the performance\n\n - ratio of variable mean in train and test: it was mostly close to 1 for this competition but I was still able to improve on LB a little bit by removing a few features with higher deviation\n\n - for continuous features, segment them into 20-50 buckets and measure diversity in the mean of y. I used kinetic energy to capture diversity but entropy works too. For binary variables something like KL divergence can be used. Higher diversity is better.\n\nwith 170 features and after extensive cleaning it became very difficult same variable would come important with one metric but useless with other. In the end I think best way was to remove a few features and see the model performance but that is too time consuming.\n\nHope this helps.\n\nRegards,", "votes": 9 }, { "id": 214016, "postDate": "2017-08-15T18:14:28.387Z", "content": "@SVJ24, thanks for sharing! That's quite some interesting metrics for measuring the feature importance. Especially the ratio of variable mean in train and test.
", "rawMarkdown": "@SVJ24, thanks for sharing! That's quite some interesting metrics for measuring the feature importance. Especially the ratio of variable mean in train and test. " } ] }, { "id": 213979, "postDate": "2017-08-15T16:18:44.150Z", "content": "Thanks for your sharing.I ignored product name.
", "rawMarkdown": "Thanks for your sharing.I ignored product name." }, { "id": 213954, "postDate": "2017-08-15T15:13:46.623Z", "content": "Congratulations. \nLearned a lot from your posts and discussions before. \nThank you!
", "rawMarkdown": "Congratulations. \nLearned a lot from your posts and discussions before. \nThank you!" } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T02:02:38Z", "competition_id": 6644, "competition_slug": "basket-analysis", "team_id": 674088, "solution_url": "/competitions/basket-analysis/writeups/svj24-my-learnings-from-18th-place-solution" }