{ "id": 38161, "title": "8th place solution - overview", "content": "

Thanks to the organizers (I <3 well sampled test sets). Also thanks to Faron for a beautiful script and making me read an interesting paper.

\n\n

In general I mostly used lightgbm in combination with a separate model for predicting none and f1 optimization. Additionally i found that due to the structure of predicting unique user product combinations, the model didn't really have a notion of user_id (order_id) groupings. Therefore adding features made from grouping of features as well as out-of-fold model predictions on a user_id basis helped quite a bit (literally taking features.groupby('user_id').mean()). Below I tried to list some of the other, maybe more interesting features I used.

\n\n

I definitely learned a lot during this competition. One of them being: you can do (almost) anything in pandas in reasonable time if you spend enough time on planning and vectorizing :)

\n\n

Features:

\n\n\n\n

Model tuning:

\n\n", "comments": [ { "id": 214409, "author_name": "Sameh Faidi", "author_url": "", "post_date": "2017-08-16T20:32:28.557000", "content": "

Congrats. Would you explain how did you create the association features? Also how much boost did the averaging of models with different sorting of features ?

", "votes": 1, "replies": [ { "id": 214418, "author_name": "gKucsko", "author_url": "", "post_date": "2017-08-16T20:51:14.807000", "content": "

thanks, I used python's mlextend package which includes association rule handling. most of the stuff you will easily find is organic products like bananas and strawberries bought together. but when you filter to high confidence and lift there are some other more interesting patterns like salt & pepper. so for those products, let's call them A and B I added a feature: if i want to predict whether client 007 bought A i add my out of fold model prediction for that client for product B (and -1 if he never bought B before). for feature sorting i think i saw variation on the order of 0.0002 in logloss

", "votes": 1, "replies": [] } ] }, { "id": 214422, "author_name": "Jing Lu", "author_url": "", "post_date": "2017-08-16T20:59:40.997000", "content": "

Congratulations, Kucsko! The computing time really bothers me. Would you care to elaborate \"you can do (almost) anything in pandas in reasonable time if you spend enough time on planning and vectorizing\"?

", "votes": 0, "replies": [ { "id": 214547, "author_name": "gKucsko", "author_url": "", "post_date": "2017-08-17T10:07:18.567000", "content": "

thanks! let me try to give a specific example. using days since last order purchased we can do a groupby cumsum shift to get something like a user specific order time with date granularity. next we want to know what is the mean difference in days between repurchases of each user product combination. for this we could do a groupby user+product apply diff, however this is extremely slow. instead we can be a bit clever and do a groupby user+product shift and then take the difference between the shifted an unshifted vector. after that we can easily take mean, med, std etc. the reason is that there are a lot of user product combinations and operations become very expensive, but applying shift is very fast and then we can take the vectorized difference, which again is very fast.\nrule of thumb, apply with axis=1 is evil and rarely really needed. groupby apply is often needed but should still be done with care, because some functions are a lot more expensive than others. especially i the return type is not a constant but a series or dataframe. always try to verctorize! :)

", "votes": 3, "replies": [] }, { "id": 214628, "author_name": "Jing Lu", "author_url": "", "post_date": "2017-08-17T17:04:20.883000", "content": "

Thanks!

", "votes": 0, "replies": [] } ] }, { "id": 214201, "author_name": "yue yu", "author_url": "", "post_date": "2017-08-16T07:54:38.863000", "content": "

Thanks a lot for sharing.\nI read it carefully. But due to my poor English, I'm not quite sure I understand your Features part correctly.\n\" note that a lot of this becomes much more efficient in dense representation using order_number\", does the \"using order_number\" mean \"combined with order number 1_2762\"?And would you like to give a specific example?

", "votes": 0, "replies": [ { "id": 214421, "author_name": "gKucsko", "author_url": "", "post_date": "2017-08-16T20:58:20.093000", "content": "

what i mean is, let's say we are interested in user 66 and product A. we can list the order history as either [1, 0, 0, 0, 1, 0] sorted by order_number (sparse representation), or we can list it as the actual order numbers [1,5] (dense representation). the second usually takes up less memory and is faster for computing order statistics. similarly in an extreme case you could pivot your whole table to be order_number * user_id * product_id, and fill all non occurring cells with nans, but of course that's a waste of resources and the wrong representation of this data

", "votes": 2, "replies": [] } ] } ], "comment_count": 7, "messages": [ { "id": 214057, "postDate": "2017-08-15T20:51:17.220Z", "content": "

Thanks to the organizers (I <3 well sampled test sets). Also thanks to Faron for a beautiful script and making me read an interesting paper.

\n\n

In general I mostly used lightgbm in combination with a separate model for predicting none and f1 optimization. Additionally i found that due to the structure of predicting unique user product combinations, the model didn't really have a notion of user_id (order_id) groupings. Therefore adding features made from grouping of features as well as out-of-fold model predictions on a user_id basis helped quite a bit (literally taking features.groupby('user_id').mean()). Below I tried to list some of the other, maybe more interesting features I used.

\n\n

I definitely learned a lot during this competition. One of them being: you can do (almost) anything in pandas in reasonable time if you spend enough time on planning and vectorizing :)

\n\n

Features:

\n\n\n\n

Model tuning:

\n\n", "rawMarkdown": "Thanks to the organizers (I <3 well sampled test sets). Also thanks to Faron for a beautiful script and making me read an interesting paper. \n\nIn general I mostly used lightgbm in combination with a separate model for predicting none and f1 optimization. Additionally i found that due to the structure of predicting unique user product combinations, the model didn't really have a notion of user_id (order_id) groupings. Therefore adding features made from grouping of features as well as out-of-fold model predictions on a user_id basis helped quite a bit (literally taking features.groupby('user_id').mean()). Below I tried to list some of the other, maybe more interesting features I used.\n\nI definitely learned a lot during this competition. One of them being: you can do (almost) anything in pandas in reasonable time if you spend enough time on planning and vectorizing :)\n\nFeatures:\n\n - various days since metrics. by creating a mapping of order id onto cumsum of days since last order i made various timeseries features such time since purchased, exp weighed moving average of last few days, days since last none order etc\n - various moments of distributions of product orders within set all user orders, such as mean ratio, skewness etc. note that a lot of this becomes much more efficient in dense representation using order_number\n - various frequency metrics, such as: user based median days between product repurchases minus # of days since user last bought the product. gives you a metric on whether a user is on-cycle or off-cycle to repurchase that product\n - association rule learning: allows you to find out things like: salt is usually bought together with pepper. problem is you can only reliably use such features for products that are bought rarely. this revealed some interesting patterns, but didn't help all too much in logloss\n - use prior error in prediction. a common tactic for timeseries predictions is to take the prediction error of a model for the previous user order: I kicked out all train orders and reordered everything so that the last order for each user becomes the train set. we can then extract an error for each user prediction. seemed to work but not help much in the basic model sadly :(\n - make user order 'sentences' out of product id purchases combined with order number. eg user 0 ordered: 1_2762 1_298 1_6378 2_298... this allows you to do things like dim reduction etc to get user clusters as well as other cross product features using sparse lgb. worked sensibly well but didn't seem very additive to manual features\n\nModel tuning:\n\n - small learning rate: even though it takes much longer, 0.01 seemed to give me significantly better results\n - average over feature samples: since i used feature fraction 0.5 i had remarkable variation in logloss depending on sorting of features. therefore i took the mean over multiple models with varying sorting of features\n - categorical features: lgb now offers a bunch of tuning metrics to reduce chances of inevitable overfitting with high cardinality categoricals. i played with this and didn't get huge success, but better than nothing", "votes": 27 }, { "id": 214409, "postDate": "2017-08-16T20:32:28.557Z", "content": "

Congrats. Would you explain how did you create the association features? Also how much boost did the averaging of models with different sorting of features ?

", "rawMarkdown": "Congrats. Would you explain how did you create the association features? Also how much boost did the averaging of models with different sorting of features ?", "votes": 1, "replies": [ { "id": 214418, "postDate": "2017-08-16T20:51:14.807Z", "content": "

thanks, I used python's mlextend package which includes association rule handling. most of the stuff you will easily find is organic products like bananas and strawberries bought together. but when you filter to high confidence and lift there are some other more interesting patterns like salt & pepper. so for those products, let's call them A and B I added a feature: if i want to predict whether client 007 bought A i add my out of fold model prediction for that client for product B (and -1 if he never bought B before). for feature sorting i think i saw variation on the order of 0.0002 in logloss

", "rawMarkdown": "thanks, I used python's mlextend package which includes association rule handling. most of the stuff you will easily find is organic products like bananas and strawberries bought together. but when you filter to high confidence and lift there are some other more interesting patterns like salt & pepper. so for those products, let's call them A and B I added a feature: if i want to predict whether client 007 bought A i add my out of fold model prediction for that client for product B (and -1 if he never bought B before). for feature sorting i think i saw variation on the order of 0.0002 in logloss", "votes": 1 } ] }, { "id": 214422, "postDate": "2017-08-16T20:59:40.997Z", "content": "

Congratulations, Kucsko! The computing time really bothers me. Would you care to elaborate \"you can do (almost) anything in pandas in reasonable time if you spend enough time on planning and vectorizing\"?

", "rawMarkdown": "Congratulations, Kucsko! The computing time really bothers me. Would you care to elaborate \"you can do (almost) anything in pandas in reasonable time if you spend enough time on planning and vectorizing\"?", "replies": [ { "id": 214547, "postDate": "2017-08-17T10:07:18.567Z", "content": "

thanks! let me try to give a specific example. using days since last order purchased we can do a groupby cumsum shift to get something like a user specific order time with date granularity. next we want to know what is the mean difference in days between repurchases of each user product combination. for this we could do a groupby user+product apply diff, however this is extremely slow. instead we can be a bit clever and do a groupby user+product shift and then take the difference between the shifted an unshifted vector. after that we can easily take mean, med, std etc. the reason is that there are a lot of user product combinations and operations become very expensive, but applying shift is very fast and then we can take the vectorized difference, which again is very fast.\nrule of thumb, apply with axis=1 is evil and rarely really needed. groupby apply is often needed but should still be done with care, because some functions are a lot more expensive than others. especially i the return type is not a constant but a series or dataframe. always try to verctorize! :)

", "rawMarkdown": "thanks! let me try to give a specific example. using days since last order purchased we can do a groupby cumsum shift to get something like a user specific order time with date granularity. next we want to know what is the mean difference in days between repurchases of each user product combination. for this we could do a groupby user+product apply diff, however this is extremely slow. instead we can be a bit clever and do a groupby user+product shift and then take the difference between the shifted an unshifted vector. after that we can easily take mean, med, std etc. the reason is that there are a lot of user product combinations and operations become very expensive, but applying shift is very fast and then we can take the vectorized difference, which again is very fast.\nrule of thumb, apply with axis=1 is evil and rarely really needed. groupby apply is often needed but should still be done with care, because some functions are a lot more expensive than others. especially i the return type is not a constant but a series or dataframe. always try to verctorize! :)", "votes": 3 }, { "id": 214628, "postDate": "2017-08-17T17:04:20.883Z", "content": "

Thanks!

", "rawMarkdown": "Thanks!" } ] }, { "id": 214201, "postDate": "2017-08-16T07:54:38.863Z", "content": "

Thanks a lot for sharing.\nI read it carefully. But due to my poor English, I'm not quite sure I understand your Features part correctly.\n\" note that a lot of this becomes much more efficient in dense representation using order_number\", does the \"using order_number\" mean \"combined with order number 1_2762\"?And would you like to give a specific example?

", "rawMarkdown": "Thanks a lot for sharing.\nI read it carefully. But due to my poor English, I'm not quite sure I understand your Features part correctly.\n\" note that a lot of this becomes much more efficient in dense representation using order_number\", does the \"using order_number\" mean \"combined with order number 1_2762\"?And would you like to give a specific example?", "replies": [ { "id": 214421, "postDate": "2017-08-16T20:58:20.093Z", "content": "

what i mean is, let's say we are interested in user 66 and product A. we can list the order history as either [1, 0, 0, 0, 1, 0] sorted by order_number (sparse representation), or we can list it as the actual order numbers [1,5] (dense representation). the second usually takes up less memory and is faster for computing order statistics. similarly in an extreme case you could pivot your whole table to be order_number * user_id * product_id, and fill all non occurring cells with nans, but of course that's a waste of resources and the wrong representation of this data

", "rawMarkdown": "what i mean is, let's say we are interested in user 66 and product A. we can list the order history as either [1, 0, 0, 0, 1, 0] sorted by order_number (sparse representation), or we can list it as the actual order numbers [1,5] (dense representation). the second usually takes up less memory and is faster for computing order statistics. similarly in an extreme case you could pivot your whole table to be order_number * user_id * product_id, and fill all non occurring cells with nans, but of course that's a waste of resources and the wrong representation of this data", "votes": 2 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T02:02:49Z", "competition_id": 6644, "competition_slug": "basket-analysis", "team_id": 722772, "solution_url": "/competitions/basket-analysis/writeups/kucsko-8th-place-solution-overview" }