Haven't seen a lot of neural network approaches posted so far, so I thought I'd share mine. I think this model could have been pushed a lot further in terms of rank, but I got a bit burnt out toward the end, and didn't spend a lot of time optimizing. (I only did 5 submissions total, and they used a model trained on my training fold, which had around 10k users held out).
\n\nCode is here. The core of the rnn model is here (implemented in tensorflow).
\n\nBasically, for each user, for each product p in that user's history, I generate a training sequence, where the RNN predicts 0/1 at each step for \"didn't order\"/\"did order\".
At each time step, the RNN gets some inputs describing the current order (day of week, hour of day...) and the previous order (number of products, whether p was in the order...). I also train embeddings for products (and departments + aisles) as part of the RNN. The embedding for p is repeated as input to the RNN at each time step.
Because the goal is just to predict *re*orders, I apply a mask to zero out the loss for all orders up to and including the first one containing p.
A few notes of interest:
\n\nMy submissions were a blend of RNN predictions + a boring gradient boosting model, with expected F1 optimization. (For your amusement, my repo contains an O(2^n) and an O(n^3) implementation of expected F-score that predate the great spoiler kernel.)
\n\nThe repo has a few other models I played with that didn't make the cut (factorization machines with libfm, learning pairwise product interactions with logistic regression), mostly because I ran out of time scrambling to put together my submissions on the last day :)
\n\nThis was my first Kaggle competition, and I have to say, I had a lot of fun and I learned a lot. Thanks to everyone on the forums for the interesting ideas and discussions.
", "comments": [ { "id": 214058, "author_name": "Yifan Xie", "author_url": "", "post_date": "2017-08-15T20:53:06.827000", "content": "Brilliant approach, thanks for sharing and I look forward learn from your code about how to implement RNN :)
", "votes": 1, "replies": [] }, { "id": 216553, "author_name": "rainbow36", "author_url": "", "post_date": "2017-08-26T16:03:19.340000", "content": "hey can you please tell me how to learn to build the RNN architecture as you have done it?
", "votes": 0, "replies": [ { "id": 216986, "author_name": "ColinMorris", "author_url": "", "post_date": "2017-08-28T21:18:18.240000", "content": "In terms of the implementation, I got started by reading through this code from Google's sketch-rnn model and trying to understand it. I used that class as a template when I implemented my RNN, and it helped a lot.
\n\nIn terms of the theory behind how they work, I found this blog post very helpful (especially the diagrams). Though I think you can get along pretty well even if you don't understand their inner workings and mostly treat them as a black box.
", "votes": 1, "replies": [] } ] }, { "id": 3253610, "author_name": "", "author_url": "", "post_date": "2025-07-25T02:02:26.457000", "content": "", "votes": 0, "replies": [] }, { "id": 214085, "author_name": "Charles Jansen", "author_url": "", "post_date": "2017-08-15T23:05:58.257000", "content": "great! thanks!
", "votes": 1, "replies": [] } ], "comment_count": 5, "messages": [ { "id": 214047, "postDate": "2017-08-15T20:24:57.547Z", "content": "Haven't seen a lot of neural network approaches posted so far, so I thought I'd share mine. I think this model could have been pushed a lot further in terms of rank, but I got a bit burnt out toward the end, and didn't spend a lot of time optimizing. (I only did 5 submissions total, and they used a model trained on my training fold, which had around 10k users held out).
\n\nCode is here. The core of the rnn model is here (implemented in tensorflow).
\n\nBasically, for each user, for each product p in that user's history, I generate a training sequence, where the RNN predicts 0/1 at each step for \"didn't order\"/\"did order\".
At each time step, the RNN gets some inputs describing the current order (day of week, hour of day...) and the previous order (number of products, whether p was in the order...). I also train embeddings for products (and departments + aisles) as part of the RNN. The embedding for p is repeated as input to the RNN at each time step.
Because the goal is just to predict *re*orders, I apply a mask to zero out the loss for all orders up to and including the first one containing p.
A few notes of interest:
\n\nMy submissions were a blend of RNN predictions + a boring gradient boosting model, with expected F1 optimization. (For your amusement, my repo contains an O(2^n) and an O(n^3) implementation of expected F-score that predate the great spoiler kernel.)
\n\nThe repo has a few other models I played with that didn't make the cut (factorization machines with libfm, learning pairwise product interactions with logistic regression), mostly because I ran out of time scrambling to put together my submissions on the last day :)
\n\nThis was my first Kaggle competition, and I have to say, I had a lot of fun and I learned a lot. Thanks to everyone on the forums for the interesting ideas and discussions.
", "rawMarkdown": "Haven't seen a lot of neural network approaches posted so far, so I thought I'd share mine. I think this model could have been pushed a lot further in terms of rank, but I got a bit burnt out toward the end, and didn't spend a lot of time optimizing. (I only did 5 submissions total, and they used a model trained on my training fold, which had around 10k users held out).\n\nCode is [here](https://github.com/colinmorris/instacart-basket-prediction). The core of the rnn model is [here](https://github.com/colinmorris/instacart-basket-prediction/blob/master/baskets/rnnmodel.py) (implemented in tensorflow). \n\nBasically, for each user, for each product `p` in that user's history, I generate a training sequence, where the RNN predicts 0/1 at each step for \"didn't order\"/\"did order\". \n\nAt each time step, the RNN gets some inputs describing the current order (day of week, hour of day...) and the previous order (number of products, whether `p` was in the order...). I also train embeddings for products (and departments + aisles) as part of the RNN. The embedding for `p` is repeated as input to the RNN at each time step.\n\nBecause the goal is just to predict **re**orders, I apply a mask to zero out the loss for all orders up to and including the first one containing `p`. \n\nA few notes of interest:\n\n- The model is trained to predict *all* reorders - the final order isn't treated any differently from the rest. I tried implementing a 'finetuning' mode where I would train an existing model for a few epochs with only the final order counting toward the loss, but it seemed to make things go haywire. I think the sparse learning signal led to some exploding/vanishing gradients? Would have liked to explore this more.\n- Anecdotally, sampling/weighting at the user level seemed to be essential to getting good results with this model - which didn't seem to be the case in my experiments with xgboost and other non-sequential methods. I had good results with the simple approach of sampling a single product per user in each epoch (the final code does something a little more complex for performance reasons). One good argument for doing this is that prolific users will generate more training sequences (because they have more products), *and* each sequence will be longer than average. So naively iterating through the dataset would lead to the model spending orders of magnitude more time on some users than others.\n- A nice aspect of using a recurrent model was that I was able to get away with very few/simple features. For example, I didn't bother to write any features encoding frequency/recency because the model should be able to figure it out based on the sequence of previous labels observed. I basically only had about 6 features + product/aisle/dept embeddings.\n\nMy submissions were a blend of RNN predictions + a boring gradient boosting model, with expected F1 optimization. (For your amusement, my repo contains an [O(2^n)](https://github.com/colinmorris/instacart-basket-prediction/blob/master/baskets/fscore.py) and an [O(n^3)](https://github.com/colinmorris/instacart-basket-prediction/blob/master/baskets/clever_fscore.py) implementation of expected F-score that predate the great spoiler kernel.)\n\nThe repo has a few other models I played with that didn't make the cut (factorization machines with libfm, learning pairwise product interactions with logistic regression), mostly because I ran out of time scrambling to put together my submissions on the last day :)\n\nThis was my first Kaggle competition, and I have to say, I had a lot of fun and I learned a *lot*. Thanks to everyone on the forums for the interesting ideas and discussions.", "votes": 43 }, { "id": 214058, "postDate": "2017-08-15T20:53:06.827Z", "content": "Brilliant approach, thanks for sharing and I look forward learn from your code about how to implement RNN :)
", "rawMarkdown": "Brilliant approach, thanks for sharing and I look forward learn from your code about how to implement RNN :)", "votes": 1 }, { "id": 216553, "postDate": "2017-08-26T16:03:19.340Z", "content": "hey can you please tell me how to learn to build the RNN architecture as you have done it?
", "rawMarkdown": "hey can you please tell me how to learn to build the RNN architecture as you have done it?", "replies": [ { "id": 216986, "postDate": "2017-08-28T21:18:18.240Z", "content": "In terms of the implementation, I got started by reading through this code from Google's sketch-rnn model and trying to understand it. I used that class as a template when I implemented my RNN, and it helped a lot.
\n\nIn terms of the theory behind how they work, I found this blog post very helpful (especially the diagrams). Though I think you can get along pretty well even if you don't understand their inner workings and mostly treat them as a black box.
", "rawMarkdown": "In terms of the implementation, I got started by reading through [this code](https://github.com/tensorflow/magenta/blob/master/magenta/models/sketch_rnn/model.py) from Google's sketch-rnn model and trying to understand it. I used that class as a template when I implemented my RNN, and it helped a lot.\n\nIn terms of the theory behind how they work, I found [this blog post](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) very helpful (especially the diagrams). Though I think you can get along pretty well even if you don't understand their inner workings and mostly treat them as a black box.", "votes": 1 } ] }, { "id": 3253610, "postDate": "2025-07-25T02:02:26.457Z", "rawMarkdown": "", "isDeleted": true }, { "id": 214085, "postDate": "2017-08-15T23:05:58.257Z", "content": "great! thanks!
", "rawMarkdown": "great! thanks!", "votes": 1 } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T02:02:12Z", "competition_id": 6644, "competition_slug": "basket-analysis", "team_id": 734194, "solution_url": "/competitions/basket-analysis/writeups/colinmorris-rnn-solution-top-50" }