# RNN Solution - top 50

Haven't seen a lot of neural network approaches posted so far, so I thought I'd share mine. I think this model could have been pushed a lot further in terms of rank, but I got a bit burnt out toward the end, and didn't spend a lot of time optimizing. (I only did 5 submissions total, and they used a model trained on my training fold, which had around 10k users held out).
Code is
[here](https://github.com/colinmorris/instacart-basket-prediction)
. The core of the rnn model is
[here](https://github.com/colinmorris/instacart-basket-prediction/blob/master/baskets/rnnmodel.py)
(implemented in tensorflow).
Basically, for each user, for each product
p
in that user's history, I generate a training sequence, where the RNN predicts 0/1 at each step for "didn't order"/"did order".
At each time step, the RNN gets some inputs describing the current order (day of week, hour of day...) and the previous order (number of products, whether
p
was in the order...). I also train embeddings for products (and departments + aisles) as part of the RNN. The embedding for
p
is repeated as input to the RNN at each time step.
Because the goal is just to predict *
re
*orders, I apply a mask to zero out the loss for all orders up to and including the first one containing
p
.
A few notes of interest:
The model is trained to predict
all
reorders - the final order isn't treated any differently from the rest. I tried implementing a 'finetuning' mode where I would train an existing model for a few epochs with only the final order counting toward the loss, but it seemed to make things go haywire. I think the sparse learning signal led to some exploding/vanishing gradients? Would have liked to explore this more.
Anecdotally, sampling/weighting at the user level seemed to be essential to getting good results with this model - which didn't seem to be the case in my experiments with xgboost and other non-sequential methods. I had good results with the simple approach of sampling a single product per user in each epoch (the final code does something a little more complex for performance reasons). One good argument for doing this is that prolific users will generate more training sequences (because they have more products),
and
each sequence will be longer than average. So naively iterating through the dataset would lead to the model spending orders of magnitude more time on some users than others.
A nice aspect of using a recurrent model was that I was able to get away with very few/simple features. For example, I didn't bother to write any features encoding frequency/recency because the model should be able to figure it out based on the sequence of previous labels observed. I basically only had about 6 features + product/aisle/dept embeddings.
My submissions were a blend of RNN predictions + a boring gradient boosting model, with expected F1 optimization. (For your amusement, my repo contains an
[O(2^n)](https://github.com/colinmorris/instacart-basket-prediction/blob/master/baskets/fscore.py)
and an
[O(n^3)](https://github.com/colinmorris/instacart-basket-prediction/blob/master/baskets/clever_fscore.py)
implementation of expected F-score that predate the great spoiler kernel.)
The repo has a few other models I played with that didn't make the cut (factorization machines with libfm, learning pairwise product interactions with logistic regression), mostly because I ran out of time scrambling to put together my submissions on the last day :)
This was my first Kaggle competition, and I have to say, I had a lot of fun and I learned a
lot
. Thanks to everyone on the forums for the interesting ideas and discussions.