# 9th place Approach

Congrats to the winners - great work!
I am committed to sharing my approach every time I get a good position. I will share my approach even though I am pretty sure the top competitors' approaches will be far more useful. Generally I did not expect to get that high, given that I never even managed to make the metric working internally - I was optimising using
log loss
the whole time . By the way, if anybody has an implementation of f1_score (not the one is sklearn that seemed to be off in my case) for python that would be great! No wonder I over-fitted private LB a bit as I used public LB to find the right cut-off to the F1 optimizer.
(I also shamelessly invite you to follow
[my new FB page](https://www.facebook.com/StackNet/)
where I will share insight about using data science tools and approaches)
Using the forums/kernels has been very useful in this competition . My score has been greatly boosted by:
[Farons F1 optimizer script](https://www.kaggle.com/mmueller/f1-score-expectation-maximization-in-o-n)
[Sh1ng fetaures](https://www.kaggle.com/c/instacart-market-basket-analysis/discussion/37697)
and
[arboretum](https://github.com/sh1ng/arboretum)
too
[Faron's Streaks](https://www.kaggle.com/mmueller/order-streaks-feature)
Features from
[this script](https://www.kaggle.com/paulantoine/light-gbm-benchmark-0-3692/code)
And the features from
[this script](https://www.kaggle.com/nickycan/lb-0-3805009-python-edition)
I give big credit for arboretum. I found it (obviously faster and) more accurate than xgboost ( but not by much) . Lightgbm was still my best (and faster) sinlge model on a set of 300 features (all the above plus some of mine ) with these params:
LightgbmClassifier boosting:dart num_leaves:22 num_iterations:3000  max_drop:5 drop_rate:0.01 categorical_feature:0,1,2* learning_rate:0.12 threads:12 max_depth:7 feature_fraction:0.7 min_data_in_leaf:10  bagging_fraction:0.85 lambda_l1:1 lambda_l2:0.01 bagging_freq:1 seed:1 verbose:false
categorical_feature:0,1,2  are product, aisle and departement.
I made a separate simple model to predict
None or not
for each offer and fed it to the f1 optimiser.
My own (extra) features included:
counts of purchased items/aisle/deps per customer in different intervals . (like last 15, 30 days etc)
Timing features. like when was the last time the item was bought.
Likelihoods on various categorical features including products, aisles, deps
Cart position of the item and features around this area were also useful. Average position of item and  features that show the shift in cart position over time
Cardinality features like number of different departments, products, asiles the customer shops from
User-based features showing the tendency of buying new items -e.g average proportion of new items in each order.
Things that did not work:
Making a model per product/aisle/department.
Collaborative filtering
Exploring the description of items. E.g text analysis was useless.
For Stacking I used both StackNet and separate Stacking (e.g. used the output of Stacknet on another meta - see below) . I ensembled around 15 models - mostly lightgbms ,
[H2o algos](http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science.html)
and
[keras](https://keras.io/)
' nns.  However i did not get that much from Stacking in this competition. It was much more important the metric's optimisation and the feature engineering.
My best final meta model was an NN using keras and theano. The structure was this:
models = Sequential()
models.add(Dense(output_dim=1024, input_dim=input_d, init='lecun_uniform')) 
models.add(Activation('relu')) 
models.add(BatchNormalization())    
models.add(Dropout(0.5))  
models.add(Dense(512, activation='relu',init='lecun_uniform'))
models.add(Activation('relu')) 
models.add(BatchNormalization())    
models.add(Dropout(0.4))  
models.add(Dense(2, init='lecun_uniform'))
models.add(Activation('softmax'))    
opt = optimizers.Adam(lr=0.01)
models.compile(loss='binary_crossentropy', optimizer=opt)
For the f1 optimizer I used
probabilities higher than 0.005.