# Faron's 3rd Place Solution

Hi fellow Kagglers,
first of all, I want to congratulate
[Bishwarup B](https://www.kaggle.com/bishwarup)
for winning this competition. Kudos! I also want to thank all the diligent forum & kernel contributors: There are alot of interesting discussions and valuable kernels, which made this competition even more interesting and a good learning trip. Besides, my special thanks go to Marios (
[KazAnova)](https://www.kaggle.com/kazanova)
and Henk (
[Triskelion](https://www.kaggle.com/triskelion)
), from whom I learnt the essential techniques to place well in kaggle competitions (without those guys I wouldn't write this post right now).
Short summary of my final solution
I ended up with using XGB and Keras exclusively for my final solution, which is an 
ensemble of around 100 base models (70% XGB & 30% Keras models). The test set predictions have been generated by a 20-times bagged Keras model with one hidden layer as stacker at the 2nd level.
Cross validation
I used 5-Fold unstratified CV (same folds has been used at base and 2nd level).
Best single model
My best single models are adapted from the public XGB & Keras kernels (early stopping removed, creation of oof-predictions). The best CV-score I got is  1124.080 (1107.44746 @ public LB) for an xgb based on the
[Encoding + Feature Comb (modkzs) (1108.72665)](https://www.kaggle.com/misfyre/allstate-claims-severity/encoding-feature-comb-modkzs-1108-72665/)
kernel by
[Misfyre](https://www.kaggle.com/misfyre)
after post processing the predictions via
yhat_opt = x1 * yhat ** x2 + x3 (yhat CV: 1125.876, public LB: 1108.95367)
In general I did not spend much time in single model tuning, because I found it easier to improve the 2nd-level CV-score by training models on different input data and/or with transformed target.
Input data
Feature engineering  like summary statistics did not help me much and hence I concentrated on different representations of the given features and their interactions.
processing on categoricals:
one hot encoding
label encoding (in lexicographical order)
creation of interactions (up to 4-way)
target loss encoding
tf-idf encoding
XGB-embedding (leaf indices of single tree with depth >75 and min_child_weight > 100 used as feature for Neural Networks)
Keras-embedding via embedding layers
removal of rare values
count of values
processing on numericals:
quantile-based binning
for NN: standard scaling, min-max scaling, Box-Cox transformation, percentile transformation
target loss encoding (except
cont14
)
count of values
Used target transformations
y
log(y + n)
sqrt(y)
boxcox(y)
y => [0,1] (for logistic regression)
Base models
XGB:
Around 70 models trained on different combinations of transformed target loss, input data and objective functions (including fairobj, reglinear, gblinear, gammareg, logreg, poission on counts)
NN:
Around 30 models trained on different combinations of transformed target loss and input data. The objective has either been "mae" or  pin-ball loss (quantile regression).
Ensembling
For me, a bagged Keras model provided the best CV-score at the 2nd level. I also noticed that Keras was the only model, which was able to correct the systematical errors all my base models suffered from. That is, there was no more gain in post processing the 2nd level predictions and CV-wise it made no difference whether the base models predictions had been post processed / optimized or not.
XGB performed around 3 MAE worse here for me and did not blend with the Keras stacker (neither did ET, RF nor friends). Another interesting observation has been, that in contrast to the base models, I was only able to get a good CV with Keras at the 2nd level when training on the  orginal scaled target loss.
Final model selection
This was easy this time, because my best local CV model also provided the best LB score.
cheers,
Faron