# 10th Place Solution

Interesting competition in a time when tabular competitions on Kaggle are quite rare and based only on synthetically generated data (oh, but also this one is a synthetic one). In addition, this competition has been somewhat reminiscent of older times for me, because I don't work anymore in insurance, having shifted to finance (instead my teammate, Matteo, still works in the sector).
Since now the competition has closed, I can reveal that I found the competition a bit puzzling because CV and public LB were always so different (unless you kept a single outlier in your training set) and now again the private has complete different values. Having seen more than 200 competitions up so far, I don't get much surprised anymore that so often public and private scores are not consistent in respect each other and that some shake-up happens. Another puzzling aspect of the completion was the task to predict on the same period of the training data. Though it is not uncommon to have to predict on the same distribution of the training data, a more realistic task would have been to predict on a subsequent period (for instance in our case on year 2006).
Anyway, here are the characteristics of our solution, which is a somehow computational intensive one due to example bootstrapping (bagging), but we dealt with it completely using Kaggle Kernels. Probably we won’t release more of our solution than what you are going to read here, unless we decide to write a paper on it.
Our solution is a blending of bagged 2nd level stacking solutions. We built many diverse models (Catboost, Lightgbm, XGBoost, Random Forest, Regularized Greedy Forests, simple neural DNN, Tabnet, Node, K-Nearest Neighbors, GLMs, Support Vector Machines) always using the cross-validation-prediction approach on the same seed (
[https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_predict.html)](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_predict.html))
.
We used a custom stratified cross-validation to reproduce the training distribution accurately.
We excluded two outlying observations based on target and features' values.
As for as feature engineering, we used both the original features and some features derived from autoencoders processing with and without multi-head attention.
We also "cleaned" key predictive variables by custom rule based functions and using some dedicated models.
We extracted further information from the target using dedicated models and reused it as features.
In the end, we devised quite a few features based on existing ones, especially features based on time since we realized that time mattered (not only both accident and claim but also the order of records).
As for the text, we used the simple raw counting of words and their n-grams and a projection of descriptions to WCIO actuarial tables using GloVe embeddings.
Each model has been run multiple times on bootstraps of the training data and then all the models were put into ensemble using a constrained linear model.
We repeated once more all the above operations using the first models in order to feed the second level ones.
Finally, we blended the first and second level models coming from different runs based on different random seeds.
We submitted the best CV model and the best LB model. We noticed that "post-processing" worked on the submission on LB, that is, you could adjust your predictions using coefficients based on accident time and your LB improved (but our CV worsened, of course). We deducted that such happened as a kind of overfitting on the tails of the public LB and we decided not to use any post-processed submission (correctly, because none of them performed better on the private).
Before submitting, we tested that our model's residuals had no time correlation and that residuals were correctly centered and distributed. We couldn't find a way to reduced the variance of the residuals of accidents from 2001-2002 onward (which was much higher than the previous years) and we believe, given the short distance of our solution to the top ones, that there is little way of doing it due to the some underlying stochastic process or other latent generative process.