# 35th place: squeezing out the last -0.0011 RMSE

I will not bore everyone with a complete overview of my 35th place solo solution.  I can't wait to read over the actual top places solutions, especially
[Little Boat](https://www.kaggle.com/xiaozhouwang)
's who I have learned so much from!!  Instead, I will go over how I squeezed the last -0.0011 out of my RMSE score, -0.0001 to -0.0003 at a time.  It took a lot of experimenting to find what worked.
First a quick overview of my basic approach.  I did a lot of feature engineering ending up with 170 numerical features derived from text properties, image properties, mining of test_active/train_active, price encodings, categorical stats, polynomial features, etc.
I had a team of GBT models (XGBoost and LightGBM) and neural net models (Keras).  My word embedding was created entirely from Avito text.  My GBT's included Tfidf and Countv on text, as well as
[TfidfEmbeddingVectorizer](https://github.com/nadbordrozd/blog_stuff/blob/master/classification_w2v/benchmarking_python3.ipynb)
features.  My NN's were BiLSTM/Conv1D and Conv1D/Conv1D similar to those in my DonorsChoose.org playground competition
[1st place solution](https://www.kaggle.com/shadowwarrior/1st-place-solution)
.
At first I linearly stacked these models using out-of-fold predictions.   I was at something like RMSE 0.2178 on the PLB, and searching for how to further reduce my RMSE.  What worked:
1.) -0.0002: I did an DataFrame corr() on all of my numerical features, and pruned out features with correlations above 0.92.
2.) -0.0003: For diversity I added into my stacker LR, GBT and NN models that worked exclusively on just numerical features, on just categorical features, on just text features and on just images.  Individually these models scored poorly, but they helped the stack.
3.) -0.0002: Non-linear stacking with XGBoost
4.) -0.0002: Add in categorical features and price into the stacker, which turn out to be helpful segmenting the predictions.
5.) -0.0001: Using CatBoost as non-linear stacker!  CatBoost didn't work well for me as a base model, but it's target encoding of categoricals helped with stacking.
6.) -0.0001: Stacker of stackers!  I stacked slight variations of my best XGBoost and CatBoost stackers.
Thank you to Kaggle & Avito for a fun competition, and fellow Kagglers for pushing me to squeeze out those last -0.0001 RMSE's!