# 136th place solution (top 4% - silver)

Our solution is an ensemble of five different architectures, some manual empiric tuning and some seeds re-runs to help stabilizing the predictions and generalizing to the private test set.
Ensemble:
2 roberta-base models, each one with a different head (5 fold CV each)
ExtraTreesRegressor on the embeddings extracted from roberta-base (5 fold CV, 2 seeds)
1 roberta-large (5 fold)
1 Roberta-large Mean Pooling model (5 fold, 2 seeds)
Total: 35 models.
All Hugging Face models were further pre-trained in the competition dataset, before the fine-tuning step.
Seed average helped improving ~0.002 on public LB score.
I want to thank some kagglers that were important to this score:
[@andretugan](https://www.kaggle.com/andretugan)
,
[@maunish](https://www.kaggle.com/maunish)
,
[@rhtsingh](https://www.kaggle.com/rhtsingh)
, and
[@rajat95gupta](https://www.kaggle.com/rajat95gupta)
. There were many others that helped one way or the other, by publicly sharing their works and thoughts.
Special thanks to my teammate
[@jbomitchell](https://www.kaggle.com/jbomitchell)
, who kept us updated with the most important public discussions and notebooks shared in this competition and also provided many valuable insights about the competition and train/test data.
Finally, thanks to everyone who joined, hosted or was a part of this competition!
Code is
[here](https://www.kaggle.com/hinepo/clpr-5-models-v1-1)
.