# 100th place : Ensemble public notebooks

Since every attempts to pre-train Roberta failed for me, I used publicly pre-trained robertas from
[@maunish](https://www.kaggle.com/maunish)
and
[@tutty1992](https://www.kaggle.com/tutty1992)
. I ensembled roberta-base and roberta-large with different strategy, and here are the methods.
1.  Roberta-Base + SVR
Fine-tuning
[@maunish](https://www.kaggle.com/maunish)
's model and predicting with linear layer, I could get
Private
Public
0.465
0.466
Then I tried to predict with SVR 5-folds wrt training data without using the linear layer. After fine-tuning the regularization parameters for each fold, I could get a slightly better score,
Private
Public
0.464
0.464
2.  Roberta-Large with different poolers and seeds
Fine-tuning
[@tutty1992](https://www.kaggle.com/tutty1992)
's model and referring to
[@jcesquiveld](https://www.kaggle.com/jcesquiveld)
, I tried to ensemble Roberta-Large models with different poolers with different seeds. Although each model is around 0.468 on public/private leaderboard, ensembling 6 different models (2 different poolers and 3 different seeds) led to higher performances as I thought,
Private
Public
0.463
0.463
Besides, I've also ensembled public notebooks and datasets from
[@rhtsingh](https://www.kaggle.com/rhtsingh)
,
[@chamecall](https://www.kaggle.com/chamecall)
,
[@rajat95gupta](https://www.kaggle.com/rajat95gupta)
and
[@nomlmean](https://www.kaggle.com/nomlmean)
to get a score
Private
Public
0.457
0.457
The codes will be made pubic after reference checkup. Hard work everyone, and thank you for everybody who shared great insights and their excellent works!!