# [88th place] solution - Team DilBERT

Acknowledgement
First of all, my team-mate
[@mananjhaveri](https://www.kaggle.com/mananjhaveri)
and I would like to thank the competition organizers for organizing this competition.
We would also like to thank
[@rhtsingh](https://www.kaggle.com/rhtsingh)
and
[@andretugan](https://www.kaggle.com/andretugan)
for their amazing notebooks. Special thanks to
[@abhishek](https://www.kaggle.com/abhishek)
for his awesome book called Approaching Almost Any Machine Learning Problem. The chapter on NLP helped us a lot.
The breakthrough (best single-model)
Our best single model was RoBERTa-Large with attention head. We also added layer wise learning rate decay and layer reinitialization used from
[this amazing notebook](https://www.kaggle.com/rhtsingh/on-stability-of-few-sample-transformer-fine-tuning)
which boosted our score.
Configuration for our best single model -
NUM_EPOCHS = 5
NUM_FOLDS = 5
BATCH_SIZE = 12
MAX_LEN = 256
LEARNING_RATE = 3e-5
WEIGHT_DECAY = 1e-2
NUM_WARMUP_STEPS = 50
EVAL_SCHEDULE = [(0.51, 50), (0.49, 25)]
With this, our best single model reached a score of 0.467 in Public LB and 0.468 in Private LB.
The breakthrough (ensemble)
We used 3 models for this -
Model 1 =  LB 0.467 -
[https://www.kaggle.com/andretugan/pre-trained-roberta-solution-in-pytorch](https://www.kaggle.com/andretugan/pre-trained-roberta-solution-in-pytorch)
Model 2 = LB 0.466 -
[https://www.kaggle.com/rajat95gupta/clrp-mean-pooling-inference](https://www.kaggle.com/rajat95gupta/clrp-mean-pooling-inference)
Model 3 = Our best single model
Model
Ratio
Model 1
0.35
Model 2
0.35
Model 3
0.3
This gave us a score of 0.457 on public as well as private LB.
A big thank you to the Kaggle community for being so supportive. This was our first competition where we landed in the medal zone and it was a very exciting journey. We look forward to more competitions like this and hopefully contribute to this community.