# Private 106th / Public 79th Solution [Top 3%]

Firstly, we would like to thanks the competition organizers for hosting such a unique competition, we learned a lot from this competition. This was our first NLP Kaggle Competition and we enjoyed it very much.
Congratulations to all the winning teams!!
Thank you to the Kaggle community for sharing such innovative ideas through notebooks and discussions, where we were able to learn a lot!!
Special thanks to the following users/community for providing amazing ideas and code that we used in our solution:
Thank you to
[@rhtsingh](https://www.kaggle.com/rhtsingh)
for providing so many resources to learn about transformers: how to pre-train them, how to fine-tune them, how to improve fine-tuning [both speed and accuracy]; theory as well as code through your amazing notebooks.
Thank you to
[@maunish](https://www.kaggle.com/maunish)
[for providing insights on Attention head and providing a clean pipeline from Pre-training to fine-tuning to scoring],
[@andretugan](https://www.kaggle.com/andretugan)
[for providing a fine-tuning notebook that solved the problem of stability in CV while training],
[@cdeotte](https://www.kaggle.com/cdeotte)
[for providing a unique method of ensembling using forward OOF technique],
[@ragnar123](https://www.kaggle.com/ragnar123)
[for providing fine-tuning notebook using TF]
[@abhishek](https://www.kaggle.com/abhishek)
[for amazing tutorials on training/finetuning transformers using pytorch],
[@rajat95gupta](https://www.kaggle.com/rajat95gupta)
and
[@jcesquiveld](https://www.kaggle.com/jcesquiveld)
[for providing insights on how to find the optimal head for a particular transformer], the
[Huggingface community](https://discuss.huggingface.co/)
[for solving all the queries albeit being small ones], and
finally my teammates
[@muktan](https://www.kaggle.com/muktan)
and
[@deepangraval](https://www.kaggle.com/deepangraval)
[being part of this 3 months journey with immense enthusiasm]
.
Brief Solution
We used Roberta-base, Roberta-large, Electra-base, and Electra-large with different heads [Mean-Pooing, CLS Token, and Attention], with/without pre-training, with/without layer-reinitialization, Gradient Accumulation, Group Differential LR, and a unique method of using 2D Attention Head that will be discussed by
[@muktan](https://www.kaggle.com/muktan)
in
[this post](https://www.kaggle.com/c/commonlitreadabilityprize/discussion/261206)
. We used one submission based solely on our CV values and another was based on LB score.
1) CV Fit [Private LB: 0.458] - We used the following
[notebook](https://www.kaggle.com/cdeotte/forward-selection-oof-ensemble-0-942-private)
to find the best ensembling score using our oof files and used 14 models which are as follows:
Model-Description
CV
Public LB
Roberta-base using Tensorflow
[@ragnar123](https://www.kaggle.com/ragnar123)
0.509
0.482
Roberta-base
[@rhtsingh](https://www.kaggle.com/rhtsingh)
0.481
0.477
Roberta-large-1
[@rhtsingh](https://www.kaggle.com/rhtsingh)
0.479
0.474
Roberta-large- [No-Pretraining+5 Layer Reinitialization]
0.481
0.473
Roberta-base
[@andretugan](https://www.kaggle.com/andretugan)
0.477
0.467
Roberta-base- [variation of the above model]
0.475
0.469
Roberta-large- [Attention Head with Differential LR]
0.474
0.460
Roberta-large- [Above Model + 5 layer Reinitialization]
0.471
0.463
Roberta-large- [Mean Pooling Head]
0.479
0.469
Roberta-large- [2D Attention for 2 4 6 8 layers from the end-
[@muktan](https://www.kaggle.com/muktan)
]
0.474
0.461
Roberta-large- [Variation of above technique]
0.475
0.465
Electra-large- [Variation of the best LB Roberta-large model]
0.479
0.469
Electra-large- [Variation of the above model]
0.487
0.480
Electra-large- [2D Attention-
[@muktan](https://www.kaggle.com/muktan)
]
0.487
0.474
Overall
0.455
0.457
2) LB Fit [Private LB: 0.458, Public LB: 0.455] - We used more-or-less the same models from the above list but we provided weights based on the LB scores and combined them in a different manner.
Training Config
Optimiser: AdamW
Weight decay: 0.5
LR: 2.5e-5 to 3.5e-5 (depending on batch size) [Group Differential LR]
Epochs: 3-5 [depending on the model]
Batch size: 16 (or 10 for the larger models)
Validation & Checkpointing every 5-10 steps [depending on the model]
Loss: MSE/RMSE
What worked for us
Pre-training Roberta-base [for large models, we were not able to pre-train ]
Differential learning rate [2.5e-5 to 3.5e-5 with a factor of 2.5 depending on the architecture]
Last 5 layer-reinitialization [we tried different techniques but this was the best amongst them]
Gradient Accumulation 2 [BS-8 for training on larger batch size]
Attention head, Mean Pooling head - [We were struggling to find custom heads for fine-tuning and these heads worked very well for the given tasks]
What didn't work for us
External-data pre-training- We tried to pre-train the model on MLM tasks using different external datasets, but we were not achieving good results w.r.t. CV or LB with that.
Stacking- We used the embeddings features from different models and added them to a Ridge/BayesianRidge Regression, but we were not getting good results with that, we concluded that as most of the features are extracted because of how trained our models using Grouped Differential LR, Stacking wasn't working for those models.
External-Features+Stacking- For some of the models, we combined the embeddings with some externals features and tried stacking using AutoML as well as Optuna but were unsuccessful as it worsened our score even more.
Tools Used
We are fresh undergraduate students with no external access to GPUs so we have been using Kaggle Kernels and Google Colab [we used 7 accounts 🤪to perform different experiments for finding hyper-parameters.]
We used Google-Docs from the beginning to first find all the techniques that are available for a readability task by reading different white papers [ML/Statistical/DL/RL] and then moved on to implement one-by-one as per the model-avail. and finally read about improving said models using different fine-tuning techniques.
We used Google-Sheets to track all the progress of the competition, and different models configurations as well.
P.S.
Our main intent was to learn about how transformers has revolutionalized the NLP community. Thus, because of this competition, we were able to achieve this goal by learning various innovative strategies to fine-tune transformer models that will eventually help us in the future.
Thank you for reading & Happy Kaggling!!