Firstly, we would like to thanks the competition organizers for hosting such a unique competition, we learned a lot from this competition. This was our first NLP Kaggle Competition and we enjoyed it very much.
\nCongratulations to all the winning teams!!
\nThank you to the Kaggle community for sharing such innovative ideas through notebooks and discussions, where we were able to learn a lot!!
\nSpecial thanks to the following users/community for providing amazing ideas and code that we used in our solution:
\nWe used Roberta-base, Roberta-large, Electra-base, and Electra-large with different heads [Mean-Pooing, CLS Token, and Attention], with/without pre-training, with/without layer-reinitialization, Gradient Accumulation, Group Differential LR, and a unique method of using 2D Attention Head that will be discussed by @muktan in this post. We used one submission based solely on our CV values and another was based on LB score.
\n1) CV Fit [Private LB: 0.458] - We used the following notebook to find the best ensembling score using our oof files and used 14 models which are as follows:
\n| Model-Description | \nCV | \nPublic LB | \n
|---|---|---|
| Roberta-base using Tensorflow @ragnar123 | \n0.509 | \n0.482 | \n
| Roberta-base @rhtsingh | \n0.481 | \n0.477 | \n
| Roberta-large-1 @rhtsingh | \n0.479 | \n0.474 | \n
| Roberta-large- [No-Pretraining+5 Layer Reinitialization] | \n0.481 | \n0.473 | \n
| Roberta-base @andretugan | \n0.477 | \n0.467 | \n
| Roberta-base- [variation of the above model] | \n0.475 | \n0.469 | \n
| Roberta-large- [Attention Head with Differential LR] | \n0.474 | \n0.460 | \n
| Roberta-large- [Above Model + 5 layer Reinitialization] | \n0.471 | \n0.463 | \n
| Roberta-large- [Mean Pooling Head] | \n0.479 | \n0.469 | \n
| Roberta-large- [2D Attention for 2 4 6 8 layers from the end- @muktan] | \n0.474 | \n0.461 | \n
| Roberta-large- [Variation of above technique] | \n0.475 | \n0.465 | \n
| Electra-large- [Variation of the best LB Roberta-large model] | \n0.479 | \n0.469 | \n
| Electra-large- [Variation of the above model] | \n0.487 | \n0.480 | \n
| Electra-large- [2D Attention- @muktan] | \n0.487 | \n0.474 | \n
| Overall | \n0.455 | \n0.457 | \n
2) LB Fit [Private LB: 0.458, Public LB: 0.455] - We used more-or-less the same models from the above list but we provided weights based on the LB scores and combined them in a different manner.
\nOur main intent was to learn about how transformers has revolutionalized the NLP community. Thus, because of this competition, we were able to achieve this goal by learning various innovative strategies to fine-tune transformer models that will eventually help us in the future.
\nThank you for reading & Happy Kaggling!!
", "comments": [], "comment_count": 0, "messages": [ { "id": 1428591, "postDate": "2021-08-03T09:37:21.943Z", "content": "Firstly, we would like to thanks the competition organizers for hosting such a unique competition, we learned a lot from this competition. This was our first NLP Kaggle Competition and we enjoyed it very much.
\nCongratulations to all the winning teams!!
\nThank you to the Kaggle community for sharing such innovative ideas through notebooks and discussions, where we were able to learn a lot!!
\nSpecial thanks to the following users/community for providing amazing ideas and code that we used in our solution:
\nWe used Roberta-base, Roberta-large, Electra-base, and Electra-large with different heads [Mean-Pooing, CLS Token, and Attention], with/without pre-training, with/without layer-reinitialization, Gradient Accumulation, Group Differential LR, and a unique method of using 2D Attention Head that will be discussed by @muktan in this post. We used one submission based solely on our CV values and another was based on LB score.
\n1) CV Fit [Private LB: 0.458] - We used the following notebook to find the best ensembling score using our oof files and used 14 models which are as follows:
\n| Model-Description | \nCV | \nPublic LB | \n
|---|---|---|
| Roberta-base using Tensorflow @ragnar123 | \n0.509 | \n0.482 | \n
| Roberta-base @rhtsingh | \n0.481 | \n0.477 | \n
| Roberta-large-1 @rhtsingh | \n0.479 | \n0.474 | \n
| Roberta-large- [No-Pretraining+5 Layer Reinitialization] | \n0.481 | \n0.473 | \n
| Roberta-base @andretugan | \n0.477 | \n0.467 | \n
| Roberta-base- [variation of the above model] | \n0.475 | \n0.469 | \n
| Roberta-large- [Attention Head with Differential LR] | \n0.474 | \n0.460 | \n
| Roberta-large- [Above Model + 5 layer Reinitialization] | \n0.471 | \n0.463 | \n
| Roberta-large- [Mean Pooling Head] | \n0.479 | \n0.469 | \n
| Roberta-large- [2D Attention for 2 4 6 8 layers from the end- @muktan] | \n0.474 | \n0.461 | \n
| Roberta-large- [Variation of above technique] | \n0.475 | \n0.465 | \n
| Electra-large- [Variation of the best LB Roberta-large model] | \n0.479 | \n0.469 | \n
| Electra-large- [Variation of the above model] | \n0.487 | \n0.480 | \n
| Electra-large- [2D Attention- @muktan] | \n0.487 | \n0.474 | \n
| Overall | \n0.455 | \n0.457 | \n
2) LB Fit [Private LB: 0.458, Public LB: 0.455] - We used more-or-less the same models from the above list but we provided weights based on the LB scores and combined them in a different manner.
\nOur main intent was to learn about how transformers has revolutionalized the NLP community. Thus, because of this competition, we were able to achieve this goal by learning various innovative strategies to fine-tune transformer models that will eventually help us in the future.
\nThank you for reading & Happy Kaggling!!
", "rawMarkdown": "Firstly, we would like to thanks the competition organizers for hosting such a unique competition, we learned a lot from this competition. This was our first NLP Kaggle Competition and we enjoyed it very much. \n\nCongratulations to all the winning teams!!\n\nThank you to the Kaggle community for sharing such innovative ideas through notebooks and discussions, where we were able to learn a lot!!\n\nSpecial thanks to the following users/community for providing amazing ideas and code that we used in our solution:\n1. Thank you to @rhtsingh for providing so many resources to learn about transformers: how to pre-train them, how to fine-tune them, how to improve fine-tuning [both speed and accuracy]; theory as well as code through your amazing notebooks.\n2. Thank you to @maunish [for providing insights on Attention head and providing a clean pipeline from Pre-training to fine-tuning to scoring], @andretugan [for providing a fine-tuning notebook that solved the problem of stability in CV while training], @cdeotte [for providing a unique method of ensembling using forward OOF technique], @ragnar123 [for providing fine-tuning notebook using TF] @abhishek [for amazing tutorials on training/finetuning transformers using pytorch], @rajat95gupta and @jcesquiveld [for providing insights on how to find the optimal head for a particular transformer], the [Huggingface community](https://discuss.huggingface.co/) [for solving all the queries albeit being small ones], and **finally my teammates @muktan and @deepangraval [being part of this 3 months journey with immense enthusiasm]**.\n\n# Brief Solution\n\nWe used Roberta-base, Roberta-large, Electra-base, and Electra-large with different heads [Mean-Pooing, CLS Token, and Attention], with/without pre-training, with/without layer-reinitialization, Gradient Accumulation, Group Differential LR, and a unique method of using 2D Attention Head that will be discussed by @muktan in **[this post](https://www.kaggle.com/c/commonlitreadabilityprize/discussion/261206)**. We used one submission based solely on our CV values and another was based on LB score.\n\n1) CV Fit [Private LB: 0.458] - We used the following [notebook](https://www.kaggle.com/cdeotte/forward-selection-oof-ensemble-0-942-private) to find the best ensembling score using our oof files and used 14 models which are as follows:\n\n| Model-Description | CV | Public LB |\n| --- | --- | --- |\n| Roberta-base using Tensorflow @ragnar123 | 0.509 | 0.482 |\n| Roberta-base @rhtsingh | 0.481 | 0.477 |\n| Roberta-large-1 @rhtsingh | 0.479 | 0.474 |\n| Roberta-large- [No-Pretraining+5 Layer Reinitialization] | 0.481 | 0.473 |\n| Roberta-base @andretugan | 0.477 | 0.467 |\n| Roberta-base- [variation of the above model] | 0.475 | 0.469 |\n| Roberta-large- [Attention Head with Differential LR] | 0.474 | 0.460 |\n| Roberta-large- [Above Model + 5 layer Reinitialization] | 0.471 | 0.463 |\n| Roberta-large- [Mean Pooling Head] | 0.479 | 0.469 |\n| Roberta-large- [2D Attention for 2 4 6 8 layers from the end- @muktan] | 0.474 | 0.461 |\n| Roberta-large- [Variation of above technique] | 0.475 | 0.465 |\n| Electra-large- [Variation of the best LB Roberta-large model] | 0.479 | 0.469 |\n| Electra-large- [Variation of the above model] | 0.487 | 0.480 |\n| Electra-large- [2D Attention- @muktan] | 0.487 | 0.474 |\n| Overall | 0.455 | 0.457 |\n\n2) LB Fit [Private LB: 0.458, Public LB: 0.455] - We used more-or-less the same models from the above list but we provided weights based on the LB scores and combined them in a different manner.\n\n# Training Config\n- Optimiser: AdamW\n - Weight decay: 0.5\n - LR: 2.5e-5 to 3.5e-5 (depending on batch size) [Group Differential LR]\n- Epochs: 3-5 [depending on the model]\n- Batch size: 16 (or 10 for the larger models)\n- Validation & Checkpointing every 5-10 steps [depending on the model]\n- Loss: MSE/RMSE\n# What worked for us\n- Pre-training Roberta-base [for large models, we were not able to pre-train ]\n- Differential learning rate [2.5e-5 to 3.5e-5 with a factor of 2.5 depending on the architecture]\n- Last 5 layer-reinitialization [we tried different techniques but this was the best amongst them]\n- Gradient Accumulation 2 [BS-8 for training on larger batch size]\n- Attention head, Mean Pooling head - [We were struggling to find custom heads for fine-tuning and these heads worked very well for the given tasks]\n# What didn't work for us\n- External-data pre-training- We tried to pre-train the model on MLM tasks using different external datasets, but we were not achieving good results w.r.t. CV or LB with that.\n- Stacking- We used the embeddings features from different models and added them to a Ridge/BayesianRidge Regression, but we were not getting good results with that, we concluded that as most of the features are extracted because of how trained our models using Grouped Differential LR, Stacking wasn't working for those models.\n- External-Features+Stacking- For some of the models, we combined the embeddings with some externals features and tried stacking using AutoML as well as Optuna but were unsuccessful as it worsened our score even more.\n# Tools Used\n- We are fresh undergraduate students with no external access to GPUs so we have been using Kaggle Kernels and Google Colab [we used 7 accounts 🤪to perform different experiments for finding hyper-parameters.]\n- We used Google-Docs from the beginning to first find all the techniques that are available for a readability task by reading different white papers [ML/Statistical/DL/RL] and then moved on to implement one-by-one as per the model-avail. and finally read about improving said models using different fine-tuning techniques.\n- We used Google-Sheets to track all the progress of the competition, and different models configurations as well.\n\n## P.S.\n\nOur main intent was to learn about how transformers has revolutionalized the NLP community. Thus, because of this competition, we were able to achieve this goal by learning various innovative strategies to fine-tune transformer models that will eventually help us in the future.\n\nThank you for reading & Happy Kaggling!!\n\n\n", "votes": 11 } ], "complete": true, "source": "live", "fetched_at": "2026-10-23T15:48:42Z", "competition_id": 25914, "competition_slug": "commonlitreadabilityprize", "team_id": 6785805, "solution_url": "/competitions/commonlitreadabilityprize/writeups/private-106th-public-79th-solution-top-3" }