![\"\"](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Aug-2021/commonlit_model.png\")
Thank you Kaggle and CommonLit for a fun and exciting competition. Thank you team for a great collaboration! @mobassir @ragnar123 @datafan07 @ivanaerlic @cdeotte Our team strategy was diversity of NLP transformer models because the train dataset was small and risk of overfitting was large. Besides diversity of backbones, our secret sauce was lots of creative model heads and creative training schedules including RAPIDS cuml SVR head which boosted CV LB +0.002!
\n
@mobassir and @ragnar123 have been sharing high quality notebooks and discussions with the Kaggle community for years. Their code and discussion sharing has personally helped me achieve gold in previous competitions! I’m excited to share in their current achievement of gold medal and becoming Kaggle Competition Masters. Please help me congratulate them. Great job guys!
\nOur team began with @mobassir and @ragnar123 who had achieved top 50 among 3000 teams. I was happy when they invited me to their team because I was struggling to get in the top 200 teams! As a team of three we decided diversity was the best approach in this competition. We invited @datafan07 and @ivanaerlic to join our team. These two Kagglers have demonstrated they can build creative diverse strong single models in their previous competitions. They succeeded once again! Thank you @datafan07 and @ivanaerlic , your models are fantastic!
\nEvery model we computed CV and LB. If both were good, we added the model to our ensemble. Our final ensemble had CV 0.452, Public LB 0.451, and Private LB 0.450. Notice that there was a strong connection between CV, Public LB, and Private LB.
\nConfiguration (CV 0.472 LB 0.463) :
\nThe main idea here was to fine-tune as many as strong transformer models using Huggingface Transformers then get their embeddings using the Sentence Transformers package. First we inspected many transformer models doing 5 fold cross validation. So we can see how they learn and generalize, then take these fine tuned models into sentence transformers to create sentence embeddings. Here we got last layer embeddings plus cls tokens, then concatenate them into one 2D array with 2048 (These can change but are similar most of the time for big models) features for each instance. Then built and trained Bayesian Ridge head where takes these embeddings as input and gives target scores as float. Actually it’s not that different from the model shared here by @datafan07. We believe these give decently generalized results but to get more accurate results there’s the second part.
\nIn the second part the main idea was combining embedding based predictions with neural network based predictions. For testing this idea first we combined bayesian predictions with a public model. This gave us a great increase in lb score (~0.01) from 0.466 to 0.454 So we planned to combine these with the other team member models. It worked pretty well with our own models too, so at late stages we concentrated on finding optimal ratios of the combination using OOF scores.
\nThank you @andretugan We used your RoBERTa-Base public notebook in our final ensemble
\nIn the last few days we began exploring additional ideas. With more time we believe some could have boosted our CV LB scores. One example is an excerpt compare model described here. Another is pseudo labeling the test data during submission and then training another RoBERTa-Base during submission. Another is NLP augmentations. Another is using external data.
\nCongratulations to all top teams. Thank you for all your sharing during and after the competition. Thanks everyone for reading our solution writeup!
", "comments": [ { "id": 1445000, "author_name": "Mobassir", "author_url": "", "post_date": "2021-08-04T07:34:30.373000", "content": "Praise be to God.
\nThank you a lot team mates and congratulations once again 🎉🔥,
\nWell done! It was great, nice team, we all contributed a lot, it was a team effort 💥
\nI am truly humbled and grateful for the opportunity to work with you all and get my first competition gold medal (a competition to remember for long long time) .Thanks everyone for all your congratulations. i learned a lot of nlp tricks here,hopefully will be applying them again in another nlp competition.Great thing about kaggle is you can start from having zero knowledge and still end up being a pro.thanks kaggle 🙌
\nall those sleepless nights that i dedicated for our invaluable slack & discord discussions just to learn from experienced data scientist like you are worth it. i will miss our awesome slack discussions a lot.Everyone did awesome. I have great fun working with you all.
\nthis motivates us(excluding @cdeotte 😄) to aim for competition GrandMaster tier.learned a lot from you all,Thank you for all your guidance and wisdom and thanks to all the kagglers that shared high quality contents,,specially maunish,torch and andretugan 👌👌🙏
Congratz to you and your team @mobassir, very well deserved :)
", "votes": 2, "replies": [] }, { "id": 1447853, "author_name": "Mobassir", "author_url": "", "post_date": "2021-08-04T16:04:25.487000", "content": "thank you @theoviel
\ni learned a lot from you in past jigsaw competition,i always try to use some of those learnings in kaggle,thank you :) 😊
Well done @mobassir!
", "votes": 2, "replies": [] }, { "id": 1455252, "author_name": "Muhammad Ahmed", "author_url": "", "post_date": "2021-08-06T13:52:12.483000", "content": "Mashallah Congrats! hope you earn more Ameen. @mobassir
", "votes": 2, "replies": [] } ] }, { "id": 1445420, "author_name": "Maksim Markeev", "author_url": "", "post_date": "2021-08-04T08:49:56.547000", "content": "Great work! As always!
\nHow you use several heads in one model? You concatenate them ?
Each model has multiple heads. Each head makes test predictions. Then we ensemble all the test predictions.
", "votes": 3, "replies": [] } ] }, { "id": 1443447, "author_name": "Shai", "author_url": "", "post_date": "2021-08-04T03:02:02.470000", "content": "Congrats @mobassir and @ragnar123. You guys deserve it.
\n@cdeotte once again you finished the job with elegance! Well done!
Great! Ty 4sharing. :D
", "votes": 1, "replies": [] }, { "id": 1445309, "author_name": "KhanhVD", "author_url": "", "post_date": "2021-08-04T08:26:41.753000", "content": "Congrats @cdeotte and team on results and thanks for sharing the writeup.
", "votes": 1, "replies": [] }, { "id": 1444290, "author_name": "yukiya", "author_url": "", "post_date": "2021-08-04T05:34:05.317000", "content": "Thank you for sharing the details.
\nI have very similar model ensemble (Roberta, Electra and Deberta) but I couldn't figure out Funnel since it looks very different from Bert.
\nCongratulations !
Congrats to team, always enjoy yr notebook sharing !
", "votes": 1, "replies": [] }, { "id": 1442388, "author_name": "Salim Khazem", "author_url": "", "post_date": "2021-08-03T22:37:39.683000", "content": "Well done, Congratulations,
\nThanks a lot for sharing your knowledge, always a pleasure to read your posts , strategies, solutions, I learned a lot of things , Thanks a lot
Thank you @cdeotte for sharing this. Also thanks to the team it was really enjoyable to work with you guys.
", "votes": 2, "replies": [] }, { "id": 1451316, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2021-08-05T10:11:18.430000", "content": "Thanks for sharing the team's solution and congratulations to @mobassir and @ragnar123 for their new titles!
", "votes": 2, "replies": [] }, { "id": 1442385, "author_name": "Sinan Calisir", "author_url": "", "post_date": "2021-08-03T22:37:13.987000", "content": "Thank you for sharing the details of your team's solution, Chris! Huge thanks to all team members. The title itself is more like a TLDR :) @datafan07 @ivanaerlic
\nAlso, huge congratulations to new Competitions Masters @mobassir @ragnar123 🔥🔥 Very well deserved!!
", "votes": 2, "replies": [] }, { "id": 1443877, "author_name": "typ0", "author_url": "", "post_date": "2021-08-04T04:25:50.797000", "content": "Congratulations, and thank you for sharing this!
", "votes": 0, "replies": [] }, { "id": 1451723, "author_name": "Devashree Madhugiri", "author_url": "", "post_date": "2021-08-05T12:17:05.610000", "content": "Great work! Thanks for sharing!
", "votes": 1, "replies": [] } ], "comment_count": 18, "messages": [ { "id": 1441187, "postDate": "2021-08-03T19:49:13.330Z", "content": "Thank you Kaggle and CommonLit for a fun and exciting competition. Thank you team for a great collaboration! @mobassir @ragnar123 @datafan07 @ivanaerlic @cdeotte Our team strategy was diversity of NLP transformer models because the train dataset was small and risk of overfitting was large. Besides diversity of backbones, our secret sauce was lots of creative model heads and creative training schedules including RAPIDS cuml SVR head which boosted CV LB +0.002!
\n
@mobassir and @ragnar123 have been sharing high quality notebooks and discussions with the Kaggle community for years. Their code and discussion sharing has personally helped me achieve gold in previous competitions! I’m excited to share in their current achievement of gold medal and becoming Kaggle Competition Masters. Please help me congratulate them. Great job guys!
\nOur team began with @mobassir and @ragnar123 who had achieved top 50 among 3000 teams. I was happy when they invited me to their team because I was struggling to get in the top 200 teams! As a team of three we decided diversity was the best approach in this competition. We invited @datafan07 and @ivanaerlic to join our team. These two Kagglers have demonstrated they can build creative diverse strong single models in their previous competitions. They succeeded once again! Thank you @datafan07 and @ivanaerlic , your models are fantastic!
\nEvery model we computed CV and LB. If both were good, we added the model to our ensemble. Our final ensemble had CV 0.452, Public LB 0.451, and Private LB 0.450. Notice that there was a strong connection between CV, Public LB, and Private LB.
\nConfiguration (CV 0.472 LB 0.463) :
\nThe main idea here was to fine-tune as many as strong transformer models using Huggingface Transformers then get their embeddings using the Sentence Transformers package. First we inspected many transformer models doing 5 fold cross validation. So we can see how they learn and generalize, then take these fine tuned models into sentence transformers to create sentence embeddings. Here we got last layer embeddings plus cls tokens, then concatenate them into one 2D array with 2048 (These can change but are similar most of the time for big models) features for each instance. Then built and trained Bayesian Ridge head where takes these embeddings as input and gives target scores as float. Actually it’s not that different from the model shared here by @datafan07. We believe these give decently generalized results but to get more accurate results there’s the second part.
\nIn the second part the main idea was combining embedding based predictions with neural network based predictions. For testing this idea first we combined bayesian predictions with a public model. This gave us a great increase in lb score (~0.01) from 0.466 to 0.454 So we planned to combine these with the other team member models. It worked pretty well with our own models too, so at late stages we concentrated on finding optimal ratios of the combination using OOF scores.
\nThank you @andretugan We used your RoBERTa-Base public notebook in our final ensemble
\nIn the last few days we began exploring additional ideas. With more time we believe some could have boosted our CV LB scores. One example is an excerpt compare model described here. Another is pseudo labeling the test data during submission and then training another RoBERTa-Base during submission. Another is NLP augmentations. Another is using external data.
\nCongratulations to all top teams. Thank you for all your sharing during and after the competition. Thanks everyone for reading our solution writeup!
", "rawMarkdown": "## 15th Place Team “Trust Your CV” and Trust Diversity!\n\nThank you Kaggle and CommonLit for a fun and exciting competition. Thank you team for a great collaboration! @mobassir @ragnar123 @datafan07 @ivanaerlic @cdeotte Our team strategy was diversity of NLP transformer models because the train dataset was small and risk of overfitting was large. Besides diversity of backbones, our secret sauce was lots of creative model heads and creative training schedules including [RAPIDS cuml SVR][3] head which boosted CV LB +0.002!\n\n\n\n## Congratulate Mobassir and Ragnar Becoming Competition Masters!\n\n@mobassir and @ragnar123 have been sharing high quality notebooks and discussions with the Kaggle community for years. Their code and discussion sharing has personally helped me achieve gold in previous competitions! I’m excited to share in their current achievement of gold medal and becoming Kaggle Competition Masters. Please help me congratulate them. Great job guys!\n\n\n## Introduction\n\nOur team began with @mobassir and @ragnar123 who had achieved top 50 among 3000 teams. I was happy when they invited me to their team because I was struggling to get in the top 200 teams! As a team of three we decided diversity was the best approach in this competition. We invited @datafan07 and @ivanaerlic to join our team. These two Kagglers have demonstrated they can build creative diverse strong single models in their previous competitions. They succeeded once again! Thank you @datafan07 and @ivanaerlic , your models are fantastic!\n\n## Trust Your CV\n\nEvery model we computed CV and LB. If both were good, we added the model to our ensemble. Our final ensemble had CV 0.452, Public LB 0.451, and Private LB 0.450. Notice that there was a strong connection between CV, Public LB, and Private LB.\n\n## Mobassir - PyTorch - DeBERTa-Large\nConfiguration (CV 0.472 LB 0.463) : \n\n* Used single seed (63)\n* Used self-attention head\n* Used layerwise learning weight decay\n* Trained for 7 epochs \n* Used learning rate 0.00003 for AdamW\n* Used batch size and validation frequency = 8\n* Used max_length = 300\n* Used abhishek thakur’s bin based stratified5fold\n* Tried SVR head but couldn’t improve cv furthermore because we think the self attention head was already trained very well\n\n## Ragnar - TensorFlow - RoBERTa-Large\n\n* CV 0.462 LB 0.459\n* first two heads - cls token head, last hidden layer head \n* third head - SVR head\n* two stage training with and without frozen backbone\n\n## Ertugrul - PyTorch - Electra-Large, Funnel-Large, RoBERTa-Large\n\n* CV 0.458 LB 0.454\n* Huggingface \n* Sentence Transformers\n* Bayesian Ridge head\n* dropout = 0 (explained [here][2])\n\nThe main idea here was to fine-tune as many as strong transformer models using Huggingface Transformers then get their embeddings using the Sentence Transformers package. First we inspected many transformer models doing 5 fold cross validation. So we can see how they learn and generalize, then take these fine tuned models into sentence transformers to create sentence embeddings. Here we got last layer embeddings plus cls tokens, then concatenate them into one 2D array with 2048 (These can change but are similar most of the time for big models) features for each instance. Then built and trained Bayesian Ridge head where takes these embeddings as input and gives target scores as float. Actually it’s not that different from the model shared here by @datafan07. We believe these give decently generalized results but to get more accurate results there’s the second part.\n\nIn the second part the main idea was combining embedding based predictions with neural network based predictions. For testing this idea first we combined bayesian predictions with a public model. This gave us a great increase in lb score (~0.01) from 0.466 to 0.454 So we planned to combine these with the other team member models. It worked pretty well with our own models too, so at late stages we concentrated on finding optimal ratios of the combination using OOF scores.\n\n## Ivan - PyTorch - Microsoft DeBERTa-Large, DeBERTa-Large-MNLI\n\n* CV 0.464 LB 0.458\n* first head - self attention head\n* second head - Bayesian Ridge head\n* dropout = 0 (explained [here][2])\n\n## Public Notebook - PyTorch - RoBERTa-Base\n\nThank you @andretugan We used your RoBERTa-Base public notebook in our final ensemble\n* CV 0.477 LB 0.467\n\n## Chris - RAPIDS cuml SVR and Post Process\n\n* Boost CV +0.002 LB +0.002\n* We learned that during inference, if we extracted the final hidden layer activations (sentence embeddings) from our transformer models and trained a RAPIDS support vector regression model and then ensemble this model with the original model head we could boost all our models’ CV and LB. The TensorFlow models boosted LB +0.002 and most PyTorch models boosted LB +0.001.\n* We also learned that the test data has a mean target that is 0.02 lower than the train target. Therefore if we subtract 0.02 from our final predictions this boosted LB +0.001\n\n## Food For Thought\nIn the last few days we began exploring additional ideas. With more time we believe some could have boosted our CV LB scores. One example is an excerpt compare model described [here][1]. Another is pseudo labeling the test data during submission and then training another RoBERTa-Base during submission. Another is NLP augmentations. Another is using external data.\n\n## Congratulations Top Teams - Thank you!\nCongratulations to all top teams. Thank you for all your sharing during and after the competition. Thanks everyone for reading our solution writeup!\n\n[1]: https://www.kaggle.com/c/commonlitreadabilityprize/discussion/257446\n[2]: https://www.kaggle.com/c/commonlitreadabilityprize/discussion/260729\n[3]: https://docs.rapids.ai/api/cuml/stable/api.html#cuml.svm.SVR\n", "votes": 92 }, { "id": 1445000, "postDate": "2021-08-04T07:34:30.373Z", "content": "Praise be to God.
\nThank you a lot team mates and congratulations once again 🎉🔥,
\nWell done! It was great, nice team, we all contributed a lot, it was a team effort 💥
\nI am truly humbled and grateful for the opportunity to work with you all and get my first competition gold medal (a competition to remember for long long time) .Thanks everyone for all your congratulations. i learned a lot of nlp tricks here,hopefully will be applying them again in another nlp competition.Great thing about kaggle is you can start from having zero knowledge and still end up being a pro.thanks kaggle 🙌
\nall those sleepless nights that i dedicated for our invaluable slack & discord discussions just to learn from experienced data scientist like you are worth it. i will miss our awesome slack discussions a lot.Everyone did awesome. I have great fun working with you all.
\nthis motivates us(excluding @cdeotte 😄) to aim for competition GrandMaster tier.learned a lot from you all,Thank you for all your guidance and wisdom and thanks to all the kagglers that shared high quality contents,,specially maunish,torch and andretugan 👌👌🙏
Congratz to you and your team @mobassir, very well deserved :)
", "rawMarkdown": "Congratz to you and your team @mobassir, very well deserved :)", "votes": 2 }, { "id": 1447853, "postDate": "2021-08-04T16:04:25.487Z", "content": "thank you @theoviel
\ni learned a lot from you in past jigsaw competition,i always try to use some of those learnings in kaggle,thank you :) 😊
Well done @mobassir!
", "rawMarkdown": "Well done @mobassir! ", "votes": 2 }, { "id": 1455252, "postDate": "2021-08-06T13:52:12.483Z", "content": "Mashallah Congrats! hope you earn more Ameen. @mobassir
", "rawMarkdown": "Mashallah Congrats! hope you earn more Ameen. @mobassir ", "votes": 2 } ] }, { "id": 1445420, "postDate": "2021-08-04T08:49:56.547Z", "content": "Great work! As always!
\nHow you use several heads in one model? You concatenate them ?
Each model has multiple heads. Each head makes test predictions. Then we ensemble all the test predictions.
", "rawMarkdown": "Each model has multiple heads. Each head makes test predictions. Then we ensemble all the test predictions.", "votes": 3 } ] }, { "id": 1443447, "postDate": "2021-08-04T03:02:02.470Z", "content": "Congrats @mobassir and @ragnar123. You guys deserve it.
\n@cdeotte once again you finished the job with elegance! Well done!
Great! Ty 4sharing. :D
", "rawMarkdown": "Great! Ty 4sharing. :D", "votes": 1 }, { "id": 1445309, "postDate": "2021-08-04T08:26:41.753Z", "content": "Congrats @cdeotte and team on results and thanks for sharing the writeup.
", "rawMarkdown": "Congrats @cdeotte and team on results and thanks for sharing the writeup.", "votes": 1 }, { "id": 1444290, "postDate": "2021-08-04T05:34:05.317Z", "content": "Thank you for sharing the details.
\nI have very similar model ensemble (Roberta, Electra and Deberta) but I couldn't figure out Funnel since it looks very different from Bert.
\nCongratulations !
Congrats to team, always enjoy yr notebook sharing !
", "rawMarkdown": "Congrats to team, always enjoy yr notebook sharing !", "votes": 1 }, { "id": 1442388, "postDate": "2021-08-03T22:37:39.683Z", "content": "Well done, Congratulations,
\nThanks a lot for sharing your knowledge, always a pleasure to read your posts , strategies, solutions, I learned a lot of things , Thanks a lot
Thank you @cdeotte for sharing this. Also thanks to the team it was really enjoyable to work with you guys.
", "rawMarkdown": "Thank you @cdeotte for sharing this. Also thanks to the team it was really enjoyable to work with you guys.", "votes": 2 }, { "id": 1451316, "postDate": "2021-08-05T10:11:18.430Z", "content": "Thanks for sharing the team's solution and congratulations to @mobassir and @ragnar123 for their new titles!
", "rawMarkdown": "Thanks for sharing the team's solution and congratulations to @mobassir and @ragnar123 for their new titles!", "votes": 2 }, { "id": 1442385, "postDate": "2021-08-03T22:37:13.987Z", "content": "Thank you for sharing the details of your team's solution, Chris! Huge thanks to all team members. The title itself is more like a TLDR :) @datafan07 @ivanaerlic
\nAlso, huge congratulations to new Competitions Masters @mobassir @ragnar123 🔥🔥 Very well deserved!!
", "rawMarkdown": "Thank you for sharing the details of your team's solution, Chris! Huge thanks to all team members. The title itself is more like a TLDR :) @datafan07 @ivanaerlic \n\nAlso, huge congratulations to new Competitions Masters @mobassir @ragnar123 🔥🔥 Very well deserved!!", "votes": 2 }, { "id": 1443877, "postDate": "2021-08-04T04:25:50.797Z", "content": "Congratulations, and thank you for sharing this!
", "rawMarkdown": "Congratulations, and thank you for sharing this!" }, { "id": 1451723, "postDate": "2021-08-05T12:17:05.610Z", "content": "Great work! Thanks for sharing!
", "rawMarkdown": "Great work! Thanks for sharing!", "votes": 1 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T23:49:09Z", "competition_id": 25914, "competition_slug": "commonlitreadabilityprize", "team_id": 6734763, "solution_url": "/competitions/commonlitreadabilityprize/writeups/trust-your-cv-15th-place-trust-your-cv-and-model-d" }