Thanks Kaggle and CommonLit for hosting this competition I learnt a lot in my first NLP competition.
\nAnd also great thanks to @rhtsingh (I learnt a lot about NLP from your notebooks.), @leighplt (I use your notebook as initial baseline.) and @konradb (I use features from your notebook).
CV
\nDropout
\nOther hyper-parameters that worked for me
\nI included one model with ITPT in final solution but just for diversity it didn't work for me in terms of both CV and LB.
\nSingle models for final inference
\n(score without dropout average)
Models with same name have different parameters, you can check respective notebook for details.
\nI use one one drop based on CV to select models, although my best single model deberta large with CV 0.4757 public LB 0.462 private LB 0.462, it is not included in my final inference notebook.
\nEnsemble
\nI use same CV split across all models and ensemble performance is evaluated on oof predictions.
Code
\nYou need to remove some private models that loaded (but not used when inference) to re-run some of the noetbooks.
", "comments": [ { "id": 1436090, "author_name": "Allohvk", "author_url": "", "post_date": "2021-08-03T14:57:39.947000", "content": "Thank you for clean documentation and kernels shared. The paragraph on dropout was particularly interesting to read
", "votes": 3, "replies": [] }, { "id": 1450648, "author_name": "Frank", "author_url": "", "post_date": "2021-08-05T06:39:05.590000", "content": "Congratulations!
\nThe code urls you shared are not working. @w5833946
", "votes": 1, "replies": [ { "id": 1450733, "author_name": "Ruby", "author_url": "", "post_date": "2021-08-05T06:58:35.327000", "content": "I didn't aware that. I will fix it later, you can check my profile and find the code that I posted.
", "votes": 0, "replies": [] } ] }, { "id": 1447836, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2021-08-04T16:01:59.870000", "content": "I didn’t have time to keep working on this one (haven’t submitted in >1 month :-( so I am glad at least the features were useful for somebody else. Congratulations!
", "votes": 1, "replies": [] }, { "id": 1442409, "author_name": "Salim Khazem", "author_url": "", "post_date": "2021-08-03T22:40:33.877000", "content": "Congratulations, thanks a lot for sharing details of your approach
", "votes": 1, "replies": [] }, { "id": 1440433, "author_name": "Chris Deotte", "author_url": "", "post_date": "2021-08-03T18:18:43.053000", "content": "Congratulations and great observation about dropout. We noticed that dropout affects models too. I wonder why that is. I posted a plot here of the difference training with and without drop.
", "votes": 1, "replies": [] } ], "comment_count": 6, "messages": [ { "id": 1435492, "postDate": "2021-08-03T14:44:21.470Z", "content": "Thanks Kaggle and CommonLit for hosting this competition I learnt a lot in my first NLP competition.
\nAnd also great thanks to @rhtsingh (I learnt a lot about NLP from your notebooks.), @leighplt (I use your notebook as initial baseline.) and @konradb (I use features from your notebook).
CV
\nDropout
\nOther hyper-parameters that worked for me
\nI included one model with ITPT in final solution but just for diversity it didn't work for me in terms of both CV and LB.
\nSingle models for final inference
\n(score without dropout average)
Models with same name have different parameters, you can check respective notebook for details.
\nI use one one drop based on CV to select models, although my best single model deberta large with CV 0.4757 public LB 0.462 private LB 0.462, it is not included in my final inference notebook.
\nEnsemble
\nI use same CV split across all models and ensemble performance is evaluated on oof predictions.
Code
\nYou need to remove some private models that loaded (but not used when inference) to re-run some of the noetbooks.
", "rawMarkdown": "Thanks Kaggle and CommonLit for hosting this competition I learnt a lot in my first NLP competition.\nAnd also great thanks to @rhtsingh (I learnt a lot about NLP from your notebooks.), @leighplt (I use your notebook as initial baseline.) and @konradb (I use features from your notebook).\n\n**CV**\n- I use k fold without shuffle since I note that the training set is not shuffled and the excerpt seems comes in batchs of different sources. I think the test set might also composed of excerpts from other sources and it is well known that test set has more modern texts. So I think use k fold without shuffle might better measure the 'out of distribution' generaliztion ability.\n- According to my early stage experiments CV without shuffle usally get cv score around 0.01 higher than shuffled ones. As LB scores are average of 5 models (I use 5 fold) and CV is given by oof prediction of 5 models but one model each time so I expect CV score to be higher than LB scores.\n- Although I think k fold without shuffle seems to be better but I didn't systematically check whether it is superior than shuffled ones or even further group by bins.\n\n**Dropout**\n- Model performance seems to have a gap between .train() and .eval() mode when apply dropout in this task.I think this might comes from potential bugs that I can't identify or the variance difference between .train() and .eval() of some layers. The scaling factor in dropout in general don't guranteen the layer to have same variance and 'Readbility' sounds like something that will related to 'variance'/'perplexity'/... Of course this might not be the real reason.\n- I was not able to solve this problem thoroughly. I use dropout in my models but use .train() to inference, this give me reasonable performace and superior to the zero dropout setting in my case. Due to randomness from dropout I average the predictions from same model but with different seed while inference, this gives me further boost.\n\n**Other hyper-parameters that worked for me**\n- Adamw 5 epochs\n- cosine learning rate scheduler\n- change learning rate exponentially across layers\n- re-initialize some layers\n- gradient accumulation for larger batch size (I only tried batch size 32 with base models and 16 with large models due to notebook time limit)\n- linear head followed by mean pooling on last hidden state (some times I use two linear layers for output but mainly due to I want to keep in track with one of my ealier experiemnt, perfromance of two or one linear layer is not much different.)\n\nI included one model with ITPT in final solution but just for diversity it didn't work for me in terms of both CV and LB.\n\n**Single models for final inference**\n(score without dropout average)\n- F2 roberta large CV 0.4839 public LB 0.470 \n- F17 roberta base CV 0.4758 public LB 0.468 \n- F23 roberta large CV 0.4792 public LB 0.465 \n- F32 deberta large CV 0.4749 public LB 0.469 \n- F41 roberta large mnli CV 0.4796 public LB 0.475 \n- F50 funnel large CV 0.480 public LB 0.465 \n- F51 electra large CV 0.479 public LB 0.469 \n\nModels with same name have different parameters, you can check respective notebook for details.\n\nI use one one drop based on CV to select models, although my best single model deberta large with CV 0.4757 public LB 0.462 private LB 0.462, it is not included in my final inference notebook.\n\n**Ensemble**\nI use same CV split across all models and ensemble performance is evaluated on oof predictions.\n- Ridge regression CV 0.4553 public LB 0.451 private LB 0.449\n- Although manual features didn't help when I integrate them with predictions of transformers by a linear model, it helps when I use lightgbm. And average predictions of lightgbm and ridge gives my final score： CV 0.4532 public LB 0.454 private LB 0.448. As public score is worse so this might not a real improvement.\n\n**Code**\n- ITPT\n[https://www.kaggle.com/w5833946/f1-itpt](url)\n- Train\n[https://www.kaggle.com/w5833946/f2-train](url)\n[https://www.kaggle.com/w5833946/f17-train/data](url)\n[https://www.kaggle.com/w5833946/f23-train](url)\n[https://www.kaggle.com/w5833946/f32-train](url)\n[https://www.kaggle.com/w5833946/f41-train](url)\n[https://www.kaggle.com/w5833946/f50-train](url)\n[https://www.kaggle.com/w5833946/f51-train](url)\n- Inference trainning set\n[https://www.kaggle.com/w5833946/f-inference-1](url)\n[https://www.kaggle.com/w5833946/f-inference-1-1/data](url)\n- Inference\n[https://www.kaggle.com/w5833946/f-inference-3-1-v4](url)\n\nYou need to remove some private models that loaded (but not used when inference) to re-run some of the noetbooks.", "votes": 36 }, { "id": 1436090, "postDate": "2021-08-03T14:57:39.947Z", "content": "Thank you for clean documentation and kernels shared. The paragraph on dropout was particularly interesting to read
", "rawMarkdown": "Thank you for clean documentation and kernels shared. The paragraph on dropout was particularly interesting to read", "votes": 3 }, { "id": 1450648, "postDate": "2021-08-05T06:39:05.590Z", "content": "Congratulations!
\nThe code urls you shared are not working. @w5833946
", "rawMarkdown": "Congratulations! \n\nThe code urls you shared are not working. @w5833946 ", "votes": 1, "replies": [ { "id": 1450733, "postDate": "2021-08-05T06:58:35.327Z", "content": "I didn't aware that. I will fix it later, you can check my profile and find the code that I posted.
", "rawMarkdown": "I didn't aware that. I will fix it later, you can check my profile and find the code that I posted." } ] }, { "id": 1447836, "postDate": "2021-08-04T16:01:59.870Z", "content": "I didn’t have time to keep working on this one (haven’t submitted in >1 month :-( so I am glad at least the features were useful for somebody else. Congratulations!
", "rawMarkdown": "I didn’t have time to keep working on this one (haven’t submitted in >1 month :-( so I am glad at least the features were useful for somebody else. Congratulations!", "votes": 1 }, { "id": 1442409, "postDate": "2021-08-03T22:40:33.877Z", "content": "Congratulations, thanks a lot for sharing details of your approach
", "rawMarkdown": "Congratulations, thanks a lot for sharing details of your approach ", "votes": 1 }, { "id": 1440433, "postDate": "2021-08-03T18:18:43.053Z", "content": "Congratulations and great observation about dropout. We noticed that dropout affects models too. I wonder why that is. I posted a plot here of the difference training with and without drop.
", "rawMarkdown": "Congratulations and great observation about dropout. We noticed that dropout affects models too. I wonder why that is. I posted a plot [here][1] of the difference training with and without drop.\n\n[1]: https://www.kaggle.com/c/commonlitreadabilityprize/discussion/260729", "votes": 1 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T23:48:32Z", "competition_id": 25914, "competition_slug": "commonlitreadabilityprize", "team_id": 6739426, "solution_url": "/competitions/commonlitreadabilityprize/writeups/ruby-5th-place-solution" }