Thanks to Kaggle and the competition hosts. The Feedback competitions were a lot of fun to work on, and it is great to hear that more competitions will be launched soon.
\nMy solution for this competition is an ensemble of 3 deberta models - one deberta-v3-base and two deberta-v3-large models. I relied on different training methods for diversity and ensembled them based on weighted average. For validation, I used MultilabelStratifiedKFold and my CV and LB were fairly correlated. Also, the essays in the training data are based on 42 topics and some topics had a lot of essays (100 - 300) while others had only a few essays (< 50). Models that performed well on the topics with fewer samples also performed well on the public LB, and on the private LB as well.
\nAWP: https://www.kaggle.com/code/wht1996/feedback-nn-train/notebook
\nThanks to everyone who shared training tips in the discussion forums, especially @wuwenmin. I joined a little late and reading all the discussions really helped.
", "comments": [ { "id": 2090945, "author_name": "Marília Prata", "author_url": "", "post_date": "2023-01-07T20:44:33.767000", "content": "Congratulations and thanks for sharing what worked and Not worked on this competition.
", "votes": 1, "replies": [] }, { "id": 2057335, "author_name": "Apa", "author_url": "", "post_date": "2022-12-07T01:37:52.347000", "content": "When I use AWP or turn on half-precision training, it is very often that the gradient is nan. I wonder if you have encountered it?
", "votes": 0, "replies": [] } ], "comment_count": 2, "messages": [ { "id": 2050428, "postDate": "2022-11-30T18:15:36.030Z", "content": "Thanks to Kaggle and the competition hosts. The Feedback competitions were a lot of fun to work on, and it is great to hear that more competitions will be launched soon.
\nMy solution for this competition is an ensemble of 3 deberta models - one deberta-v3-base and two deberta-v3-large models. I relied on different training methods for diversity and ensembled them based on weighted average. For validation, I used MultilabelStratifiedKFold and my CV and LB were fairly correlated. Also, the essays in the training data are based on 42 topics and some topics had a lot of essays (100 - 300) while others had only a few essays (< 50). Models that performed well on the topics with fewer samples also performed well on the public LB, and on the private LB as well.
\nAWP: https://www.kaggle.com/code/wht1996/feedback-nn-train/notebook
\nThanks to everyone who shared training tips in the discussion forums, especially @wuwenmin. I joined a little late and reading all the discussions really helped.
", "rawMarkdown": "\nThanks to Kaggle and the competition hosts. The Feedback competitions were a lot of fun to work on, and it is great to hear that more competitions will be launched soon. \n\n# Overview\nMy solution for this competition is an ensemble of 3 deberta models - one deberta-v3-base and two deberta-v3-large models. I relied on different training methods for diversity and ensembled them based on weighted average. For validation, I used MultilabelStratifiedKFold and my CV and LB were fairly correlated. Also, the essays in the training data are based on 42 topics and some topics had a lot of essays (100 - 300) while others had only a few essays (< 50). Models that performed well on the topics with fewer samples also performed well on the public LB, and on the private LB as well. \n\n\n# What Worked\n- Training with AWP\n - Result: +.001\n - A lot of top solutions in recent NLP competitions used AWP and it worked well for me in this competition. My strongest model was deberta-v3-large with AWP starting from the 2nd epoch. \n\n\n- Using differential learning rates\n - Result: .0008\n - Based on the discussion forums, this was one of the first things I tried and I ended up using it in all my models.\n\n- Reinitialising the last layer\n - Result: Improved the ensemble CV by .001\n - I trained this model based on the suggestions in [this post](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/360819) and it worked well in the ensemble. \n\n\n# What Didn’t Work\n- Pseudo labels \n - I tried pretraining on pseudo labels, and using them while training as well but couldn’t get it to work. Probably needed to tune it some more as it seemed to have worked for other teams. \n- MLM\n- Adding essay prompts while training\n\n# Important citations:\nAWP: https://www.kaggle.com/code/wht1996/feedback-nn-train/notebook\n\n\n# Thanks and Acknowledgements\nThanks to everyone who shared training tips in the discussion forums, especially @wuwenmin. I joined a little late and reading all the discussions really helped. \n\n ", "votes": 16 }, { "id": 2090945, "postDate": "2023-01-07T20:44:33.767Z", "content": "Congratulations and thanks for sharing what worked and Not worked on this competition.
", "rawMarkdown": "Congratulations and thanks for sharing what worked and Not worked on this competition.", "votes": 1 }, { "id": 2057335, "postDate": "2022-12-07T01:37:52.347Z", "content": "When I use AWP or turn on half-precision training, it is very often that the gradient is nan. I wonder if you have encountered it?
", "rawMarkdown": "When I use AWP or turn on half-precision training, it is very often that the gradient is nan. I wonder if you have encountered it?" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T20:02:58Z", "competition_id": 38321, "competition_slug": "feedback-prize-english-language-learning", "team_id": 9109817, "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/rai-17th-place-solution" }