Thanks to Kaggle and CommonLit for such an amazing competition! This was one of the first non-starter/playground competitions that our team has competed in. Despite our close call I’m still really happy with the results, and this has definitely equipped us with the skills needed to conquer future Kaggle competitions.
\nOur best submission had a public LB score of 0.438, which was in the bronze threshold, but ended up getting a private LB score of 0.485, which was 0.001 away from the bronze threshold :(( That being said, I still wanted to make a post detailing our approach, since our team invested a lot of time into this competition. We found that our CV was around 0.5-0.6 points higher than the public LB, and was 0.1-0.2 points higher than the private LB.
\nThank you to my teammates @yeoyunsianggeremie and @aayushpatni for helping me with this competition! You guys were amazing :))
\nCongratulations to all of the winners! And what a shakeup!
\nWe used a modification of the DebertaV3 + LGBM approach shared by @tsunotsuno in his public notebook. Big kudos to them!
\nWe utilized more features in our submission than were present in the original notebook. In particular,
\nWe ensembled 4 LLM models: 2 DebertaV3 large models trained on individual metrics, 1 deberta v3 large model trained on both metrics, and 1 deberta v3 base model trained on both metrics. The predictions from these models were then fed into individual LGBM models, that had been tuned to each LLM model using Optuna. Finally, the resultant predictions were ensembled using hill climbing.
\nWe found a variety of techniques when pretraining helped with the LLM models’ performance, including
\nautocorrect-1.1.0 because of the whole licensing issue, but we should have looked into alternatives (since this was known to be buggy)In general, we were really cautious about over-engineering our solution as we did not want to overfit to the public LB, especially given that there were not that many prompts in the training data. I do wish that we had a bigger distribution of prompts in our training data, perhaps one for each grade (grade 3-12).
\nThe notebooks we used for this competition can be found here:
\n", "comments": [ { "id": 2478585, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2023-10-12T05:10:57.727000", "content": "I think the result is secondary to learning and development @mcpenguin
\nI am sure this learning and associated improvements will help you in all your future endeavors
I really hope i can learn a lot going forward :)
", "votes": 2, "replies": [] } ] }, { "id": 2481372, "author_name": "Joakim Arvidsson", "author_url": "", "post_date": "2023-10-14T04:24:35.557000", "content": "Since the leaderboard hasn’t been finalised yet you might still have a chance at a bronze medal, if there are duplicate entries ahead of you that gets removed. @mcpenguin
", "votes": 1, "replies": [ { "id": 2481404, "author_name": "Marcus Chan", "author_url": "", "post_date": "2023-10-14T05:15:52.743000", "content": "I personally doubt it - about 100-120 people would have to be removed before we can get a medal, and the LB seems pretty finalized already.
", "votes": 0, "replies": [] } ] }, { "id": 2481355, "author_name": "Aayush Patni", "author_url": "", "post_date": "2023-10-14T04:04:46.100000", "content": "It was a pleasure working with both you and @yeoyunsianggeremie on this competition.
", "votes": 1, "replies": [] }, { "id": 2479113, "author_name": "Geremie Yeo", "author_url": "", "post_date": "2023-10-12T12:15:00.540000", "content": "Thank you @mcpenguin for the collaboration in both LLM Science Exam and CommonLit comps, really enjoyed working with you in a team! These were our first pair of non-starter, non-playground competitions and we definitely learnt a lot from both comps. I’m sure we will improve over time with practice and experience! Keep going
", "votes": 1, "replies": [] } ], "comment_count": 6, "messages": [ { "id": 2478443, "postDate": "2023-10-12T02:53:52.337Z", "content": "Thanks to Kaggle and CommonLit for such an amazing competition! This was one of the first non-starter/playground competitions that our team has competed in. Despite our close call I’m still really happy with the results, and this has definitely equipped us with the skills needed to conquer future Kaggle competitions.
\nOur best submission had a public LB score of 0.438, which was in the bronze threshold, but ended up getting a private LB score of 0.485, which was 0.001 away from the bronze threshold :(( That being said, I still wanted to make a post detailing our approach, since our team invested a lot of time into this competition. We found that our CV was around 0.5-0.6 points higher than the public LB, and was 0.1-0.2 points higher than the private LB.
\nThank you to my teammates @yeoyunsianggeremie and @aayushpatni for helping me with this competition! You guys were amazing :))
\nCongratulations to all of the winners! And what a shakeup!
\nWe used a modification of the DebertaV3 + LGBM approach shared by @tsunotsuno in his public notebook. Big kudos to them!
\nWe utilized more features in our submission than were present in the original notebook. In particular,
\nWe ensembled 4 LLM models: 2 DebertaV3 large models trained on individual metrics, 1 deberta v3 large model trained on both metrics, and 1 deberta v3 base model trained on both metrics. The predictions from these models were then fed into individual LGBM models, that had been tuned to each LLM model using Optuna. Finally, the resultant predictions were ensembled using hill climbing.
\nWe found a variety of techniques when pretraining helped with the LLM models’ performance, including
\nautocorrect-1.1.0 because of the whole licensing issue, but we should have looked into alternatives (since this was known to be buggy)In general, we were really cautious about over-engineering our solution as we did not want to overfit to the public LB, especially given that there were not that many prompts in the training data. I do wish that we had a bigger distribution of prompts in our training data, perhaps one for each grade (grade 3-12).
\nThe notebooks we used for this competition can be found here:
\n", "rawMarkdown": "Thanks to Kaggle and CommonLit for such an amazing competition! This was one of the first non-starter/playground competitions that our team has competed in. Despite our close call I’m still really happy with the results, and this has definitely equipped us with the skills needed to conquer future Kaggle competitions.\n\nOur best submission had a public LB score of 0.438, which was in the bronze threshold, but ended up getting a private LB score of 0.485, which was 0.001 away from the bronze threshold :(( That being said, I still wanted to make a post detailing our approach, since our team invested a lot of time into this competition. We found that our CV was around 0.5-0.6 points higher than the public LB, and was 0.1-0.2 points higher than the private LB.\n\nThank you to my teammates @yeoyunsianggeremie and @aayushpatni for helping me with this competition! You guys were amazing :))\n\nCongratulations to all of the winners! And what a shakeup!\n\n# Method\n\nWe used a modification of the DebertaV3 + LGBM approach shared by @tsunotsuno in his public notebook. Big kudos to them!\n\n## Feature Engineering\n\nWe utilized more features in our submission than were present in the original notebook. In particular,\n\n- we added 6-gram and 12-gram features to model copying from the text instead of paraphrasing (which was associated with lower wording scores);\n- we added polarity and subjectivity scores using the NLTK library; and\n- we added counts of the NLTK pos tags in the student summary (e.g. nouns, verbs, exclamations, etc.)\n\n## Modelling\n\nWe ensembled 4 LLM models: 2 DebertaV3 large models trained on individual metrics, 1 deberta v3 large model trained on both metrics, and 1 deberta v3 base model trained on both metrics. The predictions from these models were then fed into individual LGBM models, that had been tuned to each LLM model using Optuna. Finally, the resultant predictions were ensembled using hill climbing.\n\nWe found a variety of techniques when pretraining helped with the LLM models’ performance, including\n\n- freezing the first N layers (N=6 or 18)\n- freezing the embeddings layer\n- using a max seq len of 1024 for training and a max seq len of 512 for inference\n- using step evaluation instead of epoch evaluation\n\n# What we could have done to get that sweet, sweet competition medal\n\n- We used `autocorrect-1.1.0` because of the whole licensing issue, but we should have looked into alternatives (since this was known to be buggy)\n- We could have tried ensembling the boosting models (e.g. using CatBoost / XGB / etc.)\n\nIn general, we were really cautious about over-engineering our solution as we did not want to overfit to the public LB, especially given that there were not that many prompts in the training data. I do wish that we had a bigger distribution of prompts in our training data, perhaps one for each grade (grade 3-12).\n\n# Code\n\nThe notebooks we used for this competition can be found here:\n\n- [Inference/Submission Notebook](https://www.kaggle.com/code/mcpenguin/inference-commonlit-student-summaries)\n- [EDA Notebook](https://www.kaggle.com/code/mcpenguin/eda-commonlit-student-summaries)\n- [Utility Notebook](https://www.kaggle.com/code/mcpenguin/utility-commonlit-student-summaries)", "votes": 14 }, { "id": 2478585, "postDate": "2023-10-12T05:10:57.727Z", "content": "I think the result is secondary to learning and development @mcpenguin
\nI am sure this learning and associated improvements will help you in all your future endeavors
I really hope i can learn a lot going forward :)
", "rawMarkdown": "I really hope i can learn a lot going forward :)", "votes": 2 } ] }, { "id": 2481372, "postDate": "2023-10-14T04:24:35.557Z", "content": "Since the leaderboard hasn’t been finalised yet you might still have a chance at a bronze medal, if there are duplicate entries ahead of you that gets removed. @mcpenguin
", "rawMarkdown": "Since the leaderboard hasn’t been finalised yet you might still have a chance at a bronze medal, if there are duplicate entries ahead of you that gets removed. @mcpenguin ", "votes": 1, "replies": [ { "id": 2481404, "postDate": "2023-10-14T05:15:52.743Z", "content": "I personally doubt it - about 100-120 people would have to be removed before we can get a medal, and the LB seems pretty finalized already.
", "rawMarkdown": "I personally doubt it - about 100-120 people would have to be removed before we can get a medal, and the LB seems pretty finalized already." } ] }, { "id": 2481355, "postDate": "2023-10-14T04:04:46.100Z", "content": "It was a pleasure working with both you and @yeoyunsianggeremie on this competition.
", "rawMarkdown": "It was a pleasure working with both you and @yeoyunsianggeremie on this competition. ", "votes": 1 }, { "id": 2479113, "postDate": "2023-10-12T12:15:00.540Z", "content": "Thank you @mcpenguin for the collaboration in both LLM Science Exam and CommonLit comps, really enjoyed working with you in a team! These were our first pair of non-starter, non-playground competitions and we definitely learnt a lot from both comps. I’m sure we will improve over time with practice and experience! Keep going
", "rawMarkdown": "Thank you @mcpenguin for the collaboration in both LLM Science Exam and CommonLit comps, really enjoyed working with you in a team! These were our first pair of non-starter, non-playground competitions and we definitely learnt a lot from both comps. I’m sure we will improve over time with practice and experience! Keep going", "votes": 1 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T23:00:56Z", "competition_id": 53482, "competition_slug": "commonlit-evaluate-student-summaries", "team_id": 10837292, "solution_url": "/competitions/commonlit-evaluate-student-summaries/writeups/newformers-we-got-so-close" }