# We got so close! :(((

Thanks to Kaggle and CommonLit for such an amazing competition! This was one of the first non-starter/playground competitions that our team has competed in. Despite our close call I’m still really happy with the results, and this has definitely equipped us with the skills needed to conquer future Kaggle competitions.
Our best submission had a public LB score of 0.438, which was in the bronze threshold, but ended up getting a private LB score of 0.485, which was 0.001 away from the bronze threshold :(( That being said, I still wanted to make a post detailing our approach, since our team invested a lot of time into this competition. We found that our CV was around 0.5-0.6 points higher than the public LB, and was 0.1-0.2 points higher than the private LB.
Thank you to my teammates
[@yeoyunsianggeremie](https://www.kaggle.com/yeoyunsianggeremie)
and
[@aayushpatni](https://www.kaggle.com/aayushpatni)
for helping me with this competition! You guys were amazing :))
Congratulations to all of the winners! And what a shakeup!
Method
We used a modification of the DebertaV3 + LGBM approach shared by
[@tsunotsuno](https://www.kaggle.com/tsunotsuno)
in his public notebook. Big kudos to them!
Feature Engineering
We utilized more features in our submission than were present in the original notebook. In particular,
we added 6-gram and 12-gram features to model copying from the text instead of paraphrasing (which was associated with lower wording scores);
we added polarity and subjectivity scores using the NLTK library; and
we added counts of the NLTK pos tags in the student summary (e.g. nouns, verbs, exclamations, etc.)
Modelling
We ensembled 4 LLM models: 2 DebertaV3 large models trained on individual metrics, 1 deberta v3 large model trained on both metrics, and 1 deberta v3 base model trained on both metrics. The predictions from these models were then fed into individual LGBM models, that had been tuned to each LLM model using Optuna. Finally, the resultant predictions were ensembled using hill climbing.
We found a variety of techniques when pretraining helped with the LLM models’ performance, including
freezing the first N layers (N=6 or 18)
freezing the embeddings layer
using a max seq len of 1024 for training and a max seq len of 512 for inference
using step evaluation instead of epoch evaluation
What we could have done to get that sweet, sweet competition medal
We used
autocorrect-1.1.0
because of the whole licensing issue, but we should have looked into alternatives (since this was known to be buggy)
We could have tried ensembling the boosting models (e.g. using CatBoost / XGB / etc.)
In general, we were really cautious about over-engineering our solution as we did not want to overfit to the public LB, especially given that there were not that many prompts in the training data. I do wish that we had a bigger distribution of prompts in our training data, perhaps one for each grade (grade 3-12).
Code
The notebooks we used for this competition can be found here:
[Inference/Submission Notebook](https://www.kaggle.com/code/mcpenguin/inference-commonlit-student-summaries)
[EDA Notebook](https://www.kaggle.com/code/mcpenguin/eda-commonlit-student-summaries)
[Utility Notebook](https://www.kaggle.com/code/mcpenguin/utility-commonlit-student-summaries)