# 333rd place...sharing my findings

Firstly, huge thanks for organizing this competition and everyone who has shared their ideas in the notebooks and discussion forums!
Approach
My solution is built upon
[@tsunotsuno](https://www.kaggle.com/tsunotsuno)
's effective idea of stacking Debertav3base with LGBM. I mostly played around with adding more nlp features and tweaking with the model.
What worked
Freezing the first 4 encoder layers gave the best results
Using CLS token output for regression was effective
Adding the 6 features from
[FB3 commonlit competition](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/overview)
gave a good boost
Adding textstat features such as
automated_readability_index, smog_index, dale_chall_readability_score, gunning_fog, reading_time, difficult_words
showed improvement
Addding cosine similarity between prompt text and text embeddings also helped to a small extent
Removing
length_ratio
also showed small improvement
What didn't work
Freezing embedding layers deteriorated performance
Mean pooling/ GeM pooling
Autocorrect didn't improve score
Tried other models such as Roberta, Electra which were not as good as Debertav3base. Even ensembling Debertav3base with them couldn't help
Training separate models for each target was not effective. In fact, content scores improved when trained together with wording scores
I wanted to include
prompt_text
and also try using Debertav3large, however it didn't turn out well this time :(
Although I couldn't earn a medal, my learnings through this competition have been immense and I'm very happy to have come this far. I've already learnt a lot from the teams who have published their solutions and I look forward to participating in the next competitions!