# 88th place solution

Firstly, I would like to competition's organizers for such a great competition and congrats to the winners, you did amazing efforts! Secondly, I would like to thank my great teammates: Mark (
[@e0xextazy](https://www.kaggle.com/e0xextazy)
) , Otabek (
[@obatek](https://www.kaggle.com/obatek)
), and Sanskar (
[@odins0n](https://www.kaggle.com/odins0n)
) ! Thank you guys for the spent time, experience and emotions!
Here, I am going to describe our final solution, which could reach 0.594 and 0.592 on Public Leaderboard and Private Leaderboard leaderboard respectively, thereby getting us to 88th place and giving us the bronze medal (I and Otabek will become Kaggle Experts 😄).
Unfortunately, in the middle of the competition, we switched to the AI4Code competition and when we switched back we only had ~1 week to try out some other approaches and ideas. We used only 4x models in our ensemble:
[one of the high-scoring public notebooks](https://www.kaggle.com/code/brandonhu0215/feedback-deberta-large-lb0-619)
, Token Classification, a model with fine-tuned weights from the previous Feedback Prize competition, and a model with quite strong pre-processing. Such diversity of models permanently improves our Public Leaderboard from 0.610 to 0.594.
Cross-Validation
For classification, we used grouping by
essay_id
and stratify by
discourse_effectiveness
. Multi-label Stratified Cross-validation was used for Token Classification, i.e stratify by counting the number of entries of each
discourse_effectiveness
in the given essay.
Useful attempts
Longer training for 4-5 epochs
Using pre-trained weights from the previous Feedback Prize competition.
We used
[@cdeotte](https://www.kaggle.com/cdeotte)
's fine-tuned weights of the DeBERTa BiLSTM model.
Models diversity
3x classification models 0.61 -> replace one classification model with token-classification model 0.598 -> add one more token classification model 0.594.
Useless attempts
Masked Language Modeling pre-training
Pseudo-Labeling
We tried it yesterday when we had quite a strong ensemble, but pseudo-labels generating and then training took ~20 hours, so we just tried Hard Pseudo-Labeling, but it does not work well. Unfortunately, after reading other teams' solutions, I am very sad, that we didn't have much time two try more approaches with Pseudo-Labeling, especially Soft Pseudo-Labeling, which could give us the silver medal.
Code
Final ensemble:
[https://www.kaggle.com/code/vad13irt/fp-final-ensemble/notebook](https://www.kaggle.com/code/vad13irt/fp-final-ensemble/notebook)