# 80th Place Solution (Single Model)

First of all, thanks to the competition organizers for a great competition, and all the people who published great notebooks, which everyone was able to learn from. Additionally,
[@nbroad](https://www.kaggle.com/nbroad)
[notebook](https://www.kaggle.com/code/nbroad/token-classification-approach-fpe/notebook)
was a gamechanger, because it not only improved results drastically, but also cut down runtime ten fold.
My final solution only used a single model, as I didn't have enough time to ensemble multiple models. The model I used was of course, deberta large, as it consistently produced the best results (cv and lb).
Training Process:
As mentioned above, I used the token classification approach proposed by
[@nbroad](https://www.kaggle.com/nbroad)
as it gave a major boost in cv, and lb. Additionally, I changed the pooler to a weighted layer pooler, which seems to give the best results across many competitions/tasks. These changes helped me break 0.600 on the public leaderboard. To get the model down to 0.589 (0.590 private lb), I added an auxiliary task which stabilized training, and also of course improved cv and lb. The auxiliary task was another classification task, where the model predicted the type of discourse, in addition to it's effectiveness. For example, given an essay, the model would predict what each discourse was, and it's effectiveness.
Things that worked:
token classification
multi-task learning
weighted pooler
increased epochs for scheduler
Things that didn't work:
mlm
psuedo labeling (I had a feeling that the winners would be those who could figure out how to get this to work, but I never got convincing results in my experiments)
contrastive learning
using the last competitions task as an auxiliary task
Conclusion
I was very happy to medal in this competition, as this is my first competition medal. I am very curious to understand how the top solutions utilized the extra unlabeled data.
Training code:
[Training Notebook](https://www.kaggle.com/code/ryanbarretto/debertalargeauxtraining)