# 5th place solution

First of all, thanks to competition organizers for hosting this competition and great teammates (
[@shinomoriaoshi](https://www.kaggle.com/shinomoriaoshi)
,
[@horsek](https://www.kaggle.com/horsek)
,
[@runningz](https://www.kaggle.com/runningz)
,
[@nickycan](https://www.kaggle.com/nickycan)
).
Thanks also to the community for sharing many ideas in Notebook and Discussion.
Note: This post is a brief summary, and more detailed information will be updated or posted as a new topic by my teammates.
[tri's pipeline](https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347369)
[housuke's pipeline](https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347379)
Summary
We ensembled 6 token classification models and 1 seq classification model.
![](assets/a066b64f4c9605388f77cb20.png)
Models
We trained following models and used them for final submission.
tri(
[@shinomoriaoshi](https://www.kaggle.com/shinomoriaoshi)
)'s pipeline
token classification Deberta-v3-large
housuke(
[@horsek](https://www.kaggle.com/horsek)
)'s pipeline
token classification Deberta-v3-large
token classification Deberta-large
nakama(
[@yasufuminakama](https://www.kaggle.com/yasufuminakama)
)'s pipeline
token classification Deberta-v3-large
seq classification Deberta-v3-large
RunningZ (
[@runningz](https://www.kaggle.com/runningz)
)'s pipeline
token classification Deberta-v3-large
token classification Deberta-v2-xlarge
鲲 (
[@nickycan](https://www.kaggle.com/nickycan)
)'s pipeline
mostly engaged on efficiency track
Main methods that worked
MLM pretraining
[assets/eb200b1e6c64145e9c7f6a06.pdf](assets/eb200b1e6c64145e9c7f6a06.pdf)
Resolve encoding error
This method was used in previous 2021 Feedback Prize competition.
[https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313330](https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313330)
Mask augmentation
This method was used in previous 2021 Feedback Prize competition.
[https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313424](https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313424)
Adversarial training (AWP, FGM)
This method was used in previous 2021 Feedback Prize competition.
[https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313177](https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313177)
Multi-sample dropout
This method was used in Google QUEST Q&A Labeling competition.
[https://www.kaggle.com/competitions/google-quest-challenge/discussion/129840](https://www.kaggle.com/competitions/google-quest-challenge/discussion/129840)
Add GRU layer
Label smoothing
Add discourse_type for each discourse_text
Add [head] [tail] tokens for each discourse_text
Back translation (worked only for RunningZ's pipeline)
Pseudo labeling
2nd stage Stacking using LSTM
housuke(
[@horsek](https://www.kaggle.com/horsek)
) tried this early in the competition, and it worked very well.
3rd stage Stacking using XGBoost
After the 2nd stage Stacking, we applied 3rd stage Stacking using XGBoost, it improved result a bit.
Final Result
CV: 0.5609
Public LB: 0.555
Private LB: 0.560