# 221st place solution

Congratulations to all competitors! Here is s a summary of my solution.
Overview
My model is ensemble of the two published notebooks and my own model:
model1:
[https://www.kaggle.com/code/jingwora1/fb3-deberta-family-inference-weight-tune](https://www.kaggle.com/code/jingwora1/fb3-deberta-family-inference-weight-tune)
(by
[@jingwora1](https://www.kaggle.com/jingwora1)
)
model2:
[https://www.kaggle.com/code/quangphm/lb-0-43-simple-ensemble-deberta-base-svr](https://www.kaggle.com/code/quangphm/lb-0-43-simple-ensemble-deberta-base-svr)
(by
[@quangphm](https://www.kaggle.com/quangphm)
)
Note: The Fine-tuned DeBERTa models are excluded. The SVR side only uses deberta_large_mnli, roberta_base, roberta_large, xlnet_base and xlnet_large.
model3:  DeBERTa-v3-base/large (4 fold, seed=2024, Layer-wise Decreasing Layer Rate)
Code:
[https://www.kaggle.com/code/fx6300/ensemble-deberta-and-svr/notebook?scriptVersionId=112096434](https://www.kaggle.com/code/fx6300/ensemble-deberta-and-svr/notebook?scriptVersionId=112096434)
Ensemble Weighting Strategy
I didn't employ any optimization algorithms (e.g., Nelder-Mead) since I didn't have good validation set and this kind of algorithms work well on CV but don't always work on private LB.
Ensemble weights are:
model1: 0.8
model2: 0.1
model3: 0.1
What Worked
Layer-wise Decreasing Layer Rate
During this competition, I was highly influenced by the paper titled "How to fine-tune bert for text classification?." [0].  This paper also reports the effectiveness of Layer-wise Decreasing Layer Rate (See 5.3.4)
What Didn't Work
Metric Learning (Triplet Loss)
I couldn't find a proper algorithm for applying metric learning on regression tasks
Use flan-t5 as an encoder
Use t5 as an encoder
Further Pre-training
The above mentioned paper [0] also reports that  Further Pre-training is effective when doing it with in-domain corpus.
However, it didn't work well in my case.
If you are interested in further pre-training, check it:
[https://www.kaggle.com/code/fx6300/further-pre-train-deberta](https://www.kaggle.com/code/fx6300/further-pre-train-deberta)
Re-assign score when given target (e.g., coherence) has score x.5
I thought that score x.5 targets are unreliable.
For example, If given coherence score was 4.5, this means annotator A scored this as 5.0 and annotator B scored this as 4.0.
I thought uncontroversial instances don't cause this kind of discrepancy.
Use sentence generation probability (i.e. LM score of GPT-2) as a feature of a grammar score estimator
Use Random Forest as a vocabulary score estimator
References
[0] Sun, Chi, et al. "How to fine-tune bert for text classification?." China national conference on Chinese computational linguistics. Springer, Cham, 2019.
paper:
[assets/45f51a0884b40913a38bf721.pdf](assets/45f51a0884b40913a38bf721.pdf)