# Efficiency 17th Place Solution 

Overview
We are a group of three students representing
[Penn Data Science Group](https://penndsg.com/)
. We had a lot of fun working on this competition and thank the Competition Host and Kaggle for hosting.
Here goes our Efficiency Solution.
We pseudolabeled the 15K essays from FB1 and FB2 using our own ensemble and then finetuned deberta-v3-xsmall for Efficiency.
The ensemble has two components:
Pretrained Embeddings:
We concatenated the mean-pooled hidden states from different debertas and different layers therein and trained MLPs and KernelRidge.
The hidden states add to over 20K in dimensions; interestingly, standardization / min-max scaling (which seems imperative to SVM etc.) leads to significantly worse result. This may be due to the layer normalization.
KernelRidge uses
rbf
as kernel, with default
gamma
, and with
alpha
set to 0.075.
MLP is structurally as follows:
![](assets/cb58ab2b991b5cfa5ef99c5f.png)
We used
[Ranger](https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer)
as optimizer for the MLP. Per recommended,
beta_1
is set to 0.95, and the learning schedule is flat for the first 60-75% and then on a cosine decay for the remaining epochs. We set the flat rate as
3e-4
for the first 10 epochs, and then used cosine decay for the final 6 epochs, finishing on a learning rate of
9e-7
(in  other words,
alpha
is set to
3e-3
). The batch size is 64, and we used a weight decay of
1e-1
.
To stablize results, we trained 50 iterations and took the mean prediction.
Finetuned Models:
We kinda sucked at finetuning models, so we took
[kojimar](https://www.kaggle.com/kojimar)
's models. Specifically, we took
[deberta-v2-xlarge-mnli](https://www.kaggle.com/datasets/kojimar/0919-deberta-v2-xlarge-mnli)
,
[deberta-v2-xlarge](https://www.kaggle.com/datasets/kojimar/0919-deberta-v2-xlarge)
,
[deberta-v3-large-unscale](https://www.kaggle.com/datasets/kojimar/0926-deberta-v3-large-unscale)
, and
[20221012-deberta-v3-large](https://www.kaggle.com/datasets/kojimar/20221012-123357-deberta-v3-large)
.
This ensemble gave us an unimpressive 594th place on the private LB w/ 0.436685 and 1061th place on the public LB w/ 0.439137.
The deberta-v3-xsmall is finetuned with the 4K essays from this competition with their true labels and the 15K essays from the previous two competitions with the pseudolabels.
This model gave us 17th place on the Efficiency Private LB w/ 0.442 in 1336 seconds.
What Worked for Ensemble
MLP & Kernel Ridge for Pretrained Embeddings
Result: 0.446 CV as opposed to 0.451 CV for SVM.
These two methods allow for a nonlinear specification whilst maintaining, if trained properly, robustness.
SVM seems to be worse because of differences in objective and penalty (in direct comparison with Kernel Ridge). Actually, Bayesian Ridge with its linear specification can consistently beat SVM.
mish and Ranger on MLP
Result: 0.447 CV as opposed to 0.449 with swish and Ranger.
Including
Some
Finetuned Models
Results: 0.445 CV.
What Worked for Efficiency
Psuedolabelling
Result: The xsmall model can closely trace the CV performance of the ensemble (I don't have the data here because we could not cross-validate the entire ensemble, but just take my words that an xsmall trained with only the 4K essays do worse).
Knowledge distillation.
deberta-v3-xsmall
deberta-v3-xsmall performs better than deberta-v3-small, because it has 12 layers with hidden dimension as 384 as opposed to 6 layers with 768. The training/inference time is about the same, despite the increased layer counts.
Interestingly, a base model I late-submitted performed much worse at 0.44674 private LB.
Edit:
Weight decay on weights
Place no weight decay on the bias and normalization terms; useful for both finetuning and the training of the MLP.
What Did Not Work
Translation for data augmentation.
Finetuning techniques (Multi Sample Dropout, Layer Weight Reinitialization, Layerwise Learning Rate Adaptation).
I cannot quantify their improvements, and it seems that the increased sample size due to pseudolabelling offsets some benefits from these techniques.
I submitted a version without MSD on the last date (but forgot to select it 😫); that had a private LB of 0.44 with probably a lower runtime (I didn't bother to prune the dropout, so the model technically stills feeds forward the last layer 8 times before taking an average); then again, I also changed some hyperparameters (lowering weight decay from 1e-1 to 1e-2 and increasing
beta_1
from 0.9 to 0.95).
What we should have done
Use PyTorch? It inferences faster and without the pain of memory leakage. This would also allow for the usage efficienct data pipelines from huggingface. I suspect sub-1000s would need this and some more.
Important Citations
[kojimar](https://www.kaggle.com/kojimar)
: Excellently finetuned models; we probably should have taken more models from you.
[Chris Deotte](https://www.kaggle.com/cdeotte)
: As far as I can tell, the first person to share using pretrained embeddings.
[Ranger](https://github.com/lessw2020/Ranger-Deep-Learning-Optimizer)
: Robust and reliable optimizer.
Thanks and Acknowledgement
We thank the Competition Host and Kaggle for hosting and all those whom we cited for their insights.
Team Members
[Randy Li](https://www.kaggle.com/randyli2)
[Mangara Paul Alexander Hutagalung](https://www.kaggle.com/mangarahutagalung)
[Chris Sebastian](https://www.kaggle.com/msgchris)