# Efficiency 7th / Private 84st place solution

Overview
I took the average of 6 models with a 3 seed average in the final submission of LB.
For efficiency LB, I used deberta-v3-small with knowledge distillation and small max_len(=384), which took 0:16:42 min for submission and scored 0.438537 for public LB, 0.439835 for private LB.
![](assets/d8c7472c884e9f24448c71c4.png)
Code
LB submission:
[https://www.kaggle.com/code/tmhrkt/fb3-ensemble](https://www.kaggle.com/code/tmhrkt/fb3-ensemble)
Efficiency LB:
[https://www.kaggle.com/code/tmhrkt/fb3-efficiency?scriptVersionId=112206338](https://www.kaggle.com/code/tmhrkt/fb3-efficiency?scriptVersionId=112206338)
SVR training:
[https://www.kaggle.com/code/tmhrkt/fb3-embedding-learning-training](https://www.kaggle.com/code/tmhrkt/fb3-embedding-learning-training)
github:
[https://github.com/ktm98/kaggle-feedback-prize-3](https://github.com/ktm98/kaggle-feedback-prize-3)
What Worked For Both LB and Efficiency LB
[high Impact] Changing max_len
CV improved 0.002, 0.001 (1024 -> 512, 1024 -> 768 each)
Submission speeded up about 4 minutes (512 -> 384)
Reasoning and context: trying several max_len,
max_len<384
got worse results compared to
maxlen=1024
, 1428. And oof correlations between different max_len (0.97+ for conventions) were lower than others (0.98+). So ensembling different max_len improved my CV score.
[High Impact] Knowledge Distillation
CV improved 0.014+ for deberta-v3-small
What I did: set loss function as
criterion(y_preds, labels) + criterion(y_preds, oof_preds)
Reasoning and context: I tried KD for efficiency LB. However, the impact for both CV and LB was higher than expected, so I added KD models for LB.
What Worked For Efficiency LB
[Low Impact] num_workers=1
Mean inference speed got faster about 5 seconds for the first 200 rows of
train.csv
.
Reasoning and context: Up until midway through the competition, I was using
num_workers=0
. Changing
num_workers=1
slightly speeded up inference time.
What Did’t Work
unscale
What I did: Adding unscale described in
[nakama’s notebook](https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train/comments)
comments.
Result: CV score got worse.
XGBoost For embeddings
What I did: Train by XGBoost instead of Rapids SVR .
Result: CV score got worse.
TorchScript
What I did: Converting PyTorch model to TorchScript.
Result: Inference time was too slow. Stopped while inferencing.
Reasoning and context: TorchScript does not allow dynamic sequence length. So If the tokenized length of a sentence is shorter than the max length, the attention mask portion must also be calculated, resulting in wasted computation time.
Dynamic Quantization
What I did: Apply dynamic quantization for my pretrained model.
Result: LB score significantly dropped (0.43 -> 0.64)
Reasoning and Context: Because quantizing
nn.Linear
layers directly affects the output of the model, quantizing layers is not suitable for regression tasks.
Additional Context
CV Strategy
MultiLabelStratifiedKFold
with binned
n_words
Efficiency submissions
First, I trained 4 folds to check the performance and submitted 4 folds.
After checking the performance, the model was trained with whole FB3 data.
model
max_len
submission time
Public
Private
notes
deberta-v3-small
384
0:16:42
0.4398
0.4385
final sub
deberta-v3-small
290
0:15:07
0.4430
0.4435
deberta-v3-small
512
0:21:33
0.4388
0.4383
deberta-v3-xsmall
768
0:19:07
0.4396
0.4413
deberta-v3-xsmall
1428
0:20:11
0.4402
0.4394
2 process
Important Citations:
[https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train](https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train)
[https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference](https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference)
[https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x](https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x)
[https://www.kaggle.com/code/kojimar/fb3-single-pytorch-model-train](https://www.kaggle.com/code/kojimar/fb3-single-pytorch-model-train)
Thanks and Acknowledgements:
Thanks to the organizers for holding this interesting competition. Thanks to the kagglers for sharing very helpful codes and discussions.