# 6th Place Efficiency Solution Overview Model: 1 fold deberta-v3-small (same params 4 fold cv: 0.448930774, 1 fold public LB: 0.436674, private LB: 0.438798) Training: Pseudo Labelling + Distillation Learning (CustomDistillLoss) Inference: ONNX + ONNX Optimisation ![fb3-6th-efficiency-price-overview](assets/8596384c0fa34584ac9d841b.png) Code Teacher Models Ensemble Submission: [https://www.kaggle.com/code/wuwenmin/fb3-inference-ensemble-colwise/notebook](https://www.kaggle.com/code/wuwenmin/fb3-inference-ensemble-colwise/notebook) Efficiency LB Submission: [https://www.kaggle.com/code/wuwenmin/fb3-onnx-cpu-inference-small-fast-truncate/notebook](https://www.kaggle.com/code/wuwenmin/fb3-onnx-cpu-inference-small-fast-truncate/notebook) ONNX Optimization and Load Test: [https://www.kaggle.com/wuwenmin/fb3-debertav3-small-export-onnx-and-test](https://www.kaggle.com/wuwenmin/fb3-debertav3-small-export-onnx-and-test) What Worked for Both LB and Efficiency LB [high impact] AWP add AWP at the 3rd epoch Result: 4 folds base CV change from 0.450 to 0.449 Reasoning and context: on this fine-tuning task it's easy to get overfitting on the training set, AWP helps to reduce overfitting What worked for Effificeny LB [high impact] DistillLoss refer to this [paper](assets/721a599ad83c2dbd80eeef1f.pdf) Implement the DistillLoss mentioned in the paper and manually set the outlier threshold to 1.0 Result: 4 folds small CV change from 0.452 to 0.449 Reasoning and context: The student model can learn the knowledge from the teacher model, and setting the outlier threshold can avoid the student model from overfitting the outliers [high impact] Pre-train using pseudo labeling on FB1 data Result: 4 folds small CV change from 0.449 to 0.4485 Reasoning and context: With more data the student model able to learn more knowledge from the teacher model [Low Impact] ONNX optimization After several testing the onnx-optimizer always outputs the fast one, so I hard-coded to save the one optimised by onnx-optimizer Result: Got around 30s running time reduced Reasoning and context: the one optimised by onnx-optimiser is around 16MB small than others, thus I think it successfully removed some unnecessary calculations. [Low Impact] intra_op_num_threads No matter whether you set it to 4 or 2, the running time is almost the same Reasoning and context: I guess it's because the submission CPU only has 2 physical cores What Didn't Work Batch Inference for Efficiency I sorted the test set by the text length and tune the batch_size from 8, 16, 128, to 256 Result: the running time increased a lot Reasoning and context: add the batch increase, it will waste more computation on padding tokens Propensity Score Matching I use the propensity score matching to select the subset of FB1 data that has similar distribution as this competition. Result: With these ~5000 subset data, I failed to make the xsmall work Reasoning and context: Even some samples in FB1 data have different distribution they can also help with distillation learning (will verify this when I have time) Additional Context CV Strategy MultiLabelStratifiedKFold 4 folds with seed=42 Important Citations: [assets/721a599ad83c2dbd80eeef1f.pdf](assets/721a599ad83c2dbd80eeef1f.pdf) [https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train](https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train) [https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference](https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference) [https://www.kaggle.com/code/yasufuminakama/fb3-submission-time](https://www.kaggle.com/code/yasufuminakama/fb3-submission-time) Thanks and Acknowledgements: Thanks to the organizers for holding this interesting competition. I'm new to NLP and Transformers. Thanks to the kagglers for sharing very helpful codes and discussions, especially [@meli19](https://www.kaggle.com/meli19) and [@xyzdivergence](https://www.kaggle.com/xyzdivergence) . I've learned a lot through this competition.