![\"fb3-6th-efficiency-price-overview\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3563032%2F5c44ab0fb0cc21a53301ac3fc26b1238%2Ffb3-6th-efficiency-solution.drawio.png?generation=1669819431207768&alt=media\")
Teacher Models Ensemble Submission: https://www.kaggle.com/code/wuwenmin/fb3-inference-ensemble-colwise/notebook
\nEfficiency LB Submission: https://www.kaggle.com/code/wuwenmin/fb3-onnx-cpu-inference-small-fast-truncate/notebook
\nONNX Optimization and Load Test: https://www.kaggle.com/wuwenmin/fb3-debertav3-small-export-onnx-and-test
[high impact] DistillLoss refer to this paper
\n[high impact] Pre-train using pseudo labeling on FB1 data
\n[Low Impact] ONNX optimization
\n[Low Impact] intra_op_num_threads
Batch Inference for Efficiency
\nPropensity Score Matching
\nMultiLabelStratifiedKFold 4 folds with seed=42
Thanks to the organizers for holding this interesting competition. I'm new to NLP and Transformers. Thanks to the kagglers for sharing very helpful codes and discussions, especially @meli19 and @xyzdivergence. I've learned a lot through this competition.
", "comments": [ { "id": 2050347, "author_name": "Santhoshkumar", "author_url": "", "post_date": "2022-11-30T17:19:17.580000", "content": "Very detailed writeupđź‘Ť learnt a lot from you @wuwenmin, thanks for sharing đź‘Ź
", "votes": 1, "replies": [] }, { "id": 2050229, "author_name": "Raja Biswas", "author_url": "", "post_date": "2022-11-30T15:42:33.440000", "content": "Congratulations & many thanks for the detailed write up! Nice pipeline, will definitely checkout the distillation loss!
\nThanks for sharing ideas during the competition as well 👍 Appreciate your curiosity to learn details of the shared solutions, when it is so easy to get discouraged by shake down…personally, will try to have the same mindset when I find myself in similar situation!
", "votes": 1, "replies": [ { "id": 2050773, "author_name": "william.wu", "author_url": "", "post_date": "2022-12-01T01:43:10.163000", "content": "Congrats on your 9th LB!
\nI just treat my shake-down of this competition as a negative sample to learn from. I learned a lot from other kagglers through this competition, that's mostly important:)
", "votes": 2, "replies": [] }, { "id": 2050785, "author_name": "Raja Biswas", "author_url": "", "post_date": "2022-12-01T02:05:56.303000", "content": "fully agree!
", "votes": 0, "replies": [] } ] }, { "id": 2050213, "author_name": "Chris Deotte", "author_url": "", "post_date": "2022-11-30T15:33:17.320000", "content": "Congratulations @wuwenmin You did awesome! Thank you for all your sharing during the competition. You helped everyone. I am sorry about your shakedown on LB. Congrats on solo 6th efficiency!
", "votes": 2, "replies": [ { "id": 2050217, "author_name": "william.wu", "author_url": "", "post_date": "2022-11-30T15:36:13.193000", "content": "Thanks, @cdeotte, and Congrats on your 3rd LB! I'm new to NLP and Transformers. I've learned a lot from other kagglers through this competition. I'll keep learning and participating:)
", "votes": 3, "replies": [] } ] }, { "id": 2052113, "author_name": "DJ_Xia", "author_url": "", "post_date": "2022-12-02T00:10:42.780000", "content": "thank you for the detailed description of the comp solution, you help me a lot
", "votes": 0, "replies": [] }, { "id": 2051024, "author_name": "yukiya", "author_url": "", "post_date": "2022-12-01T06:50:56.173000", "content": "Thank you for sharing your solution and your journey throughout this competition. You're awesome !
", "votes": 0, "replies": [] }, { "id": 2050772, "author_name": "Xiang Huang", "author_url": "", "post_date": "2022-12-01T01:41:39.727000", "content": "Congrats on 6th efficiency, Thanks for sharing, your sharing helps a lot!
", "votes": 0, "replies": [] } ], "comment_count": 9, "messages": [ { "id": 2050201, "postDate": "2022-11-30T15:27:56.323Z", "content": "
Teacher Models Ensemble Submission: https://www.kaggle.com/code/wuwenmin/fb3-inference-ensemble-colwise/notebook
\nEfficiency LB Submission: https://www.kaggle.com/code/wuwenmin/fb3-onnx-cpu-inference-small-fast-truncate/notebook
\nONNX Optimization and Load Test: https://www.kaggle.com/wuwenmin/fb3-debertav3-small-export-onnx-and-test
[high impact] DistillLoss refer to this paper
\n[high impact] Pre-train using pseudo labeling on FB1 data
\n[Low Impact] ONNX optimization
\n[Low Impact] intra_op_num_threads
Batch Inference for Efficiency
\nPropensity Score Matching
\nMultiLabelStratifiedKFold 4 folds with seed=42
Thanks to the organizers for holding this interesting competition. I'm new to NLP and Transformers. Thanks to the kagglers for sharing very helpful codes and discussions, especially @meli19 and @xyzdivergence. I've learned a lot through this competition.
", "rawMarkdown": "# Overview\n\n* **Model:** 1 fold deberta-v3-small (same params 4 fold cv: 0.448930774, 1 fold public LB: 0.436674, private LB: 0.438798)\n* **Training:** Pseudo Labelling + Distillation Learning (CustomDistillLoss)\n* **Inference:** ONNX + ONNX Optimisation\n\n\n\n# Code\nTeacher Models Ensemble Submission: https://www.kaggle.com/code/wuwenmin/fb3-inference-ensemble-colwise/notebook\nEfficiency LB Submission: https://www.kaggle.com/code/wuwenmin/fb3-onnx-cpu-inference-small-fast-truncate/notebook\nONNX Optimization and Load Test: https://www.kaggle.com/wuwenmin/fb3-debertav3-small-export-onnx-and-test\n\n# What Worked for Both LB and Efficiency LB\n* [high impact] AWP\n * add AWP at the 3rd epoch\n * Result: 4 folds base CV change from 0.450 to 0.449\n * Reasoning and context: on this fine-tuning task it's easy to get overfitting on the training set, AWP helps to reduce overfitting\n \n# What worked for Effificeny LB\n* [high impact] DistillLoss refer to this [paper](https://arxiv.org/pdf/2002.12597.pdf) \n * Implement the DistillLoss mentioned in the paper and manually set the outlier threshold to 1.0\n * Result: 4 folds small CV change from 0.452 to 0.449\n * Reasoning and context: The student model can learn the knowledge from the teacher model, and setting the outlier threshold can avoid the student model from overfitting the outliers\n* [high impact] Pre-train using pseudo labeling on FB1 data\n * Result: 4 folds small CV change from 0.449 to 0.4485\n * Reasoning and context: With more data the student model able to learn more knowledge from the teacher model\n\n* [Low Impact] ONNX optimization\n * After several testing the onnx-optimizer always outputs the fast one, so I hard-coded to save the one optimised by onnx-optimizer\n * Result: Got around 30s running time reduced\n * Reasoning and context: the one optimised by onnx-optimiser is around 16MB small than others, thus I think it successfully removed some unnecessary calculations.\n\n* [Low Impact] `intra_op_num_threads`\n * No matter whether you set it to 4 or 2, the running time is almost the same\n * Reasoning and context: I guess it's because the submission CPU only has 2 physical cores\n\n# What Didn't Work\n* Batch Inference for Efficiency\n * I sorted the test set by the text length and tune the batch_size from 8, 16, 128, to 256\n * Result: the running time increased a lot\n * Reasoning and context: add the batch increase, it will waste more computation on padding tokens\n\n* Propensity Score Matching\n * I use the propensity score matching to select the subset of FB1 data that has similar distribution as this competition.\n * Result: With these ~5000 subset data, I failed to make the xsmall work\n * Reasoning and context: Even some samples in FB1 data have different distribution they can also help with distillation learning (will verify this when I have time)\n\n# Additional Context\n## CV Strategy\n`MultiLabelStratifiedKFold` 4 folds with seed=42\n\n## Important Citations:\n* https://arxiv.org/pdf/2002.12597.pdf\n* https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\n* https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference\n* https://www.kaggle.com/code/yasufuminakama/fb3-submission-time\n\n# Thanks and Acknowledgements:\nThanks to the organizers for holding this interesting competition. I'm new to NLP and Transformers. Thanks to the kagglers for sharing very helpful codes and discussions, especially @meli19 and @xyzdivergence. I've learned a lot through this competition.", "votes": 31 }, { "id": 2050347, "postDate": "2022-11-30T17:19:17.580Z", "content": "Very detailed writeupđź‘Ť learnt a lot from you @wuwenmin, thanks for sharing đź‘Ź
", "rawMarkdown": "Very detailed writeupđź‘Ť learnt a lot from you @wuwenmin, thanks for sharing đź‘Ź", "votes": 1 }, { "id": 2050229, "postDate": "2022-11-30T15:42:33.440Z", "content": "Congratulations & many thanks for the detailed write up! Nice pipeline, will definitely checkout the distillation loss!
\nThanks for sharing ideas during the competition as well 👍 Appreciate your curiosity to learn details of the shared solutions, when it is so easy to get discouraged by shake down…personally, will try to have the same mindset when I find myself in similar situation!
", "rawMarkdown": "Congratulations & many thanks for the detailed write up! Nice pipeline, will definitely checkout the distillation loss! \n\nThanks for sharing ideas during the competition as well đź‘Ť Appreciate your curiosity to learn details of the shared solutions, when it is so easy to get discouraged by shake down...personally, will try to have the same mindset when I find myself in similar situation!", "votes": 1, "replies": [ { "id": 2050773, "postDate": "2022-12-01T01:43:10.163Z", "content": "Congrats on your 9th LB!
\nI just treat my shake-down of this competition as a negative sample to learn from. I learned a lot from other kagglers through this competition, that's mostly important:)
", "rawMarkdown": "Congrats on your 9th LB!\n\nI just treat my shake-down of this competition as a negative sample to learn from. I learned a lot from other kagglers through this competition, that's mostly important:)", "votes": 2 }, { "id": 2050785, "postDate": "2022-12-01T02:05:56.303Z", "content": "fully agree!
", "rawMarkdown": "fully agree!" } ] }, { "id": 2050213, "postDate": "2022-11-30T15:33:17.320Z", "content": "Congratulations @wuwenmin You did awesome! Thank you for all your sharing during the competition. You helped everyone. I am sorry about your shakedown on LB. Congrats on solo 6th efficiency!
", "rawMarkdown": "Congratulations @wuwenmin You did awesome! Thank you for all your sharing during the competition. You helped everyone. I am sorry about your shakedown on LB. Congrats on solo 6th efficiency!", "votes": 2, "replies": [ { "id": 2050217, "postDate": "2022-11-30T15:36:13.193Z", "content": "Thanks, @cdeotte, and Congrats on your 3rd LB! I'm new to NLP and Transformers. I've learned a lot from other kagglers through this competition. I'll keep learning and participating:)
", "rawMarkdown": "Thanks, @cdeotte, and Congrats on your 3rd LB! I'm new to NLP and Transformers. I've learned a lot from other kagglers through this competition. I'll keep learning and participating:)", "votes": 3 } ] }, { "id": 2052113, "postDate": "2022-12-02T00:10:42.780Z", "content": "thank you for the detailed description of the comp solution, you help me a lot
", "rawMarkdown": "thank you for the detailed description of the comp solution, you help me a lot" }, { "id": 2051024, "postDate": "2022-12-01T06:50:56.173Z", "content": "Thank you for sharing your solution and your journey throughout this competition. You're awesome !
", "rawMarkdown": "Thank you for sharing your solution and your journey throughout this competition. You're awesome !" }, { "id": 2050772, "postDate": "2022-12-01T01:41:39.727Z", "content": "Congrats on 6th efficiency, Thanks for sharing, your sharing helps a lot!
", "rawMarkdown": "Congrats on 6th efficiency, Thanks for sharing, your sharing helps a lot!" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T20:01:27Z", "competition_id": 38321, "competition_slug": "feedback-prize-english-language-learning", "team_id": 9105652, "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/skyfish-6th-place-efficiency-solution" }