{
  "id": 369540,
  "title": "Efficiency 7th / Private 84st place solution",
  "content": "<h1>Overview</h1>\n<p>I took the average of 6 models with a 3 seed average in the final submission of LB.  <br>\nFor efficiency LB, I used deberta-v3-small with knowledge distillation and small max_len(=384), which took 0:16:42 min for submission and scored 0.438537 for public LB, 0.439835 for private LB.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5017202%2Fa680c252509a2029174d7800c59a0389%2Fsolution_summary.png?generation=1669809477192815&amp;alt=media\" alt=\"\"></p>\n<h1>Code</h1>\n<p>LB submission: <a href=\"https://www.kaggle.com/code/tmhrkt/fb3-ensemble\" target=\"_blank\">https://www.kaggle.com/code/tmhrkt/fb3-ensemble</a>  <br>\nEfficiency LB: <a href=\"https://www.kaggle.com/code/tmhrkt/fb3-efficiency?scriptVersionId=112206338\" target=\"_blank\">https://www.kaggle.com/code/tmhrkt/fb3-efficiency?scriptVersionId=112206338</a>  <br>\nSVR training: <a href=\"https://www.kaggle.com/code/tmhrkt/fb3-embedding-learning-training\" target=\"_blank\">https://www.kaggle.com/code/tmhrkt/fb3-embedding-learning-training</a>  <br>\ngithub: <a href=\"https://github.com/ktm98/kaggle-feedback-prize-3\" target=\"_blank\">https://github.com/ktm98/kaggle-feedback-prize-3</a>  </p>\n<h1>What Worked For Both LB and Efficiency LB</h1>\n<ul>\n<li>[high Impact] Changing max_len<ul>\n<li>CV improved 0.002, 0.001 (1024 -&gt; 512, 1024 -&gt; 768 each)</li>\n<li>Submission speeded up about 4 minutes (512 -&gt; 384)</li>\n<li>Reasoning and context: trying several max_len, <code>max_len&lt;384</code> got worse results compared to <code>maxlen=1024</code>, 1428. And oof correlations between different max_len (0.97+ for conventions) were lower than others (0.98+). So ensembling different max_len improved my CV score.</li></ul></li>\n<li>[High Impact] Knowledge Distillation<ul>\n<li>CV improved 0.014+ for deberta-v3-small</li>\n<li>What I did: set loss function as    <code>criterion(y_preds, labels) + criterion(y_preds, oof_preds)</code></li>\n<li>Reasoning and context: I tried KD for efficiency LB. However, the impact for both CV and LB was higher than expected, so I added KD models for LB.</li></ul></li>\n</ul>\n<h1>What Worked For Efficiency LB</h1>\n<ul>\n<li>[Low Impact] num_workers=1<ul>\n<li>Mean inference speed got faster about 5 seconds for the first 200 rows of <code>train.csv</code>.</li>\n<li>Reasoning and context: Up until midway through the competition, I was using <code>num_workers=0</code>. Changing <code>num_workers=1</code> slightly speeded up inference time.</li></ul></li>\n</ul>\n<h1>What Did’t Work</h1>\n<ul>\n<li>unscale<ul>\n<li>What I did: Adding unscale described in <a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train/comments\" target=\"_blank\">nakama’s notebook</a> comments.</li>\n<li>Result: CV score got worse.</li></ul></li>\n<li>XGBoost For embeddings<ul>\n<li>What I did: Train by XGBoost instead of Rapids SVR .</li>\n<li>Result: CV score got worse.</li></ul></li>\n<li>TorchScript<ul>\n<li>What I did: Converting PyTorch model to TorchScript.</li>\n<li>Result: Inference time was too slow. Stopped while inferencing.</li>\n<li>Reasoning and context: TorchScript does not allow dynamic sequence length. So If the tokenized length of a sentence is shorter than the max length, the attention mask portion must also be calculated, resulting in wasted computation time.</li></ul></li>\n<li>Dynamic Quantization<ul>\n<li>What I did: Apply dynamic quantization for my pretrained model.</li>\n<li>Result: LB score significantly dropped (0.43 -&gt; 0.64)</li>\n<li>Reasoning and Context: Because quantizing <code>nn.Linear</code> layers directly affects the output of the model, quantizing layers is not suitable for regression tasks.  </li></ul></li>\n</ul>\n<h1>Additional Context</h1>\n<h2>CV Strategy</h2>\n<p><code>MultiLabelStratifiedKFold</code> with binned <code>n_words</code></p>\n<h2>Efficiency submissions</h2>\n<p>First, I trained 4 folds to check the performance and submitted 4 folds.  <br>\nAfter checking the performance, the model was trained with whole FB3 data.  </p>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>max_len</th>\n<th>submission time</th>\n<th>Public</th>\n<th>Private</th>\n<th>notes</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>deberta-v3-small</td>\n<td>384</td>\n<td>0:16:42</td>\n<td>0.4398</td>\n<td>0.4385</td>\n<td>final sub</td>\n</tr>\n<tr>\n<td>deberta-v3-small</td>\n<td>290</td>\n<td>0:15:07</td>\n<td>0.4430</td>\n<td>0.4435</td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-v3-small</td>\n<td>512</td>\n<td>0:21:33</td>\n<td>0.4388</td>\n<td>0.4383</td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-v3-xsmall</td>\n<td>768</td>\n<td>0:19:07</td>\n<td>0.4396</td>\n<td>0.4413</td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-v3-xsmall</td>\n<td>1428</td>\n<td>0:20:11</td>\n<td>0.4402</td>\n<td>0.4394</td>\n<td>2 process</td>\n</tr>\n</tbody>\n</table>\n<h1>Important Citations:</h1>\n<ul>\n<li><a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train</a></li>\n<li><a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference\" target=\"_blank\">https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference</a></li>\n<li><a href=\"https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x\" target=\"_blank\">https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x</a></li>\n<li><a href=\"https://www.kaggle.com/code/kojimar/fb3-single-pytorch-model-train\" target=\"_blank\">https://www.kaggle.com/code/kojimar/fb3-single-pytorch-model-train</a></li>\n</ul>\n<h1>Thanks and Acknowledgements:</h1>\n<p>Thanks to the organizers for holding this interesting competition. Thanks to the kagglers for sharing very helpful codes and discussions.</p>",
  "comments": [
    {
      "id": 2050043,
      "author_name": "Raja Biswas",
      "author_url": "",
      "post_date": "2022-11-30T13:50:32.350000",
      "content": "<p>Congratulations!! Many thanks for sharing the amazing solution so eloquently. Really like your knowledge distillation idea! 💯</p>",
      "votes": 0,
      "replies": [
        {
          "id": 2050136,
          "author_name": "ktm",
          "author_url": "",
          "post_date": "2022-11-30T14:46:51.967000",
          "content": "<p>Thank you! Congratulations too for your 1st place of efficiency LB!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2049967,
      "author_name": "kaerururu",
      "author_url": "",
      "post_date": "2022-11-30T13:09:06.897000",
      "content": "<p>Thank you for sharing.<br>\nThis is the first time I saw that KD is used effectively in the kaggle competition.</p>",
      "votes": 0,
      "replies": [
        {
          "id": 2050133,
          "author_name": "ktm",
          "author_url": "",
          "post_date": "2022-11-30T14:44:21.403000",
          "content": "<p>Thank you. KD was a todo idea from the joining of the competition for efficiency prize!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 4,
  "messages": [
    {
      "id": 2049917,
      "postDate": "2022-11-30T12:23:28.387Z",
      "content": "<h1>Overview</h1>\n<p>I took the average of 6 models with a 3 seed average in the final submission of LB.  <br>\nFor efficiency LB, I used deberta-v3-small with knowledge distillation and small max_len(=384), which took 0:16:42 min for submission and scored 0.438537 for public LB, 0.439835 for private LB.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5017202%2Fa680c252509a2029174d7800c59a0389%2Fsolution_summary.png?generation=1669809477192815&amp;alt=media\" alt=\"\"></p>\n<h1>Code</h1>\n<p>LB submission: <a href=\"https://www.kaggle.com/code/tmhrkt/fb3-ensemble\" target=\"_blank\">https://www.kaggle.com/code/tmhrkt/fb3-ensemble</a>  <br>\nEfficiency LB: <a href=\"https://www.kaggle.com/code/tmhrkt/fb3-efficiency?scriptVersionId=112206338\" target=\"_blank\">https://www.kaggle.com/code/tmhrkt/fb3-efficiency?scriptVersionId=112206338</a>  <br>\nSVR training: <a href=\"https://www.kaggle.com/code/tmhrkt/fb3-embedding-learning-training\" target=\"_blank\">https://www.kaggle.com/code/tmhrkt/fb3-embedding-learning-training</a>  <br>\ngithub: <a href=\"https://github.com/ktm98/kaggle-feedback-prize-3\" target=\"_blank\">https://github.com/ktm98/kaggle-feedback-prize-3</a>  </p>\n<h1>What Worked For Both LB and Efficiency LB</h1>\n<ul>\n<li>[high Impact] Changing max_len<ul>\n<li>CV improved 0.002, 0.001 (1024 -&gt; 512, 1024 -&gt; 768 each)</li>\n<li>Submission speeded up about 4 minutes (512 -&gt; 384)</li>\n<li>Reasoning and context: trying several max_len, <code>max_len&lt;384</code> got worse results compared to <code>maxlen=1024</code>, 1428. And oof correlations between different max_len (0.97+ for conventions) were lower than others (0.98+). So ensembling different max_len improved my CV score.</li></ul></li>\n<li>[High Impact] Knowledge Distillation<ul>\n<li>CV improved 0.014+ for deberta-v3-small</li>\n<li>What I did: set loss function as    <code>criterion(y_preds, labels) + criterion(y_preds, oof_preds)</code></li>\n<li>Reasoning and context: I tried KD for efficiency LB. However, the impact for both CV and LB was higher than expected, so I added KD models for LB.</li></ul></li>\n</ul>\n<h1>What Worked For Efficiency LB</h1>\n<ul>\n<li>[Low Impact] num_workers=1<ul>\n<li>Mean inference speed got faster about 5 seconds for the first 200 rows of <code>train.csv</code>.</li>\n<li>Reasoning and context: Up until midway through the competition, I was using <code>num_workers=0</code>. Changing <code>num_workers=1</code> slightly speeded up inference time.</li></ul></li>\n</ul>\n<h1>What Did’t Work</h1>\n<ul>\n<li>unscale<ul>\n<li>What I did: Adding unscale described in <a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train/comments\" target=\"_blank\">nakama’s notebook</a> comments.</li>\n<li>Result: CV score got worse.</li></ul></li>\n<li>XGBoost For embeddings<ul>\n<li>What I did: Train by XGBoost instead of Rapids SVR .</li>\n<li>Result: CV score got worse.</li></ul></li>\n<li>TorchScript<ul>\n<li>What I did: Converting PyTorch model to TorchScript.</li>\n<li>Result: Inference time was too slow. Stopped while inferencing.</li>\n<li>Reasoning and context: TorchScript does not allow dynamic sequence length. So If the tokenized length of a sentence is shorter than the max length, the attention mask portion must also be calculated, resulting in wasted computation time.</li></ul></li>\n<li>Dynamic Quantization<ul>\n<li>What I did: Apply dynamic quantization for my pretrained model.</li>\n<li>Result: LB score significantly dropped (0.43 -&gt; 0.64)</li>\n<li>Reasoning and Context: Because quantizing <code>nn.Linear</code> layers directly affects the output of the model, quantizing layers is not suitable for regression tasks.  </li></ul></li>\n</ul>\n<h1>Additional Context</h1>\n<h2>CV Strategy</h2>\n<p><code>MultiLabelStratifiedKFold</code> with binned <code>n_words</code></p>\n<h2>Efficiency submissions</h2>\n<p>First, I trained 4 folds to check the performance and submitted 4 folds.  <br>\nAfter checking the performance, the model was trained with whole FB3 data.  </p>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>max_len</th>\n<th>submission time</th>\n<th>Public</th>\n<th>Private</th>\n<th>notes</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>deberta-v3-small</td>\n<td>384</td>\n<td>0:16:42</td>\n<td>0.4398</td>\n<td>0.4385</td>\n<td>final sub</td>\n</tr>\n<tr>\n<td>deberta-v3-small</td>\n<td>290</td>\n<td>0:15:07</td>\n<td>0.4430</td>\n<td>0.4435</td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-v3-small</td>\n<td>512</td>\n<td>0:21:33</td>\n<td>0.4388</td>\n<td>0.4383</td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-v3-xsmall</td>\n<td>768</td>\n<td>0:19:07</td>\n<td>0.4396</td>\n<td>0.4413</td>\n<td></td>\n</tr>\n<tr>\n<td>deberta-v3-xsmall</td>\n<td>1428</td>\n<td>0:20:11</td>\n<td>0.4402</td>\n<td>0.4394</td>\n<td>2 process</td>\n</tr>\n</tbody>\n</table>\n<h1>Important Citations:</h1>\n<ul>\n<li><a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train</a></li>\n<li><a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference\" target=\"_blank\">https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference</a></li>\n<li><a href=\"https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x\" target=\"_blank\">https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x</a></li>\n<li><a href=\"https://www.kaggle.com/code/kojimar/fb3-single-pytorch-model-train\" target=\"_blank\">https://www.kaggle.com/code/kojimar/fb3-single-pytorch-model-train</a></li>\n</ul>\n<h1>Thanks and Acknowledgements:</h1>\n<p>Thanks to the organizers for holding this interesting competition. Thanks to the kagglers for sharing very helpful codes and discussions.</p>",
      "rawMarkdown": "# Overview\n\nI took the average of 6 models with a 3 seed average in the final submission of LB.  \nFor efficiency LB, I used deberta-v3-small with knowledge distillation and small max_len(=384), which took 0:16:42 min for submission and scored 0.438537 for public LB, 0.439835 for private LB.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5017202%2Fa680c252509a2029174d7800c59a0389%2Fsolution_summary.png?generation=1669809477192815&alt=media)\n\n\n# Code\nLB submission: https://www.kaggle.com/code/tmhrkt/fb3-ensemble  \nEfficiency LB: https://www.kaggle.com/code/tmhrkt/fb3-efficiency?scriptVersionId=112206338  \nSVR training: https://www.kaggle.com/code/tmhrkt/fb3-embedding-learning-training  \ngithub: https://github.com/ktm98/kaggle-feedback-prize-3  \n\n\n# What Worked For Both LB and Efficiency LB\n* [high Impact] Changing max_len\n - CV improved 0.002, 0.001 (1024 -> 512, 1024 -> 768 each)\n - Submission speeded up about 4 minutes (512 -> 384)\n - Reasoning and context: trying several max_len, `max_len<384` got worse results compared to `maxlen=1024`, 1428. And oof correlations between different max_len (0.97+ for conventions) were lower than others (0.98+). So ensembling different max_len improved my CV score.\n* [High Impact] Knowledge Distillation\n - CV improved 0.014+ for deberta-v3-small\n - What I did: set loss function as    `criterion(y_preds, labels) + criterion(y_preds, oof_preds)`\n - Reasoning and context: I tried KD for efficiency LB. However, the impact for both CV and LB was higher than expected, so I added KD models for LB.\n# What Worked For Efficiency LB\n* [Low Impact] num_workers=1\n - Mean inference speed got faster about 5 seconds for the first 200 rows of `train.csv`.\n - Reasoning and context: Up until midway through the competition, I was using `num_workers=0`. Changing `num_workers=1` slightly speeded up inference time.\n\n\n# What Did’t Work\n* unscale\n - What I did: Adding unscale described in [nakama’s notebook](https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train/comments) comments.\n - Result: CV score got worse.\n* XGBoost For embeddings\n - What I did: Train by XGBoost instead of Rapids SVR .\n - Result: CV score got worse.\n* TorchScript\n - What I did: Converting PyTorch model to TorchScript.\n - Result: Inference time was too slow. Stopped while inferencing.\n - Reasoning and context: TorchScript does not allow dynamic sequence length. So If the tokenized length of a sentence is shorter than the max length, the attention mask portion must also be calculated, resulting in wasted computation time.\n* Dynamic Quantization\n - What I did: Apply dynamic quantization for my pretrained model.\n - Result: LB score significantly dropped (0.43 -> 0.64)\n - Reasoning and Context: Because quantizing `nn.Linear` layers directly affects the output of the model, quantizing layers is not suitable for regression tasks.  \n\n\n\n# Additional Context\n## CV Strategy\n`MultiLabelStratifiedKFold` with binned `n_words`\n\n## Efficiency submissions\nFirst, I trained 4 folds to check the performance and submitted 4 folds.  \nAfter checking the performance, the model was trained with whole FB3 data.  \n|model|max_len|submission time|Public|Private|notes|\n|:--|:--|:--|:--|:--|:--|\n|deberta-v3-small|384|0:16:42|0.4398|0.4385|final sub|\n|deberta-v3-small|290|0:15:07|0.4430|0.4435||\n|deberta-v3-small|512|0:21:33|0.4388|0.4383||\n|deberta-v3-xsmall|768|0:19:07|0.4396|0.4413||\n|deberta-v3-xsmall|1428|0:20:11|0.4402|0.4394|2 process|\n  \n\n# Important Citations:\n* https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\n* https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference\n* https://www.kaggle.com/code/cdeotte/rapids-svr-cv-0-450-lb-0-44x\n* https://www.kaggle.com/code/kojimar/fb3-single-pytorch-model-train\n\n\n# Thanks and Acknowledgements:\nThanks to the organizers for holding this interesting competition. Thanks to the kagglers for sharing very helpful codes and discussions.\n",
      "votes": 13
    },
    {
      "id": 2050043,
      "postDate": "2022-11-30T13:50:32.350Z",
      "content": "<p>Congratulations!! Many thanks for sharing the amazing solution so eloquently. Really like your knowledge distillation idea! 💯</p>",
      "rawMarkdown": "Congratulations!! Many thanks for sharing the amazing solution so eloquently. Really like your knowledge distillation idea! 💯",
      "replies": [
        {
          "id": 2050136,
          "postDate": "2022-11-30T14:46:51.967Z",
          "content": "<p>Thank you! Congratulations too for your 1st place of efficiency LB!</p>",
          "rawMarkdown": "Thank you! Congratulations too for your 1st place of efficiency LB!",
          "votes": 1
        }
      ]
    },
    {
      "id": 2049967,
      "postDate": "2022-11-30T13:09:06.897Z",
      "content": "<p>Thank you for sharing.<br>\nThis is the first time I saw that KD is used effectively in the kaggle competition.</p>",
      "rawMarkdown": "Thank you for sharing.\nThis is the first time I saw that KD is used effectively in the kaggle competition.",
      "replies": [
        {
          "id": 2050133,
          "postDate": "2022-11-30T14:44:21.403Z",
          "content": "<p>Thank you. KD was a todo idea from the joining of the competition for efficiency prize!</p>",
          "rawMarkdown": "Thank you. KD was a todo idea from the joining of the competition for efficiency prize!"
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T20:00:27Z",
  "competition_id": 38321,
  "competition_slug": "feedback-prize-english-language-learning",
  "team_id": 9205664,
  "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/ktm-efficiency-7th-private-84st-place-solution"
}