{
  "id": 370686,
  "title": "28th Place Efficiency Solution",
  "content": "<p>First of all, thanks to Kaggle and the competition hosts for organizing this competition. </p>\n<p>This was my first competition on Kaggle. I joined this competition a month ago and couldn't try lots of things due to time and hardware limitations. I focused only to the efficiency track part and selected the submissions for evaluation accordingly. In any case, I enjoyed to much to be a part of this competition and learned a lot from discussions and notebooks. Congratulations to all the winners.     <br>\n<br></p>\n<h2>Overview</h2>\n<p>My solution is an ensemble of two models (simple averaging of electra-small-discriminator and LGBM regressor models) with knowledge distillation, pseudo labeling (FB1 dataset) and extraction of new features (ngrams and other statistical features) for LGBM regressor. The most significant improvement was achieved by adding predictions from electra-small models as additional features to LGBM regressor models. Since no time left to train a single model, I used only the 'syntax' and 'phraseology' models of electra-small for faster inference.</p>\n<p>I would be very grateful if you try this solution on your own models and share the results here. Notebooks for <a href=\"https://www.kaggle.com/code/bediiaydogan/fb3-efficiency-28th-place-training\" target=\"_blank\">training</a> and <a href=\"https://www.kaggle.com/code/bediiaydogan/fb3-efficiency-28th-place-inference\" target=\"_blank\">inference</a> are publicly available.<br>\n<br></p>\n<h2>Models and Performances</h2>\n<table>\n<thead>\n<tr>\n<th>Version</th>\n<th>Model</th>\n<th>Training Data</th>\n<th>Features</th>\n<th>Private Scores</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>v1</td>\n<td>Electra-base-discriminator</td>\n<td>FB3</td>\n<td>text</td>\n<td>0.449456</td>\n</tr>\n<tr>\n<td>v2</td>\n<td>Electra-small-discriminator</td>\n<td>FB3</td>\n<td>text</td>\n<td>0.461457</td>\n</tr>\n<tr>\n<td>v3</td>\n<td>Electra-small-discriminator<br>(distilled from Model-v1)</td>\n<td>FB3</td>\n<td>text</td>\n<td>0.456003</td>\n</tr>\n<tr>\n<td>v4</td>\n<td>LGBM Regressor</td>\n<td>FB3</td>\n<td>text-feats</td>\n<td>0.494600</td>\n</tr>\n<tr>\n<td>v5</td>\n<td>LGBM Regressor</td>\n<td>FB3 + FB1</td>\n<td>text-feats</td>\n<td>0.491990</td>\n</tr>\n<tr>\n<td>v6</td>\n<td>LGBM Regressor + Model-v1</td>\n<td>FB3 + FB1</td>\n<td>text-feats +<br>model-v1 predictions</td>\n<td><strong>0.444570</strong></td>\n</tr>\n<tr>\n<td>v7</td>\n<td>LGBM Regressor + Model-v3</td>\n<td>FB3 + FB1</td>\n<td>text-feats +<br>model-v3 predictions</td>\n<td>0.448876</td>\n</tr>\n<tr>\n<td>v8&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;</td>\n<td>LGBM Regressor + Model-v3<br>(syntax and phraseology models)</td>\n<td>FB3 + FB1</td>\n<td>text-feats +<br>model-v3 predictions</td>\n<td><strong>0.452429</strong></td>\n</tr>\n</tbody>\n</table>\n<p><code>* text-feats: ngrams and statistical features extracted from full-text</code><br>\n<br></p>\n<h2>What Worked</h2>\n<ul>\n<li><p><strong>[High Impact] Predictions as Features and Ensemble</strong><br>\nAs mentioned above, adding predictions as additional features to LGBMR models and ensembling resulted in a 1% - 1.5% improvement in scores.  </p></li>\n<li><p><strong>[High Impact] Knowledge Distillation</strong></p></li>\n<li><p><strong>[Medium Impact] Pseudo Labeling</strong><br>\n<br></p></li>\n</ul>\n<h2>What Didn’t Work</h2>\n<p>I would like to write a long list of \"what didn't work\", but I haven't had the opportunity to try many things in a one month period.<br>\n<br></p>\n<h2>Team Members</h2>\n<p>Bedii A. Aydoğan <a href=\"https://www.kaggle.com/bediiaydogan\" target=\"_blank\">@bediiaydogan</a>&nbsp;&nbsp;</p>",
  "comments": [],
  "comment_count": 0,
  "messages": [
    {
      "id": 2056236,
      "postDate": "2022-12-05T22:06:00.857Z",
      "content": "<p>First of all, thanks to Kaggle and the competition hosts for organizing this competition. </p>\n<p>This was my first competition on Kaggle. I joined this competition a month ago and couldn't try lots of things due to time and hardware limitations. I focused only to the efficiency track part and selected the submissions for evaluation accordingly. In any case, I enjoyed to much to be a part of this competition and learned a lot from discussions and notebooks. Congratulations to all the winners.     <br>\n<br></p>\n<h2>Overview</h2>\n<p>My solution is an ensemble of two models (simple averaging of electra-small-discriminator and LGBM regressor models) with knowledge distillation, pseudo labeling (FB1 dataset) and extraction of new features (ngrams and other statistical features) for LGBM regressor. The most significant improvement was achieved by adding predictions from electra-small models as additional features to LGBM regressor models. Since no time left to train a single model, I used only the 'syntax' and 'phraseology' models of electra-small for faster inference.</p>\n<p>I would be very grateful if you try this solution on your own models and share the results here. Notebooks for <a href=\"https://www.kaggle.com/code/bediiaydogan/fb3-efficiency-28th-place-training\" target=\"_blank\">training</a> and <a href=\"https://www.kaggle.com/code/bediiaydogan/fb3-efficiency-28th-place-inference\" target=\"_blank\">inference</a> are publicly available.<br>\n<br></p>\n<h2>Models and Performances</h2>\n<table>\n<thead>\n<tr>\n<th>Version</th>\n<th>Model</th>\n<th>Training Data</th>\n<th>Features</th>\n<th>Private Scores</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>v1</td>\n<td>Electra-base-discriminator</td>\n<td>FB3</td>\n<td>text</td>\n<td>0.449456</td>\n</tr>\n<tr>\n<td>v2</td>\n<td>Electra-small-discriminator</td>\n<td>FB3</td>\n<td>text</td>\n<td>0.461457</td>\n</tr>\n<tr>\n<td>v3</td>\n<td>Electra-small-discriminator<br>(distilled from Model-v1)</td>\n<td>FB3</td>\n<td>text</td>\n<td>0.456003</td>\n</tr>\n<tr>\n<td>v4</td>\n<td>LGBM Regressor</td>\n<td>FB3</td>\n<td>text-feats</td>\n<td>0.494600</td>\n</tr>\n<tr>\n<td>v5</td>\n<td>LGBM Regressor</td>\n<td>FB3 + FB1</td>\n<td>text-feats</td>\n<td>0.491990</td>\n</tr>\n<tr>\n<td>v6</td>\n<td>LGBM Regressor + Model-v1</td>\n<td>FB3 + FB1</td>\n<td>text-feats +<br>model-v1 predictions</td>\n<td><strong>0.444570</strong></td>\n</tr>\n<tr>\n<td>v7</td>\n<td>LGBM Regressor + Model-v3</td>\n<td>FB3 + FB1</td>\n<td>text-feats +<br>model-v3 predictions</td>\n<td>0.448876</td>\n</tr>\n<tr>\n<td>v8&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;</td>\n<td>LGBM Regressor + Model-v3<br>(syntax and phraseology models)</td>\n<td>FB3 + FB1</td>\n<td>text-feats +<br>model-v3 predictions</td>\n<td><strong>0.452429</strong></td>\n</tr>\n</tbody>\n</table>\n<p><code>* text-feats: ngrams and statistical features extracted from full-text</code><br>\n<br></p>\n<h2>What Worked</h2>\n<ul>\n<li><p><strong>[High Impact] Predictions as Features and Ensemble</strong><br>\nAs mentioned above, adding predictions as additional features to LGBMR models and ensembling resulted in a 1% - 1.5% improvement in scores.  </p></li>\n<li><p><strong>[High Impact] Knowledge Distillation</strong></p></li>\n<li><p><strong>[Medium Impact] Pseudo Labeling</strong><br>\n<br></p></li>\n</ul>\n<h2>What Didn’t Work</h2>\n<p>I would like to write a long list of \"what didn't work\", but I haven't had the opportunity to try many things in a one month period.<br>\n<br></p>\n<h2>Team Members</h2>\n<p>Bedii A. Aydoğan <a href=\"https://www.kaggle.com/bediiaydogan\" target=\"_blank\">@bediiaydogan</a>&nbsp;&nbsp;</p>",
      "rawMarkdown": "First of all, thanks to Kaggle and the competition hosts for organizing this competition. \n\nThis was my first competition on Kaggle. I joined this competition a month ago and couldn't try lots of things due to time and hardware limitations. I focused only to the efficiency track part and selected the submissions for evaluation accordingly. In any case, I enjoyed to much to be a part of this competition and learned a lot from discussions and notebooks. Congratulations to all the winners.     \n<br>\n## Overview\n\nMy solution is an ensemble of two models (simple averaging of electra-small-discriminator and LGBM regressor models) with knowledge distillation, pseudo labeling (FB1 dataset) and extraction of new features (ngrams and other statistical features) for LGBM regressor. The most significant improvement was achieved by adding predictions from electra-small models as additional features to LGBM regressor models. Since no time left to train a single model, I used only the 'syntax' and 'phraseology' models of electra-small for faster inference.\n\nI would be very grateful if you try this solution on your own models and share the results here. Notebooks for [training](https://www.kaggle.com/code/bediiaydogan/fb3-efficiency-28th-place-training) and [inference](https://www.kaggle.com/code/bediiaydogan/fb3-efficiency-28th-place-inference) are publicly available.\n<br>\n## Models and Performances\n\n|  Version | Model                                                                                | Training Data | Features                 | Private Scores |\n| :--- | :------------------------------------------------------ | :------------- | :------------------ | :-------------- |\n| v1    | Electra-base-discriminator                               |FB3      |text                              |0.449456|\n| v2    | Electra-small-discriminator                              |FB3      |text                              |0.461457|\n| v3    | Electra-small-discriminator<br>(distilled from Model-v1)    |FB3      |text                              |0.456003    |\n| v4    | LGBM Regressor                                           |FB3      |text-feats                       |0.494600 |\n| v5    | LGBM Regressor                                           |FB3 + FB1|text-feats                       |0.491990|\n| v6   | LGBM Regressor + Model-v1                                |FB3 + FB1|text-feats +<br>model-v1 predictions|**0.444570**|\n| v7    | LGBM Regressor + Model-v3                                |FB3 + FB1|text-feats +<br>model-v3 predictions|0.448876|\n| v8&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp;   | LGBM Regressor + Model-v3<br>(syntax and phraseology models)|FB3 + FB1|text-feats +<br>model-v3 predictions|**0.452429**|\n\n`* text-feats: ngrams and statistical features extracted from full-text`\n<br>\n## What Worked\n\n* **[High Impact] Predictions as Features and Ensemble**\n  As mentioned above, adding predictions as additional features to LGBMR models and ensembling resulted in a 1% - 1.5% improvement in scores.  \n\n* **[High Impact] Knowledge Distillation**\n\n* **[Medium Impact] Pseudo Labeling**\n<br>\n## What Didn’t Work\n\nI would like to write a long list of \"what didn't work\", but I haven't had the opportunity to try many things in a one month period.\n<br>\n## Team Members\n\nBedii A. Aydoğan [@bediiaydogan](https://www.kaggle.com/bediiaydogan)&nbsp;&nbsp;",
      "votes": 5
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T20:05:48Z",
  "competition_id": 38321,
  "competition_slug": "feedback-prize-english-language-learning",
  "team_id": 9335300,
  "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/creeper-28th-place-efficiency-solution"
}