{
  "id": 369736,
  "title": "7th Place Solution",
  "content": "<p>Congrats to all the winners, and thanks to hosts for interesting competition.<br>\nI still can't believe I shaked up to gold zone.</p>\n<h1>overview</h1>\n<p>My final submission is ensemble of 16 models. The weights of the models are determined using nelder-mead.</p>\n<h1>model</h1>\n<p>The default parameters are as follows:</p>\n<ul>\n<li>max_length = 512</li>\n<li>awp <ul>\n<li>start_epoch = 3</li>\n<li>adv_lr = 0.0001</li>\n<li>adv_eps = 0.01</li></ul></li>\n<li>bce</li>\n<li>lr = 2e-5</li>\n<li>lwld<ul>\n<li>base model: lr_decay = 0.9 </li>\n<li>large model: lr_decay = 0.95 </li></ul></li>\n<li>freeze layers <ul>\n<li>large model: 1/2</li>\n<li>xlarge model: 3/4</li></ul></li>\n<li>fp16</li>\n<li>5-fold</li>\n</ul>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>changed</th>\n<th>cv</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>deberta-v3-base</td>\n<td>max_len=1024</td>\n<td>0.4517</td>\n</tr>\n<tr>\n<td>deberta-v3-base</td>\n<td>no awp</td>\n<td>0.4572</td>\n</tr>\n<tr>\n<td>deberta-v3-large</td>\n<td></td>\n<td>0.4525</td>\n</tr>\n<tr>\n<td>deberta-v3-large</td>\n<td>no freeze</td>\n<td>0.4549</td>\n</tr>\n<tr>\n<td>deberta-v3-large</td>\n<td>max_len=1024</td>\n<td>0.4566</td>\n</tr>\n<tr>\n<td>deberta-v3-large</td>\n<td>l1_loss</td>\n<td>0.4576</td>\n</tr>\n<tr>\n<td>deberta-v2-xlarge</td>\n<td>lr=5e-6</td>\n<td>0.4602</td>\n</tr>\n<tr>\n<td>deberta-xlarge</td>\n<td></td>\n<td>0.4514</td>\n</tr>\n<tr>\n<td>deberta-large</td>\n<td></td>\n<td>0.4502</td>\n</tr>\n<tr>\n<td>deberta-large</td>\n<td>no awp</td>\n<td>0.4555</td>\n</tr>\n<tr>\n<td>deberta-large</td>\n<td>l1_loss</td>\n<td>0.4557</td>\n</tr>\n<tr>\n<td>deberta-base</td>\n<td></td>\n<td>0.4567</td>\n</tr>\n<tr>\n<td>roberta-large</td>\n<td></td>\n<td>0.459</td>\n</tr>\n<tr>\n<td>muppet-roberta-large</td>\n<td>no awp</td>\n<td>0.4635</td>\n</tr>\n<tr>\n<td>distilbart-mnli-12-9</td>\n<td></td>\n<td>0.4626</td>\n</tr>\n<tr>\n<td>bart-large-finetuned-squadv1</td>\n<td></td>\n<td>0.4635</td>\n</tr>\n</tbody>\n</table>\n<h1>What worked</h1>\n<ul>\n<li><p>[High Impact] awp</p>\n<ul>\n<li>cv+0.004 ~ 0.005</li>\n<li>awp didn't work for public lb, so I added no awp model for ensemble</li></ul></li>\n<li><p>[Middle Impact] lwld</p>\n<ul>\n<li>cv+0.002~0.003</li></ul></li>\n<li><p>[Middle Impact] nelder-mead method</p></li>\n<li><p>[Low Impact] bce</p>\n<ul>\n<li>converged faster than l1_loss</li></ul></li>\n<li><p>[Low Impact] freeze layers</p>\n<ul>\n<li>stable and faster training</li></ul></li>\n</ul>\n<h1>What Didn’t Work</h1>\n<ul>\n<li>fgm</li>\n<li>Last Layer Re-initialization</li>\n<li>lgb stacking</li>\n</ul>\n<h1>Important Citation</h1>\n<ul>\n<li><p>feedback3</p>\n<ul>\n<li><a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">FB3 / Deberta-v3-base baseline [train]</a></li></ul></li>\n<li><p>feedback1</p>\n<ul>\n<li><a href=\"https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313177\" target=\"_blank\">feedback1 1st solution</a></li></ul></li>\n<li><p>commonlit</p>\n<ul>\n<li><a href=\"https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/258328\" target=\"_blank\">commonlit 2nd place solution</a></li>\n<li><a href=\"https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/260729\" target=\"_blank\">The Magic of No Dropout</a></li></ul></li>\n</ul>\n<h1>inference code</h1>\n<p><a href=\"https://www.kaggle.com/code/tanakar/fb3-ensemble-per-col/notebook\" target=\"_blank\">https://www.kaggle.com/code/tanakar/fb3-ensemble-per-col/notebook</a></p>\n<h1>Thanks and Acknowledgements:</h1>\n<p>Thanks to our hosts for hosting the competition. And thanks to the Kaggler's for sharing their helpful notebooks and discussions.</p>\n<h1>Team Members:</h1>\n<ul>\n<li><a href=\"https://www.kaggle.com/tanakar\" target=\"_blank\">@tanakar</a></li>\n</ul>",
  "comments": [
    {
      "id": 2051305,
      "author_name": "gezi",
      "author_url": "",
      "post_date": "2022-12-01T10:31:17.893000",
      "content": "<p>deberta-v3-base to get cv 4517 is cool!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2057574,
      "author_name": "Julien Mouchnino",
      "author_url": "",
      "post_date": "2022-12-07T08:12:26.793000",
      "content": "<p>Can someone explain how  Nelder-Mead method work in the setting of ensemble methods ?</p>\n<p>Lots of competitors have used the Nelder-Mead method to weigh each model. </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2057570,
      "author_name": "Julien Mouchnino",
      "author_url": "",
      "post_date": "2022-12-07T08:06:06.613000",
      "content": "<p>What is AWP ? </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2051162,
      "author_name": "Santhoshkumar",
      "author_url": "",
      "post_date": "2022-12-01T08:59:29.123000",
      "content": "<p>That's great <a href=\"https://www.kaggle.com/tanakar\" target=\"_blank\">@tanakar</a> , thanks for sharing it👍👏</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 2051172,
      "author_name": "",
      "author_url": "",
      "post_date": "2022-12-01T09:05:22.127000",
      "content": "",
      "votes": 0,
      "replies": [
        {
          "id": 2051189,
          "author_name": "Joshua Qiu",
          "author_url": "",
          "post_date": "2022-12-01T09:17:15.443000",
          "content": "<p>should be layer-wised learning rate decay</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 2051211,
          "author_name": "",
          "author_url": "",
          "post_date": "2022-12-01T09:30:14.143000",
          "content": "",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 7,
  "messages": [
    {
      "id": 2051138,
      "postDate": "2022-12-01T08:36:06.600Z",
      "content": "<p>Congrats to all the winners, and thanks to hosts for interesting competition.<br>\nI still can't believe I shaked up to gold zone.</p>\n<h1>overview</h1>\n<p>My final submission is ensemble of 16 models. The weights of the models are determined using nelder-mead.</p>\n<h1>model</h1>\n<p>The default parameters are as follows:</p>\n<ul>\n<li>max_length = 512</li>\n<li>awp <ul>\n<li>start_epoch = 3</li>\n<li>adv_lr = 0.0001</li>\n<li>adv_eps = 0.01</li></ul></li>\n<li>bce</li>\n<li>lr = 2e-5</li>\n<li>lwld<ul>\n<li>base model: lr_decay = 0.9 </li>\n<li>large model: lr_decay = 0.95 </li></ul></li>\n<li>freeze layers <ul>\n<li>large model: 1/2</li>\n<li>xlarge model: 3/4</li></ul></li>\n<li>fp16</li>\n<li>5-fold</li>\n</ul>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>changed</th>\n<th>cv</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>deberta-v3-base</td>\n<td>max_len=1024</td>\n<td>0.4517</td>\n</tr>\n<tr>\n<td>deberta-v3-base</td>\n<td>no awp</td>\n<td>0.4572</td>\n</tr>\n<tr>\n<td>deberta-v3-large</td>\n<td></td>\n<td>0.4525</td>\n</tr>\n<tr>\n<td>deberta-v3-large</td>\n<td>no freeze</td>\n<td>0.4549</td>\n</tr>\n<tr>\n<td>deberta-v3-large</td>\n<td>max_len=1024</td>\n<td>0.4566</td>\n</tr>\n<tr>\n<td>deberta-v3-large</td>\n<td>l1_loss</td>\n<td>0.4576</td>\n</tr>\n<tr>\n<td>deberta-v2-xlarge</td>\n<td>lr=5e-6</td>\n<td>0.4602</td>\n</tr>\n<tr>\n<td>deberta-xlarge</td>\n<td></td>\n<td>0.4514</td>\n</tr>\n<tr>\n<td>deberta-large</td>\n<td></td>\n<td>0.4502</td>\n</tr>\n<tr>\n<td>deberta-large</td>\n<td>no awp</td>\n<td>0.4555</td>\n</tr>\n<tr>\n<td>deberta-large</td>\n<td>l1_loss</td>\n<td>0.4557</td>\n</tr>\n<tr>\n<td>deberta-base</td>\n<td></td>\n<td>0.4567</td>\n</tr>\n<tr>\n<td>roberta-large</td>\n<td></td>\n<td>0.459</td>\n</tr>\n<tr>\n<td>muppet-roberta-large</td>\n<td>no awp</td>\n<td>0.4635</td>\n</tr>\n<tr>\n<td>distilbart-mnli-12-9</td>\n<td></td>\n<td>0.4626</td>\n</tr>\n<tr>\n<td>bart-large-finetuned-squadv1</td>\n<td></td>\n<td>0.4635</td>\n</tr>\n</tbody>\n</table>\n<h1>What worked</h1>\n<ul>\n<li><p>[High Impact] awp</p>\n<ul>\n<li>cv+0.004 ~ 0.005</li>\n<li>awp didn't work for public lb, so I added no awp model for ensemble</li></ul></li>\n<li><p>[Middle Impact] lwld</p>\n<ul>\n<li>cv+0.002~0.003</li></ul></li>\n<li><p>[Middle Impact] nelder-mead method</p></li>\n<li><p>[Low Impact] bce</p>\n<ul>\n<li>converged faster than l1_loss</li></ul></li>\n<li><p>[Low Impact] freeze layers</p>\n<ul>\n<li>stable and faster training</li></ul></li>\n</ul>\n<h1>What Didn’t Work</h1>\n<ul>\n<li>fgm</li>\n<li>Last Layer Re-initialization</li>\n<li>lgb stacking</li>\n</ul>\n<h1>Important Citation</h1>\n<ul>\n<li><p>feedback3</p>\n<ul>\n<li><a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">FB3 / Deberta-v3-base baseline [train]</a></li></ul></li>\n<li><p>feedback1</p>\n<ul>\n<li><a href=\"https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313177\" target=\"_blank\">feedback1 1st solution</a></li></ul></li>\n<li><p>commonlit</p>\n<ul>\n<li><a href=\"https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/258328\" target=\"_blank\">commonlit 2nd place solution</a></li>\n<li><a href=\"https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/260729\" target=\"_blank\">The Magic of No Dropout</a></li></ul></li>\n</ul>\n<h1>inference code</h1>\n<p><a href=\"https://www.kaggle.com/code/tanakar/fb3-ensemble-per-col/notebook\" target=\"_blank\">https://www.kaggle.com/code/tanakar/fb3-ensemble-per-col/notebook</a></p>\n<h1>Thanks and Acknowledgements:</h1>\n<p>Thanks to our hosts for hosting the competition. And thanks to the Kaggler's for sharing their helpful notebooks and discussions.</p>\n<h1>Team Members:</h1>\n<ul>\n<li><a href=\"https://www.kaggle.com/tanakar\" target=\"_blank\">@tanakar</a></li>\n</ul>",
      "rawMarkdown": "Congrats to all the winners, and thanks to hosts for interesting competition.\nI still can't believe I shaked up to gold zone.\n\n# overview\nMy final submission is ensemble of 16 models. The weights of the models are determined using nelder-mead.\n\n\n# model\n\nThe default parameters are as follows:\n\n- max_length = 512\n- awp \n  - start_epoch = 3\n  - adv_lr = 0.0001\n  - adv_eps = 0.01\n- bce\n- lr = 2e-5\n- lwld\n  - base model: lr_decay = 0.9 \n  - large model: lr_decay = 0.95 \n- freeze layers \n  - large model: 1/2\n  - xlarge model: 3/4\n- fp16\n- 5-fold\n\n\n| model | changed | cv |\n| ---- | ---- | ---- |\n| deberta-v3-base | max_len=1024 | 0.4517\n| deberta-v3-base | no awp | 0.4572\n| deberta-v3-large |  | 0.4525\n| deberta-v3-large | no freeze | 0.4549\n| deberta-v3-large | max_len=1024 | 0.4566\n| deberta-v3-large | l1_loss | 0.4576\n| deberta-v2-xlarge | lr=5e-6| 0.4602\n| deberta-xlarge |  | 0.4514\n| deberta-large |  | 0.4502\n| deberta-large | no awp | 0.4555\n| deberta-large | l1_loss | 0.4557\n| deberta-base |  | 0.4567\n| roberta-large |  | 0.459\n| muppet-roberta-large | no awp | 0.4635\n| distilbart-mnli-12-9 |  | 0.4626\n| bart-large-finetuned-squadv1 |  | 0.4635\n\n# What worked\n- [High Impact] awp\n    - cv+0.004 ~ 0.005\n    - awp didn't work for public lb, so I added no awp model for ensemble\n\n- [Middle Impact] lwld\n   - cv+0.002~0.003\n\n- [Middle Impact] nelder-mead method\n\n- [Low Impact] bce\n    - converged faster than l1_loss\n\n- [Low Impact] freeze layers\n    - stable and faster training\n\n# What Didn’t Work\n- fgm\n- Last Layer Re-initialization\n- lgb stacking\n\n# Important Citation\n- feedback3\n  - [FB3 / Deberta-v3-base baseline [train]](https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train)\n\n- feedback1\n  - [feedback1 1st solution](https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313177)\n\n- commonlit\n  - [commonlit 2nd place solution](https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/258328)\n  - [The Magic of No Dropout](https://www.kaggle.com/competitions/commonlitreadabilityprize/discussion/260729)\n\n# inference code\nhttps://www.kaggle.com/code/tanakar/fb3-ensemble-per-col/notebook\n\n# Thanks and Acknowledgements:\nThanks to our hosts for hosting the competition. And thanks to the Kaggler's for sharing their helpful notebooks and discussions.\n\n# Team Members:\n- [@tanakar](https://www.kaggle.com/tanakar)",
      "votes": 21
    },
    {
      "id": 2051305,
      "postDate": "2022-12-01T10:31:17.893Z",
      "content": "<p>deberta-v3-base to get cv 4517 is cool!</p>",
      "rawMarkdown": "deberta-v3-base to get cv 4517 is cool!",
      "votes": 1
    },
    {
      "id": 2057574,
      "postDate": "2022-12-07T08:12:26.793Z",
      "content": "<p>Can someone explain how  Nelder-Mead method work in the setting of ensemble methods ?</p>\n<p>Lots of competitors have used the Nelder-Mead method to weigh each model. </p>",
      "rawMarkdown": "Can someone explain how  Nelder-Mead method work in the setting of ensemble methods ?\n\nLots of competitors have used the Nelder-Mead method to weigh each model. \n"
    },
    {
      "id": 2057570,
      "postDate": "2022-12-07T08:06:06.613Z",
      "content": "<p>What is AWP ? </p>",
      "rawMarkdown": "What is AWP ? "
    },
    {
      "id": 2051162,
      "postDate": "2022-12-01T08:59:29.123Z",
      "content": "<p>That's great <a href=\"https://www.kaggle.com/tanakar\" target=\"_blank\">@tanakar</a> , thanks for sharing it👍👏</p>",
      "rawMarkdown": "That's great @tanakar , thanks for sharing it👍👏"
    },
    {
      "id": 2051172,
      "postDate": "2022-12-01T09:05:22.127Z",
      "rawMarkdown": "",
      "isDeleted": true,
      "replies": [
        {
          "id": 2051189,
          "postDate": "2022-12-01T09:17:15.443Z",
          "content": "<p>should be layer-wised learning rate decay</p>",
          "rawMarkdown": "should be layer-wised learning rate decay"
        },
        {
          "id": 2051211,
          "postDate": "2022-12-01T09:30:14.143Z",
          "rawMarkdown": "",
          "isDeleted": true
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T20:03:32Z",
  "competition_id": 38321,
  "competition_slug": "feedback-prize-english-language-learning",
  "team_id": 9090986,
  "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/tk-7th-place-solution"
}