{
  "id": 369564,
  "title": "14th place solution",
  "content": "<p>First of all, thanks to the host for an interesting competition and congratulations to all the winners!</p>\n<h1>Overview</h1>\n<p>My final submission is ensemble of 21 models. The weights of the models are determined using nelder-mead. </p>\n<h1>Model</h1>\n<p>Using <a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">this notebook</a> as a reference, I experimented a lot, changing model and training method.</p>\n<p>The following three things were important</p>\n<ul>\n<li>Add lstm after bert. And freeze the bert part for the first few epochs of training (model 15)</li>\n<li>Ensemble of low-cv models with negative weights(model 4, 12)</li>\n<li>Train model for each target or two targets.(model 20, 21)</li>\n</ul>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>cv</th>\n<th>weight</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1.  bart-large</td>\n<td>0.456</td>\n<td>-0.009</td>\n</tr>\n<tr>\n<td>2.  deberta-large</td>\n<td>0.456</td>\n<td>-0.089</td>\n</tr>\n<tr>\n<td>3.  deberta-xlarge</td>\n<td>0.454</td>\n<td>0.2175</td>\n</tr>\n<tr>\n<td>4.  deberta-base</td>\n<td>0.471</td>\n<td>-0.148</td>\n</tr>\n<tr>\n<td>5.  luke-large</td>\n<td>0.456</td>\n<td>0.072</td>\n</tr>\n<tr>\n<td>6.  gpt2-large</td>\n<td>0.460</td>\n<td>0.109</td>\n</tr>\n<tr>\n<td>7.  deberta-v2-xlarge</td>\n<td>0.456</td>\n<td>0.073</td>\n</tr>\n<tr>\n<td>8.  deberta-v3-large (add lstm)</td>\n<td>0.452</td>\n<td>0.025</td>\n</tr>\n<tr>\n<td>9.  gpt2</td>\n<td>0.476</td>\n<td>0.047</td>\n</tr>\n<tr>\n<td>10.  deberta-v3-large(Re-initializing layer)</td>\n<td>0.452</td>\n<td>0.073</td>\n</tr>\n<tr>\n<td>11.  deberta-v3-base (add lstm)</td>\n<td>0.453</td>\n<td>0.014</td>\n</tr>\n<tr>\n<td>12.  t5-large</td>\n<td>0.462</td>\n<td>-0.180</td>\n</tr>\n<tr>\n<td>13.  gpt-neo-125M</td>\n<td>0.479</td>\n<td>0.052</td>\n</tr>\n<tr>\n<td>14.  deberta-v3-large(pseudo label)</td>\n<td>0.450</td>\n<td>-0.013</td>\n</tr>\n<tr>\n<td>15.  deberta-v3-large(add lstm + freeze bert(2epoch))</td>\n<td>0.451</td>\n<td>0.162</td>\n</tr>\n<tr>\n<td>16.  deberta-v3-base(pseduo label + Re-initializing layer)</td>\n<td>0.454</td>\n<td>-0.055</td>\n</tr>\n<tr>\n<td>17.  deberta-v3-large(add /n token)</td>\n<td>0.454</td>\n<td>0.105</td>\n</tr>\n<tr>\n<td>18.  deberta-v3-base(add /n token)</td>\n<td>0.458</td>\n<td>0.032</td>\n</tr>\n<tr>\n<td>19.  deberta-v3-base</td>\n<td>0.456</td>\n<td>0.109</td>\n</tr>\n<tr>\n<td>20.  deberta-v3-large (train model for each target)</td>\n<td>0.452</td>\n<td>0.243</td>\n</tr>\n<tr>\n<td>21.  roberta-large (train model for two targets)</td>\n<td>0.458</td>\n<td>0.159</td>\n</tr>\n<tr>\n<td>ensemble</td>\n<td>0.4435</td>\n<td></td>\n</tr>\n</tbody>\n</table>",
  "comments": [
    {
      "id": 2050075,
      "author_name": "gezi",
      "author_url": "",
      "post_date": "2022-11-30T14:06:37.603000",
      "content": "<p>Very cool to use lstm learn first then unfreeze bert part. I tried lstm but not work here your method worth a try！<br>\nEnsemble of low-cv models with negative weights is also interesting!  Thanks for sharing!</p>",
      "votes": 3,
      "replies": [
        {
          "id": 2050083,
          "author_name": "Takoi",
          "author_url": "",
          "post_date": "2022-11-30T14:12:35.590000",
          "content": "<p>Thank you for your comment. Your U.S. Patent Phrase to Phrase Matching solution was quite helpful.</p>",
          "votes": 3,
          "replies": []
        }
      ]
    },
    {
      "id": 2050160,
      "author_name": "chumajin",
      "author_url": "",
      "post_date": "2022-11-30T15:07:09.383000",
      "content": "<p><a href=\"https://www.kaggle.com/takoihiraokazu\" target=\"_blank\">@takoihiraokazu</a> </p>\n<p>Congratulations! And thank you for sharing.<br>\nI have one question.<br>\nDo you use the same fold cuts and seeds in each model ?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2050814,
          "author_name": "Takoi",
          "author_url": "",
          "post_date": "2022-12-01T02:30:38.550000",
          "content": "<p>Thank you !<br>\nYes, all models are the same fold cuts and the same seeds.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 2051028,
          "author_name": "chumajin",
          "author_url": "",
          "post_date": "2022-12-01T06:51:53.237000",
          "content": "<p>I understand! Thank you very much.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 5,
  "messages": [
    {
      "id": 2050062,
      "postDate": "2022-11-30T13:59:30.450Z",
      "content": "<p>First of all, thanks to the host for an interesting competition and congratulations to all the winners!</p>\n<h1>Overview</h1>\n<p>My final submission is ensemble of 21 models. The weights of the models are determined using nelder-mead. </p>\n<h1>Model</h1>\n<p>Using <a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">this notebook</a> as a reference, I experimented a lot, changing model and training method.</p>\n<p>The following three things were important</p>\n<ul>\n<li>Add lstm after bert. And freeze the bert part for the first few epochs of training (model 15)</li>\n<li>Ensemble of low-cv models with negative weights(model 4, 12)</li>\n<li>Train model for each target or two targets.(model 20, 21)</li>\n</ul>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>cv</th>\n<th>weight</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>1.  bart-large</td>\n<td>0.456</td>\n<td>-0.009</td>\n</tr>\n<tr>\n<td>2.  deberta-large</td>\n<td>0.456</td>\n<td>-0.089</td>\n</tr>\n<tr>\n<td>3.  deberta-xlarge</td>\n<td>0.454</td>\n<td>0.2175</td>\n</tr>\n<tr>\n<td>4.  deberta-base</td>\n<td>0.471</td>\n<td>-0.148</td>\n</tr>\n<tr>\n<td>5.  luke-large</td>\n<td>0.456</td>\n<td>0.072</td>\n</tr>\n<tr>\n<td>6.  gpt2-large</td>\n<td>0.460</td>\n<td>0.109</td>\n</tr>\n<tr>\n<td>7.  deberta-v2-xlarge</td>\n<td>0.456</td>\n<td>0.073</td>\n</tr>\n<tr>\n<td>8.  deberta-v3-large (add lstm)</td>\n<td>0.452</td>\n<td>0.025</td>\n</tr>\n<tr>\n<td>9.  gpt2</td>\n<td>0.476</td>\n<td>0.047</td>\n</tr>\n<tr>\n<td>10.  deberta-v3-large(Re-initializing layer)</td>\n<td>0.452</td>\n<td>0.073</td>\n</tr>\n<tr>\n<td>11.  deberta-v3-base (add lstm)</td>\n<td>0.453</td>\n<td>0.014</td>\n</tr>\n<tr>\n<td>12.  t5-large</td>\n<td>0.462</td>\n<td>-0.180</td>\n</tr>\n<tr>\n<td>13.  gpt-neo-125M</td>\n<td>0.479</td>\n<td>0.052</td>\n</tr>\n<tr>\n<td>14.  deberta-v3-large(pseudo label)</td>\n<td>0.450</td>\n<td>-0.013</td>\n</tr>\n<tr>\n<td>15.  deberta-v3-large(add lstm + freeze bert(2epoch))</td>\n<td>0.451</td>\n<td>0.162</td>\n</tr>\n<tr>\n<td>16.  deberta-v3-base(pseduo label + Re-initializing layer)</td>\n<td>0.454</td>\n<td>-0.055</td>\n</tr>\n<tr>\n<td>17.  deberta-v3-large(add /n token)</td>\n<td>0.454</td>\n<td>0.105</td>\n</tr>\n<tr>\n<td>18.  deberta-v3-base(add /n token)</td>\n<td>0.458</td>\n<td>0.032</td>\n</tr>\n<tr>\n<td>19.  deberta-v3-base</td>\n<td>0.456</td>\n<td>0.109</td>\n</tr>\n<tr>\n<td>20.  deberta-v3-large (train model for each target)</td>\n<td>0.452</td>\n<td>0.243</td>\n</tr>\n<tr>\n<td>21.  roberta-large (train model for two targets)</td>\n<td>0.458</td>\n<td>0.159</td>\n</tr>\n<tr>\n<td>ensemble</td>\n<td>0.4435</td>\n<td></td>\n</tr>\n</tbody>\n</table>",
      "rawMarkdown": "First of all, thanks to the host for an interesting competition and congratulations to all the winners!\n\n# Overview\nMy final submission is ensemble of 21 models. The weights of the models are determined using nelder-mead. \n\n# Model\nUsing [this notebook](https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train) as a reference, I experimented a lot, changing model and training method.\n\nThe following three things were important\n- Add lstm after bert. And freeze the bert part for the first few epochs of training (model 15)\n- Ensemble of low-cv models with negative weights(model 4, 12)\n- Train model for each target or two targets.(model 20, 21)\n\n| model | cv | weight|\n| --- | --- |---|\n|1.  bart-large |0.456|-0.009|\n|2.  deberta-large |0.456 |-0.089|\n|3.  deberta-xlarge|0.454|0.2175|\n|4.  deberta-base |0.471|-0.148|\n|5.  luke-large|0.456|0.072|\n|6.  gpt2-large |0.460|0.109|\n|7.  deberta-v2-xlarge | 0.456 |0.073|\n|8.  deberta-v3-large (add lstm)| 0.452 |0.025|\n|9.  gpt2|0.476|0.047|\n|10.  deberta-v3-large(Re-initializing layer)|0.452|0.073|\n|11.  deberta-v3-base (add lstm) | 0.453|0.014|\n|12.  t5-large |0.462|-0.180|\n|13.  gpt-neo-125M |0.479|0.052|\n|14.  deberta-v3-large(pseudo label) |0.450|-0.013|\n|15.  deberta-v3-large(add lstm + freeze bert(2epoch)) | 0.451|0.162|\n|16.  deberta-v3-base(pseduo label + Re-initializing layer) | 0.454|-0.055|\n|17.  deberta-v3-large(add /n token)|0.454|0.105|\n|18.  deberta-v3-base(add /n token) |0.458 |0.032|\n|19.  deberta-v3-base |0.456|0.109|\n|20.  deberta-v3-large (train model for each target) |0.452|0.243|\n|21.  roberta-large (train model for two targets) |0.458|0.159|\n|ensemble |0.4435||",
      "votes": 29
    },
    {
      "id": 2050075,
      "postDate": "2022-11-30T14:06:37.603Z",
      "content": "<p>Very cool to use lstm learn first then unfreeze bert part. I tried lstm but not work here your method worth a try！<br>\nEnsemble of low-cv models with negative weights is also interesting!  Thanks for sharing!</p>",
      "rawMarkdown": "Very cool to use lstm learn first then unfreeze bert part. I tried lstm but not work here your method worth a try！\nEnsemble of low-cv models with negative weights is also interesting!  Thanks for sharing!",
      "votes": 3,
      "replies": [
        {
          "id": 2050083,
          "postDate": "2022-11-30T14:12:35.590Z",
          "content": "<p>Thank you for your comment. Your U.S. Patent Phrase to Phrase Matching solution was quite helpful.</p>",
          "rawMarkdown": "Thank you for your comment. Your U.S. Patent Phrase to Phrase Matching solution was quite helpful.",
          "votes": 3
        }
      ]
    },
    {
      "id": 2050160,
      "postDate": "2022-11-30T15:07:09.383Z",
      "content": "<p><a href=\"https://www.kaggle.com/takoihiraokazu\" target=\"_blank\">@takoihiraokazu</a> </p>\n<p>Congratulations! And thank you for sharing.<br>\nI have one question.<br>\nDo you use the same fold cuts and seeds in each model ?</p>",
      "rawMarkdown": "@takoihiraokazu \n\nCongratulations! And thank you for sharing.\nI have one question.\nDo you use the same fold cuts and seeds in each model ?",
      "votes": 1,
      "replies": [
        {
          "id": 2050814,
          "postDate": "2022-12-01T02:30:38.550Z",
          "content": "<p>Thank you !<br>\nYes, all models are the same fold cuts and the same seeds.</p>",
          "rawMarkdown": "Thank you !\nYes, all models are the same fold cuts and the same seeds.",
          "votes": 1
        },
        {
          "id": 2051028,
          "postDate": "2022-12-01T06:51:53.237Z",
          "content": "<p>I understand! Thank you very much.</p>",
          "rawMarkdown": "I understand! Thank you very much."
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T20:00:46Z",
  "competition_id": 38321,
  "competition_slug": "feedback-prize-english-language-learning",
  "team_id": 9092484,
  "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/rist-takoi-14th-place-solution"
}