{
  "id": 370974,
  "title": "18th place solution",
  "content": "<h1>18 th place Solution</h1>\n<p>First of all, I would like to thank kaggle and the hosting staff for hosting such an interesting competition. And I would like to say congrats to all winner. I could not get the solo-gold, but I learned a lot from this competition.</p>\n<h1>1. Overview (Almost the same as others)</h1>\n<p>My solution is weight average ensemble with 21 models Nealder-mead with pseudo labeling. This is almost the same as other solutions. My originality is the replacement \\n to [Chapter]. This improved the cv 0.0015. And in this discussion, I show the my CV and Private LB relationship in detail.</p>\n<h1>1.1 Configuration</h1>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4001300%2Fa2ac96703257cc1d4353d8fc871d2028%2FClipboard01.jpg?generation=1670408759841540&amp;alt=media\" alt=\"\"></p>\n<h1>1.2 Relationship of cv and lb</h1>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4001300%2F5ea21d5c1e0e384bd0e25799427bd7ae%2FClipboard02.jpg?generation=1670408854083583&amp;alt=media\" alt=\"\"></p>\n<h1>1.3 cv and score in each model</h1>\n<p>※ pseudo1 and 2 are explained on next chapter<br>\n  ※ The order is the order with the best private lb score</p>\n<table>\n<thead>\n<tr>\n<th>EXP</th>\n<th>model</th>\n<th>pseudo1</th>\n<th>pseudo2</th>\n<th>pseudo<br> dataset</th>\n<th>kfold</th>\n<th>cv</th>\n<th>private lb</th>\n<th>public lb</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>214</td>\n<td>deberta v3 large</td>\n<td>〇</td>\n<td>-</td>\n<td>FB1 only</td>\n<td>4</td>\n<td>0.4467</td>\n<td>0.4378</td>\n<td>0.4404</td>\n</tr>\n<tr>\n<td>123</td>\n<td>deberta v3 large</td>\n<td>〇</td>\n<td>-</td>\n<td>FB1 <br>+ Commonlit</td>\n<td>4</td>\n<td>0.4467</td>\n<td>0.4379</td>\n<td>0.4402</td>\n</tr>\n<tr>\n<td>200</td>\n<td>deberta v3 large</td>\n<td>-</td>\n<td>〇</td>\n<td>FB1 only</td>\n<td>4</td>\n<td>0.4471</td>\n<td>0.4381</td>\n<td>0.4396</td>\n</tr>\n<tr>\n<td>250</td>\n<td>deberta v3 large</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>15</td>\n<td>0.4470</td>\n<td>0.4382</td>\n<td>0.4410</td>\n</tr>\n<tr>\n<td>161</td>\n<td>deberta v3 large</td>\n<td>-</td>\n<td>〇</td>\n<td>FB1 <br>+ Commonlit</td>\n<td>4</td>\n<td>0.4472</td>\n<td>0.4383</td>\n<td>0.4401</td>\n</tr>\n<tr>\n<td>102</td>\n<td>deberta base</td>\n<td>〇</td>\n<td>-</td>\n<td>FB1 <br>+ Commonlit</td>\n<td>4</td>\n<td>0.4483</td>\n<td>0.4387</td>\n<td>0.4386</td>\n</tr>\n<tr>\n<td>201</td>\n<td>deberta base</td>\n<td>-</td>\n<td>〇</td>\n<td>FB1 only</td>\n<td>4</td>\n<td>0.4503</td>\n<td>0.4390</td>\n<td>0.4399</td>\n</tr>\n<tr>\n<td>163</td>\n<td>deberta base</td>\n<td>-</td>\n<td>〇</td>\n<td>FB1 <br>+ Commonlit</td>\n<td>4</td>\n<td>0.4505</td>\n<td>0.4390</td>\n<td>0.4403</td>\n</tr>\n<tr>\n<td>215</td>\n<td>deberta base</td>\n<td>〇</td>\n<td>-</td>\n<td>FB1 only</td>\n<td>4</td>\n<td>0.4486</td>\n<td>0.4391</td>\n<td>0.4389</td>\n</tr>\n<tr>\n<td>190</td>\n<td>deberta base</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>10</td>\n<td>0.4508</td>\n<td>0.4393</td>\n<td>0.4393</td>\n</tr>\n<tr>\n<td>777</td>\n<td>deberta v3 large</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>10</td>\n<td>0.4479</td>\n<td>0.4398</td>\n<td>0.4413</td>\n</tr>\n<tr>\n<td>1</td>\n<td>deberta v3 large</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>4</td>\n<td>0.4486</td>\n<td>0.4400</td>\n<td>0.4421</td>\n</tr>\n<tr>\n<td>170</td>\n<td>deberta base</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>4</td>\n<td>0.4517</td>\n<td>0.4402</td>\n<td>0.4401</td>\n</tr>\n<tr>\n<td>20</td>\n<td>deberta v3 large</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>15</td>\n<td>0.4485</td>\n<td>0.4403</td>\n<td>0.4413</td>\n</tr>\n<tr>\n<td>18</td>\n<td>deberta base</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>4</td>\n<td>0.4516</td>\n<td>0.4405</td>\n<td>0.4393</td>\n</tr>\n<tr>\n<td>131</td>\n<td>deberta v1 large</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>4</td>\n<td>0.4515</td>\n<td>0.4413</td>\n<td>0.4431</td>\n</tr>\n<tr>\n<td>164</td>\n<td>electra-large</td>\n<td>-</td>\n<td>〇</td>\n<td>FB1 <br>+ Commonlit</td>\n<td>4</td>\n<td>0.4544</td>\n<td>0.4415</td>\n<td>0.4455</td>\n</tr>\n<tr>\n<td>216</td>\n<td>electra-large</td>\n<td>〇</td>\n<td>-</td>\n<td>FB1 only</td>\n<td>4</td>\n<td>0.4507</td>\n<td>0.4416</td>\n<td>0.4454</td>\n</tr>\n<tr>\n<td>202</td>\n<td>electra-large</td>\n<td>-</td>\n<td>〇</td>\n<td>FB1 only</td>\n<td>4</td>\n<td>0.4550</td>\n<td>0.4418</td>\n<td>0.4457</td>\n</tr>\n<tr>\n<td>114</td>\n<td>electra-large</td>\n<td>〇</td>\n<td>-</td>\n<td>FB1 <br>+ Commonlit</td>\n<td>4</td>\n<td>0.4516</td>\n<td>0.4419</td>\n<td>0.4458</td>\n</tr>\n<tr>\n<td>17</td>\n<td>deberta v3 large</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>4</td>\n<td>0.4502</td>\n<td>0.4419</td>\n<td>0.4430</td>\n</tr>\n<tr>\n<td>30</td>\n<td>deberta v1 xlarge</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>4</td>\n<td>0.4522</td>\n<td>0.4420</td>\n<td>0.4423</td>\n</tr>\n<tr>\n<td>33</td>\n<td>deberta v2 xlarge</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>4</td>\n<td>0.4540</td>\n<td>0.4425</td>\n<td>0.4432</td>\n</tr>\n<tr>\n<td>29</td>\n<td>electra-large</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>4</td>\n<td>0.4546</td>\n<td>0.4432</td>\n<td>0.4494</td>\n</tr>\n</tbody>\n</table>\n<h1>2. What worked</h1>\n<h2>2.1 High Impact</h2>\n<ul>\n<li>Not using the lower (cv : 0.002 improvement)</li>\n</ul>\n<p>At first, I converted the sentence to lower. But this was no good. In this competition, we need to predict the grammer etc… So Upper word may be important.</p>\n<ul>\n<li>Preprocess (cv 0.002 improvement)</li>\n</ul>\n<p>As shown in other solution, target name concatenated with space separators and added by [SEP] (cv 0.0005 improvement). After that, I replaced further line breaks with [Chapter] (cv 0.0015 improvement). Here is a concrete example.</p>\n<pre><code>Cohesion Syntax Vocabulary Phraseology Grammar Conventions [SEP] I think that students would benefit from learning at home,because they wont have to change and get up early in the morning to shower and do there hair. taking only classes helps them because at there house they'll be pay more attention. they will be comfortable at home. [Chapter] The hardest part of school is getting ready. you wake up go brush your teeth and go to your closet and look at your cloths. 　…\n</code></pre>\n<ul>\n<li>AWP (cv 0.001 improvement)</li>\n<li>Large k-fold</li>\n<li>Nealder-mead ensemble </li>\n</ul>\n<p>I used nealder-mead ensebmle in each target. This could jump the public lb from 600th to about 130 th at that time by changing from the mean ensemble.</p>\n<ul>\n<li>Pseudo labeling 1 (1 epoch pretrain + train.csv training)</li>\n</ul>\n<p>At first, about the leakage of pseudo labeling, I have experienced in NBME competition. So I could avoid the leakage. <a href=\"https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/323156\" target=\"_blank\">my solution on NBME</a></p>\n<p>As shown in other discussions, at first, I trained the 5 models using the train.csv and inferred the the feedback 1 ( and commonlit) dataset in each. By ensembling it, the pseudo label data was made. At second, I pretrained the pseudo label data. At last, I re-trained the train.csv from the pretrained models. For example, this could make the cv 0.447060 of deberta v3 large model. </p>\n<ul>\n<li>Pseudo labeling 2 (pseudo labeled data adding to the train.csv)</li>\n</ul>\n<p>This is the simple pseudo labeling. I added the pseudo labeled data (as explained pseudo labeling 1) to the train.csv, and training it. I could make the cv 0.446743 of deberta v3 large model.</p>\n<h2>2.2 Small Impact</h2>\n<ul>\n<li>Using CommonLit data for pseudo labeling</li>\n</ul>\n<p>We were allowed to use Commonlit data <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/348973#2047629\" target=\"_blank\">reference</a>. The ensemble improved the cv a little, but only slightly.</p>\n<ul>\n<li>Changing seed</li>\n</ul>\n<p>Changing the seed and ensemble improved the cv a bit, but I think it was overfitting.</p>\n<h1>3. What didn't worked for me</h1>\n<ul>\n<li>GBDT post-process with tf-idf etc.</li>\n<li>Replacement [MASK] in sentence.</li>\n<li>Cutmix</li>\n<li>adding LSTM to model</li>\n<li>Embedding SVR with fine-tuned models</li>\n</ul>\n<h1>4.Important ciations</h1>\n<ul>\n<li>FB3 / Deberta-v3-base baseline [train] <a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">link</a></li>\n<li>FB3 / Deberta-v3-base baseline [inference] <a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference/notebook\" target=\"_blank\">link</a></li>\n<li>(fgm)FB3 Deberta v3 base[train] <a href=\"https://www.kaggle.com/code/mozattt/fgm-fb3-deberta-v3-base-train\" target=\"_blank\">link</a></li>\n</ul>\n<h1>5.Thanks and Acknowledgements:</h1>\n<p>For various reasons, I started 11 days before the deadline. At this point, the various public notebooks were out and I was able to incorporate them into my pipeline and improve my score. <a href=\"https://www.kaggle.com/nakama\" target=\"_blank\">@nakama</a> and <a href=\"https://www.kaggle.com/Mozattt\" target=\"_blank\">@Mozattt</a> deserve my deepest thanks.</p>\n<p>Also, I learned a lot from this competition, especially the Efficiency Prize Evaluation, which was very practical. In real work, it is often costly and time-consuming to prepare a GPU, and this initiative helped my work a lot. Finally, I would like to say thank you again to my host, kaggle, for organizing such a competition.</p>\n<h1>6. Team Members</h1>\n<p>chumajin</p>",
  "comments": [
    {
      "id": 2059035,
      "author_name": "DJ_Xia",
      "author_url": "",
      "post_date": "2022-12-08T12:14:22.777000",
      "content": "<p>Great writeup.congrats to your strong silver,can you explain why <strong>replacement \\n to [Chapter]</strong>?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2059054,
          "author_name": "chumajin",
          "author_url": "",
          "post_date": "2022-12-08T12:42:13.147000",
          "content": "<p><a href=\"https://www.kaggle.com/tonymarkchris\" target=\"_blank\">@tonymarkchris</a> Thank you for comment ! In this competition, one of the target is grammar. So I looked for ways to clarify things like paragraph breaks. As a result, I arrived at the word [Chapter]. The 8th team seems to have improved their score by replacing it with [PARAGRAPH] <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369524\" target=\"_blank\">link here</a></p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2058283,
      "author_name": "Marília Prata",
      "author_url": "",
      "post_date": "2022-12-07T19:10:55.943000",
      "content": "<p>You did great, as always.   Congrats for your position and thank you for sharing this embracing solution Chumajin san.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2058436,
          "author_name": "chumajin",
          "author_url": "",
          "post_date": "2022-12-08T00:17:02.357000",
          "content": "<p><a href=\"https://www.kaggle.com/mpwolke\" target=\"_blank\">@mpwolke</a> Hi, my friend. Always thank you for comment. It always motivates me👍</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2058126,
      "author_name": "Julien Mouchnino",
      "author_url": "",
      "post_date": "2022-12-07T16:22:10.190000",
      "content": "<p>What is a Feedback dataset ?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2058437,
          "author_name": "chumajin",
          "author_url": "",
          "post_date": "2022-12-08T00:18:52.627000",
          "content": "<p><a href=\"https://www.kaggle.com/julienmouchnino\" target=\"_blank\">@julienmouchnino</a> This one (past feedback competition data). </p>\n<p><a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/data\" target=\"_blank\">https://www.kaggle.com/competitions/feedback-prize-english-language-learning/data</a></p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2057841,
      "author_name": "Olqa_842",
      "author_url": "",
      "post_date": "2022-12-07T11:36:55.613000",
      "content": "<p>Good result for your short period <a href=\"https://www.kaggle.com/chumajin\" target=\"_blank\">@chumajin</a> </p>",
      "votes": 1,
      "replies": [
        {
          "id": 2057848,
          "author_name": "chumajin",
          "author_url": "",
          "post_date": "2022-12-07T11:39:48.750000",
          "content": "<p><a href=\"https://www.kaggle.com/zvr842\" target=\"_blank\">@zvr842</a> Thank you very much !! I needed another week for gold medal… 😂</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2058115,
      "author_name": "Julien Mouchnino",
      "author_url": "",
      "post_date": "2022-12-07T16:16:46.313000",
      "content": "<p>Could you show how you fine-tuned the BERT models?</p>\n<p>Nobody explains this, is it assumed to be common knowledge?</p>",
      "votes": 2,
      "replies": [
        {
          "id": 2058440,
          "author_name": "chumajin",
          "author_url": "",
          "post_date": "2022-12-08T00:20:33.370000",
          "content": "<p><a href=\"https://www.kaggle.com/julienmouchnino\" target=\"_blank\">@julienmouchnino</a> Yes, common knowledge. <br>\nYou can learn from here.</p>\n<p><a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train</a></p>\n<p><a href=\"https://www.kaggle.com/code/mozattt/fgm-fb3-deberta-v3-base-train\" target=\"_blank\">https://www.kaggle.com/code/mozattt/fgm-fb3-deberta-v3-base-train</a></p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 2058807,
          "author_name": "Julien Mouchnino",
          "author_url": "",
          "post_date": "2022-12-08T08:12:00.693000",
          "content": "<p>The first link : there is code but not so much explanation. </p>\n<p>I know the basics of fine-tuning and tranfer learning, but I would appreciate a more detailed breakdown of the process</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 2059057,
          "author_name": "chumajin",
          "author_url": "",
          "post_date": "2022-12-08T12:46:17.147000",
          "content": "<p><a href=\"https://www.kaggle.com/julienmouchnino\" target=\"_blank\">@julienmouchnino</a> how about this one (my notebook on the past competition)?</p>\n<p><a href=\"https://www.kaggle.com/code/chumajin/pytorch-bert-beginner-s-room\" target=\"_blank\">https://www.kaggle.com/code/chumajin/pytorch-bert-beginner-s-room</a></p>\n<p>If you want to learn more, you can search by other kaggle notebooks, google or studying by a book/youtube etc… Maybe most people do it including me.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 12,
  "messages": [
    {
      "id": 2057809,
      "postDate": "2022-12-07T11:08:06.913Z",
      "content": "<h1>18 th place Solution</h1>\n<p>First of all, I would like to thank kaggle and the hosting staff for hosting such an interesting competition. And I would like to say congrats to all winner. I could not get the solo-gold, but I learned a lot from this competition.</p>\n<h1>1. Overview (Almost the same as others)</h1>\n<p>My solution is weight average ensemble with 21 models Nealder-mead with pseudo labeling. This is almost the same as other solutions. My originality is the replacement \\n to [Chapter]. This improved the cv 0.0015. And in this discussion, I show the my CV and Private LB relationship in detail.</p>\n<h1>1.1 Configuration</h1>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4001300%2Fa2ac96703257cc1d4353d8fc871d2028%2FClipboard01.jpg?generation=1670408759841540&amp;alt=media\" alt=\"\"></p>\n<h1>1.2 Relationship of cv and lb</h1>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4001300%2F5ea21d5c1e0e384bd0e25799427bd7ae%2FClipboard02.jpg?generation=1670408854083583&amp;alt=media\" alt=\"\"></p>\n<h1>1.3 cv and score in each model</h1>\n<p>※ pseudo1 and 2 are explained on next chapter<br>\n  ※ The order is the order with the best private lb score</p>\n<table>\n<thead>\n<tr>\n<th>EXP</th>\n<th>model</th>\n<th>pseudo1</th>\n<th>pseudo2</th>\n<th>pseudo<br> dataset</th>\n<th>kfold</th>\n<th>cv</th>\n<th>private lb</th>\n<th>public lb</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>214</td>\n<td>deberta v3 large</td>\n<td>〇</td>\n<td>-</td>\n<td>FB1 only</td>\n<td>4</td>\n<td>0.4467</td>\n<td>0.4378</td>\n<td>0.4404</td>\n</tr>\n<tr>\n<td>123</td>\n<td>deberta v3 large</td>\n<td>〇</td>\n<td>-</td>\n<td>FB1 <br>+ Commonlit</td>\n<td>4</td>\n<td>0.4467</td>\n<td>0.4379</td>\n<td>0.4402</td>\n</tr>\n<tr>\n<td>200</td>\n<td>deberta v3 large</td>\n<td>-</td>\n<td>〇</td>\n<td>FB1 only</td>\n<td>4</td>\n<td>0.4471</td>\n<td>0.4381</td>\n<td>0.4396</td>\n</tr>\n<tr>\n<td>250</td>\n<td>deberta v3 large</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>15</td>\n<td>0.4470</td>\n<td>0.4382</td>\n<td>0.4410</td>\n</tr>\n<tr>\n<td>161</td>\n<td>deberta v3 large</td>\n<td>-</td>\n<td>〇</td>\n<td>FB1 <br>+ Commonlit</td>\n<td>4</td>\n<td>0.4472</td>\n<td>0.4383</td>\n<td>0.4401</td>\n</tr>\n<tr>\n<td>102</td>\n<td>deberta base</td>\n<td>〇</td>\n<td>-</td>\n<td>FB1 <br>+ Commonlit</td>\n<td>4</td>\n<td>0.4483</td>\n<td>0.4387</td>\n<td>0.4386</td>\n</tr>\n<tr>\n<td>201</td>\n<td>deberta base</td>\n<td>-</td>\n<td>〇</td>\n<td>FB1 only</td>\n<td>4</td>\n<td>0.4503</td>\n<td>0.4390</td>\n<td>0.4399</td>\n</tr>\n<tr>\n<td>163</td>\n<td>deberta base</td>\n<td>-</td>\n<td>〇</td>\n<td>FB1 <br>+ Commonlit</td>\n<td>4</td>\n<td>0.4505</td>\n<td>0.4390</td>\n<td>0.4403</td>\n</tr>\n<tr>\n<td>215</td>\n<td>deberta base</td>\n<td>〇</td>\n<td>-</td>\n<td>FB1 only</td>\n<td>4</td>\n<td>0.4486</td>\n<td>0.4391</td>\n<td>0.4389</td>\n</tr>\n<tr>\n<td>190</td>\n<td>deberta base</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>10</td>\n<td>0.4508</td>\n<td>0.4393</td>\n<td>0.4393</td>\n</tr>\n<tr>\n<td>777</td>\n<td>deberta v3 large</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>10</td>\n<td>0.4479</td>\n<td>0.4398</td>\n<td>0.4413</td>\n</tr>\n<tr>\n<td>1</td>\n<td>deberta v3 large</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>4</td>\n<td>0.4486</td>\n<td>0.4400</td>\n<td>0.4421</td>\n</tr>\n<tr>\n<td>170</td>\n<td>deberta base</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>4</td>\n<td>0.4517</td>\n<td>0.4402</td>\n<td>0.4401</td>\n</tr>\n<tr>\n<td>20</td>\n<td>deberta v3 large</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>15</td>\n<td>0.4485</td>\n<td>0.4403</td>\n<td>0.4413</td>\n</tr>\n<tr>\n<td>18</td>\n<td>deberta base</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>4</td>\n<td>0.4516</td>\n<td>0.4405</td>\n<td>0.4393</td>\n</tr>\n<tr>\n<td>131</td>\n<td>deberta v1 large</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>4</td>\n<td>0.4515</td>\n<td>0.4413</td>\n<td>0.4431</td>\n</tr>\n<tr>\n<td>164</td>\n<td>electra-large</td>\n<td>-</td>\n<td>〇</td>\n<td>FB1 <br>+ Commonlit</td>\n<td>4</td>\n<td>0.4544</td>\n<td>0.4415</td>\n<td>0.4455</td>\n</tr>\n<tr>\n<td>216</td>\n<td>electra-large</td>\n<td>〇</td>\n<td>-</td>\n<td>FB1 only</td>\n<td>4</td>\n<td>0.4507</td>\n<td>0.4416</td>\n<td>0.4454</td>\n</tr>\n<tr>\n<td>202</td>\n<td>electra-large</td>\n<td>-</td>\n<td>〇</td>\n<td>FB1 only</td>\n<td>4</td>\n<td>0.4550</td>\n<td>0.4418</td>\n<td>0.4457</td>\n</tr>\n<tr>\n<td>114</td>\n<td>electra-large</td>\n<td>〇</td>\n<td>-</td>\n<td>FB1 <br>+ Commonlit</td>\n<td>4</td>\n<td>0.4516</td>\n<td>0.4419</td>\n<td>0.4458</td>\n</tr>\n<tr>\n<td>17</td>\n<td>deberta v3 large</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>4</td>\n<td>0.4502</td>\n<td>0.4419</td>\n<td>0.4430</td>\n</tr>\n<tr>\n<td>30</td>\n<td>deberta v1 xlarge</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>4</td>\n<td>0.4522</td>\n<td>0.4420</td>\n<td>0.4423</td>\n</tr>\n<tr>\n<td>33</td>\n<td>deberta v2 xlarge</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>4</td>\n<td>0.4540</td>\n<td>0.4425</td>\n<td>0.4432</td>\n</tr>\n<tr>\n<td>29</td>\n<td>electra-large</td>\n<td>-</td>\n<td>-</td>\n<td>-</td>\n<td>4</td>\n<td>0.4546</td>\n<td>0.4432</td>\n<td>0.4494</td>\n</tr>\n</tbody>\n</table>\n<h1>2. What worked</h1>\n<h2>2.1 High Impact</h2>\n<ul>\n<li>Not using the lower (cv : 0.002 improvement)</li>\n</ul>\n<p>At first, I converted the sentence to lower. But this was no good. In this competition, we need to predict the grammer etc… So Upper word may be important.</p>\n<ul>\n<li>Preprocess (cv 0.002 improvement)</li>\n</ul>\n<p>As shown in other solution, target name concatenated with space separators and added by [SEP] (cv 0.0005 improvement). After that, I replaced further line breaks with [Chapter] (cv 0.0015 improvement). Here is a concrete example.</p>\n<pre><code>Cohesion Syntax Vocabulary Phraseology Grammar Conventions [SEP] I think that students would benefit from learning at home,because they wont have to change and get up early in the morning to shower and do there hair. taking only classes helps them because at there house they'll be pay more attention. they will be comfortable at home. [Chapter] The hardest part of school is getting ready. you wake up go brush your teeth and go to your closet and look at your cloths. 　…\n</code></pre>\n<ul>\n<li>AWP (cv 0.001 improvement)</li>\n<li>Large k-fold</li>\n<li>Nealder-mead ensemble </li>\n</ul>\n<p>I used nealder-mead ensebmle in each target. This could jump the public lb from 600th to about 130 th at that time by changing from the mean ensemble.</p>\n<ul>\n<li>Pseudo labeling 1 (1 epoch pretrain + train.csv training)</li>\n</ul>\n<p>At first, about the leakage of pseudo labeling, I have experienced in NBME competition. So I could avoid the leakage. <a href=\"https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/323156\" target=\"_blank\">my solution on NBME</a></p>\n<p>As shown in other discussions, at first, I trained the 5 models using the train.csv and inferred the the feedback 1 ( and commonlit) dataset in each. By ensembling it, the pseudo label data was made. At second, I pretrained the pseudo label data. At last, I re-trained the train.csv from the pretrained models. For example, this could make the cv 0.447060 of deberta v3 large model. </p>\n<ul>\n<li>Pseudo labeling 2 (pseudo labeled data adding to the train.csv)</li>\n</ul>\n<p>This is the simple pseudo labeling. I added the pseudo labeled data (as explained pseudo labeling 1) to the train.csv, and training it. I could make the cv 0.446743 of deberta v3 large model.</p>\n<h2>2.2 Small Impact</h2>\n<ul>\n<li>Using CommonLit data for pseudo labeling</li>\n</ul>\n<p>We were allowed to use Commonlit data <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/348973#2047629\" target=\"_blank\">reference</a>. The ensemble improved the cv a little, but only slightly.</p>\n<ul>\n<li>Changing seed</li>\n</ul>\n<p>Changing the seed and ensemble improved the cv a bit, but I think it was overfitting.</p>\n<h1>3. What didn't worked for me</h1>\n<ul>\n<li>GBDT post-process with tf-idf etc.</li>\n<li>Replacement [MASK] in sentence.</li>\n<li>Cutmix</li>\n<li>adding LSTM to model</li>\n<li>Embedding SVR with fine-tuned models</li>\n</ul>\n<h1>4.Important ciations</h1>\n<ul>\n<li>FB3 / Deberta-v3-base baseline [train] <a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">link</a></li>\n<li>FB3 / Deberta-v3-base baseline [inference] <a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference/notebook\" target=\"_blank\">link</a></li>\n<li>(fgm)FB3 Deberta v3 base[train] <a href=\"https://www.kaggle.com/code/mozattt/fgm-fb3-deberta-v3-base-train\" target=\"_blank\">link</a></li>\n</ul>\n<h1>5.Thanks and Acknowledgements:</h1>\n<p>For various reasons, I started 11 days before the deadline. At this point, the various public notebooks were out and I was able to incorporate them into my pipeline and improve my score. <a href=\"https://www.kaggle.com/nakama\" target=\"_blank\">@nakama</a> and <a href=\"https://www.kaggle.com/Mozattt\" target=\"_blank\">@Mozattt</a> deserve my deepest thanks.</p>\n<p>Also, I learned a lot from this competition, especially the Efficiency Prize Evaluation, which was very practical. In real work, it is often costly and time-consuming to prepare a GPU, and this initiative helped my work a lot. Finally, I would like to say thank you again to my host, kaggle, for organizing such a competition.</p>\n<h1>6. Team Members</h1>\n<p>chumajin</p>",
      "rawMarkdown": "# 18 th place Solution\n\nFirst of all, I would like to thank kaggle and the hosting staff for hosting such an interesting competition. And I would like to say congrats to all winner. I could not get the solo-gold, but I learned a lot from this competition.\n\n# 1. Overview (Almost the same as others)\n\nMy solution is weight average ensemble with 21 models Nealder-mead with pseudo labeling. This is almost the same as other solutions. My originality is the replacement \\n to [Chapter]. This improved the cv 0.0015. And in this discussion, I show the my CV and Private LB relationship in detail.\n\n# 1.1 Configuration\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4001300%2Fa2ac96703257cc1d4353d8fc871d2028%2FClipboard01.jpg?generation=1670408759841540&alt=media)\n\n\n\n# 1.2 Relationship of cv and lb \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4001300%2F5ea21d5c1e0e384bd0e25799427bd7ae%2FClipboard02.jpg?generation=1670408854083583&alt=media)\n\n\n\n\n\n\n# 1.3 cv and score in each model\n  ※ pseudo1 and 2 are explained on next chapter\n  ※ The order is the order with the best private lb score\n\n\n| EXP | model             | pseudo1| pseudo2 | pseudo<br/> dataset  | kfold | cv      | private lb | public lb |\n|-----|-------------------|---------------------|---------------------|----------------------|-------|---------|------------|-----------|\n| 214 | deberta v3 large  | 〇                   | -                   | FB1 only             | 4     | 0.4467  | 0.4378     | 0.4404    |\n| 123 | deberta v3 large  | 〇                   | -                   | FB1 <br/>+ Commonlit | 4     | 0.4467  | 0.4379     | 0.4402    |\n| 200 | deberta v3 large  | -                   | 〇                   | FB1 only             | 4     | 0.4471  | 0.4381     | 0.4396    |\n| 250 | deberta v3 large  | -                   | -                   | -                    | 15    | 0.4470  | 0.4382     | 0.4410    |\n| 161 | deberta v3 large  | -                   | 〇                   | FB1 <br/>+ Commonlit | 4     | 0.4472  | 0.4383     | 0.4401    |\n| 102 | deberta base      | 〇                   | -                   | FB1 <br/>+ Commonlit | 4     | 0.4483  | 0.4387     | 0.4386    |\n| 201 | deberta base      | -                   | 〇                   | FB1 only             | 4     | 0.4503  | 0.4390     | 0.4399    |\n| 163 | deberta base      | -                   | 〇                   | FB1 <br/>+ Commonlit | 4     | 0.4505  | 0.4390     | 0.4403    |\n| 215 | deberta base      | 〇                   | -                   | FB1 only             | 4     | 0.4486  | 0.4391     | 0.4389    |\n| 190 | deberta base      | -                   | -                   | -                    | 10    | 0.4508  | 0.4393     | 0.4393    |\n| 777 | deberta v3 large | -                   | -                   | -                    | 10    | 0.4479  | 0.4398     | 0.4413    |\n| 1   | deberta v3 large  | -                   | -                   | -                    | 4     | 0.4486  | 0.4400     | 0.4421    |\n| 170 | deberta base      | -                   | -                   | -                    | 4     | 0.4517  | 0.4402     | 0.4401    |\n| 20  | deberta v3 large  | -                   | -                   | -                    | 15    | 0.4485  | 0.4403     | 0.4413    |\n| 18  | deberta base      | -                   | -                   | -                    | 4     | 0.4516  | 0.4405     | 0.4393    |\n| 131 | deberta v1 large  | -                   | -                   | -                    | 4     | 0.4515  | 0.4413     | 0.4431    |\n| 164 | electra-large     | -                   | 〇                   | FB1 <br/>+ Commonlit | 4     | 0.4544  | 0.4415     | 0.4455    |\n| 216 | electra-large     | 〇                   | -                   | FB1 only             | 4     | 0.4507  | 0.4416     | 0.4454    |\n| 202 | electra-large     | -                   | 〇                   | FB1 only             | 4     | 0.4550  | 0.4418     | 0.4457    |\n| 114 | electra-large     | 〇                   | -                   | FB1 <br/>+ Commonlit | 4     | 0.4516  | 0.4419     | 0.4458    |\n| 17  | deberta v3 large  | -                   | -                   | -                    | 4     | 0.4502  | 0.4419     | 0.4430    |\n| 30  | deberta v1 xlarge | -                   | -                   | -                    | 4     | 0.4522  | 0.4420     | 0.4423    |\n| 33  | deberta v2 xlarge | -                   | -                   | -                    | 4     | 0.4540  | 0.4425     | 0.4432    |\n| 29  | electra-large     | -                   | -                   | -                    | 4     | 0.4546  | 0.4432     | 0.4494    |\n\n# 2. What worked\n## 2.1 High Impact\n\n* Not using the lower (cv : 0.002 improvement)\n\nAt first, I converted the sentence to lower. But this was no good. In this competition, we need to predict the grammer etc... So Upper word may be important.\n\n\n* Preprocess (cv 0.002 improvement)\n\nAs shown in other solution, target name concatenated with space separators and added by [SEP] (cv 0.0005 improvement). After that, I replaced further line breaks with [Chapter] (cv 0.0015 improvement). Here is a concrete example.\n\n```\nCohesion Syntax Vocabulary Phraseology Grammar Conventions [SEP] I think that students would benefit from learning at home,because they wont have to change and get up early in the morning to shower and do there hair. taking only classes helps them because at there house they'll be pay more attention. they will be comfortable at home. [Chapter] The hardest part of school is getting ready. you wake up go brush your teeth and go to your closet and look at your cloths. 　…\n```\n\n* AWP (cv 0.001 improvement)\n* Large k-fold\n* Nealder-mead ensemble \n\nI used nealder-mead ensebmle in each target. This could jump the public lb from 600th to about 130 th at that time by changing from the mean ensemble.\n\n* Pseudo labeling 1 (1 epoch pretrain + train.csv training)\n\nAt first, about the leakage of pseudo labeling, I have experienced in NBME competition. So I could avoid the leakage. [my solution on NBME](https://www.kaggle.com/competitions/nbme-score-clinical-patient-notes/discussion/323156)\n\nAs shown in other discussions, at first, I trained the 5 models using the train.csv and inferred the the feedback 1 ( and commonlit) dataset in each. By ensembling it, the pseudo label data was made. At second, I pretrained the pseudo label data. At last, I re-trained the train.csv from the pretrained models. For example, this could make the cv 0.447060 of deberta v3 large model. \n\n* Pseudo labeling 2 (pseudo labeled data adding to the train.csv)\n\nThis is the simple pseudo labeling. I added the pseudo labeled data (as explained pseudo labeling 1) to the train.csv, and training it. I could make the cv 0.446743 of deberta v3 large model.\n\n## 2.2 Small Impact\n\n* Using CommonLit data for pseudo labeling\n\nWe were allowed to use Commonlit data [reference](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/348973#2047629). The ensemble improved the cv a little, but only slightly.\n\n* Changing seed\n\nChanging the seed and ensemble improved the cv a bit, but I think it was overfitting.\n\n\n# 3. What didn't worked for me\n\n* GBDT post-process with tf-idf etc.\n* Replacement [MASK] in sentence.\n* Cutmix\n* adding LSTM to model\n* Embedding SVR with fine-tuned models\n\n# 4.Important ciations\n\n* FB3 / Deberta-v3-base baseline [train] [link](https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train)\n* FB3 / Deberta-v3-base baseline [inference] [link](https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference/notebook)\n* (fgm)FB3 Deberta v3 base[train] [link](https://www.kaggle.com/code/mozattt/fgm-fb3-deberta-v3-base-train)\n\n\n# 5.Thanks and Acknowledgements:\nFor various reasons, I started 11 days before the deadline. At this point, the various public notebooks were out and I was able to incorporate them into my pipeline and improve my score. @nakama and @Mozattt deserve my deepest thanks.\n\nAlso, I learned a lot from this competition, especially the Efficiency Prize Evaluation, which was very practical. In real work, it is often costly and time-consuming to prepare a GPU, and this initiative helped my work a lot. Finally, I would like to say thank you again to my host, kaggle, for organizing such a competition.\n\n# 6. Team Members\nchumajin",
      "votes": 21
    },
    {
      "id": 2059035,
      "postDate": "2022-12-08T12:14:22.777Z",
      "content": "<p>Great writeup.congrats to your strong silver,can you explain why <strong>replacement \\n to [Chapter]</strong>?</p>",
      "rawMarkdown": "Great writeup.congrats to your strong silver,can you explain why **replacement \\n to [Chapter]**?",
      "votes": 1,
      "replies": [
        {
          "id": 2059054,
          "postDate": "2022-12-08T12:42:13.147Z",
          "content": "<p><a href=\"https://www.kaggle.com/tonymarkchris\" target=\"_blank\">@tonymarkchris</a> Thank you for comment ! In this competition, one of the target is grammar. So I looked for ways to clarify things like paragraph breaks. As a result, I arrived at the word [Chapter]. The 8th team seems to have improved their score by replacing it with [PARAGRAPH] <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369524\" target=\"_blank\">link here</a></p>",
          "rawMarkdown": "@tonymarkchris Thank you for comment ! In this competition, one of the target is grammar. So I looked for ways to clarify things like paragraph breaks. As a result, I arrived at the word [Chapter]. The 8th team seems to have improved their score by replacing it with [PARAGRAPH] [link here](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369524)"
        }
      ]
    },
    {
      "id": 2058283,
      "postDate": "2022-12-07T19:10:55.943Z",
      "content": "<p>You did great, as always.   Congrats for your position and thank you for sharing this embracing solution Chumajin san.</p>",
      "rawMarkdown": "You did great, as always.   Congrats for your position and thank you for sharing this embracing solution Chumajin san.",
      "votes": 1,
      "replies": [
        {
          "id": 2058436,
          "postDate": "2022-12-08T00:17:02.357Z",
          "content": "<p><a href=\"https://www.kaggle.com/mpwolke\" target=\"_blank\">@mpwolke</a> Hi, my friend. Always thank you for comment. It always motivates me👍</p>",
          "rawMarkdown": "@mpwolke Hi, my friend. Always thank you for comment. It always motivates me👍",
          "votes": 1
        }
      ]
    },
    {
      "id": 2058126,
      "postDate": "2022-12-07T16:22:10.190Z",
      "content": "<p>What is a Feedback dataset ?</p>",
      "rawMarkdown": "What is a Feedback dataset ?",
      "votes": 1,
      "replies": [
        {
          "id": 2058437,
          "postDate": "2022-12-08T00:18:52.627Z",
          "content": "<p><a href=\"https://www.kaggle.com/julienmouchnino\" target=\"_blank\">@julienmouchnino</a> This one (past feedback competition data). </p>\n<p><a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/data\" target=\"_blank\">https://www.kaggle.com/competitions/feedback-prize-english-language-learning/data</a></p>",
          "rawMarkdown": "@julienmouchnino This one (past feedback competition data). \n\nhttps://www.kaggle.com/competitions/feedback-prize-english-language-learning/data",
          "votes": 1
        }
      ]
    },
    {
      "id": 2057841,
      "postDate": "2022-12-07T11:36:55.613Z",
      "content": "<p>Good result for your short period <a href=\"https://www.kaggle.com/chumajin\" target=\"_blank\">@chumajin</a> </p>",
      "rawMarkdown": "Good result for your short period @chumajin ",
      "votes": 1,
      "replies": [
        {
          "id": 2057848,
          "postDate": "2022-12-07T11:39:48.750Z",
          "content": "<p><a href=\"https://www.kaggle.com/zvr842\" target=\"_blank\">@zvr842</a> Thank you very much !! I needed another week for gold medal… 😂</p>",
          "rawMarkdown": "@zvr842 Thank you very much !! I needed another week for gold medal... 😂"
        }
      ]
    },
    {
      "id": 2058115,
      "postDate": "2022-12-07T16:16:46.313Z",
      "content": "<p>Could you show how you fine-tuned the BERT models?</p>\n<p>Nobody explains this, is it assumed to be common knowledge?</p>",
      "rawMarkdown": "Could you show how you fine-tuned the BERT models?\n\nNobody explains this, is it assumed to be common knowledge?\n ",
      "votes": 2,
      "replies": [
        {
          "id": 2058440,
          "postDate": "2022-12-08T00:20:33.370Z",
          "content": "<p><a href=\"https://www.kaggle.com/julienmouchnino\" target=\"_blank\">@julienmouchnino</a> Yes, common knowledge. <br>\nYou can learn from here.</p>\n<p><a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train</a></p>\n<p><a href=\"https://www.kaggle.com/code/mozattt/fgm-fb3-deberta-v3-base-train\" target=\"_blank\">https://www.kaggle.com/code/mozattt/fgm-fb3-deberta-v3-base-train</a></p>",
          "rawMarkdown": "@julienmouchnino Yes, common knowledge. \nYou can learn from here.\n\nhttps://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\n\nhttps://www.kaggle.com/code/mozattt/fgm-fb3-deberta-v3-base-train"
        },
        {
          "id": 2058807,
          "postDate": "2022-12-08T08:12:00.693Z",
          "content": "<p>The first link : there is code but not so much explanation. </p>\n<p>I know the basics of fine-tuning and tranfer learning, but I would appreciate a more detailed breakdown of the process</p>",
          "rawMarkdown": "The first link : there is code but not so much explanation. \n\nI know the basics of fine-tuning and tranfer learning, but I would appreciate a more detailed breakdown of the process\n"
        },
        {
          "id": 2059057,
          "postDate": "2022-12-08T12:46:17.147Z",
          "content": "<p><a href=\"https://www.kaggle.com/julienmouchnino\" target=\"_blank\">@julienmouchnino</a> how about this one (my notebook on the past competition)?</p>\n<p><a href=\"https://www.kaggle.com/code/chumajin/pytorch-bert-beginner-s-room\" target=\"_blank\">https://www.kaggle.com/code/chumajin/pytorch-bert-beginner-s-room</a></p>\n<p>If you want to learn more, you can search by other kaggle notebooks, google or studying by a book/youtube etc… Maybe most people do it including me.</p>",
          "rawMarkdown": "@julienmouchnino how about this one (my notebook on the past competition)?\n\n https://www.kaggle.com/code/chumajin/pytorch-bert-beginner-s-room\n\nIf you want to learn more, you can search by other kaggle notebooks, google or studying by a book/youtube etc... Maybe most people do it including me."
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T20:06:00Z",
  "competition_id": 38321,
  "competition_slug": "feedback-prize-english-language-learning",
  "team_id": 9255874,
  "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/chumajin-18th-place-solution"
}