{
  "id": 369409,
  "title": "11th Place Solution",
  "content": "<h1>11th Place Solution</h1>\n<p>First of all, I'd like to thank Kaggle and host for hosting this competition.<br>\nI still can't believe I got a solo gold.</p>\n<h2>Overview</h2>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2648907%2F0d8799582d6ad0461e9ec995a3a1d3f7%2F2022-12-06%2013.47.10.png?generation=1670305587215632&amp;alt=media\" alt=\"\"></p>\n<p>I ensembled 21 models (including models trained with Pseudo Labels). I used the Ridge-Regression and <a href=\"https://kaggler.readthedocs.io/en/latest/_modules/kaggler/ensemble/linear.html#netflix\" target=\"_blank\">Netflix method</a> for ensembling.</p>\n<p>For all of my transformer models I used the <a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">this notebook</a> and <a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference/notebook\" target=\"_blank\">this notebook</a>.</p>\n<h2>Code</h2>\n<ul>\n<li>Inference Notebook: <a href=\"https://www.kaggle.com/code/irrohas/final-many-deberta-ridge\" target=\"_blank\">https://www.kaggle.com/code/irrohas/final-many-deberta-ridge</a></li>\n<li>Code (Training): I will upload later.</li>\n</ul>\n<h2>What Worked</h2>\n<ul>\n<li><p>[Hight Impact] Ridge-Regression and Netflix method per target for ensembling</p>\n<ul>\n<li>Result: about -0.009 </li>\n<li>Reasoning and context: Adding the predictions of various models (even if the each score was low) to the ensemble, not just Deberta-v3, improved the subsequent CV. So I used 10 other models besides deberta-v3. </li></ul></li>\n<li><p>[Low Impact] Pseudo Label</p>\n<ul>\n<li>Result: about -0.003</li>\n<li>Reasoning and context: I created a pseudo label with the 'Feedback Prize - Evaluating Student Writing'(FB1) competition data. However, there were too many pseudo-labels for the training data, and using all of them was likely to result in a large leakage, so we limited the pseudo-data that could be used in each fold.<br>\nI made up to 2 steps of learning with pseudo labels.</li></ul></li>\n</ul>\n<h2>What Didn't Work</h2>\n<ul>\n<li>Stacking for 2nd stage prediction<ul>\n<li>What you did: I tried LGBM, 2DCNN, MLP, and GCN for 2nd stage prediciton.</li>\n<li>Result: worse</li>\n<li>Reasoning and context: I tried LGBM, 2DCNN, MLP, and GCN for 2nd stage prediciton. However, all of them made CV worse. Also the weight optimization by Nelder-Mead method did score better, but not as well as Ridge-Regression or Netflix-method, so we did not use it for final submission.</li></ul></li>\n<li>Pretraiend from previous competition dataset<ul>\n<li>what you did: Before training, I pretrained model with data from a previous competition ('Feedback Prize - Evaluating Student Writing' and 'Feedback Prize - Predicting Effective Arguments')</li>\n<li>Result: not change or worse</li>\n<li>Reasoning and context: Scores were almost the same or slightly worse, so I stopped considering the cost of learning.</li></ul></li>\n<li>Changing the model depending on the length of the input text<ul>\n<li>Result: worse</li>\n<li>Reasoning and context: Deberta-v3 models can handle sentences of arbitrary length, while Roberta and Funnel were limited because of 512 and 1024. However, it was determined that even if the <code>max_len</code> limit was used, the model score would remain almost the same and could be used to seed the ensemble.</li></ul></li>\n</ul>\n<h2>Additional Context</h2>\n<h3>CV Strategy</h3>\n<p>I used <a href=\"https://www.kaggle.com/code/abhishek/multi-label-stratified-folds\" target=\"_blank\">abiheshark's cv strategy</a> with 4,5 folds various seed.</p>\n<h3>Training Config</h3>\n<ul>\n<li>Oprimizer: AdamW<ul>\n<li>weight decay: 0.01</li>\n<li>initial lr: 2e-5</li></ul></li>\n<li>Loss: SmoothL1Loss</li>\n<li>No dropout</li>\n</ul>\n<h3>My Score</h3>\n<p>My each model score is here, (Including models not in use)<br>\nI use mainly mean Pooling for model's head. (I tried LSTMhead and Attentionhead, but the score not changed)</p>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>CV</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Deberta-v3-base</td>\n<td>0.4590</td>\n</tr>\n<tr>\n<td>Deberta-v3-large</td>\n<td>0.4519</td>\n</tr>\n<tr>\n<td>Roberta-large</td>\n<td>0.4583</td>\n</tr>\n<tr>\n<td>luke-large</td>\n<td>0.4590</td>\n</tr>\n<tr>\n<td>bigbird-roberta-large</td>\n<td>0.4643</td>\n</tr>\n<tr>\n<td>Deberta-large</td>\n<td>0.4615</td>\n</tr>\n<tr>\n<td>Deberta-large-mnli</td>\n<td>0.4629</td>\n</tr>\n<tr>\n<td>Funnel-large</td>\n<td>0.4686</td>\n</tr>\n<tr>\n<td>Funnel-large-base</td>\n<td>0.4644</td>\n</tr>\n<tr>\n<td>Bart-large</td>\n<td>0.4606</td>\n</tr>\n<tr>\n<td>GPT2-medium</td>\n<td>0.4658</td>\n</tr>\n<tr>\n<td>GPT2-large</td>\n<td>0.4882</td>\n</tr>\n<tr>\n<td>mpnet-base</td>\n<td>0.4952</td>\n</tr>\n<tr>\n<td>electra-large</td>\n<td>0.4731</td>\n</tr>\n<tr>\n<td>Deberta-v3-large (1st PL)</td>\n<td>0.4473</td>\n</tr>\n<tr>\n<td>Debreta-v3-large (2nd PL)</td>\n<td>0.4467</td>\n</tr>\n</tbody>\n</table>\n<p>My submissions CV score and Best Score CV are here,</p>\n<table>\n<thead>\n<tr>\n<th>submission</th>\n<th>CV</th>\n<th>My Inner Private Score Rank</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>20 models + Ridge</td>\n<td>0.4417</td>\n<td>2</td>\n</tr>\n<tr>\n<td>21 models + (0.5Ridge + 0.5Netflix)</td>\n<td>Ridge:0.4417, Netflix:0.4411</td>\n<td>3</td>\n</tr>\n<tr>\n<td>13 models + Ridge</td>\n<td>0.4421</td>\n<td>10</td>\n</tr>\n<tr>\n<td>16 models + Ridge</td>\n<td>0.4420</td>\n<td>1</td>\n</tr>\n</tbody>\n</table>\n<h3>Model Selection</h3>\n<p>I made a very large number of models, but had to choose a model due to dataset storage, Deberta-v3 had the highest score, so I chose about half of them, and the rest I chose by looking at the CV of the various models. So electra and mpnet are not in the final sub. But the best private scores were the models that included them :_(</p>\n<p>We also considered the need for a variety of Deberya-v3 models, so we used model weights from <a href=\"https://www.kaggle.com/code/kojimar/fb3-deberta-family-inference-9-28-updated\" target=\"_blank\">public Code</a>. I really appreciate.</p>\n<h2>Important Citations:</h2>\n<ul>\n<li>Code:<ul>\n<li><a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train</a></li>\n<li><a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference/notebook\" target=\"_blank\">https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference/notebook</a></li>\n<li><a href=\"https://www.kaggle.com/code/kojimar/fb3-deberta-family-inference-9-28-updated\" target=\"_blank\">https://www.kaggle.com/code/kojimar/fb3-deberta-family-inference-9-28-updated</a></li>\n<li><a href=\"https://www.kaggle.com/code/abhishek/multi-label-stratified-folds\" target=\"_blank\">https://www.kaggle.com/code/abhishek/multi-label-stratified-folds</a></li></ul></li>\n<li>Others:<ul>\n<li>netflix method: <a href=\"https://kaggler.readthedocs.io/en/latest/_modules/kaggler/ensemble/linear.html#netflix\" target=\"_blank\">https://kaggler.readthedocs.io/en/latest/_modules/kaggler/ensemble/linear.html#netflix</a></li></ul></li>\n</ul>\n<h2>Thanks and Acknowledgements:</h2>\n<p>Finally, I'd like to thank Kaggle, host for hosting such an iteresting competition and all who participated in lively discussions.<br>\nI hope my solution helps you in your future NLP competitions!</p>",
  "comments": [
    {
      "id": 2049384,
      "author_name": "quangp",
      "author_url": "",
      "post_date": "2022-11-30T05:18:57.510000",
      "content": "<p>Thanks for sharing. How did you use Ridge? Is it stacking on transfomers predictions? I tried but my result is always worse than tuning weights.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2049806,
          "author_name": "Masaya",
          "author_url": "",
          "post_date": "2022-11-30T10:58:02.407000",
          "content": "<p>Thank you for question. <br>\nI used Ridge regression for decision of model weight for each target.<br>\nI tried stacking on transformers' prediction (6 targets) too, but that resulted in worse score.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 2050073,
          "author_name": "quangp",
          "author_url": "",
          "post_date": "2022-11-30T14:06:13.393000",
          "content": "<p>Can you share more detail about that? I really want to learn since all I do was trying to stacking but It didn't work. Thanks in advance!</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 2050185,
          "author_name": "Masaya",
          "author_url": "",
          "post_date": "2022-11-30T15:20:13.583000",
          "content": "<p>The following is my opinion. I'm sorry if I'm wrong. </p>\n<p>Stacking use the all targets for prediction. For example, I used the 'syntax' and 'vocabulary' predictions besides 'cohesion' predictions for 'cohesion' prediction in 2nd stage.</p>\n<p>On the other hand, ensemble use the only 'cohesion' predictions for 'cohesion' prediction in 2nd stage.<br>\nIn the other words, I simply determine the weight by regression.</p>\n<p>I think that the reason why stacking made the score worse is that other features (i.e. 'syntax' prediction or 'vocabulary' prediction in 1st stage) are noisy. (Or other reason is in)</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 2050283,
          "author_name": "quangp",
          "author_url": "",
          "post_date": "2022-11-30T16:18:57.430000",
          "content": "<p>I understand your point now. I think your explanation makes sense. Thank you for sharing mate 👍</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2049379,
      "author_name": "gezi",
      "author_url": "",
      "post_date": "2022-11-30T05:11:23.677000",
      "content": "<p>I also tried linear lr and gbdt all worse. And optuna seems overfit cv not improve PB comparing to using manual set weight per target.<br>\nSo Ridge-Regressio with -0.009 is so cool!  Thanks for sharing!</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 2050176,
      "author_name": "kintaro1",
      "author_url": "",
      "post_date": "2022-11-30T15:16:19.283000",
      "content": "<p>solo gold congratulations！<br>\nLet me ask two questions about pseudo-labeling.</p>\n<ol>\n<li>how many data number of events did you use for each fold?</li>\n<li>When you say \"2 steps of learning with pseudo labels.\", do you mean that you trained with pseudo labels for 2 epochs and then used train data from FB3?</li>\n</ol>",
      "votes": 0,
      "replies": [
        {
          "id": 2050209,
          "author_name": "Masaya",
          "author_url": "",
          "post_date": "2022-11-30T15:31:22.847000",
          "content": "<p>Thank you for question.</p>\n<blockquote>\n  <p>how many data number of events did you use for each fold?</p>\n</blockquote>\n<p>About 3,000. In 4 folds training, I have divided the pseudo label data into 4 parts and I added each of them to training data in each fold.</p>\n<blockquote>\n  <p>When you say \"2 steps of learning with pseudo labels.\", do you mean that you trained with pseudo labels for 2 epochs and then used train data from FB3?</p>\n</blockquote>\n<p>No. I trained with both pseudo label data and FB3 data in all epochs.<br>\n\"2 steps of learning with pseudo labels.\" This means multi-step pseudo labeling. In other words, I created new data(= 2nd pseudo label data) by using the model trained on the 1st data.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 2050233,
          "author_name": "kintaro1",
          "author_url": "",
          "post_date": "2022-11-30T15:44:52.560000",
          "content": "<p>I see, now I understand！<br>\nThank you for your prompt response.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2049432,
      "author_name": "Santhoshkumar",
      "author_url": "",
      "post_date": "2022-11-30T06:35:00.110000",
      "content": "<p>great, thanks for sharing👌👍</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 10,
  "messages": [
    {
      "id": 2049375,
      "postDate": "2022-11-30T05:04:27.657Z",
      "content": "<h1>11th Place Solution</h1>\n<p>First of all, I'd like to thank Kaggle and host for hosting this competition.<br>\nI still can't believe I got a solo gold.</p>\n<h2>Overview</h2>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2648907%2F0d8799582d6ad0461e9ec995a3a1d3f7%2F2022-12-06%2013.47.10.png?generation=1670305587215632&amp;alt=media\" alt=\"\"></p>\n<p>I ensembled 21 models (including models trained with Pseudo Labels). I used the Ridge-Regression and <a href=\"https://kaggler.readthedocs.io/en/latest/_modules/kaggler/ensemble/linear.html#netflix\" target=\"_blank\">Netflix method</a> for ensembling.</p>\n<p>For all of my transformer models I used the <a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">this notebook</a> and <a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference/notebook\" target=\"_blank\">this notebook</a>.</p>\n<h2>Code</h2>\n<ul>\n<li>Inference Notebook: <a href=\"https://www.kaggle.com/code/irrohas/final-many-deberta-ridge\" target=\"_blank\">https://www.kaggle.com/code/irrohas/final-many-deberta-ridge</a></li>\n<li>Code (Training): I will upload later.</li>\n</ul>\n<h2>What Worked</h2>\n<ul>\n<li><p>[Hight Impact] Ridge-Regression and Netflix method per target for ensembling</p>\n<ul>\n<li>Result: about -0.009 </li>\n<li>Reasoning and context: Adding the predictions of various models (even if the each score was low) to the ensemble, not just Deberta-v3, improved the subsequent CV. So I used 10 other models besides deberta-v3. </li></ul></li>\n<li><p>[Low Impact] Pseudo Label</p>\n<ul>\n<li>Result: about -0.003</li>\n<li>Reasoning and context: I created a pseudo label with the 'Feedback Prize - Evaluating Student Writing'(FB1) competition data. However, there were too many pseudo-labels for the training data, and using all of them was likely to result in a large leakage, so we limited the pseudo-data that could be used in each fold.<br>\nI made up to 2 steps of learning with pseudo labels.</li></ul></li>\n</ul>\n<h2>What Didn't Work</h2>\n<ul>\n<li>Stacking for 2nd stage prediction<ul>\n<li>What you did: I tried LGBM, 2DCNN, MLP, and GCN for 2nd stage prediciton.</li>\n<li>Result: worse</li>\n<li>Reasoning and context: I tried LGBM, 2DCNN, MLP, and GCN for 2nd stage prediciton. However, all of them made CV worse. Also the weight optimization by Nelder-Mead method did score better, but not as well as Ridge-Regression or Netflix-method, so we did not use it for final submission.</li></ul></li>\n<li>Pretraiend from previous competition dataset<ul>\n<li>what you did: Before training, I pretrained model with data from a previous competition ('Feedback Prize - Evaluating Student Writing' and 'Feedback Prize - Predicting Effective Arguments')</li>\n<li>Result: not change or worse</li>\n<li>Reasoning and context: Scores were almost the same or slightly worse, so I stopped considering the cost of learning.</li></ul></li>\n<li>Changing the model depending on the length of the input text<ul>\n<li>Result: worse</li>\n<li>Reasoning and context: Deberta-v3 models can handle sentences of arbitrary length, while Roberta and Funnel were limited because of 512 and 1024. However, it was determined that even if the <code>max_len</code> limit was used, the model score would remain almost the same and could be used to seed the ensemble.</li></ul></li>\n</ul>\n<h2>Additional Context</h2>\n<h3>CV Strategy</h3>\n<p>I used <a href=\"https://www.kaggle.com/code/abhishek/multi-label-stratified-folds\" target=\"_blank\">abiheshark's cv strategy</a> with 4,5 folds various seed.</p>\n<h3>Training Config</h3>\n<ul>\n<li>Oprimizer: AdamW<ul>\n<li>weight decay: 0.01</li>\n<li>initial lr: 2e-5</li></ul></li>\n<li>Loss: SmoothL1Loss</li>\n<li>No dropout</li>\n</ul>\n<h3>My Score</h3>\n<p>My each model score is here, (Including models not in use)<br>\nI use mainly mean Pooling for model's head. (I tried LSTMhead and Attentionhead, but the score not changed)</p>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>CV</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>Deberta-v3-base</td>\n<td>0.4590</td>\n</tr>\n<tr>\n<td>Deberta-v3-large</td>\n<td>0.4519</td>\n</tr>\n<tr>\n<td>Roberta-large</td>\n<td>0.4583</td>\n</tr>\n<tr>\n<td>luke-large</td>\n<td>0.4590</td>\n</tr>\n<tr>\n<td>bigbird-roberta-large</td>\n<td>0.4643</td>\n</tr>\n<tr>\n<td>Deberta-large</td>\n<td>0.4615</td>\n</tr>\n<tr>\n<td>Deberta-large-mnli</td>\n<td>0.4629</td>\n</tr>\n<tr>\n<td>Funnel-large</td>\n<td>0.4686</td>\n</tr>\n<tr>\n<td>Funnel-large-base</td>\n<td>0.4644</td>\n</tr>\n<tr>\n<td>Bart-large</td>\n<td>0.4606</td>\n</tr>\n<tr>\n<td>GPT2-medium</td>\n<td>0.4658</td>\n</tr>\n<tr>\n<td>GPT2-large</td>\n<td>0.4882</td>\n</tr>\n<tr>\n<td>mpnet-base</td>\n<td>0.4952</td>\n</tr>\n<tr>\n<td>electra-large</td>\n<td>0.4731</td>\n</tr>\n<tr>\n<td>Deberta-v3-large (1st PL)</td>\n<td>0.4473</td>\n</tr>\n<tr>\n<td>Debreta-v3-large (2nd PL)</td>\n<td>0.4467</td>\n</tr>\n</tbody>\n</table>\n<p>My submissions CV score and Best Score CV are here,</p>\n<table>\n<thead>\n<tr>\n<th>submission</th>\n<th>CV</th>\n<th>My Inner Private Score Rank</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>20 models + Ridge</td>\n<td>0.4417</td>\n<td>2</td>\n</tr>\n<tr>\n<td>21 models + (0.5Ridge + 0.5Netflix)</td>\n<td>Ridge:0.4417, Netflix:0.4411</td>\n<td>3</td>\n</tr>\n<tr>\n<td>13 models + Ridge</td>\n<td>0.4421</td>\n<td>10</td>\n</tr>\n<tr>\n<td>16 models + Ridge</td>\n<td>0.4420</td>\n<td>1</td>\n</tr>\n</tbody>\n</table>\n<h3>Model Selection</h3>\n<p>I made a very large number of models, but had to choose a model due to dataset storage, Deberta-v3 had the highest score, so I chose about half of them, and the rest I chose by looking at the CV of the various models. So electra and mpnet are not in the final sub. But the best private scores were the models that included them :_(</p>\n<p>We also considered the need for a variety of Deberya-v3 models, so we used model weights from <a href=\"https://www.kaggle.com/code/kojimar/fb3-deberta-family-inference-9-28-updated\" target=\"_blank\">public Code</a>. I really appreciate.</p>\n<h2>Important Citations:</h2>\n<ul>\n<li>Code:<ul>\n<li><a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train</a></li>\n<li><a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference/notebook\" target=\"_blank\">https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference/notebook</a></li>\n<li><a href=\"https://www.kaggle.com/code/kojimar/fb3-deberta-family-inference-9-28-updated\" target=\"_blank\">https://www.kaggle.com/code/kojimar/fb3-deberta-family-inference-9-28-updated</a></li>\n<li><a href=\"https://www.kaggle.com/code/abhishek/multi-label-stratified-folds\" target=\"_blank\">https://www.kaggle.com/code/abhishek/multi-label-stratified-folds</a></li></ul></li>\n<li>Others:<ul>\n<li>netflix method: <a href=\"https://kaggler.readthedocs.io/en/latest/_modules/kaggler/ensemble/linear.html#netflix\" target=\"_blank\">https://kaggler.readthedocs.io/en/latest/_modules/kaggler/ensemble/linear.html#netflix</a></li></ul></li>\n</ul>\n<h2>Thanks and Acknowledgements:</h2>\n<p>Finally, I'd like to thank Kaggle, host for hosting such an iteresting competition and all who participated in lively discussions.<br>\nI hope my solution helps you in your future NLP competitions!</p>",
      "rawMarkdown": "# 11th Place Solution\n\nFirst of all, I'd like to thank Kaggle and host for hosting this competition.\nI still can't believe I got a solo gold.\n\n## Overview\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2648907%2F0d8799582d6ad0461e9ec995a3a1d3f7%2F2022-12-06%2013.47.10.png?generation=1670305587215632&alt=media)\n\nI ensembled 21 models (including models trained with Pseudo Labels). I used the Ridge-Regression and [Netflix method](https://kaggler.readthedocs.io/en/latest/_modules/kaggler/ensemble/linear.html#netflix) for ensembling.\n\nFor all of my transformer models I used the [this notebook](https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train) and [this notebook](https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference/notebook).\n\n\n## Code\n* Inference Notebook: https://www.kaggle.com/code/irrohas/final-many-deberta-ridge\n* Code (Training): I will upload later.\n\n## What Worked\n* [Hight Impact] Ridge-Regression and Netflix method per target for ensembling\n    * Result: about -0.009 \n    * Reasoning and context: Adding the predictions of various models (even if the each score was low) to the ensemble, not just Deberta-v3, improved the subsequent CV. So I used 10 other models besides deberta-v3. \n\n* [Low Impact] Pseudo Label\n    * Result: about -0.003\n    * Reasoning and context: I created a pseudo label with the 'Feedback Prize - Evaluating Student Writing'(FB1) competition data. However, there were too many pseudo-labels for the training data, and using all of them was likely to result in a large leakage, so we limited the pseudo-data that could be used in each fold.\n    I made up to 2 steps of learning with pseudo labels.\n\n## What Didn't Work\n* Stacking for 2nd stage prediction\n    * What you did: I tried LGBM, 2DCNN, MLP, and GCN for 2nd stage prediciton.\n    * Result: worse\n    * Reasoning and context: I tried LGBM, 2DCNN, MLP, and GCN for 2nd stage prediciton. However, all of them made CV worse. Also the weight optimization by Nelder-Mead method did score better, but not as well as Ridge-Regression or Netflix-method, so we did not use it for final submission.\n* Pretraiend from previous competition dataset\n    * what you did: Before training, I pretrained model with data from a previous competition ('Feedback Prize - Evaluating Student Writing' and 'Feedback Prize - Predicting Effective Arguments')\n    * Result: not change or worse\n    * Reasoning and context: Scores were almost the same or slightly worse, so I stopped considering the cost of learning.\n* Changing the model depending on the length of the input text\n    * Result: worse\n    * Reasoning and context: Deberta-v3 models can handle sentences of arbitrary length, while Roberta and Funnel were limited because of 512 and 1024. However, it was determined that even if the `max_len` limit was used, the model score would remain almost the same and could be used to seed the ensemble.\n\n## Additional Context\n### CV Strategy\nI used [abiheshark's cv strategy](https://www.kaggle.com/code/abhishek/multi-label-stratified-folds) with 4,5 folds various seed.\n\n### Training Config\n* Oprimizer: AdamW\n    * weight decay: 0.01\n    * initial lr: 2e-5\n* Loss: SmoothL1Loss\n* No dropout\n\n\n### My Score\nMy each model score is here, (Including models not in use)\nI use mainly mean Pooling for model's head. (I tried LSTMhead and Attentionhead, but the score not changed)\n\n|  model  |  CV  |\n| ---- | ---- |\n|  Deberta-v3-base |  0.4590  |\n|  Deberta-v3-large  |  0.4519  |\n|  Roberta-large  |  0.4583  |\n|  luke-large  |  0.4590 | \n| bigbird-roberta-large | 0.4643 |\n|  Deberta-large  |  0.4615  |\n|  Deberta-large-mnli | 0.4629 |\n|  Funnel-large   | 0.4686  |\n|  Funnel-large-base | 0.4644 | \n|  Bart-large     | 0.4606  |\n|  GPT2-medium     | 0.4658  |\n| GPT2-large | 0.4882 |\n|  mpnet-base  |  0.4952  | \n| electra-large | 0.4731 |\n|  Deberta-v3-large (1st PL) | 0.4473 |\n|  Debreta-v3-large (2nd PL) | 0.4467 |\n\n\nMy submissions CV score and Best Score CV are here,\n|  submission  |  CV  |  My Inner Private Score Rank |\n| ---- | ---- | ---- | \n| 20 models + Ridge | 0.4417 | 2 |\n| 21 models + (0.5Ridge + 0.5Netflix) | Ridge:0.4417, Netflix:0.4411 | 3 |\n| 13 models + Ridge | 0.4421  | 10 |\n| 16 models + Ridge | 0.4420 | 1 |\n\n\n### Model Selection\nI made a very large number of models, but had to choose a model due to dataset storage, Deberta-v3 had the highest score, so I chose about half of them, and the rest I chose by looking at the CV of the various models. So electra and mpnet are not in the final sub. But the best private scores were the models that included them :_(\n\nWe also considered the need for a variety of Deberya-v3 models, so we used model weights from [public Code](https://www.kaggle.com/code/kojimar/fb3-deberta-family-inference-9-28-updated). I really appreciate.\n\n## Important Citations:\n* Code:\n    * https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\n    * https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference/notebook\n    * https://www.kaggle.com/code/kojimar/fb3-deberta-family-inference-9-28-updated\n    * https://www.kaggle.com/code/abhishek/multi-label-stratified-folds\n* Others:\n    * netflix method: https://kaggler.readthedocs.io/en/latest/_modules/kaggler/ensemble/linear.html#netflix\n\n## Thanks and Acknowledgements:\nFinally, I'd like to thank Kaggle, host for hosting such an iteresting competition and all who participated in lively discussions.\nI hope my solution helps you in your future NLP competitions!\n",
      "votes": 30
    },
    {
      "id": 2049384,
      "postDate": "2022-11-30T05:18:57.510Z",
      "content": "<p>Thanks for sharing. How did you use Ridge? Is it stacking on transfomers predictions? I tried but my result is always worse than tuning weights.</p>",
      "rawMarkdown": "Thanks for sharing. How did you use Ridge? Is it stacking on transfomers predictions? I tried but my result is always worse than tuning weights.",
      "votes": 1,
      "replies": [
        {
          "id": 2049806,
          "postDate": "2022-11-30T10:58:02.407Z",
          "content": "<p>Thank you for question. <br>\nI used Ridge regression for decision of model weight for each target.<br>\nI tried stacking on transformers' prediction (6 targets) too, but that resulted in worse score.</p>",
          "rawMarkdown": "Thank you for question. \nI used Ridge regression for decision of model weight for each target.\nI tried stacking on transformers' prediction (6 targets) too, but that resulted in worse score.\n\n\n\n",
          "votes": 1
        },
        {
          "id": 2050073,
          "postDate": "2022-11-30T14:06:13.393Z",
          "content": "<p>Can you share more detail about that? I really want to learn since all I do was trying to stacking but It didn't work. Thanks in advance!</p>",
          "rawMarkdown": "Can you share more detail about that? I really want to learn since all I do was trying to stacking but It didn't work. Thanks in advance!"
        },
        {
          "id": 2050185,
          "postDate": "2022-11-30T15:20:13.583Z",
          "content": "<p>The following is my opinion. I'm sorry if I'm wrong. </p>\n<p>Stacking use the all targets for prediction. For example, I used the 'syntax' and 'vocabulary' predictions besides 'cohesion' predictions for 'cohesion' prediction in 2nd stage.</p>\n<p>On the other hand, ensemble use the only 'cohesion' predictions for 'cohesion' prediction in 2nd stage.<br>\nIn the other words, I simply determine the weight by regression.</p>\n<p>I think that the reason why stacking made the score worse is that other features (i.e. 'syntax' prediction or 'vocabulary' prediction in 1st stage) are noisy. (Or other reason is in)</p>",
          "rawMarkdown": "The following is my opinion. I'm sorry if I'm wrong. \n\nStacking use the all targets for prediction. For example, I used the 'syntax' and 'vocabulary' predictions besides 'cohesion' predictions for 'cohesion' prediction in 2nd stage.\n\nOn the other hand, ensemble use the only 'cohesion' predictions for 'cohesion' prediction in 2nd stage.\nIn the other words, I simply determine the weight by regression.\n\nI think that the reason why stacking made the score worse is that other features (i.e. 'syntax' prediction or 'vocabulary' prediction in 1st stage) are noisy. (Or other reason is in)",
          "votes": 1
        },
        {
          "id": 2050283,
          "postDate": "2022-11-30T16:18:57.430Z",
          "content": "<p>I understand your point now. I think your explanation makes sense. Thank you for sharing mate 👍</p>",
          "rawMarkdown": "I understand your point now. I think your explanation makes sense. Thank you for sharing mate 👍"
        }
      ]
    },
    {
      "id": 2049379,
      "postDate": "2022-11-30T05:11:23.677Z",
      "content": "<p>I also tried linear lr and gbdt all worse. And optuna seems overfit cv not improve PB comparing to using manual set weight per target.<br>\nSo Ridge-Regressio with -0.009 is so cool!  Thanks for sharing!</p>",
      "rawMarkdown": "I also tried linear lr and gbdt all worse. And optuna seems overfit cv not improve PB comparing to using manual set weight per target.\nSo Ridge-Regressio with -0.009 is so cool!  Thanks for sharing!",
      "votes": 2
    },
    {
      "id": 2050176,
      "postDate": "2022-11-30T15:16:19.283Z",
      "content": "<p>solo gold congratulations！<br>\nLet me ask two questions about pseudo-labeling.</p>\n<ol>\n<li>how many data number of events did you use for each fold?</li>\n<li>When you say \"2 steps of learning with pseudo labels.\", do you mean that you trained with pseudo labels for 2 epochs and then used train data from FB3?</li>\n</ol>",
      "rawMarkdown": "solo gold congratulations！\nLet me ask two questions about pseudo-labeling.\n1. how many data number of events did you use for each fold?\n2. When you say \"2 steps of learning with pseudo labels.\", do you mean that you trained with pseudo labels for 2 epochs and then used train data from FB3?",
      "replies": [
        {
          "id": 2050209,
          "postDate": "2022-11-30T15:31:22.847Z",
          "content": "<p>Thank you for question.</p>\n<blockquote>\n  <p>how many data number of events did you use for each fold?</p>\n</blockquote>\n<p>About 3,000. In 4 folds training, I have divided the pseudo label data into 4 parts and I added each of them to training data in each fold.</p>\n<blockquote>\n  <p>When you say \"2 steps of learning with pseudo labels.\", do you mean that you trained with pseudo labels for 2 epochs and then used train data from FB3?</p>\n</blockquote>\n<p>No. I trained with both pseudo label data and FB3 data in all epochs.<br>\n\"2 steps of learning with pseudo labels.\" This means multi-step pseudo labeling. In other words, I created new data(= 2nd pseudo label data) by using the model trained on the 1st data.</p>",
          "rawMarkdown": "Thank you for question.\n\n> how many data number of events did you use for each fold?\n\nAbout 3,000. In 4 folds training, I have divided the pseudo label data into 4 parts and I added each of them to training data in each fold.\n\n> When you say \"2 steps of learning with pseudo labels.\", do you mean that you trained with pseudo labels for 2 epochs and then used train data from FB3?\n\nNo. I trained with both pseudo label data and FB3 data in all epochs.\n\"2 steps of learning with pseudo labels.\" This means multi-step pseudo labeling. In other words, I created new data(= 2nd pseudo label data) by using the model trained on the 1st data.",
          "votes": 1
        },
        {
          "id": 2050233,
          "postDate": "2022-11-30T15:44:52.560Z",
          "content": "<p>I see, now I understand！<br>\nThank you for your prompt response.</p>",
          "rawMarkdown": "I see, now I understand！\nThank you for your prompt response."
        }
      ]
    },
    {
      "id": 2049432,
      "postDate": "2022-11-30T06:35:00.110Z",
      "content": "<p>great, thanks for sharing👌👍</p>",
      "rawMarkdown": "great, thanks for sharing👌👍"
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T19:58:13Z",
  "competition_id": 38321,
  "competition_slug": "feedback-prize-english-language-learning",
  "team_id": 9123159,
  "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/masaya-11th-place-solution"
}