{
  "id": 371203,
  "title": "42th Place Solution",
  "content": "<p>First of all, thank you for hosting this competition.<br>\nI learned a lot about the NLP of this competition.<br>\nCongrats to the winners!</p>\n<h1>Overview</h1>\n<ul>\n<li>I use 11 models(4 fold model x 4 and 10 fold model x 7).<ul>\n<li>First, I use LGBM(with optuna) for 4fold model.<br>\n※ I didn't use 10 fold model because the experiment was not completed in time for the competition deadline..</li>\n<li>Second,  I did weight ensemble. </li></ul></li>\n<li>cv=0.4426811/PublicLB=0.435475/PrivateLB=0.435906<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8119383%2Ff0b0cf8c5f4020e199bdfad6edd33ed8%2Fmy_submission.png?generation=1670509564450962&amp;alt=media\" alt=\"\"></li>\n<li>10 fold model is open model(<a href=\"https://www.kaggle.com/code/jingwora1/fb3-deberta-family-inference-weight-tune\" target=\"_blank\">link1</a>,<a href=\"https://www.kaggle.com/code/kojimar/fb3-single-pytorch-model-inference\" target=\"_blank\">link2</a>). So I share only 4 fold model detail of cv score. </li>\n</ul>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>CV score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>(*1,4,5)deberta-v3-base (pseud label fb1,2)</td>\n<td>0.4528044</td>\n</tr>\n<tr>\n<td>(*4,5)deberta-v3-large</td>\n<td>0.4568590</td>\n</tr>\n<tr>\n<td>(*2,4,5)deberta-v3-base (topics)</td>\n<td>0.4537212</td>\n</tr>\n<tr>\n<td>(*3,4,5)deberta-v3-base (original loss)</td>\n<td>0.4564600</td>\n</tr>\n</tbody>\n</table>\n<p>※ (*1,4,5) use 「Pseud Labeling」,「Concat last Four hidden」and「Dropout」.</p>\n<h1>What Worked</h1>\n<ul>\n<li>Pseudo Labeling(*1)<ul>\n<li>I use all data of fb1 and fb2.</li>\n<li>I use MultilabelStratifiedKFold to fold datasets.</li></ul></li>\n<li>Add topic(*2)<ul>\n<li>I refer <a href=\"https://www.kaggle.com/code/jdoesv/take2-feedback-essays-to-prompts\" target=\"_blank\">this code</a> and predict topic.</li>\n<li>I add predicted topic to text. (ex. [1_students_online_school_classes]I think that~)</li></ul></li>\n<li>oliginal loss(soft)(*3)<ul>\n<li>Much of the data for this competition was concentrated around 3. So I changed the loss to be more robust to outlier value.</li>\n<li>sample is below.</li></ul></li>\n</ul>\n<pre><code>self.mse = nn.MSELoss(reduction='none')\nl = self.mse(y_pred, y_true) + eps # eps=1e-9\nloss = torch.mul(torch.pow(l,alpha),torch.sqrt(l))\n# soft→alpha=1, hard→alpha=2\n</code></pre>\n<ul>\n<li>Concat last Four hidden(*4)<ul>\n<li>MLP model include important information in  last Four hidden layers.</li>\n<li>I refer <a href=\"https://www.kaggle.com/code/rhtsingh/utilizing-transformer-representations-efficiently\" target=\"_blank\">this code</a> </li></ul></li>\n<li>Dropout(*5)<ul>\n<li>Dropouts can reduce overlearning.</li>\n<li>sample is below.</li></ul></li>\n</ul>\n<pre><code>self.dropouts = nn.ModuleList([nn.Dropout(0.2) for _ in range(5)])\noutput = sum([self.fc(dropout(feature)) for dropout in self.dropouts])/5\n</code></pre>\n<ul>\n<li>LGBM<ul>\n<li>I make 6LGBM model to predict 「cohesion」, 「syntax」, 「vocabulary」, 「phraseology」, 「grammar」, and 「conventions」.</li>\n<li>I use 2stage stacking.</li></ul></li>\n</ul>\n<h1>What Didn’t Work</h1>\n<ul>\n<li>MLM<ul>\n<li>I can't get good LB and CV score</li>\n<li>I think MLM is useful when datasets is small. But This competition can use fb1,fb2 and fb3.So I think MLM does not need.</li></ul></li>\n<li>Focal loss(hard)<ul>\n<li>LB score dropped. So I didn't use.</li></ul></li>\n</ul>\n<h1>Important ciations</h1>\n<ul>\n<li>FB3 / Deberta-v3-base baseline [train] <a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">link</a></li>\n<li>FB3 / Deberta-v3-base baseline [inference] <a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference/notebook\" target=\"_blank\">link</a></li>\n<li>FB3 Deberta Family Inference [weight tune] <a href=\"https://www.kaggle.com/code/jingwora1/fb3-deberta-family-inference-weight-tune\" target=\"_blank\">link</a></li>\n<li>FB3 single pytorch model [inference] <a href=\"https://www.kaggle.com/code/kojimar/fb3-single-pytorch-model-inference\" target=\"_blank\">link</a></li>\n</ul>\n<h1>Thanks and Acknowledgements</h1>\n<p>This was almost my first time to participate in an MLP competition.<br>\nI think this competition was perfect for someone with no MLP experience because I learned a lot from the Discussion. <br>\nWe were also able to reaffirm the importance of trust CV. <br>\nAgain, thank you very much for a very enjoyable competition! </p>",
  "comments": [
    {
      "id": 2059188,
      "author_name": "IMvision12",
      "author_url": "",
      "post_date": "2022-12-08T14:56:04.233000",
      "content": "<p><a href=\"https://www.kaggle.com/tsuzuk1\" target=\"_blank\">@tsuzuk1</a> you have used Pseudo Labeling, I am a little confused about it what does and how it helps to improve accuracy, i know little about it that we train a model and do predictions on the test dataset and again we train model with training and testing dataset…</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2060168,
          "author_name": "tsuzuk1",
          "author_url": "",
          "post_date": "2022-12-09T15:51:36.603000",
          "content": "<p><a href=\"https://www.kaggle.com/imvision12\" target=\"_blank\">@imvision12</a> Thank you for your comment! I refer to this <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/352001\" target=\"_blank\">discussion</a>. Please also note the leaks in this competition, as there is some overlap between past data(fb1) and fb3 data. I can get below score in deberta-v3-base.<br>\n[without pseud label] PublicLB=0.446387/PrivateLB=0.445974<br>\n[with pseud label] PublicLB=0.440058/PrivateLB=0.440377</p>",
          "votes": 1,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 2,
  "messages": [
    {
      "id": 2059172,
      "postDate": "2022-12-08T14:29:20.993Z",
      "content": "<p>First of all, thank you for hosting this competition.<br>\nI learned a lot about the NLP of this competition.<br>\nCongrats to the winners!</p>\n<h1>Overview</h1>\n<ul>\n<li>I use 11 models(4 fold model x 4 and 10 fold model x 7).<ul>\n<li>First, I use LGBM(with optuna) for 4fold model.<br>\n※ I didn't use 10 fold model because the experiment was not completed in time for the competition deadline..</li>\n<li>Second,  I did weight ensemble. </li></ul></li>\n<li>cv=0.4426811/PublicLB=0.435475/PrivateLB=0.435906<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8119383%2Ff0b0cf8c5f4020e199bdfad6edd33ed8%2Fmy_submission.png?generation=1670509564450962&amp;alt=media\" alt=\"\"></li>\n<li>10 fold model is open model(<a href=\"https://www.kaggle.com/code/jingwora1/fb3-deberta-family-inference-weight-tune\" target=\"_blank\">link1</a>,<a href=\"https://www.kaggle.com/code/kojimar/fb3-single-pytorch-model-inference\" target=\"_blank\">link2</a>). So I share only 4 fold model detail of cv score. </li>\n</ul>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>CV score</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>(*1,4,5)deberta-v3-base (pseud label fb1,2)</td>\n<td>0.4528044</td>\n</tr>\n<tr>\n<td>(*4,5)deberta-v3-large</td>\n<td>0.4568590</td>\n</tr>\n<tr>\n<td>(*2,4,5)deberta-v3-base (topics)</td>\n<td>0.4537212</td>\n</tr>\n<tr>\n<td>(*3,4,5)deberta-v3-base (original loss)</td>\n<td>0.4564600</td>\n</tr>\n</tbody>\n</table>\n<p>※ (*1,4,5) use 「Pseud Labeling」,「Concat last Four hidden」and「Dropout」.</p>\n<h1>What Worked</h1>\n<ul>\n<li>Pseudo Labeling(*1)<ul>\n<li>I use all data of fb1 and fb2.</li>\n<li>I use MultilabelStratifiedKFold to fold datasets.</li></ul></li>\n<li>Add topic(*2)<ul>\n<li>I refer <a href=\"https://www.kaggle.com/code/jdoesv/take2-feedback-essays-to-prompts\" target=\"_blank\">this code</a> and predict topic.</li>\n<li>I add predicted topic to text. (ex. [1_students_online_school_classes]I think that~)</li></ul></li>\n<li>oliginal loss(soft)(*3)<ul>\n<li>Much of the data for this competition was concentrated around 3. So I changed the loss to be more robust to outlier value.</li>\n<li>sample is below.</li></ul></li>\n</ul>\n<pre><code>self.mse = nn.MSELoss(reduction='none')\nl = self.mse(y_pred, y_true) + eps # eps=1e-9\nloss = torch.mul(torch.pow(l,alpha),torch.sqrt(l))\n# soft→alpha=1, hard→alpha=2\n</code></pre>\n<ul>\n<li>Concat last Four hidden(*4)<ul>\n<li>MLP model include important information in  last Four hidden layers.</li>\n<li>I refer <a href=\"https://www.kaggle.com/code/rhtsingh/utilizing-transformer-representations-efficiently\" target=\"_blank\">this code</a> </li></ul></li>\n<li>Dropout(*5)<ul>\n<li>Dropouts can reduce overlearning.</li>\n<li>sample is below.</li></ul></li>\n</ul>\n<pre><code>self.dropouts = nn.ModuleList([nn.Dropout(0.2) for _ in range(5)])\noutput = sum([self.fc(dropout(feature)) for dropout in self.dropouts])/5\n</code></pre>\n<ul>\n<li>LGBM<ul>\n<li>I make 6LGBM model to predict 「cohesion」, 「syntax」, 「vocabulary」, 「phraseology」, 「grammar」, and 「conventions」.</li>\n<li>I use 2stage stacking.</li></ul></li>\n</ul>\n<h1>What Didn’t Work</h1>\n<ul>\n<li>MLM<ul>\n<li>I can't get good LB and CV score</li>\n<li>I think MLM is useful when datasets is small. But This competition can use fb1,fb2 and fb3.So I think MLM does not need.</li></ul></li>\n<li>Focal loss(hard)<ul>\n<li>LB score dropped. So I didn't use.</li></ul></li>\n</ul>\n<h1>Important ciations</h1>\n<ul>\n<li>FB3 / Deberta-v3-base baseline [train] <a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">link</a></li>\n<li>FB3 / Deberta-v3-base baseline [inference] <a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference/notebook\" target=\"_blank\">link</a></li>\n<li>FB3 Deberta Family Inference [weight tune] <a href=\"https://www.kaggle.com/code/jingwora1/fb3-deberta-family-inference-weight-tune\" target=\"_blank\">link</a></li>\n<li>FB3 single pytorch model [inference] <a href=\"https://www.kaggle.com/code/kojimar/fb3-single-pytorch-model-inference\" target=\"_blank\">link</a></li>\n</ul>\n<h1>Thanks and Acknowledgements</h1>\n<p>This was almost my first time to participate in an MLP competition.<br>\nI think this competition was perfect for someone with no MLP experience because I learned a lot from the Discussion. <br>\nWe were also able to reaffirm the importance of trust CV. <br>\nAgain, thank you very much for a very enjoyable competition! </p>",
      "rawMarkdown": "First of all, thank you for hosting this competition.\nI learned a lot about the NLP of this competition.\nCongrats to the winners!\n\n# Overview\n- I use 11 models(4 fold model x 4 and 10 fold model x 7).\n  - First, I use LGBM(with optuna) for 4fold model.\n ※ I didn't use 10 fold model because the experiment was not completed in time for the competition deadline..\n  - Second,  I did weight ensemble. \n- cv=0.4426811/PublicLB=0.435475/PrivateLB=0.435906\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8119383%2Ff0b0cf8c5f4020e199bdfad6edd33ed8%2Fmy_submission.png?generation=1670509564450962&alt=media)\n- 10 fold model is open model([link1](https://www.kaggle.com/code/jingwora1/fb3-deberta-family-inference-weight-tune),[link2](https://www.kaggle.com/code/kojimar/fb3-single-pytorch-model-inference)). So I share only 4 fold model detail of cv score. \n\n| model | CV score |\n| --- | --- |\n| (*1,4,5)deberta-v3-base (pseud label fb1,2) | 0.4528044 |\n| (*4,5)deberta-v3-large | 0.4568590 |\n| (*2,4,5)deberta-v3-base (topics) | 0.4537212 |\n| (*3,4,5)deberta-v3-base (original loss) | 0.4564600 |\n\n※ (*1,4,5) use 「Pseud Labeling」,「Concat last Four hidden」and「Dropout」.\n\n# What Worked\n- Pseudo Labeling(*1)\n   - I use all data of fb1 and fb2.\n   - I use MultilabelStratifiedKFold to fold datasets.\n- Add topic(*2)\n   -  I refer [this code](https://www.kaggle.com/code/jdoesv/take2-feedback-essays-to-prompts) and predict topic.\n   -  I add predicted topic to text. (ex. [1_students_online_school_classes]I think that~)\n- oliginal loss(soft)(*3)\n   -  Much of the data for this competition was concentrated around 3. So I changed the loss to be more robust to outlier value.\n   -  sample is below.\n```\nself.mse = nn.MSELoss(reduction='none')\nl = self.mse(y_pred, y_true) + eps # eps=1e-9\nloss = torch.mul(torch.pow(l,alpha),torch.sqrt(l))\n# soft→alpha=1, hard→alpha=2\n```\n- Concat last Four hidden(*4)\n    - MLP model include important information in  last Four hidden layers.\n    - I refer [this code](https://www.kaggle.com/code/rhtsingh/utilizing-transformer-representations-efficiently) \n-  Dropout(*5)\n    - Dropouts can reduce overlearning.\n    -  sample is below.\n```\nself.dropouts = nn.ModuleList([nn.Dropout(0.2) for _ in range(5)])\noutput = sum([self.fc(dropout(feature)) for dropout in self.dropouts])/5\n```\n- LGBM\n    - I make 6LGBM model to predict 「cohesion」, 「syntax」, 「vocabulary」, 「phraseology」, 「grammar」, and 「conventions」.\n    - I use 2stage stacking.\n\n# What Didn’t Work\n- MLM\n    - I can't get good LB and CV score\n    - I think MLM is useful when datasets is small. But This competition can use fb1,fb2 and fb3.So I think MLM does not need.\n- Focal loss(hard)\n    - LB score dropped. So I didn't use.\n\n# Important ciations\n- FB3 / Deberta-v3-base baseline [train] [link](https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train)\n- FB3 / Deberta-v3-base baseline [inference] [link](https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-inference/notebook)\n- FB3 Deberta Family Inference [weight tune] [link](https://www.kaggle.com/code/jingwora1/fb3-deberta-family-inference-weight-tune)\n- FB3 single pytorch model [inference] [link](https://www.kaggle.com/code/kojimar/fb3-single-pytorch-model-inference)\n\n# Thanks and Acknowledgements\nThis was almost my first time to participate in an MLP competition.\nI think this competition was perfect for someone with no MLP experience because I learned a lot from the Discussion. \nWe were also able to reaffirm the importance of trust CV. \nAgain, thank you very much for a very enjoyable competition! ",
      "votes": 5
    },
    {
      "id": 2059188,
      "postDate": "2022-12-08T14:56:04.233Z",
      "content": "<p><a href=\"https://www.kaggle.com/tsuzuk1\" target=\"_blank\">@tsuzuk1</a> you have used Pseudo Labeling, I am a little confused about it what does and how it helps to improve accuracy, i know little about it that we train a model and do predictions on the test dataset and again we train model with training and testing dataset…</p>",
      "rawMarkdown": "@tsuzuk1 you have used Pseudo Labeling, I am a little confused about it what does and how it helps to improve accuracy, i know little about it that we train a model and do predictions on the test dataset and again we train model with training and testing dataset...",
      "votes": 1,
      "replies": [
        {
          "id": 2060168,
          "postDate": "2022-12-09T15:51:36.603Z",
          "content": "<p><a href=\"https://www.kaggle.com/imvision12\" target=\"_blank\">@imvision12</a> Thank you for your comment! I refer to this <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/352001\" target=\"_blank\">discussion</a>. Please also note the leaks in this competition, as there is some overlap between past data(fb1) and fb3 data. I can get below score in deberta-v3-base.<br>\n[without pseud label] PublicLB=0.446387/PrivateLB=0.445974<br>\n[with pseud label] PublicLB=0.440058/PrivateLB=0.440377</p>",
          "rawMarkdown": "@imvision12 Thank you for your comment! I refer to this [discussion](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/352001). Please also note the leaks in this competition, as there is some overlap between past data(fb1) and fb3 data. I can get below score in deberta-v3-base.\n[without pseud label] PublicLB=0.446387/PrivateLB=0.445974\n[with pseud label] PublicLB=0.440058/PrivateLB=0.440377",
          "votes": 1
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T20:06:12Z",
  "competition_id": 38321,
  "competition_slug": "feedback-prize-english-language-learning",
  "team_id": 9302222,
  "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/tsuzuk1-42th-place-solution"
}