{
  "id": 369567,
  "title": "6th place solution",
  "content": "<h2>Overview</h2>\n<p>I noticed that the correlation between 6 classes is quite high in my baseline models.  So I tried to relax it and created models for each class, which I would call  “separate models” here. I also trained single models (no-separate models) but with separate heads. In the end, my solution is a class-wise weighted average of 4 separate models and 5 single models, where I used the gp_minimise function from scikit-optimize to determine the weights.  <br>\nNote: a 5-fold separate model uses 5 x 6 = 30 models and it took around 1h for GPU submission.</p>\n<p>selected submission: CV=0.4422, PublicLB=0.436362, PrivateLB=0.434121  <br>\n(best CV was my best PrivateLB)</p>\n<table>\n<thead>\n<tr>\n<th>exp</th>\n<th>type</th>\n<th>model</th>\n<th>fold</th>\n<th>CV</th>\n<th>Public LB</th>\n<th>Private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>14_v1_01</td>\n<td>separate</td>\n<td>deberta-v3-base</td>\n<td>5</td>\n<td>0.4524</td>\n<td>0.4371x</td>\n<td>0.4380x</td>\n</tr>\n<tr>\n<td>14_v1_07</td>\n<td>separate</td>\n<td>deberta-v3-base</td>\n<td>10</td>\n<td>0.4512</td>\n<td>0.4374x</td>\n<td>0.4370x</td>\n</tr>\n<tr>\n<td>14_v1_10</td>\n<td>separate</td>\n<td>deberta-v3-large</td>\n<td>5</td>\n<td>0.4518</td>\n<td>0.4411x</td>\n<td>0.4370x</td>\n</tr>\n<tr>\n<td>14_v1_12</td>\n<td>separate</td>\n<td>deberta-large</td>\n<td>5</td>\n<td>0.4534</td>\n<td>0.4409x</td>\n<td>0.4388x</td>\n</tr>\n<tr>\n<td>29_v1_02</td>\n<td>single</td>\n<td>deberta-v3-base</td>\n<td>5</td>\n<td>0.4557</td>\n<td>0.4424x</td>\n<td>0.4416x</td>\n</tr>\n<tr>\n<td>29_v1_04</td>\n<td>single</td>\n<td>deberta-v3-large</td>\n<td>5</td>\n<td>0.4531</td>\n<td>-</td>\n<td>-</td>\n</tr>\n<tr>\n<td>29_v1_11</td>\n<td>single</td>\n<td>deberta-v3-large-squad2</td>\n<td>5</td>\n<td>0.4526</td>\n<td>-</td>\n<td>-</td>\n</tr>\n<tr>\n<td>29_v1_14</td>\n<td>single</td>\n<td>deberta-v2-xlarge</td>\n<td>5</td>\n<td>0.4569</td>\n<td>-</td>\n<td>-</td>\n</tr>\n<tr>\n<td>29_v1_15</td>\n<td>single</td>\n<td>deberta-xlarge</td>\n<td>5</td>\n<td>0.4552</td>\n<td>-</td>\n<td>-</td>\n</tr>\n</tbody>\n</table>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1565399%2F0c04d6e8103e6af95ebf083575b09f3a%2Ffb3_6th_place_solution.png?generation=1669817196747414&amp;alt=media\" alt=\"\"></p>\n<p>training code: <a href=\"https://github.com/tikutikutiku/kaggle-feedback-prize-english-language-learning\" target=\"_blank\">https://github.com/tikutikutiku/kaggle-feedback-prize-english-language-learning</a><br>\ninference code: <a href=\"https://www.kaggle.com/code/tikutiku/feedback3-inference2\" target=\"_blank\">https://www.kaggle.com/code/tikutiku/feedback3-inference2</a></p>\n<h2>What Worked</h2>\n<p>[Worked] separate model </p>\n<ul>\n<li>What you did:  created models for each class</li>\n<li>Result: CV+0.003x, Public LB+0.005x, Private LB+0.003x</li>\n</ul>\n<h2>What Didn’t Work</h2>\n<p>[Didn’t Work] awp <br>\n [Didn’t Work] pre-training with fb1 data <br>\n [Didn’t Work] detector pre-training with fb1 data<br>\n [Didn’t Work] pseudo-label with fb1 data : models trained with pseudo-labels did not contribute to ensemble CV  <br>\n [Didn’t Work] random masking augmentation <br>\n [Didn’t Work] lgb/catboost stacking</p>\n<h2>Additional Context :</h2>\n<p>Since the separate model was too slow for submission, I gave up the CPU track . </p>\n<h2>Thanks and Acknowledgements:</h2>\n<p>Thanks to kaggle and the organizers for having this 3rd feedback competition! I enjoyed all of the three feedback competitions and had a great time with the team and solo play :) </p>\n<h2>Team Members:</h2>\n<p><a href=\"https://www.kaggle.com/tikutiku\" target=\"_blank\">@tikutiku</a> </p>",
  "comments": [
    {
      "id": 2050538,
      "author_name": "CPMP",
      "author_url": "",
      "post_date": "2022-11-30T20:21:56.517000",
      "content": "<p>Thanks for sharing that separate models were useful. I had it on my todo list but ran out of time. Congrats on the result!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2050205,
      "author_name": "Sergey Bochenkov",
      "author_url": "",
      "post_date": "2022-11-30T15:29:52.430000",
      "content": "<p>Great solution! I wanted to try training separate models for each class, but didn't have enough time. :)</p>",
      "votes": 1,
      "replies": []
    }
  ],
  "comment_count": 2,
  "messages": [
    {
      "id": 2050079,
      "postDate": "2022-11-30T14:10:43.697Z",
      "content": "<h2>Overview</h2>\n<p>I noticed that the correlation between 6 classes is quite high in my baseline models.  So I tried to relax it and created models for each class, which I would call  “separate models” here. I also trained single models (no-separate models) but with separate heads. In the end, my solution is a class-wise weighted average of 4 separate models and 5 single models, where I used the gp_minimise function from scikit-optimize to determine the weights.  <br>\nNote: a 5-fold separate model uses 5 x 6 = 30 models and it took around 1h for GPU submission.</p>\n<p>selected submission: CV=0.4422, PublicLB=0.436362, PrivateLB=0.434121  <br>\n(best CV was my best PrivateLB)</p>\n<table>\n<thead>\n<tr>\n<th>exp</th>\n<th>type</th>\n<th>model</th>\n<th>fold</th>\n<th>CV</th>\n<th>Public LB</th>\n<th>Private LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>14_v1_01</td>\n<td>separate</td>\n<td>deberta-v3-base</td>\n<td>5</td>\n<td>0.4524</td>\n<td>0.4371x</td>\n<td>0.4380x</td>\n</tr>\n<tr>\n<td>14_v1_07</td>\n<td>separate</td>\n<td>deberta-v3-base</td>\n<td>10</td>\n<td>0.4512</td>\n<td>0.4374x</td>\n<td>0.4370x</td>\n</tr>\n<tr>\n<td>14_v1_10</td>\n<td>separate</td>\n<td>deberta-v3-large</td>\n<td>5</td>\n<td>0.4518</td>\n<td>0.4411x</td>\n<td>0.4370x</td>\n</tr>\n<tr>\n<td>14_v1_12</td>\n<td>separate</td>\n<td>deberta-large</td>\n<td>5</td>\n<td>0.4534</td>\n<td>0.4409x</td>\n<td>0.4388x</td>\n</tr>\n<tr>\n<td>29_v1_02</td>\n<td>single</td>\n<td>deberta-v3-base</td>\n<td>5</td>\n<td>0.4557</td>\n<td>0.4424x</td>\n<td>0.4416x</td>\n</tr>\n<tr>\n<td>29_v1_04</td>\n<td>single</td>\n<td>deberta-v3-large</td>\n<td>5</td>\n<td>0.4531</td>\n<td>-</td>\n<td>-</td>\n</tr>\n<tr>\n<td>29_v1_11</td>\n<td>single</td>\n<td>deberta-v3-large-squad2</td>\n<td>5</td>\n<td>0.4526</td>\n<td>-</td>\n<td>-</td>\n</tr>\n<tr>\n<td>29_v1_14</td>\n<td>single</td>\n<td>deberta-v2-xlarge</td>\n<td>5</td>\n<td>0.4569</td>\n<td>-</td>\n<td>-</td>\n</tr>\n<tr>\n<td>29_v1_15</td>\n<td>single</td>\n<td>deberta-xlarge</td>\n<td>5</td>\n<td>0.4552</td>\n<td>-</td>\n<td>-</td>\n</tr>\n</tbody>\n</table>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1565399%2F0c04d6e8103e6af95ebf083575b09f3a%2Ffb3_6th_place_solution.png?generation=1669817196747414&amp;alt=media\" alt=\"\"></p>\n<p>training code: <a href=\"https://github.com/tikutikutiku/kaggle-feedback-prize-english-language-learning\" target=\"_blank\">https://github.com/tikutikutiku/kaggle-feedback-prize-english-language-learning</a><br>\ninference code: <a href=\"https://www.kaggle.com/code/tikutiku/feedback3-inference2\" target=\"_blank\">https://www.kaggle.com/code/tikutiku/feedback3-inference2</a></p>\n<h2>What Worked</h2>\n<p>[Worked] separate model </p>\n<ul>\n<li>What you did:  created models for each class</li>\n<li>Result: CV+0.003x, Public LB+0.005x, Private LB+0.003x</li>\n</ul>\n<h2>What Didn’t Work</h2>\n<p>[Didn’t Work] awp <br>\n [Didn’t Work] pre-training with fb1 data <br>\n [Didn’t Work] detector pre-training with fb1 data<br>\n [Didn’t Work] pseudo-label with fb1 data : models trained with pseudo-labels did not contribute to ensemble CV  <br>\n [Didn’t Work] random masking augmentation <br>\n [Didn’t Work] lgb/catboost stacking</p>\n<h2>Additional Context :</h2>\n<p>Since the separate model was too slow for submission, I gave up the CPU track . </p>\n<h2>Thanks and Acknowledgements:</h2>\n<p>Thanks to kaggle and the organizers for having this 3rd feedback competition! I enjoyed all of the three feedback competitions and had a great time with the team and solo play :) </p>\n<h2>Team Members:</h2>\n<p><a href=\"https://www.kaggle.com/tikutiku\" target=\"_blank\">@tikutiku</a> </p>",
      "rawMarkdown": "## Overview\nI noticed that the correlation between 6 classes is quite high in my baseline models.  So I tried to relax it and created models for each class, which I would call  “separate models” here. I also trained single models (no-separate models) but with separate heads. In the end, my solution is a class-wise weighted average of 4 separate models and 5 single models, where I used the gp_minimise function from scikit-optimize to determine the weights.  \nNote: a 5-fold separate model uses 5 x 6 = 30 models and it took around 1h for GPU submission.\n\nselected submission: CV=0.4422, PublicLB=0.436362, PrivateLB=0.434121  \n(best CV was my best PrivateLB)\n\n| exp | type | model | fold | CV | Public LB | Private LB |\n| --- | --- | --- | --- | --- | --- | --- |\n| 14_v1_01 | separate | deberta-v3-base | 5 | 0.4524 | 0.4371x | 0.4380x |\n| 14_v1_07 | separate | deberta-v3-base | 10 | 0.4512 | 0.4374x | 0.4370x |\n| 14_v1_10 | separate | deberta-v3-large | 5 | 0.4518 | 0.4411x | 0.4370x |\n| 14_v1_12 | separate | deberta-large | 5 | 0.4534 | 0.4409x | 0.4388x |\n| 29_v1_02 | single | deberta-v3-base | 5 | 0.4557 | 0.4424x | 0.4416x |\n| 29_v1_04 | single | deberta-v3-large | 5 | 0.4531 | - | - |\n| 29_v1_11 | single | deberta-v3-large-squad2 | 5 | 0.4526 | - | - |\n| 29_v1_14 | single | deberta-v2-xlarge | 5 | 0.4569 | - | - |\n| 29_v1_15 | single | deberta-xlarge | 5 | 0.4552 | - | - |\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1565399%2F0c04d6e8103e6af95ebf083575b09f3a%2Ffb3_6th_place_solution.png?generation=1669817196747414&alt=media)\n\n\ntraining code: https://github.com/tikutikutiku/kaggle-feedback-prize-english-language-learning\ninference code: https://www.kaggle.com/code/tikutiku/feedback3-inference2\n\n## What Worked \n[Worked] separate model \n- What you did:  created models for each class\n- Result: CV+0.003x, Public LB+0.005x, Private LB+0.003x\n\n## What Didn’t Work \n [Didn’t Work] awp \n [Didn’t Work] pre-training with fb1 data \n [Didn’t Work] detector pre-training with fb1 data\n [Didn’t Work] pseudo-label with fb1 data : models trained with pseudo-labels did not contribute to ensemble CV  \n [Didn’t Work] random masking augmentation \n [Didn’t Work] lgb/catboost stacking\n\n## Additional Context : \nSince the separate model was too slow for submission, I gave up the CPU track . \n\n## Thanks and Acknowledgements: \nThanks to kaggle and the organizers for having this 3rd feedback competition! I enjoyed all of the three feedback competitions and had a great time with the team and solo play :) \n\n## Team Members: \n@tikutiku ",
      "votes": 46
    },
    {
      "id": 2050538,
      "postDate": "2022-11-30T20:21:56.517Z",
      "content": "<p>Thanks for sharing that separate models were useful. I had it on my todo list but ran out of time. Congrats on the result!</p>",
      "rawMarkdown": "Thanks for sharing that separate models were useful. I had it on my todo list but ran out of time. Congrats on the result!",
      "votes": 1
    },
    {
      "id": 2050205,
      "postDate": "2022-11-30T15:29:52.430Z",
      "content": "<p>Great solution! I wanted to try training separate models for each class, but didn't have enough time. :)</p>",
      "rawMarkdown": "Great solution! I wanted to try training separate models for each class, but didn't have enough time. :)",
      "votes": 1
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T20:01:02Z",
  "competition_id": 38321,
  "competition_slug": "feedback-prize-english-language-learning",
  "team_id": 9091264,
  "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/tom-6th-place-solution"
}