{
  "id": 369525,
  "title": "26th place solution [lucky shake up 1117th → 26th]",
  "content": "<p>Thanks to all the participants and hosts<br>\nI was lucky to win a silver medal with big shake up.</p>\n<h1>Overview</h1>\n<p>finally cv is 0.44582 (public/private 0.439025/0.435032) (deberta-v3-large (x2) ensemble)<br>\nMLM on FB2 data (mask prob = 0.4)<br>\nCV improved slightly but LB worsened. <br>\nHowever, adding it to the ensemble seems to have boosted the score.</p>\n<p>single model cv</p>\n<ul>\n<li>0.4490 (w/o MLM) (public/private 0.439145/0.436422)</li>\n<li>0.4488 (w MLM) (public/private 0.442191/0.437079)</li>\n</ul>\n<p>In my case, LB did not correlate well with the CV improvement.<br>\nLB did not change much when I changed from a single model to an ensemble model<br>\n(This doesn't seem to be the case with the top solutions, I don't know why)<br>\nI stopped working on the competition because LB did not improve. <br>\nHowever, TrustCV seems to have been right.</p>\n<h2>What Worked</h2>\n<ul>\n<li>[High Impact] Idea<ul>\n<li>AWP</li>\n<li>attention pooling (target column-wise)</li></ul></li>\n<li>[Medium Impact] Idea<ul>\n<li>layer wise learning decay</li></ul></li>\n</ul>\n<h2>What Didn’t Work</h2>\n<ul>\n<li>smooth-l1</li>\n<li>mask augmentaion</li>\n<li>lstm(gru) head</li>\n</ul>\n<p>Small Tips<br>\nI experimented with different seed. And I worked to make sure that the cv was improved.<br>\n(This scheme is described in FB2 1st place solution)</p>\n<p>Looking forward to fb4. Thank you.</p>",
  "comments": [
    {
      "id": 2049889,
      "author_name": "Mark Baushenko",
      "author_url": "",
      "post_date": "2022-11-30T12:04:34.030000",
      "content": "<p>cool shake up, congratulations on an incredible victory!</p>",
      "votes": 1,
      "replies": []
    }
  ],
  "comment_count": 1,
  "messages": [
    {
      "id": 2049829,
      "postDate": "2022-11-30T11:16:53.217Z",
      "content": "<p>Thanks to all the participants and hosts<br>\nI was lucky to win a silver medal with big shake up.</p>\n<h1>Overview</h1>\n<p>finally cv is 0.44582 (public/private 0.439025/0.435032) (deberta-v3-large (x2) ensemble)<br>\nMLM on FB2 data (mask prob = 0.4)<br>\nCV improved slightly but LB worsened. <br>\nHowever, adding it to the ensemble seems to have boosted the score.</p>\n<p>single model cv</p>\n<ul>\n<li>0.4490 (w/o MLM) (public/private 0.439145/0.436422)</li>\n<li>0.4488 (w MLM) (public/private 0.442191/0.437079)</li>\n</ul>\n<p>In my case, LB did not correlate well with the CV improvement.<br>\nLB did not change much when I changed from a single model to an ensemble model<br>\n(This doesn't seem to be the case with the top solutions, I don't know why)<br>\nI stopped working on the competition because LB did not improve. <br>\nHowever, TrustCV seems to have been right.</p>\n<h2>What Worked</h2>\n<ul>\n<li>[High Impact] Idea<ul>\n<li>AWP</li>\n<li>attention pooling (target column-wise)</li></ul></li>\n<li>[Medium Impact] Idea<ul>\n<li>layer wise learning decay</li></ul></li>\n</ul>\n<h2>What Didn’t Work</h2>\n<ul>\n<li>smooth-l1</li>\n<li>mask augmentaion</li>\n<li>lstm(gru) head</li>\n</ul>\n<p>Small Tips<br>\nI experimented with different seed. And I worked to make sure that the cv was improved.<br>\n(This scheme is described in FB2 1st place solution)</p>\n<p>Looking forward to fb4. Thank you.</p>",
      "rawMarkdown": "Thanks to all the participants and hosts\nI was lucky to win a silver medal with big shake up.\n\n# Overview\nfinally cv is 0.44582 (public/private 0.439025/0.435032) (deberta-v3-large (x2) ensemble)\nMLM on FB2 data (mask prob = 0.4)\nCV improved slightly but LB worsened. \nHowever, adding it to the ensemble seems to have boosted the score.\n\nsingle model cv\n- 0.4490 (w/o MLM) (public/private 0.439145/0.436422)\n- 0.4488 (w MLM) (public/private 0.442191/0.437079)\n\nIn my case, LB did not correlate well with the CV improvement.\nLB did not change much when I changed from a single model to an ensemble model\n(This doesn't seem to be the case with the top solutions, I don't know why)\nI stopped working on the competition because LB did not improve. \nHowever, TrustCV seems to have been right.\n\n## What Worked\n- [High Impact] Idea\n    - AWP\n    - attention pooling (target column-wise)\n- [Medium Impact] Idea\n    - layer wise learning decay\n\n## What Didn’t Work\n- smooth-l1\n- mask augmentaion\n- lstm(gru) head\n\nSmall Tips\nI experimented with different seed. And I worked to make sure that the cv was improved.\n(This scheme is described in FB2 1st place solution)\n\nLooking forward to fb4. Thank you.",
      "votes": 15
    },
    {
      "id": 2049889,
      "postDate": "2022-11-30T12:04:34.030Z",
      "content": "<p>cool shake up, congratulations on an incredible victory!</p>",
      "rawMarkdown": "cool shake up, congratulations on an incredible victory!",
      "votes": 1
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T19:59:34Z",
  "competition_id": 38321,
  "competition_slug": "feedback-prize-english-language-learning",
  "team_id": 9091114,
  "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/ktr-26th-place-solution-lucky-shake-up-1117th-26th"
}