{
  "id": 369521,
  "title": "999 place solution🤣(Public 185)",
  "content": "<p>Congratulations to all the participants, and thanks a lot to the organizers of this competition!</p>\n<p>I got a very large shake-down, but would like to share my methods in this thread!</p>\n<h2>Main approach</h2>\n<p>Based on <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/349336\" target=\"_blank\">nakama's starter code</a>, I tried various ideas discussed in Discussion.</p>\n<p>My idea, which I think was unique, was to round the target. The target value is 9 steps from 1.0 to 5.0 but the original rubric is 5 steps from 1 to 5. I see <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/348973#2013264\" target=\"_blank\">this thread</a> and I have known that this competition target was made from two evaluators' averages. So I made two models, one rounding up 0.5 and one rounding down 0.5, and took the average of them.</p>\n<p><img src=\"https://raw.githubusercontent.com/trtd56/FeedbackPrize3/main/img/%E3%82%B9%E3%82%AF%E3%83%AA%E3%83%BC%E3%83%B3%E3%82%B7%E3%83%A7%E3%83%83%E3%83%88%202022-11-29%2020.18.51.png\" alt=\"\"></p>\n<h3>ensemble</h3>\n<p>My final submission Weight Averaging ensemble from OOF score.</p>\n<p>In addition to some DeBERTa-v3-base with different learning parameters, we used the following model:</p>\n<ul>\n<li>Longformer-base</li>\n<li>BigBird-RoBERTa-base</li>\n<li>BART-large</li>\n<li>FunnelTransformer-small</li>\n<li>simple embedding and SVR model (<a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/351577\" target=\"_blank\">Chris's approach</a>)</li>\n</ul>\n<h2>Other things that worked well</h2>\n<ul>\n<li>AWP</li>\n<li>FGM</li>\n<li>Layer-wise Learning Rate</li>\n<li>Weights Initialization</li>\n<li><a href=\"https://www.kaggle.com/code/rhtsingh/on-stability-of-few-sample-transformer-fine-tuning/notebook\" target=\"_blank\">Pre-trained Weight Decay</a></li>\n<li>Training with a short text (max_len=768)</li>\n</ul>\n<h2>Not work for me (so many…)</h2>\n<ul>\n<li>pseudo labeling<ul>\n<li>unlabeled data worked meta pseudo label only.</li></ul></li>\n<li>SiFT</li>\n<li>custom loss considering target distribution</li>\n<li>solve as a classification problem</li>\n<li>large model</li>\n<li>Custom Head<ul>\n<li>LSTM</li>\n<li>CNN</li>\n<li>Attention</li></ul></li>\n<li>Weighted Layer Pooling</li>\n<li>MLM</li>\n<li>SWA</li>\n<li>consider line breaks</li>\n<li>consider masking characters (<code>Generic_Name</code>, <code>Generic_City</code> …)</li>\n<li>stacking <a href=\"https://pypi.org/project/readability/\" target=\"_blank\">readability</a> score</li>\n</ul>\n<h2>CV vs LB</h2>\n<p>My CV and Public LB rank are well correlated, but I got large shake-down(185→999).</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1483555%2Fa8f073eb31adbdc39b81795376ece9ea%2F2022-11-30%2019.52.13.png?generation=1669805554267453&amp;alt=media\" alt=\"\"></p>\n<p>I don't use Pseudo label, so there should be no leak, but I still don't understand this part, so I will try to learn more from others' solutions.</p>\n<p>Finally, in this competition, I only used kaggle notebook and a free plan colab to train. <br>\nThank you for the nice platform that has created an opportunity for people like me who can't afford to spend money on a computing machine😊</p>",
  "comments": [],
  "comment_count": 0,
  "messages": [
    {
      "id": 2049801,
      "postDate": "2022-11-30T10:56:08.113Z",
      "content": "<p>Congratulations to all the participants, and thanks a lot to the organizers of this competition!</p>\n<p>I got a very large shake-down, but would like to share my methods in this thread!</p>\n<h2>Main approach</h2>\n<p>Based on <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/349336\" target=\"_blank\">nakama's starter code</a>, I tried various ideas discussed in Discussion.</p>\n<p>My idea, which I think was unique, was to round the target. The target value is 9 steps from 1.0 to 5.0 but the original rubric is 5 steps from 1 to 5. I see <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/348973#2013264\" target=\"_blank\">this thread</a> and I have known that this competition target was made from two evaluators' averages. So I made two models, one rounding up 0.5 and one rounding down 0.5, and took the average of them.</p>\n<p><img src=\"https://raw.githubusercontent.com/trtd56/FeedbackPrize3/main/img/%E3%82%B9%E3%82%AF%E3%83%AA%E3%83%BC%E3%83%B3%E3%82%B7%E3%83%A7%E3%83%83%E3%83%88%202022-11-29%2020.18.51.png\" alt=\"\"></p>\n<h3>ensemble</h3>\n<p>My final submission Weight Averaging ensemble from OOF score.</p>\n<p>In addition to some DeBERTa-v3-base with different learning parameters, we used the following model:</p>\n<ul>\n<li>Longformer-base</li>\n<li>BigBird-RoBERTa-base</li>\n<li>BART-large</li>\n<li>FunnelTransformer-small</li>\n<li>simple embedding and SVR model (<a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/351577\" target=\"_blank\">Chris's approach</a>)</li>\n</ul>\n<h2>Other things that worked well</h2>\n<ul>\n<li>AWP</li>\n<li>FGM</li>\n<li>Layer-wise Learning Rate</li>\n<li>Weights Initialization</li>\n<li><a href=\"https://www.kaggle.com/code/rhtsingh/on-stability-of-few-sample-transformer-fine-tuning/notebook\" target=\"_blank\">Pre-trained Weight Decay</a></li>\n<li>Training with a short text (max_len=768)</li>\n</ul>\n<h2>Not work for me (so many…)</h2>\n<ul>\n<li>pseudo labeling<ul>\n<li>unlabeled data worked meta pseudo label only.</li></ul></li>\n<li>SiFT</li>\n<li>custom loss considering target distribution</li>\n<li>solve as a classification problem</li>\n<li>large model</li>\n<li>Custom Head<ul>\n<li>LSTM</li>\n<li>CNN</li>\n<li>Attention</li></ul></li>\n<li>Weighted Layer Pooling</li>\n<li>MLM</li>\n<li>SWA</li>\n<li>consider line breaks</li>\n<li>consider masking characters (<code>Generic_Name</code>, <code>Generic_City</code> …)</li>\n<li>stacking <a href=\"https://pypi.org/project/readability/\" target=\"_blank\">readability</a> score</li>\n</ul>\n<h2>CV vs LB</h2>\n<p>My CV and Public LB rank are well correlated, but I got large shake-down(185→999).</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1483555%2Fa8f073eb31adbdc39b81795376ece9ea%2F2022-11-30%2019.52.13.png?generation=1669805554267453&amp;alt=media\" alt=\"\"></p>\n<p>I don't use Pseudo label, so there should be no leak, but I still don't understand this part, so I will try to learn more from others' solutions.</p>\n<p>Finally, in this competition, I only used kaggle notebook and a free plan colab to train. <br>\nThank you for the nice platform that has created an opportunity for people like me who can't afford to spend money on a computing machine😊</p>",
      "rawMarkdown": "Congratulations to all the participants, and thanks a lot to the organizers of this competition!\n\nI got a very large shake-down, but would like to share my methods in this thread!\n\n## Main approach\n\nBased on [nakama's starter code](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/349336), I tried various ideas discussed in Discussion.\n\nMy idea, which I think was unique, was to round the target. The target value is 9 steps from 1.0 to 5.0 but the original rubric is 5 steps from 1 to 5. I see [this thread](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/348973#2013264) and I have known that this competition target was made from two evaluators' averages. So I made two models, one rounding up 0.5 and one rounding down 0.5, and took the average of them.\n\n![](https://raw.githubusercontent.com/trtd56/FeedbackPrize3/main/img/%E3%82%B9%E3%82%AF%E3%83%AA%E3%83%BC%E3%83%B3%E3%82%B7%E3%83%A7%E3%83%83%E3%83%88%202022-11-29%2020.18.51.png)\n\n### ensemble\n\nMy final submission Weight Averaging ensemble from OOF score.\n\nIn addition to some DeBERTa-v3-base with different learning parameters, we used the following model:\n\n- Longformer-base\n- BigBird-RoBERTa-base\n- BART-large\n- FunnelTransformer-small\n- simple embedding and SVR model ([Chris's approach](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/351577))\n\n## Other things that worked well\n\n- AWP\n- FGM\n- Layer-wise Learning Rate\n- Weights Initialization\n- [Pre-trained Weight Decay](https://www.kaggle.com/code/rhtsingh/on-stability-of-few-sample-transformer-fine-tuning/notebook)\n- Training with a short text (max_len=768)\n\n## Not work for me (so many…)\n\n- pseudo labeling\n    - unlabeled data worked meta pseudo label only.\n- SiFT\n- custom loss considering target distribution\n- solve as a classification problem\n- large model\n- Custom Head\n    - LSTM\n    - CNN\n    - Attention\n- Weighted Layer Pooling\n- MLM\n- SWA\n- consider line breaks\n- consider masking characters (`Generic_Name`, `Generic_City` …)\n- stacking [readability](https://pypi.org/project/readability/) score\n\n\n## CV vs LB\n\nMy CV and Public LB rank are well correlated, but I got large shake-down(185→999).\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1483555%2Fa8f073eb31adbdc39b81795376ece9ea%2F2022-11-30%2019.52.13.png?generation=1669805554267453&alt=media)\n\nI don't use Pseudo label, so there should be no leak, but I still don't understand this part, so I will try to learn more from others' solutions.\n\n\nFinally, in this competition, I only used kaggle notebook and a free plan colab to train. \nThank you for the nice platform that has created an opportunity for people like me who can't afford to spend money on a computing machine😊",
      "votes": 29
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T19:58:59Z",
  "competition_id": 38321,
  "competition_slug": "feedback-prize-english-language-learning",
  "team_id": 9101289,
  "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/takamichi-toda-999-place-solution-public-185"
}