{
  "id": 347387,
  "title": "5th place solution",
  "content": "<p>First of all, thanks to competition organizers for hosting this competition and great teammates ( <a href=\"https://www.kaggle.com/shinomoriaoshi\" target=\"_blank\">@shinomoriaoshi</a>, <a href=\"https://www.kaggle.com/horsek\" target=\"_blank\">@horsek</a>, <a href=\"https://www.kaggle.com/runningz\" target=\"_blank\">@runningz</a>, <a href=\"https://www.kaggle.com/nickycan\" target=\"_blank\">@nickycan</a> ).<br>\nThanks also to the community for sharing many ideas in Notebook and Discussion.</p>\n<p>Note: This post is a brief summary, and more detailed information will be updated or posted as a new topic by my teammates.</p>\n<ul>\n<li><a href=\"https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347369\" target=\"_blank\">tri's pipeline</a></li>\n<li><a href=\"https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347379\" target=\"_blank\">housuke's pipeline</a></li>\n</ul>\n<h1>Summary</h1>\n<p>We ensembled 6 token classification models and 1 seq classification model.<br>\n<img src=\"https://user-images.githubusercontent.com/40639636/186302723-c501167e-a850-4fc6-93b0-583e3af72412.png\" alt=\"\"></p>\n<h1>Models</h1>\n<p>We trained following models and used them for final submission.</p>\n<ul>\n<li>tri( <a href=\"https://www.kaggle.com/shinomoriaoshi\" target=\"_blank\">@shinomoriaoshi</a> )'s pipeline<ul>\n<li>token classification Deberta-v3-large</li></ul></li>\n<li>housuke( <a href=\"https://www.kaggle.com/horsek\" target=\"_blank\">@horsek</a> )'s pipeline<ul>\n<li>token classification Deberta-v3-large</li>\n<li>token classification Deberta-large</li></ul></li>\n<li>nakama( <a href=\"https://www.kaggle.com/yasufuminakama\" target=\"_blank\">@yasufuminakama</a> )'s pipeline<ul>\n<li>token classification Deberta-v3-large</li>\n<li>seq classification Deberta-v3-large</li></ul></li>\n<li>RunningZ ( <a href=\"https://www.kaggle.com/runningz\" target=\"_blank\">@runningz</a> )'s pipeline<ul>\n<li>token classification Deberta-v3-large</li>\n<li>token classification Deberta-v2-xlarge</li></ul></li>\n<li>鲲 ( <a href=\"https://www.kaggle.com/nickycan\" target=\"_blank\">@nickycan</a> )'s pipeline<ul>\n<li>mostly engaged on efficiency track</li></ul></li>\n</ul>\n<h1>Main methods that worked</h1>\n<ul>\n<li>MLM pretraining<ul>\n<li><a href=\"https://arxiv.org/pdf/2004.10964.pdf\" target=\"_blank\">https://arxiv.org/pdf/2004.10964.pdf</a></li></ul></li>\n<li>Resolve encoding error<ul>\n<li>This method was used in previous 2021 Feedback Prize competition.</li>\n<li><a href=\"https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313330\" target=\"_blank\">https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313330</a></li></ul></li>\n<li>Mask augmentation<ul>\n<li>This method was used in previous 2021 Feedback Prize competition.</li>\n<li><a href=\"https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313424\" target=\"_blank\">https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313424</a></li></ul></li>\n<li>Adversarial training (AWP, FGM)<ul>\n<li>This method was used in previous 2021 Feedback Prize competition.</li>\n<li><a href=\"https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313177\" target=\"_blank\">https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313177</a></li></ul></li>\n<li>Multi-sample dropout<ul>\n<li>This method was used in Google QUEST Q&amp;A Labeling competition.</li>\n<li><a href=\"https://www.kaggle.com/competitions/google-quest-challenge/discussion/129840\" target=\"_blank\">https://www.kaggle.com/competitions/google-quest-challenge/discussion/129840</a></li></ul></li>\n<li>Add GRU layer</li>\n<li>Label smoothing</li>\n<li>Add discourse_type for each discourse_text</li>\n<li>Add [head] [tail] tokens for each discourse_text</li>\n<li>Back translation (worked only for RunningZ's pipeline)</li>\n<li>Pseudo labeling</li>\n</ul>\n<h1>2nd stage Stacking using LSTM</h1>\n<p>housuke( <a href=\"https://www.kaggle.com/horsek\" target=\"_blank\">@horsek</a> ) tried this early in the competition, and it worked very well.</p>\n<h1>3rd stage Stacking using XGBoost</h1>\n<p>After the 2nd stage Stacking, we applied 3rd stage Stacking using XGBoost, it improved result a bit.</p>\n<h1>Final Result</h1>\n<p>CV: 0.5609<br>\nPublic LB: 0.555<br>\nPrivate LB: 0.560</p>",
  "comments": [],
  "comment_count": 0,
  "messages": [
    {
      "id": 1911236,
      "postDate": "2022-08-24T01:40:33.033Z",
      "content": "<p>First of all, thanks to competition organizers for hosting this competition and great teammates ( <a href=\"https://www.kaggle.com/shinomoriaoshi\" target=\"_blank\">@shinomoriaoshi</a>, <a href=\"https://www.kaggle.com/horsek\" target=\"_blank\">@horsek</a>, <a href=\"https://www.kaggle.com/runningz\" target=\"_blank\">@runningz</a>, <a href=\"https://www.kaggle.com/nickycan\" target=\"_blank\">@nickycan</a> ).<br>\nThanks also to the community for sharing many ideas in Notebook and Discussion.</p>\n<p>Note: This post is a brief summary, and more detailed information will be updated or posted as a new topic by my teammates.</p>\n<ul>\n<li><a href=\"https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347369\" target=\"_blank\">tri's pipeline</a></li>\n<li><a href=\"https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347379\" target=\"_blank\">housuke's pipeline</a></li>\n</ul>\n<h1>Summary</h1>\n<p>We ensembled 6 token classification models and 1 seq classification model.<br>\n<img src=\"https://user-images.githubusercontent.com/40639636/186302723-c501167e-a850-4fc6-93b0-583e3af72412.png\" alt=\"\"></p>\n<h1>Models</h1>\n<p>We trained following models and used them for final submission.</p>\n<ul>\n<li>tri( <a href=\"https://www.kaggle.com/shinomoriaoshi\" target=\"_blank\">@shinomoriaoshi</a> )'s pipeline<ul>\n<li>token classification Deberta-v3-large</li></ul></li>\n<li>housuke( <a href=\"https://www.kaggle.com/horsek\" target=\"_blank\">@horsek</a> )'s pipeline<ul>\n<li>token classification Deberta-v3-large</li>\n<li>token classification Deberta-large</li></ul></li>\n<li>nakama( <a href=\"https://www.kaggle.com/yasufuminakama\" target=\"_blank\">@yasufuminakama</a> )'s pipeline<ul>\n<li>token classification Deberta-v3-large</li>\n<li>seq classification Deberta-v3-large</li></ul></li>\n<li>RunningZ ( <a href=\"https://www.kaggle.com/runningz\" target=\"_blank\">@runningz</a> )'s pipeline<ul>\n<li>token classification Deberta-v3-large</li>\n<li>token classification Deberta-v2-xlarge</li></ul></li>\n<li>鲲 ( <a href=\"https://www.kaggle.com/nickycan\" target=\"_blank\">@nickycan</a> )'s pipeline<ul>\n<li>mostly engaged on efficiency track</li></ul></li>\n</ul>\n<h1>Main methods that worked</h1>\n<ul>\n<li>MLM pretraining<ul>\n<li><a href=\"https://arxiv.org/pdf/2004.10964.pdf\" target=\"_blank\">https://arxiv.org/pdf/2004.10964.pdf</a></li></ul></li>\n<li>Resolve encoding error<ul>\n<li>This method was used in previous 2021 Feedback Prize competition.</li>\n<li><a href=\"https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313330\" target=\"_blank\">https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313330</a></li></ul></li>\n<li>Mask augmentation<ul>\n<li>This method was used in previous 2021 Feedback Prize competition.</li>\n<li><a href=\"https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313424\" target=\"_blank\">https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313424</a></li></ul></li>\n<li>Adversarial training (AWP, FGM)<ul>\n<li>This method was used in previous 2021 Feedback Prize competition.</li>\n<li><a href=\"https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313177\" target=\"_blank\">https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313177</a></li></ul></li>\n<li>Multi-sample dropout<ul>\n<li>This method was used in Google QUEST Q&amp;A Labeling competition.</li>\n<li><a href=\"https://www.kaggle.com/competitions/google-quest-challenge/discussion/129840\" target=\"_blank\">https://www.kaggle.com/competitions/google-quest-challenge/discussion/129840</a></li></ul></li>\n<li>Add GRU layer</li>\n<li>Label smoothing</li>\n<li>Add discourse_type for each discourse_text</li>\n<li>Add [head] [tail] tokens for each discourse_text</li>\n<li>Back translation (worked only for RunningZ's pipeline)</li>\n<li>Pseudo labeling</li>\n</ul>\n<h1>2nd stage Stacking using LSTM</h1>\n<p>housuke( <a href=\"https://www.kaggle.com/horsek\" target=\"_blank\">@horsek</a> ) tried this early in the competition, and it worked very well.</p>\n<h1>3rd stage Stacking using XGBoost</h1>\n<p>After the 2nd stage Stacking, we applied 3rd stage Stacking using XGBoost, it improved result a bit.</p>\n<h1>Final Result</h1>\n<p>CV: 0.5609<br>\nPublic LB: 0.555<br>\nPrivate LB: 0.560</p>",
      "rawMarkdown": "First of all, thanks to competition organizers for hosting this competition and great teammates ( @shinomoriaoshi, @horsek, @runningz, @nickycan ).\nThanks also to the community for sharing many ideas in Notebook and Discussion.\n\nNote: This post is a brief summary, and more detailed information will be updated or posted as a new topic by my teammates.\n- [tri's pipeline](https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347369)\n- [housuke's pipeline](https://www.kaggle.com/competitions/feedback-prize-effectiveness/discussion/347379)\n\n# Summary\nWe ensembled 6 token classification models and 1 seq classification model.\n![](https://user-images.githubusercontent.com/40639636/186302723-c501167e-a850-4fc6-93b0-583e3af72412.png)\n\n# Models\nWe trained following models and used them for final submission.\n- tri( @shinomoriaoshi )'s pipeline\n  - token classification Deberta-v3-large\n- housuke( @horsek )'s pipeline\n  - token classification Deberta-v3-large\n  - token classification Deberta-large\n- nakama( @yasufuminakama )'s pipeline\n  - token classification Deberta-v3-large\n  - seq classification Deberta-v3-large\n- RunningZ ( @runningz )'s pipeline\n  - token classification Deberta-v3-large\n  - token classification Deberta-v2-xlarge\n- 鲲 ( @nickycan )'s pipeline\n  - mostly engaged on efficiency track\n\n# Main methods that worked\n- MLM pretraining\n  - https://arxiv.org/pdf/2004.10964.pdf\n- Resolve encoding error\n  - This method was used in previous 2021 Feedback Prize competition.\n  - https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313330\n- Mask augmentation\n  - This method was used in previous 2021 Feedback Prize competition.\n  - https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313424\n- Adversarial training (AWP, FGM)\n  - This method was used in previous 2021 Feedback Prize competition.\n  - https://www.kaggle.com/competitions/feedback-prize-2021/discussion/313177\n- Multi-sample dropout\n  - This method was used in Google QUEST Q&A Labeling competition.\n  - https://www.kaggle.com/competitions/google-quest-challenge/discussion/129840\n- Add GRU layer\n- Label smoothing\n- Add discourse_type for each discourse_text\n- Add [head] [tail] tokens for each discourse_text\n- Back translation (worked only for RunningZ's pipeline)\n- Pseudo labeling\n\n# 2nd stage Stacking using LSTM\nhousuke( @horsek ) tried this early in the competition, and it worked very well.\n\n# 3rd stage Stacking using XGBoost\nAfter the 2nd stage Stacking, we applied 3rd stage Stacking using XGBoost, it improved result a bit.\n\n# Final Result\nCV: 0.5609\nPublic LB: 0.555\nPrivate LB: 0.560",
      "votes": 45
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T17:52:35Z",
  "competition_id": 35308,
  "competition_slug": "feedback-prize-effectiveness",
  "team_id": 8674999,
  "solution_url": "/competitions/feedback-prize-effectiveness/writeups/h-y-n-r-t-5th-place-solution"
}