{
  "id": 370459,
  "title": "38th place solution",
  "content": "<p>Thanks to kaggle team, host for hosting this competition and to teammates. Thanks also to the Kaggler's for sharing their helpful codes and discussions. This was my second time to participate in NLP competition after the previous fp2 competition, and I learned a lot.</p>\n<h2>Overview</h2>\n<p>My final submission is ensemble of 15 models optimized for weights with optuna. Each models are trained with various hyperparameter such as seed, num_fold, max_length, and so on.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5015548%2F2ef9b2f7ebf77e1e92394218afcf78a7%2Fsolution.png?generation=1670233325624196&amp;alt=media\" alt=\"\"></p>\n<h2>What Worked</h2>\n<ul>\n<li>SmoothL1Loss was the best in CV.</li>\n<li>Using optuna to determine ensemble weights for each target.</li>\n<li>Use various custom head.<ul>\n<li>mean pooling, lstm, concat last 4 layer</li></ul></li>\n<li>Adversarial Weight Perturbation</li>\n<li>Use various model. (not only the hugging face model, but also lightgbm)<ul>\n<li>Single model is not good, but ensemble has improved score a little.</li>\n<li>refs (lightgbm model): <a href=\"https://www.kaggle.com/code/dlaststark/fpe-no-fancy-stuff\" target=\"_blank\">https://www.kaggle.com/code/dlaststark/fpe-no-fancy-stuff</a></li></ul></li>\n</ul>\n<h2>What Didn’t Work</h2>\n<ul>\n<li>Different loss functions</li>\n<li>mixout</li>\n<li>Use text features as meta feature when I stacking models.</li>\n</ul>\n<p>These methods did not improve cv scores, but few were effective in ensembles.</p>\n<h2>Important Citations</h2>\n<ul>\n<li><a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train</a><br>\n<a href=\"https://www.kaggle.com/yasufuminakama\" target=\"_blank\">@yasufuminakama</a>. His notebook has taught me a lot.</li>\n<li><a href=\"https://www.kaggle.com/code/electro/deberta-layerwiselr-lastlayerreinit-tensorflow\" target=\"_blank\">https://www.kaggle.com/code/electro/deberta-layerwiselr-lastlayerreinit-tensorflow</a></li>\n<li><a href=\"https://www.kaggle.com/code/kojimar/fb3-single-pytorch-model-train/notebook\" target=\"_blank\">https://www.kaggle.com/code/kojimar/fb3-single-pytorch-model-train/notebook</a></li>\n</ul>\n<h2>Thanks and Acknowledgements</h2>\n<p>Finally, I'd like to thank Kaggle, host for hosting such an iteresting competition and all who participated discussions.<br>\nThank you and Congratulations to my team mates - <a href=\"https://www.kaggle.com/mitsuruueki\" target=\"_blank\">@mitsuruueki</a></p>",
  "comments": [
    {
      "id": 2056764,
      "author_name": "Olqa_842",
      "author_url": "",
      "post_date": "2022-12-06T12:17:44.460000",
      "content": "<p>Cool, but why you don't use single model? <a href=\"https://www.kaggle.com/thirota\" target=\"_blank\">@thirota</a> </p>",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 1,
  "messages": [
    {
      "id": 2054985,
      "postDate": "2022-12-04T16:18:47.327Z",
      "content": "<p>Thanks to kaggle team, host for hosting this competition and to teammates. Thanks also to the Kaggler's for sharing their helpful codes and discussions. This was my second time to participate in NLP competition after the previous fp2 competition, and I learned a lot.</p>\n<h2>Overview</h2>\n<p>My final submission is ensemble of 15 models optimized for weights with optuna. Each models are trained with various hyperparameter such as seed, num_fold, max_length, and so on.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5015548%2F2ef9b2f7ebf77e1e92394218afcf78a7%2Fsolution.png?generation=1670233325624196&amp;alt=media\" alt=\"\"></p>\n<h2>What Worked</h2>\n<ul>\n<li>SmoothL1Loss was the best in CV.</li>\n<li>Using optuna to determine ensemble weights for each target.</li>\n<li>Use various custom head.<ul>\n<li>mean pooling, lstm, concat last 4 layer</li></ul></li>\n<li>Adversarial Weight Perturbation</li>\n<li>Use various model. (not only the hugging face model, but also lightgbm)<ul>\n<li>Single model is not good, but ensemble has improved score a little.</li>\n<li>refs (lightgbm model): <a href=\"https://www.kaggle.com/code/dlaststark/fpe-no-fancy-stuff\" target=\"_blank\">https://www.kaggle.com/code/dlaststark/fpe-no-fancy-stuff</a></li></ul></li>\n</ul>\n<h2>What Didn’t Work</h2>\n<ul>\n<li>Different loss functions</li>\n<li>mixout</li>\n<li>Use text features as meta feature when I stacking models.</li>\n</ul>\n<p>These methods did not improve cv scores, but few were effective in ensembles.</p>\n<h2>Important Citations</h2>\n<ul>\n<li><a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train</a><br>\n<a href=\"https://www.kaggle.com/yasufuminakama\" target=\"_blank\">@yasufuminakama</a>. His notebook has taught me a lot.</li>\n<li><a href=\"https://www.kaggle.com/code/electro/deberta-layerwiselr-lastlayerreinit-tensorflow\" target=\"_blank\">https://www.kaggle.com/code/electro/deberta-layerwiselr-lastlayerreinit-tensorflow</a></li>\n<li><a href=\"https://www.kaggle.com/code/kojimar/fb3-single-pytorch-model-train/notebook\" target=\"_blank\">https://www.kaggle.com/code/kojimar/fb3-single-pytorch-model-train/notebook</a></li>\n</ul>\n<h2>Thanks and Acknowledgements</h2>\n<p>Finally, I'd like to thank Kaggle, host for hosting such an iteresting competition and all who participated discussions.<br>\nThank you and Congratulations to my team mates - <a href=\"https://www.kaggle.com/mitsuruueki\" target=\"_blank\">@mitsuruueki</a></p>",
      "rawMarkdown": "Thanks to kaggle team, host for hosting this competition and to teammates. Thanks also to the Kaggler's for sharing their helpful codes and discussions. This was my second time to participate in NLP competition after the previous fp2 competition, and I learned a lot.\n\n## Overview\n\nMy final submission is ensemble of 15 models optimized for weights with optuna. Each models are trained with various hyperparameter such as seed, num_fold, max_length, and so on.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5015548%2F2ef9b2f7ebf77e1e92394218afcf78a7%2Fsolution.png?generation=1670233325624196&alt=media)\n\n## What Worked\n\n* SmoothL1Loss was the best in CV.\n* Using optuna to determine ensemble weights for each target.\n* Use various custom head.\n  * mean pooling, lstm, concat last 4 layer\n* Adversarial Weight Perturbation\n* Use various model. (not only the hugging face model, but also lightgbm)\n  * Single model is not good, but ensemble has improved score a little.\n  * refs (lightgbm model): <https://www.kaggle.com/code/dlaststark/fpe-no-fancy-stuff>\n\n## What Didn’t Work\n\n* Different loss functions\n* mixout\n* Use text features as meta feature when I stacking models.\n\nThese methods did not improve cv scores, but few were effective in ensembles.\n\n## Important Citations\n\n* <https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train>\n@yasufuminakama. His notebook has taught me a lot.\n* <https://www.kaggle.com/code/electro/deberta-layerwiselr-lastlayerreinit-tensorflow>\n* <https://www.kaggle.com/code/kojimar/fb3-single-pytorch-model-train/notebook>\n\n## Thanks and Acknowledgements\n\nFinally, I'd like to thank Kaggle, host for hosting such an iteresting competition and all who participated discussions.\nThank you and Congratulations to my team mates - @mitsuruueki\n",
      "votes": 11
    },
    {
      "id": 2056764,
      "postDate": "2022-12-06T12:17:44.460Z",
      "content": "<p>Cool, but why you don't use single model? <a href=\"https://www.kaggle.com/thirota\" target=\"_blank\">@thirota</a> </p>",
      "rawMarkdown": "Cool, but why you don't use single model? @thirota "
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T20:05:34Z",
  "competition_id": 38321,
  "competition_slug": "feedback-prize-english-language-learning",
  "team_id": 9094320,
  "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/ginger-grilled-pork-38th-place-solution"
}