{
  "id": 369770,
  "title": "47th Place Solution",
  "content": "<p>First of all, thanks for hosting this competition and congratulations to all the participants. This is my first time to get medal in Kaggle, I've  learned a lot from this experience.</p>\n<h1>Overview</h1>\n<p>My solution is training a lots of models with different parameters(pooling techniques, backbones, max_len, seed) and ensemble them by weighted average. My best CV score is 0.4484.</p>\n<h1>Best Model Configuration</h1>\n<ul>\n<li>Backbone: deberta-v3-base</li>\n<li>Pooling: mean</li>\n<li>Max_len: 512</li>\n<li>Learning rate:<ul>\n<li>Encoder: 2e-4 (layer-wise lr decay: 0.25)</li>\n<li>Decoder: 1e-3</li></ul></li>\n<li>Epoch: 4 (1 for warm up)</li>\n<li>Batch size: 8</li>\n<li>Fold: 5</li>\n<li>Seed: 42</li>\n</ul>\n<h1>Parameters</h1>\n<ul>\n<li>Backbone: deberta-v3-base, deberta-v3-large, deberta-v3-small</li>\n<li>Pooling: mean, cls, max, weighted layer(use last 4 or 5 layers)</li>\n<li>Max_len: 512, 768, 1024, 1536</li>\n<li>Seed: 13, 42</li>\n<li>Epoch: 3, 4, 5</li>\n</ul>\n<h1>Ensemble</h1>\n<p>Find weights by <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/363773\" target=\"_blank\">this</a> in my models pool, filter out the model whos weight &lt; 1e-3 and re-find weights.</p>\n<h1>What worked</h1>\n<ul>\n<li>[High Impact] Layer-wise Learning Rate Decay</li>\n<li>[low Impact] Add special token: \"\\n\\n\", \"\\r\\n\\r\\n\"</li>\n<li>[low Impact] First epoch warm up</li>\n<li>[low Impact] Freeze layers</li>\n</ul>\n<h1>What Didn’t Work</h1>\n<ul>\n<li>AWP</li>\n<li>Pseudo Labels</li>\n<li>Last Layer Re-initialization</li>\n</ul>\n<h1>Important Citation</h1>\n<ul>\n<li><a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">FB3 / Deberta-v3-base baseline [train]</a></li>\n<li><a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/363773\" target=\"_blank\">Strategies For Ensembling Models Using OOF Predictions</a></li>\n<li><a href=\"https://www.kaggle.com/code/rhtsingh/utilizing-transformer-representations-efficiently\" target=\"_blank\">Utilizing Transformer Representations Efficiently</a></li>\n</ul>\n<h1>Thanks and Acknowledgements</h1>\n<p>Thanks to competition organizers for hosting this competition and everyone who shared their knowledge during the competition. I am so happy to learn a lot from it!</p>",
  "comments": [],
  "comment_count": 0,
  "messages": [
    {
      "id": 2051318,
      "postDate": "2022-12-01T10:39:02.197Z",
      "content": "<p>First of all, thanks for hosting this competition and congratulations to all the participants. This is my first time to get medal in Kaggle, I've  learned a lot from this experience.</p>\n<h1>Overview</h1>\n<p>My solution is training a lots of models with different parameters(pooling techniques, backbones, max_len, seed) and ensemble them by weighted average. My best CV score is 0.4484.</p>\n<h1>Best Model Configuration</h1>\n<ul>\n<li>Backbone: deberta-v3-base</li>\n<li>Pooling: mean</li>\n<li>Max_len: 512</li>\n<li>Learning rate:<ul>\n<li>Encoder: 2e-4 (layer-wise lr decay: 0.25)</li>\n<li>Decoder: 1e-3</li></ul></li>\n<li>Epoch: 4 (1 for warm up)</li>\n<li>Batch size: 8</li>\n<li>Fold: 5</li>\n<li>Seed: 42</li>\n</ul>\n<h1>Parameters</h1>\n<ul>\n<li>Backbone: deberta-v3-base, deberta-v3-large, deberta-v3-small</li>\n<li>Pooling: mean, cls, max, weighted layer(use last 4 or 5 layers)</li>\n<li>Max_len: 512, 768, 1024, 1536</li>\n<li>Seed: 13, 42</li>\n<li>Epoch: 3, 4, 5</li>\n</ul>\n<h1>Ensemble</h1>\n<p>Find weights by <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/363773\" target=\"_blank\">this</a> in my models pool, filter out the model whos weight &lt; 1e-3 and re-find weights.</p>\n<h1>What worked</h1>\n<ul>\n<li>[High Impact] Layer-wise Learning Rate Decay</li>\n<li>[low Impact] Add special token: \"\\n\\n\", \"\\r\\n\\r\\n\"</li>\n<li>[low Impact] First epoch warm up</li>\n<li>[low Impact] Freeze layers</li>\n</ul>\n<h1>What Didn’t Work</h1>\n<ul>\n<li>AWP</li>\n<li>Pseudo Labels</li>\n<li>Last Layer Re-initialization</li>\n</ul>\n<h1>Important Citation</h1>\n<ul>\n<li><a href=\"https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train\" target=\"_blank\">FB3 / Deberta-v3-base baseline [train]</a></li>\n<li><a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/363773\" target=\"_blank\">Strategies For Ensembling Models Using OOF Predictions</a></li>\n<li><a href=\"https://www.kaggle.com/code/rhtsingh/utilizing-transformer-representations-efficiently\" target=\"_blank\">Utilizing Transformer Representations Efficiently</a></li>\n</ul>\n<h1>Thanks and Acknowledgements</h1>\n<p>Thanks to competition organizers for hosting this competition and everyone who shared their knowledge during the competition. I am so happy to learn a lot from it!</p>",
      "rawMarkdown": "First of all, thanks for hosting this competition and congratulations to all the participants. This is my first time to get medal in Kaggle, I've  learned a lot from this experience.\n\n# Overview\nMy solution is training a lots of models with different parameters(pooling techniques, backbones, max_len, seed) and ensemble them by weighted average. My best CV score is 0.4484.\n\n# Best Model Configuration\n- Backbone: deberta-v3-base\n- Pooling: mean\n- Max_len: 512\n- Learning rate:\n  - Encoder: 2e-4 (layer-wise lr decay: 0.25)\n  - Decoder: 1e-3\n- Epoch: 4 (1 for warm up)\n- Batch size: 8\n- Fold: 5\n- Seed: 42\n\n# Parameters\n- Backbone: deberta-v3-base, deberta-v3-large, deberta-v3-small\n- Pooling: mean, cls, max, weighted layer(use last 4 or 5 layers)\n- Max_len: 512, 768, 1024, 1536\n- Seed: 13, 42\n- Epoch: 3, 4, 5\n\n# Ensemble\nFind weights by [this](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/363773) in my models pool, filter out the model whos weight < 1e-3 and re-find weights.\n\n# What worked\n- [High Impact] Layer-wise Learning Rate Decay\n- [low Impact] Add special token: \"\\n\\n\", \"\\r\\n\\r\\n\"\n- [low Impact] First epoch warm up\n- [low Impact] Freeze layers\n\n# What Didn’t Work\n- AWP\n- Pseudo Labels\n- Last Layer Re-initialization\n\n# Important Citation\n- [FB3 / Deberta-v3-base baseline [train]](https://www.kaggle.com/code/yasufuminakama/fb3-deberta-v3-base-baseline-train)\n- [Strategies For Ensembling Models Using OOF Predictions](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/363773)\n- [Utilizing Transformer Representations Efficiently](https://www.kaggle.com/code/rhtsingh/utilizing-transformer-representations-efficiently)\n\n# Thanks and Acknowledgements\nThanks to competition organizers for hosting this competition and everyone who shared their knowledge during the competition. I am so happy to learn a lot from it!",
      "votes": 6
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T20:03:50Z",
  "competition_id": 38321,
  "competition_slug": "feedback-prize-english-language-learning",
  "team_id": 9097677,
  "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/jimmyliao86204-47th-place-solution"
}