{
  "id": 369822,
  "title": "21th Place Solution",
  "content": "<h1>Overview</h1>\n<ul>\n<li>Weighted average ensemble of 21 models.</li>\n<li>Some model used a weighted average of the original prediction and the prediction trained by SVR or Ridge regression of the embedding. (Whether to use SVR or Ridge or neither was based on the pattern that would give the best CV when incorporated into the ensemble)</li>\n<li>cv=0.4407/PublicLB=0.434031/PrivateLB=0.434983<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3823496%2F4c3b2f830c5f99822cfe532cf5cf81b7%2Foverview.png?generation=1669903792363382&amp;alt=media\" alt=\"\"></li>\n</ul>\n<h1>What worked</h1>\n<ul>\n<li>Special tokens for each indicator (Multi Token)<ul>\n<li>A token for each indicator (\"[REG_COH]\", \"[REG_SYN]\", etc…) instead of a CLS token was added at the beginning and used.</li>\n<li>Depending on the backbone, DeBERTa-v3-large performed best this way (CV=0.4478)</li></ul></li>\n<li>Weighted average with SVR or Ridge regression on a per single model<ul>\n<li>Depends on the model, but can easily be improved by about 0.025 if effective</li></ul></li>\n<li>AWP<ul>\n<li>Accuracy is about 0.01 to 0.02 better on its own when used</li>\n<li>However, when incorporated into ensembles, models without AWP often improved performance, so it was ultimately applied to only some models</li></ul></li>\n<li>Classification Task Model<ul>\n<li>The stand-alone performance was not that good (cv=0.4562), but it was effective for the ensemble.</li></ul></li>\n<li>Pseudo Labeling<ul>\n<li>Basically all models improved performance (the most effective model improved by about 0.005, cv=0.445)</li>\n<li>However, PublicLB lost a lot of accuracy and only one pseudo-label model was included in the final sub</li>\n<li>But this was a huge mistake, and the biggest factor in my shakedown. (See <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369401\" target=\"_blank\">this discussion</a> for more details.)</li>\n<li>[Reference] Performance of the ensemble model with Pseudo-Label: cv=0.4410/PublicLB=0.436303/PrivateLB=0.433781</li></ul></li>\n<li>LLRD</li>\n</ul>\n<h1>What Didn’t Worked</h1>\n<ul>\n<li>Last Layer Re-initialization</li>\n<li>Post Process<ul>\n<li>I tried to correct the final forecast by multiplying the final forecast by the coefficient obtained by Nelder-Mead for each range, but LB worsened, so I did not adopt it.</li></ul></li>\n</ul>\n<h1>Important Citations:</h1>\n<ul>\n<li><a href=\"https://www.kaggle.com/code/abhishek/tez-fb3-beating-the-benchmark-inference\" target=\"_blank\">tez: fb3 beating the benchmark inference</a></li>\n</ul>\n<h1>Thanks and Acknowledgements:</h1>\n<p>Thanks to competition organizers for hosting this competition.<br>\nAnd a big thank you to all the participants who shared their wonderful notebooks and discussions.<br>\nI was a beginner who first started doing NLP with FB2 about 6 months ago, but thanks to the easy to understand competition design and many useful discussions, I now know a little more about NLP.<br>\nI sincerely hope that you will hold such a competition again. Thank you again.</p>",
  "comments": [
    {
      "id": 2056675,
      "author_name": "RB",
      "author_url": "",
      "post_date": "2022-12-06T10:58:03.400000",
      "content": "<blockquote>\n  <p>A token for each indicator (\"[REG_COH]\", \"[REG_SYN]\", etc…) instead of a CLS token was added at the beginning and used</p>\n</blockquote>\n<p>Can you please explain this a little more ? Did you concatenate these 6 tokens and then a fully connected a layer with 6 outputs ? </p>",
      "votes": 0,
      "replies": [
        {
          "id": 2059194,
          "author_name": "YumeNeko",
          "author_url": "",
          "post_date": "2022-12-08T15:10:02.453000",
          "content": "<p>For each token, we defined a fully-connected layer with an output size of 1 and concatenated them to compute the loss.<br>\nI think it is difficult to understand in words, so a simplified pseudo code is shown below.</p>\n<pre><code> (nn.Module):\n     ():\n        self.transformer = AutoModel.from_pretrained(model_name, config=config)\n        self.output_cohesion = nn.Linear(config.hidden_size, )\n        self.output_syntax = nn.Linear(config.hidden_size, )\n        self.output_vocabulary = nn.Linear(config.hidden_size, )\n        self.output_phraseology = nn.Linear(config.hidden_size, )\n        self.output_grammar = nn.Linear(config.hidden_size, )\n        self.output_conventions = nn.Linear(config.hidden_size, )\n\n     ():\n        transformer_out = self.transformer(input_ids=ids, attention_mask=mask)\n        sequence_output = transformer_out.last_hidden_state\n\n        logits_cohesion = self.output_cohesion(sequence_output[:, , :])\n        logits_syntax = self.output_syntax(sequence_output[:, , :])\n        logits_vocabulary = self.output_vocabulary(sequence_output[:, , :])\n        logits_phraseology = self.output_phraseology(sequence_output[:, , :])\n        logits_grammar = self.output_grammar(sequence_output[:, , :])\n        logits_conventions = self.output_conventions(sequence_output[:, , :])\n\n        logits = torch.cat([logits_cohesion, logits_syntax, logits_vocabulary, logits_phraseology, logits_grammar, logits_conventions], axis=)\n        loss = self.loss(logits, targets)\n</code></pre>",
          "votes": 1,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 2,
  "messages": [
    {
      "id": 2051658,
      "postDate": "2022-12-01T14:52:17.290Z",
      "content": "<h1>Overview</h1>\n<ul>\n<li>Weighted average ensemble of 21 models.</li>\n<li>Some model used a weighted average of the original prediction and the prediction trained by SVR or Ridge regression of the embedding. (Whether to use SVR or Ridge or neither was based on the pattern that would give the best CV when incorporated into the ensemble)</li>\n<li>cv=0.4407/PublicLB=0.434031/PrivateLB=0.434983<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3823496%2F4c3b2f830c5f99822cfe532cf5cf81b7%2Foverview.png?generation=1669903792363382&amp;alt=media\" alt=\"\"></li>\n</ul>\n<h1>What worked</h1>\n<ul>\n<li>Special tokens for each indicator (Multi Token)<ul>\n<li>A token for each indicator (\"[REG_COH]\", \"[REG_SYN]\", etc…) instead of a CLS token was added at the beginning and used.</li>\n<li>Depending on the backbone, DeBERTa-v3-large performed best this way (CV=0.4478)</li></ul></li>\n<li>Weighted average with SVR or Ridge regression on a per single model<ul>\n<li>Depends on the model, but can easily be improved by about 0.025 if effective</li></ul></li>\n<li>AWP<ul>\n<li>Accuracy is about 0.01 to 0.02 better on its own when used</li>\n<li>However, when incorporated into ensembles, models without AWP often improved performance, so it was ultimately applied to only some models</li></ul></li>\n<li>Classification Task Model<ul>\n<li>The stand-alone performance was not that good (cv=0.4562), but it was effective for the ensemble.</li></ul></li>\n<li>Pseudo Labeling<ul>\n<li>Basically all models improved performance (the most effective model improved by about 0.005, cv=0.445)</li>\n<li>However, PublicLB lost a lot of accuracy and only one pseudo-label model was included in the final sub</li>\n<li>But this was a huge mistake, and the biggest factor in my shakedown. (See <a href=\"https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369401\" target=\"_blank\">this discussion</a> for more details.)</li>\n<li>[Reference] Performance of the ensemble model with Pseudo-Label: cv=0.4410/PublicLB=0.436303/PrivateLB=0.433781</li></ul></li>\n<li>LLRD</li>\n</ul>\n<h1>What Didn’t Worked</h1>\n<ul>\n<li>Last Layer Re-initialization</li>\n<li>Post Process<ul>\n<li>I tried to correct the final forecast by multiplying the final forecast by the coefficient obtained by Nelder-Mead for each range, but LB worsened, so I did not adopt it.</li></ul></li>\n</ul>\n<h1>Important Citations:</h1>\n<ul>\n<li><a href=\"https://www.kaggle.com/code/abhishek/tez-fb3-beating-the-benchmark-inference\" target=\"_blank\">tez: fb3 beating the benchmark inference</a></li>\n</ul>\n<h1>Thanks and Acknowledgements:</h1>\n<p>Thanks to competition organizers for hosting this competition.<br>\nAnd a big thank you to all the participants who shared their wonderful notebooks and discussions.<br>\nI was a beginner who first started doing NLP with FB2 about 6 months ago, but thanks to the easy to understand competition design and many useful discussions, I now know a little more about NLP.<br>\nI sincerely hope that you will hold such a competition again. Thank you again.</p>",
      "rawMarkdown": "# Overview\n* Weighted average ensemble of 21 models.\n* Some model used a weighted average of the original prediction and the prediction trained by SVR or Ridge regression of the embedding. (Whether to use SVR or Ridge or neither was based on the pattern that would give the best CV when incorporated into the ensemble)\n* cv=0.4407/PublicLB=0.434031/PrivateLB=0.434983\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3823496%2F4c3b2f830c5f99822cfe532cf5cf81b7%2Foverview.png?generation=1669903792363382&alt=media)\n\n# What worked\n* Special tokens for each indicator (Multi Token)\n  * A token for each indicator (\"[REG_COH]\", \"[REG_SYN]\", etc...) instead of a CLS token was added at the beginning and used.\n  * Depending on the backbone, DeBERTa-v3-large performed best this way (CV=0.4478)\n* Weighted average with SVR or Ridge regression on a per single model\n  * Depends on the model, but can easily be improved by about 0.025 if effective\n* AWP\n  * Accuracy is about 0.01 to 0.02 better on its own when used\n  * However, when incorporated into ensembles, models without AWP often improved performance, so it was ultimately applied to only some models\n* Classification Task Model\n  * The stand-alone performance was not that good (cv=0.4562), but it was effective for the ensemble.\n* Pseudo Labeling\n  * Basically all models improved performance (the most effective model improved by about 0.005, cv=0.445)\n  * However, PublicLB lost a lot of accuracy and only one pseudo-label model was included in the final sub\n  * But this was a huge mistake, and the biggest factor in my shakedown. (See [this discussion](https://www.kaggle.com/competitions/feedback-prize-english-language-learning/discussion/369401) for more details.)\n  * [Reference] Performance of the ensemble model with Pseudo-Label: cv=0.4410/PublicLB=0.436303/PrivateLB=0.433781\n* LLRD\n\n# What Didn’t Worked\n* Last Layer Re-initialization\n* Post Process\n  * I tried to correct the final forecast by multiplying the final forecast by the coefficient obtained by Nelder-Mead for each range, but LB worsened, so I did not adopt it.\n\n# Important Citations:\n* [tez: fb3 beating the benchmark inference](https://www.kaggle.com/code/abhishek/tez-fb3-beating-the-benchmark-inference)\n\n# Thanks and Acknowledgements:\nThanks to competition organizers for hosting this competition.\nAnd a big thank you to all the participants who shared their wonderful notebooks and discussions.\nI was a beginner who first started doing NLP with FB2 about 6 months ago, but thanks to the easy to understand competition design and many useful discussions, I now know a little more about NLP.\nI sincerely hope that you will hold such a competition again. Thank you again.\n",
      "votes": 30
    },
    {
      "id": 2056675,
      "postDate": "2022-12-06T10:58:03.400Z",
      "content": "<blockquote>\n  <p>A token for each indicator (\"[REG_COH]\", \"[REG_SYN]\", etc…) instead of a CLS token was added at the beginning and used</p>\n</blockquote>\n<p>Can you please explain this a little more ? Did you concatenate these 6 tokens and then a fully connected a layer with 6 outputs ? </p>",
      "rawMarkdown": "> A token for each indicator (\"[REG_COH]\", \"[REG_SYN]\", etc…) instead of a CLS token was added at the beginning and used\n\nCan you please explain this a little more ? Did you concatenate these 6 tokens and then a fully connected a layer with 6 outputs ? ",
      "replies": [
        {
          "id": 2059194,
          "postDate": "2022-12-08T15:10:02.453Z",
          "content": "<p>For each token, we defined a fully-connected layer with an output size of 1 and concatenated them to compute the loss.<br>\nI think it is difficult to understand in words, so a simplified pseudo code is shown below.</p>\n<pre><code> (nn.Module):\n     ():\n        self.transformer = AutoModel.from_pretrained(model_name, config=config)\n        self.output_cohesion = nn.Linear(config.hidden_size, )\n        self.output_syntax = nn.Linear(config.hidden_size, )\n        self.output_vocabulary = nn.Linear(config.hidden_size, )\n        self.output_phraseology = nn.Linear(config.hidden_size, )\n        self.output_grammar = nn.Linear(config.hidden_size, )\n        self.output_conventions = nn.Linear(config.hidden_size, )\n\n     ():\n        transformer_out = self.transformer(input_ids=ids, attention_mask=mask)\n        sequence_output = transformer_out.last_hidden_state\n\n        logits_cohesion = self.output_cohesion(sequence_output[:, , :])\n        logits_syntax = self.output_syntax(sequence_output[:, , :])\n        logits_vocabulary = self.output_vocabulary(sequence_output[:, , :])\n        logits_phraseology = self.output_phraseology(sequence_output[:, , :])\n        logits_grammar = self.output_grammar(sequence_output[:, , :])\n        logits_conventions = self.output_conventions(sequence_output[:, , :])\n\n        logits = torch.cat([logits_cohesion, logits_syntax, logits_vocabulary, logits_phraseology, logits_grammar, logits_conventions], axis=)\n        loss = self.loss(logits, targets)\n</code></pre>",
          "rawMarkdown": "For each token, we defined a fully-connected layer with an output size of 1 and concatenated them to compute the loss.\nI think it is difficult to understand in words, so a simplified pseudo code is shown below.\n\n```python\nclass FeedbackModel(nn.Module):\n    def __init__(self):\n        self.transformer = AutoModel.from_pretrained(model_name, config=config)\n        self.output_cohesion = nn.Linear(config.hidden_size, 1)\n        self.output_syntax = nn.Linear(config.hidden_size, 1)\n        self.output_vocabulary = nn.Linear(config.hidden_size, 1)\n        self.output_phraseology = nn.Linear(config.hidden_size, 1)\n        self.output_grammar = nn.Linear(config.hidden_size, 1)\n        self.output_conventions = nn.Linear(config.hidden_size, 1)\n\n    def forward(self, ids, mask, targets=None):\n        transformer_out = self.transformer(input_ids=ids, attention_mask=mask)\n        sequence_output = transformer_out.last_hidden_state\n        \n        logits_cohesion = self.output_cohesion(sequence_output[:, 0, :])\n        logits_syntax = self.output_syntax(sequence_output[:, 1, :])\n        logits_vocabulary = self.output_vocabulary(sequence_output[:, 2, :])\n        logits_phraseology = self.output_phraseology(sequence_output[:, 3, :])\n        logits_grammar = self.output_grammar(sequence_output[:, 4, :])\n        logits_conventions = self.output_conventions(sequence_output[:, 5, :])\n        \n        logits = torch.cat([logits_cohesion, logits_syntax, logits_vocabulary, logits_phraseology, logits_grammar, logits_conventions], axis=1)\n        loss = self.loss(logits, targets)\n```\n",
          "votes": 1
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T20:04:11Z",
  "competition_id": 38321,
  "competition_slug": "feedback-prize-english-language-learning",
  "team_id": 9102712,
  "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/yumeneko-21th-place-solution"
}