{
  "id": 369577,
  "title": "221st place solution",
  "content": "<p>Congratulations to all competitors! Here is s a summary of my solution.</p>\n<h1>Overview</h1>\n<ul>\n<li><p>My model is ensemble of the two published notebooks and my own model:</p>\n<ul>\n<li>model1:  <a href=\"https://www.kaggle.com/code/jingwora1/fb3-deberta-family-inference-weight-tune\" target=\"_blank\">https://www.kaggle.com/code/jingwora1/fb3-deberta-family-inference-weight-tune</a> (by <a href=\"https://www.kaggle.com/jingwora1\" target=\"_blank\">@jingwora1</a>)</li>\n<li>model2: <a href=\"https://www.kaggle.com/code/quangphm/lb-0-43-simple-ensemble-deberta-base-svr\" target=\"_blank\">https://www.kaggle.com/code/quangphm/lb-0-43-simple-ensemble-deberta-base-svr</a> (by <a href=\"https://www.kaggle.com/quangphm\" target=\"_blank\">@quangphm</a>)<ul>\n<li>Note: The Fine-tuned DeBERTa models are excluded. The SVR side only uses deberta_large_mnli, roberta_base, roberta_large, xlnet_base and xlnet_large.</li></ul></li>\n<li>model3:  DeBERTa-v3-base/large (4 fold, seed=2024, Layer-wise Decreasing Layer Rate)</li></ul></li>\n<li><p>Code: <a href=\"https://www.kaggle.com/code/fx6300/ensemble-deberta-and-svr/notebook?scriptVersionId=112096434\" target=\"_blank\">https://www.kaggle.com/code/fx6300/ensemble-deberta-and-svr/notebook?scriptVersionId=112096434</a></p></li>\n</ul>\n<h1>Ensemble Weighting Strategy</h1>\n<ul>\n<li><p>I didn't employ any optimization algorithms (e.g., Nelder-Mead) since I didn't have good validation set and this kind of algorithms work well on CV but don't always work on private LB.</p></li>\n<li><p>Ensemble weights are:</p>\n<ul>\n<li>model1: 0.8</li>\n<li>model2: 0.1</li>\n<li>model3: 0.1</li></ul></li>\n</ul>\n<h1>What Worked</h1>\n<ul>\n<li>Layer-wise Decreasing Layer Rate<ul>\n<li>During this competition, I was highly influenced by the paper titled \"How to fine-tune bert for text classification?.\" [0].  This paper also reports the effectiveness of Layer-wise Decreasing Layer Rate (See 5.3.4)</li></ul></li>\n</ul>\n<h1>What Didn't Work</h1>\n<ul>\n<li>Metric Learning (Triplet Loss)<ul>\n<li>I couldn't find a proper algorithm for applying metric learning on regression tasks</li></ul></li>\n<li>Use flan-t5 as an encoder</li>\n<li>Use t5 as an encoder</li>\n<li>Further Pre-training<ul>\n<li>The above mentioned paper [0] also reports that  Further Pre-training is effective when doing it with in-domain corpus.</li>\n<li>However, it didn't work well in my case.</li>\n<li>If you are interested in further pre-training, check it:  <a href=\"https://www.kaggle.com/code/fx6300/further-pre-train-deberta\" target=\"_blank\">https://www.kaggle.com/code/fx6300/further-pre-train-deberta</a></li></ul></li>\n<li>Re-assign score when given target (e.g., coherence) has score x.5<ul>\n<li>I thought that score x.5 targets are unreliable.</li>\n<li>For example, If given coherence score was 4.5, this means annotator A scored this as 5.0 and annotator B scored this as 4.0.  </li>\n<li>I thought uncontroversial instances don't cause this kind of discrepancy.</li></ul></li>\n<li>Use sentence generation probability (i.e. LM score of GPT-2) as a feature of a grammar score estimator</li>\n<li>Use Random Forest as a vocabulary score estimator</li>\n</ul>\n<h1>References</h1>\n<ul>\n<li>[0] Sun, Chi, et al. \"How to fine-tune bert for text classification?.\" China national conference on Chinese computational linguistics. Springer, Cham, 2019.<ul>\n<li>paper: <a href=\"https://arxiv.org/pdf/1905.05583.pdf\" target=\"_blank\">https://arxiv.org/pdf/1905.05583.pdf</a></li></ul></li>\n</ul>",
  "comments": [],
  "comment_count": 0,
  "messages": [
    {
      "id": 2050157,
      "postDate": "2022-11-30T15:06:30.937Z",
      "content": "<p>Congratulations to all competitors! Here is s a summary of my solution.</p>\n<h1>Overview</h1>\n<ul>\n<li><p>My model is ensemble of the two published notebooks and my own model:</p>\n<ul>\n<li>model1:  <a href=\"https://www.kaggle.com/code/jingwora1/fb3-deberta-family-inference-weight-tune\" target=\"_blank\">https://www.kaggle.com/code/jingwora1/fb3-deberta-family-inference-weight-tune</a> (by <a href=\"https://www.kaggle.com/jingwora1\" target=\"_blank\">@jingwora1</a>)</li>\n<li>model2: <a href=\"https://www.kaggle.com/code/quangphm/lb-0-43-simple-ensemble-deberta-base-svr\" target=\"_blank\">https://www.kaggle.com/code/quangphm/lb-0-43-simple-ensemble-deberta-base-svr</a> (by <a href=\"https://www.kaggle.com/quangphm\" target=\"_blank\">@quangphm</a>)<ul>\n<li>Note: The Fine-tuned DeBERTa models are excluded. The SVR side only uses deberta_large_mnli, roberta_base, roberta_large, xlnet_base and xlnet_large.</li></ul></li>\n<li>model3:  DeBERTa-v3-base/large (4 fold, seed=2024, Layer-wise Decreasing Layer Rate)</li></ul></li>\n<li><p>Code: <a href=\"https://www.kaggle.com/code/fx6300/ensemble-deberta-and-svr/notebook?scriptVersionId=112096434\" target=\"_blank\">https://www.kaggle.com/code/fx6300/ensemble-deberta-and-svr/notebook?scriptVersionId=112096434</a></p></li>\n</ul>\n<h1>Ensemble Weighting Strategy</h1>\n<ul>\n<li><p>I didn't employ any optimization algorithms (e.g., Nelder-Mead) since I didn't have good validation set and this kind of algorithms work well on CV but don't always work on private LB.</p></li>\n<li><p>Ensemble weights are:</p>\n<ul>\n<li>model1: 0.8</li>\n<li>model2: 0.1</li>\n<li>model3: 0.1</li></ul></li>\n</ul>\n<h1>What Worked</h1>\n<ul>\n<li>Layer-wise Decreasing Layer Rate<ul>\n<li>During this competition, I was highly influenced by the paper titled \"How to fine-tune bert for text classification?.\" [0].  This paper also reports the effectiveness of Layer-wise Decreasing Layer Rate (See 5.3.4)</li></ul></li>\n</ul>\n<h1>What Didn't Work</h1>\n<ul>\n<li>Metric Learning (Triplet Loss)<ul>\n<li>I couldn't find a proper algorithm for applying metric learning on regression tasks</li></ul></li>\n<li>Use flan-t5 as an encoder</li>\n<li>Use t5 as an encoder</li>\n<li>Further Pre-training<ul>\n<li>The above mentioned paper [0] also reports that  Further Pre-training is effective when doing it with in-domain corpus.</li>\n<li>However, it didn't work well in my case.</li>\n<li>If you are interested in further pre-training, check it:  <a href=\"https://www.kaggle.com/code/fx6300/further-pre-train-deberta\" target=\"_blank\">https://www.kaggle.com/code/fx6300/further-pre-train-deberta</a></li></ul></li>\n<li>Re-assign score when given target (e.g., coherence) has score x.5<ul>\n<li>I thought that score x.5 targets are unreliable.</li>\n<li>For example, If given coherence score was 4.5, this means annotator A scored this as 5.0 and annotator B scored this as 4.0.  </li>\n<li>I thought uncontroversial instances don't cause this kind of discrepancy.</li></ul></li>\n<li>Use sentence generation probability (i.e. LM score of GPT-2) as a feature of a grammar score estimator</li>\n<li>Use Random Forest as a vocabulary score estimator</li>\n</ul>\n<h1>References</h1>\n<ul>\n<li>[0] Sun, Chi, et al. \"How to fine-tune bert for text classification?.\" China national conference on Chinese computational linguistics. Springer, Cham, 2019.<ul>\n<li>paper: <a href=\"https://arxiv.org/pdf/1905.05583.pdf\" target=\"_blank\">https://arxiv.org/pdf/1905.05583.pdf</a></li></ul></li>\n</ul>",
      "rawMarkdown": "Congratulations to all competitors! Here is s a summary of my solution.\n\n# Overview\n\n+ My model is ensemble of the two published notebooks and my own model:\n  + model1:  https://www.kaggle.com/code/jingwora1/fb3-deberta-family-inference-weight-tune (by @jingwora1)\n  + model2: https://www.kaggle.com/code/quangphm/lb-0-43-simple-ensemble-deberta-base-svr (by @quangphm)\n     + Note: The Fine-tuned DeBERTa models are excluded. The SVR side only uses deberta_large_mnli, roberta_base, roberta_large, xlnet_base and xlnet_large.\n  + model3:  DeBERTa-v3-base/large (4 fold, seed=2024, Layer-wise Decreasing Layer Rate)\n\n+ Code: https://www.kaggle.com/code/fx6300/ensemble-deberta-and-svr/notebook?scriptVersionId=112096434\n\n# Ensemble Weighting Strategy\n\n+ I didn't employ any optimization algorithms (e.g., Nelder-Mead) since I didn't have good validation set and this kind of algorithms work well on CV but don't always work on private LB.\n\n+ Ensemble weights are:\n  + model1: 0.8\n  + model2: 0.1\n  + model3: 0.1\n\n# What Worked\n\n+ Layer-wise Decreasing Layer Rate\n  + During this competition, I was highly influenced by the paper titled \"How to fine-tune bert for text classification?.\" [0].  This paper also reports the effectiveness of Layer-wise Decreasing Layer Rate (See 5.3.4)\n\n# What Didn't Work\n\n+ Metric Learning (Triplet Loss)\n  + I couldn't find a proper algorithm for applying metric learning on regression tasks\n+ Use flan-t5 as an encoder\n+ Use t5 as an encoder\n+ Further Pre-training\n  + The above mentioned paper [0] also reports that  Further Pre-training is effective when doing it with in-domain corpus.\n  + However, it didn't work well in my case.\n  + If you are interested in further pre-training, check it:  https://www.kaggle.com/code/fx6300/further-pre-train-deberta\n+ Re-assign score when given target (e.g., coherence) has score x.5\n  + I thought that score x.5 targets are unreliable.\n  + For example, If given coherence score was 4.5, this means annotator A scored this as 5.0 and annotator B scored this as 4.0.  \n  + I thought uncontroversial instances don't cause this kind of discrepancy.\n+ Use sentence generation probability (i.e. LM score of GPT-2) as a feature of a grammar score estimator\n+ Use Random Forest as a vocabulary score estimator\n\n# References\n\n+ [0] Sun, Chi, et al. \"How to fine-tune bert for text classification?.\" China national conference on Chinese computational linguistics. Springer, Cham, 2019.\n   + paper: https://arxiv.org/pdf/1905.05583.pdf\n",
      "votes": 5
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T20:01:09Z",
  "competition_id": 38321,
  "competition_slug": "feedback-prize-english-language-learning",
  "team_id": 9094024,
  "solution_url": "/competitions/feedback-prize-english-language-learning/writeups/fx-6300-221st-place-solution"
}