{
  "id": 453896,
  "title": "14th place solution: Weighted ensemble of distilroberta-base, mistral-7b",
  "content": "<p>Firstly, huge thanks to H2O.ai for organizing this competition. It gave me a great opportunity to learn and experiment with different LLMs!</p>\n<p><strong>Approach</strong></p>\n<p>The solution I tried is rather simple, basically a weighted ensemble of 2 model's probabilities - distilroberta_base and Mistral-7B. Many thanks to <a href=\"https://www.kaggle.com/phanisrikanth\" target=\"_blank\">@phanisrikanth</a> for sharing insights about the impressive performance of decoder only models for classification!</p>\n<p><strong>Things that worked for me</strong></p>\n<ul>\n<li>Using GroupKFold by question helped in cross validation</li>\n<li>Roberta and distilroberta model family showed better logloss</li>\n<li>Ensembling with decoder only model such as Mistral-7B, giving slightly higher weight to Mistral-7B (0.7-0.8)  and Distilroberta_base (0.3-0.2) showed better performance</li>\n</ul>\n<p><strong>Things that didn't work for me</strong></p>\n<ul>\n<li>Deberta family, Electra models couldn't be improved beyond 1.5-1.6 logloss</li>\n<li>Freezing parts of the model such as embeddings layers/ initial encoder layers didn't help</li>\n<li>Stacking encoder model output probabilities, hand crafted features such as word count, punctuation based features etc. with XGB seemed to worsen the logloss, maybe these features were not relevant…..</li>\n</ul>\n<p><strong>Inference code</strong><br>\n<a href=\"https://www.kaggle.com/code/rushali2406/14th-place-ensemble-distilroberta-base-mistral7b\" target=\"_blank\">https://www.kaggle.com/code/rushali2406/14th-place-ensemble-distilroberta-base-mistral7b</a></p>\n<p>Overall, this was an enriching experience for me and I hope to continue participating in NLP competitions!</p>",
  "comments": [],
  "comment_count": 0,
  "messages": [
    {
      "id": 2517008,
      "postDate": "2023-11-08T06:39:17.503Z",
      "content": "<p>Firstly, huge thanks to H2O.ai for organizing this competition. It gave me a great opportunity to learn and experiment with different LLMs!</p>\n<p><strong>Approach</strong></p>\n<p>The solution I tried is rather simple, basically a weighted ensemble of 2 model's probabilities - distilroberta_base and Mistral-7B. Many thanks to <a href=\"https://www.kaggle.com/phanisrikanth\" target=\"_blank\">@phanisrikanth</a> for sharing insights about the impressive performance of decoder only models for classification!</p>\n<p><strong>Things that worked for me</strong></p>\n<ul>\n<li>Using GroupKFold by question helped in cross validation</li>\n<li>Roberta and distilroberta model family showed better logloss</li>\n<li>Ensembling with decoder only model such as Mistral-7B, giving slightly higher weight to Mistral-7B (0.7-0.8)  and Distilroberta_base (0.3-0.2) showed better performance</li>\n</ul>\n<p><strong>Things that didn't work for me</strong></p>\n<ul>\n<li>Deberta family, Electra models couldn't be improved beyond 1.5-1.6 logloss</li>\n<li>Freezing parts of the model such as embeddings layers/ initial encoder layers didn't help</li>\n<li>Stacking encoder model output probabilities, hand crafted features such as word count, punctuation based features etc. with XGB seemed to worsen the logloss, maybe these features were not relevant…..</li>\n</ul>\n<p><strong>Inference code</strong><br>\n<a href=\"https://www.kaggle.com/code/rushali2406/14th-place-ensemble-distilroberta-base-mistral7b\" target=\"_blank\">https://www.kaggle.com/code/rushali2406/14th-place-ensemble-distilroberta-base-mistral7b</a></p>\n<p>Overall, this was an enriching experience for me and I hope to continue participating in NLP competitions!</p>",
      "rawMarkdown": "Firstly, huge thanks to H2O.ai for organizing this competition. It gave me a great opportunity to learn and experiment with different LLMs!\n\n**Approach**\n\nThe solution I tried is rather simple, basically a weighted ensemble of 2 model's probabilities - distilroberta_base and Mistral-7B. Many thanks to @phanisrikanth for sharing insights about the impressive performance of decoder only models for classification!\n\n**Things that worked for me**\n- Using GroupKFold by question helped in cross validation\n- Roberta and distilroberta model family showed better logloss\n- Ensembling with decoder only model such as Mistral-7B, giving slightly higher weight to Mistral-7B (0.7-0.8)  and Distilroberta_base (0.3-0.2) showed better performance\n\n**Things that didn't work for me**\n- Deberta family, Electra models couldn't be improved beyond 1.5-1.6 logloss\n- Freezing parts of the model such as embeddings layers/ initial encoder layers didn't help\n- Stacking encoder model output probabilities, hand crafted features such as word count, punctuation based features etc. with XGB seemed to worsen the logloss, maybe these features were not relevant.....\n\n**Inference code**\nhttps://www.kaggle.com/code/rushali2406/14th-place-ensemble-distilroberta-base-mistral7b\n\nOverall, this was an enriching experience for me and I hope to continue participating in NLP competitions!",
      "votes": 11
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T21:54:46Z",
  "competition_id": 63021,
  "competition_slug": "h2oai-predict-the-llm",
  "team_id": 11133673,
  "solution_url": "/competitions/h2oai-predict-the-llm/writeups/rush24-14th-place-solution-weighted-ensemble-of-di"
}