{
  "id": 453639,
  "title": "First Place Solution",
  "content": "<p>Firstly, I want to  thank  H2o for organizing this great competition, it was a great opportunity  to delve into the world of LLMs and try different things.</p>\n<h1>Summary</h1>\n<p>My solution is an ensemble of 3 models (Mistral-7b , Zephyr-7b and Debertav3-large) feed into a 2nd Stage stacking Model with different features . </p>\n<h2>LLMs</h2>\n<p>I was really amazed by the performance of decoder only models in classification tasks. Due to hardware limitations (a single 16 Gb gpu) I was only able to experiment with a 4-bit quantized and  Low Rank Adaptation (LoRa).I experimented with Mistral-7b and Zephyr-7b.</p>\n<h2>Encoder only Models</h2>\n<p>input: Question + sep + [R_Start] response  [R_end] + other responses.<br>\nfor pooling I only pool over tokens between [R_Start] and [R_end]  , this worked for my team in <a href=\"https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/446542\" target=\"_blank\">commonlit</a>.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8785809%2F1ca83f6f38dd05485718eb090e8b5aab%2FScreen%20Shot%202023-11-07%20at%206.59.20%20AM.png?generation=1699336770313372&amp;alt=media\" alt=\"\"></p>\n<h2>Second Stage Model</h2>\n<p>Our first stage models do not take into consideration that for each question we have the responses of the 7 LLMs.So as features for each row: (question, response) I added predictions of 1st stage models with prediction of other question response pairs along with some interaction features feed into LGBM/XGB, this was the secret sauce of my solution(+0.3/0.2 in cv/lb).</p>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>cv</th>\n<th>public</th>\n<th>private</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>deberta-v3-large</td>\n<td>1.08</td>\n<td>0.97</td>\n<td>0.916</td>\n</tr>\n<tr>\n<td>mistral-7b</td>\n<td>0.9592</td>\n<td>0.754</td>\n<td>0.802</td>\n</tr>\n<tr>\n<td>Zephyr-7b</td>\n<td>0.936</td>\n<td>0.787</td>\n<td>0.79</td>\n</tr>\n<tr>\n<td>+ 2nd stage</td>\n<td>0.607</td>\n<td>0.497</td>\n<td>0.509</td>\n</tr>\n</tbody>\n</table>\n<h1>What Didn't Work</h1>\n<p>I tried to predict the target for the 7 responses together with different approaches, I even tried a token classification approach, but this unfortunately didn't work.</p>\n<h2>Training &amp; Inference code</h2>\n<ul>\n<li>deberta-v3: <a href=\"https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-deberta-v3-large/notebook\" target=\"_blank\">https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-deberta-v3-large/notebook</a></li>\n<li>Mistral-7b: <a href=\"https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-mistral-7b-quantized-lora/notebook\" target=\"_blank\">https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-mistral-7b-quantized-lora/notebook</a></li>\n<li>Zephyr-7b: <a href=\"https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-zephyr-7b-quantized-lora/notebook\" target=\"_blank\">https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-zephyr-7b-quantized-lora/notebook</a></li>\n<li>2nd stage: <a href=\"https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-2nd-stage-stacking-model/notebook\" target=\"_blank\">https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-2nd-stage-stacking-model/notebook</a></li>\n</ul>",
  "comments": [
    {
      "id": 2517803,
      "author_name": "Raheem Nasirudeen",
      "author_url": "",
      "post_date": "2023-11-08T18:26:33.090000",
      "content": "<p>Congratulations bro. Great work</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2517412,
      "author_name": "Chiheb Mab",
      "author_url": "",
      "post_date": "2023-11-08T13:06:43.917000",
      "content": "<p>Great work <a href=\"https://www.kaggle.com/ihebch\" target=\"_blank\">@ihebch</a> Congratulation </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2516919,
      "author_name": "binga",
      "author_url": "",
      "post_date": "2023-11-08T04:35:48.763000",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/ihebch\" target=\"_blank\">@ihebch</a> for the fantastic win!</p>\n<p>It's amazing to see the second stage model giving such a massive boost in performance. This is the part I couldn't get it to working and a learning opportunity for next time.</p>\n<p>Zephyr performed significantly worse for me than the other models. I'll go back and try a few runs. Thank you for showing that it's possible to make it work.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2516945,
          "author_name": "Reacher",
          "author_url": "",
          "post_date": "2023-11-08T05:10:30.153000",
          "content": "<p>Thank you, <a href=\"https://www.kaggle.com/phanisrikanth\" target=\"_blank\">@phanisrikanth</a> , for your contributions throughout the competition.<br>\nindeed, second stage was the key in this competition for me.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2516810,
      "author_name": "Cody_Null",
      "author_url": "",
      "post_date": "2023-11-08T01:35:06.403000",
      "content": "<p>Great work <a href=\"https://www.kaggle.com/ihebch\" target=\"_blank\">@ihebch</a> very cool to see you implement pooling strategies similar to our commonlit solution! Very happy to see you win this!</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2516947,
          "author_name": "Reacher",
          "author_url": "",
          "post_date": "2023-11-08T05:11:01.200000",
          "content": "<p>Thank you <a href=\"https://www.kaggle.com/cody11null\" target=\"_blank\">@cody11null</a> !</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2516101,
      "author_name": "RaviT",
      "author_url": "",
      "post_date": "2023-11-07T13:27:29.737000",
      "content": "<p><a href=\"https://www.kaggle.com/reacher\" target=\"_blank\">@reacher</a> - Thanks for the write-up. When you appended other 6 responses, did you add any separator to distinguish the responses?</p>\n<p>input: Question + sep + [R_Start] response [R_end] + <strong><em>other responses</em></strong>.</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2516103,
          "author_name": "Reacher",
          "author_url": "",
          "post_date": "2023-11-07T13:29:55.587000",
          "content": "<p>yes, responses are separated by a [SEP] token.</p>",
          "votes": 0,
          "replies": [
            {
              "id": 2516782,
              "author_name": "RaviT",
              "author_url": "",
              "post_date": "2023-11-08T00:37:19.730000",
              "content": "<p>Thanks <a href=\"https://www.kaggle.com/ihebch\" target=\"_blank\">@ihebch</a>. The Mistral and Zephyr links are broken. Would you mind updating those?</p>\n<p>Mistral-7b: <a href=\"https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-mistral-7b-quantized-lora\" target=\"_blank\">https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-mistral-7b-quantized-lora</a><br>\nZephyr-7b: <a href=\"https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-zephyr-7b-quantized-lora\" target=\"_blank\">https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-zephyr-7b-quantized-lora</a></p>",
              "votes": 1,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2515726,
      "author_name": "Catharis",
      "author_url": "",
      "post_date": "2023-11-07T07:10:02.880000",
      "content": "<p>Well done!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2517458,
      "author_name": "Mohamed Eltayeb",
      "author_url": "",
      "post_date": "2023-11-08T13:49:08.747000",
      "content": "<p>Well done <a href=\"https://www.kaggle.com/ihebch\" target=\"_blank\">@ihebch</a> 👏 <br>\nAmazing work as always!</p>",
      "votes": 2,
      "replies": []
    }
  ],
  "comment_count": 11,
  "messages": [
    {
      "id": 2515670,
      "postDate": "2023-11-07T05:25:30.950Z",
      "content": "<p>Firstly, I want to  thank  H2o for organizing this great competition, it was a great opportunity  to delve into the world of LLMs and try different things.</p>\n<h1>Summary</h1>\n<p>My solution is an ensemble of 3 models (Mistral-7b , Zephyr-7b and Debertav3-large) feed into a 2nd Stage stacking Model with different features . </p>\n<h2>LLMs</h2>\n<p>I was really amazed by the performance of decoder only models in classification tasks. Due to hardware limitations (a single 16 Gb gpu) I was only able to experiment with a 4-bit quantized and  Low Rank Adaptation (LoRa).I experimented with Mistral-7b and Zephyr-7b.</p>\n<h2>Encoder only Models</h2>\n<p>input: Question + sep + [R_Start] response  [R_end] + other responses.<br>\nfor pooling I only pool over tokens between [R_Start] and [R_end]  , this worked for my team in <a href=\"https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/446542\" target=\"_blank\">commonlit</a>.<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8785809%2F1ca83f6f38dd05485718eb090e8b5aab%2FScreen%20Shot%202023-11-07%20at%206.59.20%20AM.png?generation=1699336770313372&amp;alt=media\" alt=\"\"></p>\n<h2>Second Stage Model</h2>\n<p>Our first stage models do not take into consideration that for each question we have the responses of the 7 LLMs.So as features for each row: (question, response) I added predictions of 1st stage models with prediction of other question response pairs along with some interaction features feed into LGBM/XGB, this was the secret sauce of my solution(+0.3/0.2 in cv/lb).</p>\n<table>\n<thead>\n<tr>\n<th>model</th>\n<th>cv</th>\n<th>public</th>\n<th>private</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>deberta-v3-large</td>\n<td>1.08</td>\n<td>0.97</td>\n<td>0.916</td>\n</tr>\n<tr>\n<td>mistral-7b</td>\n<td>0.9592</td>\n<td>0.754</td>\n<td>0.802</td>\n</tr>\n<tr>\n<td>Zephyr-7b</td>\n<td>0.936</td>\n<td>0.787</td>\n<td>0.79</td>\n</tr>\n<tr>\n<td>+ 2nd stage</td>\n<td>0.607</td>\n<td>0.497</td>\n<td>0.509</td>\n</tr>\n</tbody>\n</table>\n<h1>What Didn't Work</h1>\n<p>I tried to predict the target for the 7 responses together with different approaches, I even tried a token classification approach, but this unfortunately didn't work.</p>\n<h2>Training &amp; Inference code</h2>\n<ul>\n<li>deberta-v3: <a href=\"https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-deberta-v3-large/notebook\" target=\"_blank\">https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-deberta-v3-large/notebook</a></li>\n<li>Mistral-7b: <a href=\"https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-mistral-7b-quantized-lora/notebook\" target=\"_blank\">https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-mistral-7b-quantized-lora/notebook</a></li>\n<li>Zephyr-7b: <a href=\"https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-zephyr-7b-quantized-lora/notebook\" target=\"_blank\">https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-zephyr-7b-quantized-lora/notebook</a></li>\n<li>2nd stage: <a href=\"https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-2nd-stage-stacking-model/notebook\" target=\"_blank\">https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-2nd-stage-stacking-model/notebook</a></li>\n</ul>",
      "rawMarkdown": "Firstly, I want to  thank  H2o for organizing this great competition, it was a great opportunity  to delve into the world of LLMs and try different things.\n# Summary\nMy solution is an ensemble of 3 models (Mistral-7b , Zephyr-7b and Debertav3-large) feed into a 2nd Stage stacking Model with different features . \n## LLMs\nI was really amazed by the performance of decoder only models in classification tasks. Due to hardware limitations (a single 16 Gb gpu) I was only able to experiment with a 4-bit quantized and  Low Rank Adaptation (LoRa).I experimented with Mistral-7b and Zephyr-7b.\n## Encoder only Models\ninput: Question + sep + [R_Start] response  [R_end] + other responses.\nfor pooling I only pool over tokens between [R_Start] and [R_end]  , this worked for my team in [commonlit](https://www.kaggle.com/competitions/commonlit-evaluate-student-summaries/discussion/446542).\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8785809%2F1ca83f6f38dd05485718eb090e8b5aab%2FScreen%20Shot%202023-11-07%20at%206.59.20%20AM.png?generation=1699336770313372&alt=media)\n\n## Second Stage Model\nOur first stage models do not take into consideration that for each question we have the responses of the 7 LLMs.So as features for each row: (question, response) I added predictions of 1st stage models with prediction of other question response pairs along with some interaction features feed into LGBM/XGB, this was the secret sauce of my solution(+0.3/0.2 in cv/lb).\n| model | cv | public  | private\n| --- | --- |\n| deberta-v3-large | 1.08 | 0.97 | 0.916\n| mistral-7b | 0.9592 | 0.754 | 0.802\n| Zephyr-7b | 0.936 | 0.787 | 0.79\n| + 2nd stage | 0.607 | 0.497 | 0.509\n\n# What Didn't Work\nI tried to predict the target for the 7 responses together with different approaches, I even tried a token classification approach, but this unfortunately didn't work.\n## Training & Inference code\n- deberta-v3: https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-deberta-v3-large/notebook\n- Mistral-7b: https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-mistral-7b-quantized-lora/notebook\n- Zephyr-7b: https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-zephyr-7b-quantized-lora/notebook\n- 2nd stage: https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-2nd-stage-stacking-model/notebook\n\n",
      "votes": 40
    },
    {
      "id": 2517803,
      "postDate": "2023-11-08T18:26:33.090Z",
      "content": "<p>Congratulations bro. Great work</p>",
      "rawMarkdown": "Congratulations bro. Great work",
      "votes": 1
    },
    {
      "id": 2517412,
      "postDate": "2023-11-08T13:06:43.917Z",
      "content": "<p>Great work <a href=\"https://www.kaggle.com/ihebch\" target=\"_blank\">@ihebch</a> Congratulation </p>",
      "rawMarkdown": "Great work @ihebch Congratulation ",
      "votes": 1
    },
    {
      "id": 2516919,
      "postDate": "2023-11-08T04:35:48.763Z",
      "content": "<p>Congratulations <a href=\"https://www.kaggle.com/ihebch\" target=\"_blank\">@ihebch</a> for the fantastic win!</p>\n<p>It's amazing to see the second stage model giving such a massive boost in performance. This is the part I couldn't get it to working and a learning opportunity for next time.</p>\n<p>Zephyr performed significantly worse for me than the other models. I'll go back and try a few runs. Thank you for showing that it's possible to make it work.</p>",
      "rawMarkdown": "Congratulations @ihebch for the fantastic win!\n\nIt's amazing to see the second stage model giving such a massive boost in performance. This is the part I couldn't get it to working and a learning opportunity for next time.\n\nZephyr performed significantly worse for me than the other models. I'll go back and try a few runs. Thank you for showing that it's possible to make it work.",
      "votes": 1,
      "replies": [
        {
          "id": 2516945,
          "postDate": "2023-11-08T05:10:30.153Z",
          "content": "<p>Thank you, <a href=\"https://www.kaggle.com/phanisrikanth\" target=\"_blank\">@phanisrikanth</a> , for your contributions throughout the competition.<br>\nindeed, second stage was the key in this competition for me.</p>",
          "rawMarkdown": "Thank you, @phanisrikanth , for your contributions throughout the competition.\nindeed, second stage was the key in this competition for me.",
          "votes": 1
        }
      ]
    },
    {
      "id": 2516810,
      "postDate": "2023-11-08T01:35:06.403Z",
      "content": "<p>Great work <a href=\"https://www.kaggle.com/ihebch\" target=\"_blank\">@ihebch</a> very cool to see you implement pooling strategies similar to our commonlit solution! Very happy to see you win this!</p>",
      "rawMarkdown": "Great work @ihebch very cool to see you implement pooling strategies similar to our commonlit solution! Very happy to see you win this!",
      "votes": 1,
      "replies": [
        {
          "id": 2516947,
          "postDate": "2023-11-08T05:11:01.200Z",
          "content": "<p>Thank you <a href=\"https://www.kaggle.com/cody11null\" target=\"_blank\">@cody11null</a> !</p>",
          "rawMarkdown": "Thank you @cody11null !",
          "votes": 1
        }
      ]
    },
    {
      "id": 2516101,
      "postDate": "2023-11-07T13:27:29.737Z",
      "content": "<p><a href=\"https://www.kaggle.com/reacher\" target=\"_blank\">@reacher</a> - Thanks for the write-up. When you appended other 6 responses, did you add any separator to distinguish the responses?</p>\n<p>input: Question + sep + [R_Start] response [R_end] + <strong><em>other responses</em></strong>.</p>",
      "rawMarkdown": "@reacher - Thanks for the write-up. When you appended other 6 responses, did you add any separator to distinguish the responses?\n\ninput: Question + sep + [R_Start] response [R_end] + ***other responses***.",
      "votes": 1,
      "replies": [
        {
          "id": 2516103,
          "postDate": "2023-11-07T13:29:55.587Z",
          "content": "<p>yes, responses are separated by a [SEP] token.</p>",
          "rawMarkdown": "yes, responses are separated by a [SEP] token.",
          "replies": [
            {
              "id": 2516782,
              "postDate": "2023-11-08T00:37:19.730Z",
              "content": "<p>Thanks <a href=\"https://www.kaggle.com/ihebch\" target=\"_blank\">@ihebch</a>. The Mistral and Zephyr links are broken. Would you mind updating those?</p>\n<p>Mistral-7b: <a href=\"https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-mistral-7b-quantized-lora\" target=\"_blank\">https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-mistral-7b-quantized-lora</a><br>\nZephyr-7b: <a href=\"https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-zephyr-7b-quantized-lora\" target=\"_blank\">https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-zephyr-7b-quantized-lora</a></p>",
              "rawMarkdown": "Thanks @ihebch. The Mistral and Zephyr links are broken. Would you mind updating those?\n\nMistral-7b: https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-mistral-7b-quantized-lora\nZephyr-7b: https://www.kaggle.com/code/ihebch/h2o-ai-predict-the-llm-zephyr-7b-quantized-lora",
              "votes": 1
            }
          ]
        }
      ]
    },
    {
      "id": 2515726,
      "postDate": "2023-11-07T07:10:02.880Z",
      "content": "<p>Well done!</p>",
      "rawMarkdown": "Well done!",
      "votes": 1
    },
    {
      "id": 2517458,
      "postDate": "2023-11-08T13:49:08.747Z",
      "content": "<p>Well done <a href=\"https://www.kaggle.com/ihebch\" target=\"_blank\">@ihebch</a> 👏 <br>\nAmazing work as always!</p>",
      "rawMarkdown": "Well done @ihebch 👏 \nAmazing work as always!",
      "votes": 2
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T21:54:00Z",
  "competition_id": 63021,
  "competition_slug": "h2oai-predict-the-llm",
  "team_id": 11129919,
  "solution_url": "/competitions/h2oai-predict-the-llm/writeups/reacher-first-place-solution"
}