{
  "id": 551722,
  "title": "10st Place Solution Summary",
  "content": "<p>Thanks to the organizers and Kaggle, this is my second solo gold medal🏅, which is enough to make me a Kaggle Grandmaster. Below is a brief summary of my solution.</p>\n<h2>External data</h2>\n<p>7k external data using GPT-4O.</p>\n<h2>Retrievers</h2>\n<p>Model: Qwen/Qwen2.5-14B-Instruct,Qwen/Qwen2.5-Coder-14B-Instruct<br>\nTrain detail:<br>\nfirstly train(lora) with  7k additional data and then load the lora weight,lastly fintune the train data.<br>\nCV: train with 5 folds and then merge using lora.<br>\nI integrated a total of three recall models, including the 14b 5-folds, 14b-coder 5-folds, and the full 14b 5-folds. The training code needs to be slightly modified in some Model classes from my previously open-sourced KDD code.</p>\n<h2>Rerankers</h2>\n<p>Model:Qwen32b AWQ,training with lora<br>\nGenerated approximately For ranking, I used a multiple-choice approach. <br>\nThe input consists of questions and answers, along with the top 25 options, and the output is the probabilities of these 25 options.<br>\n During training, I only optimized the probabilities of these 25 characters, using the cross-entropy loss function. In the end, my leaderboard score was achieved using only the 5-folds (7k additional data + training data) of LoRA fusion (LoRA fusion remains effective in any scenario).</p>\n<h2>Try but not worked</h2>\n<p>Distillation：I tried to distill the content of 72b, but since I didn't see any improvement offline, I didn't submit it. I might have been wrong, after all, the offline validation data is relatively limited.</p>\n<p>The first-place solution made me realize that there is still much room for optimization in my current solution, but I am also quite satisfied～</p>",
  "comments": [
    {
      "id": 3072372,
      "author_name": "wang",
      "author_url": "",
      "post_date": "2024-12-15T05:02:38.950000",
      "content": "<p>Hello,</p>\n<p>This contest was my first attempt at working with LLMs, and thanks to your discussions, I learned a lot. I sincerely appreciate your insights.<br>\nAt the same time, I’m frustrated that I couldn’t train the recall model as effectively as I hoped, so I’d like to ask you a few questions if you don’t mind.</p>\n<p>To replicate your experiments, I started by successfully training with Mistral and then moved on to Qwen.<br>\nHere are my results:</p>\n<table>\n<thead>\n<tr>\n<th>Model</th>\n<th>CV</th>\n<th>LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>SFR-Embedding-Mistral</td>\n<td>0.35</td>\n<td>0.30</td>\n</tr>\n<tr>\n<td>Qwen2.5-14B-Instruct</td>\n<td>0.45 ~ 0.55</td>\n<td>0.35 ~ 0.39</td>\n</tr>\n</tbody>\n</table>\n<p>On the other hand, I understand that the following scores were achieved without using synthetic data:</p>\n<table>\n<thead>\n<tr>\n<th>Model</th>\n<th>CV</th>\n<th>LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>SFR-Embedding-Mistral</td>\n<td>0.490</td>\n<td>0.353</td>\n</tr>\n<tr>\n<td>?</td>\n<td>0.538</td>\n<td>0.456</td>\n</tr>\n<tr>\n<td>?</td>\n<td>0.540</td>\n<td>0.466</td>\n</tr>\n<tr>\n<td>?</td>\n<td>0.551</td>\n<td>0.478</td>\n</tr>\n</tbody>\n</table>\n<p>I’d like to understand the differences and ask the following:</p>\n<ul>\n<li>How did you train your model with SFR-Embedding-Mistral and Qwen2.5-14B-Instruct?<ul>\n<li>The settings for run_mistral_cos_argu.sh</li>\n<li>The contents of LoraConfig</li>\n<li>How you split the CV</li>\n<li>How much did the scores vary depending on the parameter settings?</li></ul></li>\n<li>What specific techniques or strategies, apart from parameter tuning, did you use to improve performance?</li>\n<li>When switching from Mistral to Qwen, what parameters did you adjust?</li>\n</ul>\n<p>Sorry for lot of question but, I’d greatly appreciate it if you could share your insights. Thank you very much!</p>",
      "votes": 5,
      "replies": [
        {
          "id": 3073046,
          "author_name": "Turbo",
          "author_url": "",
          "post_date": "2024-12-16T01:30:22.693000",
          "content": "<p>I have the same questions.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 3072678,
      "author_name": "Bootstrapping",
      "author_url": "",
      "post_date": "2024-12-15T14:33:35.467000",
      "content": "<p>Thanks for sharing your solution, I would like to ask what do you mean by lora fusion, merging to LLM or multiple lora adapter fusion?</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 3072331,
      "author_name": "xianZ_waikato",
      "author_url": "",
      "post_date": "2024-12-15T03:36:13.950000",
      "content": "<p>congratulations to be a gm! <br>\nThen may i ask how u generate your external data? or any source can help a beginner to learn this skill, cheer~</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 3072877,
      "author_name": "MrSimple",
      "author_url": "",
      "post_date": "2024-12-15T17:56:10.530000",
      "content": "<p>congratulations to be a gm!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3072803,
      "author_name": "Rudi Özgen",
      "author_url": "",
      "post_date": "2024-12-15T16:39:32.160000",
      "content": "<p>Much obliged for your valuable guidance.</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 6,
  "messages": [
    {
      "id": 3072321,
      "postDate": "2024-12-15T03:14:24.560Z",
      "content": "<p>Thanks to the organizers and Kaggle, this is my second solo gold medal🏅, which is enough to make me a Kaggle Grandmaster. Below is a brief summary of my solution.</p>\n<h2>External data</h2>\n<p>7k external data using GPT-4O.</p>\n<h2>Retrievers</h2>\n<p>Model: Qwen/Qwen2.5-14B-Instruct,Qwen/Qwen2.5-Coder-14B-Instruct<br>\nTrain detail:<br>\nfirstly train(lora) with  7k additional data and then load the lora weight,lastly fintune the train data.<br>\nCV: train with 5 folds and then merge using lora.<br>\nI integrated a total of three recall models, including the 14b 5-folds, 14b-coder 5-folds, and the full 14b 5-folds. The training code needs to be slightly modified in some Model classes from my previously open-sourced KDD code.</p>\n<h2>Rerankers</h2>\n<p>Model:Qwen32b AWQ,training with lora<br>\nGenerated approximately For ranking, I used a multiple-choice approach. <br>\nThe input consists of questions and answers, along with the top 25 options, and the output is the probabilities of these 25 options.<br>\n During training, I only optimized the probabilities of these 25 characters, using the cross-entropy loss function. In the end, my leaderboard score was achieved using only the 5-folds (7k additional data + training data) of LoRA fusion (LoRA fusion remains effective in any scenario).</p>\n<h2>Try but not worked</h2>\n<p>Distillation：I tried to distill the content of 72b, but since I didn't see any improvement offline, I didn't submit it. I might have been wrong, after all, the offline validation data is relatively limited.</p>\n<p>The first-place solution made me realize that there is still much room for optimization in my current solution, but I am also quite satisfied～</p>",
      "rawMarkdown": "Thanks to the organizers and Kaggle, this is my second solo gold medal🏅, which is enough to make me a Kaggle Grandmaster. Below is a brief summary of my solution.\n## External data\n 7k external data using GPT-4O.\n## Retrievers\nModel: Qwen/Qwen2.5-14B-Instruct,Qwen/Qwen2.5-Coder-14B-Instruct\nTrain detail:\nfirstly train(lora) with  7k additional data and then load the lora weight,lastly fintune the train data.\nCV: train with 5 folds and then merge using lora.\nI integrated a total of three recall models, including the 14b 5-folds, 14b-coder 5-folds, and the full 14b 5-folds. The training code needs to be slightly modified in some Model classes from my previously open-sourced KDD code.\n## Rerankers\nModel:Qwen32b AWQ,training with lora\nGenerated approximately For ranking, I used a multiple-choice approach. \nThe input consists of questions and answers, along with the top 25 options, and the output is the probabilities of these 25 options.\n During training, I only optimized the probabilities of these 25 characters, using the cross-entropy loss function. In the end, my leaderboard score was achieved using only the 5-folds (7k additional data + training data) of LoRA fusion (LoRA fusion remains effective in any scenario).\n## Try but not worked\nDistillation：I tried to distill the content of 72b, but since I didn't see any improvement offline, I didn't submit it. I might have been wrong, after all, the offline validation data is relatively limited.\n\n\nThe first-place solution made me realize that there is still much room for optimization in my current solution, but I am also quite satisfied～",
      "votes": 44
    },
    {
      "id": 3072372,
      "postDate": "2024-12-15T05:02:38.950Z",
      "content": "<p>Hello,</p>\n<p>This contest was my first attempt at working with LLMs, and thanks to your discussions, I learned a lot. I sincerely appreciate your insights.<br>\nAt the same time, I’m frustrated that I couldn’t train the recall model as effectively as I hoped, so I’d like to ask you a few questions if you don’t mind.</p>\n<p>To replicate your experiments, I started by successfully training with Mistral and then moved on to Qwen.<br>\nHere are my results:</p>\n<table>\n<thead>\n<tr>\n<th>Model</th>\n<th>CV</th>\n<th>LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>SFR-Embedding-Mistral</td>\n<td>0.35</td>\n<td>0.30</td>\n</tr>\n<tr>\n<td>Qwen2.5-14B-Instruct</td>\n<td>0.45 ~ 0.55</td>\n<td>0.35 ~ 0.39</td>\n</tr>\n</tbody>\n</table>\n<p>On the other hand, I understand that the following scores were achieved without using synthetic data:</p>\n<table>\n<thead>\n<tr>\n<th>Model</th>\n<th>CV</th>\n<th>LB</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>SFR-Embedding-Mistral</td>\n<td>0.490</td>\n<td>0.353</td>\n</tr>\n<tr>\n<td>?</td>\n<td>0.538</td>\n<td>0.456</td>\n</tr>\n<tr>\n<td>?</td>\n<td>0.540</td>\n<td>0.466</td>\n</tr>\n<tr>\n<td>?</td>\n<td>0.551</td>\n<td>0.478</td>\n</tr>\n</tbody>\n</table>\n<p>I’d like to understand the differences and ask the following:</p>\n<ul>\n<li>How did you train your model with SFR-Embedding-Mistral and Qwen2.5-14B-Instruct?<ul>\n<li>The settings for run_mistral_cos_argu.sh</li>\n<li>The contents of LoraConfig</li>\n<li>How you split the CV</li>\n<li>How much did the scores vary depending on the parameter settings?</li></ul></li>\n<li>What specific techniques or strategies, apart from parameter tuning, did you use to improve performance?</li>\n<li>When switching from Mistral to Qwen, what parameters did you adjust?</li>\n</ul>\n<p>Sorry for lot of question but, I’d greatly appreciate it if you could share your insights. Thank you very much!</p>",
      "rawMarkdown": "Hello,\n\nThis contest was my first attempt at working with LLMs, and thanks to your discussions, I learned a lot. I sincerely appreciate your insights.\nAt the same time, I’m frustrated that I couldn’t train the recall model as effectively as I hoped, so I’d like to ask you a few questions if you don’t mind.\n\nTo replicate your experiments, I started by successfully training with Mistral and then moved on to Qwen.\nHere are my results:\n\n| Model   | CV          | LB           |\n|---------|-------------|--------------|\n| SFR-Embedding-Mistral | 0.35        | 0.30         |\n| Qwen2.5-14B-Instruct    | 0.45 ~ 0.55 | 0.35 ~ 0.39  |\n\n\nOn the other hand, I understand that the following scores were achieved without using synthetic data:\n\n| Model   | CV   | LB    |\n|---------|------|-------|\n| SFR-Embedding-Mistral | 0.490 | 0.353 |\n| ?       | 0.538 | 0.456 |\n| ?       | 0.540 | 0.466 |\n| ?       | 0.551 | 0.478 |\n\nI’d like to understand the differences and ask the following:\n\n- How did you train your model with SFR-Embedding-Mistral and Qwen2.5-14B-Instruct?\n - The settings for run_mistral_cos_argu.sh\n - The contents of LoraConfig\n - How you split the CV\n - How much did the scores vary depending on the parameter settings?\n- What specific techniques or strategies, apart from parameter tuning, did you use to improve performance?\n- When switching from Mistral to Qwen, what parameters did you adjust?\n\nSorry for lot of question but, I’d greatly appreciate it if you could share your insights. Thank you very much!",
      "votes": 5,
      "replies": [
        {
          "id": 3073046,
          "postDate": "2024-12-16T01:30:22.693Z",
          "content": "<p>I have the same questions.</p>",
          "rawMarkdown": "I have the same questions."
        }
      ]
    },
    {
      "id": 3072678,
      "postDate": "2024-12-15T14:33:35.467Z",
      "content": "<p>Thanks for sharing your solution, I would like to ask what do you mean by lora fusion, merging to LLM or multiple lora adapter fusion?</p>",
      "rawMarkdown": "Thanks for sharing your solution, I would like to ask what do you mean by lora fusion, merging to LLM or multiple lora adapter fusion?",
      "votes": 2
    },
    {
      "id": 3072331,
      "postDate": "2024-12-15T03:36:13.950Z",
      "content": "<p>congratulations to be a gm! <br>\nThen may i ask how u generate your external data? or any source can help a beginner to learn this skill, cheer~</p>",
      "rawMarkdown": "congratulations to be a gm! \nThen may i ask how u generate your external data? or any source can help a beginner to learn this skill, cheer~",
      "votes": 2
    },
    {
      "id": 3072877,
      "postDate": "2024-12-15T17:56:10.530Z",
      "content": "<p>congratulations to be a gm!</p>",
      "rawMarkdown": "congratulations to be a gm!"
    },
    {
      "id": 3072803,
      "postDate": "2024-12-15T16:39:32.160Z",
      "content": "<p>Much obliged for your valuable guidance.</p>",
      "rawMarkdown": "Much obliged for your valuable guidance."
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-20T03:45:43Z",
  "competition_id": 82695,
  "competition_slug": "eedi-mining-misconceptions-in-mathematics",
  "team_id": 12758881,
  "solution_url": "/competitions/eedi-mining-misconceptions-in-mathematics/writeups/sayoulala-10st-place-solution-summary"
}