{
  "id": 551429,
  "title": "12th Place Solution",
  "content": "<p>I would like to express my gratitude to the Kaggle platform and Eedi for hosting this competition. Throughout this event, I have learned a great deal about recall and ranking techniques, especially on how to leverage large models for embedding contrastive learning and reranking. Ultimately, I achieved my solo gold medal, bringing me closer to becoming a Grand Master.</p>\n<h3>Overview</h3>\n<p>The primary strategy involved a Retriever combined with a reranker.</p>\n<h3>Synthetic Data Generation</h3>\n<p>Using the Chain of Thought (CoT) approach, I generated four batches of synthetic data, totaling over 10,000 external data points, based on misconceptions. The prompt was as follows:</p>\n<pre><code>You are an intelligent math problem-solving expert.\nI will provide you with a misconception-related error reason. Based on this error, generate a math single-choice question along with four options, where only one is correct, and the other three are distractors.\nThe error reason provided corresponds to one of the incorrect options among the three distractors.\nBelow are several examples:\nExample 1:\nInput: Confuses the order of operations, believes addition comes before multiplication\nOutput: [question]ConstructName:Use the order of operations to carry out calculations involving powersSubjectName:BIDMASQuestionDesc:[3 times 2+4-5]Where do the brackets need to go to make the answer equal ( 13 ) ?[Candidate answers]A.( 3 times(2+4)-5 )B.( 3 times 2+(4-5) )C.( 3 times(2+4-5) )D.Does not need brackets[Correct Answer]( 3 times(2+4)-5 )[Misconcepte Incorrect answer]Does not need brackets\n\nExample 2:\nInput: Does not know that to factorise a quadratic expression, to find two numbers that add to give the coefficient of the x term, and multiply to give the non-variable term\nOutput: [question]ConstructName:Simplify an algebraic fraction by factorising the numeratorSubjectName:Simplifying Algebraic FractionsQuestionDesc:Simplify the following, if possible: ( frac{m^{2}+2 m-3}{m-3} )[Candidate answers]A.( m+1 )B.( m+2 )C.( m-1 )D.Does not simplify[Correct Answer]Does not simplify[Misconcepte Incorrect answer]( m+1 )\n\nExample 3:\nInput: Thinks that when you cancel identical terms from the numerator and denominator, they just disappear\nOutput: [question]ConstructName:Simplify an algebraic fraction by factorising the numeratorSubjectName:Simplifying Algebraic FractionsQuestionDesc:Simplify the following, if possible: ( frac{m^{2}+2 m-3}{m-3} )[Candidate answers]A.( m+1 )B.( m+2 )C.( m-1 )D.Does not simplify[Correct Answer]Does not simplify[Misconcepte Incorrect answer]( m+2 )\n\nExample 4:\nInput: Does not know that to factorise a quadratic expression, to find two numbers that add to give the coefficient of the x term, and multiply to give the non-variable term\nOutput: [question]ConstructName:Simplify an algebraic fraction by factorising the numeratorSubjectName:Simplifying Algebraic FractionsQuestionDesc:Simplify the following, if possible: ( frac{m^{2}+2 m-3}{m-3} )[Candidate answers]A.( m+1 )B.( m+2 )C.( m-1 )D.Does not simplify[Correct Answer]Does not simplify[Misconcepte Incorrect answer]( m-1 )\nPlease generate similar questions and options based on the cases above.\nYou first need to think about the reasoning logic according to the chain of thought, then provide the specific case.\nThe output format is:\n&lt;thought&gt;Insert your thoughts on the error reason here&lt;/thought&gt;\n&lt;output&gt;Insert your question and options here&lt;/output&gt;\n</code></pre>\n<p>The external data helped improve recall from 0.430 to 0.470 (single fold).</p>\n<h3>Retriever</h3>\n<p>The overall approach involved initially using the SFR model to retrieve 100 candidates, followed by training contrastive learning with the Qwen model. The ensemble of three models was achieved by concatenating embeddings: <code>emb=concat(emb1,emb2,emb3)</code>.</p>\n<ul>\n<li>Inferred potential errors using Qwen2.5-32b-instruct-awq, combined with the original query to train Qwen2.5-14B-instruct. The 5-fold LoRA weights were averaged to obtain <code>emb1</code>.</li>\n<li>Trained Qwen2.5-14B-instruct with the original query, and averaged the 5-fold LoRA weights to get <code>emb2</code>.</li>\n<li>Used the open-source model from <a href=\"https://www.kaggle.com/code/anhvth226/eedi-11-21-14b\" target=\"_blank\">notebook</a> for <code>emb3</code>.</li>\n<li>Public score: 0.526, Private score: 0.485</li>\n</ul>\n<h3>Reranker</h3>\n<p>For the top 50 candidates retrieved by the Retriever (results obtained from 5 folds oof), 25 were randomly selected each time for 25-class prediction. The final score was achieved by ensemble weighting the probabilities of three models:</p>\n<ul>\n<li>Public score: 0.597, Private score: 0.564</li>\n<li>Qwen2.5-32b-instruct-awq origin query + vllm infer: Averaged 5-fold LoRA weights, Public: 0.570</li>\n<li>Qwen2.5-14b-instruct origin query: Averaged 5-fold LoRA weights, Public: 0.567</li>\n<li>Qwen2.5-14b-instruct query + reason: Averaged 5-fold LoRA weights, Public: 0.558</li>\n</ul>\n<h3>What Didn't Work</h3>\n<ul>\n<li>Qwen2.5-coder</li>\n<li>LGB/CatBoost for oof post reranker</li>\n<li>Reranker query + misconception last_pool =&gt; 0/1</li>\n</ul>",
  "comments": [
    {
      "id": 3070860,
      "author_name": "Bootstrapping",
      "author_url": "",
      "post_date": "2024-12-13T05:42:49.070000",
      "content": "<p>Congratulations on becoming a Kaggle Grand Master! 🎉🎉🎉</p>",
      "votes": 3,
      "replies": [
        {
          "id": 3070879,
          "author_name": "chizhu",
          "author_url": "",
          "post_date": "2024-12-13T06:09:15.220000",
          "content": "<p>Thank you😃</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 3071308,
      "author_name": "niko_fish",
      "author_url": "",
      "post_date": "2024-12-13T15:47:34.920000",
      "content": "<p>Congratulations！Your way of CoT is very suitable for this type of mathematical logic. Thank you for sharing！</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3071052,
      "author_name": "heng",
      "author_url": "",
      "post_date": "2024-12-13T09:29:03.060000",
      "content": "<p>Congratulations on becoming a Kaggle Grand Master! 🎉🎉🎉</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3070980,
      "author_name": "hinsay",
      "author_url": "",
      "post_date": "2024-12-13T07:51:49.367000",
      "content": "<p>Congratulations!!!</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 5,
  "messages": [
    {
      "id": 3070850,
      "postDate": "2024-12-13T05:22:40.097Z",
      "content": "<p>I would like to express my gratitude to the Kaggle platform and Eedi for hosting this competition. Throughout this event, I have learned a great deal about recall and ranking techniques, especially on how to leverage large models for embedding contrastive learning and reranking. Ultimately, I achieved my solo gold medal, bringing me closer to becoming a Grand Master.</p>\n<h3>Overview</h3>\n<p>The primary strategy involved a Retriever combined with a reranker.</p>\n<h3>Synthetic Data Generation</h3>\n<p>Using the Chain of Thought (CoT) approach, I generated four batches of synthetic data, totaling over 10,000 external data points, based on misconceptions. The prompt was as follows:</p>\n<pre><code>You are an intelligent math problem-solving expert.\nI will provide you with a misconception-related error reason. Based on this error, generate a math single-choice question along with four options, where only one is correct, and the other three are distractors.\nThe error reason provided corresponds to one of the incorrect options among the three distractors.\nBelow are several examples:\nExample 1:\nInput: Confuses the order of operations, believes addition comes before multiplication\nOutput: [question]ConstructName:Use the order of operations to carry out calculations involving powersSubjectName:BIDMASQuestionDesc:[3 times 2+4-5]Where do the brackets need to go to make the answer equal ( 13 ) ?[Candidate answers]A.( 3 times(2+4)-5 )B.( 3 times 2+(4-5) )C.( 3 times(2+4-5) )D.Does not need brackets[Correct Answer]( 3 times(2+4)-5 )[Misconcepte Incorrect answer]Does not need brackets\n\nExample 2:\nInput: Does not know that to factorise a quadratic expression, to find two numbers that add to give the coefficient of the x term, and multiply to give the non-variable term\nOutput: [question]ConstructName:Simplify an algebraic fraction by factorising the numeratorSubjectName:Simplifying Algebraic FractionsQuestionDesc:Simplify the following, if possible: ( frac{m^{2}+2 m-3}{m-3} )[Candidate answers]A.( m+1 )B.( m+2 )C.( m-1 )D.Does not simplify[Correct Answer]Does not simplify[Misconcepte Incorrect answer]( m+1 )\n\nExample 3:\nInput: Thinks that when you cancel identical terms from the numerator and denominator, they just disappear\nOutput: [question]ConstructName:Simplify an algebraic fraction by factorising the numeratorSubjectName:Simplifying Algebraic FractionsQuestionDesc:Simplify the following, if possible: ( frac{m^{2}+2 m-3}{m-3} )[Candidate answers]A.( m+1 )B.( m+2 )C.( m-1 )D.Does not simplify[Correct Answer]Does not simplify[Misconcepte Incorrect answer]( m+2 )\n\nExample 4:\nInput: Does not know that to factorise a quadratic expression, to find two numbers that add to give the coefficient of the x term, and multiply to give the non-variable term\nOutput: [question]ConstructName:Simplify an algebraic fraction by factorising the numeratorSubjectName:Simplifying Algebraic FractionsQuestionDesc:Simplify the following, if possible: ( frac{m^{2}+2 m-3}{m-3} )[Candidate answers]A.( m+1 )B.( m+2 )C.( m-1 )D.Does not simplify[Correct Answer]Does not simplify[Misconcepte Incorrect answer]( m-1 )\nPlease generate similar questions and options based on the cases above.\nYou first need to think about the reasoning logic according to the chain of thought, then provide the specific case.\nThe output format is:\n&lt;thought&gt;Insert your thoughts on the error reason here&lt;/thought&gt;\n&lt;output&gt;Insert your question and options here&lt;/output&gt;\n</code></pre>\n<p>The external data helped improve recall from 0.430 to 0.470 (single fold).</p>\n<h3>Retriever</h3>\n<p>The overall approach involved initially using the SFR model to retrieve 100 candidates, followed by training contrastive learning with the Qwen model. The ensemble of three models was achieved by concatenating embeddings: <code>emb=concat(emb1,emb2,emb3)</code>.</p>\n<ul>\n<li>Inferred potential errors using Qwen2.5-32b-instruct-awq, combined with the original query to train Qwen2.5-14B-instruct. The 5-fold LoRA weights were averaged to obtain <code>emb1</code>.</li>\n<li>Trained Qwen2.5-14B-instruct with the original query, and averaged the 5-fold LoRA weights to get <code>emb2</code>.</li>\n<li>Used the open-source model from <a href=\"https://www.kaggle.com/code/anhvth226/eedi-11-21-14b\" target=\"_blank\">notebook</a> for <code>emb3</code>.</li>\n<li>Public score: 0.526, Private score: 0.485</li>\n</ul>\n<h3>Reranker</h3>\n<p>For the top 50 candidates retrieved by the Retriever (results obtained from 5 folds oof), 25 were randomly selected each time for 25-class prediction. The final score was achieved by ensemble weighting the probabilities of three models:</p>\n<ul>\n<li>Public score: 0.597, Private score: 0.564</li>\n<li>Qwen2.5-32b-instruct-awq origin query + vllm infer: Averaged 5-fold LoRA weights, Public: 0.570</li>\n<li>Qwen2.5-14b-instruct origin query: Averaged 5-fold LoRA weights, Public: 0.567</li>\n<li>Qwen2.5-14b-instruct query + reason: Averaged 5-fold LoRA weights, Public: 0.558</li>\n</ul>\n<h3>What Didn't Work</h3>\n<ul>\n<li>Qwen2.5-coder</li>\n<li>LGB/CatBoost for oof post reranker</li>\n<li>Reranker query + misconception last_pool =&gt; 0/1</li>\n</ul>",
      "rawMarkdown": "I would like to express my gratitude to the Kaggle platform and Eedi for hosting this competition. Throughout this event, I have learned a great deal about recall and ranking techniques, especially on how to leverage large models for embedding contrastive learning and reranking. Ultimately, I achieved my solo gold medal, bringing me closer to becoming a Grand Master.\n\n### Overview\nThe primary strategy involved a Retriever combined with a reranker.\n\n### Synthetic Data Generation\nUsing the Chain of Thought (CoT) approach, I generated four batches of synthetic data, totaling over 10,000 external data points, based on misconceptions. The prompt was as follows:\n\n```\nYou are an intelligent math problem-solving expert.\nI will provide you with a misconception-related error reason. Based on this error, generate a math single-choice question along with four options, where only one is correct, and the other three are distractors.\nThe error reason provided corresponds to one of the incorrect options among the three distractors.\nBelow are several examples:\nExample 1:\nInput: Confuses the order of operations, believes addition comes before multiplication\nOutput: [question]\\nConstructName:Use the order of operations to carry out calculations involving powers\\nSubjectName:BIDMAS\\nQuestionDesc:\\\\[\\n3 \\\\times 2+4-5\\n\\\\]\\nWhere do the brackets need to go to make the answer equal \\\\( 13 \\\\) ?\\n\\n[Candidate answers]\\nA.\\\\( 3 \\\\times(2+4)-5 \\\\)\\nB.\\\\( 3 \\\\times 2+(4-5) \\\\)\\nC.\\\\( 3 \\\\times(2+4-5) \\\\)\\nD.Does not need brackets\\n\\n[Correct Answer]\\n\\\\( 3 \\\\times(2+4)-5 \\\\)\\n\\n[Misconcepte Incorrect answer]\\nDoes not need brackets\\n\n\nExample 2:\nInput: Does not know that to factorise a quadratic expression, to find two numbers that add to give the coefficient of the x term, and multiply to give the non-variable term\nOutput: [question]\\nConstructName:Simplify an algebraic fraction by factorising the numerator\\nSubjectName:Simplifying Algebraic Fractions\\nQuestionDesc:Simplify the following, if possible: \\\\( \\\\frac{m^{2}+2 m-3}{m-3} \\\\)\\n\\n[Candidate answers]\\nA.\\\\( m+1 \\\\)\\nB.\\\\( m+2 \\\\)\\nC.\\\\( m-1 \\\\)\\nD.Does not simplify\\n\\n[Correct Answer]\\nDoes not simplify\\n\\n[Misconcepte Incorrect answer]\\n\\\\( m+1 \\\\)\\n\n\nExample 3:\nInput: Thinks that when you cancel identical terms from the numerator and denominator, they just disappear\nOutput: [question]\\nConstructName:Simplify an algebraic fraction by factorising the numerator\\nSubjectName:Simplifying Algebraic Fractions\\nQuestionDesc:Simplify the following, if possible: \\\\( \\\\frac{m^{2}+2 m-3}{m-3} \\\\)\\n\\n[Candidate answers]\\nA.\\\\( m+1 \\\\)\\nB.\\\\( m+2 \\\\)\\nC.\\\\( m-1 \\\\)\\nD.Does not simplify\\n\\n[Correct Answer]\\nDoes not simplify\\n\\n[Misconcepte Incorrect answer]\\n\\\\( m+2 \\\\)\\n\n\nExample 4:\nInput: Does not know that to factorise a quadratic expression, to find two numbers that add to give the coefficient of the x term, and multiply to give the non-variable term\nOutput: [question]\\nConstructName:Simplify an algebraic fraction by factorising the numerator\\nSubjectName:Simplifying Algebraic Fractions\\nQuestionDesc:Simplify the following, if possible: \\\\( \\\\frac{m^{2}+2 m-3}{m-3} \\\\)\\n\\n[Candidate answers]\\nA.\\\\( m+1 \\\\)\\nB.\\\\( m+2 \\\\)\\nC.\\\\( m-1 \\\\)\\nD.Does not simplify\\n\\n[Correct Answer]\\nDoes not simplify\\n\\n[Misconcepte Incorrect answer]\\n\\\\( m-1 \\\\)\\n\nPlease generate similar questions and options based on the cases above.\nYou first need to think about the reasoning logic according to the chain of thought, then provide the specific case.\nThe output format is:\n<thought>Insert your thoughts on the error reason here</thought>\n<output>Insert your question and options here</output>\n```\n\nThe external data helped improve recall from 0.430 to 0.470 (single fold).\n\n### Retriever\nThe overall approach involved initially using the SFR model to retrieve 100 candidates, followed by training contrastive learning with the Qwen model. The ensemble of three models was achieved by concatenating embeddings: `emb=concat(emb1,emb2,emb3)`.\n- Inferred potential errors using Qwen2.5-32b-instruct-awq, combined with the original query to train Qwen2.5-14B-instruct. The 5-fold LoRA weights were averaged to obtain `emb1`.\n- Trained Qwen2.5-14B-instruct with the original query, and averaged the 5-fold LoRA weights to get `emb2`.\n- Used the open-source model from [notebook](https://www.kaggle.com/code/anhvth226/eedi-11-21-14b) for `emb3`.\n- Public score: 0.526, Private score: 0.485\n\n### Reranker\nFor the top 50 candidates retrieved by the Retriever (results obtained from 5 folds oof), 25 were randomly selected each time for 25-class prediction. The final score was achieved by ensemble weighting the probabilities of three models:\n- Public score: 0.597, Private score: 0.564\n- Qwen2.5-32b-instruct-awq origin query + vllm infer: Averaged 5-fold LoRA weights, Public: 0.570\n- Qwen2.5-14b-instruct origin query: Averaged 5-fold LoRA weights, Public: 0.567\n- Qwen2.5-14b-instruct query + reason: Averaged 5-fold LoRA weights, Public: 0.558\n\n### What Didn't Work\n- Qwen2.5-coder\n- LGB/CatBoost for oof post reranker\n- Reranker query + misconception last_pool => 0/1\n\n",
      "votes": 16
    },
    {
      "id": 3070860,
      "postDate": "2024-12-13T05:42:49.070Z",
      "content": "<p>Congratulations on becoming a Kaggle Grand Master! 🎉🎉🎉</p>",
      "rawMarkdown": "Congratulations on becoming a Kaggle Grand Master! 🎉🎉🎉",
      "votes": 3,
      "replies": [
        {
          "id": 3070879,
          "postDate": "2024-12-13T06:09:15.220Z",
          "content": "<p>Thank you😃</p>",
          "rawMarkdown": "Thank you😃"
        }
      ]
    },
    {
      "id": 3071308,
      "postDate": "2024-12-13T15:47:34.920Z",
      "content": "<p>Congratulations！Your way of CoT is very suitable for this type of mathematical logic. Thank you for sharing！</p>",
      "rawMarkdown": "Congratulations！Your way of CoT is very suitable for this type of mathematical logic. Thank you for sharing！"
    },
    {
      "id": 3071052,
      "postDate": "2024-12-13T09:29:03.060Z",
      "content": "<p>Congratulations on becoming a Kaggle Grand Master! 🎉🎉🎉</p>",
      "rawMarkdown": "Congratulations on becoming a Kaggle Grand Master! 🎉🎉🎉"
    },
    {
      "id": 3070980,
      "postDate": "2024-12-13T07:51:49.367Z",
      "content": "<p>Congratulations!!!</p>",
      "rawMarkdown": "Congratulations!!!"
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-20T03:45:49Z",
  "competition_id": 82695,
  "competition_slug": "eedi-mining-misconceptions-in-mathematics",
  "team_id": 12603090,
  "solution_url": "/competitions/eedi-mining-misconceptions-in-mathematics/writeups/chizhu-12th-place-solution"
}