{
  "id": 551498,
  "title": "3rd Place Solution (with Magic Boost)",
  "content": "<p>First, I want to thank the organizers for hosting this competition, and the Kaggle community for all the discussions and shared notebooks that helped me learn so much.</p>\n<h2>My Solution</h2>\n<p>A two-stage approach:</p>\n<h3>Stage 1: Retrieval</h3>\n<ul>\n<li>Used Qwen-14B Embedder (from <a href=\"https://www.kaggle.com/anhvth226\" target=\"_blank\">@anhvth226</a> ) ensemble with Qwen-14B embedder trained with FlagEmbedding</li>\n<li>Retrieved 35 most relevant misconceptions for each question</li>\n<li>Used publicly shared Retriever and tried my own FlagEmbedding-trained Retriever</li>\n<li>To be honest, my self-trained Retriever was pretty bad and removing it actually improved the score (lol)</li>\n</ul>\n<h3>Stage 2: Reranking</h3>\n<ul>\n<li>Used Qwen-32B-instruct-AWQ reranker</li>\n<li>Ensembled 6 different LoRAs with various training parameters and cross-validation</li>\n<li>One model included about 2000 GPT4-mini generated samples, but the improvement was… not very significant</li>\n</ul>\n<p>Scores at this point:</p>\n<ul>\n<li>Public LB: 0.590</li>\n<li>Private LB: 0.564</li>\n</ul>\n<h2>Time for Some Magic!</h2>\n<p>Here comes the most interesting part!</p>\n<p>It started when I discovered over 900 unseen misconceptions in the misconceptions table that never appeared in the training data. Plus, according to <a href=\"https://www.kaggle.com/zhudong1949\" target=\"_blank\">@zhudong1949</a> and <a href=\"https://www.kaggle.com/eugenkrylov\" target=\"_blank\">@eugenkrylov</a> findings (check here: <a href=\"https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/550875)\" target=\"_blank\">https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/550875)</a>, the testing set had many unseen subjects and only about 685 questions.</p>\n<p>So I thought: these unseen misconceptions must make up a significant portion of the testing set!</p>\n<h3>Let's Experiment</h3>\n<p>On the second-to-last day of the competition (yes, I really dared to use two submissions for experiments at this point XD), I did this LB probing:</p>\n<ul>\n<li>Predicted only 1 misconception per question</li>\n<li>Tested twice:<ol>\n<li>Using only seen misconceptions from training data: got 0.154</li>\n<li>Using only unseen misconceptions: got 0.444</li></ol></li>\n</ul>\n<p>Seeing these results, I made a bold guess that the seen-to-unseen ratio in the testing data was roughly 1:3.</p>\n<h3>The Final Magic Touch</h3>\n<p>So I implemented a simple post-processing:</p>\n<ul>\n<li>Multiplied the probabilities of all predicted unseen misconceptions by a constant C</li>\n<li>Adjusted until unseen misconceptions made up 75% of the first predictions</li>\n</ul>\n<p>This little magic trick made the scores skyrocket 😮:</p>\n<ul>\n<li>Public LB: 0.590 → 0.658</li>\n<li>Private LB: 0.564 → 0.600</li>\n</ul>\n<p>In the final version, I also randomly shuffled the order of misconceptions input to the reranker, which gave a small additional boost:</p>\n<ul>\n<li>Public LB: 0.670</li>\n<li>Private LB: 0.602</li>\n</ul>\n<h3>Code</h3>\n<p>The inference code can be found at:<br>\n<a href=\"https://www.kaggle.com/code/threerabbits/eedi-11-21-myq14b-q32b-rerank-mod-novel-local-suf\" target=\"_blank\">https://www.kaggle.com/code/threerabbits/eedi-11-21-myq14b-q32b-rerank-mod-novel-local-suf</a></p>",
  "comments": [
    {
      "id": 3071279,
      "author_name": "Liu Kelin",
      "author_url": "",
      "post_date": "2024-12-13T15:14:51.203000",
      "content": "<p>Genius!😄😄😄</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 3071275,
      "author_name": "Raja Biswas",
      "author_url": "",
      "post_date": "2024-12-13T15:11:13.690000",
      "content": "<p>Nice spot -- this would easily win the efficiency track I think!</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 3072874,
      "author_name": "MrSimple",
      "author_url": "",
      "post_date": "2024-12-15T17:55:11.037000",
      "content": "<p>Nice spot -- this would easily win the efficiency track I think!</p>",
      "votes": -4,
      "replies": []
    },
    {
      "id": 3071262,
      "author_name": "cm391",
      "author_url": "",
      "post_date": "2024-12-13T14:52:25.587000",
      "content": "<p>🤯🤯🤯</p>\n<p>Genius ! </p>",
      "votes": -3,
      "replies": []
    },
    {
      "id": 3084582,
      "author_name": "lumoumou",
      "author_url": "",
      "post_date": "2024-12-31T03:46:57.723000",
      "content": "<p>666, respect🫡</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3074310,
      "author_name": "Grand Servant",
      "author_url": "",
      "post_date": "2024-12-17T14:51:20.797000",
      "content": "<p>That is magic. Never thought there are so many unseen misconceptions there.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3073225,
      "author_name": "CsOH",
      "author_url": "",
      "post_date": "2024-12-16T07:44:00.933000",
      "content": "<p>Genius. Do you try generate more data for the unseen misconceptions?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3073192,
      "author_name": "Ashish Patel",
      "author_url": "",
      "post_date": "2024-12-16T06:41:06.190000",
      "content": "<p>Congratulations thanks for the solution </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3071654,
      "author_name": "Chew Kok Wah",
      "author_url": "",
      "post_date": "2024-12-14T04:14:37.537000",
      "content": "<p>Do you train your Lora on AWQ weight or on the unquantized weight? </p>",
      "votes": 0,
      "replies": [
        {
          "id": 3072095,
          "author_name": "Yi-Chia Chen",
          "author_url": "",
          "post_date": "2024-12-14T17:21:09.313000",
          "content": "<p>I train my LoRA directly on AWQ weights since I don't have enough VRAM to work with unquantized weights.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 3071618,
      "author_name": "Ming Sheng Choo",
      "author_url": "",
      "post_date": "2024-12-14T02:38:45.657000",
      "content": "<p>Gotta say this is a really interesting way to tackle the problem. Would like to ask how did you train your rerank model, was it Flagembedding or something else? Thanks!</p>",
      "votes": 0,
      "replies": [
        {
          "id": 3072097,
          "author_name": "Yi-Chia Chen",
          "author_url": "",
          "post_date": "2024-12-14T17:23:46.253000",
          "content": "<p>I used trl SFTTrainer for training, just like what many others have shared - giving numbers to 35 misconceptions (using exactly 1-9 and A-Z), and then training LLMs to output the most relevant numbers!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 3071617,
      "author_name": "Timmy Juicehouse",
      "author_url": "",
      "post_date": "2024-12-14T02:36:16.647000",
      "content": "<p>a brave gamble!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3071347,
      "author_name": "Sinan Calisir",
      "author_url": "",
      "post_date": "2024-12-13T16:59:08.573000",
      "content": "<p>Wow smart trick, congratulations!!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3071261,
      "author_name": "Yannan Chen",
      "author_url": "",
      "post_date": "2024-12-13T14:51:48.647000",
      "content": "<p>Such a smart hack! 🤣🤣 but maybe not what the Eedi host wants to see as a solution tho</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3071287,
      "author_name": "chizhu",
      "author_url": "",
      "post_date": "2024-12-13T15:24:53.030000",
      "content": "<p>amazing!!!</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 16,
  "messages": [
    {
      "id": 3071257,
      "postDate": "2024-12-13T14:47:06.550Z",
      "content": "<p>First, I want to thank the organizers for hosting this competition, and the Kaggle community for all the discussions and shared notebooks that helped me learn so much.</p>\n<h2>My Solution</h2>\n<p>A two-stage approach:</p>\n<h3>Stage 1: Retrieval</h3>\n<ul>\n<li>Used Qwen-14B Embedder (from <a href=\"https://www.kaggle.com/anhvth226\" target=\"_blank\">@anhvth226</a> ) ensemble with Qwen-14B embedder trained with FlagEmbedding</li>\n<li>Retrieved 35 most relevant misconceptions for each question</li>\n<li>Used publicly shared Retriever and tried my own FlagEmbedding-trained Retriever</li>\n<li>To be honest, my self-trained Retriever was pretty bad and removing it actually improved the score (lol)</li>\n</ul>\n<h3>Stage 2: Reranking</h3>\n<ul>\n<li>Used Qwen-32B-instruct-AWQ reranker</li>\n<li>Ensembled 6 different LoRAs with various training parameters and cross-validation</li>\n<li>One model included about 2000 GPT4-mini generated samples, but the improvement was… not very significant</li>\n</ul>\n<p>Scores at this point:</p>\n<ul>\n<li>Public LB: 0.590</li>\n<li>Private LB: 0.564</li>\n</ul>\n<h2>Time for Some Magic!</h2>\n<p>Here comes the most interesting part!</p>\n<p>It started when I discovered over 900 unseen misconceptions in the misconceptions table that never appeared in the training data. Plus, according to <a href=\"https://www.kaggle.com/zhudong1949\" target=\"_blank\">@zhudong1949</a> and <a href=\"https://www.kaggle.com/eugenkrylov\" target=\"_blank\">@eugenkrylov</a> findings (check here: <a href=\"https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/550875)\" target=\"_blank\">https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/550875)</a>, the testing set had many unseen subjects and only about 685 questions.</p>\n<p>So I thought: these unseen misconceptions must make up a significant portion of the testing set!</p>\n<h3>Let's Experiment</h3>\n<p>On the second-to-last day of the competition (yes, I really dared to use two submissions for experiments at this point XD), I did this LB probing:</p>\n<ul>\n<li>Predicted only 1 misconception per question</li>\n<li>Tested twice:<ol>\n<li>Using only seen misconceptions from training data: got 0.154</li>\n<li>Using only unseen misconceptions: got 0.444</li></ol></li>\n</ul>\n<p>Seeing these results, I made a bold guess that the seen-to-unseen ratio in the testing data was roughly 1:3.</p>\n<h3>The Final Magic Touch</h3>\n<p>So I implemented a simple post-processing:</p>\n<ul>\n<li>Multiplied the probabilities of all predicted unseen misconceptions by a constant C</li>\n<li>Adjusted until unseen misconceptions made up 75% of the first predictions</li>\n</ul>\n<p>This little magic trick made the scores skyrocket 😮:</p>\n<ul>\n<li>Public LB: 0.590 → 0.658</li>\n<li>Private LB: 0.564 → 0.600</li>\n</ul>\n<p>In the final version, I also randomly shuffled the order of misconceptions input to the reranker, which gave a small additional boost:</p>\n<ul>\n<li>Public LB: 0.670</li>\n<li>Private LB: 0.602</li>\n</ul>\n<h3>Code</h3>\n<p>The inference code can be found at:<br>\n<a href=\"https://www.kaggle.com/code/threerabbits/eedi-11-21-myq14b-q32b-rerank-mod-novel-local-suf\" target=\"_blank\">https://www.kaggle.com/code/threerabbits/eedi-11-21-myq14b-q32b-rerank-mod-novel-local-suf</a></p>",
      "rawMarkdown": "First, I want to thank the organizers for hosting this competition, and the Kaggle community for all the discussions and shared notebooks that helped me learn so much.\n\n## My Solution\nA two-stage approach:\n\n### Stage 1: Retrieval\n- Used Qwen-14B Embedder (from @anhvth226 ) ensemble with Qwen-14B embedder trained with FlagEmbedding\n- Retrieved 35 most relevant misconceptions for each question\n- Used publicly shared Retriever and tried my own FlagEmbedding-trained Retriever\n- To be honest, my self-trained Retriever was pretty bad and removing it actually improved the score (lol)\n\n### Stage 2: Reranking\n- Used Qwen-32B-instruct-AWQ reranker\n- Ensembled 6 different LoRAs with various training parameters and cross-validation\n- One model included about 2000 GPT4-mini generated samples, but the improvement was... not very significant\n\nScores at this point:\n- Public LB: 0.590\n- Private LB: 0.564\n\n## Time for Some Magic!\nHere comes the most interesting part!\n\nIt started when I discovered over 900 unseen misconceptions in the misconceptions table that never appeared in the training data. Plus, according to @zhudong1949 and @eugenkrylov findings (check here: https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/550875), the testing set had many unseen subjects and only about 685 questions.\n\nSo I thought: these unseen misconceptions must make up a significant portion of the testing set!\n\n### Let's Experiment\nOn the second-to-last day of the competition (yes, I really dared to use two submissions for experiments at this point XD), I did this LB probing:\n- Predicted only 1 misconception per question\n- Tested twice:\n  1. Using only seen misconceptions from training data: got 0.154\n  2. Using only unseen misconceptions: got 0.444\n\nSeeing these results, I made a bold guess that the seen-to-unseen ratio in the testing data was roughly 1:3.\n\n### The Final Magic Touch\nSo I implemented a simple post-processing:\n- Multiplied the probabilities of all predicted unseen misconceptions by a constant C\n- Adjusted until unseen misconceptions made up 75% of the first predictions\n\nThis little magic trick made the scores skyrocket 😮:\n- Public LB: 0.590 → 0.658\n- Private LB: 0.564 → 0.600\n\nIn the final version, I also randomly shuffled the order of misconceptions input to the reranker, which gave a small additional boost:\n- Public LB: 0.670\n- Private LB: 0.602\n\n### Code\nThe inference code can be found at:\nhttps://www.kaggle.com/code/threerabbits/eedi-11-21-myq14b-q32b-rerank-mod-novel-local-suf",
      "votes": 64
    },
    {
      "id": 3071279,
      "postDate": "2024-12-13T15:14:51.203Z",
      "content": "<p>Genius!😄😄😄</p>",
      "rawMarkdown": "Genius!😄😄😄",
      "votes": 1
    },
    {
      "id": 3071275,
      "postDate": "2024-12-13T15:11:13.690Z",
      "content": "<p>Nice spot -- this would easily win the efficiency track I think!</p>",
      "rawMarkdown": "Nice spot -- this would easily win the efficiency track I think!",
      "votes": 1
    },
    {
      "id": 3072874,
      "postDate": "2024-12-15T17:55:11.037Z",
      "content": "<p>Nice spot -- this would easily win the efficiency track I think!</p>",
      "rawMarkdown": "Nice spot -- this would easily win the efficiency track I think!\n\n\n",
      "votes": -4
    },
    {
      "id": 3071262,
      "postDate": "2024-12-13T14:52:25.587Z",
      "content": "<p>🤯🤯🤯</p>\n<p>Genius ! </p>",
      "rawMarkdown": "🤯🤯🤯\n\nGenius ! ",
      "votes": -3
    },
    {
      "id": 3084582,
      "postDate": "2024-12-31T03:46:57.723Z",
      "content": "<p>666, respect🫡</p>",
      "rawMarkdown": "666, respect🫡"
    },
    {
      "id": 3074310,
      "postDate": "2024-12-17T14:51:20.797Z",
      "content": "<p>That is magic. Never thought there are so many unseen misconceptions there.</p>",
      "rawMarkdown": "That is magic. Never thought there are so many unseen misconceptions there."
    },
    {
      "id": 3073225,
      "postDate": "2024-12-16T07:44:00.933Z",
      "content": "<p>Genius. Do you try generate more data for the unseen misconceptions?</p>",
      "rawMarkdown": "Genius. Do you try generate more data for the unseen misconceptions?"
    },
    {
      "id": 3073192,
      "postDate": "2024-12-16T06:41:06.190Z",
      "content": "<p>Congratulations thanks for the solution </p>",
      "rawMarkdown": "Congratulations thanks for the solution "
    },
    {
      "id": 3071654,
      "postDate": "2024-12-14T04:14:37.537Z",
      "content": "<p>Do you train your Lora on AWQ weight or on the unquantized weight? </p>",
      "rawMarkdown": "Do you train your Lora on AWQ weight or on the unquantized weight? ",
      "replies": [
        {
          "id": 3072095,
          "postDate": "2024-12-14T17:21:09.313Z",
          "content": "<p>I train my LoRA directly on AWQ weights since I don't have enough VRAM to work with unquantized weights.</p>",
          "rawMarkdown": "I train my LoRA directly on AWQ weights since I don't have enough VRAM to work with unquantized weights.",
          "votes": 1
        }
      ]
    },
    {
      "id": 3071618,
      "postDate": "2024-12-14T02:38:45.657Z",
      "content": "<p>Gotta say this is a really interesting way to tackle the problem. Would like to ask how did you train your rerank model, was it Flagembedding or something else? Thanks!</p>",
      "rawMarkdown": "Gotta say this is a really interesting way to tackle the problem. Would like to ask how did you train your rerank model, was it Flagembedding or something else? Thanks!",
      "replies": [
        {
          "id": 3072097,
          "postDate": "2024-12-14T17:23:46.253Z",
          "content": "<p>I used trl SFTTrainer for training, just like what many others have shared - giving numbers to 35 misconceptions (using exactly 1-9 and A-Z), and then training LLMs to output the most relevant numbers!</p>",
          "rawMarkdown": "I used trl SFTTrainer for training, just like what many others have shared - giving numbers to 35 misconceptions (using exactly 1-9 and A-Z), and then training LLMs to output the most relevant numbers!"
        }
      ]
    },
    {
      "id": 3071617,
      "postDate": "2024-12-14T02:36:16.647Z",
      "content": "<p>a brave gamble!</p>",
      "rawMarkdown": "a brave gamble!"
    },
    {
      "id": 3071347,
      "postDate": "2024-12-13T16:59:08.573Z",
      "content": "<p>Wow smart trick, congratulations!!</p>",
      "rawMarkdown": "Wow smart trick, congratulations!!"
    },
    {
      "id": 3071261,
      "postDate": "2024-12-13T14:51:48.647Z",
      "content": "<p>Such a smart hack! 🤣🤣 but maybe not what the Eedi host wants to see as a solution tho</p>",
      "rawMarkdown": "Such a smart hack! 🤣🤣 but maybe not what the Eedi host wants to see as a solution tho"
    },
    {
      "id": 3071287,
      "postDate": "2024-12-13T15:24:53.030Z",
      "content": "<p>amazing!!!</p>",
      "rawMarkdown": "amazing!!!"
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-20T03:48:36Z",
  "competition_id": 82695,
  "competition_slug": "eedi-mining-misconceptions-in-mathematics",
  "team_id": 12937798,
  "solution_url": "/competitions/eedi-mining-misconceptions-in-mathematics/writeups/waseda-pochi-3rd-place-solution-with-magic-boost"
}