{
  "id": 551673,
  "title": "13th Place Solution ",
  "content": "<p>Thank you Kaggle  and The Learning Agency Lab  for hosting this competition. </p>\n<p>Although we are disappointed 😢 to loose gold by one rank, we are grateful for the opportunity to learn and grow. </p>\n<h2>Summary</h2>\n<p>Our highest scoring private LB submission was -</p>\n<p><strong>(a) Knowledge Distillation</strong><br>\nQwen2.5-Math-7B-Instruct was used to solve the question in order to reach the final correct answer. The base model itself was used as it seemed quite capable and also to not overfit the Public LB. This reasoning was used to train the Embedding model for the next step</p>\n<p><strong>(b) Candidate Generation</strong><br>\nQwen2.5-14B-Instruct was finetuned with the reasoning text from step 1. Training code was adapted from <a href=\"https://www.kaggle.com/sayoulala\" target=\"_blank\">@sayoulala</a>’s github repo. This retriever was used to obtain the top 100 misconceptions</p>\n<p><strong>(c) 1st Listwise Reranking</strong><br>\nThe top 100 misconceptions from step 2 are stuffed into the prompt and sent to a finetuned Qwen2.5-32B-Instruct along with the Subject, Construct, Question, Correct Answer and the Incorrect Answer. The model generates just 1 token (the letter id) associated with the correct misconception. However, the logprobs were calculated to obtain the top 25 misconception ids rather than just 1</p>\n<p><strong>(d) 2nd Listwise Reranking</strong><br>\nThe top 25 misconceptions from stage 3 are now reranked in the same manner using a finetuned Qwen2.5-72B-Instruct</p>\n<p>All the above combined gives private 0.561 and public 0.578.</p>\n<h2>Data</h2>\n<p>CV - 4 fold based on groupkfold by misconception id.</p>\n<h3>Training Embedding Model</h3>\n<p>FlagEmbedding (adapted from <a href=\"https://www.kaggle.com/sayoulala\" target=\"_blank\">@sayoulala</a>’s github repo)</p>\n<ul>\n<li><p>Used SFR fine-tuned embedding model predictions to get hard negatives. Public LB score for this model was 0.45x which is less than public kernel, however this worked better when combined with reranker.</p></li>\n<li><p>Reasoning for the correct answer generated by Qwen 2.5 Math 7b instruct was part of the input as well. (Private 0.433 , Public 0.486)</p></li>\n</ul>\n<h3>Training Multiple Choice Reranker</h3>\n<p><strong>72B</strong></p>\n<p>LLaMa-Factory was utilized to finetune all 72B models using LoRA in a distributed setting.  All models were trained and sharded with the DeepSpeed ZeRO-3 protocol to ensure the weights, gradients, and optimizer states could all fit within the available VRAM.</p>\n<p><strong>32B</strong><br>\nTrained using Hugging Face SFT Trainer </p>\n<h3>Inference</h3>\n<ul>\n<li>Qwen 2.5 Math 7b uses vllm in fp16</li>\n<li>Qwen 2.5 14b uses bnb to fit on a single gpu - we used both GPUs to speed up inference. </li>\n<li>Qwen 2.5 32b and 72b are run in fp16 one layer at a time - we used both GPUs to speed up inference. </li>\n</ul>\n<p>Total submission time: 500 minutes (8.3 hrs)</p>\n<h4>Failed ideas</h4>\n<ul>\n<li>Ensembling 72b models</li>\n<li>Inserting step-by-step solution into context window of reranker</li>\n<li>Various ways to generate synthetic data</li>\n<li>32B Embedding model - was part of final selection but not the best submission. (Private 0.453, Public 0.470)</li>\n<li>AWQ Qwen14B embedding model - but it requires vllm &gt;= 0.6.4 for inference on Kaggle.</li>\n<li>Synthetic Data - One of the models with synthetic data was part of 3 selected submissions.</li>\n</ul>\n<hr>\n<p>Lastly, Huge Thank you to my fabulous  team mates 🌟 <a href=\"https://www.kaggle.com/nbroad\" target=\"_blank\">@nbroad</a> , 🌟 <a href=\"https://www.kaggle.com/abdullahmeda\" target=\"_blank\">@abdullahmeda</a> and 🌟 <a href=\"https://www.kaggle.com/benbla\" target=\"_blank\">@benbla</a> for your hard work and collaboration on this competition !! </p>",
  "comments": [
    {
      "id": 3076930,
      "author_name": "RB",
      "author_url": "",
      "post_date": "2024-12-20T12:00:49.227000",
      "content": "<h2>### EEDI Misconception Analyzer - Deployed on Modal (13th rank solution)</h2>\n<p>We deployed part of our solution here  - https://rashmi banthia--eedi-misconception-analyzer.modal.run  (Remove space between rashmi banthia - apparently Kaggle doesn't like same user name 🤷‍♀️)</p>\n<p>Cold start takes ~1 min or so.</p>\n<p>This deployed app is part of our solution (13th Rank) for Kaggle Competition <a href=\"https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/\" target=\"_blank\">EEDI</a>.</p>\n<p>Our solution summary can be found here - <a href=\"https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/551673\" target=\"_blank\">https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/551673</a></p>\n<p>If you want to try an example, here is a sample data - </p>\n<pre><code>{'': ,\n\n  '': '\\',\n\n  '': '   ',\n\n  '': '        ',\n\n  '': '       '}\n</code></pre>\n<p><strong>Notes:</strong></p>\n<ul>\n<li><p>This app only deploys 14B Qwen2.5 retriever model on A10G GPU on <a href=\"https://modal.com/\" target=\"_blank\">Modal</a> (no reranker) </p></li>\n<li><p>Deployed using React / FastAPI / Modal for Deployment.</p></li>\n<li><p>Most of the frontend is developed using <a href=\"https://cursor.com/\" target=\"_blank\">Cursor</a></p></li>\n<li><p>Inference is here - <code>src/eedi_api_service/inference/model_inf.py</code></p></li>\n</ul>\n<p>Source code - <a href=\"https://github.com/rashmibanthia/eedi_deploy\" target=\"_blank\">https://github.com/rashmibanthia/eedi_deploy</a></p>\n<hr>\n<p>PS: I'm unable to post this as a separate post. I'm getting \"Too many requests\" when I try to post on discussion forums. I haven't posted much in few days 🤷‍♀️ I would like to know what caused it ? How long do I need to wait to post ?  </p>\n<p>PPS: Its my user name, which is same as GitHub user id and part of Modal link - which is causing <a href=\"https://www.kaggle.com/discussions/product-feedback/320280#1762824\" target=\"_blank\">issues</a>  </p>",
      "votes": 4,
      "replies": []
    }
  ],
  "comment_count": 1,
  "messages": [
    {
      "id": 3072063,
      "postDate": "2024-12-14T16:36:16.917Z",
      "content": "<p>Thank you Kaggle  and The Learning Agency Lab  for hosting this competition. </p>\n<p>Although we are disappointed 😢 to loose gold by one rank, we are grateful for the opportunity to learn and grow. </p>\n<h2>Summary</h2>\n<p>Our highest scoring private LB submission was -</p>\n<p><strong>(a) Knowledge Distillation</strong><br>\nQwen2.5-Math-7B-Instruct was used to solve the question in order to reach the final correct answer. The base model itself was used as it seemed quite capable and also to not overfit the Public LB. This reasoning was used to train the Embedding model for the next step</p>\n<p><strong>(b) Candidate Generation</strong><br>\nQwen2.5-14B-Instruct was finetuned with the reasoning text from step 1. Training code was adapted from <a href=\"https://www.kaggle.com/sayoulala\" target=\"_blank\">@sayoulala</a>’s github repo. This retriever was used to obtain the top 100 misconceptions</p>\n<p><strong>(c) 1st Listwise Reranking</strong><br>\nThe top 100 misconceptions from step 2 are stuffed into the prompt and sent to a finetuned Qwen2.5-32B-Instruct along with the Subject, Construct, Question, Correct Answer and the Incorrect Answer. The model generates just 1 token (the letter id) associated with the correct misconception. However, the logprobs were calculated to obtain the top 25 misconception ids rather than just 1</p>\n<p><strong>(d) 2nd Listwise Reranking</strong><br>\nThe top 25 misconceptions from stage 3 are now reranked in the same manner using a finetuned Qwen2.5-72B-Instruct</p>\n<p>All the above combined gives private 0.561 and public 0.578.</p>\n<h2>Data</h2>\n<p>CV - 4 fold based on groupkfold by misconception id.</p>\n<h3>Training Embedding Model</h3>\n<p>FlagEmbedding (adapted from <a href=\"https://www.kaggle.com/sayoulala\" target=\"_blank\">@sayoulala</a>’s github repo)</p>\n<ul>\n<li><p>Used SFR fine-tuned embedding model predictions to get hard negatives. Public LB score for this model was 0.45x which is less than public kernel, however this worked better when combined with reranker.</p></li>\n<li><p>Reasoning for the correct answer generated by Qwen 2.5 Math 7b instruct was part of the input as well. (Private 0.433 , Public 0.486)</p></li>\n</ul>\n<h3>Training Multiple Choice Reranker</h3>\n<p><strong>72B</strong></p>\n<p>LLaMa-Factory was utilized to finetune all 72B models using LoRA in a distributed setting.  All models were trained and sharded with the DeepSpeed ZeRO-3 protocol to ensure the weights, gradients, and optimizer states could all fit within the available VRAM.</p>\n<p><strong>32B</strong><br>\nTrained using Hugging Face SFT Trainer </p>\n<h3>Inference</h3>\n<ul>\n<li>Qwen 2.5 Math 7b uses vllm in fp16</li>\n<li>Qwen 2.5 14b uses bnb to fit on a single gpu - we used both GPUs to speed up inference. </li>\n<li>Qwen 2.5 32b and 72b are run in fp16 one layer at a time - we used both GPUs to speed up inference. </li>\n</ul>\n<p>Total submission time: 500 minutes (8.3 hrs)</p>\n<h4>Failed ideas</h4>\n<ul>\n<li>Ensembling 72b models</li>\n<li>Inserting step-by-step solution into context window of reranker</li>\n<li>Various ways to generate synthetic data</li>\n<li>32B Embedding model - was part of final selection but not the best submission. (Private 0.453, Public 0.470)</li>\n<li>AWQ Qwen14B embedding model - but it requires vllm &gt;= 0.6.4 for inference on Kaggle.</li>\n<li>Synthetic Data - One of the models with synthetic data was part of 3 selected submissions.</li>\n</ul>\n<hr>\n<p>Lastly, Huge Thank you to my fabulous  team mates 🌟 <a href=\"https://www.kaggle.com/nbroad\" target=\"_blank\">@nbroad</a> , 🌟 <a href=\"https://www.kaggle.com/abdullahmeda\" target=\"_blank\">@abdullahmeda</a> and 🌟 <a href=\"https://www.kaggle.com/benbla\" target=\"_blank\">@benbla</a> for your hard work and collaboration on this competition !! </p>",
      "rawMarkdown": "Thank you Kaggle  and The Learning Agency Lab  for hosting this competition. \n\nAlthough we are disappointed 😢 to loose gold by one rank, we are grateful for the opportunity to learn and grow. \n\n## Summary\n\nOur highest scoring private LB submission was -\n\n**(a) Knowledge Distillation**\nQwen2.5-Math-7B-Instruct was used to solve the question in order to reach the final correct answer. The base model itself was used as it seemed quite capable and also to not overfit the Public LB. This reasoning was used to train the Embedding model for the next step\n\n**(b) Candidate Generation**\nQwen2.5-14B-Instruct was finetuned with the reasoning text from step 1. Training code was adapted from @sayoulala’s github repo. This retriever was used to obtain the top 100 misconceptions\n\n**(c) 1st Listwise Reranking**\nThe top 100 misconceptions from step 2 are stuffed into the prompt and sent to a finetuned Qwen2.5-32B-Instruct along with the Subject, Construct, Question, Correct Answer and the Incorrect Answer. The model generates just 1 token (the letter id) associated with the correct misconception. However, the logprobs were calculated to obtain the top 25 misconception ids rather than just 1\n\n**(d) 2nd Listwise Reranking**\nThe top 25 misconceptions from stage 3 are now reranked in the same manner using a finetuned Qwen2.5-72B-Instruct\n\nAll the above combined gives private 0.561 and public 0.578.\n\n## Data\n\nCV - 4 fold based on groupkfold by misconception id.\n\n### Training Embedding Model\nFlagEmbedding (adapted from @sayoulala’s github repo)\n\n- Used SFR fine-tuned embedding model predictions to get hard negatives. Public LB score for this model was 0.45x which is less than public kernel, however this worked better when combined with reranker.\n\n- Reasoning for the correct answer generated by Qwen 2.5 Math 7b instruct was part of the input as well. (Private 0.433 , Public 0.486)\n\n### Training Multiple Choice Reranker\n\n**72B**\n\nLLaMa-Factory was utilized to finetune all 72B models using LoRA in a distributed setting.  All models were trained and sharded with the DeepSpeed ZeRO-3 protocol to ensure the weights, gradients, and optimizer states could all fit within the available VRAM.\n\n**32B**\nTrained using Hugging Face SFT Trainer \n\n### Inference\n\n- Qwen 2.5 Math 7b uses vllm in fp16\n- Qwen 2.5 14b uses bnb to fit on a single gpu - we used both GPUs to speed up inference. \n- Qwen 2.5 32b and 72b are run in fp16 one layer at a time - we used both GPUs to speed up inference. \n\n\nTotal submission time: 500 minutes (8.3 hrs)\n\n#### Failed ideas\n\n- Ensembling 72b models\n- Inserting step-by-step solution into context window of reranker\n- Various ways to generate synthetic data\n- 32B Embedding model - was part of final selection but not the best submission. (Private 0.453, Public 0.470)\n- AWQ Qwen14B embedding model - but it requires vllm >= 0.6.4 for inference on Kaggle.\n- Synthetic Data - One of the models with synthetic data was part of 3 selected submissions.\n\n----\n\nLastly, Huge Thank you to my fabulous  team mates 🌟 @nbroad , 🌟 @abdullahmeda and 🌟 @benbla for your hard work and collaboration on this competition !! ",
      "votes": 23
    },
    {
      "id": 3076930,
      "postDate": "2024-12-20T12:00:49.227Z",
      "content": "<h2>### EEDI Misconception Analyzer - Deployed on Modal (13th rank solution)</h2>\n<p>We deployed part of our solution here  - https://rashmi banthia--eedi-misconception-analyzer.modal.run  (Remove space between rashmi banthia - apparently Kaggle doesn't like same user name 🤷‍♀️)</p>\n<p>Cold start takes ~1 min or so.</p>\n<p>This deployed app is part of our solution (13th Rank) for Kaggle Competition <a href=\"https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/\" target=\"_blank\">EEDI</a>.</p>\n<p>Our solution summary can be found here - <a href=\"https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/551673\" target=\"_blank\">https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/551673</a></p>\n<p>If you want to try an example, here is a sample data - </p>\n<pre><code>{'': ,\n\n  '': '\\',\n\n  '': '   ',\n\n  '': '        ',\n\n  '': '       '}\n</code></pre>\n<p><strong>Notes:</strong></p>\n<ul>\n<li><p>This app only deploys 14B Qwen2.5 retriever model on A10G GPU on <a href=\"https://modal.com/\" target=\"_blank\">Modal</a> (no reranker) </p></li>\n<li><p>Deployed using React / FastAPI / Modal for Deployment.</p></li>\n<li><p>Most of the frontend is developed using <a href=\"https://cursor.com/\" target=\"_blank\">Cursor</a></p></li>\n<li><p>Inference is here - <code>src/eedi_api_service/inference/model_inf.py</code></p></li>\n</ul>\n<p>Source code - <a href=\"https://github.com/rashmibanthia/eedi_deploy\" target=\"_blank\">https://github.com/rashmibanthia/eedi_deploy</a></p>\n<hr>\n<p>PS: I'm unable to post this as a separate post. I'm getting \"Too many requests\" when I try to post on discussion forums. I haven't posted much in few days 🤷‍♀️ I would like to know what caused it ? How long do I need to wait to post ?  </p>\n<p>PPS: Its my user name, which is same as GitHub user id and part of Modal link - which is causing <a href=\"https://www.kaggle.com/discussions/product-feedback/320280#1762824\" target=\"_blank\">issues</a>  </p>",
      "rawMarkdown": "### EEDI Misconception Analyzer - Deployed on Modal (13th rank solution)\n---\n\nWe deployed part of our solution here  - https://rashmi banthia--eedi-misconception-analyzer.modal.run  (Remove space between rashmi banthia - apparently Kaggle doesn't like same user name 🤷‍♀️)\n\n\nCold start takes ~1 min or so.\n\nThis deployed app is part of our solution (13th Rank) for Kaggle Competition [EEDI](https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/).\n\nOur solution summary can be found here - https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/551673\n\n\nIf you want to try an example, here is a sample data - \n\n```\n{'QuestionText': \"Tom and Katie are discussing the \\\\( 5 \\\\) plants with these heights:\\n\\\\( 24 \\\\mathrm{~cm}, 17 \\\\mathrm{~cm}, 42 \\\\mathrm{~cm}, 26 \\\\mathrm{~cm}, 13 \\\\mathrm{~cm} \\\\)\\nTom says if all the plants were cut in half, the range wouldn't change.\\nKatie says if all the plants grew by \\\\( 3 \\\\mathrm{~cm} \\\\) each, the range wouldn't change.\\nWho do you agree with?\",\n\n  'CorrectAnswerText': 'Only\\nKatie',\n\n  'IncorrectAnswerText': 'Both Tom and Katie',\n\n  'SubjectName': 'Range and Interquartile Range from a List of Data',\n\n  'ConstructName': 'Calculate the range from a list of data'}\n```\n\n\n**Notes:**\n\n- This app only deploys 14B Qwen2.5 retriever model on A10G GPU on [Modal](https://modal.com/) (no reranker) \n\n- Deployed using React / FastAPI / Modal for Deployment.\n\n- Most of the frontend is developed using [Cursor](https://cursor.com/)\n\n- Inference is here - `src/eedi_api_service/inference/model_inf.py`\n\n\nSource code - https://github.com/rashmibanthia/eedi_deploy\n\n---\n\nPS: I'm unable to post this as a separate post. I'm getting \"Too many requests\" when I try to post on discussion forums. I haven't posted much in few days 🤷‍♀️ I would like to know what caused it ? How long do I need to wait to post ?  \n\nPPS: Its my user name, which is same as GitHub user id and part of Modal link - which is causing [issues](https://www.kaggle.com/discussions/product-feedback/320280#1762824)  ",
      "votes": 4
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-20T03:45:35Z",
  "competition_id": 82695,
  "competition_slug": "eedi-mining-misconceptions-in-mathematics",
  "team_id": 12603398,
  "solution_url": "/competitions/eedi-mining-misconceptions-in-mathematics/writeups/qwenqwenqwenqwenqwenqwenqwenqw-13th-place-solution"
}