{
  "id": 551388,
  "title": "Private 7th (Public 2nd) Place Solution Summary",
  "content": "<p>First of all, I'd like to thank kaggle team, host for hosting this competition , and my team mates (@tereka <a href=\"https://www.kaggle.com/irrohas\" target=\"_blank\">@irrohas</a> <a href=\"https://www.kaggle.com/aerdem4\" target=\"_blank\">@aerdem4</a>)<br>\nI finally got 5th gold medal. I'm really happy to be Competitions Grand Master.</p>\n<h2>Overview</h2>\n<p>Voting Ensemble of 3pipelines (Retriever + Reranker)<br>\nAll reranker inference with vllm </p>\n<h2>Pipeline1 by Masaya</h2>\n<p>This is brief summary.<br>\nMasaya will write up detail on this <a href=\"https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/551430\" target=\"_blank\">link</a>.</p>\n<p>Retriever</p>\n<ul>\n<li>Generate test answer with pretrained qwen35B</li>\n<li>model :SFR-Embedding-2_R, Hard Negative Sampling(N=100) removed if unseen in training data.</li>\n<li>dataset: Filled unknown Misconception with GPT4o-mini</li>\n<li>candidate 60</li>\n</ul>\n<p>Reranker</p>\n<ul>\n<li>Qwen2.5 32B AWQ+LoRA (with 40 options)<ul>\n<li>options → 0~9 + A~Z + a~z (40 charactors)</li></ul></li>\n<li>dataset: only original</li>\n<li>candidate 40</li>\n<li>run twice and merge ([:40] and [20:]) </li>\n<li>left 25 with logprobs</li>\n</ul>\n<h2>Pipeline2 by Tereka+Kaeru</h2>\n<p>Retriever</p>\n<ul>\n<li>model1 :SFR-Embedding-2_R、Hard Negative Sampling(100）<ul>\n<li>dataset: original + synthesis with gemma 27B/Qwen2.5 32B</li>\n<li>lb 0.378</li></ul></li>\n<li>model2 : public model by sayoulala</li>\n<li>weighted avg model1 and model2</li>\n<li>candidate 70 and filter with masaya preds</li>\n<li>only single fold</li>\n</ul>\n<p>Reranker</p>\n<ul>\n<li>Qwen2.5 32B AWQ+LoRA (with binary options)<ul>\n<li>options → yes, not</li></ul></li>\n<li>dataset: original + synthesis unknown misconceptions with gemma2 27B/Qwen2.5 32B</li>\n<li>only single fold</li>\n</ul>\n<h2>Pipeline3 by Ahmet</h2>\n<p>Retriever</p>\n<ul>\n<li>Omitted in final pipeline</li>\n</ul>\n<p>Reranker</p>\n<ul>\n<li>Candidates are each top9 from previous pipeline</li>\n<li>Qwen2.5 32B (with 9 options)</li>\n</ul>\n<h2>Ensembling</h2>\n<ul>\n<li>Voting ensemble of each pipeline</li>\n<li>Apply rank like following, join, sum rank, sort by sum_rank</li>\n</ul>\n<pre><code>sub0\n  .with_columns(pl.col()..split())\n  .with_columns(pl.lit([/i  i  ((, , ))]).alias())\n  .explode(pl.col([, ]))\n</code></pre>\n<pre><code>submission\n  .sort(by=[, , , ], descending=[, , , ])\n  .group_by([], maintain_order=)\n  .agg(pl.col())\n</code></pre>\n<h2>Postprocessing</h2>\n<ul>\n<li>Boost missing misconception rank</li>\n</ul>\n<h2>Accelerating Tips</h2>\n<p>Retreiver</p>\n<ul>\n<li>Run parallel with cuda:0 and cuda:1</li>\n<li>Pre-calculating Misconception Vector</li>\n<li>Dynamic padding with collate_fn</li>\n</ul>\n<p>Reranker</p>\n<ul>\n<li>Inference with vllm (AWQ 4bit quantization)</li>\n<li>Set vllm parameter enabling_prefix_cache True</li>\n<li>Filter candidate with previous preds of pipeline</li>\n</ul>\n<h2>What not worked</h2>\n<p>Retrieval</p>\n<ul>\n<li>Concat vectors</li>\n<li>Avaraging many vectors</li>\n</ul>\n<p>Reranker</p>\n<ul>\n<li>Full-data trained model</li>\n<li>QwQ model</li>\n</ul>\n<p>Thank you!!</p>",
  "comments": [
    {
      "id": 3070678,
      "author_name": "Raja Biswas",
      "author_url": "",
      "post_date": "2024-12-13T00:30:53.510000",
      "content": "<p>Congratulation on becoming Competitions GM, fully deserved! 🎉<br>\nGreat solution, a lot to unpack -- looking forward to reading the detailed post. The QwQ model didn't work for me as well.</p>",
      "votes": 4,
      "replies": [
        {
          "id": 3070693,
          "author_name": "kaerururu",
          "author_url": "",
          "post_date": "2024-12-13T00:52:27.823000",
          "content": "<p>Congratulation on your 1st place too !<br>\nI'm looking forward to your robust solution.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 3071007,
      "author_name": "Ahmet Erdem",
      "author_url": "",
      "post_date": "2024-12-13T08:25:20.687000",
      "content": "<p>I want to add some details.</p>\n<ul>\n<li>We had 3 rerankers each with different nature. 1st one is single misconception yes/no classifier. 2nd one is a multiplechoice over top 40 misconceptions which have misconceptions sorted by their retrieval score. 3rd one is a multiplechoice over top 9 misconceptions which shuffle misconception order during training and also does TTA, so order independent one. They are all Qwen2.5-32B but they are diverse in terms of the way they are trained.</li>\n<li>Our postprocess boosts the misconceptions that are not seen in training set. They boost them iteratively even more unless they become first for a question answer pair. Assumption is that they are here for a reason, they match at least one test set example.</li>\n</ul>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 3073175,
      "author_name": "faraday",
      "author_url": "",
      "post_date": "2024-12-16T06:30:07.530000",
      "content": "<p>What are the examples that the approach could not cope with?<br>\nSince <a href=\"https://www.kaggle.com/aerdem4\" target=\"_blank\">@aerdem4</a> states your postprocess boosts misconceptions unseen in training, did you notice cases that it adds error? Any examples that fail just because of postprocessing?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3070994,
      "author_name": "narsil (jobs-in-data.com)",
      "author_url": "",
      "post_date": "2024-12-13T08:08:59.643000",
      "content": "<p>Congratulations and welcome to the GM family! Great work!</p>",
      "votes": 0,
      "replies": [
        {
          "id": 3071005,
          "author_name": "kaerururu",
          "author_url": "",
          "post_date": "2024-12-13T08:24:07.543000",
          "content": "<p>Thank you sooo much!!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 6,
  "messages": [
    {
      "id": 3070671,
      "postDate": "2024-12-13T00:19:34.077Z",
      "content": "<p>First of all, I'd like to thank kaggle team, host for hosting this competition , and my team mates (@tereka <a href=\"https://www.kaggle.com/irrohas\" target=\"_blank\">@irrohas</a> <a href=\"https://www.kaggle.com/aerdem4\" target=\"_blank\">@aerdem4</a>)<br>\nI finally got 5th gold medal. I'm really happy to be Competitions Grand Master.</p>\n<h2>Overview</h2>\n<p>Voting Ensemble of 3pipelines (Retriever + Reranker)<br>\nAll reranker inference with vllm </p>\n<h2>Pipeline1 by Masaya</h2>\n<p>This is brief summary.<br>\nMasaya will write up detail on this <a href=\"https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/551430\" target=\"_blank\">link</a>.</p>\n<p>Retriever</p>\n<ul>\n<li>Generate test answer with pretrained qwen35B</li>\n<li>model :SFR-Embedding-2_R, Hard Negative Sampling(N=100) removed if unseen in training data.</li>\n<li>dataset: Filled unknown Misconception with GPT4o-mini</li>\n<li>candidate 60</li>\n</ul>\n<p>Reranker</p>\n<ul>\n<li>Qwen2.5 32B AWQ+LoRA (with 40 options)<ul>\n<li>options → 0~9 + A~Z + a~z (40 charactors)</li></ul></li>\n<li>dataset: only original</li>\n<li>candidate 40</li>\n<li>run twice and merge ([:40] and [20:]) </li>\n<li>left 25 with logprobs</li>\n</ul>\n<h2>Pipeline2 by Tereka+Kaeru</h2>\n<p>Retriever</p>\n<ul>\n<li>model1 :SFR-Embedding-2_R、Hard Negative Sampling(100）<ul>\n<li>dataset: original + synthesis with gemma 27B/Qwen2.5 32B</li>\n<li>lb 0.378</li></ul></li>\n<li>model2 : public model by sayoulala</li>\n<li>weighted avg model1 and model2</li>\n<li>candidate 70 and filter with masaya preds</li>\n<li>only single fold</li>\n</ul>\n<p>Reranker</p>\n<ul>\n<li>Qwen2.5 32B AWQ+LoRA (with binary options)<ul>\n<li>options → yes, not</li></ul></li>\n<li>dataset: original + synthesis unknown misconceptions with gemma2 27B/Qwen2.5 32B</li>\n<li>only single fold</li>\n</ul>\n<h2>Pipeline3 by Ahmet</h2>\n<p>Retriever</p>\n<ul>\n<li>Omitted in final pipeline</li>\n</ul>\n<p>Reranker</p>\n<ul>\n<li>Candidates are each top9 from previous pipeline</li>\n<li>Qwen2.5 32B (with 9 options)</li>\n</ul>\n<h2>Ensembling</h2>\n<ul>\n<li>Voting ensemble of each pipeline</li>\n<li>Apply rank like following, join, sum rank, sort by sum_rank</li>\n</ul>\n<pre><code>sub0\n  .with_columns(pl.col()..split())\n  .with_columns(pl.lit([/i  i  ((, , ))]).alias())\n  .explode(pl.col([, ]))\n</code></pre>\n<pre><code>submission\n  .sort(by=[, , , ], descending=[, , , ])\n  .group_by([], maintain_order=)\n  .agg(pl.col())\n</code></pre>\n<h2>Postprocessing</h2>\n<ul>\n<li>Boost missing misconception rank</li>\n</ul>\n<h2>Accelerating Tips</h2>\n<p>Retreiver</p>\n<ul>\n<li>Run parallel with cuda:0 and cuda:1</li>\n<li>Pre-calculating Misconception Vector</li>\n<li>Dynamic padding with collate_fn</li>\n</ul>\n<p>Reranker</p>\n<ul>\n<li>Inference with vllm (AWQ 4bit quantization)</li>\n<li>Set vllm parameter enabling_prefix_cache True</li>\n<li>Filter candidate with previous preds of pipeline</li>\n</ul>\n<h2>What not worked</h2>\n<p>Retrieval</p>\n<ul>\n<li>Concat vectors</li>\n<li>Avaraging many vectors</li>\n</ul>\n<p>Reranker</p>\n<ul>\n<li>Full-data trained model</li>\n<li>QwQ model</li>\n</ul>\n<p>Thank you!!</p>",
      "rawMarkdown": "First of all, I'd like to thank kaggle team, host for hosting this competition , and my team mates (@tereka @irrohas @aerdem4)\nI finally got 5th gold medal. I'm really happy to be Competitions Grand Master.\n\n## Overview\n\nVoting Ensemble of 3pipelines (Retriever + Reranker)\nAll reranker inference with vllm \n\n## Pipeline1 by Masaya\n\nThis is brief summary.\nMasaya will write up detail on this [link](https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/551430).\n\nRetriever\n- Generate test answer with pretrained qwen35B\n- model :SFR-Embedding-2_R, Hard Negative Sampling(N=100) removed if unseen in training data.\n- dataset: Filled unknown Misconception with GPT4o-mini\n- candidate 60\n\nReranker\n- Qwen2.5 32B AWQ+LoRA (with 40 options)\n  - options → 0~9 + A~Z + a~z (40 charactors)\n- dataset: only original\n- candidate 40\n- run twice and merge ([:40] and [20:]) \n- left 25 with logprobs\n\n## Pipeline2 by Tereka+Kaeru\n\nRetriever\n- model1 :SFR-Embedding-2_R、Hard Negative Sampling(100）\n    - dataset: original + synthesis with gemma 27B/Qwen2.5 32B\n    - lb 0.378\n- model2 : public model by sayoulala\n- weighted avg model1 and model2\n- candidate 70 and filter with masaya preds\n- only single fold\n\nReranker\n- Qwen2.5 32B AWQ+LoRA (with binary options)\n  - options → yes, not\n- dataset: original + synthesis unknown misconceptions with gemma2 27B/Qwen2.5 32B\n- only single fold\n\n## Pipeline3 by Ahmet\n\nRetriever\n- Omitted in final pipeline\n\nReranker\n- Candidates are each top9 from previous pipeline\n- Qwen2.5 32B (with 9 options)\n\n## Ensembling\n- Voting ensemble of each pipeline\n- Apply rank like following, join, sum rank, sort by sum_rank\n\n```python\nsub0\n  .with_columns(pl.col(\"MisconceptionId\").str.split(\" \"))\n  .with_columns(pl.lit([1/i for i in list(range(1, 26, 1))]).alias(\"rank\"))\n  .explode(pl.col([\"MisconceptionId\", \"rank\"]))\n```\n\n```python\nsubmission\n  .sort(by=[\"QuestionId_Answer\", \"sum_rank\", \"rank1\", \"rank2\"], descending=[False, True, True, True])\n  .group_by([\"QuestionId_Answer\"], maintain_order=True)\n  .agg(pl.col(\"MisconceptionId\"))\n```\n\n## Postprocessing\n- Boost missing misconception rank\n\n## Accelerating Tips\n\nRetreiver\n- Run parallel with cuda:0 and cuda:1\n- Pre-calculating Misconception Vector\n- Dynamic padding with collate_fn\n\nReranker\n- Inference with vllm (AWQ 4bit quantization)\n- Set vllm parameter enabling_prefix_cache True\n- Filter candidate with previous preds of pipeline\n\n## What not worked\n\nRetrieval\n- Concat vectors\n- Avaraging many vectors\n\nReranker\n- Full-data trained model\n- QwQ model\n\nThank you!!",
      "votes": 58
    },
    {
      "id": 3070678,
      "postDate": "2024-12-13T00:30:53.510Z",
      "content": "<p>Congratulation on becoming Competitions GM, fully deserved! 🎉<br>\nGreat solution, a lot to unpack -- looking forward to reading the detailed post. The QwQ model didn't work for me as well.</p>",
      "rawMarkdown": "Congratulation on becoming Competitions GM, fully deserved! 🎉\nGreat solution, a lot to unpack -- looking forward to reading the detailed post. The QwQ model didn't work for me as well.",
      "votes": 4,
      "replies": [
        {
          "id": 3070693,
          "postDate": "2024-12-13T00:52:27.823Z",
          "content": "<p>Congratulation on your 1st place too !<br>\nI'm looking forward to your robust solution.</p>",
          "rawMarkdown": "Congratulation on your 1st place too !\nI'm looking forward to your robust solution.",
          "votes": 1
        }
      ]
    },
    {
      "id": 3071007,
      "postDate": "2024-12-13T08:25:20.687Z",
      "content": "<p>I want to add some details.</p>\n<ul>\n<li>We had 3 rerankers each with different nature. 1st one is single misconception yes/no classifier. 2nd one is a multiplechoice over top 40 misconceptions which have misconceptions sorted by their retrieval score. 3rd one is a multiplechoice over top 9 misconceptions which shuffle misconception order during training and also does TTA, so order independent one. They are all Qwen2.5-32B but they are diverse in terms of the way they are trained.</li>\n<li>Our postprocess boosts the misconceptions that are not seen in training set. They boost them iteratively even more unless they become first for a question answer pair. Assumption is that they are here for a reason, they match at least one test set example.</li>\n</ul>",
      "rawMarkdown": "I want to add some details.\n\n- We had 3 rerankers each with different nature. 1st one is single misconception yes/no classifier. 2nd one is a multiplechoice over top 40 misconceptions which have misconceptions sorted by their retrieval score. 3rd one is a multiplechoice over top 9 misconceptions which shuffle misconception order during training and also does TTA, so order independent one. They are all Qwen2.5-32B but they are diverse in terms of the way they are trained.\n- Our postprocess boosts the misconceptions that are not seen in training set. They boost them iteratively even more unless they become first for a question answer pair. Assumption is that they are here for a reason, they match at least one test set example.",
      "votes": 1
    },
    {
      "id": 3073175,
      "postDate": "2024-12-16T06:30:07.530Z",
      "content": "<p>What are the examples that the approach could not cope with?<br>\nSince <a href=\"https://www.kaggle.com/aerdem4\" target=\"_blank\">@aerdem4</a> states your postprocess boosts misconceptions unseen in training, did you notice cases that it adds error? Any examples that fail just because of postprocessing?</p>",
      "rawMarkdown": "What are the examples that the approach could not cope with?\nSince @aerdem4 states your postprocess boosts misconceptions unseen in training, did you notice cases that it adds error? Any examples that fail just because of postprocessing?"
    },
    {
      "id": 3070994,
      "postDate": "2024-12-13T08:08:59.643Z",
      "content": "<p>Congratulations and welcome to the GM family! Great work!</p>",
      "rawMarkdown": "Congratulations and welcome to the GM family! Great work!",
      "replies": [
        {
          "id": 3071005,
          "postDate": "2024-12-13T08:24:07.543Z",
          "content": "<p>Thank you sooo much!!</p>",
          "rawMarkdown": "Thank you sooo much!!"
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-20T03:48:28Z",
  "competition_id": 82695,
  "competition_slug": "eedi-mining-misconceptions-in-mathematics",
  "team_id": 12696994,
  "solution_url": "/competitions/eedi-mining-misconceptions-in-mathematics/writeups/terekaerumasahmet-private-7th-public-2nd-place-sol"
}