{
  "id": 551406,
  "title": "26th place solution(Public 14th)",
  "content": "<p>Thank you for hosting a wonderful contest.  <br>\nCongratulations to those who placed at the top, and I would like to praise everyone who participated.<br>\nI would like to briefly share my solution.</p>\n<h2>Overview</h2>\n<ul>\n<li>Used Retrieve Model to obtain the top 160 misconception candidates for each <code>QuestionID_Answer</code>.</li>\n<li>Rerank Model evaluated the candidates and selected the top 25.</li>\n<li>Efficient training and inference achieved through Prefix Caching and Negative Downsampling with vLLM.</li>\n</ul>\n<h2>Validation Strategy</h2>\n<ul>\n<li>Group KFold (group = question_id)</li>\n</ul>\n<h2>Training Phase</h2>\n<h3>Retriever</h3>\n<ul>\n<li>Trained using Sentence Transformer based on a public notebook.</li>\n<li>Backbone: Salesforce/SFR-Embedding-2_R.</li>\n<li>Fine-tuned with LoRA.</li>\n<li><code>num_retrieve</code>: 100.</li>\n<li>Negative samples downsampled:<ul>\n<li>For each <code>questionId_Answer</code>, randomly sampled 3 out of 100 examples for training.</li></ul></li>\n</ul>\n<h3>Reranker</h3>\n<ul>\n<li>Leveraged out-of-fold (OOF) predictions from the Retriever stage.</li>\n<li>Extracted top K (K = 100) misconception candidates for each <code>QuestionID_Answer</code> using the retriever.</li>\n<li>Input: <code>question_text + correct answer + wrong answer + misconception</code>.</li>\n<li>Task: Output <code>Yes</code> or <code>No</code>.</li>\n<li>Negative samples were downsampled in the same way as the retriever.</li>\n<li>Prompt: </li>\n</ul>\n<pre><code></code></pre>\n<ul>\n<li>Backbone: Qwen 32b AWQ or GPTQ.</li>\n<li>Fine-tuned with LoRA.</li>\n</ul>\n<h2>Inference Phase</h2>\n<h3>Retriever</h3>\n<ul>\n<li>Merged LoRA model for inference with vLLM.<ul>\n<li>since vLLM embedding models do not natively support LoRA.</li></ul></li>\n<li>Calculated similarities and selected the top K(K=160).</li>\n</ul>\n<h3>Reranker</h3>\n<ul>\n<li>Used vLLM.</li>\n<li>Forced output of either <code>Yes</code> or <code>No</code> token and output the corresponding token probabilities for reranking.</li>\n<li>Implemented Prefix Caching in vLLM for significant speedup:<ul>\n<li>Required inference for top K candidates per question-answer pair.</li>\n<li>Prompt content up to the final misconception remained the same.</li>\n<li>Prefix Caching allowed efficient inference for many candidates using the Rerank Model.</li></ul></li>\n</ul>\n<h2>What Doesn't Work</h2>\n<ul>\n<li>Train models before quantization (without AWQ or GPTQ) and attempted custom quantization.</li>\n<li>Hard Example Mining:<ul>\n<li>Improved CV but not LB.</li></ul></li>\n<li>QWQ model.</li>\n<li>Concatenating other high-ranking misconceptions in the Retriever.</li>\n<li>Concatenating other options.</li>\n</ul>",
  "comments": [
    {
      "id": 3071056,
      "author_name": "Octavi Grau",
      "author_url": "",
      "post_date": "2024-12-13T09:33:47.177000",
      "content": "<p>Congrats <a href=\"https://www.kaggle.com/fuumin621\" target=\"_blank\">@fuumin621</a> this is an excellent write-up!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 3070803,
      "author_name": "Zhuang Jia",
      "author_url": "",
      "post_date": "2024-12-13T03:59:20.737000",
      "content": "<p>Congratulations! Would like to learn the Prefix Caching in vLLM for speedup, will you release your code?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 3071002,
          "author_name": "monnu",
          "author_url": "",
          "post_date": "2024-12-13T08:20:34.567000",
          "content": "<p>I'm sorry, but I don't plan to release the code. However, the prefix_caching part can be implemented like this in vLLM:</p>\n<pre><code>llm = LLM(\n    LLM_MODEL_PATH,\n    quantization=,\n    tensor_parallel_size=, \n    gpu_memory_utilization=, \n    trust_remote_code=,\n    dtype=, \n    enforce_eager=,\n    max_model_len=,\n    enable_prefix_caching=,\n    enable_lora=,\n)\n</code></pre>",
          "votes": 1,
          "replies": []
        }
      ]
    }
  ],
  "comment_count": 3,
  "messages": [
    {
      "id": 3070770,
      "postDate": "2024-12-13T03:05:50.390Z",
      "content": "<p>Thank you for hosting a wonderful contest.  <br>\nCongratulations to those who placed at the top, and I would like to praise everyone who participated.<br>\nI would like to briefly share my solution.</p>\n<h2>Overview</h2>\n<ul>\n<li>Used Retrieve Model to obtain the top 160 misconception candidates for each <code>QuestionID_Answer</code>.</li>\n<li>Rerank Model evaluated the candidates and selected the top 25.</li>\n<li>Efficient training and inference achieved through Prefix Caching and Negative Downsampling with vLLM.</li>\n</ul>\n<h2>Validation Strategy</h2>\n<ul>\n<li>Group KFold (group = question_id)</li>\n</ul>\n<h2>Training Phase</h2>\n<h3>Retriever</h3>\n<ul>\n<li>Trained using Sentence Transformer based on a public notebook.</li>\n<li>Backbone: Salesforce/SFR-Embedding-2_R.</li>\n<li>Fine-tuned with LoRA.</li>\n<li><code>num_retrieve</code>: 100.</li>\n<li>Negative samples downsampled:<ul>\n<li>For each <code>questionId_Answer</code>, randomly sampled 3 out of 100 examples for training.</li></ul></li>\n</ul>\n<h3>Reranker</h3>\n<ul>\n<li>Leveraged out-of-fold (OOF) predictions from the Retriever stage.</li>\n<li>Extracted top K (K = 100) misconception candidates for each <code>QuestionID_Answer</code> using the retriever.</li>\n<li>Input: <code>question_text + correct answer + wrong answer + misconception</code>.</li>\n<li>Task: Output <code>Yes</code> or <code>No</code>.</li>\n<li>Negative samples were downsampled in the same way as the retriever.</li>\n<li>Prompt: </li>\n</ul>\n<pre><code></code></pre>\n<ul>\n<li>Backbone: Qwen 32b AWQ or GPTQ.</li>\n<li>Fine-tuned with LoRA.</li>\n</ul>\n<h2>Inference Phase</h2>\n<h3>Retriever</h3>\n<ul>\n<li>Merged LoRA model for inference with vLLM.<ul>\n<li>since vLLM embedding models do not natively support LoRA.</li></ul></li>\n<li>Calculated similarities and selected the top K(K=160).</li>\n</ul>\n<h3>Reranker</h3>\n<ul>\n<li>Used vLLM.</li>\n<li>Forced output of either <code>Yes</code> or <code>No</code> token and output the corresponding token probabilities for reranking.</li>\n<li>Implemented Prefix Caching in vLLM for significant speedup:<ul>\n<li>Required inference for top K candidates per question-answer pair.</li>\n<li>Prompt content up to the final misconception remained the same.</li>\n<li>Prefix Caching allowed efficient inference for many candidates using the Rerank Model.</li></ul></li>\n</ul>\n<h2>What Doesn't Work</h2>\n<ul>\n<li>Train models before quantization (without AWQ or GPTQ) and attempted custom quantization.</li>\n<li>Hard Example Mining:<ul>\n<li>Improved CV but not LB.</li></ul></li>\n<li>QWQ model.</li>\n<li>Concatenating other high-ranking misconceptions in the Retriever.</li>\n<li>Concatenating other options.</li>\n</ul>",
      "rawMarkdown": "Thank you for hosting a wonderful contest.  \nCongratulations to those who placed at the top, and I would like to praise everyone who participated.\nI would like to briefly share my solution.\n\n## Overview\n- Used Retrieve Model to obtain the top 160 misconception candidates for each `QuestionID_Answer`.\n- Rerank Model evaluated the candidates and selected the top 25.\n- Efficient training and inference achieved through Prefix Caching and Negative Downsampling with vLLM.\n\n## Validation Strategy\n- Group KFold (group = question_id)\n\n## Training Phase\n\n### Retriever\n- Trained using Sentence Transformer based on a public notebook.\n- Backbone: Salesforce/SFR-Embedding-2_R.\n- Fine-tuned with LoRA.\n- `num_retrieve`: 100.\n- Negative samples downsampled:\n  - For each `questionId_Answer`, randomly sampled 3 out of 100 examples for training.\n\n### Reranker\n- Leveraged out-of-fold (OOF) predictions from the Retriever stage.\n- Extracted top K (K = 100) misconception candidates for each `QuestionID_Answer` using the retriever.\n- Input: `question_text + correct answer + wrong answer + misconception`.\n- Task: Output `Yes` or `No`.\n- Negative samples were downsampled in the same way as the retriever.\n- Prompt: \n\n```\n<|im_start|>user\nIs the incorrect answer caused by the misconception? Answer Yes/No.\n<Problem>: {Question}\n<Construct>: {ConstructName}\n<Subject>: {SubjectName}\n<Correct Answer>: {Correct_Answer}\n<Incorrect Answer>: {Answer}\n<Misconception>: {PredMisconceptionName}\n\n<|im_end|>\n<|im_start|>assistant\nAnswer:{label}\n```\n\n- Backbone: Qwen 32b AWQ or GPTQ.\n- Fine-tuned with LoRA.\n\n## Inference Phase\n\n### Retriever\n- Merged LoRA model for inference with vLLM.\n  -  since vLLM embedding models do not natively support LoRA.\n- Calculated similarities and selected the top K(K=160).\n\n### Reranker\n- Used vLLM.\n- Forced output of either `Yes` or `No` token and output the corresponding token probabilities for reranking.\n- Implemented Prefix Caching in vLLM for significant speedup:\n  - Required inference for top K candidates per question-answer pair.\n  - Prompt content up to the final misconception remained the same.\n  - Prefix Caching allowed efficient inference for many candidates using the Rerank Model.\n\n## What Doesn't Work\n- Train models before quantization (without AWQ or GPTQ) and attempted custom quantization.\n- Hard Example Mining:\n  - Improved CV but not LB.\n- QWQ model.\n- Concatenating other high-ranking misconceptions in the Retriever.\n- Concatenating other options.",
      "votes": 18
    },
    {
      "id": 3071056,
      "postDate": "2024-12-13T09:33:47.177Z",
      "content": "<p>Congrats <a href=\"https://www.kaggle.com/fuumin621\" target=\"_blank\">@fuumin621</a> this is an excellent write-up!</p>",
      "rawMarkdown": "Congrats @fuumin621 this is an excellent write-up!"
    },
    {
      "id": 3070803,
      "postDate": "2024-12-13T03:59:20.737Z",
      "content": "<p>Congratulations! Would like to learn the Prefix Caching in vLLM for speedup, will you release your code?</p>",
      "rawMarkdown": "Congratulations! Would like to learn the Prefix Caching in vLLM for speedup, will you release your code?",
      "replies": [
        {
          "id": 3071002,
          "postDate": "2024-12-13T08:20:34.567Z",
          "content": "<p>I'm sorry, but I don't plan to release the code. However, the prefix_caching part can be implemented like this in vLLM:</p>\n<pre><code>llm = LLM(\n    LLM_MODEL_PATH,\n    quantization=,\n    tensor_parallel_size=, \n    gpu_memory_utilization=, \n    trust_remote_code=,\n    dtype=, \n    enforce_eager=,\n    max_model_len=,\n    enable_prefix_caching=,\n    enable_lora=,\n)\n</code></pre>",
          "rawMarkdown": "I'm sorry, but I don't plan to release the code. However, the prefix_caching part can be implemented like this in vLLM:\n\n```python\nllm = LLM(\n    LLM_MODEL_PATH,\n    quantization=\"gptq\",\n    tensor_parallel_size=2, \n    gpu_memory_utilization=0.95, \n    trust_remote_code=True,\n    dtype=\"half\", \n    enforce_eager=True,\n    max_model_len=768,\n    enable_prefix_caching=True,\n    enable_lora=True,\n)\n```",
          "votes": 1
        }
      ]
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-23T16:34:45Z",
  "competition_id": 82695,
  "competition_slug": "eedi-mining-misconceptions-in-mathematics",
  "team_id": 12601993,
  "solution_url": "/competitions/eedi-mining-misconceptions-in-mathematics/writeups/monnu-26th-place-solution-public-14th"
}