First of all, I would like to thank to Kaggle and the hosts for organizing this competition and providing an opportunity to learn and improve.. Although my final ranking in this competition is not high, I am sharing my solution as a personal record and in the hope that it may help others in the future.
\nFor the retrieval component, I used this Anh Vo's excellent public notebook implementation without modifications.
\nFor the reranker, I fine-tuned the Qwen2.5-32B-Instruct model. However, I couldn't run the quantization of the fine-tuned model with the vllm library. To address this, I performed inference using the transformers library instead.
Inference using the transformers library was time-consuming. Running a single inference pass on the test data with the 4-bit quantized Qwen2.5-32B-Instruct model (via bitsandbytes) took approximately 6.5 hours. This constraint prevented me from performing multiple inference runs and limited further optimization of the final score.
I fine-tuned the Qwen2.5-32B-Instruct model on an A100 GPU with 4-bit quantization using bitsandbytes. The misconception options provided to Qwen included the correct choice and the top 8 choices retrieved by the Retrieval model, shuffled together.The following prompt template was used during fine-tuning:
PROMPT = \n\n ():\n x = re.sub(, , x)\n x = re.sub(, , x) \n x = re.sub(, , x)\n x = re.sub(, , x)\n x = re.sub(, , x)\n x = re.sub(, , x)\n x = x.strip()\n x\n\n ():\n\n input_text = preprocess_text(PROMPT.(\n ConstructName=row[],\n SubjectName=row[],\n Question=row[],\n IncorrectAnswer=row[],\n CorrectAnswer=row[],\n Retrival=row[],\n )\n )\n\n messages = [\n {\n : ,\n : input_text,\n },\n {\n : ,\n : \n }\n ]\n text = tokenizer.apply_chat_template(messages, tokenize=, add_generation_prompt=)\n item = tokenizer(text, add_special_tokens=, truncation=)\n item[], item[]\nFirst I tried to get good scores only with the public notebook models, but I was able to achieve just Public LB: 0.503 / Private LB: 0.464. Incorporating the fine-tuned Qwen2.5-32B-Instruct model improved these scores to Public LB: 0.529 / Private LB: 0.493. For the final predictions, I combined a single inference run from my fine-tuned model with multiple inference runs from the public Qwen2.5-32B-Instruct-AWQ model, as implemented in this Ahmet's excellent public notebook.
", "comments": [], "comment_count": 0, "messages": [ { "id": 3071894, "postDate": "2024-12-14T13:21:05.433Z", "content": "First of all, I would like to thank to Kaggle and the hosts for organizing this competition and providing an opportunity to learn and improve.. Although my final ranking in this competition is not high, I am sharing my solution as a personal record and in the hope that it may help others in the future.
\nFor the retrieval component, I used this Anh Vo's excellent public notebook implementation without modifications.
\nFor the reranker, I fine-tuned the Qwen2.5-32B-Instruct model. However, I couldn't run the quantization of the fine-tuned model with the vllm library. To address this, I performed inference using the transformers library instead.
Inference using the transformers library was time-consuming. Running a single inference pass on the test data with the 4-bit quantized Qwen2.5-32B-Instruct model (via bitsandbytes) took approximately 6.5 hours. This constraint prevented me from performing multiple inference runs and limited further optimization of the final score.
I fine-tuned the Qwen2.5-32B-Instruct model on an A100 GPU with 4-bit quantization using bitsandbytes. The misconception options provided to Qwen included the correct choice and the top 8 choices retrieved by the Retrieval model, shuffled together.The following prompt template was used during fine-tuning:
PROMPT = \n\n ():\n x = re.sub(, , x)\n x = re.sub(, , x) \n x = re.sub(, , x)\n x = re.sub(, , x)\n x = re.sub(, , x)\n x = re.sub(, , x)\n x = x.strip()\n x\n\n ():\n\n input_text = preprocess_text(PROMPT.(\n ConstructName=row[],\n SubjectName=row[],\n Question=row[],\n IncorrectAnswer=row[],\n CorrectAnswer=row[],\n Retrival=row[],\n )\n )\n\n messages = [\n {\n : ,\n : input_text,\n },\n {\n : ,\n : \n }\n ]\n text = tokenizer.apply_chat_template(messages, tokenize=, add_generation_prompt=)\n item = tokenizer(text, add_special_tokens=, truncation=)\n item[], item[]\nFirst I tried to get good scores only with the public notebook models, but I was able to achieve just Public LB: 0.503 / Private LB: 0.464. Incorporating the fine-tuned Qwen2.5-32B-Instruct model improved these scores to Public LB: 0.529 / Private LB: 0.493. For the final predictions, I combined a single inference run from my fine-tuned model with multiple inference runs from the public Qwen2.5-32B-Instruct-AWQ model, as implemented in this Ahmet's excellent public notebook.
", "rawMarkdown": "First of all, I would like to thank to Kaggle and the hosts for organizing this competition and providing an opportunity to learn and improve.. Although my final ranking in this competition is not high, I am sharing my solution as a personal record and in the hope that it may help others in the future.\n\n## Solution Overview\n\n### Retrieval\n\nFor the retrieval component, I used [this Anh Vo's excellent public notebook](https://www.kaggle.com/code/anhvth226/eedi-11-21-14b) implementation without modifications.\n\n### Reranker\n\nFor the reranker, I fine-tuned the **Qwen2.5-32B-Instruct** model. However, I couldn't run the quantization of the fine-tuned model with the `vllm` library. To address this, I performed inference using the `transformers` library instead.\n\nInference using the `transformers` library was time-consuming. Running a single inference pass on the test data with the 4-bit quantized Qwen2.5-32B-Instruct model (via `bitsandbytes`) took approximately **6.5 hours**. This constraint prevented me from performing multiple inference runs and limited further optimization of the final score.\n\n### Fine-Tuning Details\n\nI fine-tuned the Qwen2.5-32B-Instruct model on an A100 GPU with 4-bit quantization using `bitsandbytes`. The misconception options provided to Qwen included the correct choice and the top 8 choices retrieved by the Retrieval model, shuffled together.The following prompt template was used during fine-tuning:\n\n```\nPROMPT = \"\"\"\nHere is a question about {ConstructName}({SubjectName}).\nQuestion: {Question}\nCorrect Answer: {CorrectAnswer}\nIncorrect Answer: {IncorrectAnswer}\n\nYou are a Mathematics teacher. Your task is to reason and identify the misconception behind the Incorrect Answer with the Question.\nAnswer concisely what misconception it is to lead to getting the incorrect answer.\nPick the correct misconception number from the below:\n\n{Retrival}\n\"\"\"\n\ndef preprocess_text(x):\n x = re.sub(\"http\\w+\", \"\", x)\n x = re.sub(r\"\\.+\", \".\", x) \n x = re.sub(r\"\\,+\", \",\", x)\n x = re.sub(r\"\\\\\\(\", \" \", x)\n x = re.sub(r\"\\\\\\)\", \" \", x)\n x = re.sub(r\"[ ]{1,}\", \" \", x)\n x = x.strip()\n return x\n\ndef apply_template(row):\n\n input_text = preprocess_text(PROMPT.format(\n ConstructName=row[\"ConstructName\"],\n SubjectName=row[\"SubjectName\"],\n Question=row[\"QuestionText\"],\n IncorrectAnswer=row[\"answer_name\"],\n CorrectAnswer=row[\"CorrectAnswer\"],\n Retrival=row[\"retrieval\"],\n )\n )\n\n messages = [\n {\n \"role\": \"user\",\n \"content\": input_text,\n },\n {\n \"role\": \"assistant\",\n \"content\": f\"Answer:{row.true}\"\n }\n ]\n text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)\n item = tokenizer(text, add_special_tokens=False, truncation=False)\n return item[\"input_ids\"], item[\"attention_mask\"]\n```\n\n### Results\n\nFirst I tried to get good scores only with the public notebook models, but I was able to achieve just **Public LB: 0.503 / Private LB: 0.464**. Incorporating the fine-tuned Qwen2.5-32B-Instruct model improved these scores to **Public LB: 0.529 / Private LB: 0.493**. For the final predictions, I combined a single inference run from my fine-tuned model with multiple inference runs from the public Qwen2.5-32B-Instruct-AWQ model, as implemented in [this Ahmet's excellent public notebook](https://www.kaggle.com/code/aerdem4/eedi-qwen32b-vllm-with-logits-processor-zoo).", "votes": 5 } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T03:44:47Z", "competition_id": 82695, "competition_slug": "eedi-mining-misconceptions-in-mathematics", "team_id": 12603228, "solution_url": "/competitions/eedi-mining-misconceptions-in-mathematics/writeups/ktokunaga-68th-place-solution" }