{ "id": 551412, "title": "8th Place Solution", "content": "

After reading the top solutions, I’m truly impressed by other teams’ dedicated efforts, great attention to training & inference details, and highly sophiscated methods of prompt engineering. As a novice, I feel fortunate to have achieved my final ranking and am humbled to share my solution. I think my solution, yet simple, but is still effecitve and comparatively computing efficient. I used only 14B and 32B models, with a total running time of about 6 hours.

\n

I’d like to extend my gratitude to everyone who shared their code and insights, both during and after the competition.

\n

Two key references

\n\n

Inference pipeline

\n

retrievers (4 Qwen2.5-14b models ensembled by weighted sum) → 40 misconceptions → reranker (a single Qwen2.5-32b model)→ 25 misconceptions
\n

\n

Training details

\n

In general, I find iterative hard negative mining very helpful.

\n\n


\n

Quantization & Inference details

\n\n


\n

Key improvements (Public LB)

\n\n


\n

Personal reflections

\n

The entire pipeline — from LoRA finetuning to quantization — was very new to me, and I nearly gave up more than once. I started small with Qwen2.5-0.5B to establish a workable process, but the AWQ-quantized model kept producing random letters during inference. After much frustration, I discovered that autoawq didn’t export the im-head-embed layer parameters for Qwen2.5-0.5B, forcing me to add them manually. Without @cdeotte‘s paper proving it could be done, I might have quit altogether. Thanks to that reassurance, I stuck with it and eventually got the pipeline running smoothly.
\n

\n

Code

\n", "comments": [ { "id": 3070996, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2024-12-13T08:11:37.327000", "content": "
\n

and I nearly gave up more than once

\n
\n

this is the trait for all great achievements. I had such thoughts so many times in all my competitions. Congrats for sticking to it - solo is super difficult - and now you have achieved something remarkable. You can be proud. Great work!

", "votes": 4, "replies": [ { "id": 3071210, "author_name": "Yannan Chen", "author_url": "", "post_date": "2024-12-13T13:42:37.407000", "content": "

Thanks for the kind words! It’s great to find someone who resonates! Soloing feels like venturing into the darkness, but the moment you catch a glimpse of light is truly exhilarating.

", "votes": 2, "replies": [] } ] }, { "id": 3072795, "author_name": "Rudi Özgen", "author_url": "", "post_date": "2024-12-15T16:33:14.277000", "content": "

Endless thanks for your incredible generosity.

", "votes": 1, "replies": [] }, { "id": 3071174, "author_name": "Nikhil Tumbde", "author_url": "", "post_date": "2024-12-13T12:58:50.693000", "content": "

Thanks for the solution. The enable_prefix_caching parameter and wise-FT were new to me. I have few questions for you as this is a very new space for me, it would be really helpful if you could help me understand (maybe please include these pointers in your next writeup).

\n
    \n
  1. can you share the prompts (or some idea around it) used for synthetic data generation? Also, did you generate the data for misconceptions not seen in the training data
    \n\"Added 983 synthetic samples to fill missing misconceptions.\"
    2. \n
  2. can you also share how to do this manually as I also struggled with the same ?
    3. \n
\n
\n

After much frustration, I discovered that autoawq didn’t export the im-head-embed layer parameters for Qwen2.5-0.5B, forcing me to add them manually.>

\n
", "votes": 1, "replies": [ { "id": 3071197, "author_name": "Yannan Chen", "author_url": "", "post_date": "2024-12-13T13:24:29.493000", "content": "
    \n
  1. yes, I generated samples for misconceptions not seen in the training data. When doing OOF training, the fold used for validation also contains some unique misconceptions, I generated for that too. The prompt I used was built in my native language but the key components are:
    2. \n
\n", "votes": 2, "replies": [] }, { "id": 3071203, "author_name": "Yannan Chen", "author_url": "", "post_date": "2024-12-13T13:32:23.177000", "content": "

2.This is a very weird issue. Here is how to manually adjust the params file:

\n

the safetensors file can be loaded as just simple dictionary format using lib from safetensors.torch import safe_open, load_file, save_file, with its keys being the name of each layer, and the values being the tensors.

\n

So after the file is loaded, you just need to compare with the un-quantized model's params (loaded in as dict too), see if there is any mismatch in keys. If so, modify the dict by code like quant_params[k] = unquant_params.get(k). Eventually save the modified params back.

", "votes": 2, "replies": [ { "id": 3071897, "author_name": "Nikhil Tumbde", "author_url": "", "post_date": "2024-12-14T13:25:19.430000", "content": "

thanks a lot. That helps.

", "votes": 0, "replies": [] } ] } ] }, { "id": 3071057, "author_name": "Octavi Grau", "author_url": "", "post_date": "2024-12-13T09:35:04.690000", "content": "

Congrats @yannan90 ! Congrats for the medal, for the write-up and for not giving up! Kaggle is an amazing place

", "votes": 1, "replies": [ { "id": 3071220, "author_name": "Yannan Chen", "author_url": "", "post_date": "2024-12-13T13:52:20.733000", "content": "

Thank you so much! Kaggle is truly an amazing community for learning and growth.

", "votes": 0, "replies": [] } ] }, { "id": 3229688, "author_name": "DuangDD", "author_url": "", "post_date": "2025-06-22T01:16:37.150000", "content": "

Great work！Could you please share your parameter settings for early stop, because when I was training with raw data, I stopped early before the second epoch was finished.

", "votes": 0, "replies": [] }, { "id": 3073189, "author_name": "MrSimple", "author_url": "", "post_date": "2024-12-16T06:39:00.553000", "content": "

Endless thanks for your incredible generosity.

", "votes": 0, "replies": [] }, { "id": 3078655, "author_name": "", "author_url": "", "post_date": "2024-12-22T16:22:54.913000", "content": "", "votes": 1, "replies": [] }, { "id": 3078212, "author_name": "", "author_url": "", "post_date": "2024-12-22T00:25:58.983000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 13, "messages": [ { "id": 3070804, "postDate": "2024-12-13T04:00:22.713Z", "content": "

After reading the top solutions, I’m truly impressed by other teams’ dedicated efforts, great attention to training & inference details, and highly sophiscated methods of prompt engineering. As a novice, I feel fortunate to have achieved my final ranking and am humbled to share my solution. I think my solution, yet simple, but is still effecitve and comparatively computing efficient. I used only 14B and 32B models, with a total running time of about 6 hours.

\n

I’d like to extend my gratitude to everyone who shared their code and insights, both during and after the competition.

\n

Two key references

\n\n

Inference pipeline

\n

retrievers (4 Qwen2.5-14b models ensembled by weighted sum) → 40 misconceptions → reranker (a single Qwen2.5-32b model)→ 25 misconceptions
\n

\n

Training details

\n

In general, I find iterative hard negative mining very helpful.

\n\n


\n

Quantization & Inference details

\n\n


\n

Key improvements (Public LB)

\n\n


\n

Personal reflections

\n

The entire pipeline — from LoRA finetuning to quantization — was very new to me, and I nearly gave up more than once. I started small with Qwen2.5-0.5B to establish a workable process, but the AWQ-quantized model kept producing random letters during inference. After much frustration, I discovered that autoawq didn’t export the im-head-embed layer parameters for Qwen2.5-0.5B, forcing me to add them manually. Without @cdeotte‘s paper proving it could be done, I might have quit altogether. Thanks to that reassurance, I stuck with it and eventually got the pipeline running smoothly.
\n

\n

Code

\n", "rawMarkdown": "After reading the top solutions, I’m truly impressed by other teams’ dedicated efforts, great attention to training & inference details, and highly sophiscated methods of prompt engineering. As a novice, I feel fortunate to have achieved my final ranking and am humbled to share my solution. I think my solution, yet simple, but is still effecitve and comparatively computing efficient. I used only 14B and 32B models, with a total running time of about 6 hours.\n\nI’d like to extend my gratitude to everyone who shared their code and insights, both during and after the competition.\n\n## Two key references\n- @sayoulala : https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/543519\n - Enlightened me on LoRA finetuning, hard negative mining, and recall+rerank training scheme.\n- @cdeotte : https://www.kaggle.com/competitions/lmsys-chatbot-arena/discussion/521294\n - Inspiring ideas on model quantization, wise-FT LoRA adaptation, and VLLM inference.\n
\n\n## Inference pipeline\nretrievers (4 Qwen2.5-14b models ensembled by weighted sum) → 40 misconceptions → reranker (a single Qwen2.5-32b model)→ 25 misconceptions\n
\n\n## Training details\nIn general, I find iterative hard negative mining very helpful.\n\n- Use few-shots prompt with chatgpt to generate extra training data: \n - Added 983 synthetic data samples to fill missing misconceptions. \n - training data is split in 5 CV-folds by `\"QuestionId\"`. For each fold, regenerated extra samples to cover unique misconceptions in the valid fold for training. \n- Retriever (Qwen2.5-14b): Finetuned with bnb bit4 + LoRA for FEATURE_EXTRACTION task. \n - trained on 2 A100s.\n - lora parameters: `r=32, alpha=64, target_modules=['q_proj','k_proj','v_proj','o_proj','gate_proj','up_proj','down_proj']`\n - training paramters: `train_group_size=4, effective_batch_size=64, epoch=7, lr_scheduler_type=\"cosine\"`; \n - trained in 5 CV-folds.\n - For first 4~5 epochs, model is trained without generated data, with a focus on improving map25. \n - For the final 2~3 epochs, generated data is added into training, which helps to further improve map25 and recall25.\n - I interfere manually when the valid scores (map25 & recall25) stop improving by following means:\n - reducing learning_rate gradually from 5e-4 to 1e-5\n - reducing negative data pool size from 80 to 4, for more focus on distinguishing hardest misconceptions\n- Reranker (Qwen2.5-32b): Finetuned with bnb bit4 + LoRA for CASUAL_ML task, outputting a “Yes” logit for each query–misconception pair. \n - trained on 2 A100s.\n - lora parameters: `r=32, alpha=64, target_modules=['q_proj','k_proj','v_proj','o_proj','gate_proj','up_proj','down_proj','down_proj','lm_head']`\n - training paramters: `train_group_size=4, effective_batch_size=64, epoch=4, learning_rate=5e-4, lr_scheduler_type=\"WarmupDecayLR\"`; \n - full dataset used for training reranker without validation. Negative data is generated by 4 retrievers (emsembled through weighted sum) with negative_count = 40\n\n
\n\n## Quantization & Inference details\n- Retrieving:\n - 4 retrievers (chosen from 5-fold CV based on LB scores) retrieve 4 times, then ensembled via weighted sum.\n - infer in bnb bit4+LoRA (no need to merge), so that each single retriever can fit in one T4 gpu, 4 retrievers only take 2 T4s running twice.\n- Reranking:\n - model is first merged with lora adaptor, with wise-FT hyper parameter alpha=0.8\n - model is then quantized to AWQ (calibrated using 128 samples randomly selected from training data) \n - VLLM is used for final inference. Each query is paired with 40 misconceptions retrieved, generating 40 queries for reranker to score. \"enable_prefix_caching\" is turned on to accerlerate inference.\n\n
\n\n## Key improvements (Public LB)\n- Add Synthetic data: single retriever public LB score from 0.3~ → 0.49 \n- Ensemble retrievers: 0.49 → 0.51\n- Add reranker: 0.51 → 0.57 \n- Apply Wise-FT on reranker's LoRA merging: 0.57 → 0.59\n\n
\n\n## Personal reflections\nThe entire pipeline — from LoRA finetuning to quantization — was very new to me, and I nearly gave up more than once. I started small with Qwen2.5-0.5B to establish a workable process, but the AWQ-quantized model kept producing random letters during inference. After much frustration, I discovered that autoawq didn’t export the im-head-embed layer parameters for Qwen2.5-0.5B, forcing me to add them manually. Without @cdeotte‘s paper proving it could be done, I might have quit altogether. Thanks to that reassurance, I stuck with it and eventually got the pipeline running smoothly.\n
\n\n## Code\n- Inference code: https://www.kaggle.com/code/yannan90/eedi-recall-rerank-lb-0-601-pb-0-584\n- Training code: https://github.com/yannan90/kaggle-2024-Eedi-8th-place-training-code\n", "votes": 34 }, { "id": 3070996, "postDate": "2024-12-13T08:11:37.327Z", "content": "
\n

and I nearly gave up more than once

\n
\n

this is the trait for all great achievements. I had such thoughts so many times in all my competitions. Congrats for sticking to it - solo is super difficult - and now you have achieved something remarkable. You can be proud. Great work!

", "rawMarkdown": ">and I nearly gave up more than once\n\nthis is the trait for all great achievements. I had such thoughts so many times in all my competitions. Congrats for sticking to it - solo is super difficult - and now you have achieved something remarkable. You can be proud. Great work!", "votes": 4, "replies": [ { "id": 3071210, "postDate": "2024-12-13T13:42:37.407Z", "content": "

Thanks for the kind words! It’s great to find someone who resonates! Soloing feels like venturing into the darkness, but the moment you catch a glimpse of light is truly exhilarating.

", "rawMarkdown": "Thanks for the kind words! It’s great to find someone who resonates! Soloing feels like venturing into the darkness, but the moment you catch a glimpse of light is truly exhilarating.", "votes": 2 } ] }, { "id": 3072795, "postDate": "2024-12-15T16:33:14.277Z", "content": "

Endless thanks for your incredible generosity.

", "rawMarkdown": "Endless thanks for your incredible generosity.", "votes": 1 }, { "id": 3071174, "postDate": "2024-12-13T12:58:50.693Z", "content": "

Thanks for the solution. The enable_prefix_caching parameter and wise-FT were new to me. I have few questions for you as this is a very new space for me, it would be really helpful if you could help me understand (maybe please include these pointers in your next writeup).

\n
    \n
  1. can you share the prompts (or some idea around it) used for synthetic data generation? Also, did you generate the data for misconceptions not seen in the training data
    \n\"Added 983 synthetic samples to fill missing misconceptions.\"
    2. \n
  2. can you also share how to do this manually as I also struggled with the same ?
    3. \n
\n
\n

After much frustration, I discovered that autoawq didn’t export the im-head-embed layer parameters for Qwen2.5-0.5B, forcing me to add them manually.>

\n
", "rawMarkdown": "\nThanks for the solution. The *enable_prefix_caching* parameter and wise-FT were new to me. I have few questions for you as this is a very new space for me, it would be really helpful if you could help me understand (maybe please include these pointers in your next writeup).\n\n1. can you share the prompts (or some idea around it) used for synthetic data generation? Also, did you generate the data for misconceptions not seen in the training data \n\"Added 983 synthetic samples to fill missing misconceptions.\"\n\n2. can you also share how to do this manually as I also struggled with the same ?\n>After much frustration, I discovered that autoawq didn’t export the im-head-embed layer parameters for Qwen2.5-0.5B, forcing me to add them manually.>\n\n", "votes": 1, "replies": [ { "id": 3071197, "postDate": "2024-12-13T13:24:29.493Z", "content": "
    \n
  1. yes, I generated samples for misconceptions not seen in the training data. When doing OOF training, the fold used for validation also contains some unique misconceptions, I generated for that too. The prompt I used was built in my native language but the key components are:
    2. \n
\n", "rawMarkdown": "1. yes, I generated samples for misconceptions not seen in the training data. When doing OOF training, the fold used for validation also contains some unique misconceptions, I generated for that too. The prompt I used was built in my native language but the key components are:\n - explain the task and format of each new data sample you want to AI to generate (ConstructName, SubjectName, QuestionText, CorrectAnswerText...)\n - give 3~4 examples randomly selected from existing train data as example.\n - give the new misconceptions you want AI to generate based upon. (could be one, or a list)\n - remind AI to not miss any misconceptions, return back proper format, and any other limitations you want to add.\n\n", "votes": 2 }, { "id": 3071203, "postDate": "2024-12-13T13:32:23.177Z", "content": "

2.This is a very weird issue. Here is how to manually adjust the params file:

\n

the safetensors file can be loaded as just simple dictionary format using lib from safetensors.torch import safe_open, load_file, save_file, with its keys being the name of each layer, and the values being the tensors.

\n

So after the file is loaded, you just need to compare with the un-quantized model's params (loaded in as dict too), see if there is any mismatch in keys. If so, modify the dict by code like quant_params[k] = unquant_params.get(k). Eventually save the modified params back.

", "rawMarkdown": "2.This is a very weird issue. Here is how to manually adjust the params file:\n\nthe safetensors file can be loaded as just simple dictionary format using lib `from safetensors.torch import safe_open, load_file, save_file`, with its keys being the name of each layer, and the values being the tensors.\n\nSo after the file is loaded, you just need to compare with the un-quantized model's params (loaded in as dict too), see if there is any mismatch in keys. If so, modify the dict by code like `quant_params[k] = unquant_params.get(k)`. Eventually save the modified params back.", "votes": 2, "replies": [ { "id": 3071897, "postDate": "2024-12-14T13:25:19.430Z", "content": "

thanks a lot. That helps.

", "rawMarkdown": "thanks a lot. That helps." } ] } ] }, { "id": 3071057, "postDate": "2024-12-13T09:35:04.690Z", "content": "

Congrats @yannan90 ! Congrats for the medal, for the write-up and for not giving up! Kaggle is an amazing place

", "rawMarkdown": "Congrats @yannan90 ! Congrats for the medal, for the write-up and for not giving up! Kaggle is an amazing place", "votes": 1, "replies": [ { "id": 3071220, "postDate": "2024-12-13T13:52:20.733Z", "content": "

Thank you so much! Kaggle is truly an amazing community for learning and growth.

", "rawMarkdown": "Thank you so much! Kaggle is truly an amazing community for learning and growth." } ] }, { "id": 3229688, "postDate": "2025-06-22T01:16:37.150Z", "content": "

Great work！Could you please share your parameter settings for early stop, because when I was training with raw data, I stopped early before the second epoch was finished.

", "rawMarkdown": "Great work！Could you please share your parameter settings for early stop, because when I was training with raw data, I stopped early before the second epoch was finished." }, { "id": 3073189, "postDate": "2024-12-16T06:39:00.553Z", "content": "

Endless thanks for your incredible generosity.

", "rawMarkdown": "Endless thanks for your incredible generosity.\n\n\n" }, { "id": 3078655, "postDate": "2024-12-22T16:22:54.913Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 3078212, "postDate": "2024-12-22T00:25:58.983Z", "rawMarkdown": "", "isDeleted": true } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T03:46:34Z", "competition_id": 82695, "competition_slug": "eedi-mining-misconceptions-in-mathematics", "team_id": 12680691, "solution_url": "/competitions/eedi-mining-misconceptions-in-mathematics/writeups/yannan-chen-8th-place-solution" }