Thank you to the excellent public notebook, from which I've learned a lot.
\nI made simple modifications to the public notebook and achieved 30th place.
\nReference: https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/543519
\nI fine-tuned qwen2.5-14b use [FlagEmbedding], using the unfine-tuned SFR-Embedding-2_R as the seed model, obtaining 100 negative sample instances, with epoch=20.
\nInference stage, I used load_in_8bit.
I actually did not run this part of the code.
\nReference:
\nI fine-tuned qwen-72B with accelerate+qlora [use LLaMA-Factory] , and then quantified it with GPTQ, to better adapt it to the 9-out-of-1 task. The fine-tuning brought about an improvement in performance.
\nUnlike the public notebook which uses logits_processor_zoo to get the candidate with the highest probability, I added the logprobs parameter in vllm.SamplingParams to obtain the logits for each option from 1 to 9, and then performed intra-group sorting, which also enhanced the results.
\nThe complete inference code is as follows:
\nhttps://www.kaggle.com/code/zhudong1949/recall-reranker-logit-72b
TIPS: Synthetic data improved my recall score, but the overall score did not increase, so I ultimately did not submit the version with synthetic data. This should be due to the insufficient quality of my synthetic data.
", "comments": [ { "id": 3070808, "author_name": "Nikhil Tumbde", "author_url": "", "post_date": "2024-12-13T04:11:22.600000", "content": "Hey man, great solution. I had also finetuned the 32B model and observed significant boost in LB. However, I missed fine-tuning 72B model due to time constraints. Thanks for the explanation, I'll try out your method (using LLama factory) in the future.
\nI have one question for you, any particular reason for choosing GPTQ instead of AWQ?
I believe AWQ would have required you to dequantise your model before merging.
", "votes": 0, "replies": [ { "id": 3070843, "author_name": "zhudong1949", "author_url": "", "post_date": "2024-12-13T05:15:00.413000", "content": "I directly used the quantization script from llama-factory.
\nhttps://github.com/hiyouga/LLaMA-Factory/blob/main/examples/merge_lora/llama3_gptq.yaml
", "votes": 0, "replies": [] } ] } ], "comment_count": 2, "messages": [ { "id": 3070791, "postDate": "2024-12-13T03:42:55.960Z", "content": "Thank you to the excellent public notebook, from which I've learned a lot.
\nI made simple modifications to the public notebook and achieved 30th place.
\nReference: https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/543519
\nI fine-tuned qwen2.5-14b use [FlagEmbedding], using the unfine-tuned SFR-Embedding-2_R as the seed model, obtaining 100 negative sample instances, with epoch=20.
\nInference stage, I used load_in_8bit.
I actually did not run this part of the code.
\nReference:
\nI fine-tuned qwen-72B with accelerate+qlora [use LLaMA-Factory] , and then quantified it with GPTQ, to better adapt it to the 9-out-of-1 task. The fine-tuning brought about an improvement in performance.
\nUnlike the public notebook which uses logits_processor_zoo to get the candidate with the highest probability, I added the logprobs parameter in vllm.SamplingParams to obtain the logits for each option from 1 to 9, and then performed intra-group sorting, which also enhanced the results.
\nThe complete inference code is as follows:
\nhttps://www.kaggle.com/code/zhudong1949/recall-reranker-logit-72b
TIPS: Synthetic data improved my recall score, but the overall score did not increase, so I ultimately did not submit the version with synthetic data. This should be due to the insufficient quality of my synthetic data.
", "rawMarkdown": "Thank you to the excellent public notebook, from which I've learned a lot.\n\nI made simple modifications to the public notebook and achieved 30th place.\n\n## Recall stage:\nReference: https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/543519\n\nI fine-tuned qwen2.5-14b use [[FlagEmbedding]](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/finetune/embedder/decoder_only), using the unfine-tuned SFR-Embedding-2_R as the seed model, obtaining 100 negative sample instances, with epoch=20.\nInference stage, I used load_in_8bit.\n\n## Rerank stage:\nI actually did not run this part of the code.\n\n## logits_processor_zoo + logit_score_rerank\nReference: \n- https://www.kaggle.com/code/aerdem4/eedi-qwen32b-vllm-with-logits-processor-zoo\n- https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/550223\n\nI fine-tuned qwen-72B with accelerate+qlora [[use LLaMA-Factory]](https://github.com/hiyouga/LLaMA-Factory/blob/main/examples/extras/fsdp_qlora/train.sh) , and then quantified it with GPTQ, to better adapt it to the 9-out-of-1 task. The fine-tuning brought about an improvement in performance.\n\nUnlike the public notebook which uses logits_processor_zoo to get the candidate with the highest probability, I added the logprobs parameter in vllm.SamplingParams to obtain the logits for each option from 1 to 9, and then performed intra-group sorting, which also enhanced the results.\n\nThe complete inference code is as follows:\nhttps://www.kaggle.com/code/zhudong1949/recall-reranker-logit-72b\n\nTIPS: Synthetic data improved my recall score, but the overall score did not increase, so I ultimately did not submit the version with synthetic data. This should be due to the insufficient quality of my synthetic data.\n", "votes": 15 }, { "id": 3070808, "postDate": "2024-12-13T04:11:22.600Z", "content": "Hey man, great solution. I had also finetuned the 32B model and observed significant boost in LB. However, I missed fine-tuning 72B model due to time constraints. Thanks for the explanation, I'll try out your method (using LLama factory) in the future.
\nI have one question for you, any particular reason for choosing GPTQ instead of AWQ?
I believe AWQ would have required you to dequantise your model before merging.
", "rawMarkdown": "Hey man, great solution. I had also finetuned the 32B model and observed significant boost in LB. However, I missed fine-tuning 72B model due to time constraints. Thanks for the explanation, I'll try out your method (using LLama factory) in the future.\nI have one question for you, any particular reason for choosing GPTQ instead of AWQ?\n\nI believe AWQ would have required you to dequantise your model before merging.", "replies": [ { "id": 3070843, "postDate": "2024-12-13T05:15:00.413Z", "content": "I directly used the quantization script from llama-factory.
\nhttps://github.com/hiyouga/LLaMA-Factory/blob/main/examples/merge_lora/llama3_gptq.yaml
", "rawMarkdown": "I directly used the quantization script from llama-factory.\n\nhttps://github.com/hiyouga/LLaMA-Factory/blob/main/examples/merge_lora/llama3_gptq.yaml" } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T03:45:19Z", "competition_id": 82695, "competition_slug": "eedi-mining-misconceptions-in-mathematics", "team_id": 12741398, "solution_url": "/competitions/eedi-mining-misconceptions-in-mathematics/writeups/30th-place-solution-just-public-notebook-public-35" }