{ "id": 551651, "title": "2nd place solution", "content": "

Congratulations to all the winners! Thanks to the organizers for hosting such an interesting competition. We really enjoyed our journey and got a lot of inspiration from all the competitors! I am lucky enough to be a member of such a fruitful team and would like to thank all of my team members. Here we share our solution.

\n

Preprocess

\n

We found a useful subject_metadata.csv of which the SubjectId and SubjectName are Identical to this competition from the past eedi competition hosted on NeurlPS 2022. The subject_metadata.csv contains the parent subject, so we made a vector db with this metadata to add the parent subject information to both train.csv and test.csv.

\n

Synthetic data

\n

synthetic questions

\n

We generated synthetic data 3 times, let’s call them generation1, generation2, generation3.
\nThe base approach is:

\n\n

The difference between each generation is as follows.
\nGeneration1:

\n\n

Generation2:

\n\n

Generation3:

\n\n

misconception augmentation

\n

The misconception only contains a short sentence. In order to make the embedding of the misconception more meaningful, we used LLM to generate explanation for each misconception. Since we don’t need to run inference for misconception in submission, the approach costs nothing in submission.
\nThe prompt is as follows. The explanation of llama3.1-70b-Instruct and qwen2.5-72b-Instruct out-perform gpt-4o-mini in training retriever.

\n
system_prompt_template = \n\nuser_prompt_template = \n
\n

Chain of thought

\n

We used qwen2.5-32B-Instruct-AWQ to generate chain-of-thought as additional input for the following retrieve and rerank. The prompt is as follows:

\n
system_prompt_template = \n\nuser_prompt_template =   \n
\n

Retrieve

\n

Training the Retrieve Models

\n

We trained retriever with 2 different pipeline.
\nPipeline1:

\n\n

Pipeline2:

\n\n

Single model performance is as follows.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
retrieversynthetic dataPrivate LBPublic LBInference time
Linq-AI-Research/Linq-Embed-MistralGeneration1230.4610.48450 min
Qwen/Qwen2.5-14BGeneration120.4790.50745 min
Qwen/Qwen2.5-14BGeneration1230.4850.49245 min
Qwen/Qwen2.5-32BGeneration1230.4950.536140 min
Qwen/QwQ-32B-PreviewGeneration1230.5000.531140 min
\n

Our best submission used an ensemble of Mistral and 2 x qwen2.5-14B to give enough time to 72b rerank, the private LB and public LB is 0.513, 0.530.

\n

Key Factors for Retriever Improvements

\n
    \n

  1. Using Large Models
    \nAs is shown in the table above, the larger the better, GPU and Credit Card is all you need to get Power!!! Never open the billing page during the competition.

    \n
    2. \n

  2. Synthetic question
    \nI believe most of the participants used synthetic questions, the more high quality questions, the better performance. For our team, using gpt-4o-mini to filter high quality questions is the key.

    3. \n

  3. Misconception augmentation
    \nUsing misconception augmentation significantly boosted retriever performance by about 2-4%.

    4. \n

  4. Chain of Thought
    \nCoT is also useful. But for 14B and 32B models, adding CoT to the prompt will double the inference time.

    5. \n

  5. Pooling Selection
    \nWe found that last token pooling achieved better performance than mean pooling in the Qwen model.

    6. \n
\n

Rerank

\n

We used a listwise reranker to refine the ranking of retrieved candidates. Our reranking process employed a sliding window approach: first, we used a lightweight LLM to reorder candidates ranked between 8th and 17th. Then, we leveraged larger models to finalize the rankings for the top 10 candidates.

\n

The LLMs for reranking were fine-tuned on a combination of synthetic and training data.

\n\n

Key Factors for Reranking Improvements

\n
    \n

  1. Using Large Models
    \nWe found that larger models (e.g., 72B parameters) consistently delivered stronger validation scores compared to smaller ones like 14B or 32B models. However, these larger models initially performed worse on the Public LB, leading to some concerns. Despite this, we trusted the validation scores and included the 72B model in our final submissions (special thanks to the three-submission rule!). Ultimately, the 72B model produced outstanding Private Leaderboard scores, helping us secure a prize.

    2. \n

  2. Chain of Thought
    \nThe above CoT prompts greatly improved reranking performance.

    3. \n

  3. Sliding Window
    \nInstead of increasing the number of candidates for reranking, applying the sliding window strategy multiple times to refine the top-10 rankings proved to be more effective.
    \n\"\"

    4. \n

  4. Test-Time Augmentation
    \nDuring inference, we used TTA with some models by generating prompts in reverse order and averaging their scores with those from standard prompts. This technique provided a slight boost in accuracy.

    5. \n
\n

Training the Rerank Models

\n

We developed the QLoRA training code for our LLM rerankers based on the first-place solution from atmacup17. Special thanks to @kcotton21 for sharing such excellent solutions.

\n

Here are some tips that proved effective during the reranking model training:

\n
    \n

  1. Randomizing Listwise Choices
    \nInstead of always using the top-10 candidates for prompts, we created prompts with a variety of top-N combinations such as top-3, top-5, top-15, and top-25.

    2. \n

  2. Synthetic Data
    \nThe synthetic data used for training the retriever was also helpful for training the reranker. In total, we trained on 8,000 records of training data (2 epochs) and 14,000 synthetic records, resulting in a combined dataset of 22,000 records.

    3. \n

  3. Negative Sample Mining
    \nTo mine negative samples, we used a hybrid retriever combining the fine-tuned dunzhang/stella_en_400M_v5 model and TF-IDF. Each of the 22,000 positive samples had corresponding negative samples mined using this setup.

    4. \n
\n

Training Time

\n

Qwen2.5-14B: ~2 hours on H100
\nQwen2.5-72B: ~8 hours on H100

\n

Quantization

\n

We used the intel/auto-round library for quantizing the LLM rerankers. Compared to AutoGPTQ and AutoAWQ, this library was easier to use and caused minimal accuracy loss (typically less than 2%). Additionally, it could produce models compatible with vLLM.

\n

qwen2.5-72b-Instruct have some issues to run on multi GPU due to its intermediate_size(29568). Following the workaround provided by the document of gptq, we padded the weights to 29696 and then performed quantization.

\n

For calibration, we used the training dataset. Below are the quantization parameters:

\n
bits, group_size, sym = 4, 128,   \nautoround = AutoRound(    \n    model, tokenizer, =bits, =group_size, =sym, =calib_prompts, =256,    \n    =512,    \n    =500,    \n)  \n
\n

Inference

\n

We use vLLM for inference.

\n

By setting the enabling_prefix_cache to True, we were able to save approximately 10% of the inference time.

\n

jagatkiran shared his insights on performing inference with a 72B LLM model. In this competition, larger models tend to perform better, which has been very helpful for us.

\n

We implemented the reranker using logits_processors and logprobs by assigning a weight of +100 to specific tokens. This approach helped us establish the framework for the ranker. We also tried using a classification head directly, but the results were not satisfactory, and it was not easy to perform inference with vLLM. We believe this method could become a paradigm in future competitions.

\n

Ablation

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Baseline (retriever)Qwen14BQwen72BLlama70B8-17th Qwen14BPrivate LBPublic LB
✅0.5130.530
✅✅0.5680.583
✅✅0.5930.582
✅✅✅✅0.5960.609
✅✅✅✅✅0.6040.622
\n

On the final day, we tried fine-tuning Nexusflow/Athene-V2-Chat instead of Llama70B. Unfortunately, the submission with this model got timeout due to gpu and timeout issues, but it showed highly impressive performance on the leaderboard: 0.609.

\n

train code: https://github.com/wangqihanginthesky/Eedi_kaggle\\
\ninference code: https://www.kaggle.com/code/honglihang/2nd-place-inference-code

\n

Reference

\n

LLMにおける合成データセットによる数学推論タスクの精度向上の検討
\nNeurIPS 2022 CausalML Challenge: Causal Insights for Learning Paths in Education
\nA Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models
\nQwen2.5-72B & Llama 3.3-70B on 2xT4(questionable performance)
\nOptimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs
\n1st place solution from atmacup17
\nWinning Amazon KDD Cup'24
\nQwen GPTQ Troubleshooting

", "comments": [ { "id": 3072302, "author_name": "ktr", "author_url": "", "post_date": "2024-12-15T02:26:57.653000", "content": "

Congratulations on your prize gold! I am glad to hear that my past solutions were helpful.

\n

I can see from your post that you have made good use of 72b.
\nDid you train the 72b with QLoRA, merge the Adapter, and then quantize it? Empirically, I think that merge Adapter trained with QLoRA has a significant performance degradation.

", "votes": 3, "replies": [ { "id": 3072345, "author_name": "RihanPiggy", "author_url": "", "post_date": "2024-12-15T04:05:18.547000", "content": "

Thanks very much for your solution in atmacup and congratulations!
\nYes, we trained 72b with QLoRa, merge the Adapter and then perform quantization. In our offline experiment, the performance did not show much degradation(0.7011 -> 0.6988).
\n@yokuyama developed the training of rerank and I heard that he optimized the hyper-parameter during quantize. auto-round is very powerful and easy to use, better than auto-awq in our experiments.

", "votes": 4, "replies": [ { "id": 3072365, "author_name": "ktr", "author_url": "", "post_date": "2024-12-15T04:50:25.027000", "content": "

Thank you!
\nSo there isn't much accuracy degradation.
\nI've never quantized after merging the QLoRA adapter. I'll experiment with auto-round.

", "votes": 1, "replies": [] } ] } ] }, { "id": 3072270, "author_name": "kuto", "author_url": "", "post_date": "2024-12-15T01:04:23.557000", "content": "

Congratulations and thank you for sharing your solution! It was very interesting to read.

\n

I have two questions.

\n
    \n

  1. Are the prompt and output formats different between training and inference for the reranker? I understood it as predicting Yes/No for a single misconception during training, and rearranging multiple misconceptions during inference. Is that correct?

    2. \n

  2. I didn’t quite understand the part where the special token is assigned a weight of 100. Could you explain that in a bit more detail?

    3. \n
\n
\n

We implemented the reranker using logits_processors and logprobs by assigning a weight of +100 to specific tokens.

\n
", "votes": 1, "replies": [ { "id": 3072332, "author_name": "Qihang Wang", "author_url": "", "post_date": "2024-12-15T03:36:58.233000", "content": "

Thank you for your question.

\n

The training and inference processes are consistent, both involving options like A, B, C, D …

\n

Regarding the weight of the special token: for these options (A, B, C, D,…), we apply a logit_bias of +100 to ensure that the model prioritizes these letters as output. (set scores[self.allowed_ids] += 100 in logitprocessor's call)

", "votes": 3, "replies": [ { "id": 3072403, "author_name": "kuto", "author_url": "", "post_date": "2024-12-15T06:10:51.583000", "content": "

I see that we can control the weight with logitsprocessor. I did not know that, thank you!

", "votes": 0, "replies": [] } ] } ] }, { "id": 3072392, "author_name": "MrSimple", "author_url": "", "post_date": "2024-12-15T05:33:15.227000", "content": "

Congratulations on your prize gold! I am glad to hear that my past solutions were helpful.

", "votes": 0, "replies": [] }, { "id": 3072099, "author_name": "Yi-Chia Chen", "author_url": "", "post_date": "2024-12-14T17:26:21.777000", "content": "
\n

\"We found a useful subject_metadata.csv of which the SubjectId and SubjectName are Identical to this competition from the past eedi competition hosted on NeurlPS 2022. The subject_metadata.csv contains the parent subject, so we made a vector db with this metadata to add the parent subject information to both train.csv and test.csv.\"

\n
\n

That's cool! Have you analyzed how much improvement the parent subject information brings to the model's performance?

", "votes": 0, "replies": [ { "id": 3072350, "author_name": "RihanPiggy", "author_url": "", "post_date": "2024-12-15T04:10:21.173000", "content": "

Not sure enough because I collected external data at the start of the competition. Personally I think the gain is marginal because our 14B and 32B retriever did not use this information. However, the parent subject is used as input of Chain-of-thought, Rerank and synthetic data generation, which may give the model some additional context.

", "votes": 3, "replies": [] }, { "id": 3072353, "author_name": "RihanPiggy", "author_url": "", "post_date": "2024-12-15T04:16:05.690000", "content": "

By the way, I saw that you are an double degree exchange student in waseda. I am also an alumini of waseda's double degree program!!
\nI majored political science and economics in waseda with scholarship from BaiXian, are you in the same program?

", "votes": 0, "replies": [ { "id": 3072724, "author_name": "Yi-Chia Chen", "author_url": "", "post_date": "2024-12-15T15:36:32.737000", "content": "

Thank you for your response!
\nAnd it's great to meet you, senpai! But I'm actually in a different program - the double degree partnership between National Taiwan University and Waseda.
\nAlso, congratulations! You'll become a Grandmaster after this competition, right? ✨

", "votes": 0, "replies": [] }, { "id": 3072764, "author_name": "RihanPiggy", "author_url": "", "post_date": "2024-12-15T16:12:29.227000", "content": "

Oh, really! There were students from various countries(including Taiwan) in my program, so I thought maybe you are in the same one as mine.

\n

Congratulations to you too. It's amazing to see the magic of multiplying the unseen logits, which made our team feel stomachache in the final days…
\nYes! waiting for the finalize and become Grandmaster😆 And you will become a Master with solo gold!

", "votes": 0, "replies": [] } ] } ] } ], "comment_count": 12, "messages": [ { "id": 3071931, "postDate": "2024-12-14T14:29:57.250Z", "content": "

Congratulations to all the winners! Thanks to the organizers for hosting such an interesting competition. We really enjoyed our journey and got a lot of inspiration from all the competitors! I am lucky enough to be a member of such a fruitful team and would like to thank all of my team members. Here we share our solution.

\n

Preprocess

\n

We found a useful subject_metadata.csv of which the SubjectId and SubjectName are Identical to this competition from the past eedi competition hosted on NeurlPS 2022. The subject_metadata.csv contains the parent subject, so we made a vector db with this metadata to add the parent subject information to both train.csv and test.csv.

\n

Synthetic data

\n

synthetic questions

\n

We generated synthetic data 3 times, let’s call them generation1, generation2, generation3.
\nThe base approach is:

\n\n

The difference between each generation is as follows.
\nGeneration1:

\n\n

Generation2:

\n\n

Generation3:

\n\n

misconception augmentation

\n

The misconception only contains a short sentence. In order to make the embedding of the misconception more meaningful, we used LLM to generate explanation for each misconception. Since we don’t need to run inference for misconception in submission, the approach costs nothing in submission.
\nThe prompt is as follows. The explanation of llama3.1-70b-Instruct and qwen2.5-72b-Instruct out-perform gpt-4o-mini in training retriever.

\n
system_prompt_template = \n\nuser_prompt_template = \n
\n

Chain of thought

\n

We used qwen2.5-32B-Instruct-AWQ to generate chain-of-thought as additional input for the following retrieve and rerank. The prompt is as follows:

\n
system_prompt_template = \n\nuser_prompt_template =   \n
\n

Retrieve

\n

Training the Retrieve Models

\n

We trained retriever with 2 different pipeline.
\nPipeline1:

\n\n

Pipeline2:

\n\n

Single model performance is as follows.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
retrieversynthetic dataPrivate LBPublic LBInference time
Linq-AI-Research/Linq-Embed-MistralGeneration1230.4610.48450 min
Qwen/Qwen2.5-14BGeneration120.4790.50745 min
Qwen/Qwen2.5-14BGeneration1230.4850.49245 min
Qwen/Qwen2.5-32BGeneration1230.4950.536140 min
Qwen/QwQ-32B-PreviewGeneration1230.5000.531140 min
\n

Our best submission used an ensemble of Mistral and 2 x qwen2.5-14B to give enough time to 72b rerank, the private LB and public LB is 0.513, 0.530.

\n

Key Factors for Retriever Improvements

\n
    \n

  1. Using Large Models
    \nAs is shown in the table above, the larger the better, GPU and Credit Card is all you need to get Power!!! Never open the billing page during the competition.

    \n
    2. \n

  2. Synthetic question
    \nI believe most of the participants used synthetic questions, the more high quality questions, the better performance. For our team, using gpt-4o-mini to filter high quality questions is the key.

    3. \n

  3. Misconception augmentation
    \nUsing misconception augmentation significantly boosted retriever performance by about 2-4%.

    4. \n

  4. Chain of Thought
    \nCoT is also useful. But for 14B and 32B models, adding CoT to the prompt will double the inference time.

    5. \n

  5. Pooling Selection
    \nWe found that last token pooling achieved better performance than mean pooling in the Qwen model.

    6. \n
\n

Rerank

\n

We used a listwise reranker to refine the ranking of retrieved candidates. Our reranking process employed a sliding window approach: first, we used a lightweight LLM to reorder candidates ranked between 8th and 17th. Then, we leveraged larger models to finalize the rankings for the top 10 candidates.

\n

The LLMs for reranking were fine-tuned on a combination of synthetic and training data.

\n\n

Key Factors for Reranking Improvements

\n
    \n

  1. Using Large Models
    \nWe found that larger models (e.g., 72B parameters) consistently delivered stronger validation scores compared to smaller ones like 14B or 32B models. However, these larger models initially performed worse on the Public LB, leading to some concerns. Despite this, we trusted the validation scores and included the 72B model in our final submissions (special thanks to the three-submission rule!). Ultimately, the 72B model produced outstanding Private Leaderboard scores, helping us secure a prize.

    2. \n

  2. Chain of Thought
    \nThe above CoT prompts greatly improved reranking performance.

    3. \n

  3. Sliding Window
    \nInstead of increasing the number of candidates for reranking, applying the sliding window strategy multiple times to refine the top-10 rankings proved to be more effective.
    \n\"\"

    4. \n

  4. Test-Time Augmentation
    \nDuring inference, we used TTA with some models by generating prompts in reverse order and averaging their scores with those from standard prompts. This technique provided a slight boost in accuracy.

    5. \n
\n

Training the Rerank Models

\n

We developed the QLoRA training code for our LLM rerankers based on the first-place solution from atmacup17. Special thanks to @kcotton21 for sharing such excellent solutions.

\n

Here are some tips that proved effective during the reranking model training:

\n
    \n

  1. Randomizing Listwise Choices
    \nInstead of always using the top-10 candidates for prompts, we created prompts with a variety of top-N combinations such as top-3, top-5, top-15, and top-25.

    2. \n

  2. Synthetic Data
    \nThe synthetic data used for training the retriever was also helpful for training the reranker. In total, we trained on 8,000 records of training data (2 epochs) and 14,000 synthetic records, resulting in a combined dataset of 22,000 records.

    3. \n

  3. Negative Sample Mining
    \nTo mine negative samples, we used a hybrid retriever combining the fine-tuned dunzhang/stella_en_400M_v5 model and TF-IDF. Each of the 22,000 positive samples had corresponding negative samples mined using this setup.

    4. \n
\n

Training Time

\n

Qwen2.5-14B: ~2 hours on H100
\nQwen2.5-72B: ~8 hours on H100

\n

Quantization

\n

We used the intel/auto-round library for quantizing the LLM rerankers. Compared to AutoGPTQ and AutoAWQ, this library was easier to use and caused minimal accuracy loss (typically less than 2%). Additionally, it could produce models compatible with vLLM.

\n

qwen2.5-72b-Instruct have some issues to run on multi GPU due to its intermediate_size(29568). Following the workaround provided by the document of gptq, we padded the weights to 29696 and then performed quantization.

\n

For calibration, we used the training dataset. Below are the quantization parameters:

\n
bits, group_size, sym = 4, 128,   \nautoround = AutoRound(    \n    model, tokenizer, =bits, =group_size, =sym, =calib_prompts, =256,    \n    =512,    \n    =500,    \n)  \n
\n

Inference

\n

We use vLLM for inference.

\n

By setting the enabling_prefix_cache to True, we were able to save approximately 10% of the inference time.

\n

jagatkiran shared his insights on performing inference with a 72B LLM model. In this competition, larger models tend to perform better, which has been very helpful for us.

\n

We implemented the reranker using logits_processors and logprobs by assigning a weight of +100 to specific tokens. This approach helped us establish the framework for the ranker. We also tried using a classification head directly, but the results were not satisfactory, and it was not easy to perform inference with vLLM. We believe this method could become a paradigm in future competitions.

\n

Ablation

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Baseline (retriever)Qwen14BQwen72BLlama70B8-17th Qwen14BPrivate LBPublic LB
✅0.5130.530
✅✅0.5680.583
✅✅0.5930.582
✅✅✅✅0.5960.609
✅✅✅✅✅0.6040.622
\n

On the final day, we tried fine-tuning Nexusflow/Athene-V2-Chat instead of Llama70B. Unfortunately, the submission with this model got timeout due to gpu and timeout issues, but it showed highly impressive performance on the leaderboard: 0.609.

\n

train code: https://github.com/wangqihanginthesky/Eedi_kaggle\\
\ninference code: https://www.kaggle.com/code/honglihang/2nd-place-inference-code

\n

Reference

\n

LLMにおける合成データセットによる数学推論タスクの精度向上の検討
\nNeurIPS 2022 CausalML Challenge: Causal Insights for Learning Paths in Education
\nA Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models
\nQwen2.5-72B & Llama 3.3-70B on 2xT4(questionable performance)
\nOptimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs
\n1st place solution from atmacup17
\nWinning Amazon KDD Cup'24
\nQwen GPTQ Troubleshooting

", "rawMarkdown": "Congratulations to all the winners! Thanks to the organizers for hosting such an interesting competition. We really enjoyed our journey and got a lot of inspiration from all the competitors! I am lucky enough to be a member of such a fruitful team and would like to thank all of my team members. Here we share our solution.\n\n# Preprocess\n\nWe found a useful subject_metadata.csv of which the SubjectId and SubjectName are Identical to this competition from the past eedi competition hosted on [NeurlPS 2022](https://codalab.lisn.upsaclay.fr/competitions/5626#learn_the_details-get_starting_kit). The subject_metadata.csv contains the parent subject, so we made a vector db with this metadata to add the parent subject information to both train.csv and test.csv. \n\n# Synthetic data\n\n### synthetic questions\n\nWe generated synthetic data 3 times, let’s call them generation1, generation2, generation3. \nThe base approach is:\n\n- provide LLM a misconception with few examples and let it generate questions \n- use qwen-math to solve the question to get the correct answer \n- use qwen-math to solve the question under the constraint of misconception to get the wrong answer \n- use gpt-4o-mini to score the quality of the question and choose those with score larger than 2 (max is 5)\n\nThe difference between each generation is as follows. \nGeneration1:\n\n- few shot examples: randomly sampled from train.csv\n\nGeneration2:\n\n- few shot examples: sample the question with the same misconception from train.csv and Generation1\n\nGeneration3:\n\n- The prompt of generation 3 is based on the prompt of [this tech blog](https://tech.preferred.jp/ja/blog/llm-synthetic-dataset-for-math/) \n- few shot examples: randomly sample 2 questions from train.csv, Generation1 and Generation2\n\n### misconception augmentation\n\nThe misconception only contains a short sentence. In order to make the embedding of the misconception more meaningful, we used LLM to generate explanation for each misconception. Since we don’t need to run inference for misconception in submission, the approach costs nothing in submission. \nThe prompt is as follows. The explanation of llama3.1-70b-Instruct and qwen2.5-72b-Instruct out-perform gpt-4o-mini in training retriever. \n``` \nsystem_prompt_template = 'You are an excellent math teacher about to teach students of year group 1 to 14. The subject of your lesson includes Number, Algebra, Data and Statistics, Geometry and Measure. You will be provided a misconception that your students may have. Please explain the misconception in detail and provide some short cases when the misconception will occur. No need to provide the correct approach. The explanation should be in format of \"Explanation: {explanation}\"'\n\nuser_prompt_template = 'Misconception: {misconception}' \n```\n\n# Chain of thought\n\nWe used qwen2.5-32B-Instruct-AWQ to generate chain-of-thought as additional input for the following retrieve and rerank. The prompt is as follows: \n``` \nsystem_prompt_template = \"You are an excellent math teacher about to teach students of year group 1 to 14. The detail of your lesson is as follows. Subject:{first_subject}, Topic: {second_subject}, Subtopic {third_subject}. Your students have made a mistake in the following question. Please explain the mistake step by step briefly and describe the misunderstanding behind the wrong answer at conceptual level. No need to provide the correct way to achieve the answer.\"\n\nuser_prompt_template = \"Question: {question_text}\\nCorrect Answer: {correct_text}\\nWrong Answer of your students: {answer_text}\\n\\nExplanation: \\nMisunderstanding: \" \n```\n\n# Retrieve\n### Training the Retrieve Models \nWe trained retriever with 2 different pipeline. \nPipeline1:\n\n- backbone: Linq-AI-Research/Linq-Embed-Mistral \n- loss: Arcface \n- use Chain of thought as additional input in training and inference\n\nPipeline2:\n\n- backbone: Qwen/Qwen2.5-14B, Qwen/Qwen2.5-32B, Qwen/QwQ-32B-Preview \n- loss: MultipleNegativesRankingLoss \n- w/o Chain of thought (due to inference time)\n\nSingle model performance is as follows. \n|retriever|synthetic data|Private LB|Public LB|Inference time| \n|:----|:----|:----|:----|:----| \n|Linq-AI-Research/Linq-Embed-Mistral|Generation123|0.461|0.484|50 min| \n|Qwen/Qwen2.5-14B|Generation12|0.479|0.507|45 min| \n|Qwen/Qwen2.5-14B|Generation123|0.485|0.492|45 min| \n|Qwen/Qwen2.5-32B|Generation123|0.495|0.536|140 min| \n|Qwen/QwQ-32B-Preview|Generation123|0.500|0.531|140 min| \n\nOur best submission used an ensemble of Mistral and 2 x qwen2.5-14B to give enough time to 72b rerank, the private LB and public LB is 0.513, 0.530.\n\n### Key Factors for Retriever Improvements\n\n1. **Using Large Models** \n As is shown in the table above, the larger the better, GPU and Credit Card is all you need to get Power!!! Never open the billing page during the competition.\n\n - qwen2.5-14B with Generation12: H100 about 2days \n - qwen2.5-14B with Generation123: H100 about 3days \n - qwen2.5-32B with Generation123(sampled): H100 about 5days \n - QWQ with Generation123(sampled): H100 about 5days\n\n2. **Synthetic question** \n I believe most of the participants used synthetic questions, the more high quality questions, the better performance. For our team, using gpt-4o-mini to filter high quality questions is the key.\n\n3. **Misconception augmentation** \n Using misconception augmentation significantly boosted retriever performance by about 2-4%.\n\n4. **Chain of Thought** \n CoT is also useful. But for 14B and 32B models, adding CoT to the prompt will double the inference time.\n\n5. **Pooling Selection** \n We found that last token pooling achieved better performance than mean pooling in the Qwen model.\n\n# Rerank\n\nWe used a listwise reranker to refine the ranking of retrieved candidates. Our reranking process employed a sliding window approach: first, we used a lightweight LLM to reorder candidates ranked between 8th and 17th. Then, we leveraged larger models to finalize the rankings for the top 10 candidates.\n\nThe LLMs for reranking were fine-tuned on a combination of synthetic and training data.\n\n- **Window 1 (8th ~ 17th)** \n - Qwen2.5-14B-Instruct \n- **Window 2 (1st ~ 10th)** \n - Qwen2.5-72B-Instruct \n - Llama-3.3-70B-Instruct \n\n### Key Factors for Reranking Improvements\n\n1. **Using Large Models** \n We found that larger models (e.g., 72B parameters) consistently delivered stronger validation scores compared to smaller ones like 14B or 32B models. However, these larger models initially performed worse on the Public LB, leading to some concerns. Despite this, we trusted the validation scores and included the 72B model in our final submissions (special thanks to the three-submission rule!). Ultimately, the 72B model produced outstanding Private Leaderboard scores, helping us secure a prize.\n\n2. **Chain of Thought** \n The above CoT prompts greatly improved reranking performance.\n\n3. **Sliding Window** \n Instead of increasing the number of candidates for reranking, applying the sliding window strategy multiple times to refine the top-10 rankings proved to be more effective. \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5688805%2F781fda9fa53780767ec312a7e19c284c%2Fimage1.png?generation=1734186073680419&alt=media)\n\n4. **Test-Time Augmentation** \n During inference, we used TTA with some models by generating prompts in reverse order and averaging their scores with those from standard prompts. This technique provided a slight boost in accuracy.\n\n### Training the Rerank Models \nWe developed the QLoRA training code for our LLM rerankers based on the first-place solution from [atmacup17](https://www.guruguru.science/competitions/24/discussions/21027ff1-2074-4e21-a249-b2d4170bd516/). Special thanks to [@kcotton21](https://www.kaggle.com/kcotton21) for sharing such excellent solutions.\n\nHere are some tips that proved effective during the reranking model training: \n\n1. **Randomizing Listwise Choices** \n Instead of always using the top-10 candidates for prompts, we created prompts with a variety of top-N combinations such as top-3, top-5, top-15, and top-25. \n\n2. **Synthetic Data** \n The synthetic data used for training the retriever was also helpful for training the reranker. In total, we trained on 8,000 records of training data (2 epochs) and 14,000 synthetic records, resulting in a combined dataset of 22,000 records. \n\n3. **Negative Sample Mining**\n To mine negative samples, we used a hybrid retriever combining the fine-tuned `dunzhang/stella_en_400M_v5` model and TF-IDF. Each of the 22,000 positive samples had corresponding negative samples mined using this setup. \n\n#### Training Time \nQwen2.5-14B: ~2 hours on H100 \nQwen2.5-72B: ~8 hours on H100 \n\n### Quantization \nWe used the [intel/auto-round](https://github.com/intel/auto-round) library for quantizing the LLM rerankers. Compared to AutoGPTQ and AutoAWQ, this library was easier to use and caused minimal accuracy loss (typically less than 2%). Additionally, it could produce models compatible with vLLM.\n\nqwen2.5-72b-Instruct have some issues to run on multi GPU due to its intermediate_size(29568). Following the workaround provided by the [document of gptq](https://qwen.readthedocs.io/en/latest/quantization/gptq.html#troubleshooting), we padded the weights to 29696 and then performed quantization.\n\nFor calibration, we used the training dataset. Below are the quantization parameters: \n\n```\nbits, group_size, sym = 4, 128, True \nautoround = AutoRound( \n model, tokenizer, bits=bits, group_size=group_size, sym=sym, dataset=calib_prompts, seqlen=256, \n nsamples=512, \n iters=500, \n) \n```\n\n### Inference \nWe use vLLM for inference.\n\nBy setting the `enabling_prefix_cache` to `True`, we were able to save approximately 10% of the inference time.\n\n[jagatkiran](https://www.kaggle.com/jagatkiran) shared his insights on performing [inference with a 72B LLM model](https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/550223). In this competition, larger models tend to perform better, which has been very helpful for us.\n\nWe implemented the reranker using `logits_processors` and `logprobs` by assigning a weight of +100 to specific tokens. This approach helped us establish the framework for the ranker. We also tried using a classification head directly, but the results were not satisfactory, and it was not easy to perform inference with vLLM. We believe this method could become a paradigm in future competitions.\n\n### Ablation \n|Baseline (retriever)|Qwen14B|Qwen72B|Llama70B|8-17th Qwen14B|Private LB|Public LB| \n|:----|:----|:----|:----|:----|:----|:----| \n|✅| | | | |0.513|0.530| \n|✅|✅| | | |0.568|0.583| \n|✅| |✅| | |0.593|0.582| \n|✅|✅|✅|✅| |0.596|0.609| \n|✅|✅|✅|✅|✅|**0.604**|0.622|\n\nOn the final day, we tried fine-tuning Nexusflow/Athene-V2-Chat instead of Llama70B. Unfortunately, the submission with this model got timeout due to gpu and timeout issues, but it showed highly impressive performance on the leaderboard: 0.609.\n\ntrain code: https://github.com/wangqihanginthesky/Eedi_kaggle\\\ninference code: https://www.kaggle.com/code/honglihang/2nd-place-inference-code\n\n# Reference \n[LLMにおける合成データセットによる数学推論タスクの精度向上の検討](https://tech.preferred.jp/ja/blog/llm-synthetic-dataset-for-math/) \n[NeurIPS 2022 CausalML Challenge: Causal Insights for Learning Paths in Education](https://codalab.lisn.upsaclay.fr/competitions/5626#learn_the_details-overview) \n[A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models](https://arxiv.org/pdf/2310.09497) \n[Qwen2.5-72B & Llama 3.3-70B on 2xT4(questionable performance)](https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/550223) \n[Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs](https://arxiv.org/abs/2309.05516) \n[1st place solution from atmacup17](https://www.guruguru.science/competitions/24/discussions/21027ff1-2074-4e21-a249-b2d4170bd516/) \n[Winning Amazon KDD Cup'24](https://openreview.net/forum?id=sv0E1mBhu8) \n[Qwen GPTQ Troubleshooting](https://qwen.readthedocs.io/en/latest/quantization/gptq.html#troubleshooting)", "votes": 38 }, { "id": 3072302, "postDate": "2024-12-15T02:26:57.653Z", "content": "

Congratulations on your prize gold! I am glad to hear that my past solutions were helpful.

\n

I can see from your post that you have made good use of 72b.
\nDid you train the 72b with QLoRA, merge the Adapter, and then quantize it? Empirically, I think that merge Adapter trained with QLoRA has a significant performance degradation.

", "rawMarkdown": "Congratulations on your prize gold! I am glad to hear that my past solutions were helpful.\n\nI can see from your post that you have made good use of 72b.\nDid you train the 72b with QLoRA, merge the Adapter, and then quantize it? Empirically, I think that merge Adapter trained with QLoRA has a significant performance degradation.", "votes": 3, "replies": [ { "id": 3072345, "postDate": "2024-12-15T04:05:18.547Z", "content": "

Thanks very much for your solution in atmacup and congratulations!
\nYes, we trained 72b with QLoRa, merge the Adapter and then perform quantization. In our offline experiment, the performance did not show much degradation(0.7011 -> 0.6988).
\n@yokuyama developed the training of rerank and I heard that he optimized the hyper-parameter during quantize. auto-round is very powerful and easy to use, better than auto-awq in our experiments.

", "rawMarkdown": "Thanks very much for your solution in atmacup and congratulations!\nYes, we trained 72b with QLoRa, merge the Adapter and then perform quantization. In our offline experiment, the performance did not show much degradation(0.7011 -> 0.6988).\n@yokuyama developed the training of rerank and I heard that he optimized the hyper-parameter during quantize. auto-round is very powerful and easy to use, better than auto-awq in our experiments.", "votes": 4, "replies": [ { "id": 3072365, "postDate": "2024-12-15T04:50:25.027Z", "content": "

Thank you!
\nSo there isn't much accuracy degradation.
\nI've never quantized after merging the QLoRA adapter. I'll experiment with auto-round.

", "rawMarkdown": "Thank you!\nSo there isn't much accuracy degradation.\nI've never quantized after merging the QLoRA adapter. I'll experiment with auto-round.", "votes": 1 } ] } ] }, { "id": 3072270, "postDate": "2024-12-15T01:04:23.557Z", "content": "

Congratulations and thank you for sharing your solution! It was very interesting to read.

\n

I have two questions.

\n
    \n

  1. Are the prompt and output formats different between training and inference for the reranker? I understood it as predicting Yes/No for a single misconception during training, and rearranging multiple misconceptions during inference. Is that correct?

    2. \n

  2. I didn’t quite understand the part where the special token is assigned a weight of 100. Could you explain that in a bit more detail?

    3. \n
\n
\n

We implemented the reranker using logits_processors and logprobs by assigning a weight of +100 to specific tokens.

\n
", "rawMarkdown": "Congratulations and thank you for sharing your solution! It was very interesting to read.\n\nI have two questions.\n\n1. Are the prompt and output formats different between training and inference for the reranker? I understood it as predicting Yes/No for a single misconception during training, and rearranging multiple misconceptions during inference. Is that correct?\n\n2. I didn’t quite understand the part where the special token is assigned a weight of 100. Could you explain that in a bit more detail?\n\n> We implemented the reranker using logits_processors and logprobs by assigning a weight of +100 to specific tokens.\n", "votes": 1, "replies": [ { "id": 3072332, "postDate": "2024-12-15T03:36:58.233Z", "content": "

Thank you for your question.

\n

The training and inference processes are consistent, both involving options like A, B, C, D …

\n

Regarding the weight of the special token: for these options (A, B, C, D,…), we apply a logit_bias of +100 to ensure that the model prioritizes these letters as output. (set scores[self.allowed_ids] += 100 in logitprocessor's call)

", "rawMarkdown": "Thank you for your question.\n\nThe training and inference processes are consistent, both involving options like A, B, C, D ...\n\nRegarding the weight of the special token: for these options (A, B, C, D,...), we apply a logit_bias of +100 to ensure that the model prioritizes these letters as output. (set `scores[self.allowed_ids] += 100` in logitprocessor's __call__)\n", "votes": 3, "replies": [ { "id": 3072403, "postDate": "2024-12-15T06:10:51.583Z", "content": "

I see that we can control the weight with logitsprocessor. I did not know that, thank you!

", "rawMarkdown": "I see that we can control the weight with logitsprocessor. I did not know that, thank you!" } ] } ] }, { "id": 3072392, "postDate": "2024-12-15T05:33:15.227Z", "content": "

Congratulations on your prize gold! I am glad to hear that my past solutions were helpful.

", "rawMarkdown": "Congratulations on your prize gold! I am glad to hear that my past solutions were helpful.\n\n" }, { "id": 3072099, "postDate": "2024-12-14T17:26:21.777Z", "content": "
\n

\"We found a useful subject_metadata.csv of which the SubjectId and SubjectName are Identical to this competition from the past eedi competition hosted on NeurlPS 2022. The subject_metadata.csv contains the parent subject, so we made a vector db with this metadata to add the parent subject information to both train.csv and test.csv.\"

\n
\n

That's cool! Have you analyzed how much improvement the parent subject information brings to the model's performance?

", "rawMarkdown": "> \"We found a useful subject_metadata.csv of which the SubjectId and SubjectName are Identical to this competition from the past eedi competition hosted on NeurlPS 2022. The subject_metadata.csv contains the parent subject, so we made a vector db with this metadata to add the parent subject information to both train.csv and test.csv.\"\n\nThat's cool! Have you analyzed how much improvement the parent subject information brings to the model's performance?", "replies": [ { "id": 3072350, "postDate": "2024-12-15T04:10:21.173Z", "content": "

Not sure enough because I collected external data at the start of the competition. Personally I think the gain is marginal because our 14B and 32B retriever did not use this information. However, the parent subject is used as input of Chain-of-thought, Rerank and synthetic data generation, which may give the model some additional context.

", "rawMarkdown": "Not sure enough because I collected external data at the start of the competition. Personally I think the gain is marginal because our 14B and 32B retriever did not use this information. However, the parent subject is used as input of Chain-of-thought, Rerank and synthetic data generation, which may give the model some additional context.", "votes": 3 }, { "id": 3072353, "postDate": "2024-12-15T04:16:05.690Z", "content": "

By the way, I saw that you are an double degree exchange student in waseda. I am also an alumini of waseda's double degree program!!
\nI majored political science and economics in waseda with scholarship from BaiXian, are you in the same program?

", "rawMarkdown": "By the way, I saw that you are an double degree exchange student in waseda. I am also an alumini of waseda's double degree program!!\nI majored political science and economics in waseda with scholarship from BaiXian, are you in the same program?", "replies": [ { "id": 3072724, "postDate": "2024-12-15T15:36:32.737Z", "content": "

Thank you for your response!
\nAnd it's great to meet you, senpai! But I'm actually in a different program - the double degree partnership between National Taiwan University and Waseda.
\nAlso, congratulations! You'll become a Grandmaster after this competition, right? ✨

", "rawMarkdown": "Thank you for your response!\nAnd it's great to meet you, senpai! But I'm actually in a different program - the double degree partnership between National Taiwan University and Waseda.\nAlso, congratulations! You'll become a Grandmaster after this competition, right? ✨" }, { "id": 3072764, "postDate": "2024-12-15T16:12:29.227Z", "content": "

Oh, really! There were students from various countries(including Taiwan) in my program, so I thought maybe you are in the same one as mine.

\n

Congratulations to you too. It's amazing to see the magic of multiplying the unseen logits, which made our team feel stomachache in the final days…
\nYes! waiting for the finalize and become Grandmaster😆 And you will become a Master with solo gold!

", "rawMarkdown": "Oh, really! There were students from various countries(including Taiwan) in my program, so I thought maybe you are in the same one as mine.\n\nCongratulations to you too. It's amazing to see the magic of multiplying the unseen logits, which made our team feel stomachache in the final days...\nYes! waiting for the finalize and become Grandmaster😆 And you will become a Master with solo gold!" } ] } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T03:47:24Z", "competition_id": 82695, "competition_slug": "eedi-mining-misconceptions-in-mathematics", "team_id": 12624966, "solution_url": "/competitions/eedi-mining-misconceptions-in-mathematics/writeups/cqyr-2nd-place-solution" }