Hopefully it's not too late to post my solutionš
\nI am so excited that I got my first silver medal.
\nSince last year, I have started participating in kaggle competitions in my spare time of work. As a beginner, sometimes I feel it really hard to construct a runnable pipeline for submission, but thanks to the kaggle community users for their selfless sharing, I learned a lot of tricks and techniques, and many useful insights to solve the given task. Also, I would like to thank Kaggle and EEDI for hosting such an interesting competition.
Like many other players, my solution is a 2-stage approach:
\nThe retriever is a Qwen-14b-awq from https://www.kaggle.com/code/anhvth226/eedi-11-21-14b. For each query, select the top 25 likely misconceptions.
\nGiven a question, a correct answer, a wrong answer and a list of possible misconceptions (9 candidates each query) from the previous stage, let Qwen2.5-32b-awq predict the most likely misconception index.
\nPlease refer to the hot public notebook https://www.kaggle.com/code/jagatkiran/qwen14b-retrieval-qwen32b-logits-processor-zoo.
\nAnd my infer notebook: https://www.kaggle.com/code/doublezeta/notebook0d961c3b94
I finetuned (lora) a Qwen2.5-32b-awq reranker via trl.SFTTrainer. Here are my hyper parameters
\npeft_config = LoraConfig(\n =,\n =,\n =64,\n =16,\n =0.05,\n target_modules=[, , , , , , ],\n =\n)\ntraining_args = SFTConfig(\n =,\n =tokenizer.model_max_length,\n =1e-4,\n =4,\n =1,\n =0.001,\n =3,\n =,\n =,\n =,\n =0.03,\n)\nMy demo training notebook: https://www.kaggle.com/code/doublezeta/train-reranker Actually I did the training in Colab.
\nI construct my SFT training dataset by
\nThe competition overview mentions \"The goal is to create a model that not only aligns with known misconceptions but also generalizes to new, emerging misconceptions\".
\nThere are ~900 misconceptions not appeared in the train set. I suppose the private LB would focus more on the unseen misconceptions.
\nTherefore I slightly adjust my strategy for constructing the SFT training set. Among the 8 wrong candidates of each sample, 4 are from the retrieval result, and 4 are from the 900 unseen misconceptions.
\nI believe increasing the ratio of the unseen misconceptions can help the LLM enhance the ability of reasoning upon these misconceptions.
It indeed brought me sort of improvement on the private board.
\n| \n | Public Score | \nPrivate Score | \n
|---|---|---|
| with Augmentation | \n0.539 | \n0.531 | \n
| without Augmentation | \n0.554 | \n0.514 | \n
Notebook to build SFT training set : https://www.kaggle.com/code/doublezeta/train-data-sft
\nI found similar insight in the 3rd place solution: https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/551498, maybe I should push it furtherš„
\nI noticed that the reranker just select the most likely misconception without changing the order of the rest candidates.
\nInspired by
\nhttps://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/543519
\nhttps://openreview.net/pdf?id=vhLAb1dpIw,
\nI used LLM as a pointwise scorer by getting the Yes token's logprob.
\nI finetuned (lora) a Qwen2.5-32b-awq. During inference, I prompt it to predict \"Yes\" or \"No\" for given a question, a correct answer, an incorrect answer, and a possible misconception.
\nBut the Public LB is poor (0.3+), I guess it's because
enable_prefix_caching=TrueGreat job!I
", "votes": 0, "replies": [] }, { "id": 3081797, "author_name": "marble", "author_url": "", "post_date": "2024-12-27T08:01:29.027000", "content": "Great job!I wonder what gpu do you use for trainning the rerank model.
", "votes": 0, "replies": [ { "id": 3081808, "author_name": "Double Zeta", "author_url": "", "post_date": "2024-12-27T08:44:05.590000", "content": "~7 hours on A100
", "votes": 0, "replies": [] } ] } ], "comment_count": 3, "messages": [ { "id": 3080447, "postDate": "2024-12-25T07:47:02.377Z", "content": "Hopefully it's not too late to post my solutionš
\nI am so excited that I got my first silver medal.
\nSince last year, I have started participating in kaggle competitions in my spare time of work. As a beginner, sometimes I feel it really hard to construct a runnable pipeline for submission, but thanks to the kaggle community users for their selfless sharing, I learned a lot of tricks and techniques, and many useful insights to solve the given task. Also, I would like to thank Kaggle and EEDI for hosting such an interesting competition.
Like many other players, my solution is a 2-stage approach:
\nThe retriever is a Qwen-14b-awq from https://www.kaggle.com/code/anhvth226/eedi-11-21-14b. For each query, select the top 25 likely misconceptions.
\nGiven a question, a correct answer, a wrong answer and a list of possible misconceptions (9 candidates each query) from the previous stage, let Qwen2.5-32b-awq predict the most likely misconception index.
\nPlease refer to the hot public notebook https://www.kaggle.com/code/jagatkiran/qwen14b-retrieval-qwen32b-logits-processor-zoo.
\nAnd my infer notebook: https://www.kaggle.com/code/doublezeta/notebook0d961c3b94
I finetuned (lora) a Qwen2.5-32b-awq reranker via trl.SFTTrainer. Here are my hyper parameters
\npeft_config = LoraConfig(\n =,\n =,\n =64,\n =16,\n =0.05,\n target_modules=[, , , , , , ],\n =\n)\ntraining_args = SFTConfig(\n =,\n =tokenizer.model_max_length,\n =1e-4,\n =4,\n =1,\n =0.001,\n =3,\n =,\n =,\n =,\n =0.03,\n)\nMy demo training notebook: https://www.kaggle.com/code/doublezeta/train-reranker Actually I did the training in Colab.
\nI construct my SFT training dataset by
\nThe competition overview mentions \"The goal is to create a model that not only aligns with known misconceptions but also generalizes to new, emerging misconceptions\".
\nThere are ~900 misconceptions not appeared in the train set. I suppose the private LB would focus more on the unseen misconceptions.
\nTherefore I slightly adjust my strategy for constructing the SFT training set. Among the 8 wrong candidates of each sample, 4 are from the retrieval result, and 4 are from the 900 unseen misconceptions.
\nI believe increasing the ratio of the unseen misconceptions can help the LLM enhance the ability of reasoning upon these misconceptions.
It indeed brought me sort of improvement on the private board.
\n| \n | Public Score | \nPrivate Score | \n
|---|---|---|
| with Augmentation | \n0.539 | \n0.531 | \n
| without Augmentation | \n0.554 | \n0.514 | \n
Notebook to build SFT training set : https://www.kaggle.com/code/doublezeta/train-data-sft
\nI found similar insight in the 3rd place solution: https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/551498, maybe I should push it furtherš„
\nI noticed that the reranker just select the most likely misconception without changing the order of the rest candidates.
\nInspired by
\nhttps://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/543519
\nhttps://openreview.net/pdf?id=vhLAb1dpIw,
\nI used LLM as a pointwise scorer by getting the Yes token's logprob.
\nI finetuned (lora) a Qwen2.5-32b-awq. During inference, I prompt it to predict \"Yes\" or \"No\" for given a question, a correct answer, an incorrect answer, and a possible misconception.
\nBut the Public LB is poor (0.3+), I guess it's because
enable_prefix_caching=TrueGreat job!I
", "rawMarkdown": "Great job!I" }, { "id": 3081797, "postDate": "2024-12-27T08:01:29.027Z", "content": "Great job!I wonder what gpu do you use for trainning the rerank model.
", "rawMarkdown": "Great job!I wonder what gpu do you use for trainning the rerank model.", "replies": [ { "id": 3081808, "postDate": "2024-12-27T08:44:05.590Z", "content": "~7 hours on A100
", "rawMarkdown": "~7 hours on A100 " } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T03:44:59Z", "competition_id": 82695, "competition_slug": "eedi-mining-misconceptions-in-mathematics", "team_id": 12635200, "solution_url": "/competitions/eedi-mining-misconceptions-in-mathematics/writeups/double-zeta-31st-place-solution" }