I must show my appreciation with respect to the eedi organization and kaggle team, for hosting such a stunning competition. This is my first participation of a kaggle competition, and I am very happy to take the gold medal.
\nWe generated SubjectName, ConstructName, QuestionText, Correct Answer, Incorrect Answer by providing misconceptions to GPT-4o.
\nNext, start filtering:
train.csv, we filtered generated data with base model (training Qwen 2.5 14B Instruct without synthetic data scored 0.447 on LB).train.csv, do nothing.Input the SubjectName, ConstructName, QuestionText, Correct Answer and Incorrect Answer into Qwen 2.5 32B Instruct AWQ to generate a briefly rationale (up to 50 words).
\nPrompt:
Prompt = \nFollowing @sayoulala and his discussion https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/543519.
\n32B inference referer to https://www.kaggle.com/code/cdeotte/infer-34b-with-vllm.
\n| Model | \nPublic LB | \nPrivate LB | \n
|---|---|---|
| Retriever | \n0.547 | \n0.491 | \n
| Binary classification reranker | \n0.606 | \n0.542 | \n
| Causal LM reranker | \n0.641 | \n0.571 | \n
btw, we started training Reranker only a week before the competition cut-off and not much time for attemptðŸ˜
", "comments": [ { "id": 3070999, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2024-12-13T08:13:48.823000", "content": "Great work! Lots to learn here. Thanks for the write up
", "votes": 0, "replies": [] }, { "id": 3070967, "author_name": "hinsay", "author_url": "", "post_date": "2024-12-13T07:37:02.163000", "content": "Inference codeinfer
\nTraining the retriever is referenced fromretriever
\nTraining the reranker is referenced fromreranker
training and inference codes are available here
", "votes": 0, "replies": [] } ] }, { "id": 3070920, "author_name": "James owen", "author_url": "", "post_date": "2024-12-13T07:03:27.390000", "content": "hi@zuoyouzuo, congratulations on your achievement. will you publish your complete code, I can't wait to learn your code. To be honest, I learned a lot from your public notebook, which is very useful for me as a new llm
", "votes": 0, "replies": [ { "id": 3070959, "author_name": "Okinawa", "author_url": "", "post_date": "2024-12-13T07:29:30.373000", "content": "Thanks @xuyilehhh, I'll make the inference code public soon.
", "votes": 0, "replies": [] } ] }, { "id": 3070872, "author_name": "Nikhil Tumbde", "author_url": "", "post_date": "2024-12-13T06:00:58.723000", "content": "Hi, thanks for the solution. Would you be sharing a detailed solution or inference code? I really wanted to understand how to properly use synthetic data.
\nI also have the following questions for you:
\nWhat do you mean by \"we filtered generated data with base model\" ?
\nI still couldn't understand how the generated data and rationales were used in modelling, can you please explain?
Base model means the model do not use synthetic data, training on single fold trainset scored 447 on LB.
\nI'll make the inference code public soon.
Hi. Can you elaborate on the \"filtered\" part? I am unable to understand that. Really appreciate your response.
", "votes": 0, "replies": [] }, { "id": 3072061, "author_name": "Okinawa", "author_url": "", "post_date": "2024-12-14T16:31:52.373000", "content": "Because the synthetic data generated by GPT4o is inaccurate, these data are filtered by the retriever with the highest score on LB. Specifically, we dropped the synthetic data where their misconception not in recall top 5 or 10 from retriever.
", "votes": 1, "replies": [] }, { "id": 3072335, "author_name": "Nikhil Tumbde", "author_url": "", "post_date": "2024-12-15T03:46:27.100000", "content": "Understood. Thanks for the clarification 😊
", "votes": 0, "replies": [] } ] } ] }, { "id": 3070869, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2024-12-13T05:55:07.443000", "content": "Nice job! Can you share the private lb scores in the table for retriever, binary classification reranker, and CLM reranker?
", "votes": 0, "replies": [ { "id": 3070969, "author_name": "Okinawa", "author_url": "", "post_date": "2024-12-13T07:40:48.190000", "content": "Thanks @nbroad , I've made my private score public.
", "votes": 1, "replies": [] } ] }, { "id": 3070846, "author_name": "xbustc", "author_url": "", "post_date": "2024-12-13T05:18:03.683000", "content": "Big con! I wonder whether you tried the cross entropy loss to train reranker just like flagemedding reranker?
", "votes": 0, "replies": [ { "id": 3070851, "author_name": "Okinawa", "author_url": "", "post_date": "2024-12-13T05:28:57.333000", "content": "yes, we modified code and llm prompt adapting for our solution
", "votes": 0, "replies": [] } ] }, { "id": 3070963, "author_name": "", "author_url": "", "post_date": "2024-12-13T07:32:09.940000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 17, "messages": [ { "id": 3070822, "postDate": "2024-12-13T04:34:38.297Z", "content": "I must show my appreciation with respect to the eedi organization and kaggle team, for hosting such a stunning competition. This is my first participation of a kaggle competition, and I am very happy to take the gold medal.
\nWe generated SubjectName, ConstructName, QuestionText, Correct Answer, Incorrect Answer by providing misconceptions to GPT-4o.
\nNext, start filtering:
train.csv, we filtered generated data with base model (training Qwen 2.5 14B Instruct without synthetic data scored 0.447 on LB).train.csv, do nothing.Input the SubjectName, ConstructName, QuestionText, Correct Answer and Incorrect Answer into Qwen 2.5 32B Instruct AWQ to generate a briefly rationale (up to 50 words).
\nPrompt:
Prompt = \nFollowing @sayoulala and his discussion https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/543519.
\n32B inference referer to https://www.kaggle.com/code/cdeotte/infer-34b-with-vllm.
\n| Model | \nPublic LB | \nPrivate LB | \n
|---|---|---|
| Retriever | \n0.547 | \n0.491 | \n
| Binary classification reranker | \n0.606 | \n0.542 | \n
| Causal LM reranker | \n0.641 | \n0.571 | \n
btw, we started training Reranker only a week before the competition cut-off and not much time for attemptðŸ˜
", "rawMarkdown": "I must show my appreciation with respect to the eedi organization and kaggle team, for hosting such a stunning competition. This is my first participation of a kaggle competition, and I am very happy to take the gold medal.\n\n### Overview\n1. Synthetic data generation (GPT-4o)\n2. Rationale generation (Qwen 2.5 32B Instruct AWQ with vllm)\n3. Retriever (Qwen 2.5 14B Instruct)\n4. Reranker (Qwen 2.5 14B Instruct and Qwen 2.5 32B Instruct with vllm)\n\n### Synthetic data generation\nWe generated SubjectName, ConstructName, QuestionText, Correct Answer, Incorrect Answer by providing misconceptions to GPT-4o.\nNext, start filtering:\n- For misconceptions in `train.csv`, we filtered generated data with base model (training Qwen 2.5 14B Instruct without synthetic data scored 0.447 on LB).\n- For misconceptions not in `train.csv`, do nothing.\n\n### Rationale generation\nInput the SubjectName, ConstructName, QuestionText, Correct Answer and Incorrect Answer into Qwen 2.5 32B Instruct AWQ to generate a briefly rationale (up to 50 words).\nPrompt:\n```python\nPrompt = \"\"\"Here is a question about {ConstructName} ({SubjectName}).\nQuestion: {Question}\nCorrect Answer: {CorrectAnswer}\nIncorrect Answer: {IncorrectAnswer}\n\nGenerate a brief rationale for the Incorrect Answer in a sentences, within 50 tokens. Describe the reasoning process that might lead to this choice, including any logical errors or misconceptions.\n\nKeep the explanation concise and focused on the key points, and ensure it does not exceed 50 tokens\n***Important***\n1.There is no need to analyze the relationship between the wrong answer and the correct answer. Simply describe the rationale of the incorrect answer in a few words.\n2.There is no need to have fields such as \"Incorrect Answer:\" and \"Incorrect Answer is\". Just give the rationale directly.\n3.Output results in English.\n\"\"\"\n```\n\n### Retriever\nFollowing @sayoulala and his discussion https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/543519.\n- Upsample original trainset question once for training, where misconceptions occur less than 3.\n- Training retriever by contrastive learning and using sfr for hard negative sample mining.\n\n### Reranker\n- Binary classification reranker (14B), predict whether incorrect answer and misconception are matched. (we didn't use it finally because of unefficiency)\n- Causal LM reranker (32B), using lm head take yes token logits as ranking score.Once trained, AWQ quantization is performed.\n\n32B inference referer to https://www.kaggle.com/code/cdeotte/infer-34b-with-vllm.\n\n| Model | Public LB | Private LB |\n| --- | --- | --- |\n| Retriever | 0.547 | 0.491 |\n| Binary classification reranker | 0.606 | 0.542 |\n| Causal LM reranker | 0.641 | 0.571 |\n\n\nbtw, we started training Reranker only a week before the competition cut-off and not much time for attemptðŸ˜\n", "votes": 27 }, { "id": 3070999, "postDate": "2024-12-13T08:13:48.823Z", "content": "Great work! Lots to learn here. Thanks for the write up
", "rawMarkdown": "Great work! Lots to learn here. Thanks for the write up" }, { "id": 3070967, "postDate": "2024-12-13T07:37:02.163Z", "content": "Inference codeinfer
\nTraining the retriever is referenced fromretriever
\nTraining the reranker is referenced fromreranker
training and inference codes are available here
", "rawMarkdown": "training and inference codes are available here" } ] }, { "id": 3070920, "postDate": "2024-12-13T07:03:27.390Z", "content": "hi@zuoyouzuo, congratulations on your achievement. will you publish your complete code, I can't wait to learn your code. To be honest, I learned a lot from your public notebook, which is very useful for me as a new llm
", "rawMarkdown": "hi@zuoyouzuo, congratulations on your achievement. will you publish your complete code, I can't wait to learn your code. To be honest, I learned a lot from your public notebook, which is very useful for me as a new llm", "replies": [ { "id": 3070959, "postDate": "2024-12-13T07:29:30.373Z", "content": "Thanks @xuyilehhh, I'll make the inference code public soon.
", "rawMarkdown": "Thanks @xuyilehhh, I'll make the inference code public soon." } ] }, { "id": 3070872, "postDate": "2024-12-13T06:00:58.723Z", "content": "Hi, thanks for the solution. Would you be sharing a detailed solution or inference code? I really wanted to understand how to properly use synthetic data.
\nI also have the following questions for you:
\nWhat do you mean by \"we filtered generated data with base model\" ?
\nI still couldn't understand how the generated data and rationales were used in modelling, can you please explain?
Base model means the model do not use synthetic data, training on single fold trainset scored 447 on LB.
\nI'll make the inference code public soon.
Hi. Can you elaborate on the \"filtered\" part? I am unable to understand that. Really appreciate your response.
", "rawMarkdown": "Hi. Can you elaborate on the \"filtered\" part? I am unable to understand that. Really appreciate your response." }, { "id": 3072061, "postDate": "2024-12-14T16:31:52.373Z", "content": "Because the synthetic data generated by GPT4o is inaccurate, these data are filtered by the retriever with the highest score on LB. Specifically, we dropped the synthetic data where their misconception not in recall top 5 or 10 from retriever.
", "rawMarkdown": "Because the synthetic data generated by GPT4o is inaccurate, these data are filtered by the retriever with the highest score on LB. Specifically, we dropped the synthetic data where their misconception not in recall top 5 or 10 from retriever.", "votes": 1 }, { "id": 3072335, "postDate": "2024-12-15T03:46:27.100Z", "content": "Understood. Thanks for the clarification 😊
", "rawMarkdown": "Understood. Thanks for the clarification 😊" } ] } ] }, { "id": 3070869, "postDate": "2024-12-13T05:55:07.443Z", "content": "Nice job! Can you share the private lb scores in the table for retriever, binary classification reranker, and CLM reranker?
", "rawMarkdown": "Nice job! Can you share the private lb scores in the table for retriever, binary classification reranker, and CLM reranker?", "replies": [ { "id": 3070969, "postDate": "2024-12-13T07:40:48.190Z", "content": "Thanks @nbroad , I've made my private score public.
", "rawMarkdown": "Thanks @nbroad , I've made my private score public.", "votes": 1 } ] }, { "id": 3070846, "postDate": "2024-12-13T05:18:03.683Z", "content": "Big con! I wonder whether you tried the cross entropy loss to train reranker just like flagemedding reranker?
", "rawMarkdown": "Big con! I wonder whether you tried the cross entropy loss to train reranker just like flagemedding reranker? ", "replies": [ { "id": 3070851, "postDate": "2024-12-13T05:28:57.333Z", "content": "yes, we modified code and llm prompt adapting for our solution
", "rawMarkdown": "yes, we modified code and llm prompt adapting for our solution" } ] }, { "id": 3070963, "postDate": "2024-12-13T07:32:09.940Z", "rawMarkdown": "", "isDeleted": true } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T03:47:37Z", "competition_id": 82695, "competition_slug": "eedi-mining-misconceptions-in-mathematics", "team_id": 12674927, "solution_url": "/competitions/eedi-mining-misconceptions-in-mathematics/writeups/study-hard-private-9th-public-7th-place-solution" }