First, I would like to extend my heartfelt gratitude to the organizers for hosting such a fantastic competition. This has been the most engaging contest I’ve ever participated in.
\nMy solution is quite straightforward. I used a retrieval model to select 25 candidates and re-ranked them using a single model. I applied this approach with different combinations of training data. For the 25 selected candidates, duplicate IDs had their scores averaged, while unique IDs retained their original scores.
\nUsing GPT-4o, I provided 10 sample questions and the misconceptions I aimed to generate. Questions were then created by reverse-engineering from the misconceptions.
\nThis process might seem counterintuitive. However, the role of the retrieval model was not to identify a single correct answer, but rather to ensure that the correct answer was included within the top 25 candidates. Therefore, the model needed to adopt a broader focus to achieve this objective.
\nThe scores for overlapping IDs from the two prediction results were averaged, while unique ID scores were retained. Surprisingly, this approach worked well. Since the retriever and reranker models had different strengths due to their random negative sampling, this ensembling method boosted the score by 0.02.
\nThis competition allowed me to achieve my best-ever ranking. I stayed in the gold medal range for a long time during the contest, which was incredibly exciting. Though I narrowly missed the gold medal, I am eager to apply these learnings in future competitions.
\nThank you all very much!
\nhttps://www.kaggle.com/code/tkyiws/sub-embedder-reranker-ensemble-v3?scriptVersionId=212088687
", "comments": [ { "id": 3076044, "author_name": "shion5", "author_url": "", "post_date": "2024-12-19T15:20:04.290000", "content": "Thank you for the sharing. So much informative!
", "votes": 0, "replies": [] }, { "id": 3075830, "author_name": "john", "author_url": "", "post_date": "2024-12-19T10:19:58.287000", "content": "Congratulations! Maybe the luck is accumulated for the next time.
\nI have question about Retriever Training process.
Yes, \"Do the same thing for Qwen25-32B as in Step 2\" is correct.
\nNext, 30 negative examples are randomly selected for each sample.
\nDue to memory constraints, the batch size was set to 1, which also helped avoid inconsistencies.
bravo, I like this solution, simple & efficient!
", "votes": 0, "replies": [] } ] } ] }, { "id": 3071351, "author_name": "Sinan Calisir", "author_url": "", "post_date": "2024-12-13T17:01:53.933000", "content": "Congratulations on your strong finish! Are you planning to release your code as well? I didn't participate in this one but I want to study the top solutions.
", "votes": 0, "replies": [ { "id": 3071371, "author_name": "iws", "author_url": "", "post_date": "2024-12-13T17:23:22.893000", "content": "Thank you very much. I have published the code. Please take a look!
", "votes": 0, "replies": [] } ] } ], "comment_count": 6, "messages": [ { "id": 3071288, "postDate": "2024-12-13T15:24:59.457Z", "content": "First, I would like to extend my heartfelt gratitude to the organizers for hosting such a fantastic competition. This has been the most engaging contest I’ve ever participated in.
\nMy solution is quite straightforward. I used a retrieval model to select 25 candidates and re-ranked them using a single model. I applied this approach with different combinations of training data. For the 25 selected candidates, duplicate IDs had their scores averaged, while unique IDs retained their original scores.
\nUsing GPT-4o, I provided 10 sample questions and the misconceptions I aimed to generate. Questions were then created by reverse-engineering from the misconceptions.
\nThis process might seem counterintuitive. However, the role of the retrieval model was not to identify a single correct answer, but rather to ensure that the correct answer was included within the top 25 candidates. Therefore, the model needed to adopt a broader focus to achieve this objective.
\nThe scores for overlapping IDs from the two prediction results were averaged, while unique ID scores were retained. Surprisingly, this approach worked well. Since the retriever and reranker models had different strengths due to their random negative sampling, this ensembling method boosted the score by 0.02.
\nThis competition allowed me to achieve my best-ever ranking. I stayed in the gold medal range for a long time during the contest, which was incredibly exciting. Though I narrowly missed the gold medal, I am eager to apply these learnings in future competitions.
\nThank you all very much!
\nhttps://www.kaggle.com/code/tkyiws/sub-embedder-reranker-ensemble-v3?scriptVersionId=212088687
", "rawMarkdown": "First, I would like to extend my heartfelt gratitude to the organizers for hosting such a fantastic competition. This has been the most engaging contest I’ve ever participated in.\n\n---\n\n## Model Overview and Features\n\n### Overview\n\nMy solution is quite straightforward. I used a retrieval model to select 25 candidates and re-ranked them using a single model. I applied this approach with different combinations of training data. For the 25 selected candidates, duplicate IDs had their scores averaged, while unique IDs retained their original scores.\n\n---\n\n### Score Results\n\n- **Final Score:** 0.553 (private) / 0.617 (public)\n - **Retriever Model 1:** 0.513 (public) / 0.455 (private) - Qwen2.5-32B\n - **Retriever Model 2:** 0.515 (public) / 0.450 (private) - Qwen2.5-32B\n - **Reranker Model 1:** 0.610 (public) / 0.538 (private) - Qwen2.5-32B-Instruct-bnb-4bit\n - **Reranker Model 2:** 0.596 (public) / 0.536 (private) - Qwen2.5-32B-Instruct-bnb-4bit\n\n---\n\n### Synthetic Data Generation\n\nUsing GPT-4o, I provided 10 sample questions and the misconceptions I aimed to generate. Questions were then created by reverse-engineering from the misconceptions.\n\n- **Misconceptions never appearing in competition data:** 2 types (985 samples × 2)\n- **Misconceptions appearing only once:** 1 type (847 samples)\n\n---\n\n### Training Methodology\n\n#### Retriever Training:\n1. Extracted the top 200 candidates using the SFR-Embedding-2\\_R pre-trained model.\n2. Used Qwen25-14B to randomly select 30 hard negatives from the top 200 candidates and conducted contrastive learning. Extracted the top 200 candidates again.\n3. Repeated the same process using Qwen25-32B.\n\nThis process might seem counterintuitive. However, the role of the retrieval model was not to identify a single correct answer, but rather to ensure that the correct answer was included within the top 25 candidates. Therefore, the model needed to adopt a broader focus to achieve this objective.\n\n#### Reranker Training:\n1. Extracted the top 25 candidates for each question using the retriever model.\n2. Randomly selected 2-25 options from these candidates. If there was no correct answer, one option was replaced with the correct answer.\n3. Conducted instruction tuning to predict the correct answer from the set of options. Varying the number of options helped capture the core nature of the task.\n\n---\n\n### Ensembling Method\n\nThe scores for overlapping IDs from the two prediction results were averaged, while unique ID scores were retained. Surprisingly, this approach worked well. Since the retriever and reranker models had different strengths due to their random negative sampling, this ensembling method boosted the score by 0.02.\n\n---\n\n## Final Thoughts\n\nThis competition allowed me to achieve my best-ever ranking. I stayed in the gold medal range for a long time during the contest, which was incredibly exciting. Though I narrowly missed the gold medal, I am eager to apply these learnings in future competitions.\n\nThank you all very much!\n\n### Inference Code (Version 2)\nhttps://www.kaggle.com/code/tkyiws/sub-embedder-reranker-ensemble-v3?scriptVersionId=212088687\n\n", "votes": 12 }, { "id": 3076044, "postDate": "2024-12-19T15:20:04.290Z", "content": "Thank you for the sharing. So much informative!
", "rawMarkdown": "Thank you for the sharing. So much informative!" }, { "id": 3075830, "postDate": "2024-12-19T10:19:58.287Z", "content": "Congratulations! Maybe the luck is accumulated for the next time.
\nI have question about Retriever Training process.
Yes, \"Do the same thing for Qwen25-32B as in Step 2\" is correct.
\nNext, 30 negative examples are randomly selected for each sample.
\nDue to memory constraints, the batch size was set to 1, which also helped avoid inconsistencies.
bravo, I like this solution, simple & efficient!
", "rawMarkdown": "bravo, I like this solution, simple & efficient!" } ] } ] }, { "id": 3071351, "postDate": "2024-12-13T17:01:53.933Z", "content": "Congratulations on your strong finish! Are you planning to release your code as well? I didn't participate in this one but I want to study the top solutions.
", "rawMarkdown": "Congratulations on your strong finish! Are you planning to release your code as well? I didn't participate in this one but I want to study the top solutions. ", "replies": [ { "id": 3071371, "postDate": "2024-12-13T17:23:22.893Z", "content": "Thank you very much. I have published the code. Please take a look!
", "rawMarkdown": "Thank you very much. I have published the code. Please take a look!" } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T03:46:52Z", "competition_id": 82695, "competition_slug": "eedi-mining-misconceptions-in-mathematics", "team_id": 12608061, "solution_url": "/competitions/eedi-mining-misconceptions-in-mathematics/writeups/iws-15th-place-solution" }