![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2064220%2F98b4dea553bdc5d6d050b10cc99c5cb9%2Fsolution.drawio%20(5).png?generation=1734097628389406&alt=media\")
First, I would like to thank the organizers for hosting such an interesting competition with plenty of room for innovation and no major issues. I also want to thank my teammates who helped create a smooth communication environment for my first team-up experience.
\n
Cross validation
\nSynthetic data generation
\nKnowledge Distillation(KD)
\nWe input the QuestionText, correct answer, and incorrect answer into the LLM (Qwen 2.5 32B Instruct) to estimate the Misconception. Using this information improves the biencoder's cv+0.04. While the Listwise reranker mentioned later uses the same model as KD, it showed an increase of about cv+0.01 compared to when it's not included.
\nPROMPT_FORMAT = \nCandidate generation
\nListwise reranker
\nOverview
\nprompt
\nNA_PROMPT_FORMAT = \nmisconception_list
\nIt contains 52 choices of uppercase + lowercase alphabets and their corresponding MisconceptionNames. They are ordered by the Biencoder output.
\nA: Thinks sign parallel lines just means opposite\nB: Does recognise the notation parallel sides\n.\n.\n.\ny: Thinks that co-interior angles can lie on opposite sides of a transversal\nz: Believes the gradient of perpendicular lines just have opposite signs<|im_end|>\nlabel_choice
\nrank
\nTraining
\nInference
\n| \n | 52x1 | \n52x2 | \n52x3 | \n
|---|---|---|---|
| fold 4 | \n0.623(0.627)[0572] | \n0.626(0.633)[0.581] | \n0.630(0.631)[0.581] | \n
| fold 3 | \n\n | (0.644)[0.581] | \n\n |
| fold (3 + 4) | \n\n | (0.650)[0.590] | \n\n |
| fold (2(60%sample) + 3 + 4) | \n\n | (0.653)[0.597] | \n\n |
My local machine is 1 x RTX 4090.
\nI rented an A100 (from vast.ai) for training a 32B model, and including experiments, it cost me about $350 in total…
\nTraining one 32B Listwise model took about 7 hours on the A100.
\nWe released training code. → Eedi-5th-solution
", "comments": [ { "id": 3070708, "author_name": "Nikita Babych", "author_url": "", "post_date": "2024-12-13T01:14:23.930000", "content": "Congratulations! I enjoyed competing with you on the public leaderboard.
\nIt’s very interesting that you were able to achieve such a high score without ensembling models on the candidate generation stage. It was quite important in my solution.
Since Qwen likes a lot to preserve the order of the init list when it is not sure.
", "votes": 1, "replies": [ { "id": 3070728, "author_name": "Ebi", "author_url": "", "post_date": "2024-12-13T01:46:47.357000", "content": "Congratulations on achieving Solo Gold!
\nAlthough it's not mentioned, we use the biencoder corresponding to each fold for the listwise reranker in each fold.
\n\nIt’s very interesting that you were able to achieve such a high score without ensembling models on the candidate generation stage. It was quite important in my solution.
\n
I think the answer to this is because we reranked the retriever results by 32B model to the top 104, not the top 50 like you did. We were able to obtain a high recall even though the performance of the retriever model was not high.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3075740, "author_name": "Santiago Mota", "author_url": "", "post_date": "2024-12-19T07:39:42.197000", "content": "Congrats. Nice job on the competition and thank for the explanation and releasing the code
", "votes": 1, "replies": [] }, { "id": 3075728, "author_name": "Nikhil Tumbde", "author_url": "", "post_date": "2024-12-19T07:18:17.060000", "content": "Hi @ebinan92, thanks for publishing the code and congratulations for securing the gold medal.
\nI wanted to confirm one thing regarding the quantization, I can see that the re-ranker (32B) was finetuned in BF16, so I believe you directly merged the lora weights with the model weights. Is that right?
Also, any reason for using DataCollatorForSeq2Seq instead of DataCollatorForCompletionOnlyLM ? Am I missing something?
", "votes": 1, "replies": [ { "id": 3076572, "author_name": "Ebi", "author_url": "", "post_date": "2024-12-20T04:49:32.030000", "content": "\n\nI wanted to confirm one thing regarding the quantization, I can see that the re-ranker (32B) was finetuned in BF16, so I believe you directly merged the lora weights with the model weights. Is that right?
\n
Yes, that's correct. We're not using QLoRA to avoid performance degradation during merging.
\n\n\nAlso, any reason for using DataCollatorForSeq2Seq instead of DataCollatorForCompletionOnlyLM ? Am I missing something?
\n
For fine-tuning with unsloth, I'm using DataCollatorForSeq2Seq based on this issue as a reference.
\nAs you mentioned, I think it would also be possible to use DataCollatorForCompletionOnlyLM.
\n\nYes, that's correct. We're not using QLoRA to avoid performance degradation during merging.
\n
Also, as you used training argument as BF16 instead of FP16, all the computations are carried out in BF16 and the model weights are also saved as BF16 which helps avoid any performance degradations.
", "votes": 1, "replies": [] } ] } ] }, { "id": 3075573, "author_name": "john", "author_url": "", "post_date": "2024-12-19T03:26:22.813000", "content": "Congratulations. Thank you for sharing your solution and code! The code is structed very well!
\nI have some questions about your code for training retriever.
Short answer:
\nI chose these parameters because they showed the best recall@100 (my main target metric for bi-encoder) after testing several combinations.
Detailed explanation:
\nThanks for your explain.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3071868, "author_name": "Camaro", "author_url": "", "post_date": "2024-12-14T12:21:21.130000", "content": "Congratulations and thank you for sharing your solution and clean code!
\nI have a question about data generation.
\n\n\nFor these few shot samples, we insert problems and incorrect answers that have similar MisconceptionNames to the one being generated.
\n
Adding similar MisconceptionNames as few-shot examples is a clever approach. Have you checked how it compares to random sampling?
", "votes": 1, "replies": [ { "id": 3071876, "author_name": "Ebi", "author_url": "", "post_date": "2024-12-14T12:35:45.937000", "content": "I'm glad the code I shared was helpful.
\n\n\nAdding similar MisconceptionNames as few-shot examples is a clever approach. Have you checked how it compares to random sampling?
\n
Yes, We have confirmed this. There is an effect of about +0.011~0.014 in all of local cv, private lb, and public lb.
", "votes": 1, "replies": [] }, { "id": 3071877, "author_name": "Camaro", "author_url": "", "post_date": "2024-12-14T12:37:23.913000", "content": "Thanks for the quick answer! I'll give it a try.
\nA few more questions:
\n\n\nIf I understand your code correctly, it seems to generate one sample per misconception ID. Did generating more samples not improve the score?
\n
We tried it, but the local cv got worse, so we didn't submit it. However, after reading some solutions, it seemed important to bring MisconceptionId, which is not included in the trainng data, to the top of the recommendation results, so now I suspect that if we had submitted it, the public lb would have increased.
\n\n\nYou are using Gemini Flash 1.5. When I tried generating with this model, it sometimes produced incorrect answers. Have you tried other models, such as O1 or al least Gemini Pro?
\n
Sorry about this. I made a mistake in sharing the code and we actually used gemini-1.5-pro. We compared the accuracy of biencoder with gemini-1.5-flash and gemini-1.5-pro, but the scores were not different at all (both local cv and public LB).
\nSo we stopped the experiment of changing the LLM used to generate synthetic data here, but now I think it might be effective for Listwise reranking, because there was also no significant change in the biencoder score when we experimented with changing the few-shot examples.
Great work and write up! congrats
", "votes": 1, "replies": [] }, { "id": 3070771, "author_name": "James owen", "author_url": "", "post_date": "2024-12-13T03:05:52.540000", "content": "Congratulations ebi , looking forward to your complete code and dataset
", "votes": 1, "replies": [] }, { "id": 3072390, "author_name": "MrSimple", "author_url": "", "post_date": "2024-12-15T05:31:42.653000", "content": "Congratulations and thank you for sharing your solution and clean code!
", "votes": 0, "replies": [] }, { "id": 3084546, "author_name": "", "author_url": "", "post_date": "2024-12-31T02:15:38.653000", "content": "", "votes": 0, "replies": [] }, { "id": 3070692, "author_name": "", "author_url": "", "post_date": "2024-12-13T00:50:17.443000", "content": "Congrats, nice pipeline. Hope to see the code release soon !
", "votes": 2, "replies": [ { "id": 3070699, "author_name": "", "author_url": "", "post_date": "2024-12-13T00:59:01.343000", "content": "And @ebinan92 are you finetuning the embedding model? i see the embedding part seems not.
", "votes": 1, "replies": [ { "id": 3070707, "author_name": "Ebi", "author_url": "", "post_date": "2024-12-13T01:11:08.180000", "content": "Do you refer to the MisconceptionName embedding when generating synthetic data? Yes, no fine-tuning is done here.
", "votes": 1, "replies": [] } ] } ] }, { "id": 3076676, "author_name": "TFI", "author_url": "", "post_date": "2024-12-20T07:16:29.787000", "content": "Thank you for sharing & congrats!
", "votes": 0, "replies": [] } ], "comment_count": 23, "messages": [ { "id": 3070688, "postDate": "2024-12-13T00:40:42.827Z", "content": "First, I would like to thank the organizers for hosting such an interesting competition with plenty of room for innovation and no major issues. I also want to thank my teammates who helped create a smooth communication environment for my first team-up experience.
\n
Cross validation
\nSynthetic data generation
\nKnowledge Distillation(KD)
\nWe input the QuestionText, correct answer, and incorrect answer into the LLM (Qwen 2.5 32B Instruct) to estimate the Misconception. Using this information improves the biencoder's cv+0.04. While the Listwise reranker mentioned later uses the same model as KD, it showed an increase of about cv+0.01 compared to when it's not included.
\nPROMPT_FORMAT = \nCandidate generation
\nListwise reranker
\nOverview
\nprompt
\nNA_PROMPT_FORMAT = \nmisconception_list
\nIt contains 52 choices of uppercase + lowercase alphabets and their corresponding MisconceptionNames. They are ordered by the Biencoder output.
\nA: Thinks sign parallel lines just means opposite\nB: Does recognise the notation parallel sides\n.\n.\n.\ny: Thinks that co-interior angles can lie on opposite sides of a transversal\nz: Believes the gradient of perpendicular lines just have opposite signs<|im_end|>\nlabel_choice
\nrank
\nTraining
\nInference
\n| \n | 52x1 | \n52x2 | \n52x3 | \n
|---|---|---|---|
| fold 4 | \n0.623(0.627)[0572] | \n0.626(0.633)[0.581] | \n0.630(0.631)[0.581] | \n
| fold 3 | \n\n | (0.644)[0.581] | \n\n |
| fold (3 + 4) | \n\n | (0.650)[0.590] | \n\n |
| fold (2(60%sample) + 3 + 4) | \n\n | (0.653)[0.597] | \n\n |
My local machine is 1 x RTX 4090.
\nI rented an A100 (from vast.ai) for training a 32B model, and including experiments, it cost me about $350 in total…
\nTraining one 32B Listwise model took about 7 hours on the A100.
\nWe released training code. → Eedi-5th-solution
", "rawMarkdown": "# Comment\n\nFirst, I would like to thank the organizers for hosting such an interesting competition with plenty of room for innovation and no major issues. I also want to thank my teammates who helped create a smooth communication environment for my first team-up experience.\n\n# Solution summary\n.png?generation=1734097628389406&alt=media)\n\n1. **Synthetic data generation**\n * We create problems using LLM with few shot prompting for each MisconceptionId that is not included in the training dataset.\n * For these few shot samples, we insert problems and incorrect answers that have similar MisconceptionNames to the one being generated.\n2. **Knowledge Distillation**\n * We input the questiontext, correct answer, and incorrect answer into the LLM (Qwen 2.5 32B Instruct) to generate the reasoning behind the incorrect answer. This content is used as input for all subsequent models (Biencoder, Listwise reranker).\n3. **Candidate generation**\n * We use a biencoder model to narrow down the Misconception candidates for each problem and incorrect answer.\n4. **Listwise reranking**\n * We input the Misconception candidates sorted by the Biencoder, 52 at a time (corresponding to the number of uppercase and lowercase alphabet letters used as options), into the fine-tuned LLM(Qwen 2.5 32B Instruct) to obtain probability for each option. We then sort the Misconceptions based on these probability.\n * For this process, we use the top 104 candidates from the biencoder. In other words, we perform inference twice with 52 candidates each time, extract the probability, and combine the results.\n5. **Ensemble**\n * Since the Listwise reranker uses GPTQ + vLLM inference, a single inference takes only about 120 minutes, allowing the results to be ensembled across three folds. (But due to the random delays discussed [here](https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/550743), our best sub could not include all results from the third fold. I hope that Kaggle staff will address these issues in the future.)\n\n# Detailed solution\n\n1. **Cross validation**\n * Initially, we used GroupKFold(k=5) based on QuestionId, but since the lb was significantly lower than local cv, we suspected there might be many MisconceptionIds that only appear in the test data. Based on this, we switched to GroupKFold(k=5) using SubjectId. This change resulted in a higher proportion of MisconceptionIds that only appear in validation data, which tended to make the cv closer to lb.\n * example local cv 0.626, lb: 0.633\n \n2. **Synthetic data generation**\n * Synthetic data generation is essential to include MisconceptionIds without existing problems in the prediction results. For each MisconceptionName without problems, we generated one set of existing dataset information (QuestionName, SubjectName, ConstructName, Correct Answer, Incorrect Answer) using few shot prompting (4 shots) with LLM (gemini-1.5-pro).\n * For the few shot prompting samples, we embedded all MisconceptionNames using the model (dunzhang/stella_en_1.5B_v5) and included problems corresponding to similar MisconceptionNames. This resulted in approximately +0.01 improvement in cv compared to using random samples.\n \n3. **Knowledge Distillation(KD)**\n * We input the QuestionText, correct answer, and incorrect answer into the LLM (Qwen 2.5 32B Instruct) to estimate the Misconception. Using this information improves the biencoder's cv+0.04. While the Listwise reranker mentioned later uses the same model as KD, it showed an increase of about cv+0.01 compared to when it's not included.\n * prompt\n \n ```python\n PROMPT_FORMAT = \"\"\"<|im_start|>system\n You will be given a math problem and its correct and incorrect answer.\n First explain why the correct answer is correct, and finally explain reasons and misconceptions for incorrect answer.\n Please briefly explain in 200 words or less.<|im_end|>\n <|im_start|>user\n Problem: {QuestionText}\\nCorrect Answer: {Correct}\\nIncorrect Answer: {Answer}.<|im_end|>\n <|im_start|>assistant\n \"\"\"\n ```\n \n \n4. **Candidate generation**\n * We extract MisconceptionName candidates for each problem and incorrect answer pair using Biencoder. While we perform negative mining, hard negatives were not effective, so we include slightly relaxed samples.\n * Training parameters\n * library: SentenceTransfomer\n * model: dunzhang/stella_en_1.5B_v5\n * negative mining params\n * range_min: 512\n * num_negatives: 2\n * lora config\n * r: 32\n * lora_alpha: 64\n * loss: CachedMultipleNegativesRankingLoss\n * epoch: 1\n * cv: 0.41, lb: 0.39\n \n5. **Listwise reranker**\n - Overview\n - The reranking part using LLM appears to be the most unique aspect of this competition, and I expect that how this part was implemented significantly impacted the scores. Our teammate @kcotton21 had already come up with the idea of generating single tokens and using their logits for sorting at the beginning of the competition (reference: https://arxiv.org/abs/2406.15657), and our solution is based on this.\n - prompt\n - After various trial and error, we ultimately performed regular fine-tuning using the following prompt to have the LLM generate appropriate choices (which must be single tokens to enable sorting by logits(probability)).\n ```python\n NA_PROMPT_FORMAT = \"\"\"<|im_start|>system\n You will be given math problem, overview of ther problem, correct answer, incorrect answer, and incorrect reason.\n Please return the most appropriate option from the list of misconceptions. Do not output anything other than options. If there are no suitable \n options, return NA.<|im_end|>\n <|im_start|>user\n # Math Problem\n Problem: {QuestionText}\\nOverview: ({SubjectName}){ConstructName}\\nCorrectAnswer: {Correct}\\nIncorrectAnswer: \n {Answer}\\nIncorrectReason: {kd}\n\n # Misconception List (rank: {rank})\n {misconception_list}<|im_end|>\n <|im_start|>assistant\n {label_choice}\"\"\"\n ```\n\n - misconception_list\n - It contains 52 choices of uppercase + lowercase alphabets and their corresponding MisconceptionNames. They are ordered by the Biencoder output.\n \n ```python\n A: Thinks sign for parallel lines just means opposite\n B: Does not recognise the notation for parallel sides\n .\n .\n .\n y: Thinks that co-interior angles can lie on opposite sides of a transversal\n z: Believes the gradient of perpendicular lines just have opposite signs<|im_end|>\n ```\n \n - label_choice\n - This is the correct answer choice used for fine-tuning. If the correct answer is not included in the choices, \"NA\" (which is also a single token in Qwen 2.5 32B) is inserted.\n - rank\n - Indicates which range of choices {start_idx - end_idx} from the Biencoder's output is included.\n - Training\n - We divided the top 208 outputs from the biencoder into non-overlapping groups of 52 each (1-52, 53-104, 105-156, 157-208) and performed fine-tuning.\n - Training parameters\n - loss\n - This is the standard cross-entropy loss used in regular fine-tuning, calculated across the entire vocabulary. We chose this simpler implementation method since calculating cross-entropy limited to only the candidate choices + \"NA\" token showed little difference in results.\n - model: Qwen/Qwen2.5-32B-Instruct\n - lora_config\n - r: 24\n - lora_alpha: 48\n - lr: 5e-5\n - scheduler: cosine\n - warmup_steps: 10\n - epoch : 1.0\n - global batch size: 8\n - library: unsloth\n - Inference\n - From the biencoder output, we take the top 104 items and split them into two sets (1-52 and 53-104). Each set is input to the model, generating only single tokens to obtain probabilities for each option. The results from both sets are then combined and sorted.\n - While adding to make it top 156 (52x3 inference) showed a slight improvement in local CV, it wasn't reflected in the leaderboard score, so we didn't adopt this approach.\n - Quantized to 4-bit using GPTQ and inference is performed using vLLM\n\n# cv(public)[private] scores\n\n| | 52x1 | 52x2 | 52x3 |\n| --- | --- | --- | --- |\n| fold 4 | 0.623(0.627)[0572] | 0.626(0.633)[0.581] | 0.630(0.631)[0.581] |\n| fold 3 | | (0.644)[0.581] | |\n| fold (3 + 4) | | (0.650)[0.590] | |\n| fold (2(60%sample) + 3 + 4) | | (0.653)[0.597] | |\n\n# Computational costs\n\nMy local machine is 1 x RTX 4090.\n\nI rented an A100 (from [vast.ai](http://vast.ai/)) for training a 32B model, and including experiments, it cost me about $350 in total...\n\nTraining one 32B Listwise model took about 7 hours on the A100.\n\n# What didn't work\n- Options used in the Listwise reranker\n - As single token choices, we tried combinations of two alphabets, combinations of alphabets and numbers 0-9, and japanese hiragana, but all of these approaches performed worse than using just the 52 alphabets.\n- utilization of QwQ-32B-preview\n- multi step reraking (Qwen 2.5 7B(104 candidate) → Qwen 2.5 32B(52 cadindate))\n- adding examples of problems and misconceptions in the reraking prompts.\n\n# Code release\nWe released training code. → [Eedi-5th-solution](https://github.com/ebinan92/Eedi-5th-solution?tab=readme-ov-file)", "votes": 76 }, { "id": 3070708, "postDate": "2024-12-13T01:14:23.930Z", "content": "Congratulations! I enjoyed competing with you on the public leaderboard.
\nIt’s very interesting that you were able to achieve such a high score without ensembling models on the candidate generation stage. It was quite important in my solution.
Since Qwen likes a lot to preserve the order of the init list when it is not sure.
", "rawMarkdown": "Since Qwen likes a lot to preserve the order of the init list when it is not sure. ", "votes": 1, "replies": [ { "id": 3070728, "postDate": "2024-12-13T01:46:47.357Z", "content": "Congratulations on achieving Solo Gold!
\nAlthough it's not mentioned, we use the biencoder corresponding to each fold for the listwise reranker in each fold.
\n\nIt’s very interesting that you were able to achieve such a high score without ensembling models on the candidate generation stage. It was quite important in my solution.
\n
I think the answer to this is because we reranked the retriever results by 32B model to the top 104, not the top 50 like you did. We were able to obtain a high recall even though the performance of the retriever model was not high.
", "rawMarkdown": ">It’s very interesting that you were able to achieve such a high score without ensembling models on the candidate generation stage. It was quite important in my solution.\n\nI think the answer to this is because we reranked the retriever results by 32B model to the top 104, not the top 50 like you did. We were able to obtain a high recall even though the performance of the retriever model was not high." } ] } ] }, { "id": 3075740, "postDate": "2024-12-19T07:39:42.197Z", "content": "Congrats. Nice job on the competition and thank for the explanation and releasing the code
", "rawMarkdown": "Congrats. Nice job on the competition and thank for the explanation and releasing the code", "votes": 1 }, { "id": 3075728, "postDate": "2024-12-19T07:18:17.060Z", "content": "Hi @ebinan92, thanks for publishing the code and congratulations for securing the gold medal.
\nI wanted to confirm one thing regarding the quantization, I can see that the re-ranker (32B) was finetuned in BF16, so I believe you directly merged the lora weights with the model weights. Is that right?
Also, any reason for using DataCollatorForSeq2Seq instead of DataCollatorForCompletionOnlyLM ? Am I missing something?
", "rawMarkdown": "Hi @ebinan92, thanks for publishing the code and congratulations for securing the gold medal.\nI wanted to confirm one thing regarding the quantization, I can see that the re-ranker (32B) was finetuned in BF16, so I believe you directly merged the lora weights with the model weights. Is that right? \n\nAlso, any reason for using DataCollatorForSeq2Seq instead of DataCollatorForCompletionOnlyLM ? Am I missing something?", "votes": 1, "replies": [ { "id": 3076572, "postDate": "2024-12-20T04:49:32.030Z", "content": "\n\nI wanted to confirm one thing regarding the quantization, I can see that the re-ranker (32B) was finetuned in BF16, so I believe you directly merged the lora weights with the model weights. Is that right?
\n
Yes, that's correct. We're not using QLoRA to avoid performance degradation during merging.
\n\n\nAlso, any reason for using DataCollatorForSeq2Seq instead of DataCollatorForCompletionOnlyLM ? Am I missing something?
\n
For fine-tuning with unsloth, I'm using DataCollatorForSeq2Seq based on this issue as a reference.
\nAs you mentioned, I think it would also be possible to use DataCollatorForCompletionOnlyLM.
\n\nYes, that's correct. We're not using QLoRA to avoid performance degradation during merging.
\n
Also, as you used training argument as BF16 instead of FP16, all the computations are carried out in BF16 and the model weights are also saved as BF16 which helps avoid any performance degradations.
", "rawMarkdown": ">Yes, that's correct. We're not using QLoRA to avoid performance degradation during merging.\n\nAlso, as you used training argument as BF16 instead of FP16, all the computations are carried out in BF16 and the model weights are also saved as BF16 which helps avoid any performance degradations. ", "votes": 1 } ] } ] }, { "id": 3075573, "postDate": "2024-12-19T03:26:22.813Z", "content": "Congratulations. Thank you for sharing your solution and code! The code is structed very well!
\nI have some questions about your code for training retriever.
Short answer:
\nI chose these parameters because they showed the best recall@100 (my main target metric for bi-encoder) after testing several combinations.
Detailed explanation:
\nThanks for your explain.
", "rawMarkdown": "Thanks for your explain." } ] } ] }, { "id": 3071868, "postDate": "2024-12-14T12:21:21.130Z", "content": "Congratulations and thank you for sharing your solution and clean code!
\nI have a question about data generation.
\n\n\nFor these few shot samples, we insert problems and incorrect answers that have similar MisconceptionNames to the one being generated.
\n
Adding similar MisconceptionNames as few-shot examples is a clever approach. Have you checked how it compares to random sampling?
", "rawMarkdown": "Congratulations and thank you for sharing your solution and clean code!\n\nI have a question about data generation.\n\n> For these few shot samples, we insert problems and incorrect answers that have similar MisconceptionNames to the one being generated.\n\nAdding similar MisconceptionNames as few-shot examples is a clever approach. Have you checked how it compares to random sampling?", "votes": 1, "replies": [ { "id": 3071876, "postDate": "2024-12-14T12:35:45.937Z", "content": "I'm glad the code I shared was helpful.
\n\n\nAdding similar MisconceptionNames as few-shot examples is a clever approach. Have you checked how it compares to random sampling?
\n
Yes, We have confirmed this. There is an effect of about +0.011~0.014 in all of local cv, private lb, and public lb.
", "rawMarkdown": "I'm glad the code I shared was helpful.\n>Adding similar MisconceptionNames as few-shot examples is a clever approach. Have you checked how it compares to random sampling?\n\n\nYes, We have confirmed this. There is an effect of about +0.011~0.014 in all of local cv, private lb, and public lb.", "votes": 1 }, { "id": 3071877, "postDate": "2024-12-14T12:37:23.913Z", "content": "Thanks for the quick answer! I'll give it a try.
\nA few more questions:
\n\n\nIf I understand your code correctly, it seems to generate one sample per misconception ID. Did generating more samples not improve the score?
\n
We tried it, but the local cv got worse, so we didn't submit it. However, after reading some solutions, it seemed important to bring MisconceptionId, which is not included in the trainng data, to the top of the recommendation results, so now I suspect that if we had submitted it, the public lb would have increased.
\n\n\nYou are using Gemini Flash 1.5. When I tried generating with this model, it sometimes produced incorrect answers. Have you tried other models, such as O1 or al least Gemini Pro?
\n
Sorry about this. I made a mistake in sharing the code and we actually used gemini-1.5-pro. We compared the accuracy of biencoder with gemini-1.5-flash and gemini-1.5-pro, but the scores were not different at all (both local cv and public LB).
\nSo we stopped the experiment of changing the LLM used to generate synthetic data here, but now I think it might be effective for Listwise reranking, because there was also no significant change in the biencoder score when we experimented with changing the few-shot examples.
Great work and write up! congrats
", "rawMarkdown": "Great work and write up! congrats", "votes": 1 }, { "id": 3070771, "postDate": "2024-12-13T03:05:52.540Z", "content": "Congratulations ebi , looking forward to your complete code and dataset
", "rawMarkdown": "Congratulations ebi , looking forward to your complete code and dataset", "votes": 1 }, { "id": 3072390, "postDate": "2024-12-15T05:31:42.653Z", "content": "Congratulations and thank you for sharing your solution and clean code!
", "rawMarkdown": "Congratulations and thank you for sharing your solution and clean code!\n\n" }, { "id": 3084546, "postDate": "2024-12-31T02:15:38.653Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3070692, "postDate": "2024-12-13T00:50:17.443Z", "content": "Congrats, nice pipeline. Hope to see the code release soon !
", "rawMarkdown": "Congrats, nice pipeline. Hope to see the code release soon !", "votes": 2, "isDeleted": true, "replies": [ { "id": 3070699, "postDate": "2024-12-13T00:59:01.343Z", "content": "And @ebinan92 are you finetuning the embedding model? i see the embedding part seems not.
", "rawMarkdown": "And @ebinan92 are you finetuning the embedding model? i see the embedding part seems not.", "votes": 1, "isDeleted": true, "replies": [ { "id": 3070707, "postDate": "2024-12-13T01:11:08.180Z", "content": "Do you refer to the MisconceptionName embedding when generating synthetic data? Yes, no fine-tuning is done here.
", "rawMarkdown": "Do you refer to the MisconceptionName embedding when generating synthetic data? Yes, no fine-tuning is done here.", "votes": 1 } ] } ] }, { "id": 3076676, "postDate": "2024-12-20T07:16:29.787Z", "content": "Thank you for sharing & congrats!
", "rawMarkdown": "Thank you for sharing & congrats!" } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T03:48:11Z", "competition_id": 82695, "competition_slug": "eedi-mining-misconceptions-in-mathematics", "team_id": 12647413, "solution_url": "/competitions/eedi-mining-misconceptions-in-mathematics/writeups/ebi-ktr-5th-place-solution" }