![\"Inference](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2125251%2F3b2245b7c0baaab58b9d9232c6ff9bae%2Finference_pipeline.png?generation=1734198970117751&alt=media\")
I would like to thank Kaggle and EEDI for hosting this interesting competition! It was challenging and provided many opportunities to learn/apply new techniques. As always, I'm grateful to the Kaggle community for innovative ideas and engaging discussions. I'm excited to share my detailed solution and hope others will find it helpful!
\nGiven a diagnostic math MCQ, together with the correct answer and an incorrect answer, the task was to recommend top 25 misconceptions (sorted by their affinity to the incorrect answer) from a pool of 2.5k+ misconceptions. \nEven though the standard retrieve-rerank framework was a natural fit, it came with a few challenges:
\nNovice Learner i.e. generating the incorrect answer stemming from a specific misconception or (2) an Expert Tutor i.e. identifying the misconception that explains a given incorrect answer. LLMs excel at solving math problems, but they are not so great at counterfactual reasoning.\n\nInitial efforts to use pre-trained language models have not been successful, likely due to the complexity of the mathematical content in the questions.
\n
At the start of the competition, I had the following hypotheses on what could be important for a good solution:
\nThese hypotheses guided me to design my experiments throughout the competition and helped me to stick to an idea even though the initial attempts were not successful.
\nI implemented a retrieve-and-rerank system that involved a sequence of four steps:
\ntop candidate similarity score - constant. Candidates with similarity scores higher than the dynamic threshold are kept. Qwen/Qwen2.5-14B ranker to process all retrieved candidates and shortlist top 8.Qwen/Qwen2.5-32B ranker to process top 8 candidates and narrow them down to top 5.Qwen/Qwen2.5-72B ranker to finalize the ranking of top 5 candidates.A schematic diagram of the pipeline and the leaderboard scores from each stages are shown below:\n
The final 25 misconception predictions comprised of:
\nThe training data mix for both the retriever and ranker models included competition data (1.8k examples) and synthetic data (10k examples). The synthetic data played a crucial role in improving both raw performance and generalization capability with respect to unseen misconceptions.
\nIn an ideal scenario, synthetic MCQs should have the following properties:
\nI started generating diagnostic MCQs with a simple prompt that specified the expected format and a misconception of interest. However, it resulted in a dataset that violated almost all the properties to varying degrees. It was expected since
\nI experimented with several strategies to address the issues and managed to create a high quality synthetic dataset with a grouped synthetic data generation approach and incorporating LLM based filtering. Specifically, I first created clusters of closely related misconceptions leveraging co-occurrence statistics of misconceptions in retriever/ranker predictions on validation data. This notebook demonstrates the clustering process. Here is an example of a misconception cluster:
- Thinks x^2 - a^2x is a valid form of difference of two squares\n- When factorising a quadratic without a non variable term, tries to double bracket factorise\n- Incorrectly factorises a quadratic\n- Believes the constant in an expanded quadratic comes from adding the two numbers in the brackets\n- Believes the coefficent of x in an expanded quadratic comes from multiplying the two numbers in the brackets\n- Does not recognise difference of two squares\n- Does not realise a quadratic must be in the form ax^2+bx+c=0 to be factorised\n- Does not know how to identify common factors from algebraic terms\n- Believes the difference of 2 squares method of factorising also works for the addition of 2 squares\n- When factorising, believes they can choose different factors for each term\n- Believes they can factorise a difference of two squares by placing the constant in both brackets without square rooting\nI next generated 5 new MCQs for each cluster using Claude 3.5 Sonnet with a refined prompt that included 4-5 reference examples on misconceptions from the cluster. I used the metaprompt tool from Anthropic cookbook for prompt engineering. Here is the final prompt used for data generation:
You will be generating Multiple Choice Questions (MCQs) that diagnose specific mathematical misconceptions. Here are the misconceptions you should focus on:\n\n<misconceptions>\n{cluster_misconceptions}\n</misconceptions>\n\nHere are reference MCQs that demonstrate how to effectively diagnose these misconceptions:\n\n<reference_mcqs>\n{reference_mcqs}\n</reference_mcqs>\n\nYour task is to generate {num_mcqs} new MCQs that diagnose misconceptions not already covered by the reference MCQs.\n\nFirst, analyze the reference MCQs carefully:\n1. For each reference MCQ, identify in your <analysis> tags:\n - Which misconception it targets\n - How the incorrect answers map to specific misconceptions\n - What makes the question effective at diagnosing the misconception\n2. Note the style, difficulty level, and precision of language used\n\nThen, in your <planning> tags:\n- List which misconceptions still need coverage\n- For each needed misconception, brainstorm mathematical contexts where it commonly appears\n- Design questions where the misconception leads naturally to specific wrong answers\n- Take notes on how you can craft new MCQs that adheres to the reference MCQs' style, difficulty level, and precision of language\n\nFinally, generate new MCQs following these important guidelines:\n- Make sure each incorrect answer maps clearly to exactly one misconception\n- Use precise mathematical language matching the style of reference MCQs\n- Make questions challenging enough that students must demonstrate real understanding\n- Ensure wrong answers are plausible and stem from genuine misconceptions, not careless errors\n- Use the exact wording of misconceptions as given in the misconceptions list\n- Pay careful attention to subtle differences between the misconceptions and observe which one is the most appropriate for a given incorrect answer\n- Keep the construct name and subject name as short as possible hiding the details of the misconception\n- Questions should be of higher difficulty level than reference MCQs\nI used LLM-as-a-judge to filter out low quality synthetic MCQs. I prompted GPT-4o to rate the quality of the synthetic MCQs as below:
You will analyze how well an incorrect answer reflects a suspected misconception in a mathematics problem. Your goal is to determine whether there is a clear, logical connection between the misconception and the wrong answer.\n\nHere is the problem with both correct and incorrect answers. The suspected misconception is also provided:\n<problem>\n{PROBLEM_DATA}\n</problem>\n\nFirst, analyze the problem in your scratchpad:\n<scratchpad>\n1. Solve the problem independently to verify the correct answer\n2. Examine how someone holding the suspected misconception would approach the problem\n3. Trace the logical path from misconception to incorrect answer\n4. Identify any gaps or inconsistencies in this connection\n</scratchpad>\n\nThen provide your evaluation using this format:\n<evaluation>\n1. Brief explanation of how the misconception could lead to the wrong answer\n2. Score from 0-10 based on these criteria:\n - 10: Perfect alignment - wrong answer is direct result of misconception\n - 8-9: Strong alignment - clear logical path from misconception to answer\n - 5-7: Moderate alignment - connection exists but has some gaps\n - 1-4: Weak alignment - connection is unclear or requires assumptions\n - 0: No alignment - misconception does not explain wrong answer\n</evaluation>\n\nImportant guidelines:\n- Focus solely on the logical connection between misconception and wrong answer\n- Do not speculate about other possible misconceptions\n- Be specific about how the misconception leads to the error\n- Flag and deduct scores if any assumptions are required to connect misconception to answer\n- Consider whether a student with this misconception would consistently arrive at this wrong answer\nI selected GPT-4o as judge based on vibe test as it felt more sensitive to the scoring scheme and as consistent as Claude 3.5 Sonnet.
While creating synthetic data, LLM produced many misconceptions (4k+) that were not part of the host provided misconception pool. I decided to keep the external misconceptions to improve generalization capability of the retriever and ranker models.
\nNaively merging the external misconceptions with existing ones would have introduced conflicts and noise, since many of them were just rephrasing of the existing misconceptions. I adopted the following steps to handle this carefully:
\nthenlper/gte-base) with very high threshold (0.995)The final dataset is uploaded here: Eedi - Mining Misconceptions in Mathematics. It contains
\nReference: I'd highly recommend reading the synthetic data generation post from answer.ai: How To T̶r̶a̶i̶n̶ Synthesize Your D̶r̶a̶g̶o̶n̶ Data.
\nTo distill the reasoning capability of an advanced LLM in my solution pipeline, I first created a dataset containing thought processes generated by Claude 3.5 Sonnet. I provided Claude with: (i) the problem statement, (ii) the correct answer, (iii) an incorrect answer, (iv) the ground truth misconception, and (v) several closely related misconceptions. Using this information, I prompted Claude to generate the likely thought process that led a student to select the incorrect answer. The prompt is shown below:
You will analyze a student's incorrect answer to identify the specific reasoning flaw that led to their error.\nYour goal is to explain precisely how their misconception caused them to arrive at the wrong answer.\n\nHere is the problem information:\n<problem_data>\n# Question: Simplify the expression: \\[x \\cdot y \\cdot x\\]\n# Correct Answer: \\(x^2y\\)\n# Incorrect Answer: \\(x^2\\)\n# Primary Misconception: Ignores variables without explicit coefficients when multiplying\n</problem_data>\n\nHere are related misconceptions that are similar but do not explain this specific error as precisely:\n<related_misconceptions>\n- Thinks only like terms can be multiplied\n- Fails to combine all instances of the same variable\n- Incorrectly identifies an incomplete variable factor\n- Does not understand how to multiply algebraic terms\n</related_misconceptions>\n\nFirst, examine all components of the problem carefully:\n1. The problem statement and question asked\n2. The correct answer and solution method\n3. The student's incorrect answer\n4. The primary misconception given\n5. The related misconceptions that should be distinguished from the primary one\n\nThen, reconstruct the student's likely thought process:\n- Identify the exact point where their reasoning diverged from the correct solution path\n- Note which specific mathematical operations or concepts they misapplied\n- Connect their error directly to the stated primary misconception\n- Verify that this explanation better fits the error than the related misconceptions\n\nWrite your analysis in <evaluation> tags, following this structure:\n- Show the correct calculation first\n- Show the incorrect calculations that demonstrate the error\n- Explain the specific flaw in the student's reasoning\n- Demonstrate how the misconception led to this particular error\n- Distinguish from the related misconceptions\n- Keep your explanation to 5-6 clear, non-repetitive sentences\n- Focus solely on the reasoning that produced this specific error\n\nGuidelines for writing your explanation:\n- Do not restate the problem or name the misconception\n- Be precise about the mathematical concepts involved\n- Show exactly how the misconception led to the error\n- Distinguish from related misconceptions\n- Avoid repetition\n- Stay focused on this specific error\nUsing this dataset, I fine-tuned three Qwen 2.5 series models (Qwen/Qwen2.5-Math-7B, Qwen/Qwen2.5-14B, and Qwen/Qwen2.5-32B) to act as reasoners. Notably, I omitted the true and related misconceptions from the input for fine-tuning. During inference, these models would see: (i) the problem statement, (ii) the correct answer, and (iii) an incorrect answer. They would then generate the likely thought process of a student behind selecting the incorrect answer. I used vllm for CoT generation during inference with temperature=0.7, top_p=0.8, repetition_penalty=1.0, max_tokens=256.
The CoT dataset is uploaded here: Eedi - CoT Dataset.
\nI split the competition dataset into 5 folds, initially based on the QuestionId field. However, this produced overly optimistic estimates with a large gap between validation and leaderboard scores. Next, I tried GroupKFold on SubjectId, but this resulted in overly pessimistic estimates. I finally settled on GroupKFold with ConstructId, which yielded a narrow Validation/LB gap and good correlation up to ~0.62 public LB score range.
During development, I always trained my models on folds 1-4 and validated on fold 0. The synthetic data was marked as fold 99 and was included in the training set. The final models were trained on the entire dataset (full fit) using the same hyper parameters from the development stage.
\nI used the standard MultipleNegativesRankingLoss to fine-tune the retrievers, while referencing the FlagEmbedding codebase for developing training scripts. At this stage, I tracked both map@25 and recall@32 metrics and prioritized optimizing for recall@32, since recall was more important for overall pipeline performance.
I fine-tuned several retrievers with the following scores:\n![\"Retriever](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2125251%2Fee7ac74c895fd5221738bf2ae8f42cee%2Fretriever_scores.png?generation=1734199290617188&alt=media\")
Incorporating hard negatives into training batches and distilling re-ranker scores consistently improved map@25 performance, but did not positively impact recall@32. For my final submission selections, I chose the high-recall Qwen/Qwen2.5-14B encoder (Model 3) instead of the best map@25 encoder (Model 4). Following the same logic, I also excluded the BGE encoder (Model 2). Notably, my best encoder-only submission (Model 1 + Model 2 + Model 4) scored 0.524 on public LB and 0.475 on private LB.
Retrievers were fine-tuned using LoRA with: r=64, alpha=128, learning_rate_lora_a=1e-5, learning_rate_lora_b=5e-5, LoRA on all linear layers, batch_size=128, and epochs=12. A few key factors that improved recall performance were:
Qwen/Qwen2.5-14B encoder with all available synthetic data (before the curation step in section 4.1.1).What didn't work:
\nThe Qwen 14B ranker did most of the heavy lifting by processing all retrieved misconceptions (32-64 candidates) and identifying top 8 candidates. This model was trained with a pointwise approach, where it sees one misconception at a time in the context window. The model input was structured as follows:\n![\"Pointwise](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2125251%2Fa64d8d1be0b9fd0d27ddc7c8b47b7644%2Fpointwise_ranker.png?generation=1734199367236460&alt=media\")
The logits were computed by taking the difference between the 'Yes' and 'No' token scores, as below:
\noutputs = self.model(input_ids=input_ids, attention_mask=attention_mask, output_hidden_states=True)\nlogits_yes = outputs.logits[:, -1, self.yes_loc] # Yes token logit at the last position [bs]\nlogits_no = outputs.logits[:, -1, self.no_loc] # No token logit at the last position [bs]\nlogits = logits_yes - logits_no # [bs]\n\nlogits = logits.reshape(-1, self.group_size)\nlabels = labels.to(logits.device).reshape(-1) \nce_loss = self.loss_fn(logits, labels)\nEach batch contained 1 positive and batch_size-1 negative misconceptions for a given Question-Incorrect Answer combination. The label was set to the index of the positive misconception and cross entropy loss was used for training. The models was trained with LoRA with r=64, alpha=128, LoRA on all linear layers. The language modelling head (lm_head) of the LLM (Qwen/Qwen2.5-14B) was frozen during fine-tuning.
I found the following 4 strategies to be particularly significant for boosting the reranker performance and its generalization capability:
\nTo retain and leverage the LLM's in-context learning capabilities, I optionally included a few reference misconception examples (0 to 2) in the model input context (as shown in the diagram above). Not all training examples contained these few-shot demonstrations. The objective was to encourage the model to use reference examples when available, and otherwise rely on its internal reasoning. During inference, I used 1 or 2 examples from the entire training set as demonstrations. This approach was particularly effective for the 14B ranker's performance boosting private LB score from 0.495 to 0.531 (+0.036). At this point, I was only using the competition data for training.
\nNext, I incorporated the synthetic examples (post curation) into the training pipeline through pseudo labeling. I first fine-tuned two pointwise 72B models (Qwen/Qwen2.5-Math-72B and Qwen/Qwen2.5-72B) using only the competition dataset. These models were then used to generate pseudo labels for the synthetic examples. Finally, I used competition + pseudo-labeled data to fine-tune the next iteration of the 14B rerankers. This strategy boosted the 14B reranker's private LB score from 0.531 to 0.575 (+0.044).
Next, I experimented with increasing the number of negative samples per positive. Showing the rankers a large number of negatives during training helped to improve performance. For each positive, I next increased the number of negatives to 24 from earlier 16. At this point, I also added scaled number of synthetic examples 2x. These two modifications boosted the 14B reranker's private LB score from 0.575 to 0.596 (+0.021).
\nI finally added the fine-tuned CoT reasoners (Section 4.2) into training and inference pipeline. During training, CoTs produced from fine-tuned Qwen/Qwen2.5-14B reasoner were optionally added to the model input. 50% of training data used CoTs and 50% did not. This encouraged the model to use external reasoning when available and otherwise rely on its internal reasoning. CoT boosted the 14B reranker's private LB score from 0.596 to 0.615 (+0.019).
The impact of these strategies is summarized in the following table:\n![\"Ablation\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2125251%2F933a9cbaf89521f3873ce3220955d411%2F14b_ablation_updated.png?generation=1734200026698363&alt=media\")
The live eval during fine-tuning showed consistent improvement in the 14B reranker's performance.\n![\"Live](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2125251%2F32caa0b7b996ea5af12050a4dc23f1c9%2Flive_eval_14b_ranker.png?generation=1734199467358905&alt=media\")
Qwen/Qwen2.5-32B)I fine-tuned the Qwen/Qwen2.5-32B model in exactly the same way as the 14B reranker (with a different seed). My best 32B model had validation score of 0.663 vs that of 0.646 for the 14B reranker. During inference, the 32B reranker was used to process top 8 candidates from the 14B reranker and narrow them down to top 5 candidates. Incorporating the 32B reranker boosted private LB score from 0.615 to 0.625 (+0.010).
Qwen/Qwen2.5-72B)The top 5 candidates were finally ranked in a listwise manner using a fine-tuned Qwen/Qwen2.5-72B model. Here, the model sees all of the top 5 candidates at once. An example model input is shown below:\n![\"Listwise\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2125251%2Fd044104a02153e758cbeda0b6914a14d%2Flistwise_model_input.png?generation=1734199518083387&alt=media\")
This model had access to the even more information than pointwise re-rankers i.e.
\nThe model had the best MAP@5 among all of the rerankers (validation MAP@5 of 0.661). LLM based Listwise rankers typically suffer from candidate position bias. One way to mitigate this is to run multiple inference runs by shuffling the position of the candidate misconceptions (Test time augmentation (TTA) like impact). I was unable to run TTA during inference due to time constraints. Incorporating the 72B reranker boosted private LB score from 0.625 to 0.638 (+0.013).
I quantized the fine-tuned rankers using AutoAWQ. I used task specific calibration datasets for each model to mitigate performance degradation. I tested the impact of calibration dataset with one of my 32B rerankers. The results are shown below:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2125251%2F66dae22248f881fdf8a065979695c6e2%2Fcalibration_update.png?generation=1734200937122595&alt=media\")
Qwen/Qwen2.5-32B but got worse results. The QwQ model likely requires additional research to find out its proper usage.Detailed is an understatement!
\nThis is great info to share, Raja. Congrats again!
", "votes": 5, "replies": [] }, { "id": 3154496, "author_name": "Orange", "author_url": "", "post_date": "2025-03-20T02:42:06.973000", "content": "Hi, thanks for sharing such an excellent proposal—it’s a solid RAG framework concept. Here's my understanding of your approach to training the reasoner:
\nMy question is: How do you ensure that the generated text—the CoT content—is accurate? Do you employ a curation step, similar to the one used in the synthesized data process, to filter for high-quality CoTs?
", "votes": 1, "replies": [ { "id": 3155013, "author_name": "Raja Biswas", "author_url": "", "post_date": "2025-03-20T16:05:37.953000", "content": "Hi @youwbpyr, Thanks. Yes, I used the generated query-CoT pairs to distill Claude 3.5's reasoning capability into student models. For this competition, I didn't curate the generated CoTs. However, I think generating multiple CoTs and picking the best one as per a process reward model / llm-as-a-judge could improve student models reasoning abilities.
", "votes": 2, "replies": [] } ] }, { "id": 3077833, "author_name": "Geremie Yeo", "author_url": "", "post_date": "2024-12-21T12:56:21.633000", "content": "Hi Raja, thanks for the detailed solution! Congrats for the solo win🥇
\nWould you mind if I ask a few questions ?
\nGreat job on trying out lots of experiments within a short period of three months. What GPU specifications/clusters did you use to fine tune the models ?
\nContext: I rented H100s from RunPod and finetuning using LoRA for a 14B Yes/No reranker took > 5 hours with our team’s external data. For initial experiments I tested various numbers of hard negatives (1,2,4,8,16,24) and easily training these six models can take more than 2 days total (just for one idea!). Towards the end I finetuned 32B and the training on just 4 negatives took > 20 hours for a single experiment (5 hours without external data)! I trained for 2 epochs using the SFTTrainer. Felt that there may be a faster way…
\nDo you have tips on selecting experiments?
\nContext: From the MCTS and EEDI competitions, I realised that I’m able to think of lots of ideas to solve the problem. However, I felt my ability to decide which one to prioritise needs improvement. For example in EEDI, I spent 3 days customizing TextGrad to optimize prompts for Qwen to hopefully improve the reranking performance (similar to this). However the improvement is very marginal and possibly not worth 3 full days of effort…Maybe I will get better at this as I gain more experience 😅
\nReally inspiring that you got the correct approach from the beginning and have 3 months to get the most out of it 👍
", "votes": 4, "replies": [ { "id": 3078167, "author_name": "Raja Biswas", "author_url": "", "post_date": "2024-12-21T21:54:46.657000", "content": "Hi bogoconic1!
\nThanks a lot! You've been on an impressive path and performing well across different types of competitions!
\n\n\nGreat job on trying out lots of experiments within a short period of three months. What GPU specifications/clusters did you use to fine tune the models ?
\n
I used 2xH100 GPUs from vast.ai. I trained my models using the HF accelerate with a few basic speed optimizations e.g. using (1) torch.compile (2) flash-attention (3) LoRA instead of QLoRA (4) padding sequence lengths to multiples of 8. The 14B re-ranker was also taking ~5hrs for a full run in my setup. But, I very rarely did a full run: (a) ~50% of training was giving good indications on the experiment, (b) I primarily experimented with the 7B models. For 32B, I just copied the best settings from 14B. IMO key is to keep experiment setup as lean as possible.
\n\n\nDo you have tips on selecting experiments?
\n
I often struggle with this too. It's difficult to get the exploration vs exploitation balance right. Although there is no silver bullet, I think the following points help in NLP comps:
\nWill add to this list if I can think of other points tomorrow. I'm not sure if these are too obvious or very opinionated…
", "votes": 10, "replies": [] } ] }, { "id": 3122439, "author_name": "YuhaoWu123", "author_url": "", "post_date": "2025-02-12T16:21:11.237000", "content": "hi～ I would like to ask you another question. In your notebook at https://gist.github.com/rbiswasfc/8d2bfec5c2a358e8beeb2df390111f9d, you used a file named ranker_oof_df.csv. In your solution, you mentioned, “Specifically, I first created clusters of closely related misconceptions leveraging co-occurrence statistics of misconceptions in retriever/ranker predictions on validation data.” My question is, when you refer to “validation data” here, do you mean the predictions on the validation data obtained by training a model using the training set, which is split from the original entire dataset into training and valid sets? thanks a lot！
", "votes": 1, "replies": [ { "id": 3123110, "author_name": "Raja Biswas", "author_url": "", "post_date": "2025-02-13T11:41:31.767000", "content": "Hi @yuhaowu123, yes, exactly. Using top retrieved candidates on the validation split.
", "votes": 0, "replies": [ { "id": 3141956, "author_name": "", "author_url": "", "post_date": "2025-03-06T02:04:19.437000", "content": "", "votes": 0, "replies": [] } ] } ] }, { "id": 3081076, "author_name": "CodeCavalier", "author_url": "", "post_date": "2024-12-26T07:06:16.510000", "content": "Superb, good luck in your future ventures!
", "votes": 1, "replies": [] }, { "id": 3078878, "author_name": "Miles", "author_url": "", "post_date": "2024-12-23T00:42:09.683000", "content": "Congrats for the winning! Did you try to train the 14b reranker with synthetic data but without distillation from 72b? I wonder how it would perform.
", "votes": 1, "replies": [ { "id": 3078984, "author_name": "Raja Biswas", "author_url": "", "post_date": "2024-12-23T04:54:24.810000", "content": "Interesting, this should be a good experiment! However, I didn't try this thinking synthetic data misconceptions still may be noisy (lesser quality than competition data) even after curation…
", "votes": 0, "replies": [] } ] }, { "id": 3078557, "author_name": "YuhaoWu123", "author_url": "", "post_date": "2024-12-22T13:49:41.320000", "content": "Thank you very much for your insights! This is a very interesting and solid solution. However, I have a few questions regarding the prompt design that I’d like to clarify:
\nYou mentioned that the misconception you provided are very similar, but for a single question (Q), the three incorrect answers (A) might correspond to very different types of misconception. In that case, does each reference correspond to one QA pair, or does one Q correspond to three different A options ?
The prompt does not explicitly mention that the types of misconception in the generated samples can be new. However, it does state that you need to first identify which misconception need to be leverage, which implies selecting misconception from those provided that were not covered by the reference questions. If this is the case, how can we ensure that new types of misconception will appear in the constructed dataset?
The prompt mentions that the tag needs to indicate which misconception each incorrect answer corresponds to. Does this mean that the reference questions only provide the incorrect answers without providing the misconception？
Many of my above questions stem from uncertainty about the content within the tag and the tag in a complete prompt. Would you be able to provide me with an example of a fully constructed prompt including these tags?Thanks a lot again！~~~
Here is an example prompt:
\nYou will be generating Multiple Choice Questions (MCQs) diagnose specific mathematical misconceptions. Here are the misconceptions you should focus on:\n\n<misconceptions>\n- Thinks the perimeter a square grid the number squares surrounding the rectangle\n- Calculates area asked perimeter\n- Counts a side length more than once counting perimeter due marking a starting point\n- Thinks the perimeter a square grid the number squares inside are touching the perimeter the shape\n- Calculates perimeter asked area\n- When calculating perimeter, only adds two sides\n- Believes lines inside the shape are included finding perimeter\n- When calculating perimeter, only adds the marked lengths\n- When calculating perimeter, multiplies instead adds\n- Thinks only labelled dimensions need be included calculating the perimeter\n- Confuses perimeter area\n- Fails use the properties simple shapes work out missing lengths\n- Does know we can use the conservation perimeter find missing lengths\n- Makes an assumption about line segments being equal within a shape\n- Mixes up width height a rectangle\n- Does know how find missing lengths a composite shape\n</misconceptions>\n\nHere are reference MCQs demonstrate how effectively diagnose these misconceptions:\n\n<reference_mcqs>\n[{: , : , : , : , : , : , : , : , : , : , : , : }, {: , : , : , : , : , : , : , : , : , : , : , : }, {: , : , : , : , : , : , : , : , : , : , : , : }, {: , : , : , : , : , : , : , : , : , : , : , : }, {: , : , : , : , : , : , : , : , : , : , : , : }]\n</reference_mcqs>\n\nYour task generate MCQs diagnose misconceptions already covered the reference MCQs.\n\nFirst, analyze the reference MCQs carefully:\n For each reference MCQ, identify your <analysis> tags:\n - Which misconception targets\n - How the incorrect answers specific misconceptions\n - What makes the question effective at diagnosing the misconception\n Note the style, difficulty level, precision language used\n\nThen, your <planning> tags:\n- List which misconceptions still need coverage\n- For each needed misconception, brainstorm mathematical contexts where commonly appears\n- Design questions where the misconception leads naturally specific wrong answers\n- Take notes how you can craft MCQs adheres the reference MCQs\n\nYou mentioned that the misconception you provided are very similar, but for a single question (Q), the three incorrect answers (A) might correspond to very different types of misconception. In that case, does each reference correspond to one QA pair, or does one Q correspond to three different A options ?
\n
For each MCQ, the LLM would generate
\n \n \n \n \n \n \n \n \n \n \n \n \ntherefore, incorrect answers and misconceptions can be paired properly.
\n\n\nThe prompt does not explicitly mention that the types of misconception in the generated samples can be new. However, it does state that you need to first identify which misconception need to be leverage, which implies selecting misconception from those provided that were not covered by the reference questions. If this is the case, how can we ensure that new types of misconception will appear in the constructed dataset?
\n
Nice spot. Even if we specify the possible misconceptions, LLM still likes to generate additional ones. Before I settled on this prompt, my earlier prompts would generate even more of new misconceptions.
\n\n\nThe prompt mentions that the tag needs to indicate which misconception each incorrect answer corresponds to. Does this mean that the reference questions only provide the incorrect answers without providing the misconception？
\n
I have provided a full prompt example here. Hope it helps!
", "votes": 0, "replies": [ { "id": 3140911, "author_name": "lixianyan11", "author_url": "", "post_date": "2025-03-05T03:58:35.747000", "content": "Do the misconceptions generated by the LLM, such as misconception_a_name, misconception_b_name, misconception_c_name, and misconception_d_name, tend to choose from the given misconception pool of the cluster?
", "votes": 0, "replies": [] } ] }, { "id": 3140906, "author_name": "lixianyan11", "author_url": "", "post_date": "2025-03-05T03:48:03.497000", "content": "I would like to ask: If only the misconceptions are provided as clusters, where do the other answers and corresponding misconceptions generated by the LLM for a new question come from?
", "votes": 0, "replies": [] } ] }, { "id": 3077562, "author_name": "Yannan Chen", "author_url": "", "post_date": "2024-12-21T05:39:43.933000", "content": "Thank you so much for this detailed solution. I'm so impressed with how much attention you've paid to details, and how you fully utilze the limited GPU resources to perform so many rounds of reranking. May I ask:
\nThanks again! Congratulations on 1st place! The solution really deserves it.
", "votes": 1, "replies": [ { "id": 3077760, "author_name": "Raja Biswas", "author_url": "", "post_date": "2024-12-21T11:19:04.580000", "content": "Thanks for the kind words! Congratulations to you too on solo gold! 🎉
\n\n\nyou mentioned \"add recovery adapters\" to quantization, what is it? did you apply it eventually?
\n
The idea comes from the Apple Intelligence report (Section 5.2).
\n\n\n… The quantized model often shows a moderate level of quality loss. Therefore, instead of directly passing the quantized model to application teams for feature development, we attach a set of parameter-efficient LoRA adapters for quality recovery. We make sure that these LoRA adapters training recipes are consistent with pre-training and post-training processes.
\n
I didn't get to try this in this competition.
\n\n\nWhat are the important tricks to fit 72b model for inference into 2 T4 GPUs?
\n
I mainly followed this great post from @jagatkiran! Didn't have any inference tricks for the 72b model. For 14b ranker, prefix caching was key.
\n\n\nWhat total running time did you manange to achieve during competition? Any out of time or out of memory experienced near the end of the competition?
\n
My best selected submission was ~510mins. I didn't face OOM but had a few out of time near the end…
\nThanks!
", "votes": 2, "replies": [ { "id": 3078220, "author_name": "Yannan Chen", "author_url": "", "post_date": "2024-12-22T00:36:24.273000", "content": "Thank you for your explanation! I also noticed that for your retriever, you used different learning rate for lora_a and lora _b, this is an interesting detail. May I ask what is the rationale behind that?
", "votes": 1, "replies": [] }, { "id": 3078977, "author_name": "Raja Biswas", "author_url": "", "post_date": "2024-12-23T04:48:38.020000", "content": "I followed the LoRA+ paper: LoRA+: Efficient Low Rank Adaptation of Large Models -- which shows faster convergence and better perf with different LRs for LoRA A and B.
\n\n", "votes": 1, "replies": [] } ] } ] }, { "id": 3073545, "author_name": "xyzlsy", "author_url": "", "post_date": "2024-12-16T15:42:50.033000", "content": "…Low Rank Adaptation (LoRA) as originally introduced in Hu et al. (2021) leads to suboptimal finetuning of models with large width (embedding dimension). This is due to the fact that adapter matrices A and B in LoRA are updated with the same learning rate. Using scaling arguments for large width networks, we demonstrate that using the same learning rate for A and B does not allow efficient feature learning. We then show that this suboptimality of LoRA can be corrected simply by setting different learning rates for the LoRA adapter matrices A and B with a well-chosen ratio. We call this proposed algorithm LoRA+. In our extensive experiments, LoRA+ improves performance (1-2 % improvements) and finetuning speed (up to ∼ 2X SpeedUp), at the same computational cost as LoRA.
\n
I want to know if your recall model is created by combining model 1, model 2, and model 3, and then concatenating the last token, or if another method is used.
", "votes": 1, "replies": [ { "id": 3074399, "author_name": "Raja Biswas", "author_url": "", "post_date": "2024-12-17T16:22:45.953000", "content": "I averaged similarity scores from these models.
", "votes": 0, "replies": [] } ] }, { "id": 3073495, "author_name": "xyzlsy", "author_url": "", "post_date": "2024-12-16T14:53:16.193000", "content": "Did you merge the training set and the synthetic dataset together and then perform the 5-fold split, or did you perform the 5-fold split on the training set and then directly add the synthetic data to the training set?
", "votes": 1, "replies": [ { "id": 3073504, "author_name": "Raja Biswas", "author_url": "", "post_date": "2024-12-16T15:00:42.240000", "content": "5-fold split on the training set and then directly add the synthetic data to the training set
", "votes": 1, "replies": [] } ] }, { "id": 3073391, "author_name": "Khairul Islam25", "author_url": "", "post_date": "2024-12-16T13:18:32.093000", "content": "Your explanation is excellent bro, Thanks.
", "votes": 1, "replies": [] }, { "id": 3073360, "author_name": "Santiago Mota", "author_url": "", "post_date": "2024-12-16T11:59:14.937000", "content": "Congrats both. Result ans explication.
", "votes": 1, "replies": [] }, { "id": 3072882, "author_name": "MrSimple", "author_url": "", "post_date": "2024-12-15T18:00:13.827000", "content": "This is undoubtedly very insightful! Thank you for sharing this amazing information! @conjuring92
", "votes": 1, "replies": [] }, { "id": 3072696, "author_name": "Bootstrapping", "author_url": "", "post_date": "2024-12-15T15:00:31.440000", "content": "Thanks for sharing such a detailed solution!
", "votes": 1, "replies": [] }, { "id": 3072550, "author_name": "Andrey Galichin", "author_url": "", "post_date": "2024-12-15T11:36:55.657000", "content": "A very insightful and beautiful solution, thank you!
\nWe have also tried to generate Chain-of-Thought using gpt-4o in the early stage of the competition, but found that we couldn't correctly formalize the way of thinking for the model, and quite often the model explicitly mentioned the misconception in its response (although we've tried to encourage it not to do so).
\nHaven't you met this problem during the prompt construction; if yes, can you please tell, what exact prompting modifications helped to solve it?
", "votes": 1, "replies": [ { "id": 3073139, "author_name": "Raja Biswas", "author_url": "", "post_date": "2024-12-16T05:10:59.153000", "content": "Yes, I too noticed Claude mentioning the misconception name at the start of CoT. I fixed it with prompt engineering by asking LLM to follow this sequence:
\nI see a lot of people used the LM Head with Yes/No probabilities as the binary classifier. Any thoughts on whether that is the best approach when fine tuning, or is it better to add a new classifier head? I tried both and the classifier head worked better for me, but it was also my first time doing any serious comparision between these two approaches. Appreciate if you share your thoughts. Thanks also for such an immensely detailed post, learnt a lot.
", "votes": 1, "replies": [ { "id": 3073138, "author_name": "Raja Biswas", "author_url": "", "post_date": "2024-12-16T05:08:13.313000", "content": "Thanks Dipam!
\nI actually trained most of my models with a new classification head and switched to Yes/No tokens approach in the later stages -- mainly to take advantage of vllm inference + in-built prefix caching. Both Yes/No approach and classifier head gave similar scores here.
", "votes": 3, "replies": [ { "id": 3078523, "author_name": "Atharva Ingle", "author_url": "", "post_date": "2024-12-22T12:32:24.967000", "content": "Thanks for the detailed writeup, a lot to learn here. I am a bit confused in this part:
\nQuestion: Since you are finetuning the causal lm to output Yes/No token, why the lm_head was frozen. What was the reasoning behind this ? As we want the lm_head to have high probs for either Yes/No token, freezing it will avoid it to learn this mapping. Can you explain this part ?
\n\n\nThe logits were computed by taking the difference between the 'Yes' and 'No' token scores, as below:
\n
outputs = .model(input_ids=input_ids, attention_mask=attention_mask, output_hidden_states=)\nlogits_yes = outputs.logits[:, -, .yes_loc] \nlogits_no = outputs.logits[:, -, .no_loc] \nlogits = logits_yes - logits_no \nlogits = logits.reshape(-, .group_size)\nlabels = labels.to(logits.device).reshape(-) \nce_loss = .loss_fn(logits, labels)\n\n", "votes": 1, "replies": [] }, { "id": 3079011, "author_name": "Raja Biswas", "author_url": "", "post_date": "2024-12-23T05:59:35.763000", "content": "Each batch contained 1 positive and batch_size-1 negative misconceptions for a given Question-Incorrect Answer combination. The label was set to the index of the positive misconception and cross entropy loss was used for training. The models was trained with LoRA with r=64, alpha=128, LoRA on all linear layers. The language modelling head (lm_head) of the LLM (Qwen/Qwen2.5-14B) was frozen during fine-tuning.
\n
Hey @atharvaingle, on freezing the lm_head:
logit(y=Yes) = features . W_lm_head_yes : The Yes/No token logits are controlled by both the LM head weights at the location of Yes/No tokens and model hidden states at end of the sequence. Even if we freeze the LM head, the logits can be adjusted by the network by updating other params such that features are moving as per loss during fine-tuning.Thanks this helps a lot.
", "votes": 1, "replies": [] }, { "id": 3086113, "author_name": "Chenyu Lin", "author_url": "", "post_date": "2025-01-02T02:26:49.143000", "content": "If possible, could you show me which part of the code freezes the weights for this section? I couldn’t find it. Thank you very much.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3072329, "author_name": "James owen", "author_url": "", "post_date": "2024-12-15T03:31:27.977000", "content": "Hi raja, I have carefully read the detailed plan you posted. I have truly learned a lot. As a beginner in LLM, your plan involves a lot of knowledge that I have not yet understood. Please allow me to express my sincerest gratitude to you and look forward to your performance in the next competition
", "votes": 1, "replies": [] }, { "id": 3074742, "author_name": "Lao Xiaodong", "author_url": "", "post_date": "2024-12-18T02:40:30.673000", "content": "Looking forward to the release of your training code.
", "votes": 2, "replies": [ { "id": 3077764, "author_name": "Raja Biswas", "author_url": "", "post_date": "2024-12-21T11:20:42.470000", "content": "Currently working on the code release, so sorry for the delay…
", "votes": 0, "replies": [] }, { "id": 3080817, "author_name": "Raja Biswas", "author_url": "", "post_date": "2024-12-25T19:16:04.187000", "content": "published the code here: https://github.com/rbiswasfc/eedi-mining-misconceptions
", "votes": 1, "replies": [] } ] }, { "id": 3073598, "author_name": "Nikhil Tumbde", "author_url": "", "post_date": "2024-12-16T16:30:47.543000", "content": "Hi @conjuring92, thank you so much for the detailed explanation. It is quite a lot to unpack.
\nIn my efforts towards learning, I am trying to summarize my understanding of your approach. However, I have a few questions and would really appreciate if you (or anyone on the forum) can help me get more clarity.
\n", "votes": 2, "replies": [ { "id": 3074397, "author_name": "Raja Biswas", "author_url": "", "post_date": "2024-12-17T16:22:09.333000", "content": "Summary
\n1. Data Generation
\na. Create clusters out of similar misconceptions
\n
\n b. Generate 5 MCQs (using Claude 3.5 sonnet) for each clusters by giving cluster misconceptions and related 4-5 questions as context. (data quality can be questionable)
\n c. Use GPT-4o to evaluate the quality of generated MCQs on a scale from 0-10 and filter out the bad ones.
\n d. Step b gave additional misconceptions (not in misconception_mapping). Filtered out the misconceptions similar to the ones in the given mapping to avoid conflicts (using vectors from embedding model).
\n At this point the author has original misc clusters, and generated MCQs with misconceptions mapped to them (both original and generated)2. COT Distillation
\nOverall, the idea here is to distill the knowledge of claude into smaller models. So we get generations from claude and use that to finetune smaller models.
\n
\n a. Generate COT (on original train dataset) using Claude using the question context, misc and related misc in the prompt.
\n b. Finetune 3 models (7,14,32B) on the generated text and same prompt but exclude the misconceptions and related misc (as during inference they won’t be present)
\n At this point the author has 3 models (reasoners) with the capability of generating COT reasons for each incorrect answer using only the questions context (no misconception).3. Models - retriever
\na. Finetune retrievers using MNR loss for 12 epochs on entire data (except 1 fold) from step 1.d. {some confusion here will update once it is clarified}
\n
\n b. Pre-training the finetuned qwen-14B on synthetic data from step 1.b gave recall boost.4. Model –reranker
\noverall 2 pointwise rerankers (yes/no output for each misconception) and 1 listwise used.
\na. Pointwise rerankers (14/32B)
\ni. Pseudo labels on synthetic dataset (from 1.d) generated using models (Qwen/Qwen2.5-Math-72B and Qwen/Qwen2.5-72B) finetuned on train dataset.
\n
\n ii. Entire dataset used for finetuning pointwise reranker. COT was also added to 50% of the training dataset.
\n iii. Other details-\n
\n- The prompt contains few shot examples.
\n- The re-ranker was trained on logit difference of ‘yes’ and ‘no’ tokens as predictions using cross entropy loss.
\n- 1 positive and (BS – 1) negative samples taken for each batch. 1:24 positive:negative ratio.
\n- LM head was frozen during finetuning.
\nb. Listwise reranker
\ni. Predicts 1 out of top 5 choices.
\n
\n ii. {Training strategy not shared}5. Quantization
\na. Fp16 finetuning was performed and the models were quantized using AWQ.
\n
\n b. Calibration dataset used {some confusion here will update once it is clarified.}
Thanks Nikhil for the questions. Please find below my replies.
\n\n\nHow many negative samples were used for retriever finetuning?
\n
I used the standard in-batch negatives setup. Therefore, for a batch_size of 128, there were 127 negatives and 1 positive for each query (question + incorrect answer).
\n\n\nCan you elaborate on “Ensuring only one demonstration per misconception appeared in each training batch, as multiple demonstrations would introduce label noise through in-batch negatives”. Do you mean 1 misconception won’t appear as true label twice in a batch?
\n
Say, In a batch, the positive examples are (q1, m1), (q2, m2)..., (q128, m128). For the in-batch negatives setup, this means q1 will have m1 as positive and [m2, m3, ..m128] as negatives. Similarly, for q2 you will have m2 as positive and [m1, m3, ...m128] as negatives. During training, we will compute similarity scores between all query and misconception combinations (128x128 matrix). The diagonal elements would correspond to the positives and off-diagonals should be negatives. In this competition, we had many-to-one relationship for the query to content mapping. Hence, a random batching strategy would lead to cases where multiple queries belonging to the same misconception would end up in a given batch. In this case, you will have false negatives in the off diagonals -- leading to noisy, sub optimal convergence.
\n\nHow were the negative samples obtained for retriever models 4/ 3? Using a pretrained embedding model?
\n
I just used in-batch negatives. I, however, experimented mined hard negatives obtained from fine-tuned retrievers (+ de-noised with re-ranker predictions Reference), but it was ineffective to improve long tail recall.
\n\n\nCan you explain the re-ranker distillation part?
\n
You can refer to the distillation loss in the FlagEmbedding repo.
\n\n\nAWQ - which datasets were used for calibration?
\n
I basically used the a subset of re-ranker training dataset as calibration dataset.
", "votes": 3, "replies": [ { "id": 3074759, "author_name": "Nikhil Tumbde", "author_url": "", "post_date": "2024-12-18T03:15:24.390000", "content": "Thanks a lot for the response! That helps.
", "votes": 1, "replies": [] }, { "id": 3182330, "author_name": "White_Ink", "author_url": "", "post_date": "2025-04-19T06:04:40.330000", "content": "Thanks a lot for the response! That helps.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3072154, "author_name": "Saurabh Badole", "author_url": "", "post_date": "2024-12-14T19:09:44.270000", "content": "This is undoubtedly very insightful! Thank you for sharing this amazing information! @conjuring92
", "votes": 2, "replies": [] }, { "id": 3121905, "author_name": "YuhaoWu123", "author_url": "", "post_date": "2025-02-12T05:06:11.393000", "content": "Hello, I’d like to ask another question. The problem ultimately uses the MAP@K of the correct answers, but your code calculates the MAP@K for all the top K answers. I’m curious why the metric in the code is inconsistent with the metric required by the problem. the code i mentioned is ⬇️
\n for i, p in enumerate(predicted):
\n # first condition checks whether it is valid prediction
\n # second condition checks if prediction is not repeated
\n if p in actual and p not in predicted[:i]:
\n num_hits += 1.0
\n score += num_hits / (i + 1.0)
/ (len(actual), k)\nlooking forward for your reply，thanks a lot！
", "votes": 0, "replies": [ { "id": 3123112, "author_name": "Raja Biswas", "author_url": "", "post_date": "2025-02-13T11:42:27.030000", "content": "Metric computation looks okay to me. Here is a detailed post on the metric: https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/534077
", "votes": 0, "replies": [] } ] }, { "id": 3087919, "author_name": "Joaquin Robador", "author_url": "", "post_date": "2025-01-04T04:38:36", "content": "GOAT. Amazing
", "votes": 0, "replies": [] }, { "id": 3081241, "author_name": "未来鸟@", "author_url": "", "post_date": "2024-12-26T13:46:33.623000", "content": "RAJA BISWAS, I am very grateful that you have published the training code！ I would like to know how long it took you to install the flash_attn package. It has been almost an hour since I ran pip install \"flash_attn==2.6.3\" --no-build-isolation, and it still hasn't completed the installation.
", "votes": 0, "replies": [ { "id": 3081264, "author_name": "Raja Biswas", "author_url": "", "post_date": "2024-12-26T14:14:33.623000", "content": "that's strange, it was ~3mins iirc, please try: pip install flash-attn --no-build-isolation
Three minutes is really fast! I'm happy that I finally managed to install it, but it took me almost an hour. This might be because I used a VPN, as I can't directly connect to vast.ai from my country.
", "votes": 0, "replies": [] }, { "id": 3081606, "author_name": "未来鸟@", "author_url": "", "post_date": "2024-12-27T01:52:17.697000", "content": "Hello RAJA BISWAS,
\nI would like to know if the input_dataset in the configuration file conf_qwen14b_pretrain was named eedi-embed-pretrain-mix-final instead of eedi-embed-pretrain-mix-expanded when you trained Qwen/Qwen2.5-14B on the synthetic dataset. I couldn't find the file eedi-embed-pretrain-mix-expanded on Kaggle.
\nThank you!
", "votes": 0, "replies": [] }, { "id": 3081692, "author_name": "Raja Biswas", "author_url": "", "post_date": "2024-12-27T04:52:03.260000", "content": "Thanks for the catch! It should be conjuring92/eedi-embed-pretrain-mix-final. I just pushed the fix.
Please feel free to create an issue in the repo if you encounter other issues!
", "votes": 0, "replies": [] }, { "id": 3133227, "author_name": "未来鸟@", "author_url": "", "post_date": "2025-02-25T06:13:30.100000", "content": "I would like to know if you can provide the calibration datasets for quantizing a fine-tuned model using AWQ, such as rbiswasfc/eedi-awq-calibration, rbiswasfc/eedi-awq-calibration-tutor, and rbiswasfc/eedi-awq-calibration-cot
", "votes": 0, "replies": [] }, { "id": 3133584, "author_name": "Raja Biswas", "author_url": "", "post_date": "2025-02-25T14:07:41.307000", "content": "Here are the datasets:
\nMany thanks！
", "votes": 0, "replies": [] } ] } ] }, { "id": 3078935, "author_name": "Miles", "author_url": "", "post_date": "2024-12-23T03:09:18.463000", "content": "How was it achieved to inference 72b model on 2 T4 GPUs? They are 2*16=32GB Vram. AWQ of 72b takes 36GB vram on parameters only. And if offload to CPU, it would be so slow as not able to complete the inference within time.
", "votes": 0, "replies": [] }, { "id": 3078921, "author_name": "yanqiangmiffy", "author_url": "", "post_date": "2024-12-23T02:40:53.943000", "content": "Looking forward to training code
", "votes": 0, "replies": [ { "id": 3080820, "author_name": "Raja Biswas", "author_url": "", "post_date": "2024-12-25T19:17:34.867000", "content": "Just pushed the code here.
", "votes": 2, "replies": [] } ] }, { "id": 3074526, "author_name": "Amardeep Singh", "author_url": "", "post_date": "2024-12-17T18:56:14.323000", "content": "Your explanation is excellent bro, Thanks.
", "votes": 0, "replies": [] }, { "id": 3142482, "author_name": "", "author_url": "", "post_date": "2025-03-06T12:36:23.377000", "content": "", "votes": 0, "replies": [] }, { "id": 3141983, "author_name": "", "author_url": "", "post_date": "2025-03-06T02:42:58.807000", "content": "", "votes": 0, "replies": [ { "id": 3141984, "author_name": "", "author_url": "", "post_date": "2025-03-06T02:43:33.153000", "content": "", "votes": 0, "replies": [] }, { "id": 3142013, "author_name": "", "author_url": "", "post_date": "2025-03-06T03:19:58.493000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 3078331, "author_name": "", "author_url": "", "post_date": "2024-12-22T07:02:24.763000", "content": "", "votes": 0, "replies": [ { "id": 3101726, "author_name": "James KJ ", "author_url": "", "post_date": "2025-01-21T08:15:49.537000", "content": "Hi, did you get the csv file ?
", "votes": 1, "replies": [] } ] }, { "id": 3077461, "author_name": "", "author_url": "", "post_date": "2024-12-21T02:03:29.063000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 67, "messages": [ { "id": 3072128, "postDate": "2024-12-14T18:18:44.017Z", "content": "I would like to thank Kaggle and EEDI for hosting this interesting competition! It was challenging and provided many opportunities to learn/apply new techniques. As always, I'm grateful to the Kaggle community for innovative ideas and engaging discussions. I'm excited to share my detailed solution and hope others will find it helpful!
\nGiven a diagnostic math MCQ, together with the correct answer and an incorrect answer, the task was to recommend top 25 misconceptions (sorted by their affinity to the incorrect answer) from a pool of 2.5k+ misconceptions. \nEven though the standard retrieve-rerank framework was a natural fit, it came with a few challenges:
\nNovice Learner i.e. generating the incorrect answer stemming from a specific misconception or (2) an Expert Tutor i.e. identifying the misconception that explains a given incorrect answer. LLMs excel at solving math problems, but they are not so great at counterfactual reasoning.\n\nInitial efforts to use pre-trained language models have not been successful, likely due to the complexity of the mathematical content in the questions.
\n
At the start of the competition, I had the following hypotheses on what could be important for a good solution:
\nThese hypotheses guided me to design my experiments throughout the competition and helped me to stick to an idea even though the initial attempts were not successful.
\nI implemented a retrieve-and-rerank system that involved a sequence of four steps:
\ntop candidate similarity score - constant. Candidates with similarity scores higher than the dynamic threshold are kept. Qwen/Qwen2.5-14B ranker to process all retrieved candidates and shortlist top 8.Qwen/Qwen2.5-32B ranker to process top 8 candidates and narrow them down to top 5.Qwen/Qwen2.5-72B ranker to finalize the ranking of top 5 candidates.A schematic diagram of the pipeline and the leaderboard scores from each stages are shown below:\n
The final 25 misconception predictions comprised of:
\nThe training data mix for both the retriever and ranker models included competition data (1.8k examples) and synthetic data (10k examples). The synthetic data played a crucial role in improving both raw performance and generalization capability with respect to unseen misconceptions.
\nIn an ideal scenario, synthetic MCQs should have the following properties:
\nI started generating diagnostic MCQs with a simple prompt that specified the expected format and a misconception of interest. However, it resulted in a dataset that violated almost all the properties to varying degrees. It was expected since
\nI experimented with several strategies to address the issues and managed to create a high quality synthetic dataset with a grouped synthetic data generation approach and incorporating LLM based filtering. Specifically, I first created clusters of closely related misconceptions leveraging co-occurrence statistics of misconceptions in retriever/ranker predictions on validation data. This notebook demonstrates the clustering process. Here is an example of a misconception cluster:
- Thinks x^2 - a^2x is a valid form of difference of two squares\n- When factorising a quadratic without a non variable term, tries to double bracket factorise\n- Incorrectly factorises a quadratic\n- Believes the constant in an expanded quadratic comes from adding the two numbers in the brackets\n- Believes the coefficent of x in an expanded quadratic comes from multiplying the two numbers in the brackets\n- Does not recognise difference of two squares\n- Does not realise a quadratic must be in the form ax^2+bx+c=0 to be factorised\n- Does not know how to identify common factors from algebraic terms\n- Believes the difference of 2 squares method of factorising also works for the addition of 2 squares\n- When factorising, believes they can choose different factors for each term\n- Believes they can factorise a difference of two squares by placing the constant in both brackets without square rooting\nI next generated 5 new MCQs for each cluster using Claude 3.5 Sonnet with a refined prompt that included 4-5 reference examples on misconceptions from the cluster. I used the metaprompt tool from Anthropic cookbook for prompt engineering. Here is the final prompt used for data generation:
You will be generating Multiple Choice Questions (MCQs) that diagnose specific mathematical misconceptions. Here are the misconceptions you should focus on:\n\n<misconceptions>\n{cluster_misconceptions}\n</misconceptions>\n\nHere are reference MCQs that demonstrate how to effectively diagnose these misconceptions:\n\n<reference_mcqs>\n{reference_mcqs}\n</reference_mcqs>\n\nYour task is to generate {num_mcqs} new MCQs that diagnose misconceptions not already covered by the reference MCQs.\n\nFirst, analyze the reference MCQs carefully:\n1. For each reference MCQ, identify in your <analysis> tags:\n - Which misconception it targets\n - How the incorrect answers map to specific misconceptions\n - What makes the question effective at diagnosing the misconception\n2. Note the style, difficulty level, and precision of language used\n\nThen, in your <planning> tags:\n- List which misconceptions still need coverage\n- For each needed misconception, brainstorm mathematical contexts where it commonly appears\n- Design questions where the misconception leads naturally to specific wrong answers\n- Take notes on how you can craft new MCQs that adheres to the reference MCQs' style, difficulty level, and precision of language\n\nFinally, generate new MCQs following these important guidelines:\n- Make sure each incorrect answer maps clearly to exactly one misconception\n- Use precise mathematical language matching the style of reference MCQs\n- Make questions challenging enough that students must demonstrate real understanding\n- Ensure wrong answers are plausible and stem from genuine misconceptions, not careless errors\n- Use the exact wording of misconceptions as given in the misconceptions list\n- Pay careful attention to subtle differences between the misconceptions and observe which one is the most appropriate for a given incorrect answer\n- Keep the construct name and subject name as short as possible hiding the details of the misconception\n- Questions should be of higher difficulty level than reference MCQs\nI used LLM-as-a-judge to filter out low quality synthetic MCQs. I prompted GPT-4o to rate the quality of the synthetic MCQs as below:
You will analyze how well an incorrect answer reflects a suspected misconception in a mathematics problem. Your goal is to determine whether there is a clear, logical connection between the misconception and the wrong answer.\n\nHere is the problem with both correct and incorrect answers. The suspected misconception is also provided:\n<problem>\n{PROBLEM_DATA}\n</problem>\n\nFirst, analyze the problem in your scratchpad:\n<scratchpad>\n1. Solve the problem independently to verify the correct answer\n2. Examine how someone holding the suspected misconception would approach the problem\n3. Trace the logical path from misconception to incorrect answer\n4. Identify any gaps or inconsistencies in this connection\n</scratchpad>\n\nThen provide your evaluation using this format:\n<evaluation>\n1. Brief explanation of how the misconception could lead to the wrong answer\n2. Score from 0-10 based on these criteria:\n - 10: Perfect alignment - wrong answer is direct result of misconception\n - 8-9: Strong alignment - clear logical path from misconception to answer\n - 5-7: Moderate alignment - connection exists but has some gaps\n - 1-4: Weak alignment - connection is unclear or requires assumptions\n - 0: No alignment - misconception does not explain wrong answer\n</evaluation>\n\nImportant guidelines:\n- Focus solely on the logical connection between misconception and wrong answer\n- Do not speculate about other possible misconceptions\n- Be specific about how the misconception leads to the error\n- Flag and deduct scores if any assumptions are required to connect misconception to answer\n- Consider whether a student with this misconception would consistently arrive at this wrong answer\nI selected GPT-4o as judge based on vibe test as it felt more sensitive to the scoring scheme and as consistent as Claude 3.5 Sonnet.
While creating synthetic data, LLM produced many misconceptions (4k+) that were not part of the host provided misconception pool. I decided to keep the external misconceptions to improve generalization capability of the retriever and ranker models.
\nNaively merging the external misconceptions with existing ones would have introduced conflicts and noise, since many of them were just rephrasing of the existing misconceptions. I adopted the following steps to handle this carefully:
\nthenlper/gte-base) with very high threshold (0.995)The final dataset is uploaded here: Eedi - Mining Misconceptions in Mathematics. It contains
\nReference: I'd highly recommend reading the synthetic data generation post from answer.ai: How To T̶r̶a̶i̶n̶ Synthesize Your D̶r̶a̶g̶o̶n̶ Data.
\nTo distill the reasoning capability of an advanced LLM in my solution pipeline, I first created a dataset containing thought processes generated by Claude 3.5 Sonnet. I provided Claude with: (i) the problem statement, (ii) the correct answer, (iii) an incorrect answer, (iv) the ground truth misconception, and (v) several closely related misconceptions. Using this information, I prompted Claude to generate the likely thought process that led a student to select the incorrect answer. The prompt is shown below:
You will analyze a student's incorrect answer to identify the specific reasoning flaw that led to their error.\nYour goal is to explain precisely how their misconception caused them to arrive at the wrong answer.\n\nHere is the problem information:\n<problem_data>\n# Question: Simplify the expression: \\[x \\cdot y \\cdot x\\]\n# Correct Answer: \\(x^2y\\)\n# Incorrect Answer: \\(x^2\\)\n# Primary Misconception: Ignores variables without explicit coefficients when multiplying\n</problem_data>\n\nHere are related misconceptions that are similar but do not explain this specific error as precisely:\n<related_misconceptions>\n- Thinks only like terms can be multiplied\n- Fails to combine all instances of the same variable\n- Incorrectly identifies an incomplete variable factor\n- Does not understand how to multiply algebraic terms\n</related_misconceptions>\n\nFirst, examine all components of the problem carefully:\n1. The problem statement and question asked\n2. The correct answer and solution method\n3. The student's incorrect answer\n4. The primary misconception given\n5. The related misconceptions that should be distinguished from the primary one\n\nThen, reconstruct the student's likely thought process:\n- Identify the exact point where their reasoning diverged from the correct solution path\n- Note which specific mathematical operations or concepts they misapplied\n- Connect their error directly to the stated primary misconception\n- Verify that this explanation better fits the error than the related misconceptions\n\nWrite your analysis in <evaluation> tags, following this structure:\n- Show the correct calculation first\n- Show the incorrect calculations that demonstrate the error\n- Explain the specific flaw in the student's reasoning\n- Demonstrate how the misconception led to this particular error\n- Distinguish from the related misconceptions\n- Keep your explanation to 5-6 clear, non-repetitive sentences\n- Focus solely on the reasoning that produced this specific error\n\nGuidelines for writing your explanation:\n- Do not restate the problem or name the misconception\n- Be precise about the mathematical concepts involved\n- Show exactly how the misconception led to the error\n- Distinguish from related misconceptions\n- Avoid repetition\n- Stay focused on this specific error\nUsing this dataset, I fine-tuned three Qwen 2.5 series models (Qwen/Qwen2.5-Math-7B, Qwen/Qwen2.5-14B, and Qwen/Qwen2.5-32B) to act as reasoners. Notably, I omitted the true and related misconceptions from the input for fine-tuning. During inference, these models would see: (i) the problem statement, (ii) the correct answer, and (iii) an incorrect answer. They would then generate the likely thought process of a student behind selecting the incorrect answer. I used vllm for CoT generation during inference with temperature=0.7, top_p=0.8, repetition_penalty=1.0, max_tokens=256.
The CoT dataset is uploaded here: Eedi - CoT Dataset.
\nI split the competition dataset into 5 folds, initially based on the QuestionId field. However, this produced overly optimistic estimates with a large gap between validation and leaderboard scores. Next, I tried GroupKFold on SubjectId, but this resulted in overly pessimistic estimates. I finally settled on GroupKFold with ConstructId, which yielded a narrow Validation/LB gap and good correlation up to ~0.62 public LB score range.
During development, I always trained my models on folds 1-4 and validated on fold 0. The synthetic data was marked as fold 99 and was included in the training set. The final models were trained on the entire dataset (full fit) using the same hyper parameters from the development stage.
\nI used the standard MultipleNegativesRankingLoss to fine-tune the retrievers, while referencing the FlagEmbedding codebase for developing training scripts. At this stage, I tracked both map@25 and recall@32 metrics and prioritized optimizing for recall@32, since recall was more important for overall pipeline performance.
I fine-tuned several retrievers with the following scores:\n
Incorporating hard negatives into training batches and distilling re-ranker scores consistently improved map@25 performance, but did not positively impact recall@32. For my final submission selections, I chose the high-recall Qwen/Qwen2.5-14B encoder (Model 3) instead of the best map@25 encoder (Model 4). Following the same logic, I also excluded the BGE encoder (Model 2). Notably, my best encoder-only submission (Model 1 + Model 2 + Model 4) scored 0.524 on public LB and 0.475 on private LB.
Retrievers were fine-tuned using LoRA with: r=64, alpha=128, learning_rate_lora_a=1e-5, learning_rate_lora_b=5e-5, LoRA on all linear layers, batch_size=128, and epochs=12. A few key factors that improved recall performance were:
Qwen/Qwen2.5-14B encoder with all available synthetic data (before the curation step in section 4.1.1).What didn't work:
\nThe Qwen 14B ranker did most of the heavy lifting by processing all retrieved misconceptions (32-64 candidates) and identifying top 8 candidates. This model was trained with a pointwise approach, where it sees one misconception at a time in the context window. The model input was structured as follows:\n
The logits were computed by taking the difference between the 'Yes' and 'No' token scores, as below:
\noutputs = self.model(input_ids=input_ids, attention_mask=attention_mask, output_hidden_states=True)\nlogits_yes = outputs.logits[:, -1, self.yes_loc] # Yes token logit at the last position [bs]\nlogits_no = outputs.logits[:, -1, self.no_loc] # No token logit at the last position [bs]\nlogits = logits_yes - logits_no # [bs]\n\nlogits = logits.reshape(-1, self.group_size)\nlabels = labels.to(logits.device).reshape(-1) \nce_loss = self.loss_fn(logits, labels)\nEach batch contained 1 positive and batch_size-1 negative misconceptions for a given Question-Incorrect Answer combination. The label was set to the index of the positive misconception and cross entropy loss was used for training. The models was trained with LoRA with r=64, alpha=128, LoRA on all linear layers. The language modelling head (lm_head) of the LLM (Qwen/Qwen2.5-14B) was frozen during fine-tuning.
I found the following 4 strategies to be particularly significant for boosting the reranker performance and its generalization capability:
\nTo retain and leverage the LLM's in-context learning capabilities, I optionally included a few reference misconception examples (0 to 2) in the model input context (as shown in the diagram above). Not all training examples contained these few-shot demonstrations. The objective was to encourage the model to use reference examples when available, and otherwise rely on its internal reasoning. During inference, I used 1 or 2 examples from the entire training set as demonstrations. This approach was particularly effective for the 14B ranker's performance boosting private LB score from 0.495 to 0.531 (+0.036). At this point, I was only using the competition data for training.
\nNext, I incorporated the synthetic examples (post curation) into the training pipeline through pseudo labeling. I first fine-tuned two pointwise 72B models (Qwen/Qwen2.5-Math-72B and Qwen/Qwen2.5-72B) using only the competition dataset. These models were then used to generate pseudo labels for the synthetic examples. Finally, I used competition + pseudo-labeled data to fine-tune the next iteration of the 14B rerankers. This strategy boosted the 14B reranker's private LB score from 0.531 to 0.575 (+0.044).
Next, I experimented with increasing the number of negative samples per positive. Showing the rankers a large number of negatives during training helped to improve performance. For each positive, I next increased the number of negatives to 24 from earlier 16. At this point, I also added scaled number of synthetic examples 2x. These two modifications boosted the 14B reranker's private LB score from 0.575 to 0.596 (+0.021).
\nI finally added the fine-tuned CoT reasoners (Section 4.2) into training and inference pipeline. During training, CoTs produced from fine-tuned Qwen/Qwen2.5-14B reasoner were optionally added to the model input. 50% of training data used CoTs and 50% did not. This encouraged the model to use external reasoning when available and otherwise rely on its internal reasoning. CoT boosted the 14B reranker's private LB score from 0.596 to 0.615 (+0.019).
The impact of these strategies is summarized in the following table:\n
The live eval during fine-tuning showed consistent improvement in the 14B reranker's performance.\n
Qwen/Qwen2.5-32B)I fine-tuned the Qwen/Qwen2.5-32B model in exactly the same way as the 14B reranker (with a different seed). My best 32B model had validation score of 0.663 vs that of 0.646 for the 14B reranker. During inference, the 32B reranker was used to process top 8 candidates from the 14B reranker and narrow them down to top 5 candidates. Incorporating the 32B reranker boosted private LB score from 0.615 to 0.625 (+0.010).
Qwen/Qwen2.5-72B)The top 5 candidates were finally ranked in a listwise manner using a fine-tuned Qwen/Qwen2.5-72B model. Here, the model sees all of the top 5 candidates at once. An example model input is shown below:\n
This model had access to the even more information than pointwise re-rankers i.e.
\nThe model had the best MAP@5 among all of the rerankers (validation MAP@5 of 0.661). LLM based Listwise rankers typically suffer from candidate position bias. One way to mitigate this is to run multiple inference runs by shuffling the position of the candidate misconceptions (Test time augmentation (TTA) like impact). I was unable to run TTA during inference due to time constraints. Incorporating the 72B reranker boosted private LB score from 0.625 to 0.638 (+0.013).
I quantized the fine-tuned rankers using AutoAWQ. I used task specific calibration datasets for each model to mitigate performance degradation. I tested the impact of calibration dataset with one of my 32B rerankers. The results are shown below:
\n
Qwen/Qwen2.5-32B but got worse results. The QwQ model likely requires additional research to find out its proper usage.Detailed is an understatement!
\nThis is great info to share, Raja. Congrats again!
", "rawMarkdown": "Detailed is an understatement!\n\nThis is great info to share, Raja. Congrats again!", "votes": 5 }, { "id": 3154496, "postDate": "2025-03-20T02:42:06.973Z", "content": "Hi, thanks for sharing such an excellent proposal—it’s a solid RAG framework concept. Here's my understanding of your approach to training the reasoner:
\nMy question is: How do you ensure that the generated text—the CoT content—is accurate? Do you employ a curation step, similar to the one used in the synthesized data process, to filter for high-quality CoTs?
", "rawMarkdown": "Hi, thanks for sharing such an excellent proposal—it’s a solid RAG framework concept. Here's my understanding of your approach to training the reasoner:\n\n1. First, you use a prompt with Claude 3.5 to generate a chain-of-thought (CoT) that might lead to selecting the wrong answer.\n2. Then, with the same input, you filter out the error causes and near-error causes and combine the resulting text with the generated CoT to form a query-CoT text pair for fine-tuning the student model.\n\nMy question is: How do you ensure that the generated text—the CoT content—is accurate? Do you employ a curation step, similar to the one used in the synthesized data process, to filter for high-quality CoTs?", "votes": 1, "replies": [ { "id": 3155013, "postDate": "2025-03-20T16:05:37.953Z", "content": "Hi @youwbpyr, Thanks. Yes, I used the generated query-CoT pairs to distill Claude 3.5's reasoning capability into student models. For this competition, I didn't curate the generated CoTs. However, I think generating multiple CoTs and picking the best one as per a process reward model / llm-as-a-judge could improve student models reasoning abilities.
", "rawMarkdown": "Hi @youwbpyr, Thanks. Yes, I used the generated query-CoT pairs to distill Claude 3.5's reasoning capability into student models. For this competition, I didn't curate the generated CoTs. However, I think generating multiple CoTs and picking the best one as per a process reward model / llm-as-a-judge could improve student models reasoning abilities. ", "votes": 2 } ] }, { "id": 3077833, "postDate": "2024-12-21T12:56:21.633Z", "content": "Hi Raja, thanks for the detailed solution! Congrats for the solo win🥇
\nWould you mind if I ask a few questions ?
\nGreat job on trying out lots of experiments within a short period of three months. What GPU specifications/clusters did you use to fine tune the models ?
\nContext: I rented H100s from RunPod and finetuning using LoRA for a 14B Yes/No reranker took > 5 hours with our team’s external data. For initial experiments I tested various numbers of hard negatives (1,2,4,8,16,24) and easily training these six models can take more than 2 days total (just for one idea!). Towards the end I finetuned 32B and the training on just 4 negatives took > 20 hours for a single experiment (5 hours without external data)! I trained for 2 epochs using the SFTTrainer. Felt that there may be a faster way…
\nDo you have tips on selecting experiments?
\nContext: From the MCTS and EEDI competitions, I realised that I’m able to think of lots of ideas to solve the problem. However, I felt my ability to decide which one to prioritise needs improvement. For example in EEDI, I spent 3 days customizing TextGrad to optimize prompts for Qwen to hopefully improve the reranking performance (similar to this). However the improvement is very marginal and possibly not worth 3 full days of effort…Maybe I will get better at this as I gain more experience 😅
\nReally inspiring that you got the correct approach from the beginning and have 3 months to get the most out of it 👍
", "rawMarkdown": "Hi Raja, thanks for the detailed solution! Congrats for the solo win🥇\n\nWould you mind if I ask a few questions ?\n\n**Great job on trying out lots of experiments within a short period of three months. What GPU specifications/clusters did you use to fine tune the models ?**\n\nContext: I rented H100s from RunPod and finetuning using LoRA for a 14B Yes/No reranker took > 5 hours with our team’s external data. For initial experiments I tested various numbers of hard negatives (1,2,4,8,16,24) and easily training these six models can take more than 2 days total (just for one idea!). Towards the end I finetuned 32B and the training on just 4 negatives took > 20 hours for a single experiment (5 hours without external data)! I trained for 2 epochs using the SFTTrainer. Felt that there may be a faster way…\n\n**Do you have tips on selecting experiments?**\n\nContext: From the MCTS and EEDI competitions, I realised that I’m able to think of lots of ideas to solve the problem. However, I felt my ability to decide which one to prioritise needs improvement. For example in EEDI, I spent 3 days customizing TextGrad to optimize prompts for Qwen to hopefully improve the reranking performance (similar to [this](https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/540759)). However the improvement is very marginal and possibly not worth 3 full days of effort…Maybe I will get better at this as I gain more experience 😅\n\nReally inspiring that you got the correct approach from the beginning and have 3 months to get the most out of it 👍\n\n\n\n", "votes": 4, "replies": [ { "id": 3078167, "postDate": "2024-12-21T21:54:46.657Z", "content": "Hi bogoconic1!
\nThanks a lot! You've been on an impressive path and performing well across different types of competitions!
\n\n\nGreat job on trying out lots of experiments within a short period of three months. What GPU specifications/clusters did you use to fine tune the models ?
\n
I used 2xH100 GPUs from vast.ai. I trained my models using the HF accelerate with a few basic speed optimizations e.g. using (1) torch.compile (2) flash-attention (3) LoRA instead of QLoRA (4) padding sequence lengths to multiples of 8. The 14B re-ranker was also taking ~5hrs for a full run in my setup. But, I very rarely did a full run: (a) ~50% of training was giving good indications on the experiment, (b) I primarily experimented with the 7B models. For 32B, I just copied the best settings from 14B. IMO key is to keep experiment setup as lean as possible.
\n\n\nDo you have tips on selecting experiments?
\n
I often struggle with this too. It's difficult to get the exploration vs exploitation balance right. Although there is no silver bullet, I think the following points help in NLP comps:
\nWill add to this list if I can think of other points tomorrow. I'm not sure if these are too obvious or very opinionated…
", "rawMarkdown": "Hi bogoconic1!\n\nThanks a lot! You've been on an impressive path and performing well across different types of competitions!\n\n>Great job on trying out lots of experiments within a short period of three months. What GPU specifications/clusters did you use to fine tune the models ?\n\nI used 2xH100 GPUs from [vast.ai](https://cloud.vast.ai/create/). I trained my models using the HF accelerate with a few basic speed optimizations e.g. using (1) torch.compile (2) flash-attention (3) LoRA instead of QLoRA (4) padding sequence lengths to multiples of 8. The 14B re-ranker was also taking ~5hrs for a full run in my setup. But, I very rarely did a full run: (a) ~50% of training was giving good indications on the experiment, (b) I primarily experimented with the 7B models. For 32B, I just copied the best settings from 14B. IMO key is to keep experiment setup as lean as possible.\n\n> Do you have tips on selecting experiments?\n\nI often struggle with this too. It's difficult to get the exploration vs exploitation balance right. Although there is no silver bullet, I think the following points help in NLP comps:\n1. Try to find the best set of backbones to fine-tune from\n2. Once you find a good backbone & have a decent baseline, focus on data related experiments\n3. Write down takeaways after each experiment -- include related plots, error analysis etc. This should inform you what to do next\n4. Techniques like distillation/pseudo labeling almost always help with performance and generalization\n\nWill add to this list if I can think of other points tomorrow. I'm not sure if these are too obvious or very opinionated...", "votes": 10 } ] }, { "id": 3122439, "postDate": "2025-02-12T16:21:11.237Z", "content": "hi～ I would like to ask you another question. In your notebook at https://gist.github.com/rbiswasfc/8d2bfec5c2a358e8beeb2df390111f9d, you used a file named ranker_oof_df.csv. In your solution, you mentioned, “Specifically, I first created clusters of closely related misconceptions leveraging co-occurrence statistics of misconceptions in retriever/ranker predictions on validation data.” My question is, when you refer to “validation data” here, do you mean the predictions on the validation data obtained by training a model using the training set, which is split from the original entire dataset into training and valid sets? thanks a lot！
", "rawMarkdown": "hi～ I would like to ask you another question. In your notebook at https://gist.github.com/rbiswasfc/8d2bfec5c2a358e8beeb2df390111f9d, you used a file named ranker_oof_df.csv. In your solution, you mentioned, “Specifically, I first created clusters of closely related misconceptions leveraging co-occurrence statistics of misconceptions in retriever/ranker predictions on validation data.” My question is, when you refer to “validation data” here, do you mean the predictions on the validation data obtained by training a model using the training set, which is split from the original entire dataset into training and valid sets? thanks a lot！", "votes": 1, "replies": [ { "id": 3123110, "postDate": "2025-02-13T11:41:31.767Z", "content": "Hi @yuhaowu123, yes, exactly. Using top retrieved candidates on the validation split.
", "rawMarkdown": "Hi @yuhaowu123, yes, exactly. Using top retrieved candidates on the validation split.", "replies": [ { "id": 3141956, "postDate": "2025-03-06T02:04:19.437Z", "rawMarkdown": "", "isDeleted": true } ] } ] }, { "id": 3081076, "postDate": "2024-12-26T07:06:16.510Z", "content": "Superb, good luck in your future ventures!
", "rawMarkdown": "Superb, good luck in your future ventures!", "votes": 1 }, { "id": 3078878, "postDate": "2024-12-23T00:42:09.683Z", "content": "Congrats for the winning! Did you try to train the 14b reranker with synthetic data but without distillation from 72b? I wonder how it would perform.
", "rawMarkdown": "Congrats for the winning! Did you try to train the 14b reranker with synthetic data but without distillation from 72b? I wonder how it would perform.", "votes": 1, "replies": [ { "id": 3078984, "postDate": "2024-12-23T04:54:24.810Z", "content": "Interesting, this should be a good experiment! However, I didn't try this thinking synthetic data misconceptions still may be noisy (lesser quality than competition data) even after curation…
", "rawMarkdown": "Interesting, this should be a good experiment! However, I didn't try this thinking synthetic data misconceptions still may be noisy (lesser quality than competition data) even after curation..." } ] }, { "id": 3078557, "postDate": "2024-12-22T13:49:41.320Z", "content": "Thank you very much for your insights! This is a very interesting and solid solution. However, I have a few questions regarding the prompt design that I’d like to clarify:
\nYou mentioned that the misconception you provided are very similar, but for a single question (Q), the three incorrect answers (A) might correspond to very different types of misconception. In that case, does each reference correspond to one QA pair, or does one Q correspond to three different A options ?
The prompt does not explicitly mention that the types of misconception in the generated samples can be new. However, it does state that you need to first identify which misconception need to be leverage, which implies selecting misconception from those provided that were not covered by the reference questions. If this is the case, how can we ensure that new types of misconception will appear in the constructed dataset?
The prompt mentions that the tag needs to indicate which misconception each incorrect answer corresponds to. Does this mean that the reference questions only provide the incorrect answers without providing the misconception？
Many of my above questions stem from uncertainty about the content within the tag and the tag in a complete prompt. Would you be able to provide me with an example of a fully constructed prompt including these tags?Thanks a lot again！~~~
Here is an example prompt:
\nYou will be generating Multiple Choice Questions (MCQs) diagnose specific mathematical misconceptions. Here are the misconceptions you should focus on:\n\n<misconceptions>\n- Thinks the perimeter a square grid the number squares surrounding the rectangle\n- Calculates area asked perimeter\n- Counts a side length more than once counting perimeter due marking a starting point\n- Thinks the perimeter a square grid the number squares inside are touching the perimeter the shape\n- Calculates perimeter asked area\n- When calculating perimeter, only adds two sides\n- Believes lines inside the shape are included finding perimeter\n- When calculating perimeter, only adds the marked lengths\n- When calculating perimeter, multiplies instead adds\n- Thinks only labelled dimensions need be included calculating the perimeter\n- Confuses perimeter area\n- Fails use the properties simple shapes work out missing lengths\n- Does know we can use the conservation perimeter find missing lengths\n- Makes an assumption about line segments being equal within a shape\n- Mixes up width height a rectangle\n- Does know how find missing lengths a composite shape\n</misconceptions>\n\nHere are reference MCQs demonstrate how effectively diagnose these misconceptions:\n\n<reference_mcqs>\n[{: , : , : , : , : , : , : , : , : , : , : , : }, {: , : , : , : , : , : , : , : , : , : , : , : }, {: , : , : , : , : , : , : , : , : , : , : , : }, {: , : , : , : , : , : , : , : , : , : , : , : }, {: , : , : , : , : , : , : , : , : , : , : , : }]\n</reference_mcqs>\n\nYour task generate MCQs diagnose misconceptions already covered the reference MCQs.\n\nFirst, analyze the reference MCQs carefully:\n For each reference MCQ, identify your <analysis> tags:\n - Which misconception targets\n - How the incorrect answers specific misconceptions\n - What makes the question effective at diagnosing the misconception\n Note the style, difficulty level, precision language used\n\nThen, your <planning> tags:\n- List which misconceptions still need coverage\n- For each needed misconception, brainstorm mathematical contexts where commonly appears\n- Design questions where the misconception leads naturally specific wrong answers\n- Take notes how you can craft MCQs adheres the reference MCQs\n\nYou mentioned that the misconception you provided are very similar, but for a single question (Q), the three incorrect answers (A) might correspond to very different types of misconception. In that case, does each reference correspond to one QA pair, or does one Q correspond to three different A options ?
\n
For each MCQ, the LLM would generate
\n \n \n \n \n \n \n \n \n \n \n \n \ntherefore, incorrect answers and misconceptions can be paired properly.
\n\n\nThe prompt does not explicitly mention that the types of misconception in the generated samples can be new. However, it does state that you need to first identify which misconception need to be leverage, which implies selecting misconception from those provided that were not covered by the reference questions. If this is the case, how can we ensure that new types of misconception will appear in the constructed dataset?
\n
Nice spot. Even if we specify the possible misconceptions, LLM still likes to generate additional ones. Before I settled on this prompt, my earlier prompts would generate even more of new misconceptions.
\n\n\nThe prompt mentions that the tag needs to indicate which misconception each incorrect answer corresponds to. Does this mean that the reference questions only provide the incorrect answers without providing the misconception？
\n
I have provided a full prompt example here. Hope it helps!
", "rawMarkdown": "> You mentioned that the misconception you provided are very similar, but for a single question (Q), the three incorrect answers (A) might correspond to very different types of misconception. In that case, does each reference correspond to one QA pair, or does one Q correspond to three different A options ?\n\nFor each MCQ, the LLM would generate\n```\n subject_name: str, # The subject area of the question\n construct_name: str, # The specific mathematical construct being tested\n question_text: str, # The text of the question\n answer_a_text: str, # Text for answer choice A\n answer_b_text: str, # Text for answer choice B\n answer_c_text: str, # Text for answer choice C\n answer_d_text: str, # Text for answer choice D\n correct_answer: str, # The letter (A, B, C, or D) of the correct answer\n misconception_a_name: str, # The misconception name targeted by answer A (if any)\n misconception_b_name: str, # The misconception name targeted by answer B (if any)\n misconception_c_name: str, # The misconception name targeted by answer C (if any)\n misconception_d_name: str, # The misconception name targeted by answer D (if any)\n```\ntherefore, incorrect answers and misconceptions can be paired properly.\n\n> The prompt does not explicitly mention that the types of misconception in the generated samples can be new. However, it does state that you need to first identify which misconception need to be leverage, which implies selecting misconception from those provided that were not covered by the reference questions. If this is the case, how can we ensure that new types of misconception will appear in the constructed dataset?\n\nNice spot. Even if we specify the possible misconceptions, LLM still likes to generate additional ones. Before I settled on this prompt, my earlier prompts would generate even more of new misconceptions. \n\n> The prompt mentions that the tag needs to indicate which misconception each incorrect answer corresponds to. Does this mean that the reference questions only provide the incorrect answers without providing the misconception？\n\nI have provided a full prompt example [here](https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/551688#3079014). Hope it helps!", "replies": [ { "id": 3140911, "postDate": "2025-03-05T03:58:35.747Z", "content": "Do the misconceptions generated by the LLM, such as misconception_a_name, misconception_b_name, misconception_c_name, and misconception_d_name, tend to choose from the given misconception pool of the cluster?
", "rawMarkdown": "Do the misconceptions generated by the LLM, such as misconception_a_name, misconception_b_name, misconception_c_name, and misconception_d_name, tend to choose from the given misconception pool of the cluster?" } ] }, { "id": 3140906, "postDate": "2025-03-05T03:48:03.497Z", "content": "I would like to ask: If only the misconceptions are provided as clusters, where do the other answers and corresponding misconceptions generated by the LLM for a new question come from?
", "rawMarkdown": "I would like to ask: If only the misconceptions are provided as clusters, where do the other answers and corresponding misconceptions generated by the LLM for a new question come from? " } ] }, { "id": 3077562, "postDate": "2024-12-21T05:39:43.933Z", "content": "Thank you so much for this detailed solution. I'm so impressed with how much attention you've paid to details, and how you fully utilze the limited GPU resources to perform so many rounds of reranking. May I ask:
\nThanks again! Congratulations on 1st place! The solution really deserves it.
", "rawMarkdown": "Thank you so much for this detailed solution. I'm so impressed with how much attention you've paid to details, and how you fully utilze the limited GPU resources to perform so many rounds of reranking. May I ask:\n\n- you mentioned \"add recovery adapters\" to quantization, what is it? did you apply it eventually?\n- What are the important tricks to fit 72b model for inference into 2 T4 GPUs?\n- What total running time did you manange to achieve during competition? Any out of time or out of memory experienced near the end of the competition?\n\nThanks again! Congratulations on 1st place! The solution really deserves it.", "votes": 1, "replies": [ { "id": 3077760, "postDate": "2024-12-21T11:19:04.580Z", "content": "Thanks for the kind words! Congratulations to you too on solo gold! 🎉
\n\n\nyou mentioned \"add recovery adapters\" to quantization, what is it? did you apply it eventually?
\n
The idea comes from the Apple Intelligence report (Section 5.2).
\n\n\n… The quantized model often shows a moderate level of quality loss. Therefore, instead of directly passing the quantized model to application teams for feature development, we attach a set of parameter-efficient LoRA adapters for quality recovery. We make sure that these LoRA adapters training recipes are consistent with pre-training and post-training processes.
\n
I didn't get to try this in this competition.
\n\n\nWhat are the important tricks to fit 72b model for inference into 2 T4 GPUs?
\n
I mainly followed this great post from @jagatkiran! Didn't have any inference tricks for the 72b model. For 14b ranker, prefix caching was key.
\n\n\nWhat total running time did you manange to achieve during competition? Any out of time or out of memory experienced near the end of the competition?
\n
My best selected submission was ~510mins. I didn't face OOM but had a few out of time near the end…
\nThanks!
", "rawMarkdown": "Thanks for the kind words! Congratulations to you too on solo gold! 🎉\n\n> you mentioned \"add recovery adapters\" to quantization, what is it? did you apply it eventually?\n\nThe idea comes from the [Apple Intelligence report](https://arxiv.org/pdf/2407.21075) (Section 5.2). \n> ... The quantized model often shows a moderate level of quality loss. Therefore, instead of directly passing the quantized model to application teams for feature development, we attach a set of parameter-efficient LoRA adapters for quality recovery. We make sure that these LoRA adapters training recipes are consistent with pre-training and post-training processes.\n\nI didn't get to try this in this competition.\n\n> What are the important tricks to fit 72b model for inference into 2 T4 GPUs?\n\nI mainly followed [this great post](https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/550223) from @jagatkiran! Didn't have any inference tricks for the 72b model. For 14b ranker, prefix caching was key.\n\n>What total running time did you manange to achieve during competition? Any out of time or out of memory experienced near the end of the competition?\n\nMy best selected submission was ~510mins. I didn't face OOM but had a few out of time near the end...\n\nThanks!", "votes": 2, "replies": [ { "id": 3078220, "postDate": "2024-12-22T00:36:24.273Z", "content": "Thank you for your explanation! I also noticed that for your retriever, you used different learning rate for lora_a and lora _b, this is an interesting detail. May I ask what is the rationale behind that?
", "rawMarkdown": "Thank you for your explanation! I also noticed that for your retriever, you used different learning rate for lora_a and lora _b, this is an interesting detail. May I ask what is the rationale behind that?", "votes": 1 }, { "id": 3078977, "postDate": "2024-12-23T04:48:38.020Z", "content": "I followed the LoRA+ paper: LoRA+: Efficient Low Rank Adaptation of Large Models -- which shows faster convergence and better perf with different LRs for LoRA A and B.
\n\n", "rawMarkdown": "I followed the LoRA+ paper: [LoRA+: Efficient Low Rank Adaptation of Large Models](https://arxiv.org/abs/2402.12354) -- which shows faster convergence and better perf with different LRs for LoRA A and B.\n\n>...Low Rank Adaptation (LoRA) as originally introduced in Hu et al. (2021) leads to suboptimal finetuning of models with large width (embedding dimension). This is due to the fact that adapter matrices A and B in LoRA are updated with the same learning rate. Using scaling arguments for large width networks, we demonstrate that using the same learning rate for A and B does not allow efficient feature learning. We then show that this suboptimality of LoRA can be corrected simply by setting different learning rates for the LoRA adapter matrices A and B with a well-chosen ratio. We call this proposed algorithm LoRA+. In our extensive experiments, LoRA+ improves performance (1-2 % improvements) and finetuning speed (up to ∼ 2X SpeedUp), at the same computational cost as LoRA.", "votes": 1 } ] } ] }, { "id": 3073545, "postDate": "2024-12-16T15:42:50.033Z", "content": "…Low Rank Adaptation (LoRA) as originally introduced in Hu et al. (2021) leads to suboptimal finetuning of models with large width (embedding dimension). This is due to the fact that adapter matrices A and B in LoRA are updated with the same learning rate. Using scaling arguments for large width networks, we demonstrate that using the same learning rate for A and B does not allow efficient feature learning. We then show that this suboptimality of LoRA can be corrected simply by setting different learning rates for the LoRA adapter matrices A and B with a well-chosen ratio. We call this proposed algorithm LoRA+. In our extensive experiments, LoRA+ improves performance (1-2 % improvements) and finetuning speed (up to ∼ 2X SpeedUp), at the same computational cost as LoRA.
\n
I want to know if your recall model is created by combining model 1, model 2, and model 3, and then concatenating the last token, or if another method is used.
", "rawMarkdown": "I want to know if your recall model is created by combining model 1, model 2, and model 3, and then concatenating the last token, or if another method is used.", "votes": 1, "replies": [ { "id": 3074399, "postDate": "2024-12-17T16:22:45.953Z", "content": "I averaged similarity scores from these models.
", "rawMarkdown": "I averaged similarity scores from these models." } ] }, { "id": 3073495, "postDate": "2024-12-16T14:53:16.193Z", "content": "Did you merge the training set and the synthetic dataset together and then perform the 5-fold split, or did you perform the 5-fold split on the training set and then directly add the synthetic data to the training set?
", "rawMarkdown": "Did you merge the training set and the synthetic dataset together and then perform the 5-fold split, or did you perform the 5-fold split on the training set and then directly add the synthetic data to the training set?", "votes": 1, "replies": [ { "id": 3073504, "postDate": "2024-12-16T15:00:42.240Z", "content": "5-fold split on the training set and then directly add the synthetic data to the training set
", "rawMarkdown": "5-fold split on the training set and then directly add the synthetic data to the training set", "votes": 1 } ] }, { "id": 3073391, "postDate": "2024-12-16T13:18:32.093Z", "content": "Your explanation is excellent bro, Thanks.
", "rawMarkdown": "Your explanation is excellent bro, Thanks.", "votes": 1 }, { "id": 3073360, "postDate": "2024-12-16T11:59:14.937Z", "content": "Congrats both. Result ans explication.
", "rawMarkdown": "Congrats both. Result ans explication. ", "votes": 1 }, { "id": 3072882, "postDate": "2024-12-15T18:00:13.827Z", "content": "This is undoubtedly very insightful! Thank you for sharing this amazing information! @conjuring92
", "rawMarkdown": "This is undoubtedly very insightful! Thank you for sharing this amazing information! @conjuring92\n\n\n", "votes": 1 }, { "id": 3072696, "postDate": "2024-12-15T15:00:31.440Z", "content": "Thanks for sharing such a detailed solution!
", "rawMarkdown": "Thanks for sharing such a detailed solution!", "votes": 1 }, { "id": 3072550, "postDate": "2024-12-15T11:36:55.657Z", "content": "A very insightful and beautiful solution, thank you!
\nWe have also tried to generate Chain-of-Thought using gpt-4o in the early stage of the competition, but found that we couldn't correctly formalize the way of thinking for the model, and quite often the model explicitly mentioned the misconception in its response (although we've tried to encourage it not to do so).
\nHaven't you met this problem during the prompt construction; if yes, can you please tell, what exact prompting modifications helped to solve it?
", "rawMarkdown": "A very insightful and beautiful solution, thank you!\n\nWe have also tried to generate Chain-of-Thought using gpt-4o in the early stage of the competition, but found that we couldn't correctly formalize the way of thinking for the model, and quite often the model explicitly mentioned the misconception in its response (although we've tried to encourage it not to do so).\n\nHaven't you met this problem during the prompt construction; if yes, can you please tell, what exact prompting modifications helped to solve it?", "votes": 1, "replies": [ { "id": 3073139, "postDate": "2024-12-16T05:10:59.153Z", "content": "Yes, I too noticed Claude mentioning the misconception name at the start of CoT. I fixed it with prompt engineering by asking LLM to follow this sequence:
\nI see a lot of people used the LM Head with Yes/No probabilities as the binary classifier. Any thoughts on whether that is the best approach when fine tuning, or is it better to add a new classifier head? I tried both and the classifier head worked better for me, but it was also my first time doing any serious comparision between these two approaches. Appreciate if you share your thoughts. Thanks also for such an immensely detailed post, learnt a lot.
", "rawMarkdown": "I see a lot of people used the LM Head with Yes/No probabilities as the binary classifier. Any thoughts on whether that is the best approach when fine tuning, or is it better to add a new classifier head? I tried both and the classifier head worked better for me, but it was also my first time doing any serious comparision between these two approaches. Appreciate if you share your thoughts. Thanks also for such an immensely detailed post, learnt a lot.", "votes": 1, "replies": [ { "id": 3073138, "postDate": "2024-12-16T05:08:13.313Z", "content": "Thanks Dipam!
\nI actually trained most of my models with a new classification head and switched to Yes/No tokens approach in the later stages -- mainly to take advantage of vllm inference + in-built prefix caching. Both Yes/No approach and classifier head gave similar scores here.
", "rawMarkdown": "Thanks Dipam!\n\nI actually trained most of my models with a new classification head and switched to Yes/No tokens approach in the later stages -- mainly to take advantage of vllm inference + in-built prefix caching. Both Yes/No approach and classifier head gave similar scores here.", "votes": 3, "replies": [ { "id": 3078523, "postDate": "2024-12-22T12:32:24.967Z", "content": "Thanks for the detailed writeup, a lot to learn here. I am a bit confused in this part:
\nQuestion: Since you are finetuning the causal lm to output Yes/No token, why the lm_head was frozen. What was the reasoning behind this ? As we want the lm_head to have high probs for either Yes/No token, freezing it will avoid it to learn this mapping. Can you explain this part ?
\n\n\nThe logits were computed by taking the difference between the 'Yes' and 'No' token scores, as below:
\n
outputs = .model(input_ids=input_ids, attention_mask=attention_mask, output_hidden_states=)\nlogits_yes = outputs.logits[:, -, .yes_loc] \nlogits_no = outputs.logits[:, -, .no_loc] \nlogits = logits_yes - logits_no \nlogits = logits.reshape(-, .group_size)\nlabels = labels.to(logits.device).reshape(-) \nce_loss = .loss_fn(logits, labels)\n\n", "rawMarkdown": "Thanks for the detailed writeup, a lot to learn here. I am a bit confused in this part:\n\nQuestion: Since you are finetuning the causal lm to output Yes/No token, why the lm_head was frozen. What was the reasoning behind this ? As we want the lm_head to have high probs for either Yes/No token, freezing it will avoid it to learn this mapping. Can you explain this part ?\n\n\n>The logits were computed by taking the difference between the 'Yes' and 'No' token scores, as below:\n```python\noutputs = self.model(input_ids=input_ids, attention_mask=attention_mask, output_hidden_states=True)\nlogits_yes = outputs.logits[:, -1, self.yes_loc] # Yes token logit at the last position [bs]\nlogits_no = outputs.logits[:, -1, self.no_loc] # No token logit at the last position [bs]\nlogits = logits_yes - logits_no # [bs]\nlogits = logits.reshape(-1, self.group_size)\nlabels = labels.to(logits.device).reshape(-1) \nce_loss = self.loss_fn(logits, labels)\n```\n>Each batch contained 1 positive and batch_size-1 negative misconceptions for a given Question-Incorrect Answer combination. The label was set to the index of the positive misconception and cross entropy loss was used for training. The models was trained with LoRA with r=64, alpha=128, LoRA on all linear layers. The language modelling head (lm_head) of the LLM (Qwen/Qwen2.5-14B) was frozen during fine-tuning.", "votes": 1 }, { "id": 3079011, "postDate": "2024-12-23T05:59:35.763Z", "content": "Each batch contained 1 positive and batch_size-1 negative misconceptions for a given Question-Incorrect Answer combination. The label was set to the index of the positive misconception and cross entropy loss was used for training. The models was trained with LoRA with r=64, alpha=128, LoRA on all linear layers. The language modelling head (lm_head) of the LLM (Qwen/Qwen2.5-14B) was frozen during fine-tuning.
\n
Hey @atharvaingle, on freezing the lm_head:
logit(y=Yes) = features . W_lm_head_yes : The Yes/No token logits are controlled by both the LM head weights at the location of Yes/No tokens and model hidden states at end of the sequence. Even if we freeze the LM head, the logits can be adjusted by the network by updating other params such that features are moving as per loss during fine-tuning.Thanks this helps a lot.
", "rawMarkdown": "Thanks this helps a lot.", "votes": 1 }, { "id": 3086113, "postDate": "2025-01-02T02:26:49.143Z", "content": "If possible, could you show me which part of the code freezes the weights for this section? I couldn’t find it. Thank you very much.
", "rawMarkdown": "If possible, could you show me which part of the code freezes the weights for this section? I couldn’t find it. Thank you very much." } ] } ] }, { "id": 3072329, "postDate": "2024-12-15T03:31:27.977Z", "content": "Hi raja, I have carefully read the detailed plan you posted. I have truly learned a lot. As a beginner in LLM, your plan involves a lot of knowledge that I have not yet understood. Please allow me to express my sincerest gratitude to you and look forward to your performance in the next competition
", "rawMarkdown": "Hi raja, I have carefully read the detailed plan you posted. I have truly learned a lot. As a beginner in LLM, your plan involves a lot of knowledge that I have not yet understood. Please allow me to express my sincerest gratitude to you and look forward to your performance in the next competition", "votes": 1 }, { "id": 3074742, "postDate": "2024-12-18T02:40:30.673Z", "content": "Looking forward to the release of your training code.
", "rawMarkdown": "Looking forward to the release of your training code.", "votes": 2, "replies": [ { "id": 3077764, "postDate": "2024-12-21T11:20:42.470Z", "content": "Currently working on the code release, so sorry for the delay…
", "rawMarkdown": "Currently working on the code release, so sorry for the delay..." }, { "id": 3080817, "postDate": "2024-12-25T19:16:04.187Z", "content": "published the code here: https://github.com/rbiswasfc/eedi-mining-misconceptions
", "rawMarkdown": "published the code here: https://github.com/rbiswasfc/eedi-mining-misconceptions", "votes": 1 } ] }, { "id": 3073598, "postDate": "2024-12-16T16:30:47.543Z", "content": "Hi @conjuring92, thank you so much for the detailed explanation. It is quite a lot to unpack.
\nIn my efforts towards learning, I am trying to summarize my understanding of your approach. However, I have a few questions and would really appreciate if you (or anyone on the forum) can help me get more clarity.
\n", "rawMarkdown": "Hi @conjuring92, thank you so much for the detailed explanation. It is quite a lot to unpack.\nIn my efforts towards learning, I am trying to summarize my understanding of your approach. However, I have a few questions and would really appreciate if you (or anyone on the forum) can help me get more clarity.\n\n# Questions\n1.\tHow many negative samples were used for retriever finetuning?\n2.\tCan you elaborate on *“Ensuring only one demonstration per misconception appeared in each training batch, as multiple demonstrations would introduce label noise through in-batch negatives”*. Do you mean 1 misconception won’t appear as true label twice in a batch?\n3.\tHow were the negative samples obtained for retriever models 4/ 3? Using a pretrained embedding model?\n4.\tCan you explain the re-ranker distillation part?\n5.\tAWQ - which datasets were used for calibration? \n\n\n># Summary\n## 1.\tData Generation\na.\tCreate clusters out of similar misconceptions\nb.\tGenerate 5 MCQs (using Claude 3.5 sonnet) for each clusters by giving cluster misconceptions and related 4-5 questions as context. (data quality can be questionable)\nc.\tUse GPT-4o to evaluate the quality of generated MCQs on a scale from 0-10 and filter out the bad ones.\nd.\tStep b gave additional misconceptions (not in misconception_mapping). Filtered out the misconceptions similar to the ones in the given mapping to avoid conflicts (using vectors from embedding model).\nAt this point the author has original misc clusters, and generated MCQs with misconceptions mapped to them (both original and generated)\n## 2.\tCOT Distillation\nOverall, the idea here is to distill the knowledge of claude into smaller models. So we get generations from claude and use that to finetune smaller models.\na.\tGenerate COT (on original train dataset) using Claude using the question context, misc and related misc in the prompt. \nb.\tFinetune 3 models (7,14,32B) on the generated text and same prompt but exclude the misconceptions and related misc (as during inference they won’t be present)\nAt this point the author has 3 models (reasoners) with the capability of generating COT reasons for each incorrect answer using only the questions context (no misconception).\n## 3.\tModels - retriever\na.\tFinetune retrievers using MNR loss for 12 epochs on entire data (except 1 fold) from step 1.d. {some confusion here will update once it is clarified}\nb.\tPre-training the finetuned qwen-14B on synthetic data from step 1.b gave recall boost. \n## 4.\tModel –reranker\n*overall 2 pointwise rerankers (yes/no output for each misconception) and 1 listwise used.*\n### a.\tPointwise rerankers (14/32B)\ni.\tPseudo labels on synthetic dataset (from 1.d) generated using models (Qwen/Qwen2.5-Math-72B and Qwen/Qwen2.5-72B) finetuned on train dataset.\nii.\tEntire dataset used for finetuning pointwise reranker. COT was also added to 50% of the training dataset.\niii.\tOther details-\n1.\tThe prompt contains few shot examples.\n2.\tThe re-ranker was trained on logit difference of ‘yes’ and ‘no’ tokens as predictions using cross entropy loss. \n3.\t1 positive and (BS – 1) negative samples taken for each batch. 1:24 positive:negative ratio.\n4.\tLM head was frozen during finetuning.\n### b.\tListwise reranker\ni.\tPredicts 1 out of top 5 choices.\nii.\t*{Training strategy not shared}*\n## 5.\tQuantization\na.\tFp16 finetuning was performed and the models were quantized using AWQ.\nb.\tCalibration dataset used *{some confusion here will update once it is clarified.}*\n\n", "votes": 2, "replies": [ { "id": 3074397, "postDate": "2024-12-17T16:22:09.333Z", "content": "Summary
\n1. Data Generation
\na. Create clusters out of similar misconceptions
\n
\n b. Generate 5 MCQs (using Claude 3.5 sonnet) for each clusters by giving cluster misconceptions and related 4-5 questions as context. (data quality can be questionable)
\n c. Use GPT-4o to evaluate the quality of generated MCQs on a scale from 0-10 and filter out the bad ones.
\n d. Step b gave additional misconceptions (not in misconception_mapping). Filtered out the misconceptions similar to the ones in the given mapping to avoid conflicts (using vectors from embedding model).
\n At this point the author has original misc clusters, and generated MCQs with misconceptions mapped to them (both original and generated)2. COT Distillation
\nOverall, the idea here is to distill the knowledge of claude into smaller models. So we get generations from claude and use that to finetune smaller models.
\n
\n a. Generate COT (on original train dataset) using Claude using the question context, misc and related misc in the prompt.
\n b. Finetune 3 models (7,14,32B) on the generated text and same prompt but exclude the misconceptions and related misc (as during inference they won’t be present)
\n At this point the author has 3 models (reasoners) with the capability of generating COT reasons for each incorrect answer using only the questions context (no misconception).3. Models - retriever
\na. Finetune retrievers using MNR loss for 12 epochs on entire data (except 1 fold) from step 1.d. {some confusion here will update once it is clarified}
\n
\n b. Pre-training the finetuned qwen-14B on synthetic data from step 1.b gave recall boost.4. Model –reranker
\noverall 2 pointwise rerankers (yes/no output for each misconception) and 1 listwise used.
\na. Pointwise rerankers (14/32B)
\ni. Pseudo labels on synthetic dataset (from 1.d) generated using models (Qwen/Qwen2.5-Math-72B and Qwen/Qwen2.5-72B) finetuned on train dataset.
\n
\n ii. Entire dataset used for finetuning pointwise reranker. COT was also added to 50% of the training dataset.
\n iii. Other details-\n
\n- The prompt contains few shot examples.
\n- The re-ranker was trained on logit difference of ‘yes’ and ‘no’ tokens as predictions using cross entropy loss.
\n- 1 positive and (BS – 1) negative samples taken for each batch. 1:24 positive:negative ratio.
\n- LM head was frozen during finetuning.
\nb. Listwise reranker
\ni. Predicts 1 out of top 5 choices.
\n
\n ii. {Training strategy not shared}5. Quantization
\na. Fp16 finetuning was performed and the models were quantized using AWQ.
\n
\n b. Calibration dataset used {some confusion here will update once it is clarified.}
Thanks Nikhil for the questions. Please find below my replies.
\n\n\nHow many negative samples were used for retriever finetuning?
\n
I used the standard in-batch negatives setup. Therefore, for a batch_size of 128, there were 127 negatives and 1 positive for each query (question + incorrect answer).
\n\n\nCan you elaborate on “Ensuring only one demonstration per misconception appeared in each training batch, as multiple demonstrations would introduce label noise through in-batch negatives”. Do you mean 1 misconception won’t appear as true label twice in a batch?
\n
Say, In a batch, the positive examples are (q1, m1), (q2, m2)..., (q128, m128). For the in-batch negatives setup, this means q1 will have m1 as positive and [m2, m3, ..m128] as negatives. Similarly, for q2 you will have m2 as positive and [m1, m3, ...m128] as negatives. During training, we will compute similarity scores between all query and misconception combinations (128x128 matrix). The diagonal elements would correspond to the positives and off-diagonals should be negatives. In this competition, we had many-to-one relationship for the query to content mapping. Hence, a random batching strategy would lead to cases where multiple queries belonging to the same misconception would end up in a given batch. In this case, you will have false negatives in the off diagonals -- leading to noisy, sub optimal convergence.
\n\nHow were the negative samples obtained for retriever models 4/ 3? Using a pretrained embedding model?
\n
I just used in-batch negatives. I, however, experimented mined hard negatives obtained from fine-tuned retrievers (+ de-noised with re-ranker predictions Reference), but it was ineffective to improve long tail recall.
\n\n\nCan you explain the re-ranker distillation part?
\n
You can refer to the distillation loss in the FlagEmbedding repo.
\n\n\nAWQ - which datasets were used for calibration?
\n
I basically used the a subset of re-ranker training dataset as calibration dataset.
", "rawMarkdown": "Thanks Nikhil for the questions. Please find below my replies.\n\n> How many negative samples were used for retriever finetuning?\n\nI used the standard **in-batch negatives** setup. Therefore, for a batch_size of 128, there were 127 negatives and 1 positive for each query (question + incorrect answer).\n\n> Can you elaborate on “Ensuring only one demonstration per misconception appeared in each training batch, as multiple demonstrations would introduce label noise through in-batch negatives”. Do you mean 1 misconception won’t appear as true label twice in a batch?\n\nSay, In a batch, the positive examples are `(q1, m1), (q2, m2)..., (q128, m128)`. For the in-batch negatives setup, this means `q1` will have `m1` as positive and `[m2, m3, ..m128]` as negatives. Similarly, for `q2` you will have `m2` as positive and `[m1, m3, ...m128]` as negatives. During training, we will compute similarity scores between all query and misconception combinations (`128x128` matrix). The diagonal elements would correspond to the positives and off-diagonals should be negatives. In this competition, we had many-to-one relationship for the query to content mapping. Hence, a random batching strategy would lead to cases where multiple queries belonging to the same misconception would end up in a given batch. In this case, you will have false negatives in the off diagonals -- leading to noisy, sub optimal convergence.\n\n> How were the negative samples obtained for retriever models 4/ 3? Using a pretrained embedding model?\n\nI just used in-batch negatives. I, however, experimented mined hard negatives obtained from fine-tuned retrievers (+ de-noised with re-ranker predictions [Reference](https://arxiv.org/abs/2407.15831)), but it was ineffective to improve long tail recall.\n\n> Can you explain the re-ranker distillation part?\n\nYou can refer to the [distillation loss](https://github.com/FlagOpen/FlagEmbedding/blob/master/FlagEmbedding/abc/finetune/embedder/AbsModeling.py#L280) in the [FlagEmbedding](https://github.com/FlagOpen/FlagEmbedding) repo.\n\n\n> AWQ - which datasets were used for calibration?\n\nI basically used the a subset of re-ranker training dataset as calibration dataset.", "votes": 3, "replies": [ { "id": 3074759, "postDate": "2024-12-18T03:15:24.390Z", "content": "Thanks a lot for the response! That helps.
", "rawMarkdown": "Thanks a lot for the response! That helps.", "votes": 1 }, { "id": 3182330, "postDate": "2025-04-19T06:04:40.330Z", "content": "Thanks a lot for the response! That helps.
", "rawMarkdown": "Thanks a lot for the response! That helps." } ] } ] }, { "id": 3072154, "postDate": "2024-12-14T19:09:44.270Z", "content": "This is undoubtedly very insightful! Thank you for sharing this amazing information! @conjuring92
", "rawMarkdown": "This is undoubtedly very insightful! Thank you for sharing this amazing information! @conjuring92 ", "votes": 2 }, { "id": 3121905, "postDate": "2025-02-12T05:06:11.393Z", "content": "Hello, I’d like to ask another question. The problem ultimately uses the MAP@K of the correct answers, but your code calculates the MAP@K for all the top K answers. I’m curious why the metric in the code is inconsistent with the metric required by the problem. the code i mentioned is ⬇️
\n for i, p in enumerate(predicted):
\n # first condition checks whether it is valid prediction
\n # second condition checks if prediction is not repeated
\n if p in actual and p not in predicted[:i]:
\n num_hits += 1.0
\n score += num_hits / (i + 1.0)
/ (len(actual), k)\nlooking forward for your reply，thanks a lot！
", "rawMarkdown": "Hello, I’d like to ask another question. The problem ultimately uses the MAP@K of the correct answers, but your code calculates the MAP@K for all the top K answers. I’m curious why the metric in the code is inconsistent with the metric required by the problem. the code i mentioned is ⬇️\n for i, p in enumerate(predicted):\n # first condition checks whether it is valid prediction\n # second condition checks if prediction is not repeated\n if p in actual and p not in predicted[:i]:\n num_hits += 1.0\n score += num_hits / (i + 1.0)\n\n return score / min(len(actual), k)\nlooking forward for your reply，thanks a lot！", "replies": [ { "id": 3123112, "postDate": "2025-02-13T11:42:27.030Z", "content": "Metric computation looks okay to me. Here is a detailed post on the metric: https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/534077
", "rawMarkdown": "Metric computation looks okay to me. Here is a detailed post on the metric: https://www.kaggle.com/competitions/eedi-mining-misconceptions-in-mathematics/discussion/534077" } ] }, { "id": 3087919, "postDate": "2025-01-04T04:38:36Z", "content": "GOAT. Amazing
", "rawMarkdown": "GOAT. Amazing" }, { "id": 3081241, "postDate": "2024-12-26T13:46:33.623Z", "content": "RAJA BISWAS, I am very grateful that you have published the training code！ I would like to know how long it took you to install the flash_attn package. It has been almost an hour since I ran pip install \"flash_attn==2.6.3\" --no-build-isolation, and it still hasn't completed the installation.
", "rawMarkdown": "RAJA BISWAS, I am very grateful that you have published the training code！ I would like to know how long it took you to install the flash_attn package. It has been almost an hour since I ran pip install \"flash_attn==2.6.3\" --no-build-isolation, and it still hasn't completed the installation.", "replies": [ { "id": 3081264, "postDate": "2024-12-26T14:14:33.623Z", "content": "that's strange, it was ~3mins iirc, please try: pip install flash-attn --no-build-isolation
Three minutes is really fast! I'm happy that I finally managed to install it, but it took me almost an hour. This might be because I used a VPN, as I can't directly connect to vast.ai from my country.
", "rawMarkdown": "Three minutes is really fast! I'm happy that I finally managed to install it, but it took me almost an hour. This might be because I used a VPN, as I can't directly connect to vast.ai from my country." }, { "id": 3081606, "postDate": "2024-12-27T01:52:17.697Z", "content": "Hello RAJA BISWAS,
\nI would like to know if the input_dataset in the configuration file conf_qwen14b_pretrain was named eedi-embed-pretrain-mix-final instead of eedi-embed-pretrain-mix-expanded when you trained Qwen/Qwen2.5-14B on the synthetic dataset. I couldn't find the file eedi-embed-pretrain-mix-expanded on Kaggle.
\nThank you!
", "rawMarkdown": "Hello RAJA BISWAS,\n\nI would like to know if the input_dataset in the configuration file conf_qwen14b_pretrain was named eedi-embed-pretrain-mix-final instead of eedi-embed-pretrain-mix-expanded when you trained Qwen/Qwen2.5-14B on the synthetic dataset. I couldn't find the file eedi-embed-pretrain-mix-expanded on Kaggle.\n\nThank you!" }, { "id": 3081692, "postDate": "2024-12-27T04:52:03.260Z", "content": "Thanks for the catch! It should be conjuring92/eedi-embed-pretrain-mix-final. I just pushed the fix.
Please feel free to create an issue in the repo if you encounter other issues!
", "rawMarkdown": "Thanks for the catch! It should be `conjuring92/eedi-embed-pretrain-mix-final`. I just pushed the fix.\n\nPlease feel free to create an issue in the repo if you encounter other issues!" }, { "id": 3133227, "postDate": "2025-02-25T06:13:30.100Z", "content": "I would like to know if you can provide the calibration datasets for quantizing a fine-tuned model using AWQ, such as rbiswasfc/eedi-awq-calibration, rbiswasfc/eedi-awq-calibration-tutor, and rbiswasfc/eedi-awq-calibration-cot
", "rawMarkdown": "I would like to know if you can provide the calibration datasets for quantizing a fine-tuned model using AWQ, such as rbiswasfc/eedi-awq-calibration, rbiswasfc/eedi-awq-calibration-tutor, and rbiswasfc/eedi-awq-calibration-cot" }, { "id": 3133584, "postDate": "2025-02-25T14:07:41.307Z", "content": "Here are the datasets:
\nMany thanks！
", "rawMarkdown": "Many thanks！" } ] } ] }, { "id": 3078935, "postDate": "2024-12-23T03:09:18.463Z", "content": "How was it achieved to inference 72b model on 2 T4 GPUs? They are 2*16=32GB Vram. AWQ of 72b takes 36GB vram on parameters only. And if offload to CPU, it would be so slow as not able to complete the inference within time.
", "rawMarkdown": "How was it achieved to inference 72b model on 2 T4 GPUs? They are 2*16=32GB Vram. AWQ of 72b takes 36GB vram on parameters only. And if offload to CPU, it would be so slow as not able to complete the inference within time." }, { "id": 3078921, "postDate": "2024-12-23T02:40:53.943Z", "content": "Looking forward to training code
", "rawMarkdown": "Looking forward to training code", "replies": [ { "id": 3080820, "postDate": "2024-12-25T19:17:34.867Z", "content": "Just pushed the code here.
", "rawMarkdown": "Just pushed the code [here](https://github.com/rbiswasfc/eedi-mining-misconceptions). ", "votes": 2 } ] }, { "id": 3074526, "postDate": "2024-12-17T18:56:14.323Z", "content": "Your explanation is excellent bro, Thanks.
", "rawMarkdown": "Your explanation is excellent bro, Thanks." }, { "id": 3142482, "postDate": "2025-03-06T12:36:23.377Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3141983, "postDate": "2025-03-06T02:42:58.807Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 3141984, "postDate": "2025-03-06T02:43:33.153Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3142013, "postDate": "2025-03-06T03:19:58.493Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 3078331, "postDate": "2024-12-22T07:02:24.763Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 3101726, "postDate": "2025-01-21T08:15:49.537Z", "content": "Hi, did you get the csv file ?
", "rawMarkdown": "Hi, did you get the csv file ?", "votes": 1 } ] }, { "id": 3077461, "postDate": "2024-12-21T02:03:29.063Z", "rawMarkdown": "", "isDeleted": true } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T03:47:50Z", "competition_id": 82695, "competition_slug": "eedi-mining-misconceptions-in-mathematics", "team_id": 12601869, "solution_url": "/competitions/eedi-mining-misconceptions-in-mathematics/writeups/mth-101-1st-place-detailed-solution" }