{
  "id": 551559,
  "title": "4th Place Solution",
  "content": "<p>First, I would like to express my gratitude to the host for organizing such an incredible competition. I also want to thank my teammates, <a href=\"https://www.kaggle.com/charmq\" target=\"_blank\">charmq</a> and <a href=\"https://www.kaggle.com/kami634\" target=\"_blank\">kami</a> for their collaboration.</p>\n<h2>Summary</h2>\n<ul>\n<li><strong>Approach to the Competition</strong><ul>\n<li>Since most misconceptions in the test set were anticipated to be absent in the train set, we focused on improving the accuracy for these unseen misconceptions.</li></ul></li>\n<li><strong>Solution Overview</strong><ul>\n<li><strong>Data Generation</strong><ul>\n<li>Data was generated using <code>Qwen2.5-72B-Instruct-AWQ</code>.</li>\n<li>The process specifically targeted misconceptions not present in the training dataset.</li></ul></li>\n<li><strong>Misconception Generation</strong><ul>\n<li>Used <code>Qwen2.5-32B-instruct-AWQ</code> to generate misconceptions from Questions, Answers, etc.</li></ul></li>\n<li><strong>Retriever</strong><ul>\n<li>Training<ul>\n<li>Created two models and fine-tuned them using LoRA.<ul>\n<li>Qwen2.5-14B-instruct</li>\n<li>Qwen2.5-32B-instruct</li></ul></li>\n<li>During training, in addition to the provided data such as QuestionText and AnswerText, misconceptions generated in the previous misconception generation phase were also added to the text.</li></ul></li>\n<li>Inference &amp; Retrieval:<ul>\n<li>Conducted retrieval by concatenating the outputs of Qwen2.5-14B-instruct and Qwen2.5-32B-instruct. The outputs were generated for several folds.</li>\n<li>For each Question-Answer pair, retrieved the following numbers of misconceptions:<ul>\n<li>25 most similar misconceptions from all misconceptions.</li>\n<li>15 most similar misconceptions specifically from those not present in the train set.</li>\n<li>Removed duplicates.</li></ul></li></ul></li></ul></li>\n<li><strong>Reranker</strong><ul>\n<li><strong>Training</strong><ul>\n<li>Fine-tuned multiple Qwen2.5-32B-instruct models using LoRA, adjusting the number of negative samples and adding generated data to the training dataset.</li></ul></li>\n<li><strong>Inferene</strong><ul>\n<li>Ensembled the LoRA components of the trained models to create three models.</li>\n<li>Finally, ensembled the outputs of these three models.</li></ul></li></ul></li>\n<li><strong>Post-Processing</strong><ul>\n<li>Adjusted the predictions for misconceptions that existed in the train set by reducing their scores:<ul>\n<li>Prediction Score * 0.40</li></ul></li></ul></li>\n<li>During data generation and inference (e.g., Retreival、Rerank), processes were accelerated using <code>vLLM</code></li></ul></li>\n</ul>\n<h2>Solution</h2>\n<h3>Validation Strategy</h3>\n<ul>\n<li>Group KFold (group = <code>QuestionId</code>)</li>\n<li>5 folds</li>\n<li>Evaluated not only the overall fold scores but also the scores for validation data extracted specifically for misconceptions not included in the training data.</li>\n</ul>\n<h3>Data Generation</h3>\n<ul>\n<li>Used <code>Qwen2.5-72B-Instruct-AWQ</code> to generate new questions, answer choices, and incorrect answers corresponding to each misconception that does not appear in the training dataset.</li>\n<li>Approximately 8000 questions were created, and 2500 of them were randomly sampled for training the reranker model.</li>\n<li>For each misconception, 100 randomly sampled questions from the training dataset were added to the prompt as input examples for data generation.</li>\n</ul>\n<p>The prompt for data generation is as follows, with the maximum token count being approximately 20000.</p>\n<pre><code>\n</code></pre>\n<p>Here are examples of the generated questions. The 72B model seems to have high question-generation abilities.</p>\n<pre><code>:  the circumference  a circle given the radius\n: \n :  the radius  a circle is \\(  \\) cm, what is the circumference  the circle?\n A : \\(  \\) cm\n B : \\(  \\) cm\n C : \\(  \\) cm\n D : \\(  \\) cm\n: B\n : A\n:  circumference is radius x pi\n</code></pre>\n<pre><code>:  algebraic fractions by identifying and cancelling common factors\n:   \n :  the following algebraic :\n\\[\n\\frac{6x^2y}{9xy^}\n\\]\n A : \\( \\frac{2x}{3y} \\)\n B : \\( \\frac{6x}{9y} \\)\n C : \\( \\frac{2xy}{3y^} \\)\n D : \\( \\frac{6x^}{9y^} \\)\n: A\n : B\n:  identify a common factor when simplifying algebraic fractions\n</code></pre>\n<p>These examples are mathematically valid, and it was crucial to include a large number of examples (100 cases) in the prompt to generate valid questions.</p>\n<h3><strong>Misconception Generation</strong></h3>\n<ul>\n<li>Used <code>Qwen2.5-32B-instruct</code> to generate misconceptions. The following prompt was used:</li>\n</ul>\n<pre><code>\n</code></pre>\n<h3><strong>Retriever</strong></h3>\n<ul>\n<li>The retrieval model was evaluated by checking not only MAP@25 but also the top-25 recall</li>\n<li>Retrieval models used for Final Submission<ul>\n<li>Ensemble of the following models<ul>\n<li><code>Qwen2.5-14B-instruct</code><ul>\n<li>Misconceptions generated during the Misconception Generation step were also added to the text</li>\n<li>Loss : MultipleNegativesRankingLoss</li>\n<li>Fine-tuned with LoRA:<ul>\n<li><code>LoRA_rank</code>: 32</li>\n<li><code>LoRA_alpha</code>: 64</li></ul></li>\n<li>Trained with 1 batch consisting of 1 positive sample and 47 negative samples for each question-answer pair.<ul>\n<li>Increasing the number of negatives was critical.</li>\n<li>Negatives used for training were restricted to those associated with positive samples within the training data.</li>\n<li>Negatives were selected randomly.</li></ul></li>\n<li>map 25 :  0.511</li>\n<li>recall 25 : 0.923</li></ul></li>\n<li><code>Qwen2.5-32B-instruct-GPTQ-Int4</code><ul>\n<li>Almost same as 14B</li>\n<li>Trained with 1 batch consisting of 1 positive sample and 255 negative samples for each question-answer pair.</li>\n<li>10 epochs (each fold took 10 hours on an A100).</li>\n<li>Due to the long inference time (50min/fold), only 2 folds were used for the final submission.</li>\n<li>map 25 : 0.554</li>\n<li>recall 25 : 0.926</li>\n<li>map 25 for generated data: ~0.7</li>\n<li>recall 25 for generated data: ~0.99</li></ul></li></ul></li></ul></li>\n<li><strong>Inference During Submission</strong><ul>\n<li><code>vllm</code> was utilized to speed up the embedding calculation. Since it could not be used directly, some modifications were made to its implementation to adapt it to our use case. This approach enabled efficient inference while maintaining accuracy.</li>\n<li>The embeddings from the above models were concatenated to generate the final representation.</li>\n<li>Number of Retrieved Items<ul>\n<li>25 items from all misconceptions.</li>\n<li>15 items from misconceptions not present in the training data:<ul>\n<li>This was prioritized because most misconceptions in the test set were predicted to be absent in the training data.</li></ul></li>\n<li>Duplicates between the above were removed.</li></ul></li></ul></li>\n<li>Retrieval for Rerank Candidate<ul>\n<li>The following model ensemble was used as candidates for reranking:<ul>\n<li>Salesforce/SFR-Embedding-2_R</li>\n<li>BAAI/bge-large-en-v1.5</li>\n<li>Alibaba-NLP/gte-large-en-v1.5</li></ul></li>\n<li>The recall top 25 is lower compared to those used for submission, so it was not used for submission.</li>\n<li>The rerank model trained with these candidates achieved a better public score.</li></ul></li>\n</ul>\n<h2>Reranker</h2>\n<ul>\n<li>Performed inference using three final models.</li>\n<li><strong>Model 1 &amp; Model 2</strong><ul>\n<li><code>Qwen2.5-32B-Instruct-GPTQ-Int4</code><ul>\n<li>Trained across 4 folds.</li>\n<li>Fine-tuned using LoRA.</li>\n<li>Used a ratio of 1 positive : 9 negatives during training.</li>\n<li>Limited negatives to those present in the positives.<ul>\n<li>This approach improved CV.</li></ul></li></ul></li>\n<li><strong>Fold 1 results:</strong><ul>\n<li>MAP@25: 0.653</li>\n<li>Evaluated using only misconceptions not present in the training data: MAP@25: 0.601</li></ul></li>\n<li><strong>Model 1:</strong><ul>\n<li>Ensembled the LoRA components of fold 1 and fold 2.</li></ul></li>\n<li><strong>Model 2:</strong><ul>\n<li>Ensembled the LoRA components of fold 3 and fold 4.</li></ul></li></ul></li>\n<li><strong>Model 3</strong><ul>\n<li><code>Qwen2.5-32B-Instruct-GPTQ-Int4</code><ul>\n<li>In addition to the competition data, 2500 generated samples were used for training.</li>\n<li>Trained across 2 folds and ensembled the LoRA components of them.</li>\n<li>Used a ratio of 1 positive : 19 negatives during training.</li>\n<li>2epochs (each fold took 10 hours on 4 x A100).</li></ul></li>\n<li><strong>Fold 1 results:</strong><ul>\n<li>MAP@25: 0.664</li>\n<li>Evaluated using only misconceptions not present in the training data: MAP@25: 0.605</li></ul></li></ul></li>\n</ul>\n<p>The private score was boosted by 0.01 through ensembling a reranker model (Model 3) trained with data generated by the 72B model.</p>\n<h2>PostProcess</h2>\n<ul>\n<li>Adjusted the predictions for misconceptions that existed in the train set by reducing their scores:<ul>\n<li>Prediction Score * 0.40<ul>\n<li>Other coefficients were tested, but 0.40 resulted in the highest public score.</li></ul></li></ul></li>\n</ul>\n<h2>code</h2>\n<p>takoi part : <a href=\"https://github.com/TakoiHirokazu/kaggle-Eedi-4th-solution\" target=\"_blank\">https://github.com/TakoiHirokazu/kaggle-Eedi-4th-solution</a><br>\ncharmq part : <a href=\"https://github.com/charmq00/kaggle_eedi_public\" target=\"_blank\">https://github.com/charmq00/kaggle_eedi_public</a><br>\nInference : <a href=\"https://www.kaggle.com/code/charmq/eedi-pp040-ret15-exp345-341-348-multi-32b-ret-c015\" target=\"_blank\">https://www.kaggle.com/code/charmq/eedi-pp040-ret15-exp345-341-348-multi-32b-ret-c015</a></p>",
  "comments": [
    {
      "id": 3071936,
      "author_name": "Camaro",
      "author_url": "",
      "post_date": "2024-12-14T14:37:29.033000",
      "content": "<p>Congrats on 4th place, and thank you for the detailed summary!</p>\n<p>I have a few questions:</p>\n<ol>\n<li><p>You mentioned that approximately 8,000 questions were created, but only 2,500 were randomly sampled for training the reranker model. Why was only 2,500 selected? Was using a smaller subset more effective than using all 8,000?</p></li>\n<li><p>Regarding the Misconception Generation process, is it correct to assume that you did not fine-tune Qwen2.5-32B-instruct for this task?</p></li>\n<li><p>About the training setup:</p>\n<ul>\n<li>You said each batch consisted of one positive sample and 47 negative samples for each question-answer pair. Could you clarify the batch size here? Do you mean that the batch size is effectively one sample in this context?</li></ul></li>\n<li><p>You mentioned modifying vllm to speed up embedding calculation since it could not be used directly. Do you have any plans to release this modified implementation (or ideally, the entire codebase)?</p></li>\n<li><p>You also noted that restricting negatives to those present in the positives improved cv. Did this approach also lead to improvements on the LB?</p></li>\n</ol>",
      "votes": 3,
      "replies": [
        {
          "id": 3072611,
          "author_name": "Takoi",
          "author_url": "",
          "post_date": "2024-12-15T12:50:24.293000",
          "content": "<p>Thank you! Here are the responses to your questions:</p>\n<ol>\n<li><p>Using all 8,000 questions would take too much time for training, so only 2,500 were selected to reduce the training time.</p></li>\n<li><p>That is correct. Fine-tuning Qwen2.5-32B-instruct was not performed. Initial tests without fine-tuning showed improvements in both CV and public scores, so the decision was made to proceed without it.</p></li>\n<li><p>In the discussion, the focus was on one question-answer within the batch, explaining the number of negatives for that specific case. Apologies if the explanation was unclear.<br>\nThe batch size is 6. Specifically, six question-answers with their corresponding positive misconceptions are prepared using a data loader. Additionally, 42 misconceptions are added. From the perspective of a single question-answer, five of the other positive misconceptions within the same batch are treated as negatives, resulting in a total of 47 negatives.</p></li>\n<li><p>At this time, there are no plans to share the modified implementation or the entire codebase with anyone other than the competition host.</p></li>\n<li><p>For retrieval, submissions without applying the restriction mentioned were not made, so the impact is unclear. The decision was based solely on CV results. Regarding reranking, both CV and public scores improved, and the private score also showed improvement. Comparing a single rerank model, both public and private scores improved by approximately 0.025.</p></li>\n</ol>",
          "votes": 1,
          "replies": [
            {
              "id": 3073383,
              "author_name": "w2",
              "author_url": "",
              "post_date": "2024-12-16T12:47:07.513000",
              "content": "<p>May I ask what 'Limited negatives to those present in the positives' mean? I'm confused about it but it boosts a lot. Really appreciate it if you can share your insights.</p>",
              "votes": 1,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 3073553,
      "author_name": "Santiago Mota",
      "author_url": "",
      "post_date": "2024-12-16T15:49:55.577000",
      "content": "<p>Congrats and thanks for the detailed explanation </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 3073211,
      "author_name": "CsOH",
      "author_url": "",
      "post_date": "2024-12-16T07:23:35.507000",
      "content": "<p>Congratulation!  Thanks for sharing your solution detailedly. I've learned a lot from your solution.<br>\nOne question is that how you ensembled the LoRA components of fold 1 and fold 2. By simply average the parameter?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 3073375,
          "author_name": "Takoi",
          "author_url": "",
          "post_date": "2024-12-16T12:22:49.100000",
          "content": "<p>Thank you.<br>\nYes, that's correct. I simply averaged the parameters.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 3073191,
      "author_name": "MrSimple",
      "author_url": "",
      "post_date": "2024-12-16T06:39:15.647000",
      "content": "<p>Thanks for sharing!</p>",
      "votes": 1,
      "replies": []
    }
  ],
  "comment_count": 7,
  "messages": [
    {
      "id": 3071538,
      "postDate": "2024-12-13T22:33:33.317Z",
      "content": "<p>First, I would like to express my gratitude to the host for organizing such an incredible competition. I also want to thank my teammates, <a href=\"https://www.kaggle.com/charmq\" target=\"_blank\">charmq</a> and <a href=\"https://www.kaggle.com/kami634\" target=\"_blank\">kami</a> for their collaboration.</p>\n<h2>Summary</h2>\n<ul>\n<li><strong>Approach to the Competition</strong><ul>\n<li>Since most misconceptions in the test set were anticipated to be absent in the train set, we focused on improving the accuracy for these unseen misconceptions.</li></ul></li>\n<li><strong>Solution Overview</strong><ul>\n<li><strong>Data Generation</strong><ul>\n<li>Data was generated using <code>Qwen2.5-72B-Instruct-AWQ</code>.</li>\n<li>The process specifically targeted misconceptions not present in the training dataset.</li></ul></li>\n<li><strong>Misconception Generation</strong><ul>\n<li>Used <code>Qwen2.5-32B-instruct-AWQ</code> to generate misconceptions from Questions, Answers, etc.</li></ul></li>\n<li><strong>Retriever</strong><ul>\n<li>Training<ul>\n<li>Created two models and fine-tuned them using LoRA.<ul>\n<li>Qwen2.5-14B-instruct</li>\n<li>Qwen2.5-32B-instruct</li></ul></li>\n<li>During training, in addition to the provided data such as QuestionText and AnswerText, misconceptions generated in the previous misconception generation phase were also added to the text.</li></ul></li>\n<li>Inference &amp; Retrieval:<ul>\n<li>Conducted retrieval by concatenating the outputs of Qwen2.5-14B-instruct and Qwen2.5-32B-instruct. The outputs were generated for several folds.</li>\n<li>For each Question-Answer pair, retrieved the following numbers of misconceptions:<ul>\n<li>25 most similar misconceptions from all misconceptions.</li>\n<li>15 most similar misconceptions specifically from those not present in the train set.</li>\n<li>Removed duplicates.</li></ul></li></ul></li></ul></li>\n<li><strong>Reranker</strong><ul>\n<li><strong>Training</strong><ul>\n<li>Fine-tuned multiple Qwen2.5-32B-instruct models using LoRA, adjusting the number of negative samples and adding generated data to the training dataset.</li></ul></li>\n<li><strong>Inferene</strong><ul>\n<li>Ensembled the LoRA components of the trained models to create three models.</li>\n<li>Finally, ensembled the outputs of these three models.</li></ul></li></ul></li>\n<li><strong>Post-Processing</strong><ul>\n<li>Adjusted the predictions for misconceptions that existed in the train set by reducing their scores:<ul>\n<li>Prediction Score * 0.40</li></ul></li></ul></li>\n<li>During data generation and inference (e.g., Retreival、Rerank), processes were accelerated using <code>vLLM</code></li></ul></li>\n</ul>\n<h2>Solution</h2>\n<h3>Validation Strategy</h3>\n<ul>\n<li>Group KFold (group = <code>QuestionId</code>)</li>\n<li>5 folds</li>\n<li>Evaluated not only the overall fold scores but also the scores for validation data extracted specifically for misconceptions not included in the training data.</li>\n</ul>\n<h3>Data Generation</h3>\n<ul>\n<li>Used <code>Qwen2.5-72B-Instruct-AWQ</code> to generate new questions, answer choices, and incorrect answers corresponding to each misconception that does not appear in the training dataset.</li>\n<li>Approximately 8000 questions were created, and 2500 of them were randomly sampled for training the reranker model.</li>\n<li>For each misconception, 100 randomly sampled questions from the training dataset were added to the prompt as input examples for data generation.</li>\n</ul>\n<p>The prompt for data generation is as follows, with the maximum token count being approximately 20000.</p>\n<pre><code>\n</code></pre>\n<p>Here are examples of the generated questions. The 72B model seems to have high question-generation abilities.</p>\n<pre><code>:  the circumference  a circle given the radius\n: \n :  the radius  a circle is \\(  \\) cm, what is the circumference  the circle?\n A : \\(  \\) cm\n B : \\(  \\) cm\n C : \\(  \\) cm\n D : \\(  \\) cm\n: B\n : A\n:  circumference is radius x pi\n</code></pre>\n<pre><code>:  algebraic fractions by identifying and cancelling common factors\n:   \n :  the following algebraic :\n\\[\n\\frac{6x^2y}{9xy^}\n\\]\n A : \\( \\frac{2x}{3y} \\)\n B : \\( \\frac{6x}{9y} \\)\n C : \\( \\frac{2xy}{3y^} \\)\n D : \\( \\frac{6x^}{9y^} \\)\n: A\n : B\n:  identify a common factor when simplifying algebraic fractions\n</code></pre>\n<p>These examples are mathematically valid, and it was crucial to include a large number of examples (100 cases) in the prompt to generate valid questions.</p>\n<h3><strong>Misconception Generation</strong></h3>\n<ul>\n<li>Used <code>Qwen2.5-32B-instruct</code> to generate misconceptions. The following prompt was used:</li>\n</ul>\n<pre><code>\n</code></pre>\n<h3><strong>Retriever</strong></h3>\n<ul>\n<li>The retrieval model was evaluated by checking not only MAP@25 but also the top-25 recall</li>\n<li>Retrieval models used for Final Submission<ul>\n<li>Ensemble of the following models<ul>\n<li><code>Qwen2.5-14B-instruct</code><ul>\n<li>Misconceptions generated during the Misconception Generation step were also added to the text</li>\n<li>Loss : MultipleNegativesRankingLoss</li>\n<li>Fine-tuned with LoRA:<ul>\n<li><code>LoRA_rank</code>: 32</li>\n<li><code>LoRA_alpha</code>: 64</li></ul></li>\n<li>Trained with 1 batch consisting of 1 positive sample and 47 negative samples for each question-answer pair.<ul>\n<li>Increasing the number of negatives was critical.</li>\n<li>Negatives used for training were restricted to those associated with positive samples within the training data.</li>\n<li>Negatives were selected randomly.</li></ul></li>\n<li>map 25 :  0.511</li>\n<li>recall 25 : 0.923</li></ul></li>\n<li><code>Qwen2.5-32B-instruct-GPTQ-Int4</code><ul>\n<li>Almost same as 14B</li>\n<li>Trained with 1 batch consisting of 1 positive sample and 255 negative samples for each question-answer pair.</li>\n<li>10 epochs (each fold took 10 hours on an A100).</li>\n<li>Due to the long inference time (50min/fold), only 2 folds were used for the final submission.</li>\n<li>map 25 : 0.554</li>\n<li>recall 25 : 0.926</li>\n<li>map 25 for generated data: ~0.7</li>\n<li>recall 25 for generated data: ~0.99</li></ul></li></ul></li></ul></li>\n<li><strong>Inference During Submission</strong><ul>\n<li><code>vllm</code> was utilized to speed up the embedding calculation. Since it could not be used directly, some modifications were made to its implementation to adapt it to our use case. This approach enabled efficient inference while maintaining accuracy.</li>\n<li>The embeddings from the above models were concatenated to generate the final representation.</li>\n<li>Number of Retrieved Items<ul>\n<li>25 items from all misconceptions.</li>\n<li>15 items from misconceptions not present in the training data:<ul>\n<li>This was prioritized because most misconceptions in the test set were predicted to be absent in the training data.</li></ul></li>\n<li>Duplicates between the above were removed.</li></ul></li></ul></li>\n<li>Retrieval for Rerank Candidate<ul>\n<li>The following model ensemble was used as candidates for reranking:<ul>\n<li>Salesforce/SFR-Embedding-2_R</li>\n<li>BAAI/bge-large-en-v1.5</li>\n<li>Alibaba-NLP/gte-large-en-v1.5</li></ul></li>\n<li>The recall top 25 is lower compared to those used for submission, so it was not used for submission.</li>\n<li>The rerank model trained with these candidates achieved a better public score.</li></ul></li>\n</ul>\n<h2>Reranker</h2>\n<ul>\n<li>Performed inference using three final models.</li>\n<li><strong>Model 1 &amp; Model 2</strong><ul>\n<li><code>Qwen2.5-32B-Instruct-GPTQ-Int4</code><ul>\n<li>Trained across 4 folds.</li>\n<li>Fine-tuned using LoRA.</li>\n<li>Used a ratio of 1 positive : 9 negatives during training.</li>\n<li>Limited negatives to those present in the positives.<ul>\n<li>This approach improved CV.</li></ul></li></ul></li>\n<li><strong>Fold 1 results:</strong><ul>\n<li>MAP@25: 0.653</li>\n<li>Evaluated using only misconceptions not present in the training data: MAP@25: 0.601</li></ul></li>\n<li><strong>Model 1:</strong><ul>\n<li>Ensembled the LoRA components of fold 1 and fold 2.</li></ul></li>\n<li><strong>Model 2:</strong><ul>\n<li>Ensembled the LoRA components of fold 3 and fold 4.</li></ul></li></ul></li>\n<li><strong>Model 3</strong><ul>\n<li><code>Qwen2.5-32B-Instruct-GPTQ-Int4</code><ul>\n<li>In addition to the competition data, 2500 generated samples were used for training.</li>\n<li>Trained across 2 folds and ensembled the LoRA components of them.</li>\n<li>Used a ratio of 1 positive : 19 negatives during training.</li>\n<li>2epochs (each fold took 10 hours on 4 x A100).</li></ul></li>\n<li><strong>Fold 1 results:</strong><ul>\n<li>MAP@25: 0.664</li>\n<li>Evaluated using only misconceptions not present in the training data: MAP@25: 0.605</li></ul></li></ul></li>\n</ul>\n<p>The private score was boosted by 0.01 through ensembling a reranker model (Model 3) trained with data generated by the 72B model.</p>\n<h2>PostProcess</h2>\n<ul>\n<li>Adjusted the predictions for misconceptions that existed in the train set by reducing their scores:<ul>\n<li>Prediction Score * 0.40<ul>\n<li>Other coefficients were tested, but 0.40 resulted in the highest public score.</li></ul></li></ul></li>\n</ul>\n<h2>code</h2>\n<p>takoi part : <a href=\"https://github.com/TakoiHirokazu/kaggle-Eedi-4th-solution\" target=\"_blank\">https://github.com/TakoiHirokazu/kaggle-Eedi-4th-solution</a><br>\ncharmq part : <a href=\"https://github.com/charmq00/kaggle_eedi_public\" target=\"_blank\">https://github.com/charmq00/kaggle_eedi_public</a><br>\nInference : <a href=\"https://www.kaggle.com/code/charmq/eedi-pp040-ret15-exp345-341-348-multi-32b-ret-c015\" target=\"_blank\">https://www.kaggle.com/code/charmq/eedi-pp040-ret15-exp345-341-348-multi-32b-ret-c015</a></p>",
      "rawMarkdown": "First, I would like to express my gratitude to the host for organizing such an incredible competition. I also want to thank my teammates, [charmq](https://www.kaggle.com/charmq) and [kami](https://www.kaggle.com/kami634) for their collaboration.\n\n## Summary\n\n- **Approach to the Competition**\n    - Since most misconceptions in the test set were anticipated to be absent in the train set, we focused on improving the accuracy for these unseen misconceptions.\n- **Solution Overview**\n    - **Data Generation**\n        - Data was generated using `Qwen2.5-72B-Instruct-AWQ`.\n        - The process specifically targeted misconceptions not present in the training dataset.\n    - **Misconception Generation**\n        - Used `Qwen2.5-32B-instruct-AWQ` to generate misconceptions from Questions, Answers, etc.\n    - **Retriever**\n        - Training\n            - Created two models and fine-tuned them using LoRA.\n                - Qwen2.5-14B-instruct\n                - Qwen2.5-32B-instruct\n            - During training, in addition to the provided data such as QuestionText and AnswerText, misconceptions generated in the previous misconception generation phase were also added to the text.\n        - Inference & Retrieval:\n            - Conducted retrieval by concatenating the outputs of Qwen2.5-14B-instruct and Qwen2.5-32B-instruct. The outputs were generated for several folds.\n            - For each Question-Answer pair, retrieved the following numbers of misconceptions:\n                - 25 most similar misconceptions from all misconceptions.\n                - 15 most similar misconceptions specifically from those not present in the train set.\n                - Removed duplicates.\n    - **Reranker**\n        - **Training**\n            - Fine-tuned multiple Qwen2.5-32B-instruct models using LoRA, adjusting the number of negative samples and adding generated data to the training dataset.\n        - **Inferene**\n            - Ensembled the LoRA components of the trained models to create three models.\n            - Finally, ensembled the outputs of these three models.\n    - **Post-Processing**\n        - Adjusted the predictions for misconceptions that existed in the train set by reducing their scores:\n            - Prediction Score * 0.40\n    - During data generation and inference (e.g., Retreival、Rerank), processes were accelerated using `vLLM`\n\n## Solution\n\n### Validation Strategy\n\n- Group KFold (group = `QuestionId`)\n- 5 folds\n- Evaluated not only the overall fold scores but also the scores for validation data extracted specifically for misconceptions not included in the training data.\n\n### Data Generation\n\n- Used `Qwen2.5-72B-Instruct-AWQ` to generate new questions, answer choices, and incorrect answers corresponding to each misconception that does not appear in the training dataset.\n- Approximately 8000 questions were created, and 2500 of them were randomly sampled for training the reranker model.\n- For each misconception, 100 randomly sampled questions from the training dataset were added to the prompt as input examples for data generation.\n\nThe prompt for data generation is as follows, with the maximum token count being approximately 20000.\n\n```jsx\n\"\"\"You are an expert in mathematics. \nRefer to the examples below to create new problem with given misconception. \n\nMisconception: {MisconceptionText}\n\nThe output format shoud be below.\n\n\nConstructName:  \nSubjectName: \nMath problem: \nAnswer A text: \nAnswer B text: \nAnswer C text: \nAnswer D text: \nAnswer: \nIncorrect answer: \n\n\nThe examples are below\n\nExample 1: \n\nConstructName: {ConstructName_1}\nSubjectName: {SubjectName_1}\nMath problem: {QuestionText_1}\nAnswer A text: {AnswerAText_1}\nAnswer B text: {AnswerBText_1}\nAnswer C text: {AnswerCText_1}\nAnswer D text: {AnswerDText_1}\nAnswer: {CorrectAnswer_1}\nIncorrect answer: {IncorrectAnswer_1}\nMisconception: {MisconceptionText_1} \n\n...\n\nExample 100: \n\nConstructName: {ConstructName_100}\nSubjectName: {SubjectName_100}\nMath problem: {QuestionText_100}\nAnswer A text: {AnswerAText_100}\nAnswer B text: {AnswerBText_100}\nAnswer C text: {AnswerCText_100}\nAnswer D text: {AnswerDText_100}\nAnswer: {CorrectAnswer_100}\nIncorrect answer: {IncorrectAnswer_100}\nMisconception: {MisconceptionText_100} \n\n\"\"\"\n```\n\nHere are examples of the generated questions. The 72B model seems to have high question-generation abilities.\n\n```jsx\nConstructName: Calculate the circumference of a circle given the radius\nSubjectName: Circles\nMath problem: If the radius of a circle is \\( 7 \\) cm, what is the circumference of the circle?\nAnswer A text: \\( 22 \\) cm\nAnswer B text: \\( 44 \\) cm\nAnswer C text: \\( 14 \\) cm\nAnswer D text: \\( 154 \\) cm\nAnswer: B\nIncorrect answer: A\nMisconception: Thinks circumference is radius x pi\n```\n\n```jsx\nConstructName: Simplify algebraic fractions by identifying and cancelling common factors\nSubjectName: Simplifying Algebraic Fractions\nMath problem: Simplify the following algebraic fraction:\n\\[\n\\frac{6x^2y}{9xy^2}\n\\]\nAnswer A text: \\( \\frac{2x}{3y} \\)\nAnswer B text: \\( \\frac{6x}{9y} \\)\nAnswer C text: \\( \\frac{2xy}{3y^2} \\)\nAnswer D text: \\( \\frac{6x^2}{9y^2} \\)\nAnswer: A\nIncorrect answer: B\nMisconception: Cannot identify a common factor when simplifying algebraic fractions\n```\n\nThese examples are mathematically valid, and it was crucial to include a large number of examples (100 cases) in the prompt to generate valid questions.\n\n### **Misconception Generation**\n\n- Used `Qwen2.5-32B-instruct` to generate misconceptions. The following prompt was used:\n\n```jsx\n\"\"\"You are an expert in mathematics.\nRefer to the examples below to identify and describe the misconception that led to the incorrect answer.\nExample1\nConstructName: Recognise and use efficient methods for mental multiplication\nSubjectName: Mental Multiplication and Division\nMath problem: Tom and Katie are discussing ways to calculate\\\\( 21\\\\times 12\\\\) mentally. Tom does\\\\( 12\\\\times 7\\\\) and then multiplies his answer by\\\\( 3\\\\); Katie does\\\\( 21\\\\times 6\\\\) and then doubles her answer. Who would get the correct answer?\nIncorrect answer: Only Katie\nMisconception: Does not correctly apply the distributive property of multiplication\n\nExample2\nConstructName: Multiply a decimal by an integer\nSubjectName: Mental Multiplication and Division\nMath problem:\\\\( 9.4\\\\times 50=\\\\)\nIncorrect answer:\\\\( 4700\\\\)\nMisconception: When multiplying a decimal by an integer, ignores decimal point and just multiplies the digits\n\nConstructName:{ConstructName}\nSubjectName:{SubjectName}Math problem:{QuestionText}\nIncorrect answer:{AnswerText}\nMisconception:\n\"\"\"\n```\n\n### **Retriever**\n\n- The retrieval model was evaluated by checking not only MAP@25 but also the top-25 recall\n- Retrieval models used for Final Submission\n    - Ensemble of the following models\n        - `Qwen2.5-14B-instruct`\n            - Misconceptions generated during the Misconception Generation step were also added to the text\n            - Loss : MultipleNegativesRankingLoss\n            - Fine-tuned with LoRA:\n                - `LoRA_rank`: 32\n                - `LoRA_alpha`: 64\n            - Trained with 1 batch consisting of 1 positive sample and 47 negative samples for each question-answer pair.\n                - Increasing the number of negatives was critical.\n                - Negatives used for training were restricted to those associated with positive samples within the training data.\n                - Negatives were selected randomly.\n            - map 25 :  0.511\n            - recall 25 : 0.923\n        - `Qwen2.5-32B-instruct-GPTQ-Int4`\n            - Almost same as 14B\n            - Trained with 1 batch consisting of 1 positive sample and 255 negative samples for each question-answer pair.\n            - 10 epochs (each fold took 10 hours on an A100).\n            - Due to the long inference time (50min/fold), only 2 folds were used for the final submission.\n            - map 25 : 0.554\n            - recall 25 : 0.926\n            - map 25 for generated data: ~0.7\n            - recall 25 for generated data: ~0.99\n- **Inference During Submission**\n    - `vllm` was utilized to speed up the embedding calculation. Since it could not be used directly, some modifications were made to its implementation to adapt it to our use case. This approach enabled efficient inference while maintaining accuracy.\n    - The embeddings from the above models were concatenated to generate the final representation.\n    - Number of Retrieved Items\n        - 25 items from all misconceptions.\n        - 15 items from misconceptions not present in the training data:\n            - This was prioritized because most misconceptions in the test set were predicted to be absent in the training data.\n        - Duplicates between the above were removed.\n- Retrieval for Rerank Candidate\n    - The following model ensemble was used as candidates for reranking:\n        - Salesforce/SFR-Embedding-2_R\n        - BAAI/bge-large-en-v1.5\n        - Alibaba-NLP/gte-large-en-v1.5\n    - The recall top 25 is lower compared to those used for submission, so it was not used for submission.\n    - The rerank model trained with these candidates achieved a better public score.\n## Reranker\n\n- Performed inference using three final models.\n- **Model 1 & Model 2**\n    - `Qwen2.5-32B-Instruct-GPTQ-Int4`\n        - Trained across 4 folds.\n        - Fine-tuned using LoRA.\n        - Used a ratio of 1 positive : 9 negatives during training.\n        - Limited negatives to those present in the positives.\n            - This approach improved CV.\n    - **Fold 1 results:**\n        - MAP@25: 0.653\n        - Evaluated using only misconceptions not present in the training data: MAP@25: 0.601\n    - **Model 1:**\n        - Ensembled the LoRA components of fold 1 and fold 2.\n    - **Model 2:**\n        - Ensembled the LoRA components of fold 3 and fold 4.\n- **Model 3**\n    - `Qwen2.5-32B-Instruct-GPTQ-Int4`\n        - In addition to the competition data, 2500 generated samples were used for training.\n        - Trained across 2 folds and ensembled the LoRA components of them.\n        - Used a ratio of 1 positive : 19 negatives during training.\n        - 2epochs (each fold took 10 hours on 4 x A100).\n    - **Fold 1 results:**\n        - MAP@25: 0.664\n        - Evaluated using only misconceptions not present in the training data: MAP@25: 0.605\n\nThe private score was boosted by 0.01 through ensembling a reranker model (Model 3) trained with data generated by the 72B model.\n\n## PostProcess\n\n- Adjusted the predictions for misconceptions that existed in the train set by reducing their scores:\n    - Prediction Score * 0.40\n        - Other coefficients were tested, but 0.40 resulted in the highest public score.\n\n## code\ntakoi part : https://github.com/TakoiHirokazu/kaggle-Eedi-4th-solution\ncharmq part : https://github.com/charmq00/kaggle_eedi_public\nInference : https://www.kaggle.com/code/charmq/eedi-pp040-ret15-exp345-341-348-multi-32b-ret-c015",
      "votes": 36
    },
    {
      "id": 3071936,
      "postDate": "2024-12-14T14:37:29.033Z",
      "content": "<p>Congrats on 4th place, and thank you for the detailed summary!</p>\n<p>I have a few questions:</p>\n<ol>\n<li><p>You mentioned that approximately 8,000 questions were created, but only 2,500 were randomly sampled for training the reranker model. Why was only 2,500 selected? Was using a smaller subset more effective than using all 8,000?</p></li>\n<li><p>Regarding the Misconception Generation process, is it correct to assume that you did not fine-tune Qwen2.5-32B-instruct for this task?</p></li>\n<li><p>About the training setup:</p>\n<ul>\n<li>You said each batch consisted of one positive sample and 47 negative samples for each question-answer pair. Could you clarify the batch size here? Do you mean that the batch size is effectively one sample in this context?</li></ul></li>\n<li><p>You mentioned modifying vllm to speed up embedding calculation since it could not be used directly. Do you have any plans to release this modified implementation (or ideally, the entire codebase)?</p></li>\n<li><p>You also noted that restricting negatives to those present in the positives improved cv. Did this approach also lead to improvements on the LB?</p></li>\n</ol>",
      "rawMarkdown": "Congrats on 4th place, and thank you for the detailed summary!\n\nI have a few questions:\n\n1. You mentioned that approximately 8,000 questions were created, but only 2,500 were randomly sampled for training the reranker model. Why was only 2,500 selected? Was using a smaller subset more effective than using all 8,000?\n\n2. Regarding the Misconception Generation process, is it correct to assume that you did not fine-tune Qwen2.5-32B-instruct for this task?\n\n3. About the training setup:\n   - You said each batch consisted of one positive sample and 47 negative samples for each question-answer pair. Could you clarify the batch size here? Do you mean that the batch size is effectively one sample in this context?\n\n4. You mentioned modifying vllm to speed up embedding calculation since it could not be used directly. Do you have any plans to release this modified implementation (or ideally, the entire codebase)?\n\n5. You also noted that restricting negatives to those present in the positives improved cv. Did this approach also lead to improvements on the LB?\n",
      "votes": 3,
      "replies": [
        {
          "id": 3072611,
          "postDate": "2024-12-15T12:50:24.293Z",
          "content": "<p>Thank you! Here are the responses to your questions:</p>\n<ol>\n<li><p>Using all 8,000 questions would take too much time for training, so only 2,500 were selected to reduce the training time.</p></li>\n<li><p>That is correct. Fine-tuning Qwen2.5-32B-instruct was not performed. Initial tests without fine-tuning showed improvements in both CV and public scores, so the decision was made to proceed without it.</p></li>\n<li><p>In the discussion, the focus was on one question-answer within the batch, explaining the number of negatives for that specific case. Apologies if the explanation was unclear.<br>\nThe batch size is 6. Specifically, six question-answers with their corresponding positive misconceptions are prepared using a data loader. Additionally, 42 misconceptions are added. From the perspective of a single question-answer, five of the other positive misconceptions within the same batch are treated as negatives, resulting in a total of 47 negatives.</p></li>\n<li><p>At this time, there are no plans to share the modified implementation or the entire codebase with anyone other than the competition host.</p></li>\n<li><p>For retrieval, submissions without applying the restriction mentioned were not made, so the impact is unclear. The decision was based solely on CV results. Regarding reranking, both CV and public scores improved, and the private score also showed improvement. Comparing a single rerank model, both public and private scores improved by approximately 0.025.</p></li>\n</ol>",
          "rawMarkdown": "Thank you! Here are the responses to your questions:\n\n1. Using all 8,000 questions would take too much time for training, so only 2,500 were selected to reduce the training time.\n\n2. That is correct. Fine-tuning Qwen2.5-32B-instruct was not performed. Initial tests without fine-tuning showed improvements in both CV and public scores, so the decision was made to proceed without it.\n\n3. In the discussion, the focus was on one question-answer within the batch, explaining the number of negatives for that specific case. Apologies if the explanation was unclear.\nThe batch size is 6. Specifically, six question-answers with their corresponding positive misconceptions are prepared using a data loader. Additionally, 42 misconceptions are added. From the perspective of a single question-answer, five of the other positive misconceptions within the same batch are treated as negatives, resulting in a total of 47 negatives.\n\n4. At this time, there are no plans to share the modified implementation or the entire codebase with anyone other than the competition host.\n\n5. For retrieval, submissions without applying the restriction mentioned were not made, so the impact is unclear. The decision was based solely on CV results. Regarding reranking, both CV and public scores improved, and the private score also showed improvement. Comparing a single rerank model, both public and private scores improved by approximately 0.025.",
          "votes": 1,
          "replies": [
            {
              "id": 3073383,
              "postDate": "2024-12-16T12:47:07.513Z",
              "content": "<p>May I ask what 'Limited negatives to those present in the positives' mean? I'm confused about it but it boosts a lot. Really appreciate it if you can share your insights.</p>",
              "rawMarkdown": "May I ask what 'Limited negatives to those present in the positives' mean? I'm confused about it but it boosts a lot. Really appreciate it if you can share your insights.",
              "votes": 1
            }
          ]
        }
      ]
    },
    {
      "id": 3073553,
      "postDate": "2024-12-16T15:49:55.577Z",
      "content": "<p>Congrats and thanks for the detailed explanation </p>",
      "rawMarkdown": "Congrats and thanks for the detailed explanation ",
      "votes": 1
    },
    {
      "id": 3073211,
      "postDate": "2024-12-16T07:23:35.507Z",
      "content": "<p>Congratulation!  Thanks for sharing your solution detailedly. I've learned a lot from your solution.<br>\nOne question is that how you ensembled the LoRA components of fold 1 and fold 2. By simply average the parameter?</p>",
      "rawMarkdown": "Congratulation!  Thanks for sharing your solution detailedly. I've learned a lot from your solution.\nOne question is that how you ensembled the LoRA components of fold 1 and fold 2. By simply average the parameter?",
      "replies": [
        {
          "id": 3073375,
          "postDate": "2024-12-16T12:22:49.100Z",
          "content": "<p>Thank you.<br>\nYes, that's correct. I simply averaged the parameters.</p>",
          "rawMarkdown": "Thank you.\nYes, that's correct. I simply averaged the parameters."
        }
      ]
    },
    {
      "id": 3073191,
      "postDate": "2024-12-16T06:39:15.647Z",
      "content": "<p>Thanks for sharing!</p>",
      "rawMarkdown": "Thanks for sharing!",
      "votes": 1
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-20T03:48:20Z",
  "competition_id": 82695,
  "competition_slug": "eedi-mining-misconceptions-in-mathematics",
  "team_id": 12686681,
  "solution_url": "/competitions/eedi-mining-misconceptions-in-mathematics/writeups/takoi-charmq-kami-4th-place-solution"
}