{
  "id": 518543,
  "title": "53rd Place Solution",
  "content": "<p>Our team struggled to improve our LB score and it was a very difficult competition. We were lucky to get a silver medal in the big shakeup. I would like to thank all the Kagglers who shared their notebooks and discussions!</p>\n<p>Here is our <a href=\"https://www.kaggle.com/code/takeshimorimura/53th-place-solution-aimo-progress-prize-1\" target=\"_blank\">code</a>. The base notebook is <a href=\"https://www.kaggle.com/anrenk\" target=\"_blank\">@anrenk</a>'s <a href=\"https://www.kaggle.com/code/anrenk/aimo-llm-class-deepseek\" target=\"_blank\">AIMO LLM Class DeepSeek</a></p>\n<h2>Improvements</h2>\n<ul>\n<li>Referring to <a href=\"https://www.kaggle.com/competitions/ai-mathematical-olympiad-prize/discussion/509742\" target=\"_blank\">this discussion</a>, <strong>we implemented asynchronous inference using vLLM to maximize the number of <code>n_repetitions</code></strong>.<ul>\n<li>vLLM leverages HW performance better than regular Transformers, enabling very fast inference. In a rough benchmark, vLLM was 1.5 times faster than regular Transformers under the same conditions, and asynchronous inference using AsyncLLMEngine was nearly 6 times faster.</li></ul></li>\n<li>Like the base notebook, we used Deepseek-math's Code and Text answers, but instead of simply selecting the answer with the most responses, we created <strong>multiple-choice questions from the top 5 answers, had Deepseek-math solve them again, and selected the answer based on the original number of responses and the new results</strong>.<ul>\n<li>Example prompt for solving multiple-choice questions:</li></ul></li>\n</ul>\n<pre><code>The   a math problem  you  solve, along  the answer choices.\nProblem: \"There exists a unique increasing geometric sequence of five 2-digit positive integers. What is their sum?\"\n: \"A) 496, B) 310, C) 452, D) 242, E) 500\"\nChoose the letter  the  that best answers the problem,  provide your answer  \\boxed{}.\n</code></pre>\n<ul>\n<li>We also fixed minor bugs in <code>process_text_output</code> and <code>naive_parse</code>, and added error handling from the original notebook. <strong>These seem to have improved the robustness of the score.</strong><ul>\n<li>If the code output by LLM results in an error, result_output is also set to -1.</li>\n<li>I modified the naive_parse function to extract the string \"answer\" and apply the function to the string that follows it (if \"answer\" is not found, result_output is set to -1).</li>\n<li>In the process_code function, I have modified it so that if the code output by the LLM results in a timeout error, this is explicitly returned to the LLM.</li></ul></li>\n</ul>\n<h2>Submission Selection and Validation</h2>\n<ul>\n<li>For validation, we tried MATH level5, AIME83-86, and AIME2023-2024, but ultimately <strong>used AIME83-86, which seemed to have the best correlation with the Public LB.</strong></li>\n<li>The final submissions were the two with the highest validation scores and Public LB.</li>\n</ul>\n<h2>Other things tried but not adopted</h2>\n<ul>\n<li>Considered creating a LightGBM model to <strong>predict the problem type</strong> (\"Algebra\", \"Counting &amp; Probability\", \"Geometry\", \"Intermediate Algebra\", \"Number Theory\", \"Precalculus\", \"Prealgebra\") from the <code>last_hidden_state</code> of deepseek-math using the MATH dataset as training data, and changing parameters like <code>n_repetitions</code> for each problem type.<ul>\n<li>The model was able to predict the type with about <strong>80% accuracy</strong>, but the use of vLLM made it difficult to obtain the <code>hidden_state</code>, so it was not adopted.</li></ul></li>\n<li>Using formatters like <code>ruff</code> or <code>black</code> to fix code formatting and typical errors.<ul>\n<li>This idea came to us near the end of the competition, and we couldn't verify how to effectively incorporate it into the prompt, so we decided not to use it.</li></ul></li>\n<li>Zero-Shot EoT Prompting (<a href=\"https://arxiv.org/abs/2402.05376\" target=\"_blank\">https://arxiv.org/abs/2402.05376</a>)</li>\n<li>Progressive-Hint Prompting (<a href=\"https://arxiv.org/abs/2304.09797\" target=\"_blank\">https://arxiv.org/abs/2304.09797</a>)</li>\n</ul>",
  "comments": [
    {
      "id": 2911588,
      "author_name": "田运来",
      "author_url": "",
      "post_date": "2024-07-08T12:09:09.660000",
      "content": "<p>How can i get the 'accelerate-0-29-3/accelerate-0.29.3-py3-none-any.whl'😝</p>",
      "votes": 1,
      "replies": [
        {
          "id": 2912707,
          "author_name": "bobfromjapan",
          "author_url": "",
          "post_date": "2024-07-09T03:36:02.327000",
          "content": "<p>You can find accelerate 0.29.3 whl from following link(provided by <a href=\"https://www.kaggle.com/anrenk\" target=\"_blank\">@anrenk</a> ). </p>\n<p><a href=\"https://www.kaggle.com/datasets/anrenk/accelerate-0-29-3\" target=\"_blank\">https://www.kaggle.com/datasets/anrenk/accelerate-0-29-3</a></p>",
          "votes": 3,
          "replies": []
        }
      ]
    },
    {
      "id": 2909646,
      "author_name": "",
      "author_url": "",
      "post_date": "2024-07-07T07:16:29.083000",
      "content": "<p>Congratulations for this achievements</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 2912275,
      "author_name": "JM",
      "author_url": "",
      "post_date": "2024-07-08T19:16:38.900000",
      "content": "<blockquote>\n  <p>Like the base notebook, we used Deepseek-math's Code and Text answers, but instead of simply selecting the answer with the most responses, we created multiple-choice questions from the top 5 answers, had Deepseek-math solve them again, and selected the answer based on the original number of responses and the new results.</p>\n</blockquote>\n<p>I tried this also myself but it introduced its own variance in multiple choice selection again lol</p>",
      "votes": 2,
      "replies": [
        {
          "id": 2912706,
          "author_name": "bobfromjapan",
          "author_url": "",
          "post_date": "2024-07-09T03:35:00.427000",
          "content": "<p>As you mentioned, turning the questions into multiple-choice didn't improve accuracy much, and the answers were also inconsistent. <br>\nAdding different problem-solving methods might have helped for leveraging ensemble effect, but we might have just gotten lucky with a good seed :)</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 2911587,
      "author_name": "",
      "author_url": "",
      "post_date": "2024-07-08T12:08:22.947000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 6,
  "messages": [
    {
      "id": 2909456,
      "postDate": "2024-07-07T02:05:59.930Z",
      "content": "<p>Our team struggled to improve our LB score and it was a very difficult competition. We were lucky to get a silver medal in the big shakeup. I would like to thank all the Kagglers who shared their notebooks and discussions!</p>\n<p>Here is our <a href=\"https://www.kaggle.com/code/takeshimorimura/53th-place-solution-aimo-progress-prize-1\" target=\"_blank\">code</a>. The base notebook is <a href=\"https://www.kaggle.com/anrenk\" target=\"_blank\">@anrenk</a>'s <a href=\"https://www.kaggle.com/code/anrenk/aimo-llm-class-deepseek\" target=\"_blank\">AIMO LLM Class DeepSeek</a></p>\n<h2>Improvements</h2>\n<ul>\n<li>Referring to <a href=\"https://www.kaggle.com/competitions/ai-mathematical-olympiad-prize/discussion/509742\" target=\"_blank\">this discussion</a>, <strong>we implemented asynchronous inference using vLLM to maximize the number of <code>n_repetitions</code></strong>.<ul>\n<li>vLLM leverages HW performance better than regular Transformers, enabling very fast inference. In a rough benchmark, vLLM was 1.5 times faster than regular Transformers under the same conditions, and asynchronous inference using AsyncLLMEngine was nearly 6 times faster.</li></ul></li>\n<li>Like the base notebook, we used Deepseek-math's Code and Text answers, but instead of simply selecting the answer with the most responses, we created <strong>multiple-choice questions from the top 5 answers, had Deepseek-math solve them again, and selected the answer based on the original number of responses and the new results</strong>.<ul>\n<li>Example prompt for solving multiple-choice questions:</li></ul></li>\n</ul>\n<pre><code>The   a math problem  you  solve, along  the answer choices.\nProblem: \"There exists a unique increasing geometric sequence of five 2-digit positive integers. What is their sum?\"\n: \"A) 496, B) 310, C) 452, D) 242, E) 500\"\nChoose the letter  the  that best answers the problem,  provide your answer  \\boxed{}.\n</code></pre>\n<ul>\n<li>We also fixed minor bugs in <code>process_text_output</code> and <code>naive_parse</code>, and added error handling from the original notebook. <strong>These seem to have improved the robustness of the score.</strong><ul>\n<li>If the code output by LLM results in an error, result_output is also set to -1.</li>\n<li>I modified the naive_parse function to extract the string \"answer\" and apply the function to the string that follows it (if \"answer\" is not found, result_output is set to -1).</li>\n<li>In the process_code function, I have modified it so that if the code output by the LLM results in a timeout error, this is explicitly returned to the LLM.</li></ul></li>\n</ul>\n<h2>Submission Selection and Validation</h2>\n<ul>\n<li>For validation, we tried MATH level5, AIME83-86, and AIME2023-2024, but ultimately <strong>used AIME83-86, which seemed to have the best correlation with the Public LB.</strong></li>\n<li>The final submissions were the two with the highest validation scores and Public LB.</li>\n</ul>\n<h2>Other things tried but not adopted</h2>\n<ul>\n<li>Considered creating a LightGBM model to <strong>predict the problem type</strong> (\"Algebra\", \"Counting &amp; Probability\", \"Geometry\", \"Intermediate Algebra\", \"Number Theory\", \"Precalculus\", \"Prealgebra\") from the <code>last_hidden_state</code> of deepseek-math using the MATH dataset as training data, and changing parameters like <code>n_repetitions</code> for each problem type.<ul>\n<li>The model was able to predict the type with about <strong>80% accuracy</strong>, but the use of vLLM made it difficult to obtain the <code>hidden_state</code>, so it was not adopted.</li></ul></li>\n<li>Using formatters like <code>ruff</code> or <code>black</code> to fix code formatting and typical errors.<ul>\n<li>This idea came to us near the end of the competition, and we couldn't verify how to effectively incorporate it into the prompt, so we decided not to use it.</li></ul></li>\n<li>Zero-Shot EoT Prompting (<a href=\"https://arxiv.org/abs/2402.05376\" target=\"_blank\">https://arxiv.org/abs/2402.05376</a>)</li>\n<li>Progressive-Hint Prompting (<a href=\"https://arxiv.org/abs/2304.09797\" target=\"_blank\">https://arxiv.org/abs/2304.09797</a>)</li>\n</ul>",
      "rawMarkdown": "Our team struggled to improve our LB score and it was a very difficult competition. We were lucky to get a silver medal in the big shakeup. I would like to thank all the Kagglers who shared their notebooks and discussions!\n\nHere is our [code](https://www.kaggle.com/code/takeshimorimura/53th-place-solution-aimo-progress-prize-1). The base notebook is @anrenk's [AIMO LLM Class DeepSeek](https://www.kaggle.com/code/anrenk/aimo-llm-class-deepseek)\n\n## Improvements\n\n- Referring to [this discussion](https://www.kaggle.com/competitions/ai-mathematical-olympiad-prize/discussion/509742), **we implemented asynchronous inference using vLLM to maximize the number of `n_repetitions`**.\n  - vLLM leverages HW performance better than regular Transformers, enabling very fast inference. In a rough benchmark, vLLM was 1.5 times faster than regular Transformers under the same conditions, and asynchronous inference using AsyncLLMEngine was nearly 6 times faster.\n- Like the base notebook, we used Deepseek-math's Code and Text answers, but instead of simply selecting the answer with the most responses, we created **multiple-choice questions from the top 5 answers, had Deepseek-math solve them again, and selected the answer based on the original number of responses and the new results**.\n  - Example prompt for solving multiple-choice questions:\n\n```\nThe following is a math problem for you to solve, along with the answer choices.\nProblem: \"There exists a unique increasing geometric sequence of five 2-digit positive integers. What is their sum?\"\nOptions: \"A) 496, B) 310, C) 452, D) 242, E) 500\"\nChoose the letter of the option that best answers the problem, and provide your answer within \\boxed{}.\n```\n\n- We also fixed minor bugs in `process_text_output` and `naive_parse`, and added error handling from the original notebook. **These seem to have improved the robustness of the score.**\n  - If the code output by LLM results in an error, result_output is also set to -1.\n  - I modified the naive_parse function to extract the string \"answer\" and apply the function to the string that follows it (if \"answer\" is not found, result_output is set to -1).\n  - In the process_code function, I have modified it so that if the code output by the LLM results in a timeout error, this is explicitly returned to the LLM.\n\n## Submission Selection and Validation\n\n- For validation, we tried MATH level5, AIME83-86, and AIME2023-2024, but ultimately **used AIME83-86, which seemed to have the best correlation with the Public LB.**\n- The final submissions were the two with the highest validation scores and Public LB.\n\n## Other things tried but not adopted\n\n- Considered creating a LightGBM model to **predict the problem type** (\"Algebra\", \"Counting & Probability\", \"Geometry\", \"Intermediate Algebra\", \"Number Theory\", \"Precalculus\", \"Prealgebra\") from the `last_hidden_state` of deepseek-math using the MATH dataset as training data, and changing parameters like `n_repetitions` for each problem type.\n  - The model was able to predict the type with about **80% accuracy**, but the use of vLLM made it difficult to obtain the `hidden_state`, so it was not adopted.\n- Using formatters like `ruff` or `black` to fix code formatting and typical errors.\n  - This idea came to us near the end of the competition, and we couldn't verify how to effectively incorporate it into the prompt, so we decided not to use it.\n- Zero-Shot EoT Prompting (https://arxiv.org/abs/2402.05376)\n- Progressive-Hint Prompting (https://arxiv.org/abs/2304.09797)",
      "votes": 16
    },
    {
      "id": 2911588,
      "postDate": "2024-07-08T12:09:09.660Z",
      "content": "<p>How can i get the 'accelerate-0-29-3/accelerate-0.29.3-py3-none-any.whl'😝</p>",
      "rawMarkdown": "How can i get the 'accelerate-0-29-3/accelerate-0.29.3-py3-none-any.whl'😝",
      "votes": 1,
      "replies": [
        {
          "id": 2912707,
          "postDate": "2024-07-09T03:36:02.327Z",
          "content": "<p>You can find accelerate 0.29.3 whl from following link(provided by <a href=\"https://www.kaggle.com/anrenk\" target=\"_blank\">@anrenk</a> ). </p>\n<p><a href=\"https://www.kaggle.com/datasets/anrenk/accelerate-0-29-3\" target=\"_blank\">https://www.kaggle.com/datasets/anrenk/accelerate-0-29-3</a></p>",
          "rawMarkdown": "You can find accelerate 0.29.3 whl from following link(provided by @anrenk ). \n\nhttps://www.kaggle.com/datasets/anrenk/accelerate-0-29-3",
          "votes": 3
        }
      ]
    },
    {
      "id": 2909646,
      "postDate": "2024-07-07T07:16:29.083Z",
      "content": "<p>Congratulations for this achievements</p>",
      "rawMarkdown": "Congratulations for this achievements",
      "votes": 1
    },
    {
      "id": 2912275,
      "postDate": "2024-07-08T19:16:38.900Z",
      "content": "<blockquote>\n  <p>Like the base notebook, we used Deepseek-math's Code and Text answers, but instead of simply selecting the answer with the most responses, we created multiple-choice questions from the top 5 answers, had Deepseek-math solve them again, and selected the answer based on the original number of responses and the new results.</p>\n</blockquote>\n<p>I tried this also myself but it introduced its own variance in multiple choice selection again lol</p>",
      "rawMarkdown": ">Like the base notebook, we used Deepseek-math's Code and Text answers, but instead of simply selecting the answer with the most responses, we created multiple-choice questions from the top 5 answers, had Deepseek-math solve them again, and selected the answer based on the original number of responses and the new results.\n\nI tried this also myself but it introduced its own variance in multiple choice selection again lol",
      "votes": 2,
      "replies": [
        {
          "id": 2912706,
          "postDate": "2024-07-09T03:35:00.427Z",
          "content": "<p>As you mentioned, turning the questions into multiple-choice didn't improve accuracy much, and the answers were also inconsistent. <br>\nAdding different problem-solving methods might have helped for leveraging ensemble effect, but we might have just gotten lucky with a good seed :)</p>",
          "rawMarkdown": "As you mentioned, turning the questions into multiple-choice didn't improve accuracy much, and the answers were also inconsistent. \nAdding different problem-solving methods might have helped for leveraging ensemble effect, but we might have just gotten lucky with a good seed :)",
          "votes": 1
        }
      ]
    },
    {
      "id": 2911587,
      "postDate": "2024-07-08T12:08:22.947Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-20T06:54:31Z",
  "competition_id": 73231,
  "competition_slug": "ai-mathematical-olympiad-prize",
  "team_id": 11874374,
  "solution_url": "/competitions/ai-mathematical-olympiad-prize/writeups/moribob-53rd-place-solution"
}