{ "id": 518960, "title": "4th place solution", "content": "

First, we would like to thank Kaggle and XTX Markets for bringing us this exciting competition. Through this competition, we have learned a lot. We also want to thank every team member for their hard work. They are truly amazing, with many excellent ideas and engineering skills.

\n

Our solution is based on the great work of AbdurRafae, Improved Code Interpretation (kaggle.com). His Early Sharing Prize is well-deserved. Similarly, Anren's reorganization of this code provided us with great help.

\n

Below is a brief introduction to our solution's overall process and some unique tricks. Our solution code in here

\n

4th Dual gpu reasoning and comparing answers (kaggle.com)

\n

Local CV Strategy

\n

We used this 71k dataset: AIMO-24: Processor (Art Of Problem Solving) (kaggle.com). From all AMC_12A problems, we selected the latest 50 for our local validation (excluding problems with the [asy] string in the text, which will be explained later). This showed good correlation with the public leaderboard.

\n

Model Selection

\n

We used deepseek-math-7b-rl, with parameters: temperature of 0.9, top_p of 1.0, and max tokens of 2048. This model with code tools, can achieve 58.8% on MATH benchmarks.

\n

We deployed and ran this model in parallel on two T4 GPUs to maximize self-consistency for final prediction accuracy. Dual-GPU inference increased the total repetitions per problem from 21 to 30-40.

\n

Time Management

\n

TimeManager Class

\n

We created a TimeManager class to manage time allocation. It tracks the time spent on each problem and dynamically adjusts the number of attempts for subsequent problems to ensure as many problems are solved as possible within the given time.

\n
 :\n     ():\n        self.total_problems = total_problems  \n        self.total_time = total_time  \n        self.default_attempts = default_attempts  \n        self.time_spent =   \n        self.current_problem =   \n        self.last_problem_time =   \n\n     ():\n        self.time_spent += time_for_problem  \n        self.current_problem +=   \n        self.last_problem_time = time_for_problem  \n\n     ():\n         self.current_problem >= self.total_problems:\n             \n\n        remaining_time = self.total_time - self.time_spent  \n        remaining_problems = self.total_problems - self.current_problem  \n        average_time_per_problem = remaining_time / remaining_problems  \n\n         self.current_problem ==   self.time_spent / self.current_problem <= average_time_per_problem:\n            next_attempts = self.default_attempts\n        :\n            next_attempts = (, (self.default_attempts * (average_time_per_problem / (self.time_spent / self.current_problem)) * ))\n\n         next_attempts\n\n     ():\n        remaining_time = self.total_time - self.time_spent  \n        ()  \n        ()  \n        ()  \n        ()  \n
\n

Reducing Attempts for Difficult Problems

\n

We manually reduced the number of attempts for problems we believed the model could not solve, providing more attempts for the remaining problems. Specifically, we only allowed 3 repetitions for problems containing the [asy] symbol. This symbol indicates a geometric problem with a figure, surrounded by [asy] in LaTeX. In our local tests, the model rarely answered these problems correctly, and even when it did, we suspected the problem might have been in the model's training set.

\n

Reducing Waiting Time Between Dual GPU Runs

\n

When using ThreadPoolExecutor for parallel inference, one GPU often finishes faster. If we see the first completed GPU has enough candidate answers, we will terminate the slower GPU's attempts early. And then, merge the candidate answers generated by the two gpu's. This ensures more efficient use of dual GPUs and increases the average number of repetitions for all problems.

\n

Candidate Answer Generation

\n

DeepSeekMath is initialized with DeepSeek-Coder-v1.5 7B, which can effectively solve and prove mathematical problems by writing programs. We followed Anren's method with two prompts, letting the model solve problems using COT and write code methods.

\n

We made some improvements:

\n
    \n

  1. Limiting the Model's Code Modify Opportunities

    \n

    We limited the model to only 3 code modify opportunities per repetition, as input text that is too long significantly reduces the output quality of LLMs. This feature is controlled by the parameter while_limit, which we set to 6(as each code input and output processing requires two while loop).

    2. \n

  2. Weighted Count Sorting for Candidate Answers

    \n

    We discovered that the model tends to output small integers like 0, 1, 2, 3, 4, 5 when answering incorrectly. We reduced the weight of these numbers to 0.25, while other numbers retained the normal weight of 1.

    3. \n

  3. Fixing Some Incorrect Behaviors

    \n

    For example, when the code output is not a valid answer (e.g., error, decimal, complex number), the original method still tried to parse the text answer, which was almost always incorrect. In such cases, we skipped recording the text answer.

    4. \n
\n

Other

\n

Our logs recorded detailed data for each experiment, including candidate answers, program code, code accuracy, text accuracy, etc. These detailed logs greatly facilitated our quick experiments and result observations. Our local experiment score was around 21, with pass1@ reaching 32 and pass2@ reaching 23.

\n

Of course, some luck might be needed in the competition. Our solution on the public leaderboard, using different seeds, could result in a 3-point score difference.

\n

Detailed Local Log (parameters slightly different)
\n\"pic\"

", "comments": [ { "id": 2912941, "author_name": "", "author_url": "", "post_date": "2024-07-09T06:31:51.663000", "content": "

congratulation for 4th p[ace. and please share best source for learn this kind of models and algorithms.

", "votes": -1, "replies": [] } ], "comment_count": 1, "messages": [ { "id": 2912812, "postDate": "2024-07-09T04:51:35.060Z", "content": "

First, we would like to thank Kaggle and XTX Markets for bringing us this exciting competition. Through this competition, we have learned a lot. We also want to thank every team member for their hard work. They are truly amazing, with many excellent ideas and engineering skills.

\n

Our solution is based on the great work of AbdurRafae, Improved Code Interpretation (kaggle.com). His Early Sharing Prize is well-deserved. Similarly, Anren's reorganization of this code provided us with great help.

\n

Below is a brief introduction to our solution's overall process and some unique tricks. Our solution code in here

\n

4th Dual gpu reasoning and comparing answers (kaggle.com)

\n

Local CV Strategy

\n

We used this 71k dataset: AIMO-24: Processor (Art Of Problem Solving) (kaggle.com). From all AMC_12A problems, we selected the latest 50 for our local validation (excluding problems with the [asy] string in the text, which will be explained later). This showed good correlation with the public leaderboard.

\n

Model Selection

\n

We used deepseek-math-7b-rl, with parameters: temperature of 0.9, top_p of 1.0, and max tokens of 2048. This model with code tools, can achieve 58.8% on MATH benchmarks.

\n

We deployed and ran this model in parallel on two T4 GPUs to maximize self-consistency for final prediction accuracy. Dual-GPU inference increased the total repetitions per problem from 21 to 30-40.

\n

Time Management

\n

TimeManager Class

\n

We created a TimeManager class to manage time allocation. It tracks the time spent on each problem and dynamically adjusts the number of attempts for subsequent problems to ensure as many problems are solved as possible within the given time.

\n
 :\n     ():\n        self.total_problems = total_problems  \n        self.total_time = total_time  \n        self.default_attempts = default_attempts  \n        self.time_spent =   \n        self.current_problem =   \n        self.last_problem_time =   \n\n     ():\n        self.time_spent += time_for_problem  \n        self.current_problem +=   \n        self.last_problem_time = time_for_problem  \n\n     ():\n         self.current_problem >= self.total_problems:\n             \n\n        remaining_time = self.total_time - self.time_spent  \n        remaining_problems = self.total_problems - self.current_problem  \n        average_time_per_problem = remaining_time / remaining_problems  \n\n         self.current_problem ==   self.time_spent / self.current_problem <= average_time_per_problem:\n            next_attempts = self.default_attempts\n        :\n            next_attempts = (, (self.default_attempts * (average_time_per_problem / (self.time_spent / self.current_problem)) * ))\n\n         next_attempts\n\n     ():\n        remaining_time = self.total_time - self.time_spent  \n        ()  \n        ()  \n        ()  \n        ()  \n
\n

Reducing Attempts for Difficult Problems

\n

We manually reduced the number of attempts for problems we believed the model could not solve, providing more attempts for the remaining problems. Specifically, we only allowed 3 repetitions for problems containing the [asy] symbol. This symbol indicates a geometric problem with a figure, surrounded by [asy] in LaTeX. In our local tests, the model rarely answered these problems correctly, and even when it did, we suspected the problem might have been in the model's training set.

\n

Reducing Waiting Time Between Dual GPU Runs

\n

When using ThreadPoolExecutor for parallel inference, one GPU often finishes faster. If we see the first completed GPU has enough candidate answers, we will terminate the slower GPU's attempts early. And then, merge the candidate answers generated by the two gpu's. This ensures more efficient use of dual GPUs and increases the average number of repetitions for all problems.

\n

Candidate Answer Generation

\n

DeepSeekMath is initialized with DeepSeek-Coder-v1.5 7B, which can effectively solve and prove mathematical problems by writing programs. We followed Anren's method with two prompts, letting the model solve problems using COT and write code methods.

\n

We made some improvements:

\n
    \n

  1. Limiting the Model's Code Modify Opportunities

    \n

    We limited the model to only 3 code modify opportunities per repetition, as input text that is too long significantly reduces the output quality of LLMs. This feature is controlled by the parameter while_limit, which we set to 6(as each code input and output processing requires two while loop).

    2. \n

  2. Weighted Count Sorting for Candidate Answers

    \n

    We discovered that the model tends to output small integers like 0, 1, 2, 3, 4, 5 when answering incorrectly. We reduced the weight of these numbers to 0.25, while other numbers retained the normal weight of 1.

    3. \n

  3. Fixing Some Incorrect Behaviors

    \n

    For example, when the code output is not a valid answer (e.g., error, decimal, complex number), the original method still tried to parse the text answer, which was almost always incorrect. In such cases, we skipped recording the text answer.

    4. \n
\n

Other

\n

Our logs recorded detailed data for each experiment, including candidate answers, program code, code accuracy, text accuracy, etc. These detailed logs greatly facilitated our quick experiments and result observations. Our local experiment score was around 21, with pass1@ reaching 32 and pass2@ reaching 23.

\n

Of course, some luck might be needed in the competition. Our solution on the public leaderboard, using different seeds, could result in a 3-point score difference.

\n

Detailed Local Log (parameters slightly different)
\n\"pic\"

", "rawMarkdown": "First, we would like to thank Kaggle and XTX Markets for bringing us this exciting competition. Through this competition, we have learned a lot. We also want to thank every team member for their hard work. They are truly amazing, with many excellent ideas and engineering skills.\n\nOur solution is based on the great work of [AbdurRafae](https://www.kaggle.com/abdurrafae), [Improved Code Interpretation (kaggle.com)](https://www.kaggle.com/code/abdurrafae/improved-code-interpretation). His Early Sharing Prize is well-deserved. Similarly, [Anren](https://www.kaggle.com/anrenk)'s reorganization of this code provided us with great help.\n\nBelow is a brief introduction to our solution's overall process and some unique tricks. Our solution code in here\n\n[4th Dual gpu reasoning and comparing answers (kaggle.com)](https://www.kaggle.com/code/winter2003/4th-dual-gpu-reasoning-and-comparing-answers)\n\n\n\n### Local CV Strategy\n\nWe used this 71k dataset: [AIMO-24: Processor (Art Of Problem Solving) (kaggle.com)](https://www.kaggle.com/code/dinhttrandrise/aimo-24-processor-art-of-problem-solving/output?select=problems.csv). From all AMC_12A problems, we selected the latest 50 for our local validation (excluding problems with the `[asy]` string in the text, which will be explained later). This showed good correlation with the public leaderboard.\n\n\n\n### Model Selection\n\nWe used `deepseek-math-7b-rl`, with parameters: temperature of 0.9, top_p of 1.0, and max tokens of 2048. This model with code tools, can achieve 58.8% on MATH benchmarks.\n\nWe deployed and ran this model in parallel on two T4 GPUs to maximize self-consistency for final prediction accuracy. Dual-GPU inference increased the total repetitions per problem from 21 to 30-40.\n\n\n\n### Time Management\n\n**TimeManager Class**\n\nWe created a `TimeManager` class to manage time allocation. It tracks the time spent on each problem and dynamically adjusts the number of attempts for subsequent problems to ensure as many problems are solved as possible within the given time.\n\n```python\nclass TimeManager:\n def __init__(self, total_problems, total_time, default_attempts):\n self.total_problems = total_problems # Total number of problems\n self.total_time = total_time # Total time available\n self.default_attempts = default_attempts # Default number of attempts\n self.time_spent = 0 # Time spent so far\n self.current_problem = 0 # Current problem index\n self.last_problem_time = 0 # Time spent on the last problem\n \n def update_time_spent(self, time_for_problem):\n self.time_spent += time_for_problem # Add time spent on the problem to the total time spent\n self.current_problem += 1 # Increment the problem index\n self.last_problem_time = time_for_problem # Record the time spent on the current problem\n \n def get_next_attempts(self):\n if self.current_problem >= self.total_problems:\n return 0\n \n remaining_time = self.total_time - self.time_spent # Remaining time\n remaining_problems = self.total_problems - self.current_problem # Remaining problems\n average_time_per_problem = remaining_time / remaining_problems # Average time per problem\n \n if self.current_problem == 0 or self.time_spent / self.current_problem <= average_time_per_problem:\n next_attempts = self.default_attempts\n else:\n next_attempts = max(1, int(self.default_attempts * (average_time_per_problem / (self.time_spent / self.current_problem)) * 0.8))\n \n return next_attempts\n \n def print_status(self):\n remaining_time = self.total_time - self.time_spent # Remaining time\n print(f\"Time spent: {self.time_spent} seconds\") # Print total time spent\n print(f\"Remaining time: {remaining_time} seconds\") # Print remaining time\n print(f\"Current problem: {self.current_problem}/{self.total_problems}\") # Print current problem progress\n print(f\"Time spent on current problem: {self.last_problem_time} seconds\") # Print time spent on the current problem\n```\n\n**Reducing Attempts for Difficult Problems**\n\nWe manually reduced the number of attempts for problems we believed the model could not solve, providing more attempts for the remaining problems. Specifically, we only allowed 3 repetitions for problems containing the `[asy]` symbol. This symbol indicates a geometric problem with a figure, surrounded by `[asy]` in LaTeX. In our local tests, the model rarely answered these problems correctly, and even when it did, we suspected the problem might have been in the model's training set.\n\n**Reducing Waiting Time Between Dual GPU Runs**\n\nWhen using `ThreadPoolExecutor` for parallel inference, one GPU often finishes faster. If we see the first completed GPU has enough candidate answers, we will terminate the slower GPU's attempts early. And then, merge the candidate answers generated by the two gpu's. This ensures more efficient use of dual GPUs and increases the average number of repetitions for all problems.\n\n\n\n### Candidate Answer Generation\n\nDeepSeekMath is initialized with [DeepSeek-Coder-v1.5 7B](https://huggingface.co/deepseek-ai/deepseek-coder-7b-base-v1.5), which can effectively solve and prove mathematical problems by writing programs. We followed [Anren](https://www.kaggle.com/anrenk)'s method with two prompts, letting the model solve problems using COT and write code methods.\n\nWe made some improvements:\n\n1. **Limiting the Model's Code Modify Opportunities**\n\n We limited the model to only 3 code modify opportunities per repetition, as input text that is too long significantly reduces the output quality of LLMs. This feature is controlled by the parameter `while_limit`, which we set to 6(as each code input and output processing requires two while loop).\n\n2. **Weighted Count Sorting for Candidate Answers**\n\n We discovered that the model tends to output small integers like 0, 1, 2, 3, 4, 5 when answering incorrectly. We reduced the weight of these numbers to 0.25, while other numbers retained the normal weight of 1.\n\n3. **Fixing Some Incorrect Behaviors**\n\n For example, when the code output is not a valid answer (e.g., error, decimal, complex number), the original method still tried to parse the text answer, which was almost always incorrect. In such cases, we skipped recording the text answer.\n\n\n\n### Other\n\nOur logs recorded detailed data for each experiment, including candidate answers, program code, code accuracy, text accuracy, etc. These detailed logs greatly facilitated our quick experiments and result observations. Our local experiment score was around 21, with pass1@ reaching 32 and pass2@ reaching 23.\n\nOf course, some luck might be needed in the competition. Our solution on the public leaderboard, using different seeds, could result in a 3-point score difference.\n\n\n\n**Detailed Local Log (parameters slightly different)**\n![pic](https://ibb.co/b258Q4M)", "votes": 36 }, { "id": 2912941, "postDate": "2024-07-09T06:31:51.663Z", "content": "

congratulation for 4th p[ace. and please share best source for learn this kind of models and algorithms.

", "rawMarkdown": "congratulation for 4th p[ace. and please share best source for learn this kind of models and algorithms. ", "votes": -1 } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T06:54:42Z", "competition_id": 73231, "competition_slug": "ai-mathematical-olympiad-prize", "team_id": 12035484, "solution_url": "/competitions/ai-mathematical-olympiad-prize/writeups/codeinter-4th-place-solution" }