![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F6050419%2F461f2d44a407cb5a9c238c4934d84d3b%2FScreenshot%202026-04-24%20at%209.52.36AM.png?generation=1776992015179159&alt=media\")
This solution builds on Sachchidananda Daki's public notebook (v45), which scored 40/50 on the public leaderboard using vLLM serving, Harmony encoding, a sandboxed Jupyter tool, parallel attempts, entropy-weighted voting, early-stopping, and adaptive time budgeting. My changes were to the prompts and several hyperparameters (detailed in the Prompts and Configuration sections below), which raised the score to 45/50. Thanks also to andreasbis/aimo-3-utils for the offline wheel bundle.
\nThe solver runs gpt-oss-120b on a single H100 via vLLM, with fp8 KV cache, prefix caching, and a 64K context. For each problem, the system issues 8 independently seeded attempts in parallel at temperature=1.0 and reasoning_effort=HIGH. Each attempt has access to a stateful Jupyter sandbox (math, numpy, sympy, itertools, collections, mpmath with 64 digit precision) that the model can call mid-generation via Harmony tool-use messages. Output is streamed token by token, and whenever a } appears, the last 64 tokens are scanned. If \\boxed{…} is found, the attempt terminates immediately with that answer.
Once an attempt produces a final answer, it is pooled with the others. If four attempts agree on the same answer, the remaining attempts are cancelled (early stop). Otherwise the system waits for all 8 attempts to complete and then selects the answer using entropy-weighted voting.
\nTime is budgeted globally within the notebook's total runtime limit of 17,400s. Each problem reserves a base budget of 300s but may extend up to 900s if remaining time exceeds what the remaining problems need.
\n
The rest of this section zooms in on the pieces referenced above: the per-attempt turn loop, the entropy-weighted voting used for tie-breaking, the three prompts, the configuration, and how answers are extracted from streamed output.
\nThe model can reason freely, but it cannot execute Python on its own. When it writes code inside a tool-call message and emits the hand-off token, generation has to stop so the orchestrator can actually run that code in a sandbox and feed the real output back. Without this pause, the model would just keep generating and hallucinate what it thinks the output should be. A turn is exactly one round of generation bounded by these hand-offs.
\nAfter a turn ends, the orchestrator looks at the last message. If it's on the final channel, the boxed answer is extracted and the attempt is done. If it's a Python tool call, the code runs in the sandbox and the real output is appended as a tool-response message before the next turn begins.
\nDuring generation, at every token position the model produces we record the top-5 log-probabilities for that position. At the end of the attempt, each log-probability is converted to a probability (via exp), and the Shannon entropy at that position is computed as -sum(p_i * log2(p_i)) across those 5 values. Summing those per-position entropies and dividing by the token count gives the attempt's mean entropy: a single number where low means the model was consistently sure of itself, high means it was uncertain throughout.
Each attempt's vote is then weighted by 1 / mean_entropy. Weights are summed per candidate answer, and the answer with the highest total weight wins.
\nA worked example makes the behavior clearer. Suppose a problem launched 8 attempts and 3 failed to produce a valid answer (timed out, crashed, or never emitted a \\boxed{}). The remaining 5 valid attempts look like this:
| Attempt | \nAnswer | \nMean entropy | \n
|---|---|---|
| 1 | \n8687 | \n0.75 | \n
| 2 | \n8687 | \n0.69 | \n
| 3 | \n48673 | \n1.77 | \n
| 4 | \n48673 | \n1.12 | \n
| 5 | \n14 | \n0.88 | \n
No answer reached 4 agreements, so early-stop didn't fire, so we fall into the voting path.
\nStep 1: Invert each entropy into a weight (1 / entropy):
| Attempt | \nAnswer | \nEntropy | \nWeight | \n
|---|---|---|---|
| 1 | \n8687 | \n0.75 | \n1.334 | \n
| 2 | \n8687 | \n0.69 | \n1.449 | \n
| 3 | \n48673 | \n1.77 | \n0.565 | \n
| 4 | \n48673 | \n1.12 | \n0.893 | \n
| 5 | \n14 | \n0.88 | \n1.136 | \n
Step 2: Sum weights per candidate answer:
\nStep 3: Pick the highest. 8687 wins with 2.783.
\n8687 and 48673 both had 2 raw votes, so a plain majority vote would have tied them. But 8687's attempts had low entropy (0.75 and 0.69), while 48673's were much noisier (1.77 and 1.12), so 8687 wins the weighted vote. Answer 14 was individually confident (entropy 0.88) but only had one attempt behind it, so it couldn't catch up.
\nThree prompts are used. All three are deliberately minimal. They state the contract and the available tooling, and do not prescribe reasoning steps.
\nThe v45 baseline used a much more elaborate set of prompts (~65 lines total), explicitly scaffolding the model's reasoning through an UNDERSTAND → EXPLORE → PLAN → EXECUTE → VERIFY protocol, along with detailed mathematical reasoning principles, verification requirements, and library best-practices:
\nSystem prompt:
\nsystem_prompt = (\n 'You are an elite mathematical problem solver with expertise at the International '\n 'Mathematical Olympiad (IMO) level. Your goal is to find the correct answer through '\n 'rigorous mathematical reasoning.\\n\\n'\n '# Problem-Solving Approach:\\n'\n '1. UNDERSTAND: Carefully read and rephrase the problem in your own words.\\n'\n '2. EXPLORE: Consider multiple solution strategies. Think about relevant theorems.\\n'\n '3. PLAN: Select the most promising approach and outline key steps before executing.\\n'\n '4. EXECUTE: Work through your solution methodically. Show all reasoning steps clearly.\\n'\n '5. VERIFY: Check your answer by substituting back, testing edge cases.\\n\\n'\n '# Mathematical Reasoning Principles:\\n'\n '- Break complex problems into smaller, manageable sub-problems\\n'\n ...\n '# Verification Requirements:\\n'\n ...\n '# Output Format:\\n'\n ...\n)\nTool prompt:
\ntool_prompt = (\n 'Use this tool to execute Python code for:\\n'\n '- Complex calculations that would be error-prone by hand\\n'\n ...\n 'Remember: Code should support your mathematical reasoning, not replace it. '\n 'Explain what you\\'re computing and why before running code.'\n)\nPreference prompt:
\npreference_prompt = (\n 'You have access to `math`, `numpy`, and `sympy` for:\\n\\n'\n '# Symbolic Computation (sympy):\\n'\n ...\n '# Numerical Computation (numpy):\\n'\n ...\n)\nThis baseline scored 40/50 on the public leaderboard.
\nThe new prompts collapse this to ~10 lines total, removing the reasoning scaffold entirely and keeping only the task contract and tool description.
\nSystem prompt (sets the task contract):
\nsystem_prompt = (\n 'Solve the following math problem step by step.\\n'\n 'Use Python code for calculations, verification, or brute-force search.\\n'\n 'The answer is a non-negative integer in [0, 99999].\\n'\n 'Put your final answer in \\\\boxed{}.'\n)\nTool prompt (describes the Python sandbox, attached to the Harmony ToolNamespaceConfig):
tool_prompt = (\n 'Execute Python code in a stateful Jupyter notebook. '\n 'Available: math, numpy, sympy, itertools, collections, mpmath (64-digit precision). '\n 'Use print() to see results.'\n)\nPreference prompt (appended to each problem; nudges library choice, not reasoning):
\npreference_prompt = (\n 'Use sympy for exact symbolic computation. '\n 'For number theory: use sympy.ntheory (factorint, divisors, totient, isprime). '\n 'Verify analytical solutions with numerical checks or brute-force when feasible.'\n)\nWith these changes (along with the configuration tweaks described in the next section), the score moved from 40/50 to 45/50. The prompt simplification appears to be a major factor, though it was not isolated from the other hyperparameter changes described in the Configuration section.
\nAs DeepSeek-R1 noted, heavy chain-of-thought scaffolding (UNDERSTAND → EXPLORE → PLAN → EXECUTE → VERIFY style) can interfere with models that already have strong internal reasoning. I believe gpt-oss-120b at reasoning_effort=HIGH is such a model, so I tried simplifying the prompts. They specify what the model must do (answer format, answer range) and what it has access to (tooling), but not how to think. Reasoning is left entirely to the model.
Compared to v45, fewer attempts run in parallel (16 → 8), but each gets more compute and time: workers per problem doubled (8 → 16), the early-stop threshold rose from 3 to 4 agreements, and the per-problem time budget grew (145s → 300s base, 250s → 900s ceiling). The full final configuration is below.
\n| Parameter | \nValue | \nNotes | \n
|---|---|---|
| Model | \ngpt-oss-120b | \nserved via vLLM, fp8 KV cache, prefix caching | \n
reasoning_effort | \nHIGH | \nHarmony system content | \n
| Context | \n65,536 tokens | \nmax_model_len | \n
| Per-turn max tokens | \n4,096 | \ntruncation guard | \n
| Temperature | \n1.0 | \n\n |
min_p | \n0.02 | \n\n |
| Attempts per problem | \n8 | \nparallel, independently seeded | \n
| Workers | \n16 | \nworker threads per problem | \n
| Turns per attempt | \n128 | \nmax tool-use turns | \n
| Early-stop threshold | \n4 agreements | \n\n |
| Voting | \nentropy-weighted | \nweight = 1 / mean_token_entropy | \n
| Base per-problem timeout | \n300s | \n\n |
| Max per-problem timeout | \n900s | \nextended if notebook time allows | \n
| Notebook runtime limit | \n17,400s | \nabout 4h 50m | \n
| Seed | \n42 | \nper-attempt seed = (seed + i)^2 | \n
During streaming, each chunk is watched for a } character. When one appears, the last 64 tokens of text are searched with the regex \\\\boxed\\s*\\{\\s*([0-9,]+)\\s*\\}. The trailing match wins, commas are stripped, and the value is kept if it falls in [0, 99999]. If the full message completes without a \\boxed{} match, a fallback regex final\\s+answer\\s+is\\s*([0-9,]+) is applied. Attempts without a valid answer contribute no vote.
The following variants were tested during development and did not improve on the configuration above:
\nTwo-round retry: If the initial 8 attempts produced ≤2 agreement, launch another 8 attempts with an additional 900s budget, then entropy-vote across all 16. Result: 40/50.
Temperature sweep: A sweep over {0.85, 1.00, 1.15} was run on 10 problems from reference.csv. The nine easy/medium problems were solved at all three settings. On the single hard problem, out of 8 parallel attempts, temp 0.85 yielded 1 correct, temp 1.00 yielded 0, and temp 1.15 yielded 3. Another run at temp 1.15 on the same problem yielded different hit counts, suggesting that factors beyond temperature alone (such as vLLM internal state, KV cache reuse, or scheduling nondeterminism) also affect outcomes. The signal was judged too noisy to justify deviating from temp 1.0.
Concept-first agentic loop: Four-step pipeline: (a) LLM enumerates relevant concepts for the problem, (b) LLM drafts a strategy from {problem + concepts}, (c) LLM identifies potential pitfalls and wrong assumptions, (d) LLM solves the problem with the full accumulated context. On the hard problems where the baseline already struggled, the extra stages did not produce more correct answers, and on some runs zero out of 8 attempts reached the correct answer.
Solver/verifier loop: The solver proposes a solving strategy, and the verifier returns either approved or rejected. The loop continues until a strategy gets approved, and the solver then uses that approved strategy to solve the problem. This also did not work out well. The verifier often approved flawed strategies or rejected reasonable ones, and the resulting solutions were no more accurate than single-shot solving. Correct and incorrect answers frequently tied in the final vote.
AIMO2-style verifier wave (from NVIDIA's AIMO2 solution): NVIDIA's AIMO2 protocol generates 64 candidates and runs a tournament-style verification: 32 times, the candidates are grouped into sets of 8, the best one from each group is picked, and then the best one is picked among those winners. The final answer is decided by majority vote across the 32 tournament winners. I tried a simpler version. If the initial 8 attempts had consensus ≤2, a second stage summarizes the 8 solutions and runs 8 verifier passes, where each pass sees the summarized candidates in a randomly shuffled order and picks the best one. The candidate with the most votes across the 8 passes wins. This did not work out well. The pipeline often ended up with the correct answer getting two votes and an incorrect one also getting two votes, with no clear winner. The approach might have worked better with more verifier passes, but I did not try that due to the time budget. One possible factor is that NVIDIA's GenSelect used a dedicated selector model trained on matched data, while I was using an off-the-shelf gpt-oss-120b for both solving and verification.
Per-answer verifier scoring wave: If the initial 8 attempts had ≤2 consensus, a second stage runs where every unique candidate answer is shown to the verifier 8 times. Each pass assigns a score in {1, 2, 3} reflecting how likely the candidate is correct. Scores are summed per candidate across all passes, and the candidate with the highest total wins. Same underlying problem as (5). The verifier was not a meaningfully stronger judge than the solver.
Across the variants I tested, adding structure to guide the model's reasoning (extra stages, verifier loops, scaffolded prompts) did not improve accuracy. The configuration above (minimal prompts, single-shot parallel sampling, entropy voting) outperformed all tested elaborations. NVIDIA's AIMO2 solution showed that verifier-based selection can work well, but it relied on a dedicated selector trained on matched data. It seems that imposing structure on how a strong reasoner thinks tends to hurt rather than help, unless the model has been trained for that specific structure.
\nAcross every variant I tested, adding structure (verifier loops, scaffolded prompts, multi-stage pipelines) hurt rather than helped. The minimal configuration outperformed all elaborations. It seems that for a model with strong internal reasoning, prompt and pipeline design should clarify the contract, not direct the thinking. Future work likely lies in better selection across attempts (a stronger judge than the solver itself), not in further structuring the solver.
", "comments": [ { "id": 3447776, "author_name": "jayhun", "author_url": "", "post_date": "2026-04-23T22:05:14.563000", "content": "The code is shared; too bad, I cannot access it somehow. 😭
", "votes": 0, "replies": [ { "id": 3447821, "author_name": "Seungjun Lee", "author_url": "", "post_date": "2026-04-24T03:23:39.293000", "content": "Thanks for pointing that out. I just made it public. You should be able to see it now :)
", "votes": 0, "replies": [] } ] }, { "id": 3452683, "author_name": "Yu Liwei", "author_url": "", "post_date": "2026-05-04T06:21:41.023000", "content": "Thanks for your share. I wonder the model is a fine-tune version or original version?
", "votes": 0, "replies": [ { "id": 3456302, "author_name": "Seungjun Lee", "author_url": "", "post_date": "2026-05-11T17:07:30.213000", "content": "Thanks! It's the Unsloth version of gpt-oss-120b, not fine-tuned.
", "votes": 0, "replies": [] } ] }, { "id": 3452541, "author_name": "Rustam Bazarbayev", "author_url": "", "post_date": "2026-05-03T17:48:23.667000", "content": "Many of my submissions I lean on how solved my model the most hard problem. If the model successfully solved it I sent. but my model got score lower than 43
", "votes": 0, "replies": [] }, { "id": 3448189, "author_name": "mrx", "author_url": "", "post_date": "2026-04-24T21:13:33.730000", "content": "The private LB is not yet finalized
", "votes": 0, "replies": [] } ], "comment_count": 6, "messages": [ { "id": 3447708, "postDate": "2026-04-23T18:45:37.427Z", "content": "This solution builds on Sachchidananda Daki's public notebook (v45), which scored 40/50 on the public leaderboard using vLLM serving, Harmony encoding, a sandboxed Jupyter tool, parallel attempts, entropy-weighted voting, early-stopping, and adaptive time budgeting. My changes were to the prompts and several hyperparameters (detailed in the Prompts and Configuration sections below), which raised the score to 45/50. Thanks also to andreasbis/aimo-3-utils for the offline wheel bundle.
\nThe solver runs gpt-oss-120b on a single H100 via vLLM, with fp8 KV cache, prefix caching, and a 64K context. For each problem, the system issues 8 independently seeded attempts in parallel at temperature=1.0 and reasoning_effort=HIGH. Each attempt has access to a stateful Jupyter sandbox (math, numpy, sympy, itertools, collections, mpmath with 64 digit precision) that the model can call mid-generation via Harmony tool-use messages. Output is streamed token by token, and whenever a } appears, the last 64 tokens are scanned. If \\boxed{…} is found, the attempt terminates immediately with that answer.
Once an attempt produces a final answer, it is pooled with the others. If four attempts agree on the same answer, the remaining attempts are cancelled (early stop). Otherwise the system waits for all 8 attempts to complete and then selects the answer using entropy-weighted voting.
\nTime is budgeted globally within the notebook's total runtime limit of 17,400s. Each problem reserves a base budget of 300s but may extend up to 900s if remaining time exceeds what the remaining problems need.
\n
The rest of this section zooms in on the pieces referenced above: the per-attempt turn loop, the entropy-weighted voting used for tie-breaking, the three prompts, the configuration, and how answers are extracted from streamed output.
\nThe model can reason freely, but it cannot execute Python on its own. When it writes code inside a tool-call message and emits the hand-off token, generation has to stop so the orchestrator can actually run that code in a sandbox and feed the real output back. Without this pause, the model would just keep generating and hallucinate what it thinks the output should be. A turn is exactly one round of generation bounded by these hand-offs.
\nAfter a turn ends, the orchestrator looks at the last message. If it's on the final channel, the boxed answer is extracted and the attempt is done. If it's a Python tool call, the code runs in the sandbox and the real output is appended as a tool-response message before the next turn begins.
\nDuring generation, at every token position the model produces we record the top-5 log-probabilities for that position. At the end of the attempt, each log-probability is converted to a probability (via exp), and the Shannon entropy at that position is computed as -sum(p_i * log2(p_i)) across those 5 values. Summing those per-position entropies and dividing by the token count gives the attempt's mean entropy: a single number where low means the model was consistently sure of itself, high means it was uncertain throughout.
Each attempt's vote is then weighted by 1 / mean_entropy. Weights are summed per candidate answer, and the answer with the highest total weight wins.
\nA worked example makes the behavior clearer. Suppose a problem launched 8 attempts and 3 failed to produce a valid answer (timed out, crashed, or never emitted a \\boxed{}). The remaining 5 valid attempts look like this:
| Attempt | \nAnswer | \nMean entropy | \n
|---|---|---|
| 1 | \n8687 | \n0.75 | \n
| 2 | \n8687 | \n0.69 | \n
| 3 | \n48673 | \n1.77 | \n
| 4 | \n48673 | \n1.12 | \n
| 5 | \n14 | \n0.88 | \n
No answer reached 4 agreements, so early-stop didn't fire, so we fall into the voting path.
\nStep 1: Invert each entropy into a weight (1 / entropy):
| Attempt | \nAnswer | \nEntropy | \nWeight | \n
|---|---|---|---|
| 1 | \n8687 | \n0.75 | \n1.334 | \n
| 2 | \n8687 | \n0.69 | \n1.449 | \n
| 3 | \n48673 | \n1.77 | \n0.565 | \n
| 4 | \n48673 | \n1.12 | \n0.893 | \n
| 5 | \n14 | \n0.88 | \n1.136 | \n
Step 2: Sum weights per candidate answer:
\nStep 3: Pick the highest. 8687 wins with 2.783.
\n8687 and 48673 both had 2 raw votes, so a plain majority vote would have tied them. But 8687's attempts had low entropy (0.75 and 0.69), while 48673's were much noisier (1.77 and 1.12), so 8687 wins the weighted vote. Answer 14 was individually confident (entropy 0.88) but only had one attempt behind it, so it couldn't catch up.
\nThree prompts are used. All three are deliberately minimal. They state the contract and the available tooling, and do not prescribe reasoning steps.
\nThe v45 baseline used a much more elaborate set of prompts (~65 lines total), explicitly scaffolding the model's reasoning through an UNDERSTAND → EXPLORE → PLAN → EXECUTE → VERIFY protocol, along with detailed mathematical reasoning principles, verification requirements, and library best-practices:
\nSystem prompt:
\nsystem_prompt = (\n 'You are an elite mathematical problem solver with expertise at the International '\n 'Mathematical Olympiad (IMO) level. Your goal is to find the correct answer through '\n 'rigorous mathematical reasoning.\\n\\n'\n '# Problem-Solving Approach:\\n'\n '1. UNDERSTAND: Carefully read and rephrase the problem in your own words.\\n'\n '2. EXPLORE: Consider multiple solution strategies. Think about relevant theorems.\\n'\n '3. PLAN: Select the most promising approach and outline key steps before executing.\\n'\n '4. EXECUTE: Work through your solution methodically. Show all reasoning steps clearly.\\n'\n '5. VERIFY: Check your answer by substituting back, testing edge cases.\\n\\n'\n '# Mathematical Reasoning Principles:\\n'\n '- Break complex problems into smaller, manageable sub-problems\\n'\n ...\n '# Verification Requirements:\\n'\n ...\n '# Output Format:\\n'\n ...\n)\nTool prompt:
\ntool_prompt = (\n 'Use this tool to execute Python code for:\\n'\n '- Complex calculations that would be error-prone by hand\\n'\n ...\n 'Remember: Code should support your mathematical reasoning, not replace it. '\n 'Explain what you\\'re computing and why before running code.'\n)\nPreference prompt:
\npreference_prompt = (\n 'You have access to `math`, `numpy`, and `sympy` for:\\n\\n'\n '# Symbolic Computation (sympy):\\n'\n ...\n '# Numerical Computation (numpy):\\n'\n ...\n)\nThis baseline scored 40/50 on the public leaderboard.
\nThe new prompts collapse this to ~10 lines total, removing the reasoning scaffold entirely and keeping only the task contract and tool description.
\nSystem prompt (sets the task contract):
\nsystem_prompt = (\n 'Solve the following math problem step by step.\\n'\n 'Use Python code for calculations, verification, or brute-force search.\\n'\n 'The answer is a non-negative integer in [0, 99999].\\n'\n 'Put your final answer in \\\\boxed{}.'\n)\nTool prompt (describes the Python sandbox, attached to the Harmony ToolNamespaceConfig):
tool_prompt = (\n 'Execute Python code in a stateful Jupyter notebook. '\n 'Available: math, numpy, sympy, itertools, collections, mpmath (64-digit precision). '\n 'Use print() to see results.'\n)\nPreference prompt (appended to each problem; nudges library choice, not reasoning):
\npreference_prompt = (\n 'Use sympy for exact symbolic computation. '\n 'For number theory: use sympy.ntheory (factorint, divisors, totient, isprime). '\n 'Verify analytical solutions with numerical checks or brute-force when feasible.'\n)\nWith these changes (along with the configuration tweaks described in the next section), the score moved from 40/50 to 45/50. The prompt simplification appears to be a major factor, though it was not isolated from the other hyperparameter changes described in the Configuration section.
\nAs DeepSeek-R1 noted, heavy chain-of-thought scaffolding (UNDERSTAND → EXPLORE → PLAN → EXECUTE → VERIFY style) can interfere with models that already have strong internal reasoning. I believe gpt-oss-120b at reasoning_effort=HIGH is such a model, so I tried simplifying the prompts. They specify what the model must do (answer format, answer range) and what it has access to (tooling), but not how to think. Reasoning is left entirely to the model.
Compared to v45, fewer attempts run in parallel (16 → 8), but each gets more compute and time: workers per problem doubled (8 → 16), the early-stop threshold rose from 3 to 4 agreements, and the per-problem time budget grew (145s → 300s base, 250s → 900s ceiling). The full final configuration is below.
\n| Parameter | \nValue | \nNotes | \n
|---|---|---|
| Model | \ngpt-oss-120b | \nserved via vLLM, fp8 KV cache, prefix caching | \n
reasoning_effort | \nHIGH | \nHarmony system content | \n
| Context | \n65,536 tokens | \nmax_model_len | \n
| Per-turn max tokens | \n4,096 | \ntruncation guard | \n
| Temperature | \n1.0 | \n\n |
min_p | \n0.02 | \n\n |
| Attempts per problem | \n8 | \nparallel, independently seeded | \n
| Workers | \n16 | \nworker threads per problem | \n
| Turns per attempt | \n128 | \nmax tool-use turns | \n
| Early-stop threshold | \n4 agreements | \n\n |
| Voting | \nentropy-weighted | \nweight = 1 / mean_token_entropy | \n
| Base per-problem timeout | \n300s | \n\n |
| Max per-problem timeout | \n900s | \nextended if notebook time allows | \n
| Notebook runtime limit | \n17,400s | \nabout 4h 50m | \n
| Seed | \n42 | \nper-attempt seed = (seed + i)^2 | \n
During streaming, each chunk is watched for a } character. When one appears, the last 64 tokens of text are searched with the regex \\\\boxed\\s*\\{\\s*([0-9,]+)\\s*\\}. The trailing match wins, commas are stripped, and the value is kept if it falls in [0, 99999]. If the full message completes without a \\boxed{} match, a fallback regex final\\s+answer\\s+is\\s*([0-9,]+) is applied. Attempts without a valid answer contribute no vote.
The following variants were tested during development and did not improve on the configuration above:
\nTwo-round retry: If the initial 8 attempts produced ≤2 agreement, launch another 8 attempts with an additional 900s budget, then entropy-vote across all 16. Result: 40/50.
Temperature sweep: A sweep over {0.85, 1.00, 1.15} was run on 10 problems from reference.csv. The nine easy/medium problems were solved at all three settings. On the single hard problem, out of 8 parallel attempts, temp 0.85 yielded 1 correct, temp 1.00 yielded 0, and temp 1.15 yielded 3. Another run at temp 1.15 on the same problem yielded different hit counts, suggesting that factors beyond temperature alone (such as vLLM internal state, KV cache reuse, or scheduling nondeterminism) also affect outcomes. The signal was judged too noisy to justify deviating from temp 1.0.
Concept-first agentic loop: Four-step pipeline: (a) LLM enumerates relevant concepts for the problem, (b) LLM drafts a strategy from {problem + concepts}, (c) LLM identifies potential pitfalls and wrong assumptions, (d) LLM solves the problem with the full accumulated context. On the hard problems where the baseline already struggled, the extra stages did not produce more correct answers, and on some runs zero out of 8 attempts reached the correct answer.
Solver/verifier loop: The solver proposes a solving strategy, and the verifier returns either approved or rejected. The loop continues until a strategy gets approved, and the solver then uses that approved strategy to solve the problem. This also did not work out well. The verifier often approved flawed strategies or rejected reasonable ones, and the resulting solutions were no more accurate than single-shot solving. Correct and incorrect answers frequently tied in the final vote.
AIMO2-style verifier wave (from NVIDIA's AIMO2 solution): NVIDIA's AIMO2 protocol generates 64 candidates and runs a tournament-style verification: 32 times, the candidates are grouped into sets of 8, the best one from each group is picked, and then the best one is picked among those winners. The final answer is decided by majority vote across the 32 tournament winners. I tried a simpler version. If the initial 8 attempts had consensus ≤2, a second stage summarizes the 8 solutions and runs 8 verifier passes, where each pass sees the summarized candidates in a randomly shuffled order and picks the best one. The candidate with the most votes across the 8 passes wins. This did not work out well. The pipeline often ended up with the correct answer getting two votes and an incorrect one also getting two votes, with no clear winner. The approach might have worked better with more verifier passes, but I did not try that due to the time budget. One possible factor is that NVIDIA's GenSelect used a dedicated selector model trained on matched data, while I was using an off-the-shelf gpt-oss-120b for both solving and verification.
Per-answer verifier scoring wave: If the initial 8 attempts had ≤2 consensus, a second stage runs where every unique candidate answer is shown to the verifier 8 times. Each pass assigns a score in {1, 2, 3} reflecting how likely the candidate is correct. Scores are summed per candidate across all passes, and the candidate with the highest total wins. Same underlying problem as (5). The verifier was not a meaningfully stronger judge than the solver.
Across the variants I tested, adding structure to guide the model's reasoning (extra stages, verifier loops, scaffolded prompts) did not improve accuracy. The configuration above (minimal prompts, single-shot parallel sampling, entropy voting) outperformed all tested elaborations. NVIDIA's AIMO2 solution showed that verifier-based selection can work well, but it relied on a dedicated selector trained on matched data. It seems that imposing structure on how a strong reasoner thinks tends to hurt rather than help, unless the model has been trained for that specific structure.
\nAcross every variant I tested, adding structure (verifier loops, scaffolded prompts, multi-stage pipelines) hurt rather than helped. The minimal configuration outperformed all elaborations. It seems that for a model with strong internal reasoning, prompt and pipeline design should clarify the contract, not direct the thinking. Future work likely lies in better selection across attempts (a stronger judge than the solver itself), not in further structuring the solver.
", "rawMarkdown": "# AIMO Progress Prize 3\n\n## Acknowledgments\n\nThis solution builds on [Sachchidananda Daki's public notebook (v45)](https://www.kaggle.com/code/sachchidanandadaki/aimo-3-submission), which scored 40/50 on the public leaderboard using vLLM serving, Harmony encoding, a sandboxed Jupyter tool, parallel attempts, entropy-weighted voting, early-stopping, and adaptive time budgeting. My changes were to the prompts and several hyperparameters (detailed in the Prompts and Configuration sections below), which raised the score to 45/50. Thanks also to andreasbis/aimo-3-utils for the offline wheel bundle.\n\n## Overview\n\nThe solver runs `gpt-oss-120b` on a single H100 via vLLM, with fp8 KV cache, prefix caching, and a 64K context. For each problem, the system issues 8 independently seeded attempts in parallel at temperature=1.0 and reasoning_effort=HIGH. Each attempt has access to a stateful Jupyter sandbox (math, numpy, sympy, itertools, collections, mpmath with 64 digit precision) that the model can call mid-generation via Harmony tool-use messages. Output is streamed token by token, and whenever a } appears, the last 64 tokens are scanned. If \\boxed{...} is found, the attempt terminates immediately with that answer.\n\nOnce an attempt produces a final answer, it is pooled with the others. If four attempts agree on the same answer, the remaining attempts are cancelled (early stop). Otherwise the system waits for all 8 attempts to complete and then selects the answer using entropy-weighted voting.\n\nTime is budgeted globally within the notebook's total runtime limit of 17,400s. Each problem reserves a base budget of 300s but may extend up to 900s if remaining time exceeds what the remaining problems need.\n\n\n\n\nThe rest of this section zooms in on the pieces referenced above: the per-attempt turn loop, the entropy-weighted voting used for tie-breaking, the three prompts, the configuration, and how answers are extracted from streamed output.\n\n### Turn loop\n\nThe model can reason freely, but it cannot execute Python on its own. When it writes code inside a tool-call message and emits the hand-off token, generation has to stop so the orchestrator can actually run that code in a sandbox and feed the real output back. Without this pause, the model would just keep generating and hallucinate what it thinks the output should be. A turn is exactly one round of generation bounded by these hand-offs.\n\nAfter a turn ends, the orchestrator looks at the last message. If it's on the final channel, the boxed answer is extracted and the attempt is done. If it's a Python tool call, the code runs in the sandbox and the real output is appended as a tool-response message before the next turn begins. \n\n### Entropy-weighted voting\n\nDuring generation, at every token position the model produces we record the top-5 log-probabilities for that position. At the end of the attempt, each log-probability is converted to a probability (via `exp`), and the Shannon entropy at that position is computed as `-sum(p_i * log2(p_i))` across those 5 values. Summing those per-position entropies and dividing by the token count gives the attempt's mean entropy: a single number where low means the model was consistently sure of itself, high means it was uncertain throughout.\n\nEach attempt's vote is then weighted by 1 / mean_entropy. Weights are summed per candidate answer, and the answer with the highest total weight wins.\n\nA worked example makes the behavior clearer. Suppose a problem launched 8 attempts and 3 failed to produce a valid answer (timed out, crashed, or never emitted a `\\boxed{}`). The remaining 5 valid attempts look like this:\n\n| Attempt | Answer | Mean entropy |\n|---|---|---|\n| 1 | 8687 | 0.75 |\n| 2 | 8687 | 0.69 |\n| 3 | 48673 | 1.77 |\n| 4 | 48673 | 1.12 |\n| 5 | 14 | 0.88 |\n\nNo answer reached 4 agreements, so early-stop didn't fire, so we fall into the voting path.\n\n**Step 1: Invert each entropy into a weight (`1 / entropy`):**\n\n| Attempt | Answer | Entropy | Weight |\n|---|---|---|---|\n| 1 | 8687 | 0.75 | 1.334 |\n| 2 | 8687 | 0.69 | 1.449 |\n| 3 | 48673 | 1.77 | 0.565 |\n| 4 | 48673 | 1.12 | 0.893 |\n| 5 | 14 | 0.88 | 1.136 |\n\n**Step 2: Sum weights per candidate answer:**\n\n- 8687: 1.334 + 1.449 = 2.783\n- 48673: 0.565 + 0.893 = 1.458\n- 14: 1.136\n\n**Step 3: Pick the highest. 8687 wins with 2.783.**\n\n8687 and 48673 both had 2 raw votes, so a plain majority vote would have tied them. But 8687's attempts had low entropy (0.75 and 0.69), while 48673's were much noisier (1.77 and 1.12), so 8687 wins the weighted vote. Answer 14 was individually confident (entropy 0.88) but only had one attempt behind it, so it couldn't catch up.\n\n## Prompts\n\nThree prompts are used. All three are deliberately minimal. They state the contract and the available tooling, and do not prescribe reasoning steps.\n\nThe v45 baseline used a much more elaborate set of prompts (~65 lines total), explicitly scaffolding the model's reasoning through an UNDERSTAND → EXPLORE → PLAN → EXECUTE → VERIFY protocol, along with detailed mathematical reasoning principles, verification requirements, and library best-practices:\n\n**System prompt:**\n\n```python\nsystem_prompt = (\n 'You are an elite mathematical problem solver with expertise at the International '\n 'Mathematical Olympiad (IMO) level. Your goal is to find the correct answer through '\n 'rigorous mathematical reasoning.\\n\\n'\n '# Problem-Solving Approach:\\n'\n '1. UNDERSTAND: Carefully read and rephrase the problem in your own words.\\n'\n '2. EXPLORE: Consider multiple solution strategies. Think about relevant theorems.\\n'\n '3. PLAN: Select the most promising approach and outline key steps before executing.\\n'\n '4. EXECUTE: Work through your solution methodically. Show all reasoning steps clearly.\\n'\n '5. VERIFY: Check your answer by substituting back, testing edge cases.\\n\\n'\n '# Mathematical Reasoning Principles:\\n'\n '- Break complex problems into smaller, manageable sub-problems\\n'\n ...\n '# Verification Requirements:\\n'\n ...\n '# Output Format:\\n'\n ...\n)\n```\n\n**Tool prompt:**\n\n```python\ntool_prompt = (\n 'Use this tool to execute Python code for:\\n'\n '- Complex calculations that would be error-prone by hand\\n'\n ...\n 'Remember: Code should support your mathematical reasoning, not replace it. '\n 'Explain what you\\'re computing and why before running code.'\n)\n```\n\n**Preference prompt:**\n\n```python\npreference_prompt = (\n 'You have access to `math`, `numpy`, and `sympy` for:\\n\\n'\n '# Symbolic Computation (sympy):\\n'\n ...\n '# Numerical Computation (numpy):\\n'\n ...\n)\n```\n\nThis baseline scored 40/50 on the public leaderboard.\n\nThe new prompts collapse this to ~10 lines total, removing the reasoning scaffold entirely and keeping only the task contract and tool description.\n\n**System prompt** (sets the task contract):\n```python\nsystem_prompt = (\n 'Solve the following math problem step by step.\\n'\n 'Use Python code for calculations, verification, or brute-force search.\\n'\n 'The answer is a non-negative integer in [0, 99999].\\n'\n 'Put your final answer in \\\\boxed{}.'\n)\n```\n**Tool prompt** (describes the Python sandbox, attached to the Harmony `ToolNamespaceConfig`):\n```python\ntool_prompt = (\n 'Execute Python code in a stateful Jupyter notebook. '\n 'Available: math, numpy, sympy, itertools, collections, mpmath (64-digit precision). '\n 'Use print() to see results.'\n)\n```\n**Preference prompt** (appended to each problem; nudges library choice, not reasoning):\n```python\npreference_prompt = (\n 'Use sympy for exact symbolic computation. '\n 'For number theory: use sympy.ntheory (factorint, divisors, totient, isprime). '\n 'Verify analytical solutions with numerical checks or brute-force when feasible.'\n)\n```\n\nWith these changes (along with the configuration tweaks described in the next section), the score moved from 40/50 to 45/50. The prompt simplification appears to be a major factor, though it was not isolated from the other hyperparameter changes described in the Configuration section.\n\nAs DeepSeek-R1 noted, heavy chain-of-thought scaffolding (UNDERSTAND → EXPLORE → PLAN → EXECUTE → VERIFY style) can interfere with models that already have strong internal reasoning. I believe gpt-oss-120b at `reasoning_effort=HIGH` is such a model, so I tried simplifying the prompts. They specify *what* the model must do (answer format, answer range) and *what* it has access to (tooling), but not *how* to think. Reasoning is left entirely to the model.\n\n### Configuration\n\nCompared to v45, fewer attempts run in parallel (16 → 8), but each gets more compute and time: workers per problem doubled (8 → 16), the early-stop threshold rose from 3 to 4 agreements, and the per-problem time budget grew (145s → 300s base, 250s → 900s ceiling). The full final configuration is below.\n\n| Parameter | Value | Notes |\n|---|---|---|\n| Model | `gpt-oss-120b` | served via vLLM, fp8 KV cache, prefix caching |\n| `reasoning_effort` | HIGH | Harmony system content |\n| Context | 65,536 tokens | `max_model_len` |\n| Per-turn max tokens | 4,096 | truncation guard |\n| Temperature | 1.0 | |\n| `min_p` | 0.02 | |\n| Attempts per problem | 8 | parallel, independently seeded |\n| Workers | 16 | worker threads per problem |\n| Turns per attempt | 128 | max tool-use turns |\n| Early-stop threshold | 4 agreements | |\n| Voting | entropy-weighted | `weight = 1 / mean_token_entropy` |\n| Base per-problem timeout | 300s | |\n| Max per-problem timeout | 900s | extended if notebook time allows |\n| Notebook runtime limit | 17,400s | about 4h 50m |\n| Seed | 42 | per-attempt seed = `(seed + i)^2` |\n\n### Answer extraction\n\nDuring streaming, each chunk is watched for a `}` character. When one appears, the last 64 tokens of text are searched with the regex `\\\\boxed\\s*\\{\\s*([0-9,]+)\\s*\\}`. The trailing match wins, commas are stripped, and the value is kept if it falls in [0, 99999]. If the full message completes without a `\\boxed{}` match, a fallback regex `final\\s+answer\\s+is\\s*([0-9,]+)` is applied. Attempts without a valid answer contribute no vote.\n\n## What didn't work\n\nThe following variants were tested during development and did not improve on the configuration above:\n\n1. **Two-round retry**: If the initial 8 attempts produced ≤2 agreement, launch another 8 attempts with an additional 900s budget, then entropy-vote across all 16. Result: 40/50. \n\n2. **Temperature sweep**: A sweep over {0.85, 1.00, 1.15} was run on 10 problems from reference.csv. The nine easy/medium problems were solved at all three settings. On the single hard problem, out of 8 parallel attempts, temp 0.85 yielded 1 correct, temp 1.00 yielded 0, and temp 1.15 yielded 3. Another run at temp 1.15 on the same problem yielded different hit counts, suggesting that factors beyond temperature alone (such as vLLM internal state, KV cache reuse, or scheduling nondeterminism) also affect outcomes. The signal was judged too noisy to justify deviating from temp 1.0.\n\n3. **Concept-first agentic loop**: Four-step pipeline: (a) LLM enumerates relevant concepts for the problem, (b) LLM drafts a strategy from {problem + concepts}, (c) LLM identifies potential pitfalls and wrong assumptions, (d) LLM solves the problem with the full accumulated context. On the hard problems where the baseline already struggled, the extra stages did not produce more correct answers, and on some runs zero out of 8 attempts reached the correct answer.\n\n4. **Solver/verifier loop**: The solver proposes a solving strategy, and the verifier returns either approved or rejected. The loop continues until a strategy gets approved, and the solver then uses that approved strategy to solve the problem. This also did not work out well. The verifier often approved flawed strategies or rejected reasonable ones, and the resulting solutions were no more accurate than single-shot solving. Correct and incorrect answers frequently tied in the final vote.\n\n5. **AIMO2-style verifier wave (from NVIDIA's AIMO2 solution)**: NVIDIA's AIMO2 protocol generates 64 candidates and runs a tournament-style verification: 32 times, the candidates are grouped into sets of 8, the best one from each group is picked, and then the best one is picked among those winners. The final answer is decided by majority vote across the 32 tournament winners. I tried a simpler version. If the initial 8 attempts had consensus ≤2, a second stage summarizes the 8 solutions and runs 8 verifier passes, where each pass sees the summarized candidates in a randomly shuffled order and picks the best one. The candidate with the most votes across the 8 passes wins. This did not work out well. The pipeline often ended up with the correct answer getting two votes and an incorrect one also getting two votes, with no clear winner. The approach might have worked better with more verifier passes, but I did not try that due to the time budget. One possible factor is that NVIDIA's GenSelect used a dedicated selector model trained on matched data, while I was using an off-the-shelf gpt-oss-120b for both solving and verification.\n\n6. **Per-answer verifier scoring wave**: If the initial 8 attempts had ≤2 consensus, a second stage runs where every unique candidate answer is shown to the verifier 8 times. Each pass assigns a score in {1, 2, 3} reflecting how likely the candidate is correct. Scores are summed per candidate across all passes, and the candidate with the highest total wins. Same underlying problem as (5). The verifier was not a meaningfully stronger judge than the solver.\n\nAcross the variants I tested, adding structure to guide the model's reasoning (extra stages, verifier loops, scaffolded prompts) did not improve accuracy. The configuration above (minimal prompts, single-shot parallel sampling, entropy voting) outperformed all tested elaborations. NVIDIA's AIMO2 solution showed that verifier-based selection can work well, but it relied on a dedicated selector trained on matched data. It seems that imposing structure on how a strong reasoner thinks tends to hurt rather than help, unless the model has been trained for that specific structure.\n\n\n## Conclusion\n\nAcross every variant I tested, adding structure (verifier loops, scaffolded prompts, multi-stage pipelines) hurt rather than helped. The minimal configuration outperformed all elaborations. It seems that for a model with strong internal reasoning, prompt and pipeline design should clarify the contract, not direct the thinking. Future work likely lies in better selection across attempts (a stronger judge than the solver itself), not in further structuring the solver.", "votes": 19 }, { "id": 3447776, "postDate": "2026-04-23T22:05:14.563Z", "content": "The code is shared; too bad, I cannot access it somehow. 😭
", "rawMarkdown": "The code is shared; too bad, I cannot access it somehow. 😭", "replies": [ { "id": 3447821, "postDate": "2026-04-24T03:23:39.293Z", "content": "Thanks for pointing that out. I just made it public. You should be able to see it now :)
", "rawMarkdown": "Thanks for pointing that out. I just made it public. You should be able to see it now :)" } ] }, { "id": 3452683, "postDate": "2026-05-04T06:21:41.023Z", "content": "Thanks for your share. I wonder the model is a fine-tune version or original version?
", "rawMarkdown": "Thanks for your share. I wonder the model is a fine-tune version or original version?", "replies": [ { "id": 3456302, "postDate": "2026-05-11T17:07:30.213Z", "content": "Thanks! It's the Unsloth version of gpt-oss-120b, not fine-tuned.
", "rawMarkdown": "Thanks! It's the Unsloth version of gpt-oss-120b, not fine-tuned. " } ] }, { "id": 3452541, "postDate": "2026-05-03T17:48:23.667Z", "content": "Many of my submissions I lean on how solved my model the most hard problem. If the model successfully solved it I sent. but my model got score lower than 43
", "rawMarkdown": "Many of my submissions I lean on how solved my model the most hard problem. If the model successfully solved it I sent. but my model got score lower than 43" }, { "id": 3448189, "postDate": "2026-04-24T21:13:33.730Z", "content": "The private LB is not yet finalized
", "rawMarkdown": "### 45 / 50 public · 45 / 50 private?\n\nThe private LB is not yet finalized" } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T04:25:09Z", "competition_id": 118448, "competition_slug": "ai-mathematical-olympiad-progress-prize-3", "team_id": 15217078, "solution_url": "/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/less-prompting-more-trust" }