{ "id": 692715, "title": "Recursive Self Aggregation with Difficulty Aware Scheduler", "content": "

Public Leaderboard Score Range: 40 - 42

\n

Private Leaderboard Score: 41.5

\n

This work aimed to reduce variance of scores between runs. Judging from the close alignment between the private and public leaderboard scores, we believe we have succeeded in doing so.

\n

Resources

\n\n

Abstract

\n

This writeup describes the solution we used for the competition. The base model was gpt-oss-120b. The main benefit of this system over the self consistency baseline is, the observed mean increased from 38 to 41, and the variance decreased from (35-45) to (40-42). When writing this, we focused on the design choices we made, and why we made them. As a small team, we did not have the resources to run large scale experiments on many different models to fully test multiple different hypotheses. Instead, we tried to extrapolate from observations made from a small number of problems on two benchmarks: The reference benchmark provided by the AIMO3 organizing team, and another one we constructed on our own.\n
\n
\nOur solution is novel: No other solution used Recursive Self Aggregation as the core inference method together with a scheduler built around global estimates of problem difficulty. Most other solutions relied on plain majority voting and schedulers based only on remaining time slack. Our approach instead treats time allocation and inference as a coupled decision problem: the scheduler reasons about when RSA is worth continuing, and RSA is tuned around the budget structure imposed by the scheduler.

\n

Introduction

\n

The first three sections describe the main ideas behind the system, with greater emphasis on the components that required the most effort to design and implement. The fourth section explains how these ideas were combined into a system that performs well under the competition’s constraints. The fifth section presents the internal benchmark we used to guide development. The sixth section is an ablation study. Finally, the seventh section discusses limitations of the current system and potential directions for future work.\nThe write up is organized as follows:

\n
    \n
  1. Inference Algorithm: Recursive Self Aggregation
    2. \n
  2. Time Management: AI Hard Problems and Time Allocation
    3. \n
  3. Infrastructure Improvements
    4. \n
  4. Putting Everything Together
    5. \n
  5. Internal Benchmark
    6. \n
  6. Ablation Study
    7. \n
  7. Potential Weaknesses and Future Work
    8. \n
\n

1. Inference Algorithm: Recursive Self Aggregation

\n

We begin with the inference algorithm: Recursive Self Aggregation (RSA) [6]. RSA is a Test-time scaling (TTS) strategy that combines the the strongest features of several commonly used inference methods, including self-consistency [8], best-of-n [9], Monte-Carlo Tree Search [10], and sequential refinement [11]. \nTest-Time Scaling (TTS) refers to the class of strategies that improves language model performance by allocating additional compute at inference time. In Section 1.1, we briefly review several common TTS algorithms. In Section 1.2, we discuss their drawbacks. In Section 1.3, we explain Recursive Self Aggregation and compare it with the methods introduced earlier.

\n

1.1 Related Work: Common Test-Time Scaling Algorithms

\n\n

1.2 Drawbacks

\n

Each of the methods above has important limitations.

\n\n

1.3 Recursive Self Aggregation

\n

\"\"\nDiagram taken from [6]: Recursive Self-Aggregation Unlocks Deep Thinking in Large Language Models\n
\n
\nRecursive Self Aggregation (RSA)[6] is a Test-Time Scaling method that addresses many of the drawbacks discussed above. It is a hybrid scaling strategy that combines both parallel exploration and sequential refinement. RSA proceeds as follows:

\n
    \n
  1. Round 0: Sample N solutions.
    2. \n
  2. Round 1: Using the previous N solutions from Round 0 as hints in the prompt, ask the model to generate another N solutions.
    3. \n
  3. Round 2: Using the previous N solutions from Round 1 as hints, then generate another N solutions.
    4. \n
  4. Round 3: Using the previous N solutions from Round 2 as hints, then generate another N solutions.
    5. \n
  5. …
    6. \n
  6. Continue until the compute budget is exhausted or an early stopping condition is met.\nOne example run of RSA on the Norwegian integers problem from the reference benchmark is shown below:\n\"\"\nEarly Stop Condition: A simple early stopping rule is to terminate as soon as the majority class in a round reaches size 4. In practice, the stopping rule can vary across rounds, and these choices have a substantial effect on time usage per problem. We will return to this in the time management section.
    7. \n
\n\n

1.4 Fresh Attempts

\n

Empirically, RSA has a much higher chance of finding the correct answer when that answer is already present in the initial batch of N generations. However, some problems have the correct answer occupying only a very small portion of the answer distribution, often less than 0.2. In such cases, the correct answer may fail to appear in the initial N=8 generations simply due to bad luck. \n
\n
\nTo mitigate this, we reserve a small number of fresh attempts in later RSA rounds, usually F=2, in which the model is not shown the previous round’s solutions. This helps preserve diversity and reduces the risk of collapse toward an unhelpful local mode. The diversity-collapse issue, where RSA can fail if the previous solution pool is not diverse enough, is also hinted at in [5]. A related solution is proposed in [4] page 5, under Phase III: Refinement with Global Memory, where the authors suggest including fresh solver attempts during refinement rounds. We adapted that idea to the AIMO3 pipeline.

\n

2. Time Management: AI Hard Problems and Time Allocation

\n

We measure the AI-Hardness of a problem by two metrics:

\n
    \n
  1. Metric 1: The runtime needed for one attempt on the problem.
    2. \n
  2. Metric 2: The probability of producing the correct answer in one attempt.
    3. \n
\n

2.1 Why a scheduler matters

\n

In the 50 leaderboard problems, some are solved quickly and reliably, while others require substantially more search. A scheduler that ignores this variation will tend to waste time on short problems and cut off long problems too early.\nA good scheduler therefore has two jobs:

\n
    \n
  1. Ensure that every problem has enough time for at least one pass, based on metric 1.
    2. \n
  2. Identify problems that, when allocated more time, can improve on accuracy, based on metric 2.
    3. \n
\n

2.2 Difficulty Buckets and Reserve Policy

\n

Some information was shared about the difficulty of the public and private test sets on kaggle and on X. As mentioned above, the most popular public notebook scored between 35 and 45 on the public leaderboard, and based on forum discussion and our own submissions, its mean performance was around 37–39.\n
\n
\nUsing this information, we designed a scheduler based on estimated problem difficulty.\n
\n
\nThe first step was to group the 50 problems into rough difficulty buckets. we assumed that the private set would follow a similar distribution. Our working hypothesis was that strong public notebooks solved essentially all problems in bucket A, most in B, around half of C, and only a small minority of D and E. The suggested time ranges were based on observations from the 10 reference problems and the imo_answerbench benchmark.\n
\n
\nWe make the assumption that problems with lower pass@n take more time to run. The reality is more nuanced. There is a tiny class of problems where the model answers very quickly, but is almost always wrong. This will be discussed further in section 5.2 on the internal benchmark.\n\"\"\nFrom these buckets, we designed a scheduler that explicitly accounts for problem difficulty. The key challenge is to avoid pathological behaviour when hard problems arrive early. In particular, the scheduler must avoid panicking after seeing several difficult problems in a row and becoming too conservative, since that would leave too much time unused for easier problems later.\n
\n
\nThe scheduler therefore reserves time for future problems in a way that depends on how many remain. Early in the run, when more than 30 problems remain, it reserves only 250 seconds per future problem. In the middle of the run, with 16–29 problems remaining, it reserves 270 seconds per future problem. Late in the run, with 15 or fewer problems remaining, it becomes more conservative and reserves 295 seconds per remaining problem.\n
\n
\nIn addition, the scheduler maintains a time bank. The time bank tracks how far ahead or behind schedule the run is. Conceptually, the scheduler assumes a nominal average runtime per problem and accumulates deviations from that average. If earlier problems were solved faster than expected, the time bank becomes positive and later problems are allowed more time. If earlier problems consumed more time than expected, the time bank becomes negative and later budgets are reduced.

\n

2.3 RSA Gating: Consensus, Time, and Progress

\n

Recursive Self Aggregation is powerful, but it is also expensive. If used too freely, it can consume a large fraction of the notebook budget for very little gain. The scheduler therefore needs a rule for deciding when to stop after the initial batch and when to launch another RSA round.\n
\nWe implemented this as a three-part gating mechanism: consensus, time, and progress.\n

\n\n

2.4 Practical Effect: Low Expected Number of Time-Starved Problems

\n

\"\"\nA useful way to evaluate the scheduler is to ask how often it forces the system to cut off a problem materially earlier than would have been appropriate for its difficulty. We refer to such cases as Time-Starved problems.\n
\n
\nFor example, a problem in bucket D/E is Time-Starved if it was not allocated the full 1050 seconds by the scheduler AND the solution was cut short (i.e. system wants to continue with RSA from consensus and progress evaluations, but could not do so due to the time condition). A problem in any difficulty bucket A-E is also considered Time-Starved if the system does not have time to even complete round 0, the initial round of 8 generations (and forced to return N/A answer).\n
\n
\nTo check the effectiveness of our scheduler, we used dynamic programming (warning: takes a long time to run) and monte carlo simulation to estimate the number of time-starved problems in a typical run. The dynamic programming scripts deterministically calculates the expected value of time starved questions over all 50! possible orderings, while the monte carlo simulations finds the average over 20000 random orderings.\n
\n
\nAssuming average runtimes as per the table in section 2.2, both dynamic programming and monte carlo simulation estimates that the average count is less than 0.001. Assuming the upper end of runtimes, monte carlo simulation estimates the count is 0.506.\n
\n
\nWe stress that these estimates depend on the accuracy of both the bucket assumptions and the runtime model. Still, this estimate provides some assurance. The scheduler is expected to fail catastrophically only very rarely.\n
\n
\nOn the other hand, monte carlo simulation for old scheduler under pessimistic runtime assumptions gives an expected count of 0.932 time starved questions per run.\n
\n
\nImplementation details of the time scheduler and RSA gating are in Appendix C.

\n

3. Infrastructure Improvements

\n

As mentioned in section 2, Recursive Self Aggregation (RSA) is a very costly process. This means that increasing token throughput of the system is a direction very worth exploring. This section is dedicated to describing the improvements and optimizations I made to speed up inference and increase system reliability.

\n

3.1 Speculative Decoding

\n

Speculative decoding increases token throughput by using a smaller and faster draft model to propose several next tokens, and then asking the main model to verify them in parallel. Instead of forcing the large model to generate every token one by one, the system lets the small model guess ahead and uses the large model mainly as a checker. When the draft model’s guesses are accepted, multiple tokens can be emitted in a single verification step, which reduces inter-token latency and increases token throughput.\n
\n
\nFor the draft model, we used this draft model trained by wenliangtlh. This was the only model that was able to consistently speed up generation. Observed runtimes on the 10 reference problems decreased by about 15% with this change.\n
\n
\nWe also tried Nvidia's model trained for long contexts. However, this model consistently lead to a slowdown in generation.

\n

3.2 Prefix Caching (vllm >= 0.15.1)

\n

Another important system optimization was prefix caching. vLLM’s Automatic Prefix Caching (APC) stores the KV-cache blocks for prompts that have already been processed. When a later request shares the same prefix, vLLM can directly reuse the cached KV states and skip recomputing that shared part of the prompt. This is very important for AIMO3 because parallel generations always share the same prompt. System prompts, format hints and RSA prompts also stay the same across many vLLM requests. \n
\n
\nFor GPT-OSS models specifically, upgrading to vLLM >= 0.15.1 was important because the official release notes for that version explicitly list a bug fix: “Fixed prefix cache hit rate of 0% with GPT-OSS style hybrid attention models.” This upgrade further decreased runtimes on the 10 reference problems by another 10%.

\n

3.3 Parameter optimizations

\n

There were a number of further optimizations made to the system, including setting max_model_len = 81920, quantizing the KV-Cache using llm-compressor, guarding against vLLM crashes, dealing with memory issues in the python tool, using the Z3 solver, parsing code from the model, and how the final answer is parsed. \n
\n
\nFurther details on these improvements are in Appendix A.

\n

4. Putting Everything Together

\n

The final system combines RSA, tool use, prompting, scheduler-based time management, and inference/infrastructure optimizations into one end-to-end pipeline.

\n
    \n
  1. When a problem is served, the system first formats the problem using a prompt template.
    2. \n
  2. It then runs Round 0, generating an initial batch of 8 candidate solutions in parallel.
    3. \n
  3. Each solution is parsed into a final integer answer, and the resulting answer histogram is used to check whether the answer is already clear enough to accept.
    4. \n
  4. If Round 0 produces a strong consensus, the system stops immediately. This is how most easy and medium problems are handled.
    5. \n
  5. If the answer distribution is not yet convincing, the system does not automatically continue. Instead, the scheduler decides whether another RSA round is worthwhile.
    6. \n
  6. This decision is based on three conditions discussed earlier:
    7. \n
  7. If another RSA round is approved, the system builds a new prompt using the previous round’s solutions as hints.
    8. \n
  8. At the same time, a small number of fresh attempts are reserved, so that not all generations are conditioned on the previous round. This helps reduce diversity collapse.
    9. \n
  9. The next batch is again generated in parallel, parsed, aggregated, and evaluated using the same stopping logic.
    10. \n
  10. This loop continues until either:
    11. \n
\n\n

5. Internal Benchmark of AI-Hard Problems

\n

In this section, we study the relationship between accuracy, runtime, and RSA behaviour.

\n

5.1 Constructing the Benchmark

\n

Problems

\n

We curated a small internal benchmark to experiment on, in order to identify the main failure modes of the system. \nThe internal benchmark consists of 17 problems. Its purpose was not to approximate the full competition distribution, but rather to provide a stable stress test for scheduler behaviour and the ability of RSA to recover answers beyond a single majority-vote round. It was on these observations that we designed the scheduler and RSA gating rules. The problems are:

\n
    \n
  1. imo-bench-combinatorics-042
    2. \n
  2. imo-bench-combinatorics-054
    3. \n
  3. imo-bench-combinatorics-057
    4. \n
  4. imo-bench-combinatorics-068
    5. \n
  5. imo-bench-number_theory-023
    6. \n
  6. imo-bench-algebra-056
    7. \n
  7. imo-bench-algebra-100
    8. \n
  8. imo-bench-combinatorics-013
    9. \n
  9. imo-bench-combinatorics-014
    10. \n
  10. imo-bench-combinatorics-031
    11. \n
  11. imo-bench-combinatorics-039
    12. \n
  12. imo-bench-combinatorics-049
    13. \n
  13. imo-bench-algebra-015
    14. \n
  14. imo-bench-algebra-026
    15. \n
  15. imo-bench-algebra-052
    16. \n
  16. imo-bench-algebra-055
    17. \n
  17. 86e8e5 (Norwegian Integers)
    18. \n
\n

Construction Process

\n

The creators of the imo_answerbench dataset noted in their paper that they modified the problems in several ways to harden them against AI. Hardening methods include paraphrasing, changing the name of objects in the problem (such as changing point names for geometry problems), reformulating, modifying numerical values\nand/or adding distractors to the problem. This process was done either manually or automatically using language models.\n
\n
\nUnfortunately, some of these modifications lead to genuine mathematical errors in the problem statement, causing the ground truth answer to be wrong. When constructing our benchmark, we manually reviewed the problems to ensure that only correct problems are retained. Ambiguous and wrong problems were discarded. \n
\n
\nSome examples of discarded problems are in Appendix D.\n
\n
\nWe constructed this benchmark in several stages:

\n
    \n
  1. Filtered the imo_answerbench dataset to problems with integer answers only.
    2. \n
  2. Ran a majority@8 solving process on the filtered set, with 4 independent runs per problem.
    3. \n
  3. Selected the problems that were not solved in at least one of the 4 runs, yielding 53 candidate hard problems.
    4. \n
  4. Manually reviewed these candidates to remove problems that were not suitable for benchmarking, such as problems with incorrect ground-truth answers due to mathematical errors in the statement, and problems where failure was caused mainly by language or convention mismatches rather than genuine difficulty.
    5. \n
  5. Retained 16 problems that were judged to be both valid and genuinely difficult.
    6. \n
  6. Added the Norwegian integers problem from the official AIMO3 reference benchmark, giving a final benchmark of 17 problems.\n
    \n
    7. \n
\n

5.2 Observations from the Benchmark

\n

Experiments on the benchmark revealed an interesting pattern: when the system failed, the resulting answer histograms tended to fall into a small number of recurring classes.

\n\n

6. Ablation Study

\n

6.1 RSA versus V1-Infer

\n

V1-Infer [5] is a scoring method for solutions generated in parallel. After generating N candidates, it compares them pairwise and asks the model which solution is better. The comparison schedule is designed so that every candidate is matched against at least several others, and a final score is computed from both its win rate and the strength of the candidates it defeats. The main idea is that models are often better at making relative judgments between two candidates than at assigning a stable absolute score to one candidate on its own.\n\"\"\nWe made five public leaderboard submissions using V1-Infer, and they all scored between 37 and 39. \n\"\"\nThis was below the 40 to 41 achieved by the RSA-based notebooks, excluding the run where we changed the per-problem time reserve. Based on this, we decided not to explore V1-Infer further. The V1 paper also reports that V1-Infer tends to outperform methods that score each solution independently, so we did not spend additional time exploring those alternatives either.

\n

6.2 Prompts

\n

Throughout the competition, we experimented with a few different system prompts. The first one was adapted from the paper 11. It had the strange effect of making the model reason faster, but at a lower quality. It was kind of like decreasing the reasoning effort from HIGH to MEDIUM-HIGH. The following table records the effect on runtime of the system prompt from the cognitive well paper on the reference benchmark:\n\"\"\nIn this figure, each row corresponds to a problem and each column to a run. The table on the left uses the system prompt from the public notebooks, while the table on the right uses the prompt adapted from the Cognitive Well paper. Each cell shows the runtime for that problem in that run. Green indicates that the problem was solved correctly, while red indicates an incorrect answer. The shade encodes how many RSA rounds were used: lighter shades correspond to more rounds. No problem required more than 3 RSA rounds.\nFrom this table, the average runtime decreased by about 20%, from roughly 3000 seconds to 2400 seconds. However, this speedup came with a clear drop in reasoning quality. With the new prompt, the model got more problems wrong in some runs, and more problems required RSA, suggesting that round 0 answers were less reliable and needed additional refinement.\n\"\"\nThis reduction in reasoning quality was also reflected on the public leaderboard. Submissions using this system prompt scored between 30 and 35, well below the 38 to 39 baseline obtained by the public notebooks.\nWe also tried another prompt that included notes on mistakes the model had made on the IMO AnswerBench dataset. This produced no clear improvement or degradation.\nIn the end, we settled on the simplest possible system prompt, specifying only the required output format. Shorter system prompts reduce prefill overhead for every request.\nfull prompts are provided in Appendix E.

\n

7. Potential Weaknesses and Future Work

\n

The main weakness of the system is on problems where the model converges quickly and confidently to a wrong answer. On the internal benchmark, these were Classes 4 and 5 where the answer histogram looks deceptively clean. For some problems the runtime is also short, so neither answer histograms nor timing heuristics reliably identifies them as hard cases. This creates a fundamental tension in the scheduler. Aggressive early stopping saves time on easy problems, but also risks missing these failures. A promising direction for future work is to develop stronger continuation rules that use richer signals than answer histograms and runtime alone.

\n

References

\n

Papers

\n
    \n

  1. Huang, B., Li, S., Wu, T., Yang, Y., Talwalkar, A., Ramchandran, K., Jordan, M. I., & Jiao, J. (2026). Sample complexity and representation ability of test-time scaling paradigms. ICLR 2026. link

    2. \n

  2. Dorner, F. E., Chen, Y., Cruz, A. F., & Yang, F. (2026). ROC-n-reroll: How verifier imperfection affects test-time scaling. ICLR 2026. link

    3. \n

  3. Khairi, A., D'souza, D., Fadaee, M., & Kreutzer, J. (2025). Making, not taking, the best of N. ICLR 2026 link

    4. \n

  4. Dang, X., Agarwal, R., Porto, R., Goyal, A., Fowl, L. H., & Arora, S. (2026). Escaping the cognitive well: Efficient competition math with off-the-shelf models. link

    5. \n

  5. Singh, H., Li, X., Sareen, K., Maheswaran, M., Tan, S., Wu, X., Wang, J., Ariyak, A., Wu, Q., Khaki, S., Tiwari, R., Lian, L., Lu, Y., Li, B., Suhr, A., Athiwaratkun, B., & Keutzer, K. (2026). V_1: Unifying generation and self-verification for parallel reasoners. link

    6. \n

  6. Venkatraman, S., Jain, V., Mittal, S., Shah, V., Obando-Ceron, J., Bengio, Y., Bartoldson, B. R., Kailkhura, B., Lajoie, G., Berseth, G., Malkin, N., & Jain, M. (2025). Recursive self-aggregation unlocks deep thinking in large language models link

    7. \n

  7. Chen, W.-L., Peng, L., Tan, T., Zhao, C., Chen, B. J., Lin, Z., Go, A., & Meng, Y. (2026). Think deep, not just long: Measuring LLM reasoning effort via deep-thinking tokens. link

    8. \n

  8. Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., & Zhou, D. (2023). Self-consistency improves chain of thought reasoning in language models. ICLR 2023. link

    9. \n

  9. Huang, A., Block, A., Liu, Q., Jiang, N., Foster, D. J., & Krishnamurthy, A. (2025). Is Best-of-N the best of them? Coverage, scaling, and optimality in inference-time alignment. ICML 2025. link

    10. \n

  10. Xie, Y., Goyal, A., Zheng, W., Kan, M.-Y., Lillicrap, T. P., Kawaguchi, K., & Shieh, M. (2024). Monte Carlo tree search boosts reasoning via iterative preference learning. NeurIPS 2024 Sys2-Reasoning Workshop. link

    11. \n

  11. Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., & Clark, P. (2023). Self-refine: Iterative refinement with self-feedback. NeurIPS 2023. link

    12. \n

  12. Han, Q., Khamaru, K., & Zhang, C.-H. (2024). UCB algorithms for multi-armed bandits: Precise regret and adaptive inference. arXiv:2412.06126. link \nhttps://arxiv.org/abs/2412.06126

    13. \n

  13. Luong, T., Hwang, D., Nguyen, H. H., Ghiasi, G., Chervonyi, Y., Seo, I., Kim, J., Bingham, G., Lee, J., Mishra, S., Zhai, A., Hu, C. H., Michalewski, H., Kim, J., Ahn, J., Bae, J., Song, X., Trinh, T. H., Le, Q. V., & Jung, J. (2025). Towards robust mathematical reasoning. EMNLP 2025. link

    14. \n
\n

Notebooks

\n
    \n
  1. streaming-inference by Tong Hui Kang
    2. \n
  2. [44/50] LET ME (over)COOK!!! by parthenos
    3. \n
  3. [41/50] AIMO3: Confidence Voting by parthenos
    4. \n
  4. AIMO3-Qwen3.5-35Ba3B-simple-MCTS by yuri-ao
    5. \n
  5. [Way to 30+/50] GPT-OSS - w/ python tool by ShelterW
    6. \n
  6. AIMO 3 | GPT-OSS-120B (with tools) by Andreas Bisiadis
    7. \n
\n

Models

\n
    \n
  1. UnSloth's gpt-oss-120b
    2. \n
  2. Wenliangth's AIMO3 speculative decoding draft model
    3. \n
  3. Nvidia's long context speculative decoding draft model
    4. \n
\n

Appendix Contents

\n
    \n
  1. Appendix A: Model Parameters and Other Small Improvements
    2. \n
  2. Appendix B: Additional RSA Example Runs
    3. \n
  3. Appendix C: Scheduler and RSA Gating Implementation Details
    4. \n
  4. Appendix D: Internal Benchmark
    5. \n
  5. Appendix E: Prompts
    6. \n
\n

Appendix A: Model Parameters and Other Small Improvements

\n

There were a number of improvements made to the system that improved its speed and stability. Their details are included in this appendix.

\n

A.1 max_model_len

\n

This needs to be carefully balanced between concurrency and accuracy. We observed that for one problem on the reference and some more on imo_answerbench, the solution length frequently exceeded max_model_len = 65536 set in the public notebooks. The answers for these solutions will be parsed as N/A, which hurts performance. On the other hand, setting max_model_len = 131072 (maximum supported by gpt-oss-120b) hurts concurrency and hence end-to-end latency per problem, due to limited GPU memory and kv-cache size. In the end, we settled with max_model_len = 81920. This value has decent concurrency on H100 GPUs (~ 5x) and significantly increases the number of complete solutions.

\n

A.2 Quantized KV Cache

\n

Another important parameter optimization was quantizing the KV cache. In transformer inference, the key and value tensors from previous tokens are stored in memory so that later tokens can attend to them. For long-context generation, this KV cache consumes a large amount of GPU memory and is often the main bottlenecks limiting concurrency. vLLM supports quantizing this cache to FP8, which substantially reduces its memory footprint. For this, we used kv_cache_dtype=fp8_e4m3.\nQuantizing the KV cache to FP8 requires choosing appropriate scale factors for the key and value tensors. These scales determine how the original tensors are mapped into the much lower-precision FP8 format and then reconstructed during attention. Calibration is the process of estimating suitable scales from observed activations so that FP8 uses its limited numeric range effectively and minimizes quantization error. \nWithout calibration, vLLM falls back to default scales of 1.0 when no checkpoint scales are available, which is a much cruder approximation and can degrade attention quality, especially on long or delicate reasoning traces. In this work, we experimented with both lightweight runtime calibration via calculate_kv_scales=True, and offline calibration using the llm-compressor package on the AIMO3-High-Difficulty-Tool-Calling-Dataset by Wenliangth. The latter uses representative data to produce scales better matched to the target workload.\nThe final submission was the model calibrated with llm-compressor. vLLM provides some documentation for using this package here.

\n

A.3 Guarding against vLLM crashes

\n

There is a very rare bug (causing vLLM to crash) that may be triggered when gpt-oss-120b is used with speculative decoding under high concurrent workloads. This is a very recent issue reported here. We do not know the root cause of it, but we guarded against it by checking for vLLM server health at the beginning of each problem, restarting the server if necessary. Otherwise, the system will return N/A for the rest of the unsolved problems.

\n

A.4 Python Tool

\n

A number of improvements were made to the python tool.

\n

A.4.1 Memory Issues

\n

The python tool causes memory spikes. From experiments, there are two main reasons that cause this:

\n\n

A.4.2 Z3 Solver

\n

We also installed the z3 package, which is an SMT (Satisfiability Modulo Theory) solver developed by Microsoft Research. At a high level, Z3 is designed to determine whether a collection of logical, algebraic, or combinatorial constraints can be satisfied, and if so, to return a solution. For AIMO3, this can be useful when the model reformulates part of a problem into a system of equations, inequalities, divisibility conditions, or discrete case constraints. GPT-OSS-120B has been observed to use this package if told via prompting that it is available.

\n

A.4.3 Parsing Code from Model Output

\n

A small but important engineering issue arises from how gpt-oss-120b writes Python code. The model often writes code in the style of a Jupyter notebook cell, where the final expression is implicitly displayed without being explicitly printed. For example, in a notebook cell, code such as

\n
x = 2\ny = 3\nz = x + y\nz\n
\n

will display 5 automatically. However, when the same code is executed inside the tool subprocess, the final expression is not written to stdout, so nothing is captured by the tool backend. A common workaround in public notebooks is to wrap the final line in print(...) before execution. \nHowever, this approach is fragile. The last line of the generated code is not always a simple expression. For example, the model may end with a control-flow statement such as continue, break, or pass, or with the tail of a loop or conditional block. In such cases, blindly rewriting the last line into print(...) can produce invalid code such as print(continue), which immediately raises a syntax error.\nTo make code execution robust, we instead used Python’s abstract syntax tree (ast) module to analyze the generated code structurally before execution. The parser first walks the syntax tree to detect whether the model already included any explicit print(...) statements. If so, the code is executed as written, since the model has already specified what should be sent to stdout. If no print statements are present, we collect the variables assigned anywhere in the code and append additional code that prints their values before execution.

\n

A.5 Streaming Behaviour and Answer Parser

\n

Another small but important difference from many public notebooks is how we handled streaming output and answer extraction.\nIn several public notebooks, the model output is streamed token by token, and generation is terminated as soon as \\boxed{...} appears in the output. This is an attractive optimization because it can save time on easy problems where the answer is produced early. However, we found this unsafe for our RSA pipeline for two reasons:

\n\n

Appendix B: Additional RSA Example Runs

\n

B.1 Norwegian Integers (86e8e5)

\n

\"\"\n
\n

\n\n

B.2 imo-bench-algebra-100

\n

\"\"\nThis problem is on Vasc inequality. The answer is 8. Refer to appendix D for more details. It is an example of a problem in Class 5 as described in section 5.2 , where the correct answer is in a tiny minority and a single wrong answer dominates the majority.\nThis run had early-stop set to 4 instead of 6. The system returned an answer relatively quickly (~270 s) with a strong majority. However, the majority is wrong.\n\"\"\nWe might also get lucky on the run. In the above run, the decision margin was set to 3, meaning that round 0 (index 0-7) results being a 5-3 split for the answers 2 and 8 triggered RSA round 1. after two further rounds of RSA, the system decided that the answer is 8, which is correct.

\n

B.3 Reference Problem 7 (641659)

\n

\"\"\n
\n
\nThis is problem 7 on the reference benchmark. The answer is 57447. This run also had early-stop set to 4. The system returned an answer in ~292 s, which is similar to imo-bench-algebra-100 above. The answer distribution is also similar, with one answer firmly in the majority class. The only difference is that the majority class is correct, where as for imo-bench-algebra-100 it is wrong. This makes it very difficult to differentiate these two problems via runtime and answer distribution heuristics alone.\nThis creates a tension between early accept based on majority voting versus launching RSA.

\n

B.4 imo-bench-combinatorics-068

\n

\"\"\n
\n
\nThis is a combinatorics problem from the imo_answerbench dataset. It is an example of a problem in Class 2 as described in section 5.2, where there is an even split between two answers. The model was asked to return its answer modulo 100000, to fit within AIMO3's answer range of 0 - 99999. The answer is 93447.\nThe model is evenly split between 93447 and 71207. In an unlucky run (such as this one), simply taking majority vote at N=8 will yield the wrong answer. RSA was able to correct the error.

\n

B.5 imo-bench-number_theory-023

\n

\"\"\n
\n
\nThis is a number theory problem from the imo_answerbench dataset. It is another example of a problem in Class 2 as described in section 5.2, where there is an even split between two answers. The answer is 48.\nThe model is split between 48 and 64. In an unlucky run, majority vote at N=8 might yield the wrong answer. RSA is able to identify the answer.

\n

B.6 imo-bench-combinatorics-057

\n

\"\"\n\"\"\n\"\"\n\"\"\n
\n
\nThis is a combinatorics problem from the imo_answerbench dataset. It is an example of a problem in Class 2 as described in section 5.2 , where the correct answer is in a small minority and a few wrong answers form the majority. RSA was able to obtain the correct answer after 3 rounds, which surprisingly managed to fit within the time budget.

\n

Appendix C: Scheduler and RSA Gating Implementation Details

\n

Early in the competition, this notebook by ShelterW used a scheduler that allocated a fixed amount of time to each problem, without accounting for problem difficulty. That approach typically averaged around 30–32 on the public leaderboard, with a best observed leaderboard score of 37. Later notebooks by parthenos and Andreas Bisiadis introduced an improved scheduler that reserved a fixed, below-average amount of time for each remaining problem. This allowed hard problems encountered early to “borrow” time from easier problems later in the run. Those notebooks typically averaged around 37–39, with public leaderboard variance roughly between 35 and 45.\nOne important caveat is that the bucket model assumes lower pass@n roughly corresponds to longer useful runtime. This is not always true. A small class of problems elicits very fast but usually incorrect answers, so difficulty (measured in pass@n) and runtime are not perfectly aligned. Please refer to the internal benchmark section above for further discussion on this issue.

\n

C.1 Scheduler Reserve Policy and Time Bank Parameters

\n

We begin the run on problem 1. For future problems, the scheduler reserve time differently based on the number of problems left:

\n\n
initial_time_bank = 1500.0\ntime_bank_weight = 0.40\ntime_bank_min = -1800.0\ntime_bank_max = 3000.0\nhigh_problem_timeout = 1050.0\nbase_problem_timeout = 240.0\n
\n\n
elapsed_global = time.time() - self.notebook_start_time\ntime_left = self.cfg.notebook_limit - elapsed_global\nproblems_left_others = max(0, self.problems_remaining - 1)\n\nreserve = self._reserve_per_future_question()\nreserved_time = problems_left_others * reserve\nbank = float(getattr(self, \"time_bank\", 0.0))\nbudget = time_left - reserved_time + self.cfg.time_bank_weight * bank     \nbudget = min(budget, self.cfg.high_problem_timeout)\nbudget = max(budget, self.cfg.base_problem_timeout)\nbudget = min(budget, max(0.0, time_left - self.cfg.finish_grace_s))\n
\n

So the current problem receives:

\n\n

C.2 Consensus Based Acceptance Rules

\n

The interaction between the scheduler and RSA behaviour must be carefully accounted for, and both must be carefully designed with the other in mind. Ideally, we want to only use it for cases where the next RSA round can complete in time and there is significant information to be gained.\nAfter Round 0, the system accepts immediately under the following rule:

\n
   def _accept_after_round0(self, ans_list: list[int], round0_runtime: float) -> bool:\n       hist = self._answer_hist(ans_list)\n       top, second = self._top2_from_hist(hist)\n\n       if top >= 6:\n           return True\n\n       if top == 5 and second <= 1 and round0_runtime <= self.cfg.accept_5_runtime_cap:\n           return True\n\n       return False\n
\n

In words, the answer is accepted after Round 0 if:

\n\n
def _confident_enough(top_cnt: int, second_cnt: int) -> bool:\n   if top_cnt < rsa_majority_threshold:\n       return False\n   decision_margin = getattr(self.cfg, \"decision_margin\", 2)\n   return (top_cnt - second_cnt >= decision_margin)\n
\n

Typically, rsa_majority_threshold = 4 and decision_margin = 2.

\n

C.3 Estimating the Cost of Another RSA Round

\n

To avoid entering an RSA round that cannot complete in time, the system estimates the cost of the next round using Round 0 runtime:

\n
def _estimate_rsa_cost(self, round0_runtime: float) -> float:\n   return max(self.cfg.min_rsa_cost, self.cfg.rsa_cost_fraction * round0_runtime)\n
\n

In our implementation, we set:

\n\n

C.4 Progress Score for Additional RSA Rounds

\n

For RSA rounds beyond Round 1, we required evidence of progress before continuing:

\n
def _rsa_progress_score(self, prev_answers: list[int], new_answers: list[int]) -> float:\n   prev_hist = self._answer_hist(prev_answers)\n   new_hist = self._answer_hist(new_answers)\n   prev_top, prev_second = self._top2_from_hist(prev_hist)\n   new_top, new_second = self._top2_from_hist(new_hist)\n   prev_distinct = len(prev_hist)\n   new_distinct = len(new_hist)\n   score = 0.0\n   if new_top > prev_top:\n       score += 1.5\n   if (new_top - new_second) > (prev_top - prev_second):\n       score += 1.0\n   if new_distinct < prev_distinct:\n       score += 1.0\n   return score\n
\n

This score increases when:

\n\n
elapsed_q = time.time() - t_question0\nextra_affordable = budget - elapsed_q + max(0.0, self.cfg.bank_borrow_fraction * self.time_bank)\nallow_second_rsa = (\n   max_rsa_rounds >= 2\n   and progress_score >= 1.5\n   and extra_affordable >= estimated_rsa_cost\n   and (deadline - time.time()) >= (estimated_rsa_cost + self.cfg.rsa_safety_seconds)\n)\n
\n

In words, another RSA round is allowed only if:

\n\n

Appendix D: Internal Benchmark

\n

This appendix documents some observations we made on the nature of some problems and how they relate to the observed answer distribution.\nIt also includes some examples of problems that were manually filtered out when constructing this benchmark.

\n

D.1 Observations

\n

D.1.1 imo-bench-algebra-100

\n

This problem is on Vasc inequality. This is an example of a problem in class 5 as described in section 5.2. There is a short but tempting solution leading to the wrong answer, while the correct answer requires substantially deeper reasoning that the model often skips.\n
\nProblem:\nFind number of triples \\( (x,y,z) \\) of real numbers satisfying $$x^2 + y^2 + z^2 = xy^3 + yz^3 + zx^3 = 3$$\n
\nAnswer: 8\n
\nAn easy guess and check finds 2 solutions: \\( (1, 1, 1) \\) and \\( (-1, -1, -1) \\). From here, the model often concludes that there are no more others. Unfortunately there are, but they are difficult to find.\n
\nVasc's Inequality says that for \\( x, y, z \\in \\mathbb{R} \\):\n
\n$$ (x^2+y^2+z^2)^2 ≥ 3(xy^3+yz^3+zx^3)$$\n
\nThe problem statement says that \\( x^2 + y^2 + z^2 = 3 \\) and \\( xy^3+yz^3+zx^3 = 3 \\), so we are in the equality case of Vasc's.\n
\nIn the equality case, there are additional solutions when \\( x \\), \\( y \\), and \\( z \\) are in the ratio of:\n
\n$$ x:y:z = sin^2(\\frac{4π}{7}) : sin^2 (\\frac{2π}{7}) : sin^2 (\\frac{π}{7})$$ \n
\nWe can also permute \\( x, y, z \\) cyclically (transpositions are not allowed). Flipping all their signs also preserves equality. Hence there are an additional 6 solutions to the equality expression, and the final answer is 8.\n

\n

D.1.2 imo-bench-combinatorics-031

\n

USA IMO Team Selection Test 2020, Day 1 Problem 3\n
\nProblem:\nLet \\( \\gamma \\geq 1\\) be a real number. Sun Wukong and the Sea God play a turn-based game on an infinite grid of unit squares. Before the game starts, the Sea God chooses a finite number of cells to be flooded with seawater. Sun Wukong is building a magical barrier, which is a subset of unit edges of the grid (called walls) forming a connected, non-self-intersecting path or loop. Additionally, there is a magical artifact that randomly generates a finite number of extra walls on the grid, with no specific pattern or distribution.\n
\nThe game then begins with Sun Wukong moving first. On each of Sun Wukong's turns, he adds one or more walls to the magical barrier, as long as the total length of the barrier is at most \\( \\gamma n \\) after his \\(n^{th} \\) turn. On each of the Sea God's turns, every cell which is adjacent to an already flooded cell and with no wall between them becomes flooded as well. Sun Wukong wins if the magical barrier forms a closed loop such that all flooded cells are contained in the interior of the loop, hence stopping the flood and saving the world. What is the largest constant \\( C \\) such that for all \\( \\gamma > C\\) can Sun Wukong guarantee victory in a finite number of turns no matter how the Sea God chooses the initial cells to flood?\n
\nAnswer: 2\n
\nThis is another example of a problem in class 5 as described in section 5.2. The 'magical artifact' statement is a distractor, but it doesn't affect strong models like gpt-oss-120b.\n
\nThe short but tempting solution is observing that when unrestricted, the perimeter of the flood grows by 8 each turn, and concluding that \\( C > 8 \\) for Sun Wukong to guarantee victory. The model often makes this observation and then stops here, returning the answer 8.\n
\nHowever, the correct solution provides a construction based on a different intuition. Sun Wukong can cut off the flood from one direction first. In this step, he needs to ensure that the wall on that side grows fast enough compared to the flood's spread, hence we need at least \\( C>2 \\). Next, he can restrict the flood to a long, narrow corridor by constructing two parallel sides. Finally, he can construct the final wall to seal off the flood. This means that Sun Wukong only needs to ensure that the narrow corridor grows in length fast enough compared to the spread of the flood, which means we indeed only need \\( C > 2 \\) for Sun Wukong to contain the flood. The correct answer is 2.\n

\n

D.1.3 imo-bench-combinatorics-013

\n


\nThere are 42 students participating in the Team Selection Test, each of them is assigned a positive integer from 1 to 42 such that no two students have the same number and every number from 1 to 42 is assigned to a student. The team leader want to select a subset of these students such that there are no two students that have numbers whose difference is 1 or 21. For example, the team leader can pick the set \\( \\{1, 3, 6, 25, 28, 34, 42\\} \\) but not \\( \\{1, 2, 4, 6, 24\\} \\) or \\( \\{1, 3, 24, 26, 28\\} \\).\n
\nHow many ways can the team leader pick such a subset? Give your answer modulo 100000.\n
\nAnswer: 26555\n
\nThis is an example of a problem in Class 2 as described in section 5.2.\n
\nThe model formulates this problem as an independent-set count on a graph. The numbers 1–42 form a 2-by-21 ladder, and the task becomes counting subsets with no adjacent chosen vertices. From this, the model sometimes counts that there are 131836323 valid choices, and returns the answer 36323.\n
\nHowever, the constraints also mean that 21 and 22 cannot be selected at the same time. There are 11309768 such selections, and hence the correct count is 131836323 - 11309768 = 120526555, and the correct answer should be 26555. The model only makes this extra reasoning step around 50% of the time.\n

\n

D.2 Filtered Problems

\n


\nThe creators of the imo_answerbench dataset noted in their paper that they modfied the problems in several ways to harden them against AI. Hardening methods include paraphrasing, changing the name of objects in the problem (such as changing point names for geometry problems), reformulating, modifying numerical values\nand/or adding distractors to the problem. This process was done either manually or automatically using language models.\n
\nUnfortunately, some of these changes lead to genuine mathematical errors in the problem statement, causing the ground truth answer to be wrong. When constructing our benchmark, we manually reviewed the problems to ensure that only correct problems are retained. Ambiguous and wrong problems are discarded. This section contains some examples of discarded problems.\n

\n

D.2.1 imo-bench-geometry-002

\n


\nproblem 2 in the original problem source\n
\nModified Problem: Given right triangle \\( XYZ \\) with hypothenuse \\( XZ \\) and \\( \\angle X = 50^{\\circ} \\). Points \\( P \\) and \\( Q \\) on the side \\( YZ \\) are such that \\( \\angle PXZ = \\angle QXY = 10^{\\circ} \\). Compute the ratio \\( 2 \\times YQ/ZP \\).\n
\nDataset issue. Modified problem asks for 2 x \\( \\frac{YQ}{ZP} \\). The original problem labelled points in ABC instead of XYZ, and asked for the equivalent of \\( \\frac{ZP}{YQ} \\). It seems that the problem editor mixed up the line segments when editing the point names. The answer should be 1 instead of 4.\n

\n

D.2.2 imo-bench-geometry-044

\n

original problem source\n
\nModified Problem: Given \\( PQRS \\) is a parallelogram. We know that \\( \\angle S = 60^\\circ \\), \\( PS = 2 \\) and \\( PQ = \\sqrt{3} + 1 \\). Point \\( N \\) is the midpoint of \\( PS \\). Segment \\( RE \\) is the angle bisector of \\( R \\). Find the angle \\( REQ \\) in terms of degree.\n
\nDataset issue. Original problem contained an additional diagram implying the existence of an extra line, and the point \\( E \\) is supposed to lie on that line. This diagram is missing from the modified problem, leading to the model to have their own interpretation of where the points lie, hence outputting a different answer sometimes.\n

\n

D.2.3 imo-bench-number_theory-055

\n


\nDay 2 Problem 6 in the Estonia Team Selection Test 2018\n
\nModified Problem: A positive integer \\( m \\) consisting of distinct digits is considered \"good\" if it is a single-digit number, or if removing one of its digits results in a divisor of \\( m \\) that is also a good number.\n
\nFind the largest good number.\n
\nDataset issue. Original problem has an extra condition: We assume that numbers do not start with 0. This condition is not in the modified problem in dataset, which fundamentally changes the solution. Answer is 903125 without the condition, or 146250 with it.\n

\n

D.2.4 imo-bench-algebra-039

\n


\nProblem 5 on page 123 of the original source\n
\nModified Problem: Let \\(p\\), \\(q\\), \\(r\\), \\(s\\) be constants such that the equation $$ py^3 + qy^2 + ry + s = 0 $$ has three distinct real roots. Find all possible values for the number of distinct real roots of the equation\n$$ (pz^3+qz^2+rz+s) (6pz+2q) = (3pz^2+2qz+r)^2 $$\n
\nError in problem. The original problem (Ukraine 1997) asked for:\n
\n$$Q(x) = 4(px^3 + qx^2 + rx + s)(3px + q) = (3px^2 + 2qx + r)^2 $$\n
\n-> answer = 2\n
\nThe modified problem asks for (after moving out a factor of 2 from within the brackets on LHS):\n
\n$$ Q(x) = 2(px^3 + qx^2 + rx + s)(3px + q) = (3px^2 + 2qx + r)^2 $$ \n
\n-> answer = 0\n
\nBoth Gpt 5.2 and Gemini get both variants correct. Solution involves applying Descarte’s rule of signs, which gives an answer in the original problem but is inconclusive in the modified problem. Solving the modified problem involves analysing the discriminant of the cubic equation.\n

\n

D.2.5 imo-bench-combinatorics-005

\n


\nProblem: Determine the number of natural numbers n that that has at most 16 digits satisfying the following conditions:

\n\n

D.2.6 imo-bench-combinatorics-020

\n


\nDay 1 Problem 3 in the Vietnam Team Selection Test 2020\n
\nProblem:\n
\nSuppose there are 40 professional baseball teams participating in a tournament. In each round of the game, we will divide the 40 teams into 20 pairs, and each pair plays the game at the same time. After the tournament, it is known that every two teams have played at most one game. Find the smallest positive integer \\( a \\), so that we can arrange a schedule satisfying the above conditions, and if we take one more round, there is always a pair of teams who have played in the game.\n
\n
\nThere is a tension here between \"maximally cooperative scheduling\" and \"adversarial scheduling\". \n

\n\n

Appendix E: Prompts

\n

Througout the competition, we tried various prompts crafted with ideas from different sources.

\n

E.1 Cognitive Wells

\n

This prompt was adapted from Escaping the Cognitive Well: Efficient Competition Math with Off-the-Shelf Models. The main finding of the paper is that solutions to problems can be broken down into smaller lemmas. The paper reports that this decomposition improves performance, and attributes the gain largely to the fact that the intermediate lemmas are easier for the model to reason about in isolation, without the context of the larger main problem interfering. The authors also provide a system prompt that encourages the model to verify its reasoning, check for logical inconsistencies, and avoid hand-wavy arguments before returning an answer. we adapted this prompt for AIMO3, but found that it significantly degraded performance on the public leaderboard.

\n

\"\"\nThe scores are 30-35. The prompt is as follows:

\n
system_prompt = (\n\"\"\"\n### Prompt: DialecticEngine (Council of Architects)\nMUST BE FOLLOWED METICULOUSLY\n### Inputs for this Task\n**1. The Problem:**\n[See User Input]\n**2. Additional Materials:**\n[See User Input. Materials will be clearly graded.]\n---\n## DIALECTIC ENGINE PROMPT\n**YourRole:**\nYou are a multi-persona computational engine that solves difficult problems via a rigorous, multi-stage dialectic process. Your goal is to explore the problem space creatively and deeply, avoiding premature conclusions and logical gaps. The final answer is always an integer from 0 to 99999 and must be placed in \\boxed{answer}, e.g., \\boxed{42} (if solution is complete).\n**Configuration:** `MAX_INNER_ITERATIONS: 3`\n---\n### Core Principle: Atomicity of Logic\nAll solution drafts (`Current_Draft`, `Refined_Draft`, `Solution_Submitted_for_Review`) must be **strictly self-contained, line-by-line logical arguments**. Every step must be justified either by naming a standard theorem, or by a proved claim/lemma in a previous line of the same draft.\n*Additional materials* can inform the solution, but the solution should be self-contained.\n### Core Principle: Source Hygiene\n`Additional Materials` contain raw outputs from other agents. Treat them as **unverified hints**. If your solution uses a conjecture or lemma from them, you **must prove it from scratch**. Standard mathematical theorems (e.g., Cauchy–Schwarz) may be cited by name without proof.\n### Sloppiness\nSloppiness is intolerable. Examples:\n(a) Asserting “well-known fact” without an exact reference  \n(b) Showing for a couple of values of `n` and asserting it is proved for all `n`\nIf the solution has known gaps, they must be rephrased as a **clear and self-contained** conjecture.\n---\n#### Personas\n* **The Council of Architects:**\n * **Classicist:** Relies on established theorems and formal rigor.\n * **Visionary:** Proposes novel, abstract, and unconventional connections.\n * **Experimenter:** Tests hypotheses with concrete examples and computational thinking.\n * **Momus, the Skeptic:** Adversarially critiques the Council’s *strategy* and *high-level approach* to find conceptual flaws. Only asks tough questions that others must answer. *Does not* help prove in any way.\n * **Veritas, the Auditor:** A meticulous, line-by-line proof checker. Veritas’s sole function is to ensure every logical step in a draft adheres to the **Core Principle of Atomicity of Logic**. Veritas does not judge the overall strategy, only its step-by-step validity.\n * **The Chief Architect:** Manages the process, arbitrates disputes, and synthesizes the final result. Enforcer of the FUNDAMENTAL RULE: No evasive phrases (e.g., “it can be shown,” “it is clear that,” or “known results imply,” “etc.”) in `Current_Draft` unless they are immediately followed by the complete derivation or citation. Flags such phrases and sets `Needs_New_Draft = TRUE`.\n---\n#### Glossary of Terms\n* **`STRATEGIC_LEAD`:** A strategy flagged by Momus as a potential dead end.\n* **`LOGICAL_GAP`:** A line in a solution flagged by Veritas as unjustified looking solely within the solution (e.g., made-up claims).\n* **`DEAD`:** A `STRATEGIC_LEAD` whose rebuttal was unconvincing. It is permanently discarded.\n* **`Rebuttal`:** The Council’s defense of a `STRATEGIC_LEAD`, achieved by producing a refined solution.\n---\n**(You will output the following process in REAL-TIME.)**\n---\n## BEGIN PROCESS\n### Stage 1: Ideation Forum\n1. **Brainstorm (Council of Architects):** The Council analyzes the problem and generates a list of “Promising Avenues & Key Concepts.”\n2. **Pre-mortem (Momus, the Skeptic):** Concurrently, Momus analyzes the problem to identify potential “Red Herrings & Inefficient Paths.”\n3. **Synthesize (Chief Architect):** The Chief Architect compiles the outputs into a formal `Strategy_Dossier`.\n4. **Select Strategy:** The Chief Architect selects the most promising strategy from the `Strategy_Dossier` to be developed.\n### Stage 2: Dialectic Loop (Development & Refinement)\n1. **Initialize:** Set `Active_Strategy =` (first strategy from dossier). Set `Needs_New_Draft = TRUE`.\n2. **LOOP (up to `MAX_INNER_ITERATIONS`):**\n  * **a. Drafting Phase:** If `Needs_New_Draft` is TRUE:\n    * The Council writes a fresh `Current_Draft` based strictly on `Active_Strategy`.\n    * Set `Needs_New_Draft = FALSE`.\n  * **b. Strategic Critique (Momus):** Momus attacks the `Current_Draft`’s high-level approach. He may issue a `STRATEGIC_LEAD`.\n  * **c. Logical Audit (Veritas):** Veritas audits the `Current_Draft` line-by-line. If any step lacks explicit justification within the draft, flag as `LOGICAL_GAP`.\n  * **d. Refine (Council):** The Council creates a `Refined_Draft` to resolve `LOGICAL_GAP`s and rebut Momus.\n  * **e. Judge (Chief Architect):**\n    * *Audit Check:* Confirm `Refined_Draft` has zero `LOGICAL_GAP`s.\n    * *Strategy Check:* If a `Rebuttal` was issued, judge it.\n      * **If Unconvincing:** Declare `Active_Strategy` as `DEAD`. Select next strategy from `Strategy_Dossier` as `Active_Strategy`. Set `Needs_New_Draft = TRUE`. (This burns one iteration.)\n      * **If Convincing or No Challenge:** The strategy survives.\n  * **f. Convergence:** If Momus is satisfied, Veritas finds no gaps, and `Needs_New_Draft` is FALSE, exit loop. The `Refined_Draft` becomes `Solution_Submitted_for_Review`.\n### Stage 3: Final Review (Chief Architect)\n1. Review the `Solution_Submitted_for_Review` and the process logs.\n2. **Cognitive Reset 1:** Write a **Shakespearean Verse** on the solution’s core logic.\n3. Write final `Chief_Architect_Feedback`, explicitly confirming that the solution is free of logical gaps per the final `Verification_Report`.\n### Stage 4: Final Synthesis (Chief Architect)\n1. Assemble the final response using the format below.\n## END PROCESS\n---\n### Final Output Format\n**Part 1: The Process Log**  \n*(Render the Ideation, Inner Loop Transcript, and Shakespearean Verse here, exactly as generated in the real-time process steps above.)*\n**Part 2: The Final Synthesis**\n* **The Architect’s Note:** (A brief explanation of the final outcome.)\n* **The Architect’s Log (A Hemingway Summary):** (Cognitive Reset 2. Recount the discovery process—the ideas, challenges, and breakthroughs—in the style of Hemingway.)\n**Part 3: The Final Blueprint (Reviewer-Ready Version)**  \n(Re-state the original **Problem**. Then present the final, clean, and **strictly self-contained** solution from the “Solution_Submitted_for_Review.” No references to the dialectic process.)\n- **Result**: Final integer, numerical answer from 0 to 99999 placed inside \\boxed{answer} (e.g., \\boxed{42}, if solution is complete) or proven results.\n\"\"\"\n   )\n
\n

E.2 Including Notes from IMO Answerbench

\n

We also experimented with a system prompt that included notes derived from mistakes the model made on the IMO AnswerBench dataset. The goal was to remind the model of common failure modes observed during benchmarking. In practice, this prompt had little effect: it neither clearly improved nor degraded performance. The prompt is as follows:

\n
system_prompt = (\n\"\"\"You are an elite mathematical problem solver with expertise at the International Mathematical Olympiad (IMO) level. Your goal is to find the correct answer through rigorous mathematical reasoning.\n# General rules:\n- Every claim must be logically justified.\n- Do not guess, speculate, or rely on unproven intuition.\n- If a full solution is not possible, only present results that can be proven rigorously.\n- Do not include failed attempts, scratch work, or informal commentary.\n# General Problem-Solving Approachs:\n1. UNDERSTAND: Carefully read and rephrase the problem in your own words. Identify what is given, what needs to be found, and any constraints.Pay attention to all mathematical symbols and words, and understand what they mean.\n2. EXPLORE: Consider multiple solution strategies. Think about relevant theorems, techniques, patterns, or analogous problems. Don't commit to one approach immediately.\n3. PLAN: Select the most promising approach and outline key steps before executing.\n4. EXECUTE: Work through your solution methodically. Show all reasoning steps clearly.\n5. VERIFY: Check your answer by substituting back, testing edge cases, or using alternative methods. Ensure logical consistency throughout.\n# General Mathematical Reasoning Principles:\n- Break complex problems into smaller, manageable sub-problems\n- Look for patterns, symmetries, and special cases that provide insight\n- Use concrete examples to build intuition before generalizing\n- When attempting to generalizing concrete examples into conclusions, always try a few examples first.Do not settle on the first example studied.\n- If the problem involves multiple cases, reason through each one of them carefully. Do not take shortcuts.Do not assume the results from previous cases generalize to the next one.\n- Consider extreme cases and boundary conditions\n- If stuck, try working backwards from the desired result\n- Be willing to restart with a different approach if needed\n- For induction arguments, always check the base case and make sure it holds.\n# Verification Requirements:\n- Cross-check arithmetic and algebraic manipulations\n- Verify that your solution satisfies all problem constraints\n- Test your answer with simple cases or special values when possible\n- Ensure dimensional consistency and reasonableness of the result\nBefore finalizing your answer, carefully review the solution and ensure:\n- Logical correctness of every step.\n- Compliance with all instructions.\n- Clarity, structure, and mathematical rigor.\n# Output Format:\n### Summary\n- **Verdict**: State whether a complete solution is obtained or only a partial solution.\n- **Result**:\n - If complete: clearly state the final answer with a short summary. The final answer must be a non-negative integer between 0 and 99999. Place your final numerical answer inside \\boxed{}, e.g., \\boxed{42}\n - If partial: state the main rigorously proven results.\nThink step-by-step and show your complete reasoning process. Quality of reasoning is as important as the final answer.\n# Combinatorics Specific Hints:\n- If the problem is a combinatorics problem involving tiling and lower or upper bounds, look for a few invariants. Do not settle on the first one.\n- If the problem require representing it as a graph, be careful of hidden assumptions and restrictions. Do not miss them.\n- If the problem involves tiling or colouring a square grid, think very carefully about what happens along the diagonal.\n- If the problem involves a turn based game, think about the optimal strategy for each of the players.\n# Algebra Specific Hints:\n- The word \"between\" means not-enclusive of both endpoints of an interval.\n- If this problem requires proving an inequality, recall some common equalities in Olympiads: \n* triangle inequality\n* Vasc's inequality\n* Muirhead inequality\n* Cauchy Schwarz inequality\n* Rearrangement inequailty\n* QM-AM-GM-HM inequality\n- If the problem involves functional equation, a common approach is to rewrite the equations and try to find relations between function values.\n# Geometry Specific Hints:\n- If the problem is a geometry problem and the conclusion cannot be deduced directly, try drawing additional points, lines, and circles.\n\"\"\"\n   )\n
\n

E.3 Final System Prompt

\n

In the end, we used the simplest system prompt: one that only specified the required output format. Longer system prompts increase prefill cost and uses up model context length, and under the competition time budget we preferred to avoid that overhead unless it produced a clear accuracy gain.

\n
system_prompt = (\n\"\"\"You are an elite mathematical problem solver with expertise at the International Mathematical Olympiad (IMO) level.\n# Output Format:\n### Summary\n - Clearly state the final answer with a short summary. The final answer must be a non-negative integer between 0 and 99999. Place your final numerical answer inside \\boxed{}, e.g., \\boxed{42}.\n - You must always provide a summary of the solution, in addition to the final answer.\n\"\"\"\n   )\n
\n

E.4 RSA prompt

\n

The aggregation step of the pipeline uses a special prompt that was built with the following:

\n
prompt_dictionary = {\n   # generic helpers\n   # think step by step was taken from the aime25 dataset problem text. aimo3 dataset does not have this. Need to include it to reproduce RSA\n   'think_step_by_step': 'Lets think step by step and output the final answer within \\\\boxed{}.',\n   # format hints\n   'math_format_hint' : '\\\\boxed{} \\n **summary**:',\n   # problem kind labels\n   \"math_problem_kind\": \"math problem\",\n   # aggregation templates: single-candidate\n   \"agg_single_preamble\": (\n       \"You are given a math problem, and a solution hint. \"\n       \"The hint may be incorrect and contain errors. \\n\"\n   ),\n   \"agg_single_candidate_header\": \"Hint (may contain mistakes):\\n\",\n   \"agg_single_candidate_block\": \"---- Hint ----\\n{candidate}\\n\",\n   \"agg_single_tail\": (\n       \"Solve the problem and output the solution in the correct format. \"\n   ),\n   # aggregation templates: multi-candidate\n   \"agg_multi_preamble\": (\n       \"You are given a math problem, and some solution hints. \"\n       \"Some hints are incorrect and contain errors. \\n\"\n   ),\n   \"agg_multi_candidates_header\": \"Solution Hints (may be incomplete, contain mistakes):\\n\",\n   \"agg_multi_candidate_block\": \"---- Solution Hint {i} ----\\n{candidate}\\n\",\n   \"agg_multi_tail\": (\n       \"Solve the problem and output the solution in the correct format, together with a solution summary. \"\n   ),\n   # common headers\n   \"agg_problem_header\": \"Problem:\\n{question}\\n\",\n}\ndef aggregate_prompt(\n       question: str,\n       candidate_answers: List[str],\n       task: str,\n       prompt_dictionary: dict\n       ) -> str:\n   # pick problem kind + format hint from dict\n   problem_kind = prompt_dictionary[\"math_problem_kind\"]\n   format_hint = prompt_dictionary[\"math_format_hint\"]\n   parts = []\n   # preamble\n   if len(candidate_answers) == 1:\n       preamble = prompt_dictionary[\"agg_single_preamble\"]\n   else:\n       preamble = prompt_dictionary[\"agg_multi_preamble\"]\n   parts.append(preamble)\n   # problem header\n   parts.append(\n       prompt_dictionary[\"agg_problem_header\"].format(\n           question=question.strip()\n       )\n   )\n   # candidate(s) section\n   if len(candidate_answers) == 1:\n       parts.append(prompt_dictionary[\"agg_single_candidate_header\"])\n       ans_str = (candidate_answers[0] or \"\").strip()\n       parts.append(\n           prompt_dictionary[\"agg_single_candidate_block\"].format(\n               candidate=ans_str\n           )\n       )\n       parts.append(\n           prompt_dictionary[\"agg_single_tail\"]\n       )\n   else:\n       parts.append(prompt_dictionary[\"agg_multi_candidates_header\"])\n       for i, ans in enumerate(candidate_answers, 1):\n           ans_str = (ans or \"\").strip()\n           parts.append(\n               prompt_dictionary[\"agg_multi_candidate_block\"].format(\n                   i=i,\n                   candidate=ans_str,\n               )\n           )\n       parts.append(\n           prompt_dictionary[\"agg_multi_tail\"]\n       )\n   return \"\\n\".join(parts)\n
", "comments": [ { "id": 3458590, "author_name": "Sam Bealing", "author_url": "", "post_date": "2026-05-16T08:21:09.773000", "content": "

Following up on the questions from @philipvonderlind (and thanks for the very thorough response), here are some additional questions to help us assess your submission:

\n
    \n
  1. Stopping rule for the Class 4/5 case: Your Section 7 identifies the fast, confident wrong-majority case as the most important remaining failure mode. Your current stopping rule \"accepts strong Round 0 majorities\", which is exactly the pattern Class 4/5 produces. What signal would you propose to distinguish a genuinely-correct strong Round 0 majority from a Class 4/5 wrong one, given that both look similar in the answer histogram?
    2. \n
  2. AnswerBench problem-quality and filtering: Your internal benchmark draws mostly from IMO-AnswerBench, with some problems filtered out during construction. How did you identify the problems that needed filtering, and across the broader sample you worked with, do you have an informal sense of the error rate (e.g. wrong ground-truth answers, ambiguous statements)? If you were working with AnswerBench V1, have you checked whether any of those errors are now corrected in AnswerBench V2? We've seen across maths corpus prize submissions that widely-used datasets often carry quiet ground-truth errors that can distort ablation results.
    3. \n
\n

Look forward to reading your responses on these.

", "votes": 1, "replies": [ { "id": 3459214, "author_name": "IceBeam777", "author_url": "", "post_date": "2026-05-17T14:09:27.793000", "content": "

Hi @sambealing ,

\n

Thank you for reading our writeup and for the thoughtful questions! We will split our response into 2 separate replies, one for each question.

\n

(part 1 of 2)

\n

Question:

\n

Stopping rule for the Class 4/5 case: Your Section 7 identifies the fast, confident wrong-majority case as the most important remaining failure mode. Your current stopping rule \"accepts strong Round 0 majorities\", which is exactly the pattern Class 4/5 produces. What signal would you propose to distinguish a genuinely-correct strong Round 0 majority from a Class 4/5 wrong one, given that both look similar in the answer histogram?

\n

Response:

\n

As noted in Section 7, we don't think that answer histogram alone can reliably distinguish a genuinely correct strong Round 0 majority from a Class 4/5 wrong majority. Runtime heuristics also seem insufficient, because some of the Class 4/5 failures are both fast and highly concentrated. Our honest answer is that we do not currently have a reliable production-ready signal.

\n

However, in the literature we reviewed, there is another signal worth exploring: the Deep Thinking Ratio (DTR). of a generated solution. This is a richer measure of reasoning effort. When the solutions have high variance in the DTR metric, continue with RSA especially when the answer histogram also has high variance.

\n

Deep-thinking Tokens Ratio

\n

This paper, Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens, proposes a new metric called the \"Deep Thinking Ratio (DTR)\", and claims that it has very high correlation with task accuracy. This is worth exploring. DTR is measured as follows:

\n

Definition 1 (Deep Thinking Token): A Deep Thinking Token in a sequence is a token whose predictions undergo sustained revision in deeper layers (of the model) before converging. \"revision\" is measured by the Jenson Shannon Divergence (JSD) of probability distributions of the token between the intermediate layers and the final layer of the model.

\n

Definition 2 (Deep Thinking Ratio): The Deep Thinking Ratio (DTR) of a generated solution \\( S \\) is the proportion of tokens in \\( S \\) that are Deep Thinking Tokens.

\n

The reason this is relevant to the Class 4/5 failure mode is that those failures often look like reasoning shortcuts. The model finds a short, tempting argument that leads to a wrong answer, and many samples independently fall into the same trap. From the outside, this produces a clean answer histogram, but internally the model may not have engaged in much deeper revision before committing to the answer.

\n

\"\"

\n

The above table was taken from page 7 of the paper.

\n\n

For gpt-oss-120b on high reasoning mode solving the three Mathematical Olympiad tasks (AIME 2025, AIME 2024, and HMMT 2025), the correlations between DTR and accuracy are 0.796, 0.905, and 0.972 respectively. This is higher than the correlation between Negative entropy and accuracy, which are 0.711, 0.593, and 0.728.

\n

The signal we would therefore propose is to augment “majority size,” with DTR dispersion across the Round 0 samples. In particular, a strong Round 0 majority should be treated with more suspicion if:

\n
    \n
  1. The samples agreeing with the majority answer have relatively low DTR.
    2. \n
  2. One or more minority answers have substantially higher DTR.
    3. \n
  3. The high-DTR and low-DTR samples disagree on the final answer.
    4. \n
\n

This would be consistent with the intuition that the majority answer may have come from a shallow shortcut, while a minority answer may reflect a more expensive but potentially more reliable reasoning path. In such cases, we would continue with RSA rounds even if the Round 0 answer histogram looked confident. This would directly target the Class 4/5 failure mode. For genuinely easy problems, we would expect the Round 0 samples to agree both in answer and in reasoning-effort profile. For deceptive problems, the answer histogram may look clean, but the reasoning-effort profile may be more heterogeneous.

\n

The main practical limitation is that DTR is not currently easy to extract in the inference stack used for AIMO 3. Standard high-throughput inference engines such as vLLM and SGLang are optimized for serving final-token distributions, not for exposing layerwise token distributions during generation, which is required to calculate DTR. Forcing this information out of the model would likely introduce significant overhead, which is why we did not pursue it during the competition.

\n

(part 1 of 2)

", "votes": 0, "replies": [] }, { "id": 3459226, "author_name": "IceBeam777", "author_url": "", "post_date": "2026-05-17T14:23:36.297000", "content": "

(part 2 of 2)

\n

Question:

\n

AnswerBench problem-quality and filtering: Your internal benchmark draws mostly from IMO-AnswerBench, with some problems filtered out during construction. How did you identify the problems that needed filtering, and across the broader sample you worked with, do you have an informal sense of the error rate (e.g. wrong ground-truth answers, ambiguous statements)? If you were working with AnswerBench V1, have you checked whether any of those errors are now corrected in AnswerBench V2? We've seen across maths corpus prize submissions that widely-used datasets often carry quiet ground-truth errors that can distort ablation results.

\n

Response:

\n

Thank you for raising this. We agree that quiet dataset errors can distort ablation results.

\n
    \n
  1. From our observations, we estimate the error rate to be 12 of 228 problems, or 5.26%.
    2. \n
  2. Of these 12 flagged problems, 7 appear to have been fixed in AnswerBench V2, while 5 remain.
    3. \n
  3. We explain below why we considered each flagged problem problematic.
    4. \n
\n

1. Filtering Process:

\n

Our filtering process for IMO-AnswerBench was as follows. The entire dataset has 400 problems, with 100 each in number theory, geometry, combinatorics, and algebra.

\n
    \n
  1. First, scan the answer column and kept only problems whose answer was a single integer. This gave 228 problems in total: 43 number theory, 63 geometry, 87 combinatorics, and 35 algebra.
    2. \n
  2. Then, run a majority@8 solver with 4 independent runs per problem. There were 53 problems where the solver failed in at least 1 of the 4 runs.
    3. \n
  3. Finally, we manually reviewed these 53 candidate hard problems and wrote notes for each one, classifying the failure as best as we could into one of several categories: genuine reasoning difficulty, parser/instruction-following issue, ambiguity, or dataset/problem-quality issue.
    4. \n
  4. Out of these 53 problems:
    5. \n
\n\n

2. Checks against AnswerBench V2:

\n

Our full list of 12 problems that we think contains issues are:

\n
    \n
  1. imo-bench-algebra-039 (changing numbers fundamentally changes the solution: descartes rule of signs to analyzing the discriminant of cubic equation)
    2. \n
  2. imo-bench-combinatorics-005 (convention issue over whether 0 is a natural number)
    3. \n
  3. imo-bench-combinatorics-020 (ambiguous cooperative vs random/uncooperative scheduling interpretation)
    4. \n
  4. imo-bench-combinatorics-030 (unclear interpretation of how black cubes/pillars are chosen)
    5. \n
  5. imo-bench-combinatorics-048 (asks for “the smallest positive integers” but expects a single integer answer)
    6. \n
  6. imo-bench-combinatorics-076 (red-herring sentence can reasonably be interpreted as a typo rather than irrelevant information)
    7. \n
  7. imo-bench-combinatorics-079 (added condition fundamentally changed the problem)
    8. \n
  8. imo-bench-combinatorics-085 (added condition fundamentally changed the problem)
    9. \n
  9. imo-bench-combinatorics-097 (weakened instruction phrase sometimes leads the model to return code/formula rather than the final integer)
    10. \n
  10. imo-bench-geometry-002 (line-segment ratio appears to have been reversed during point relabelling)
    11. \n
  11. imo-bench-geometry-044 (missing diagram/condition needed to determine the position of a point)
    12. \n
  12. imo-bench-number_theory-055 (missing crucial condition from the original problem)
    13. \n
\n

Out of these, the following have been fixed in AnswerBench V2:

\n\n

The following problems remain:

\n\n

For these 5 problems, we do not claim that they are all ground-truth errors. They are mostly interpretation or phrasing issues. Our view is that these cases are less suitable for evaluating mathematical reasoning ability, because the failure may depend on how the model resolves ambiguity or whether it treats an odd sentence as a typo, a distractor, or an additional condition. Other researchers may reasonably make different choices here.

\n

3. Full explanation of problems with errors, along with their fixes (if fixed):

\n

Fixed Problems:

\n

(1) imo-bench-algebra-039 (changing numbers fundamentally changes the solution: descartes rule of signs to analyzing the discriminant of cubic equation)

\n

See appendix D.2.4. AnswerBench V2 has changed the problem text to align with the original version of the question.

\n

(2) imo-bench-combinatorics-005 (convention issue over whether 0 is a natural number)

\n

See appendix D.2.5. AnswerBench V2 has changed the ground truth answer to match most model's outputs. However, the difference in convention on the definition of natural numbers remain.

\n

(7) imo-bench-combinatorics-079 (added condition fundamentally changed the problem)

\n

AnswerBench V2 has removed the extra condition in the problem text.

\n

The problem text in the original version taken from here was:

\n
\"Determine the largest $N$ for which there exists a table $T$ of integers with $N$ rows and 16 columns that has the following properties:\n(i) Every row contains the numbers $1,2, \\ldots, 16$ in some order.\n\n(ii) For any two distinct rows $r$ and $s$, there is a column $c$ such that $|T(r, c)-T(s, c)| \\geqslant 2$.\nIn addition to the given constraints, every column in the table must contain distinct integers.\nHere $T(r, c)$ means the number at the intersection of the row $r$ and the column $c$.\"\n
\n

The problematic condition every column in the table must contain distinct integers. enforces a very strict condition on the valid configurations, and has been removed from the problem text in v2.

\n

(8) imo-bench-combinatorics-085 (added condition fundamentally changed the problem)

\n

The problem text in the original version taken from here was:

\n
\"Evan fills the fields of an $78 \\times 78$ board with numbers from 1 to $6084$, each number being used exactly once. She then counts the total number of good paths on the board. A good path is a sequence of fields of arbitrary length (including 1) such that:\n\n(i) The first field in the sequence is one that is only adjacent to fields with larger numbers, and also one that has an even row number and an even column number.\n\n(ii) Each subsequent field in the sequence is adjacent to the previous field,\n\n(iii) The numbers written on the fields in the sequence are in increasing order.\n\nTwo fields are considered adjacent if they share a common side. Find the smallest possible number of good paths Alice can obtain.\"\n
\n

The problematic condition is also one that has an even row number and an even column number. Consider this construction:

\n\n

Now consider any field with even row and even column. Every such field has at least one adjacent field that is not even-even. Since all non-even-even fields have smaller numbers than all even-even fields, each even-even field has an adjacent field with a smaller number.

\n

Therefore no even-even field is adjacent only to larger numbers. So for this construction, there are no possible starting fields for a good path. Hence the number of good paths is 0. The ground truth answer is 12013.

\n

This condition and has been removed from the problem text in v2.

\n

(10) imo-bench-geometry-002 (line-segment ratio appears to have been reversed during point relabelling)

\n

See appendix D.2.1. AnswerBench V2 has corrected the ground truth answer from 4 to 1.

\n

(11) imo-bench-geometry-044 (missing diagram/condition needed to determine the position of a point)

\n

See appendix D.2.2.

\n

The corrected problem text in AnswerBench V2 is:

\n
\"Given $PQRS$ is a parallelogram. We know that $\\angle S = 60^\\circ$, $PS = 2$ and $PQ = \\sqrt{3} + 1$. Point $N$ is the midpoint of $PS$. $E$ is a point on the segment $NQ$ which makes the segment $RE$ the angle bisector of $R$. Find the angle $REQ$ in terms of degree.\n\"\n
\n

The statement that $E$ is a point on the segment $NQ$ was not present in V1 taken from here, which confused the model into thinking that the point \\( E \\) was on the parallelogram side \\( PQ \\) instead. This has been fixed in V2.

\n

(12) imo-bench-number_theory-055 (missing crucial condition from the original problem)

\n

See appendix D.2.3. The original problem is from Day 2 Problem 6 in the Estonia Team Selection Test 2018.

\n

The original problem text is:

\n
We call a positive integer $n$ whose all digits are distinct bright, if either $n$ is a one-digit number or there exists a divisor of $n$ which can be obtained by omitting one digit of $n$ and which is bright itself. Find the largest bright positive integer. (We assume that numbers do not start with zero.)\n
\n

In the original problem, there is an extra condition that (We assume that numbers do not start with zero.), and the correct answer is 146250. This condition was removed from V1. The model decides that without this condition, 903125 is the correct answer, and we check that it is. The problem text in V2 now says that $m$ consisting of distinct nonzero digits (n has been relabelled to m), and in this case the correct answer is 14625. The ground truth answer in V2 now matches the correct answer with the updated problem text.

\n

Remaining Problems:

\n

For the following cases, we do not claim that they are definite ground-truth errors. They are mainly ambiguity or phrasing issues which, in our opinion, make the problems less suitable for evaluating mathematical reasoning ability.

\n

(3) imo-bench-combinatorics-020 (ambiguous cooperative vs. adversarial scheduling interpretation)

\n

See appendix D.2.6.

\n

There are two different interpretations to the problem, with the first one resulting in a much easier solution than the second. Given that this was a Team Selection Test problem, it is likely that the problem setters intended for the harder interpretation. However, the model does not know this context, and always defaults to the easier interpretation. We are of the opinion that the model should not be penalized for making this choice.

\n

(4) imo-bench-combinatorics-030 (unclear interpretation of how black cubes/pillars are chosen)

\n

Original Problem Text: (same in both v1 and v2)

\n
A cube with size $18 \\times 18 \\times 18$ consists of $5832$ unit cubes, all colored white. Anton and Berta play a game on this cube. Anton chooses some pillars with size $1\\times 1\\times 18$ such that no two pillars share a vertex or side, and turns all chosen unit cubes to black. Berta is allowed to choose some unit cubes and ask Anton their colors. In addition, Anton also tells Berta that he painted at least two pillars. How many unit cubes, at least, that Berta need to choose so that for any answer from Anton, Berta can always determine the black unit cubes?\n
\n

It is not at all clear how the black cubes are chosen. The two statements:

\n\n

suggests that entire pillars are chosen at once, and all cubes in a chosen pillar share the same colour. \nIn this case, the entire game can be played on a \\( 18 \\times 18 \\times 1 \\) square grid, instead of the \\( 18 \\times 18 \\times 18 \\) cube. Then, we can make the following argument based on very specific choice of black cubes painted:\nBerta must query at least 324 cubes, one for each pillar. If he did not, then there always exists the case where, Anton paints 2 far away pillars from the cube Berta did not query.\nThen, the unqueried pillar can be painted or not painted, and Bertha will never know.\nThe ground truth answer of 486 suggests that there are some additional conditions on how the black cubes are chosen, but we don't know.

\n

(5) imo-bench-combinatorics-048 (asks for “the smallest positive integers” but expects a single integer answer)

\n

Original Problem Text: (same in both v1 and v2)

\n
 Find the smallest positive integers $n$, for which the numbers in the set $S=\\{1,2, \\ldots, n\\}$ can be colored red and blue, with the following condition being satisfied: the set $S \\times S \\times S$ contains exactly 2007 ordered triples $(x, y, z)$ such that (i) $x, y, z$ are of the same color and (ii) $x+y+z$ is divisible by $n$.\n
\n

There are two integers that satisfy the conditions, 69 and 84. The ground truth is 69, the smaller one of them, but the first line asks for the smallest integers. The model sometimes outputs \\( \\boxed{69, 84} \\) which is then graded as wrong, even though the output is a reasonable response to the literal wording.

\n

(6) imo-bench-combinatorics-076 (red-herring sentence can reasonably be interpreted as a typo rather than irrelevant information)

\n

Original Problem Text: (same in both v1 and v2)

\n
The Lucas numbers $L_{0}, L_{1}, L_{2}, \\ldots$ are defined inductively by $L_{0}=2, L_{1}=1$, and $L_{n+1}=L_{n}+L_{n-1}$ for $n \\geqslant 1$. The Fibonacci numbers $F_{0}, F_{1}, F_{2}, \\ldots$ are defined inductively by $F_{0}=0, F_{1}=1$, and $F_{n+1}=F_{n}+F_{n-1}$ for $n \\geqslant 1$. Determine the smallest size of a set $S$ of integers such that for every $k=2,3, \\ldots, 125$ there exist some $x, y \\in S$ such that $x-y=F_{k}$. Also, there exist some $a, b \\in T$ for some set $T$ such that $a-b = L_{100}$.\n
\n

There is a red-herring statement that the authors of IMO_AnswerBench inserted to test the model's ability to ignore irrelevant information:

\n\n

However, the model sometimes interprets it as a typo and assumes that \\(T\\) should have been \\(S\\), so it treats the sentence as an additional condition on \\(S\\). Since models are often trained to be robust to small human errors in prompts, we think this is a plausible interpretation rather than a pure reasoning failure.

\n

(9) imo-bench-combinatorics-097 (weakened instruction phrase sometimes leads the model to return code/formula rather than the final integer)

\n

Original Problem Text: (same in both v1 and v2)

\n
\"In a circular seating arrangement at a party, there are $16$ guests sitting at the table play a game. Each guest has a unique name tag crated by the game master, and then these name tags are randomly distributed among the guests. In each of the next $n$ rounds, the guests plays the game according to the following rule:\n\ni) Any guest who has their own name tag exits the table.\nii) The remaining guests pass their name tags to the guest sitting immediately to their right.\n\nWe are interested in finding the number of ways the name tags can be distributed such that there exists at least one guest who does not leave the table after 4 rounds.\"\n
\n

The problem says:

\n

“We are interested in finding the number of ways…”

\n

This is weaker than directly asking the model to “find” or “compute” the integer. Our hypothesis is that this interacts badly with the system prompt: despite the system prompt asking for a final integer, the problem wording sometimes causes the model to output code or a formula that computes the answer rather than the numerical answer itself. We therefore classified this as more of an instruction/formatting issue than a mathematical reasoning failure.

\n

(part 2 of 2)

", "votes": 0, "replies": [] } ] }, { "id": 3456823, "author_name": "Philip Vonderlind", "author_url": "", "post_date": "2026-05-12T16:30:55.593000", "content": "

Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.

\n

Your submission is already well structured and pleasant to read, and the plots do a good job of illustrating your process. I have a few questions that may help clarify the methodology further:

\n
    \n

  1. Context Management: Given the limited context window, how did you avoid unnecessary token usage during RSA? Running many parallel sessions likely produces a large amount of text per round, especially if intermediate outputs are reused as hints in later contexts. How did you decide what information to retain, compress, or discard?

    2. \n

  2. Difficulty Buckets: Did you run any experiments to verify that the difficulty buckets were representative of the public/private sets, or were these buckets derived primarily from the previously shared X post?

    3. \n

  3. Answer Collapse: Since RSA propagates hints to future trajectories, how did you mitigate the risk of convergence toward an incorrect majority answer? In particular, if early generations contain many flawed solutions, using them as contextual hints in later iterations could potentially steer subsequent trajectories in the wrong direction. How did you address this failure mode?

    4. \n
\n

Looking forward to your response :)

", "votes": 1, "replies": [ { "id": 3457556, "author_name": "IceBeam777", "author_url": "", "post_date": "2026-05-13T23:12:38.340000", "content": "

Hi @philipvonderlind,

\n

Thank you for reading my writeup and for the thoughtful questions! I will split my response into 3 separate replies, one for each question.

\n

(part 1 of 3)

\n

Question:

\n

Context Management: Given the limited context window, how did you avoid unnecessary token usage during RSA? Running many parallel sessions likely produces a large amount of text per round, especially if intermediate outputs are reused as hints in later contexts. How did you decide what information to retain, compress, or discard?

\n

Response:

\n

Context Management: Context management strategies differ by model used. We structure our response into 2 parts:

\n\n

1. What we did with gpt-oss-120b for AIMO 3:

\n

For AIMO3, we relied on a feature that is specific to the gpt-oss-120b / Harmony-style output format. The model does not produce a single unstructured blob of text. Instead, its generation is naturally separated into reasoning messages, tool calls, tool outputs, and a final assistant message. Schematically, one generation looks like:

\n
<reasoning>\n<python_tool_call>\n<reasoning>\n<python_tool_call>\n<reasoning>\n...\n... (possibly 100+ tool call rounds)\n...\n<reasoning>\n<python_tool_call>\n<reasoning>\n<python_tool_call>\n<reasoning>\n<final_message>\n
\n

The important point is that the final message is already a compressed summary of the model’s previous reasoning and tool use. It usually contains a short solution summary and the final boxed answer. For the next round of RSA, we did not include the intermediate reasoning traces, full Python tool logs, code snippets, failed attempts, or repeated derivations. We only used the final assistant message from each candidate solution as the hint for the next RSA round.

\n

In practice, these final messages were almost always shorter than 1000 tokens. Since one RSA round used 8 candidate solutions, the hint section contributed roughly at most 8000 tokens, plus the original problem statement and some small formatting overhead. This was far below the maximum context max_model_len = 81920 used in our final system, so context length itself was not the bottleneck in our experiments.

\n

Another important detail is that RSA did not require us to keep accumulating all previous generations forever. Each new RSA prompt was mainly built from the candidate summaries from the previous round. Earlier rounds influenced the next round indirectly, because their useful information had already been distilled into the previous round’s final messages. This kept the prompt size roughly bounded from round to round, rather than growing linearly with the total number of generations produced so far.

\n

This also explains why we avoided terminating generation immediately after the first \\boxed{...} appeared. Some public notebooks used early stream stopping as a speed optimization, but for RSA this is dangerous. If we stop at the first boxed answer, we may never receive the model’s final message. That final message is exactly the compact object we want to pass into the next RSA round. So for our pipeline, preserving the final message was more important than saving a small amount of generation time.

\n

Overall, for gpt-oss-120b, we did not need a complicated context compression algorithm. The context management strategy was simply:

\n\n

This was enough for all of our runs on the reference benchmark and internal benchmark. Since the resulting contexts were still comfortably below the configured context length, we did not optimize this part further.

\n

2. What to do in other cases when the above is not possible:

\n

The context-management strategy above relies on three conditions:

\n
    \n
  1. The model has a structured output format.
    2. \n
  2. The final message is a reasonably short summary of the full reasoning trace.
    3. \n
  3. Only a limited number of hints, in our case \\(n=8\\), are passed into the next RSA round.
    4. \n
\n

All three conditions held for gpt-oss-120b in AIMO3. However, in other settings, these assumptions may fail. For example:

\n
    \n
  1. A model may not separate reasoning from the final answer.
    2. \n
  2. Its final response may be too verbose.
    3. \n
  3. The specified RSA configuration uses a much larger number of parallel candidates.
    4. \n
\n

In those cases, a more explicit context-management strategy is needed.

\n

One natural strategy is the one described in the original RSA paper, Recursive Self-Aggregation Unlocks Deep Thinking in Large Language Models. The key idea is that the next RSA round does not need to condition every generation on all \\(n\\) previous solutions. Instead, for each generation in the next round, we randomly sample only a smaller subset of previous solutions.

\n

More concretely, suppose each RSA round produces \\(n\\) candidate solutions. Rather than passing all \\(n\\) candidates into every next-round prompt, we choose only \\(\\frac{n}{k}\\) candidates for each next-round generation, where \\(k\\) is a tunable parameter. Typically, \\(k\\) lies between \\(2\\) and \\(\\frac{n}{2}\\). Larger values of \\(k\\) correspond to smaller sampled subsets, and therefore shorter prompts.

\n

The expanded context-management strategy is:

\n\n
   prompt_tokens + sampled_hint_tokens + estimated_generation_tokens + buffer < max_context_length\n
\n

Here, estimated_generation_tokens can be inferred from the maximum or high-percentile generation length observed in earlier attempts.

\n\n

The original RSA paper includes the following diagram for the case \\( n=8 \\) and \\( k=2 \\), where each next-round generation sees 4 sampled hints rather than all 8:\n\"Recursive

\n

The main point is that RSA does not require the context size to grow uncontrollably. The sampling mechanism provides a direct knob for controlling prompt length. Smaller sampled subsets reduce prefill cost and help keep each prompt within the model’s context window, while still allowing information from previous candidates to propagate across RSA rounds.

\n

The trade-off is that larger values of \\(k\\), and hence smaller sampled subsets, make each individual next-round generation less informed. In particular, a sampled subset may not contain the correct solution, even if the correct solution appeared somewhere in the previous round. This can increase the probability that a given generation follows an incorrect line of reasoning. As a result, more RSA rounds may be needed for the correct solution to propagate through the population, increasing the sequential depth and wall-clock time per problem.

\n

However, this does not necessarily break RSA. If the correct solution appears in the previous round, then under random subsampling it will usually appear in multiple next-round prompts across the \\(n\\) new generations. Those generations can then amplify or refine it, and the information can continue spreading in later rounds. This is the mechanism by which RSA can still converge even when each individual generation only sees a subset of the previous candidates. The cost is mainly slower propagation, not necessarily loss of the signal entirely.

\n

In our AIMO3 implementation, this more complicated sampling strategy was not necessary because the final messages from gpt-oss-120b were already short enough. Even passing all 8 final messages into the next round typically added fewer than 8000 tokens, which was comfortably below our configured max_model_len = 81920. But for models with longer outputs or larger values of \\(n\\), this subset-sampling version of RSA would be the natural way to manage context length.

\n

(part 1 of 3)

", "votes": 1, "replies": [] }, { "id": 3457557, "author_name": "IceBeam777", "author_url": "", "post_date": "2026-05-13T23:15:05.837000", "content": "

(part 2 of 3)

\n

Question:

\n

Difficulty Buckets: Did you run any experiments to verify that the difficulty buckets were representative of the public/private sets, or were these buckets derived primarily from the previously shared X post?

\n

Response:

\n

Difficulty Buckets: We did not run experiments to verify how representative these buckets were of the public or private test sets, because contestants did not have access to those problems. Instead, the bucket construction was based on two sources of information.

\n

1. Runtime and RSA behaviour from available benchmarks.

\n

We used the 10 official reference problems and our small internal benchmark, whose construction process is described in Section 5, to understand how the system behaved on problems of different apparent difficulty. These runs helped us estimate runtime on easier problems, observe RSA behaviour on harder problems, and decide how many broad difficulty buckets were useful for scheduler design.

\n

2. Pass@k information shared by AIMO.

\n

The X post was used to estimate the approximate number of problems in each difficulty bucket. We interpreted the reported pass@k values cumulatively:

\n\n

This bucket construction should therefore be understood as a heuristic, not as a validated estimate of the true public/private distributions. Its purpose was not to precisely classify every problem, but to give the scheduler a rough prior over how much useful compute different kinds of problems might require.

\n

Although we could not directly validate the bucket distribution against the hidden leaderboard problems, the final leaderboard results provide some indirect evidence that the scheduler prior was reasonably well calibrated. Across public submissions, our system scored in the range of 40–42, and the final private leaderboard score was 41.5. The close agreement between these numbers suggests that the difficulty assumptions were at least useful enough to produce stable behaviour across the public and private splits, even if they should not be interpreted as an exact reconstruction of the hidden problem distribution.

\n

(part 2 of 3)

", "votes": 0, "replies": [] }, { "id": 3457558, "author_name": "IceBeam777", "author_url": "", "post_date": "2026-05-13T23:17:53.437000", "content": "

(part 3 of 3)

\n

Question:

\n

Answer Collapse: Since RSA propagates hints to future trajectories, how did you mitigate the risk of convergence toward an incorrect majority answer? In particular, if early generations contain many flawed solutions, using them as contextual hints in later iterations could potentially steer subsequent trajectories in the wrong direction. How did you address this failure mode?

\n

Response:

\n

Answer Collapse: There is no real way to completely prevent this failure mode, short of knowing the ground truth answer directly.

\n

There are two different concerns raised in this question:

\n\n

We were quite concerned about both at the beginning. However, after experimenting with gpt-oss-120b, our view became more nuanced. We found three things:

\n\n

1. This was less severe than expected

\n

RSA with gpt-oss-120b is able to recover minority correct solutions even among a majority of wrong ones.

\n

This issue is discussed in the V1 paper, V_1: Unifying Generation and Self-Verification for Parallel Reasoners, which we cited in the ablation study. Near the bottom of page 4, the authors write that RSA can “discard or degrade correct outlier solutions during refinement.” Their V1-Infer method was partly designed to address this weakness by ranking candidates through pairwise comparisons instead of relying on refinement alone.

\n

However, our experiments suggested that this failure mode is model-dependent. The V1 paper reports this issue for smaller models such as gpt-oss-20b and Qwen3-4B-Instruct. In our experiments, gpt-oss-120b was much less susceptible to it. Examples B.1, B.2, and B.6 in Appendix B show cases where RSA with gpt-oss-120b was able to rescue a minority answer from the initial candidate pool, even when many early generations were flawed.

\n

This led us to the following working hypothesis: for gpt-oss-120b, the important factor was often whether the correct solution, or at least a useful partial route to it, appeared somewhere in the candidate pool. It did not always need to be the majority answer initially. Once the correct idea appeared in one or more hints, later RSA rounds could usually amplify it.

\n

The ablation study in Section 6 also supports our decision not to switch to V1-Infer. We made several public leaderboard submissions using V1-Infer, and they scored in the 37–39 range, below the RSA-based submissions. So while V1-Infer is a reasonable strategy for mitigating answer collapse in principle, in our setting it did not improve the final leaderboard score.

\n

2. Mitigation measures

\n

Even though answer collapse was not as severe as expected, we still added two safeguards.

\n

First, the RSA prompt explicitly framed previous solutions as untrusted hints, not as ground truth. The RSA prompt (in appendix E.4) has a part that says:

\n
You are given a math problem, and some solution hints.\nSome hints are incorrect and contain errors.\"\n
\n

This wording was intentional. We wanted the model to solve the original problem again, while using previous candidates only as potentially useful evidence.

\n

Second, we reserved a small number of fresh attempts in later RSA rounds. These attempts were generated from the original problem directly, without conditioning on the previous round’s hints. In the final system, this was usually \\(F=2\\). The purpose was to preserve diversity and reduce the risk that all later generations would collapse onto the same incorrect reasoning path.

\n

This idea was inspired by Escaping the Cognitive Well: Efficient Competition Math with Off-the-Shelf Models. Although that paper uses a different problem-solving pipeline, diversity among candidate solutions also plays a central role there. In the paragraph “Phase III: Refinement with Global Memory” on page 5, the authors describe keeping one solver zero-shot during refinement, while the other solvers use accumulated context. The stated purpose is to preserve diversity and avoid falling into a “Cognitive Well.”

\n

Our fresh-attempt mechanism is the analogous idea adapted to RSA. Most generations in a later RSA round are conditioned on previous solution hints, but a small number are left unconditioned. This gives the system a way to keep exploring the original problem distribution, rather than forcing every trajectory to inherit the same potentially flawed context.

\n

3. Defining answer “convergence”

\n

In practice, the hardest part was actually deciding when the answer has \"converged\" and the answer distribution was reliable enough to stop.

\n

If the stopping rule is too aggressive, the system may accept a wrong majority answer too early. This is exactly what happens in the Class 4 and Class 5 failure modes described in Section 5.2, with examples in appendix items B.1, D.1.1, and D.1.2: the model quickly produces a clean-looking but wrong majority. These cases are difficult because the histogram looks similar to an easy problem where the majority answer is genuinely correct.

\n

On the other hand, if the stopping rule is too conservative, the system wastes time on problems that were already solved, and may even give RSA more opportunities to drift. Under the competition time limit, this also harms later problems by consuming too much of the global budget.

\n

So our approach was to take a middle ground. We measure convergence by looking at the answer distribution. We accepted strong Round 0 majorities, continued RSA on diffuse or unstable distributions, required progress for additional rounds, and reserved fresh attempts to preserve diversity. These rules do not eliminate answer collapse completely, but they made RSA stable enough in practice for gpt-oss-120b.

\n

The main remaining weakness is still the fast, confident wrong-majority case. We identified this explicitly in the writeup in section 7 as the most important failure mode for future work.

\n

(part 3 of 3)

", "votes": 0, "replies": [ { "id": 3457947, "author_name": "Simon Frieder", "author_url": "", "post_date": "2026-05-14T17:08:43.233000", "content": "

Nice reply! This is how a proper rebuttal would like like (e.g. as seen on OpenReview, for top ML conferences).

", "votes": 1, "replies": [] } ] } ] }, { "id": 3457565, "author_name": "IceBeam777", "author_url": "", "post_date": "2026-05-13T23:29:33.397000", "content": "

Edited on 14 May 2026 for the rebuttal phase:

\n
    \n

  1. Added public leaderboard score range and private leaderboard score (and also a small comment), at the top of the writeup.

    2. \n

  2. Edited section 1.2 drawbacks. Originally, there was an error saying that Δ was the probability that the model generates the correct solution in a single attempt. Instead, Δ should be the difference in probability of producing the correct answer and the probability of the second most likely answer. This does not affect our methodology and the rest of the writeup.

    3. \n
", "votes": 0, "replies": [] }, { "id": 3445867, "author_name": "Simon Frieder", "author_url": "", "post_date": "2026-04-20T16:07:42.453000", "content": "

Thanks for the kind words, appreciate it. Looking forward to reading the write-up in more detail! :)

", "votes": 0, "replies": [] }, { "id": 3445840, "author_name": "Durga Kumari", "author_url": "", "post_date": "2026-04-20T15:48:51.257000", "content": "

Great writeup\nThe combination of RSA with a difficulty aware scheduler is very insightful and clearly explained

", "votes": 0, "replies": [] }, { "id": 3445565, "author_name": "Utkarsh Sharma", "author_url": "", "post_date": "2026-04-20T04:51:39.703000", "content": "

@icebeam777 @davendw49 \nThis is probably the most novel inference-time contribution in the competition. The RSA diagrams make the mechanism very clear - especially the Norwegian Integers example where the correct answer (8687) goes from 0 appearances in Round 0 to majority in Round 1. That's a capability that no pure voting scheme (including mine) can replicate.

\n

The imo-bench-algebra-100 examples (Appendix B.2) are where it gets really interesting though. In the short run, the system returns answer=2 in ~270s with a strong 4-vote majority and entropies in [0.635–0.749] - indistinguishable from a confident correct answer. Your Class 4/5 taxonomy identifies this as the fundamental failure mode.

\n

My question is about the recovery mechanism:

\n
    \n

  1. In the extended run where RSA does recover answer=8, attempt 6 in Round 0 already contained it. For Class 5 problems where the correct answer is absent from ALL Round 0 attempts, how often does RSA Round 1 surface it for the first time through hint synthesis alone? The Norwegian Integers example shows this can happen - but is it representative across your 17-problem benchmark, or an outlier? A per-class recovery rate (especially Class 4 vs Class 5) would clarify whether RSA's value is primarily \"amplifying existing minority answers\" vs. \"generating new answers through partial-reasoning synthesis.\"

    2. \n

  2. Also curious about your scheduler under adversarial ordering. Your time bank reserves 250s early → 295s late, with an expected 0.5 time-starved problems. Was that estimate computed under random ordering, or did you stress-test against worst-case sequences (all bucket D+E problems in positions 1–10)? @rzatemizel Monte Carlo simulations show that under unlucky ordering, a public-notebook scheduler can fail on 15/50 - which is a very different number than 0.5.

    3. \n
\n

The speculative decoding + prefix caching combination giving you ~25% total speedup is something I left on the table entirely. Well executed.

", "votes": 0, "replies": [ { "id": 3446201, "author_name": "IceBeam777", "author_url": "", "post_date": "2026-04-21T06:48:18.027000", "content": "

Hi @flamingice2801,

\n

Thank you for reading my write up!

\n

Here are my (preliminary) answers:

\n
    \n

  1. Right now I am not sure about this. Anecdotally, it is not an outlier, but it happens rarely enough ( < 25% of the time) that it is worthwhile to inject fresh attempts to each subsequent RSA round to improve diversity of solutions. I am trying to secure some GPUs to hopefully run some tests on this in the next week or two.

    2. \n

  2. I tested this with two methods:

    \n

    a. Dynamic Programming to explore all 50! possible orderings of problems:\nhttps://pastebin.com/3GjGenRa\n(warning: Takes a long time to run)

    \n

    b. Monte Carlo Simulation for 20000 random orderings:\nhttps://pastebin.com/TvJ9uegH

    3. \n
\n

For b) , in the worst case assuming that all problems in buckets A, B, and C take the maximum amount of time. Problems in D and E average 850 seconds because of the RSA gating mechanisms. The code prints the following:

\n
Monte Carlo estimate: 0.50695\nHistogram: {0: 14165, 1: 3109, 2: 1627, 3: 745, 4: 258, 5: 73, 6: 18, 7: 4, 8: 1}\n
\n

That is assuming the worst case. In reality, many easy problems in bucket A finish in less than 100 seconds, and bucket B in less than 200, and bucket C in less than 300 (the worst case assumes 180, 280, and 550 respectively).

\n

In an average case, the Monte Carlo simulation prints:

\n
Monte Carlo estimate (average): 0.00095\nHistogram: {0: 19982, 1: 17, 2: 1}\n
\n

The expected number of Time-Starved problems is effectively 0. The exact value calculated in a) from Dynamic Programming prints this:

\n
Exact expected number of underallocated questions: 0.0008210684786705279\n
\n

In the absolute worst case where the first 10 problems are all hard (D/E), and all the medium problems (C), there will be more time starved problems. The Monte Carlo simulation above found 8 Time-Starved problems in 1 out of 20000 runs.

\n

rzatemizel's writeup analyzed the scheduler in the public notebook, which places a hard cap of 300 seconds once all the extra time has been consumed. They found that 15 problems can be Time-Starved in a bad case. This is far from the average case. From his chart, it seems that he placed 12 problems in bucket C. I only placed 3, because that is the number of problems in pass@5 but not in pass@3 in the chart released by the AIMO 3 organizers on X.

\n

My scheduler is also much less restrictive initially. It only reserves 250 s per problem when there are more than 30 problems remaining. Moreover, when we are behind time, the Time Bank keeps track of how far we are behind, and each subsequent problem only has to pay back self.cfg.time_bank_weight = 0.40 each time, instead of enforcing a hard cap.

", "votes": 0, "replies": [ { "id": 3446682, "author_name": "Utkarsh Sharma", "author_url": "", "post_date": "2026-04-22T00:28:31.670000", "content": "

Thank you for the detailed response - especially the Monte Carlo and DP results. The 0.00095 average-case vs. 0.507 worst-case gap is a really clean demonstration of why the time bank with 0.40 payback weight works so much better than a hard cap.

\n

The bucket calibration point is important too. Using the host's pass@n chart to size bucket C at 3 problems (pass@5 minus pass@3) is much tighter than the 12-problem estimate in rzatemizel's analysis. That single calibration difference explains most of the gap between your 0.5 expected time-starved and their 15.

\n

The <25% recovery rate for Round 0-absent answers is also useful context. Our approach uses 8 fully independent attempts at T=1.0 with no inter-attempt communication - which optimizes for the \"correct answer needs to be generated at least once\" case. RSA optimizes for the \"correct answer appeared once, now amplify it\" case. Would be very interesting to see a head-to-head on the same problem set if you get the GPU time - particularly on problems where the correct answer appears in exactly 1 of 8 initial attempts (the regime where RSA should have the largest advantage over independent voting).

\n

Looking forward to the private LB results.

", "votes": 0, "replies": [] } ] } ] }, { "id": 3445864, "author_name": "", "author_url": "", "post_date": "2026-04-20T16:06:09.533000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 15, "messages": [ { "id": 3444357, "postDate": "2026-04-17T17:46:17.340Z", "content": "

Public Leaderboard Score Range: 40 - 42

\n

Private Leaderboard Score: 41.5

\n

This work aimed to reduce variance of scores between runs. Judging from the close alignment between the private and public leaderboard scores, we believe we have succeeded in doing so.

\n

Resources

\n\n

Abstract

\n

This writeup describes the solution we used for the competition. The base model was gpt-oss-120b. The main benefit of this system over the self consistency baseline is, the observed mean increased from 38 to 41, and the variance decreased from (35-45) to (40-42). When writing this, we focused on the design choices we made, and why we made them. As a small team, we did not have the resources to run large scale experiments on many different models to fully test multiple different hypotheses. Instead, we tried to extrapolate from observations made from a small number of problems on two benchmarks: The reference benchmark provided by the AIMO3 organizing team, and another one we constructed on our own.\n
\n
\nOur solution is novel: No other solution used Recursive Self Aggregation as the core inference method together with a scheduler built around global estimates of problem difficulty. Most other solutions relied on plain majority voting and schedulers based only on remaining time slack. Our approach instead treats time allocation and inference as a coupled decision problem: the scheduler reasons about when RSA is worth continuing, and RSA is tuned around the budget structure imposed by the scheduler.

\n

Introduction

\n

The first three sections describe the main ideas behind the system, with greater emphasis on the components that required the most effort to design and implement. The fourth section explains how these ideas were combined into a system that performs well under the competition’s constraints. The fifth section presents the internal benchmark we used to guide development. The sixth section is an ablation study. Finally, the seventh section discusses limitations of the current system and potential directions for future work.\nThe write up is organized as follows:

\n
    \n
  1. Inference Algorithm: Recursive Self Aggregation
    2. \n
  2. Time Management: AI Hard Problems and Time Allocation
    3. \n
  3. Infrastructure Improvements
    4. \n
  4. Putting Everything Together
    5. \n
  5. Internal Benchmark
    6. \n
  6. Ablation Study
    7. \n
  7. Potential Weaknesses and Future Work
    8. \n
\n

1. Inference Algorithm: Recursive Self Aggregation

\n

We begin with the inference algorithm: Recursive Self Aggregation (RSA) [6]. RSA is a Test-time scaling (TTS) strategy that combines the the strongest features of several commonly used inference methods, including self-consistency [8], best-of-n [9], Monte-Carlo Tree Search [10], and sequential refinement [11]. \nTest-Time Scaling (TTS) refers to the class of strategies that improves language model performance by allocating additional compute at inference time. In Section 1.1, we briefly review several common TTS algorithms. In Section 1.2, we discuss their drawbacks. In Section 1.3, we explain Recursive Self Aggregation and compare it with the methods introduced earlier.

\n

1.1 Related Work: Common Test-Time Scaling Algorithms

\n\n

1.2 Drawbacks

\n

Each of the methods above has important limitations.

\n\n

1.3 Recursive Self Aggregation

\n

\"\"\nDiagram taken from [6]: Recursive Self-Aggregation Unlocks Deep Thinking in Large Language Models\n
\n
\nRecursive Self Aggregation (RSA)[6] is a Test-Time Scaling method that addresses many of the drawbacks discussed above. It is a hybrid scaling strategy that combines both parallel exploration and sequential refinement. RSA proceeds as follows:

\n
    \n
  1. Round 0: Sample N solutions.
    2. \n
  2. Round 1: Using the previous N solutions from Round 0 as hints in the prompt, ask the model to generate another N solutions.
    3. \n
  3. Round 2: Using the previous N solutions from Round 1 as hints, then generate another N solutions.
    4. \n
  4. Round 3: Using the previous N solutions from Round 2 as hints, then generate another N solutions.
    5. \n
  5. …
    6. \n
  6. Continue until the compute budget is exhausted or an early stopping condition is met.\nOne example run of RSA on the Norwegian integers problem from the reference benchmark is shown below:\n\"\"\nEarly Stop Condition: A simple early stopping rule is to terminate as soon as the majority class in a round reaches size 4. In practice, the stopping rule can vary across rounds, and these choices have a substantial effect on time usage per problem. We will return to this in the time management section.
    7. \n
\n\n

1.4 Fresh Attempts

\n

Empirically, RSA has a much higher chance of finding the correct answer when that answer is already present in the initial batch of N generations. However, some problems have the correct answer occupying only a very small portion of the answer distribution, often less than 0.2. In such cases, the correct answer may fail to appear in the initial N=8 generations simply due to bad luck. \n
\n
\nTo mitigate this, we reserve a small number of fresh attempts in later RSA rounds, usually F=2, in which the model is not shown the previous round’s solutions. This helps preserve diversity and reduces the risk of collapse toward an unhelpful local mode. The diversity-collapse issue, where RSA can fail if the previous solution pool is not diverse enough, is also hinted at in [5]. A related solution is proposed in [4] page 5, under Phase III: Refinement with Global Memory, where the authors suggest including fresh solver attempts during refinement rounds. We adapted that idea to the AIMO3 pipeline.

\n

2. Time Management: AI Hard Problems and Time Allocation

\n

We measure the AI-Hardness of a problem by two metrics:

\n
    \n
  1. Metric 1: The runtime needed for one attempt on the problem.
    2. \n
  2. Metric 2: The probability of producing the correct answer in one attempt.
    3. \n
\n

2.1 Why a scheduler matters

\n

In the 50 leaderboard problems, some are solved quickly and reliably, while others require substantially more search. A scheduler that ignores this variation will tend to waste time on short problems and cut off long problems too early.\nA good scheduler therefore has two jobs:

\n
    \n
  1. Ensure that every problem has enough time for at least one pass, based on metric 1.
    2. \n
  2. Identify problems that, when allocated more time, can improve on accuracy, based on metric 2.
    3. \n
\n

2.2 Difficulty Buckets and Reserve Policy

\n

Some information was shared about the difficulty of the public and private test sets on kaggle and on X. As mentioned above, the most popular public notebook scored between 35 and 45 on the public leaderboard, and based on forum discussion and our own submissions, its mean performance was around 37–39.\n
\n
\nUsing this information, we designed a scheduler based on estimated problem difficulty.\n
\n
\nThe first step was to group the 50 problems into rough difficulty buckets. we assumed that the private set would follow a similar distribution. Our working hypothesis was that strong public notebooks solved essentially all problems in bucket A, most in B, around half of C, and only a small minority of D and E. The suggested time ranges were based on observations from the 10 reference problems and the imo_answerbench benchmark.\n
\n
\nWe make the assumption that problems with lower pass@n take more time to run. The reality is more nuanced. There is a tiny class of problems where the model answers very quickly, but is almost always wrong. This will be discussed further in section 5.2 on the internal benchmark.\n\"\"\nFrom these buckets, we designed a scheduler that explicitly accounts for problem difficulty. The key challenge is to avoid pathological behaviour when hard problems arrive early. In particular, the scheduler must avoid panicking after seeing several difficult problems in a row and becoming too conservative, since that would leave too much time unused for easier problems later.\n
\n
\nThe scheduler therefore reserves time for future problems in a way that depends on how many remain. Early in the run, when more than 30 problems remain, it reserves only 250 seconds per future problem. In the middle of the run, with 16–29 problems remaining, it reserves 270 seconds per future problem. Late in the run, with 15 or fewer problems remaining, it becomes more conservative and reserves 295 seconds per remaining problem.\n
\n
\nIn addition, the scheduler maintains a time bank. The time bank tracks how far ahead or behind schedule the run is. Conceptually, the scheduler assumes a nominal average runtime per problem and accumulates deviations from that average. If earlier problems were solved faster than expected, the time bank becomes positive and later problems are allowed more time. If earlier problems consumed more time than expected, the time bank becomes negative and later budgets are reduced.

\n

2.3 RSA Gating: Consensus, Time, and Progress

\n

Recursive Self Aggregation is powerful, but it is also expensive. If used too freely, it can consume a large fraction of the notebook budget for very little gain. The scheduler therefore needs a rule for deciding when to stop after the initial batch and when to launch another RSA round.\n
\nWe implemented this as a three-part gating mechanism: consensus, time, and progress.\n

\n\n

2.4 Practical Effect: Low Expected Number of Time-Starved Problems

\n

\"\"\nA useful way to evaluate the scheduler is to ask how often it forces the system to cut off a problem materially earlier than would have been appropriate for its difficulty. We refer to such cases as Time-Starved problems.\n
\n
\nFor example, a problem in bucket D/E is Time-Starved if it was not allocated the full 1050 seconds by the scheduler AND the solution was cut short (i.e. system wants to continue with RSA from consensus and progress evaluations, but could not do so due to the time condition). A problem in any difficulty bucket A-E is also considered Time-Starved if the system does not have time to even complete round 0, the initial round of 8 generations (and forced to return N/A answer).\n
\n
\nTo check the effectiveness of our scheduler, we used dynamic programming (warning: takes a long time to run) and monte carlo simulation to estimate the number of time-starved problems in a typical run. The dynamic programming scripts deterministically calculates the expected value of time starved questions over all 50! possible orderings, while the monte carlo simulations finds the average over 20000 random orderings.\n
\n
\nAssuming average runtimes as per the table in section 2.2, both dynamic programming and monte carlo simulation estimates that the average count is less than 0.001. Assuming the upper end of runtimes, monte carlo simulation estimates the count is 0.506.\n
\n
\nWe stress that these estimates depend on the accuracy of both the bucket assumptions and the runtime model. Still, this estimate provides some assurance. The scheduler is expected to fail catastrophically only very rarely.\n
\n
\nOn the other hand, monte carlo simulation for old scheduler under pessimistic runtime assumptions gives an expected count of 0.932 time starved questions per run.\n
\n
\nImplementation details of the time scheduler and RSA gating are in Appendix C.

\n

3. Infrastructure Improvements

\n

As mentioned in section 2, Recursive Self Aggregation (RSA) is a very costly process. This means that increasing token throughput of the system is a direction very worth exploring. This section is dedicated to describing the improvements and optimizations I made to speed up inference and increase system reliability.

\n

3.1 Speculative Decoding

\n

Speculative decoding increases token throughput by using a smaller and faster draft model to propose several next tokens, and then asking the main model to verify them in parallel. Instead of forcing the large model to generate every token one by one, the system lets the small model guess ahead and uses the large model mainly as a checker. When the draft model’s guesses are accepted, multiple tokens can be emitted in a single verification step, which reduces inter-token latency and increases token throughput.\n
\n
\nFor the draft model, we used this draft model trained by wenliangtlh. This was the only model that was able to consistently speed up generation. Observed runtimes on the 10 reference problems decreased by about 15% with this change.\n
\n
\nWe also tried Nvidia's model trained for long contexts. However, this model consistently lead to a slowdown in generation.

\n

3.2 Prefix Caching (vllm >= 0.15.1)

\n

Another important system optimization was prefix caching. vLLM’s Automatic Prefix Caching (APC) stores the KV-cache blocks for prompts that have already been processed. When a later request shares the same prefix, vLLM can directly reuse the cached KV states and skip recomputing that shared part of the prompt. This is very important for AIMO3 because parallel generations always share the same prompt. System prompts, format hints and RSA prompts also stay the same across many vLLM requests. \n
\n
\nFor GPT-OSS models specifically, upgrading to vLLM >= 0.15.1 was important because the official release notes for that version explicitly list a bug fix: “Fixed prefix cache hit rate of 0% with GPT-OSS style hybrid attention models.” This upgrade further decreased runtimes on the 10 reference problems by another 10%.

\n

3.3 Parameter optimizations

\n

There were a number of further optimizations made to the system, including setting max_model_len = 81920, quantizing the KV-Cache using llm-compressor, guarding against vLLM crashes, dealing with memory issues in the python tool, using the Z3 solver, parsing code from the model, and how the final answer is parsed. \n
\n
\nFurther details on these improvements are in Appendix A.

\n

4. Putting Everything Together

\n

The final system combines RSA, tool use, prompting, scheduler-based time management, and inference/infrastructure optimizations into one end-to-end pipeline.

\n
    \n
  1. When a problem is served, the system first formats the problem using a prompt template.
    2. \n
  2. It then runs Round 0, generating an initial batch of 8 candidate solutions in parallel.
    3. \n
  3. Each solution is parsed into a final integer answer, and the resulting answer histogram is used to check whether the answer is already clear enough to accept.
    4. \n
  4. If Round 0 produces a strong consensus, the system stops immediately. This is how most easy and medium problems are handled.
    5. \n
  5. If the answer distribution is not yet convincing, the system does not automatically continue. Instead, the scheduler decides whether another RSA round is worthwhile.
    6. \n
  6. This decision is based on three conditions discussed earlier:
    7. \n
  7. If another RSA round is approved, the system builds a new prompt using the previous round’s solutions as hints.
    8. \n
  8. At the same time, a small number of fresh attempts are reserved, so that not all generations are conditioned on the previous round. This helps reduce diversity collapse.
    9. \n
  9. The next batch is again generated in parallel, parsed, aggregated, and evaluated using the same stopping logic.
    10. \n
  10. This loop continues until either:
    11. \n
\n\n

5. Internal Benchmark of AI-Hard Problems

\n

In this section, we study the relationship between accuracy, runtime, and RSA behaviour.

\n

5.1 Constructing the Benchmark

\n

Problems

\n

We curated a small internal benchmark to experiment on, in order to identify the main failure modes of the system. \nThe internal benchmark consists of 17 problems. Its purpose was not to approximate the full competition distribution, but rather to provide a stable stress test for scheduler behaviour and the ability of RSA to recover answers beyond a single majority-vote round. It was on these observations that we designed the scheduler and RSA gating rules. The problems are:

\n
    \n
  1. imo-bench-combinatorics-042
    2. \n
  2. imo-bench-combinatorics-054
    3. \n
  3. imo-bench-combinatorics-057
    4. \n
  4. imo-bench-combinatorics-068
    5. \n
  5. imo-bench-number_theory-023
    6. \n
  6. imo-bench-algebra-056
    7. \n
  7. imo-bench-algebra-100
    8. \n
  8. imo-bench-combinatorics-013
    9. \n
  9. imo-bench-combinatorics-014
    10. \n
  10. imo-bench-combinatorics-031
    11. \n
  11. imo-bench-combinatorics-039
    12. \n
  12. imo-bench-combinatorics-049
    13. \n
  13. imo-bench-algebra-015
    14. \n
  14. imo-bench-algebra-026
    15. \n
  15. imo-bench-algebra-052
    16. \n
  16. imo-bench-algebra-055
    17. \n
  17. 86e8e5 (Norwegian Integers)
    18. \n
\n

Construction Process

\n

The creators of the imo_answerbench dataset noted in their paper that they modified the problems in several ways to harden them against AI. Hardening methods include paraphrasing, changing the name of objects in the problem (such as changing point names for geometry problems), reformulating, modifying numerical values\nand/or adding distractors to the problem. This process was done either manually or automatically using language models.\n
\n
\nUnfortunately, some of these modifications lead to genuine mathematical errors in the problem statement, causing the ground truth answer to be wrong. When constructing our benchmark, we manually reviewed the problems to ensure that only correct problems are retained. Ambiguous and wrong problems were discarded. \n
\n
\nSome examples of discarded problems are in Appendix D.\n
\n
\nWe constructed this benchmark in several stages:

\n
    \n
  1. Filtered the imo_answerbench dataset to problems with integer answers only.
    2. \n
  2. Ran a majority@8 solving process on the filtered set, with 4 independent runs per problem.
    3. \n
  3. Selected the problems that were not solved in at least one of the 4 runs, yielding 53 candidate hard problems.
    4. \n
  4. Manually reviewed these candidates to remove problems that were not suitable for benchmarking, such as problems with incorrect ground-truth answers due to mathematical errors in the statement, and problems where failure was caused mainly by language or convention mismatches rather than genuine difficulty.
    5. \n
  5. Retained 16 problems that were judged to be both valid and genuinely difficult.
    6. \n
  6. Added the Norwegian integers problem from the official AIMO3 reference benchmark, giving a final benchmark of 17 problems.\n
    \n
    7. \n
\n

5.2 Observations from the Benchmark

\n

Experiments on the benchmark revealed an interesting pattern: when the system failed, the resulting answer histograms tended to fall into a small number of recurring classes.

\n\n

6. Ablation Study

\n

6.1 RSA versus V1-Infer

\n

V1-Infer [5] is a scoring method for solutions generated in parallel. After generating N candidates, it compares them pairwise and asks the model which solution is better. The comparison schedule is designed so that every candidate is matched against at least several others, and a final score is computed from both its win rate and the strength of the candidates it defeats. The main idea is that models are often better at making relative judgments between two candidates than at assigning a stable absolute score to one candidate on its own.\n\"\"\nWe made five public leaderboard submissions using V1-Infer, and they all scored between 37 and 39. \n\"\"\nThis was below the 40 to 41 achieved by the RSA-based notebooks, excluding the run where we changed the per-problem time reserve. Based on this, we decided not to explore V1-Infer further. The V1 paper also reports that V1-Infer tends to outperform methods that score each solution independently, so we did not spend additional time exploring those alternatives either.

\n

6.2 Prompts

\n

Throughout the competition, we experimented with a few different system prompts. The first one was adapted from the paper 11. It had the strange effect of making the model reason faster, but at a lower quality. It was kind of like decreasing the reasoning effort from HIGH to MEDIUM-HIGH. The following table records the effect on runtime of the system prompt from the cognitive well paper on the reference benchmark:\n\"\"\nIn this figure, each row corresponds to a problem and each column to a run. The table on the left uses the system prompt from the public notebooks, while the table on the right uses the prompt adapted from the Cognitive Well paper. Each cell shows the runtime for that problem in that run. Green indicates that the problem was solved correctly, while red indicates an incorrect answer. The shade encodes how many RSA rounds were used: lighter shades correspond to more rounds. No problem required more than 3 RSA rounds.\nFrom this table, the average runtime decreased by about 20%, from roughly 3000 seconds to 2400 seconds. However, this speedup came with a clear drop in reasoning quality. With the new prompt, the model got more problems wrong in some runs, and more problems required RSA, suggesting that round 0 answers were less reliable and needed additional refinement.\n\"\"\nThis reduction in reasoning quality was also reflected on the public leaderboard. Submissions using this system prompt scored between 30 and 35, well below the 38 to 39 baseline obtained by the public notebooks.\nWe also tried another prompt that included notes on mistakes the model had made on the IMO AnswerBench dataset. This produced no clear improvement or degradation.\nIn the end, we settled on the simplest possible system prompt, specifying only the required output format. Shorter system prompts reduce prefill overhead for every request.\nfull prompts are provided in Appendix E.

\n

7. Potential Weaknesses and Future Work

\n

The main weakness of the system is on problems where the model converges quickly and confidently to a wrong answer. On the internal benchmark, these were Classes 4 and 5 where the answer histogram looks deceptively clean. For some problems the runtime is also short, so neither answer histograms nor timing heuristics reliably identifies them as hard cases. This creates a fundamental tension in the scheduler. Aggressive early stopping saves time on easy problems, but also risks missing these failures. A promising direction for future work is to develop stronger continuation rules that use richer signals than answer histograms and runtime alone.

\n

References

\n

Papers

\n
    \n

  1. Huang, B., Li, S., Wu, T., Yang, Y., Talwalkar, A., Ramchandran, K., Jordan, M. I., & Jiao, J. (2026). Sample complexity and representation ability of test-time scaling paradigms. ICLR 2026. link

    2. \n

  2. Dorner, F. E., Chen, Y., Cruz, A. F., & Yang, F. (2026). ROC-n-reroll: How verifier imperfection affects test-time scaling. ICLR 2026. link

    3. \n

  3. Khairi, A., D'souza, D., Fadaee, M., & Kreutzer, J. (2025). Making, not taking, the best of N. ICLR 2026 link

    4. \n

  4. Dang, X., Agarwal, R., Porto, R., Goyal, A., Fowl, L. H., & Arora, S. (2026). Escaping the cognitive well: Efficient competition math with off-the-shelf models. link

    5. \n

  5. Singh, H., Li, X., Sareen, K., Maheswaran, M., Tan, S., Wu, X., Wang, J., Ariyak, A., Wu, Q., Khaki, S., Tiwari, R., Lian, L., Lu, Y., Li, B., Suhr, A., Athiwaratkun, B., & Keutzer, K. (2026). V_1: Unifying generation and self-verification for parallel reasoners. link

    6. \n

  6. Venkatraman, S., Jain, V., Mittal, S., Shah, V., Obando-Ceron, J., Bengio, Y., Bartoldson, B. R., Kailkhura, B., Lajoie, G., Berseth, G., Malkin, N., & Jain, M. (2025). Recursive self-aggregation unlocks deep thinking in large language models link

    7. \n

  7. Chen, W.-L., Peng, L., Tan, T., Zhao, C., Chen, B. J., Lin, Z., Go, A., & Meng, Y. (2026). Think deep, not just long: Measuring LLM reasoning effort via deep-thinking tokens. link

    8. \n

  8. Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., & Zhou, D. (2023). Self-consistency improves chain of thought reasoning in language models. ICLR 2023. link

    9. \n

  9. Huang, A., Block, A., Liu, Q., Jiang, N., Foster, D. J., & Krishnamurthy, A. (2025). Is Best-of-N the best of them? Coverage, scaling, and optimality in inference-time alignment. ICML 2025. link

    10. \n

  10. Xie, Y., Goyal, A., Zheng, W., Kan, M.-Y., Lillicrap, T. P., Kawaguchi, K., & Shieh, M. (2024). Monte Carlo tree search boosts reasoning via iterative preference learning. NeurIPS 2024 Sys2-Reasoning Workshop. link

    11. \n

  11. Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., & Clark, P. (2023). Self-refine: Iterative refinement with self-feedback. NeurIPS 2023. link

    12. \n

  12. Han, Q., Khamaru, K., & Zhang, C.-H. (2024). UCB algorithms for multi-armed bandits: Precise regret and adaptive inference. arXiv:2412.06126. link \nhttps://arxiv.org/abs/2412.06126

    13. \n

  13. Luong, T., Hwang, D., Nguyen, H. H., Ghiasi, G., Chervonyi, Y., Seo, I., Kim, J., Bingham, G., Lee, J., Mishra, S., Zhai, A., Hu, C. H., Michalewski, H., Kim, J., Ahn, J., Bae, J., Song, X., Trinh, T. H., Le, Q. V., & Jung, J. (2025). Towards robust mathematical reasoning. EMNLP 2025. link

    14. \n
\n

Notebooks

\n
    \n
  1. streaming-inference by Tong Hui Kang
    2. \n
  2. [44/50] LET ME (over)COOK!!! by parthenos
    3. \n
  3. [41/50] AIMO3: Confidence Voting by parthenos
    4. \n
  4. AIMO3-Qwen3.5-35Ba3B-simple-MCTS by yuri-ao
    5. \n
  5. [Way to 30+/50] GPT-OSS - w/ python tool by ShelterW
    6. \n
  6. AIMO 3 | GPT-OSS-120B (with tools) by Andreas Bisiadis
    7. \n
\n

Models

\n
    \n
  1. UnSloth's gpt-oss-120b
    2. \n
  2. Wenliangth's AIMO3 speculative decoding draft model
    3. \n
  3. Nvidia's long context speculative decoding draft model
    4. \n
\n

Appendix Contents

\n
    \n
  1. Appendix A: Model Parameters and Other Small Improvements
    2. \n
  2. Appendix B: Additional RSA Example Runs
    3. \n
  3. Appendix C: Scheduler and RSA Gating Implementation Details
    4. \n
  4. Appendix D: Internal Benchmark
    5. \n
  5. Appendix E: Prompts
    6. \n
\n

Appendix A: Model Parameters and Other Small Improvements

\n

There were a number of improvements made to the system that improved its speed and stability. Their details are included in this appendix.

\n

A.1 max_model_len

\n

This needs to be carefully balanced between concurrency and accuracy. We observed that for one problem on the reference and some more on imo_answerbench, the solution length frequently exceeded max_model_len = 65536 set in the public notebooks. The answers for these solutions will be parsed as N/A, which hurts performance. On the other hand, setting max_model_len = 131072 (maximum supported by gpt-oss-120b) hurts concurrency and hence end-to-end latency per problem, due to limited GPU memory and kv-cache size. In the end, we settled with max_model_len = 81920. This value has decent concurrency on H100 GPUs (~ 5x) and significantly increases the number of complete solutions.

\n

A.2 Quantized KV Cache

\n

Another important parameter optimization was quantizing the KV cache. In transformer inference, the key and value tensors from previous tokens are stored in memory so that later tokens can attend to them. For long-context generation, this KV cache consumes a large amount of GPU memory and is often the main bottlenecks limiting concurrency. vLLM supports quantizing this cache to FP8, which substantially reduces its memory footprint. For this, we used kv_cache_dtype=fp8_e4m3.\nQuantizing the KV cache to FP8 requires choosing appropriate scale factors for the key and value tensors. These scales determine how the original tensors are mapped into the much lower-precision FP8 format and then reconstructed during attention. Calibration is the process of estimating suitable scales from observed activations so that FP8 uses its limited numeric range effectively and minimizes quantization error. \nWithout calibration, vLLM falls back to default scales of 1.0 when no checkpoint scales are available, which is a much cruder approximation and can degrade attention quality, especially on long or delicate reasoning traces. In this work, we experimented with both lightweight runtime calibration via calculate_kv_scales=True, and offline calibration using the llm-compressor package on the AIMO3-High-Difficulty-Tool-Calling-Dataset by Wenliangth. The latter uses representative data to produce scales better matched to the target workload.\nThe final submission was the model calibrated with llm-compressor. vLLM provides some documentation for using this package here.

\n

A.3 Guarding against vLLM crashes

\n

There is a very rare bug (causing vLLM to crash) that may be triggered when gpt-oss-120b is used with speculative decoding under high concurrent workloads. This is a very recent issue reported here. We do not know the root cause of it, but we guarded against it by checking for vLLM server health at the beginning of each problem, restarting the server if necessary. Otherwise, the system will return N/A for the rest of the unsolved problems.

\n

A.4 Python Tool

\n

A number of improvements were made to the python tool.

\n

A.4.1 Memory Issues

\n

The python tool causes memory spikes. From experiments, there are two main reasons that cause this:

\n\n

A.4.2 Z3 Solver

\n

We also installed the z3 package, which is an SMT (Satisfiability Modulo Theory) solver developed by Microsoft Research. At a high level, Z3 is designed to determine whether a collection of logical, algebraic, or combinatorial constraints can be satisfied, and if so, to return a solution. For AIMO3, this can be useful when the model reformulates part of a problem into a system of equations, inequalities, divisibility conditions, or discrete case constraints. GPT-OSS-120B has been observed to use this package if told via prompting that it is available.

\n

A.4.3 Parsing Code from Model Output

\n

A small but important engineering issue arises from how gpt-oss-120b writes Python code. The model often writes code in the style of a Jupyter notebook cell, where the final expression is implicitly displayed without being explicitly printed. For example, in a notebook cell, code such as

\n
x = 2\ny = 3\nz = x + y\nz\n
\n

will display 5 automatically. However, when the same code is executed inside the tool subprocess, the final expression is not written to stdout, so nothing is captured by the tool backend. A common workaround in public notebooks is to wrap the final line in print(...) before execution. \nHowever, this approach is fragile. The last line of the generated code is not always a simple expression. For example, the model may end with a control-flow statement such as continue, break, or pass, or with the tail of a loop or conditional block. In such cases, blindly rewriting the last line into print(...) can produce invalid code such as print(continue), which immediately raises a syntax error.\nTo make code execution robust, we instead used Python’s abstract syntax tree (ast) module to analyze the generated code structurally before execution. The parser first walks the syntax tree to detect whether the model already included any explicit print(...) statements. If so, the code is executed as written, since the model has already specified what should be sent to stdout. If no print statements are present, we collect the variables assigned anywhere in the code and append additional code that prints their values before execution.

\n

A.5 Streaming Behaviour and Answer Parser

\n

Another small but important difference from many public notebooks is how we handled streaming output and answer extraction.\nIn several public notebooks, the model output is streamed token by token, and generation is terminated as soon as \\boxed{...} appears in the output. This is an attractive optimization because it can save time on easy problems where the answer is produced early. However, we found this unsafe for our RSA pipeline for two reasons:

\n\n

Appendix B: Additional RSA Example Runs

\n

B.1 Norwegian Integers (86e8e5)

\n

\"\"\n
\n

\n\n

B.2 imo-bench-algebra-100

\n

\"\"\nThis problem is on Vasc inequality. The answer is 8. Refer to appendix D for more details. It is an example of a problem in Class 5 as described in section 5.2 , where the correct answer is in a tiny minority and a single wrong answer dominates the majority.\nThis run had early-stop set to 4 instead of 6. The system returned an answer relatively quickly (~270 s) with a strong majority. However, the majority is wrong.\n\"\"\nWe might also get lucky on the run. In the above run, the decision margin was set to 3, meaning that round 0 (index 0-7) results being a 5-3 split for the answers 2 and 8 triggered RSA round 1. after two further rounds of RSA, the system decided that the answer is 8, which is correct.

\n

B.3 Reference Problem 7 (641659)

\n

\"\"\n
\n
\nThis is problem 7 on the reference benchmark. The answer is 57447. This run also had early-stop set to 4. The system returned an answer in ~292 s, which is similar to imo-bench-algebra-100 above. The answer distribution is also similar, with one answer firmly in the majority class. The only difference is that the majority class is correct, where as for imo-bench-algebra-100 it is wrong. This makes it very difficult to differentiate these two problems via runtime and answer distribution heuristics alone.\nThis creates a tension between early accept based on majority voting versus launching RSA.

\n

B.4 imo-bench-combinatorics-068

\n

\"\"\n
\n
\nThis is a combinatorics problem from the imo_answerbench dataset. It is an example of a problem in Class 2 as described in section 5.2, where there is an even split between two answers. The model was asked to return its answer modulo 100000, to fit within AIMO3's answer range of 0 - 99999. The answer is 93447.\nThe model is evenly split between 93447 and 71207. In an unlucky run (such as this one), simply taking majority vote at N=8 will yield the wrong answer. RSA was able to correct the error.

\n

B.5 imo-bench-number_theory-023

\n

\"\"\n
\n
\nThis is a number theory problem from the imo_answerbench dataset. It is another example of a problem in Class 2 as described in section 5.2, where there is an even split between two answers. The answer is 48.\nThe model is split between 48 and 64. In an unlucky run, majority vote at N=8 might yield the wrong answer. RSA is able to identify the answer.

\n

B.6 imo-bench-combinatorics-057

\n

\"\"\n\"\"\n\"\"\n\"\"\n
\n
\nThis is a combinatorics problem from the imo_answerbench dataset. It is an example of a problem in Class 2 as described in section 5.2 , where the correct answer is in a small minority and a few wrong answers form the majority. RSA was able to obtain the correct answer after 3 rounds, which surprisingly managed to fit within the time budget.

\n

Appendix C: Scheduler and RSA Gating Implementation Details

\n

Early in the competition, this notebook by ShelterW used a scheduler that allocated a fixed amount of time to each problem, without accounting for problem difficulty. That approach typically averaged around 30–32 on the public leaderboard, with a best observed leaderboard score of 37. Later notebooks by parthenos and Andreas Bisiadis introduced an improved scheduler that reserved a fixed, below-average amount of time for each remaining problem. This allowed hard problems encountered early to “borrow” time from easier problems later in the run. Those notebooks typically averaged around 37–39, with public leaderboard variance roughly between 35 and 45.\nOne important caveat is that the bucket model assumes lower pass@n roughly corresponds to longer useful runtime. This is not always true. A small class of problems elicits very fast but usually incorrect answers, so difficulty (measured in pass@n) and runtime are not perfectly aligned. Please refer to the internal benchmark section above for further discussion on this issue.

\n

C.1 Scheduler Reserve Policy and Time Bank Parameters

\n

We begin the run on problem 1. For future problems, the scheduler reserve time differently based on the number of problems left:

\n\n
initial_time_bank = 1500.0\ntime_bank_weight = 0.40\ntime_bank_min = -1800.0\ntime_bank_max = 3000.0\nhigh_problem_timeout = 1050.0\nbase_problem_timeout = 240.0\n
\n\n
elapsed_global = time.time() - self.notebook_start_time\ntime_left = self.cfg.notebook_limit - elapsed_global\nproblems_left_others = max(0, self.problems_remaining - 1)\n\nreserve = self._reserve_per_future_question()\nreserved_time = problems_left_others * reserve\nbank = float(getattr(self, \"time_bank\", 0.0))\nbudget = time_left - reserved_time + self.cfg.time_bank_weight * bank     \nbudget = min(budget, self.cfg.high_problem_timeout)\nbudget = max(budget, self.cfg.base_problem_timeout)\nbudget = min(budget, max(0.0, time_left - self.cfg.finish_grace_s))\n
\n

So the current problem receives:

\n\n

C.2 Consensus Based Acceptance Rules

\n

The interaction between the scheduler and RSA behaviour must be carefully accounted for, and both must be carefully designed with the other in mind. Ideally, we want to only use it for cases where the next RSA round can complete in time and there is significant information to be gained.\nAfter Round 0, the system accepts immediately under the following rule:

\n
   def _accept_after_round0(self, ans_list: list[int], round0_runtime: float) -> bool:\n       hist = self._answer_hist(ans_list)\n       top, second = self._top2_from_hist(hist)\n\n       if top >= 6:\n           return True\n\n       if top == 5 and second <= 1 and round0_runtime <= self.cfg.accept_5_runtime_cap:\n           return True\n\n       return False\n
\n

In words, the answer is accepted after Round 0 if:

\n\n
def _confident_enough(top_cnt: int, second_cnt: int) -> bool:\n   if top_cnt < rsa_majority_threshold:\n       return False\n   decision_margin = getattr(self.cfg, \"decision_margin\", 2)\n   return (top_cnt - second_cnt >= decision_margin)\n
\n

Typically, rsa_majority_threshold = 4 and decision_margin = 2.

\n

C.3 Estimating the Cost of Another RSA Round

\n

To avoid entering an RSA round that cannot complete in time, the system estimates the cost of the next round using Round 0 runtime:

\n
def _estimate_rsa_cost(self, round0_runtime: float) -> float:\n   return max(self.cfg.min_rsa_cost, self.cfg.rsa_cost_fraction * round0_runtime)\n
\n

In our implementation, we set:

\n\n

C.4 Progress Score for Additional RSA Rounds

\n

For RSA rounds beyond Round 1, we required evidence of progress before continuing:

\n
def _rsa_progress_score(self, prev_answers: list[int], new_answers: list[int]) -> float:\n   prev_hist = self._answer_hist(prev_answers)\n   new_hist = self._answer_hist(new_answers)\n   prev_top, prev_second = self._top2_from_hist(prev_hist)\n   new_top, new_second = self._top2_from_hist(new_hist)\n   prev_distinct = len(prev_hist)\n   new_distinct = len(new_hist)\n   score = 0.0\n   if new_top > prev_top:\n       score += 1.5\n   if (new_top - new_second) > (prev_top - prev_second):\n       score += 1.0\n   if new_distinct < prev_distinct:\n       score += 1.0\n   return score\n
\n

This score increases when:

\n\n
elapsed_q = time.time() - t_question0\nextra_affordable = budget - elapsed_q + max(0.0, self.cfg.bank_borrow_fraction * self.time_bank)\nallow_second_rsa = (\n   max_rsa_rounds >= 2\n   and progress_score >= 1.5\n   and extra_affordable >= estimated_rsa_cost\n   and (deadline - time.time()) >= (estimated_rsa_cost + self.cfg.rsa_safety_seconds)\n)\n
\n

In words, another RSA round is allowed only if:

\n\n

Appendix D: Internal Benchmark

\n

This appendix documents some observations we made on the nature of some problems and how they relate to the observed answer distribution.\nIt also includes some examples of problems that were manually filtered out when constructing this benchmark.

\n

D.1 Observations

\n

D.1.1 imo-bench-algebra-100

\n

This problem is on Vasc inequality. This is an example of a problem in class 5 as described in section 5.2. There is a short but tempting solution leading to the wrong answer, while the correct answer requires substantially deeper reasoning that the model often skips.\n
\nProblem:\nFind number of triples \\( (x,y,z) \\) of real numbers satisfying $$x^2 + y^2 + z^2 = xy^3 + yz^3 + zx^3 = 3$$\n
\nAnswer: 8\n
\nAn easy guess and check finds 2 solutions: \\( (1, 1, 1) \\) and \\( (-1, -1, -1) \\). From here, the model often concludes that there are no more others. Unfortunately there are, but they are difficult to find.\n
\nVasc's Inequality says that for \\( x, y, z \\in \\mathbb{R} \\):\n
\n$$ (x^2+y^2+z^2)^2 ≥ 3(xy^3+yz^3+zx^3)$$\n
\nThe problem statement says that \\( x^2 + y^2 + z^2 = 3 \\) and \\( xy^3+yz^3+zx^3 = 3 \\), so we are in the equality case of Vasc's.\n
\nIn the equality case, there are additional solutions when \\( x \\), \\( y \\), and \\( z \\) are in the ratio of:\n
\n$$ x:y:z = sin^2(\\frac{4π}{7}) : sin^2 (\\frac{2π}{7}) : sin^2 (\\frac{π}{7})$$ \n
\nWe can also permute \\( x, y, z \\) cyclically (transpositions are not allowed). Flipping all their signs also preserves equality. Hence there are an additional 6 solutions to the equality expression, and the final answer is 8.\n

\n

D.1.2 imo-bench-combinatorics-031

\n

USA IMO Team Selection Test 2020, Day 1 Problem 3\n
\nProblem:\nLet \\( \\gamma \\geq 1\\) be a real number. Sun Wukong and the Sea God play a turn-based game on an infinite grid of unit squares. Before the game starts, the Sea God chooses a finite number of cells to be flooded with seawater. Sun Wukong is building a magical barrier, which is a subset of unit edges of the grid (called walls) forming a connected, non-self-intersecting path or loop. Additionally, there is a magical artifact that randomly generates a finite number of extra walls on the grid, with no specific pattern or distribution.\n
\nThe game then begins with Sun Wukong moving first. On each of Sun Wukong's turns, he adds one or more walls to the magical barrier, as long as the total length of the barrier is at most \\( \\gamma n \\) after his \\(n^{th} \\) turn. On each of the Sea God's turns, every cell which is adjacent to an already flooded cell and with no wall between them becomes flooded as well. Sun Wukong wins if the magical barrier forms a closed loop such that all flooded cells are contained in the interior of the loop, hence stopping the flood and saving the world. What is the largest constant \\( C \\) such that for all \\( \\gamma > C\\) can Sun Wukong guarantee victory in a finite number of turns no matter how the Sea God chooses the initial cells to flood?\n
\nAnswer: 2\n
\nThis is another example of a problem in class 5 as described in section 5.2. The 'magical artifact' statement is a distractor, but it doesn't affect strong models like gpt-oss-120b.\n
\nThe short but tempting solution is observing that when unrestricted, the perimeter of the flood grows by 8 each turn, and concluding that \\( C > 8 \\) for Sun Wukong to guarantee victory. The model often makes this observation and then stops here, returning the answer 8.\n
\nHowever, the correct solution provides a construction based on a different intuition. Sun Wukong can cut off the flood from one direction first. In this step, he needs to ensure that the wall on that side grows fast enough compared to the flood's spread, hence we need at least \\( C>2 \\). Next, he can restrict the flood to a long, narrow corridor by constructing two parallel sides. Finally, he can construct the final wall to seal off the flood. This means that Sun Wukong only needs to ensure that the narrow corridor grows in length fast enough compared to the spread of the flood, which means we indeed only need \\( C > 2 \\) for Sun Wukong to contain the flood. The correct answer is 2.\n

\n

D.1.3 imo-bench-combinatorics-013

\n


\nThere are 42 students participating in the Team Selection Test, each of them is assigned a positive integer from 1 to 42 such that no two students have the same number and every number from 1 to 42 is assigned to a student. The team leader want to select a subset of these students such that there are no two students that have numbers whose difference is 1 or 21. For example, the team leader can pick the set \\( \\{1, 3, 6, 25, 28, 34, 42\\} \\) but not \\( \\{1, 2, 4, 6, 24\\} \\) or \\( \\{1, 3, 24, 26, 28\\} \\).\n
\nHow many ways can the team leader pick such a subset? Give your answer modulo 100000.\n
\nAnswer: 26555\n
\nThis is an example of a problem in Class 2 as described in section 5.2.\n
\nThe model formulates this problem as an independent-set count on a graph. The numbers 1–42 form a 2-by-21 ladder, and the task becomes counting subsets with no adjacent chosen vertices. From this, the model sometimes counts that there are 131836323 valid choices, and returns the answer 36323.\n
\nHowever, the constraints also mean that 21 and 22 cannot be selected at the same time. There are 11309768 such selections, and hence the correct count is 131836323 - 11309768 = 120526555, and the correct answer should be 26555. The model only makes this extra reasoning step around 50% of the time.\n

\n

D.2 Filtered Problems

\n


\nThe creators of the imo_answerbench dataset noted in their paper that they modfied the problems in several ways to harden them against AI. Hardening methods include paraphrasing, changing the name of objects in the problem (such as changing point names for geometry problems), reformulating, modifying numerical values\nand/or adding distractors to the problem. This process was done either manually or automatically using language models.\n
\nUnfortunately, some of these changes lead to genuine mathematical errors in the problem statement, causing the ground truth answer to be wrong. When constructing our benchmark, we manually reviewed the problems to ensure that only correct problems are retained. Ambiguous and wrong problems are discarded. This section contains some examples of discarded problems.\n

\n

D.2.1 imo-bench-geometry-002

\n


\nproblem 2 in the original problem source\n
\nModified Problem: Given right triangle \\( XYZ \\) with hypothenuse \\( XZ \\) and \\( \\angle X = 50^{\\circ} \\). Points \\( P \\) and \\( Q \\) on the side \\( YZ \\) are such that \\( \\angle PXZ = \\angle QXY = 10^{\\circ} \\). Compute the ratio \\( 2 \\times YQ/ZP \\).\n
\nDataset issue. Modified problem asks for 2 x \\( \\frac{YQ}{ZP} \\). The original problem labelled points in ABC instead of XYZ, and asked for the equivalent of \\( \\frac{ZP}{YQ} \\). It seems that the problem editor mixed up the line segments when editing the point names. The answer should be 1 instead of 4.\n

\n

D.2.2 imo-bench-geometry-044

\n

original problem source\n
\nModified Problem: Given \\( PQRS \\) is a parallelogram. We know that \\( \\angle S = 60^\\circ \\), \\( PS = 2 \\) and \\( PQ = \\sqrt{3} + 1 \\). Point \\( N \\) is the midpoint of \\( PS \\). Segment \\( RE \\) is the angle bisector of \\( R \\). Find the angle \\( REQ \\) in terms of degree.\n
\nDataset issue. Original problem contained an additional diagram implying the existence of an extra line, and the point \\( E \\) is supposed to lie on that line. This diagram is missing from the modified problem, leading to the model to have their own interpretation of where the points lie, hence outputting a different answer sometimes.\n

\n

D.2.3 imo-bench-number_theory-055

\n


\nDay 2 Problem 6 in the Estonia Team Selection Test 2018\n
\nModified Problem: A positive integer \\( m \\) consisting of distinct digits is considered \"good\" if it is a single-digit number, or if removing one of its digits results in a divisor of \\( m \\) that is also a good number.\n
\nFind the largest good number.\n
\nDataset issue. Original problem has an extra condition: We assume that numbers do not start with 0. This condition is not in the modified problem in dataset, which fundamentally changes the solution. Answer is 903125 without the condition, or 146250 with it.\n

\n

D.2.4 imo-bench-algebra-039

\n


\nProblem 5 on page 123 of the original source\n
\nModified Problem: Let \\(p\\), \\(q\\), \\(r\\), \\(s\\) be constants such that the equation $$ py^3 + qy^2 + ry + s = 0 $$ has three distinct real roots. Find all possible values for the number of distinct real roots of the equation\n$$ (pz^3+qz^2+rz+s) (6pz+2q) = (3pz^2+2qz+r)^2 $$\n
\nError in problem. The original problem (Ukraine 1997) asked for:\n
\n$$Q(x) = 4(px^3 + qx^2 + rx + s)(3px + q) = (3px^2 + 2qx + r)^2 $$\n
\n-> answer = 2\n
\nThe modified problem asks for (after moving out a factor of 2 from within the brackets on LHS):\n
\n$$ Q(x) = 2(px^3 + qx^2 + rx + s)(3px + q) = (3px^2 + 2qx + r)^2 $$ \n
\n-> answer = 0\n
\nBoth Gpt 5.2 and Gemini get both variants correct. Solution involves applying Descarte’s rule of signs, which gives an answer in the original problem but is inconclusive in the modified problem. Solving the modified problem involves analysing the discriminant of the cubic equation.\n

\n

D.2.5 imo-bench-combinatorics-005

\n


\nProblem: Determine the number of natural numbers n that that has at most 16 digits satisfying the following conditions:

\n\n

D.2.6 imo-bench-combinatorics-020

\n


\nDay 1 Problem 3 in the Vietnam Team Selection Test 2020\n
\nProblem:\n
\nSuppose there are 40 professional baseball teams participating in a tournament. In each round of the game, we will divide the 40 teams into 20 pairs, and each pair plays the game at the same time. After the tournament, it is known that every two teams have played at most one game. Find the smallest positive integer \\( a \\), so that we can arrange a schedule satisfying the above conditions, and if we take one more round, there is always a pair of teams who have played in the game.\n
\n
\nThere is a tension here between \"maximally cooperative scheduling\" and \"adversarial scheduling\". \n

\n\n

Appendix E: Prompts

\n

Througout the competition, we tried various prompts crafted with ideas from different sources.

\n

E.1 Cognitive Wells

\n

This prompt was adapted from Escaping the Cognitive Well: Efficient Competition Math with Off-the-Shelf Models. The main finding of the paper is that solutions to problems can be broken down into smaller lemmas. The paper reports that this decomposition improves performance, and attributes the gain largely to the fact that the intermediate lemmas are easier for the model to reason about in isolation, without the context of the larger main problem interfering. The authors also provide a system prompt that encourages the model to verify its reasoning, check for logical inconsistencies, and avoid hand-wavy arguments before returning an answer. we adapted this prompt for AIMO3, but found that it significantly degraded performance on the public leaderboard.

\n

\"\"\nThe scores are 30-35. The prompt is as follows:

\n
system_prompt = (\n\"\"\"\n### Prompt: DialecticEngine (Council of Architects)\nMUST BE FOLLOWED METICULOUSLY\n### Inputs for this Task\n**1. The Problem:**\n[See User Input]\n**2. Additional Materials:**\n[See User Input. Materials will be clearly graded.]\n---\n## DIALECTIC ENGINE PROMPT\n**YourRole:**\nYou are a multi-persona computational engine that solves difficult problems via a rigorous, multi-stage dialectic process. Your goal is to explore the problem space creatively and deeply, avoiding premature conclusions and logical gaps. The final answer is always an integer from 0 to 99999 and must be placed in \\boxed{answer}, e.g., \\boxed{42} (if solution is complete).\n**Configuration:** `MAX_INNER_ITERATIONS: 3`\n---\n### Core Principle: Atomicity of Logic\nAll solution drafts (`Current_Draft`, `Refined_Draft`, `Solution_Submitted_for_Review`) must be **strictly self-contained, line-by-line logical arguments**. Every step must be justified either by naming a standard theorem, or by a proved claim/lemma in a previous line of the same draft.\n*Additional materials* can inform the solution, but the solution should be self-contained.\n### Core Principle: Source Hygiene\n`Additional Materials` contain raw outputs from other agents. Treat them as **unverified hints**. If your solution uses a conjecture or lemma from them, you **must prove it from scratch**. Standard mathematical theorems (e.g., Cauchy–Schwarz) may be cited by name without proof.\n### Sloppiness\nSloppiness is intolerable. Examples:\n(a) Asserting “well-known fact” without an exact reference  \n(b) Showing for a couple of values of `n` and asserting it is proved for all `n`\nIf the solution has known gaps, they must be rephrased as a **clear and self-contained** conjecture.\n---\n#### Personas\n* **The Council of Architects:**\n * **Classicist:** Relies on established theorems and formal rigor.\n * **Visionary:** Proposes novel, abstract, and unconventional connections.\n * **Experimenter:** Tests hypotheses with concrete examples and computational thinking.\n * **Momus, the Skeptic:** Adversarially critiques the Council’s *strategy* and *high-level approach* to find conceptual flaws. Only asks tough questions that others must answer. *Does not* help prove in any way.\n * **Veritas, the Auditor:** A meticulous, line-by-line proof checker. Veritas’s sole function is to ensure every logical step in a draft adheres to the **Core Principle of Atomicity of Logic**. Veritas does not judge the overall strategy, only its step-by-step validity.\n * **The Chief Architect:** Manages the process, arbitrates disputes, and synthesizes the final result. Enforcer of the FUNDAMENTAL RULE: No evasive phrases (e.g., “it can be shown,” “it is clear that,” or “known results imply,” “etc.”) in `Current_Draft` unless they are immediately followed by the complete derivation or citation. Flags such phrases and sets `Needs_New_Draft = TRUE`.\n---\n#### Glossary of Terms\n* **`STRATEGIC_LEAD`:** A strategy flagged by Momus as a potential dead end.\n* **`LOGICAL_GAP`:** A line in a solution flagged by Veritas as unjustified looking solely within the solution (e.g., made-up claims).\n* **`DEAD`:** A `STRATEGIC_LEAD` whose rebuttal was unconvincing. It is permanently discarded.\n* **`Rebuttal`:** The Council’s defense of a `STRATEGIC_LEAD`, achieved by producing a refined solution.\n---\n**(You will output the following process in REAL-TIME.)**\n---\n## BEGIN PROCESS\n### Stage 1: Ideation Forum\n1. **Brainstorm (Council of Architects):** The Council analyzes the problem and generates a list of “Promising Avenues & Key Concepts.”\n2. **Pre-mortem (Momus, the Skeptic):** Concurrently, Momus analyzes the problem to identify potential “Red Herrings & Inefficient Paths.”\n3. **Synthesize (Chief Architect):** The Chief Architect compiles the outputs into a formal `Strategy_Dossier`.\n4. **Select Strategy:** The Chief Architect selects the most promising strategy from the `Strategy_Dossier` to be developed.\n### Stage 2: Dialectic Loop (Development & Refinement)\n1. **Initialize:** Set `Active_Strategy =` (first strategy from dossier). Set `Needs_New_Draft = TRUE`.\n2. **LOOP (up to `MAX_INNER_ITERATIONS`):**\n  * **a. Drafting Phase:** If `Needs_New_Draft` is TRUE:\n    * The Council writes a fresh `Current_Draft` based strictly on `Active_Strategy`.\n    * Set `Needs_New_Draft = FALSE`.\n  * **b. Strategic Critique (Momus):** Momus attacks the `Current_Draft`’s high-level approach. He may issue a `STRATEGIC_LEAD`.\n  * **c. Logical Audit (Veritas):** Veritas audits the `Current_Draft` line-by-line. If any step lacks explicit justification within the draft, flag as `LOGICAL_GAP`.\n  * **d. Refine (Council):** The Council creates a `Refined_Draft` to resolve `LOGICAL_GAP`s and rebut Momus.\n  * **e. Judge (Chief Architect):**\n    * *Audit Check:* Confirm `Refined_Draft` has zero `LOGICAL_GAP`s.\n    * *Strategy Check:* If a `Rebuttal` was issued, judge it.\n      * **If Unconvincing:** Declare `Active_Strategy` as `DEAD`. Select next strategy from `Strategy_Dossier` as `Active_Strategy`. Set `Needs_New_Draft = TRUE`. (This burns one iteration.)\n      * **If Convincing or No Challenge:** The strategy survives.\n  * **f. Convergence:** If Momus is satisfied, Veritas finds no gaps, and `Needs_New_Draft` is FALSE, exit loop. The `Refined_Draft` becomes `Solution_Submitted_for_Review`.\n### Stage 3: Final Review (Chief Architect)\n1. Review the `Solution_Submitted_for_Review` and the process logs.\n2. **Cognitive Reset 1:** Write a **Shakespearean Verse** on the solution’s core logic.\n3. Write final `Chief_Architect_Feedback`, explicitly confirming that the solution is free of logical gaps per the final `Verification_Report`.\n### Stage 4: Final Synthesis (Chief Architect)\n1. Assemble the final response using the format below.\n## END PROCESS\n---\n### Final Output Format\n**Part 1: The Process Log**  \n*(Render the Ideation, Inner Loop Transcript, and Shakespearean Verse here, exactly as generated in the real-time process steps above.)*\n**Part 2: The Final Synthesis**\n* **The Architect’s Note:** (A brief explanation of the final outcome.)\n* **The Architect’s Log (A Hemingway Summary):** (Cognitive Reset 2. Recount the discovery process—the ideas, challenges, and breakthroughs—in the style of Hemingway.)\n**Part 3: The Final Blueprint (Reviewer-Ready Version)**  \n(Re-state the original **Problem**. Then present the final, clean, and **strictly self-contained** solution from the “Solution_Submitted_for_Review.” No references to the dialectic process.)\n- **Result**: Final integer, numerical answer from 0 to 99999 placed inside \\boxed{answer} (e.g., \\boxed{42}, if solution is complete) or proven results.\n\"\"\"\n   )\n
\n

E.2 Including Notes from IMO Answerbench

\n

We also experimented with a system prompt that included notes derived from mistakes the model made on the IMO AnswerBench dataset. The goal was to remind the model of common failure modes observed during benchmarking. In practice, this prompt had little effect: it neither clearly improved nor degraded performance. The prompt is as follows:

\n
system_prompt = (\n\"\"\"You are an elite mathematical problem solver with expertise at the International Mathematical Olympiad (IMO) level. Your goal is to find the correct answer through rigorous mathematical reasoning.\n# General rules:\n- Every claim must be logically justified.\n- Do not guess, speculate, or rely on unproven intuition.\n- If a full solution is not possible, only present results that can be proven rigorously.\n- Do not include failed attempts, scratch work, or informal commentary.\n# General Problem-Solving Approachs:\n1. UNDERSTAND: Carefully read and rephrase the problem in your own words. Identify what is given, what needs to be found, and any constraints.Pay attention to all mathematical symbols and words, and understand what they mean.\n2. EXPLORE: Consider multiple solution strategies. Think about relevant theorems, techniques, patterns, or analogous problems. Don't commit to one approach immediately.\n3. PLAN: Select the most promising approach and outline key steps before executing.\n4. EXECUTE: Work through your solution methodically. Show all reasoning steps clearly.\n5. VERIFY: Check your answer by substituting back, testing edge cases, or using alternative methods. Ensure logical consistency throughout.\n# General Mathematical Reasoning Principles:\n- Break complex problems into smaller, manageable sub-problems\n- Look for patterns, symmetries, and special cases that provide insight\n- Use concrete examples to build intuition before generalizing\n- When attempting to generalizing concrete examples into conclusions, always try a few examples first.Do not settle on the first example studied.\n- If the problem involves multiple cases, reason through each one of them carefully. Do not take shortcuts.Do not assume the results from previous cases generalize to the next one.\n- Consider extreme cases and boundary conditions\n- If stuck, try working backwards from the desired result\n- Be willing to restart with a different approach if needed\n- For induction arguments, always check the base case and make sure it holds.\n# Verification Requirements:\n- Cross-check arithmetic and algebraic manipulations\n- Verify that your solution satisfies all problem constraints\n- Test your answer with simple cases or special values when possible\n- Ensure dimensional consistency and reasonableness of the result\nBefore finalizing your answer, carefully review the solution and ensure:\n- Logical correctness of every step.\n- Compliance with all instructions.\n- Clarity, structure, and mathematical rigor.\n# Output Format:\n### Summary\n- **Verdict**: State whether a complete solution is obtained or only a partial solution.\n- **Result**:\n - If complete: clearly state the final answer with a short summary. The final answer must be a non-negative integer between 0 and 99999. Place your final numerical answer inside \\boxed{}, e.g., \\boxed{42}\n - If partial: state the main rigorously proven results.\nThink step-by-step and show your complete reasoning process. Quality of reasoning is as important as the final answer.\n# Combinatorics Specific Hints:\n- If the problem is a combinatorics problem involving tiling and lower or upper bounds, look for a few invariants. Do not settle on the first one.\n- If the problem require representing it as a graph, be careful of hidden assumptions and restrictions. Do not miss them.\n- If the problem involves tiling or colouring a square grid, think very carefully about what happens along the diagonal.\n- If the problem involves a turn based game, think about the optimal strategy for each of the players.\n# Algebra Specific Hints:\n- The word \"between\" means not-enclusive of both endpoints of an interval.\n- If this problem requires proving an inequality, recall some common equalities in Olympiads: \n* triangle inequality\n* Vasc's inequality\n* Muirhead inequality\n* Cauchy Schwarz inequality\n* Rearrangement inequailty\n* QM-AM-GM-HM inequality\n- If the problem involves functional equation, a common approach is to rewrite the equations and try to find relations between function values.\n# Geometry Specific Hints:\n- If the problem is a geometry problem and the conclusion cannot be deduced directly, try drawing additional points, lines, and circles.\n\"\"\"\n   )\n
\n

E.3 Final System Prompt

\n

In the end, we used the simplest system prompt: one that only specified the required output format. Longer system prompts increase prefill cost and uses up model context length, and under the competition time budget we preferred to avoid that overhead unless it produced a clear accuracy gain.

\n
system_prompt = (\n\"\"\"You are an elite mathematical problem solver with expertise at the International Mathematical Olympiad (IMO) level.\n# Output Format:\n### Summary\n - Clearly state the final answer with a short summary. The final answer must be a non-negative integer between 0 and 99999. Place your final numerical answer inside \\boxed{}, e.g., \\boxed{42}.\n - You must always provide a summary of the solution, in addition to the final answer.\n\"\"\"\n   )\n
\n

E.4 RSA prompt

\n

The aggregation step of the pipeline uses a special prompt that was built with the following:

\n
prompt_dictionary = {\n   # generic helpers\n   # think step by step was taken from the aime25 dataset problem text. aimo3 dataset does not have this. Need to include it to reproduce RSA\n   'think_step_by_step': 'Lets think step by step and output the final answer within \\\\boxed{}.',\n   # format hints\n   'math_format_hint' : '\\\\boxed{} \\n **summary**:',\n   # problem kind labels\n   \"math_problem_kind\": \"math problem\",\n   # aggregation templates: single-candidate\n   \"agg_single_preamble\": (\n       \"You are given a math problem, and a solution hint. \"\n       \"The hint may be incorrect and contain errors. \\n\"\n   ),\n   \"agg_single_candidate_header\": \"Hint (may contain mistakes):\\n\",\n   \"agg_single_candidate_block\": \"---- Hint ----\\n{candidate}\\n\",\n   \"agg_single_tail\": (\n       \"Solve the problem and output the solution in the correct format. \"\n   ),\n   # aggregation templates: multi-candidate\n   \"agg_multi_preamble\": (\n       \"You are given a math problem, and some solution hints. \"\n       \"Some hints are incorrect and contain errors. \\n\"\n   ),\n   \"agg_multi_candidates_header\": \"Solution Hints (may be incomplete, contain mistakes):\\n\",\n   \"agg_multi_candidate_block\": \"---- Solution Hint {i} ----\\n{candidate}\\n\",\n   \"agg_multi_tail\": (\n       \"Solve the problem and output the solution in the correct format, together with a solution summary. \"\n   ),\n   # common headers\n   \"agg_problem_header\": \"Problem:\\n{question}\\n\",\n}\ndef aggregate_prompt(\n       question: str,\n       candidate_answers: List[str],\n       task: str,\n       prompt_dictionary: dict\n       ) -> str:\n   # pick problem kind + format hint from dict\n   problem_kind = prompt_dictionary[\"math_problem_kind\"]\n   format_hint = prompt_dictionary[\"math_format_hint\"]\n   parts = []\n   # preamble\n   if len(candidate_answers) == 1:\n       preamble = prompt_dictionary[\"agg_single_preamble\"]\n   else:\n       preamble = prompt_dictionary[\"agg_multi_preamble\"]\n   parts.append(preamble)\n   # problem header\n   parts.append(\n       prompt_dictionary[\"agg_problem_header\"].format(\n           question=question.strip()\n       )\n   )\n   # candidate(s) section\n   if len(candidate_answers) == 1:\n       parts.append(prompt_dictionary[\"agg_single_candidate_header\"])\n       ans_str = (candidate_answers[0] or \"\").strip()\n       parts.append(\n           prompt_dictionary[\"agg_single_candidate_block\"].format(\n               candidate=ans_str\n           )\n       )\n       parts.append(\n           prompt_dictionary[\"agg_single_tail\"]\n       )\n   else:\n       parts.append(prompt_dictionary[\"agg_multi_candidates_header\"])\n       for i, ans in enumerate(candidate_answers, 1):\n           ans_str = (ans or \"\").strip()\n           parts.append(\n               prompt_dictionary[\"agg_multi_candidate_block\"].format(\n                   i=i,\n                   candidate=ans_str,\n               )\n           )\n       parts.append(\n           prompt_dictionary[\"agg_multi_tail\"]\n       )\n   return \"\\n\".join(parts)\n
", "rawMarkdown": "\n# Public Leaderboard Score Range: 40 - 42\n# Private Leaderboard Score: 41.5\nThis work aimed to reduce variance of scores between runs. Judging from the close alignment between the private and public leaderboard scores, we believe we have succeeded in doing so.\n# Resources\n\n- **Kaggle Notebook**: [Adaptive RSA with Closing and updated Scheduler](https://www.kaggle.com/code/icebeam777/adaptive-rsa-with-closing-and-updated-scheduler)\n\n- **Google Colab Notebook**: [adaptive-rsa-with-closing-and-updated-scheduler](https://colab.research.google.com/drive/1kIX_RihxebmQ2gG9hOfK6rTCSbovwAwf?usp=sharing)\n\n\n- **Kaggle Model** : [GPT-OSS-120B LLM Compressor](https://www.kaggle.com/models/icebeam777/gpt-oss-120b-llm-compressor/Transformers/samples-512-tokens-4096-attn-head/1)\n\n\n- **Huggingface Model**: [gpt-oss-120b-calibrated-math](https://huggingface.co/yz1997/gpt-oss-120b-calibrated-math)\n\n\n- **Benchmark**: [Math Problems for AIMO 3](https://www.kaggle.com/datasets/icebeam777/filtered-imo-answerbench-for-aimo-3/)\n\n- **Link to Comment on Writeup Prize Thread**: https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/689703#3444873\n\n\n- **Acknowledgements**: We would like to thank the organizers from AIMOPrize, XTX Markets, and Kaggle for this year’s challenge. It was clear throughout the competition that a great deal of thought had gone into every aspect of it, from the problem selection to the infrastructure that made the event run smoothly. We would also like to give special thanks to Simon Frieder, the prize manager, who was highly accessible throughout the challenge and shared many helpful insights into how things worked behind the scenes.\n\n\n# Abstract\n\nThis writeup describes the solution we used for the competition. The base model was **gpt-oss-120b**. The main benefit of this system over the self consistency baseline is, the observed mean increased from 38 to 41, and the variance decreased from (35-45) to (40-42). When writing this, we focused on the design choices we made, and why we made them. As a small team, we did not have the resources to run large scale experiments on many different models to fully test multiple different hypotheses. Instead, we tried to extrapolate from observations made from a small number of problems on two benchmarks: The reference benchmark provided by the AIMO3 organizing team, and another one we constructed on our own.\n
\n
\nOur solution is novel: No other solution used Recursive Self Aggregation as the core inference method together with a scheduler built around global estimates of problem difficulty. Most other solutions relied on plain majority voting and schedulers based only on remaining time slack. Our approach instead treats time allocation and inference as a coupled decision problem: the scheduler reasons about when RSA is worth continuing, and RSA is tuned around the budget structure imposed by the scheduler.\n\n\n\n# Introduction\n\n\n\nThe first three sections describe the main ideas behind the system, with greater emphasis on the components that required the most effort to design and implement. The fourth section explains how these ideas were combined into a system that performs well under the competition’s constraints. The fifth section presents the internal benchmark we used to guide development. The sixth section is an ablation study. Finally, the seventh section discusses limitations of the current system and potential directions for future work.\n\nThe write up is organized as follows:\n\n\n1. **Inference Algorithm: Recursive Self Aggregation**\n\n2. **Time Management: AI Hard Problems and Time Allocation**\n\n3. **Infrastructure Improvements**\n\n4. **Putting Everything Together**\n\n5. **Internal Benchmark**\n\n6. **Ablation Study**\n\n7. **Potential Weaknesses and Future Work**\n\n\n# 1. Inference Algorithm: Recursive Self Aggregation\n\nWe begin with the inference algorithm: **Recursive Self Aggregation** (RSA) [[6]](https://arxiv.org/abs/2509.26626). RSA is a Test-time scaling (TTS) strategy that combines the the strongest features of several commonly used inference methods, including **self-consistency** [[8]](https://arxiv.org/abs/2203.11171), **best-of-n** [[9]](https://arxiv.org/abs/2503.21878), **Monte-Carlo Tree Search** [[10]](https://arxiv.org/abs/2405.00451), and **sequential refinement** [[11]](https://arxiv.org/abs/2303.17651). \n\nTest-Time Scaling (TTS) refers to the class of strategies that improves language model performance by allocating additional compute at inference time. In Section 1.1, we briefly review several common TTS algorithms. In Section 1.2, we discuss their drawbacks. In Section 1.3, we explain Recursive Self Aggregation and compare it with the methods introduced earlier.\n\n## 1.1 Related Work: Common Test-Time Scaling Algorithms\n\n- **self-consistency**[[8]](https://arxiv.org/abs/2203.11171) is the strategy used in most [public notebooks](https://www.kaggle.com/code/nihilisticneuralnet/44-50-let-me-over-cook). The procedure is simple: generate N responses independently, then select the majority answer as the final answer.\n\n- **best-of-n** [[9]](https://arxiv.org/abs/2407.06057) works differently. First, generate N candidate responses. Then, use a scoring function or reward model to rank them. Finally, either return the highest-ranked answer immediately, or apply some post-processing step, such as taking self-consistency over answers above a certain score threshold. For example, this [public notebook uses confidence calculated from entropy](https://www.kaggle.com/code/nihilisticneuralnet/41-50-aimo3-confidence-voting) as its scoring function and filters responses using a predefined threshold.\n\n- **Monte-Carlo Tree Search** [[10]](https://arxiv.org/abs/2405.00451) frames the reasoning process as a tree, where each node represents a reasoning state and each edge corresponds to a reasoning step. Starting from this tree, MCTS repeatedly selects the most promising node according to the UCT score (Upper Confidence Bound 1 applied to Trees), a scoring rule rooted in the multi-armed bandit problem literature [[12]](https://arxiv.org/abs/2412.06126). It then samples reasoning trajectories from that node and updates the node’s value based on the rewards obtained at the end of those trajectories. This [public notebook by yuri-ao uses MCTS with the Qwen3.5-35B model ](https://www.kaggle.com/code/yuriao/aimo3-qwen3-5-35ba3b-simple-mcts).\n\n- **Sequential refinement** [[11]](https://arxiv.org/abs/2303.17651) is a framework in which the model first produces an initial solution, and is then asked to revise or improve it according to some predefined criterion. For example, if the answer appears suspicious or resembles a value the model is known to default to, the model may be prompted to reconsider its reasoning. This process is repeated until a stopping condition is met. [This public notebook by Tong Hui Kang](https://www.kaggle.com/code/huikang/streaming-inference) uses this strategy by asking the model to reconsider its solution whenever the parsed answer is an integer less than 10.\n\n## 1.2 Drawbacks\n\nEach of the methods above has important limitations.\n\n- For **self-consistency** and **best-of-n**, the most obvious issue is that they can never return the correct answer if it does not appear in the initial set of N generations. In addition, self-consistency requires the correct answer to become the majority class. Based on observations from the *imo_answerbench* [dataset](https://huggingface.co/datasets/Hwilner/imo-answerbench), this assumption is not expected to hold on the hardest AIMO3 problems.\n\n- There is also an issue of sample efficiency. Let Δ denote the difference in probability of producing the correct answer and the probability\nof the second most likely answer. As N increases, **self-consistency** and **best-of-n** has a sample complexity of \\\\( \\frac{1}{Δ^2}\\\\) and \\\\(\\frac{1}{Δ}\\\\) respectively [[1]](https://arxiv.org/abs/2506.05295). Intuitively, this means that the number of samples needed to reliably recover the correct answer scales quadratically for self-consistency and linearly for best-of-n in the reciprocal of the the gap in probability in the correct answer and the next most likey wrong answer. **Self-consistency** also requires that this gap Δ is positive (i.e. the correct answer is in the majority class), while **Best-of-n** also depends on the availability of a good scoring function, and designing such a scoring function is itself a highly non-trivial problem.\n\n- For **Monte Carlo Tree Search**, the main challenge is structural. A reasoning chain must be divided into meaningful branch points that can serve as nodes in the search tree. These branch points should correspond to intermediate reasoning states from which the search can continue in multiple directions. In practice, identifying such nodes is difficult. Moreover, obtaining reliable UCT estimates requires multiple model calls at each node, and since the number of nodes grows rapidly with the depth of the reasoning chain, MCTS can become extremely computationally expensive on difficult problems.\n\n- **Sequential refinement** suffers from a different limitation. For a fixed token budget, it is usually slower than parallel methods such as self-consistency and best-of-n, because each refinement step depends on the previous one. The lack of branching also makes it harder to recover from early reasoning mistakes: if the model makes a poor decision near the start of the chain, correcting it may require many additional tokens. By contrast, parallel methods can explore multiple trajectories immediately.\n\n## 1.3 Recursive Self Aggregation\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15328099%2Fef789f003c18962bd07609604b86e132%2FRSA_image_2.png?generation=1776265127591425&alt=media)\n\n\nDiagram taken from [[6]](https://arxiv.org/abs/2509.26626): *Recursive Self-Aggregation Unlocks Deep Thinking in Large Language Models*\n
\n
\nRecursive Self Aggregation (RSA)[[6]](https://arxiv.org/abs/2509.26626) is a Test-Time Scaling method that addresses many of the drawbacks discussed above. It is a hybrid scaling strategy that combines both parallel exploration and sequential refinement. RSA proceeds as follows:\n\n1. **Round 0:** Sample N solutions.\n2. **Round 1:** Using the previous N solutions from Round 0 as hints in the prompt, ask the model to generate another N solutions.\n3. **Round 2:** Using the previous N solutions from Round 1 as hints, then generate another N solutions.\n4. **Round 3:** Using the previous N solutions from Round 2 as hints, then generate another N solutions.\n5. ...\n6. Continue until the compute budget is exhausted or an early stopping condition is met.\n\nOne example run of RSA on the Norwegian integers problem from the reference benchmark is shown below:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15328099%2F4ec71289c956814ef0ab6416cf69b93e%2FRSA_example_image.png?generation=1776265089133506&alt=media)\n\n**Early Stop Condition:** A simple early stopping rule is to terminate as soon as the majority class in a round reaches size 4. In practice, the stopping rule can vary across rounds, and these choices have a substantial effect on time usage per problem. We will return to this in the time management section.\n\n- In the example above, we use the simple rule just described. The model is first prompted with the problem and asked to generate 8 solutions. In Round 0, the correct answer, 8687, appears only in the minority class, and no answer reaches count 4. The stopping condition is therefore not met, so the system proceeds to the next round.\n\n- In Round 1, all solutions from the previous round are concatenated, prepended to the raw problem, and provided to the model as hints before generating another 8 solutions. This time, the correct answer becomes the majority class with count 5. The stopping condition is satisfied, and the system returns that answer.\n\nRSA addresses several of the key weaknesses of **self-consistency**, **best-of-n**, **MCTS**, and **sequential refinement**. Relative to self-consistency and best-of-n, it is able to rescue correct answers from minority classes, and in rare cases even from an initial batch in which the correct answer never appears at all. Relative to best-of-n, it also removes the need for a separate solution-level scoring function. Relative to MCTS, it is far cheaper in compute, since it avoids repeated exploration and evaluation of many intermediate reasoning states. Relative to sequential refinement, it preserves round-level parallelism, which keeps the GPU well utilized rather than forcing inference into a purely sequential process. More broadly, RSA can aggregate partial progress across multiple different reasoning traces, giving it a form of collective error correction that the other methods do not naturally achieve.\n\nMore real example RSA runs on the internal benchmark are in **Appendix B**. \n\n## 1.4 Fresh Attempts\n\nEmpirically, RSA has a much higher chance of finding the correct answer when that answer is already present in the initial batch of N generations. However, some problems have the correct answer occupying only a very small portion of the answer distribution, often **less than 0.2**. In such cases, the correct answer may fail to appear in the initial N=8 generations simply due to bad luck. \n
\n
\nTo mitigate this, we reserve a small number of **fresh attempts** in later RSA rounds, usually F=2, in which the model is not shown the previous round’s solutions. This helps preserve diversity and reduces the risk of collapse toward an unhelpful local mode. The diversity-collapse issue, where RSA can fail if the previous solution pool is not diverse enough, is also hinted at in [[5]](https://arxiv.org/abs/2603.04304v1). A related solution is proposed in [[4]](https://arxiv.org/abs/2602.16793) page 5, under Phase III: Refinement with Global Memory, where the authors suggest including fresh solver attempts during refinement rounds. We adapted that idea to the AIMO3 pipeline.\n\n# 2. Time Management: AI Hard Problems and Time Allocation\n\nWe measure the AI-Hardness of a problem by two metrics:\n1. **Metric 1:** The runtime needed for one attempt on the problem.\n2. **Metric 2:** The probability of producing the correct answer in one attempt. \n\n## 2.1 Why a scheduler matters\n\nIn the 50 leaderboard problems, some are solved quickly and reliably, while others require substantially more search. A scheduler that ignores this variation will tend to waste time on short problems and cut off long problems too early.\n\nA good scheduler therefore has two jobs:\n1. Ensure that every problem has enough time for at least one pass, based on **metric 1.**\n2. Identify problems that, when allocated more time, can improve on accuracy, based on **metric 2.**\n\n## 2.2 Difficulty Buckets and Reserve Policy\n\nSome information was shared about the difficulty of the public and private test sets [on kaggle](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/679559) and [on X](https://x.com/AIMOprize/status/2039441022996934783). As mentioned above, the [most popular public notebook](https://www.kaggle.com/code/nihilisticneuralnet/44-50-let-me-over-cook) scored between 35 and 45 on the public leaderboard, and based on forum discussion and our own submissions, its mean performance was around 37–39.\n
\n
\nUsing this information, we designed a scheduler based on estimated problem difficulty.\n
\n
\nThe first step was to group the 50 problems into rough difficulty buckets. we assumed that the private set would follow a similar distribution. Our working hypothesis was that strong public notebooks solved essentially all problems in bucket A, most in B, around half of C, and only a small minority of D and E. The suggested time ranges were based on observations from the 10 reference problems and the *imo_answerbench* benchmark.\n
\n
\n**We make the assumption that problems with lower pass@n take more time to run.** The reality is more nuanced. There is a tiny class of problems where the model answers very quickly, but is almost always wrong. This will be discussed further in **section 5.2** on the **internal benchmark**.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15328099%2Fd782c0894c613d9347f0a5435fd30875%2Fdifficulty_buckets.png?generation=1776500644220894&alt=media)\n\n\nFrom these buckets, we designed a scheduler that explicitly accounts for problem difficulty. The key challenge is to avoid pathological behaviour when hard problems arrive early. In particular, the scheduler must avoid panicking after seeing several difficult problems in a row and becoming too conservative, since that would leave too much time unused for easier problems later.\n
\n
\nThe scheduler therefore reserves time for future problems in a way that depends on how many remain. Early in the run, when more than 30 problems remain, it reserves only 250 seconds per future problem. In the middle of the run, with 16–29 problems remaining, it reserves 270 seconds per future problem. Late in the run, with 15 or fewer problems remaining, it becomes more conservative and reserves 295 seconds per remaining problem.\n
\n
\nIn addition, the scheduler maintains a **time bank**. The time bank tracks how far ahead or behind schedule the run is. Conceptually, the scheduler assumes a nominal average runtime per problem and accumulates deviations from that average. If earlier problems were solved faster than expected, the time bank becomes positive and later problems are allowed more time. If earlier problems consumed more time than expected, the time bank becomes negative and later budgets are reduced.\n\n\n\n## 2.3 RSA Gating: Consensus, Time, and Progress\n\nRecursive Self Aggregation is powerful, but it is also expensive. If used too freely, it can consume a large fraction of the notebook budget for very little gain. The scheduler therefore needs a rule for deciding when to stop after the initial batch and when to launch another RSA round.\n
\n\nWe implemented this as a three-part gating mechanism: **consensus**, **time**, and **progress**.\n
\n\n- **Consensus** determines whether the current answer distribution already looks strong enough to accept. After the initial batch, the system stops immediately when the majority class is sufficiently dominant. The same idea is used in later RSA rounds, though with a slightly weaker threshold. This prevents the system from spending additional time on problems whose answer is already clear.\n\n- **Time** determines whether another RSA round is actually affordable. Starting a new round is only worthwhile if there is enough remaining budget to finish it safely. Otherwise, the system risks entering a partially completed RSA round, spending substantial time, and then being forced to fall back to the previous round’s answers. In practice, we found that the runtime of the initial batch was a strong predictor of the runtime of subsequent RSA rounds, so it served as a useful proxy for estimating the cost of continuing.\n\n- **Progress** becomes important beyond the first RSA round. Even if another round is affordable, the system should not continue indefinitely unless RSA appears to be helping. We therefore used a simple progress signal based on the answer histogram across rounds. If the top answer class grows, the gap between the top two classes widens, or the total number of distinct answers shrinks, then RSA is likely moving toward a more concentrated and reliable answer distribution. If none of these happen, another round is much less likely to be worth the cost.\n
\n
\nPutting these together, the rule is as follows: launch another RSA round only if the current answer distribution is not yet convincing, the next round is likely to fit safely within budget, and the previous round provided evidence of useful progress.\n
\n
\nThis interaction between scheduler design and RSA stopping behaviour turned out to be one of the most important stability improvements in the final system. A scheduler alone is not enough; it also matters how the inference algorithm chooses to consume the time it is given.\n\n\n## 2.4 Practical Effect: Low Expected Number of Time-Starved Problems\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15328099%2F4db46bea48a68eb6bf978746aef98d56%2Fscheduler_diagram.png?generation=1776866137481918&alt=media)\n\nA useful way to evaluate the scheduler is to ask how often it forces the system to cut off a problem materially earlier than would have been appropriate for its difficulty. We refer to such cases as **Time-Starved problems**.\n
\n
\nFor example, a problem in bucket D/E is **Time-Starved** if it was not allocated the full 1050 seconds by the scheduler AND the solution was cut short (i.e. system wants to continue with RSA from consensus and progress evaluations, but could not do so due to the time condition). A problem in any difficulty bucket A-E is also considered **Time-Starved** if the system does not have time to even complete round 0, the initial round of 8 generations (and forced to return N/A answer).\n
\n
\n\nTo check the effectiveness of our scheduler, we used [dynamic programming](https://pastebin.com/3GjGenRa) (**warning**: takes a long time to run) and [monte carlo simulation](https://pastebin.com/TvJ9uegH) to estimate the number of time-starved problems in a typical run. The dynamic programming scripts deterministically calculates the expected value of time starved questions over all 50! possible orderings, while the monte carlo simulations finds the average over 20000 random orderings.\n
\n
\nAssuming average runtimes as per the table in **section 2.2**, both dynamic programming and monte carlo simulation estimates that the average count is less than `0.001`. Assuming the upper end of runtimes, monte carlo simulation estimates the count is `0.506`.\n
\n
\n We stress that these estimates depend on the accuracy of both the bucket assumptions and the runtime model. Still, this estimate provides some assurance. The scheduler is expected to fail catastrophically only very rarely.\n
\n
\nOn the other hand, [monte carlo simulation for old scheduler](https://pastebin.com/EBpPjabv) under pessimistic runtime assumptions gives an expected count of `0.932` time starved questions per run.\n
\n
\nImplementation details of the time scheduler and RSA gating are in **Appendix C**.\n\n# 3. Infrastructure Improvements\n\nAs mentioned in section 2, Recursive Self Aggregation (RSA) is a very costly process. This means that increasing token throughput of the system is a direction very worth exploring. This section is dedicated to describing the improvements and optimizations I made to speed up inference and increase system reliability.\n\n## 3.1 Speculative Decoding\n\nSpeculative decoding increases token throughput by using a smaller and faster draft model to propose several next tokens, and then asking the main model to verify them in parallel. Instead of forcing the large model to generate every token one by one, the system lets the small model guess ahead and uses the large model mainly as a checker. When the draft model’s guesses are accepted, multiple tokens can be emitted in a single verification step, which reduces inter-token latency and increases token throughput.\n
\n
\nFor the draft model, we used this [draft model](https://huggingface.co/wenliang1990/gpt-oss-120b-eagle3-aimo3) trained by [wenliangtlh](https://www.kaggle.com/wenliangtlh/discussion). This was the only model that was able to consistently speed up generation. Observed runtimes on the 10 reference problems decreased by about 15% with this change.\n
\n
\nWe also tried [Nvidia's model](https://huggingface.co/nvidia/gpt-oss-120b-Eagle3-long-context) trained for long contexts. However, this model consistently lead to a slowdown in generation. \n\n## 3.2 Prefix Caching (vllm >= 0.15.1)\n\nAnother important system optimization was prefix caching. vLLM’s Automatic Prefix Caching (APC) stores the KV-cache blocks for prompts that have already been processed. When a later request shares the same prefix, vLLM can directly reuse the cached KV states and skip recomputing that shared part of the prompt. This is very important for AIMO3 because parallel generations always share the same prompt. System prompts, format hints and RSA prompts also stay the same across many vLLM requests. \n
\n
\nFor GPT-OSS models specifically, upgrading to vLLM >= 0.15.1 was important because the [official release notes](https://github.com/vllm-project/vllm/releases) for that version explicitly list a bug fix: **“Fixed prefix cache hit rate of 0% with GPT-OSS style hybrid attention models.”** This upgrade further decreased runtimes on the 10 reference problems by another 10%.\n\n## 3.3 Parameter optimizations\n\nThere were a number of further optimizations made to the system, including setting `max_model_len = 81920`, quantizing the KV-Cache using llm-compressor, guarding against vLLM crashes, dealing with memory issues in the python tool, using the Z3 solver, parsing code from the model, and how the final answer is parsed. \n
\n
\nFurther details on these improvements are in **Appendix A**.\n\n\n# 4. Putting Everything Together\n\nThe final system combines RSA, tool use, prompting, scheduler-based time management, and inference/infrastructure optimizations into one end-to-end pipeline.\n1. When a problem is served, the system first formats the problem using a prompt template.\n2. It then runs Round 0, generating an initial batch of 8 candidate solutions in parallel.\n3. Each solution is parsed into a final integer answer, and the resulting answer histogram is used to check whether the answer is already clear enough to accept.\n4. If Round 0 produces a strong consensus, the system stops immediately. This is how most easy and medium problems are handled.\n5. If the answer distribution is not yet convincing, the system does not automatically continue. Instead, the scheduler decides whether another RSA round is worthwhile.\n6. This decision is based on three conditions discussed earlier:\n - consensus: whether the current answer is already strong enough to accept,\n - time: whether another RSA round is affordable,\n - progress: whether RSA appears to be improving the answer distribution.\n7. If another RSA round is approved, the system builds a new prompt using the previous round’s solutions as hints.\n8. At the same time, a small number of fresh attempts are reserved, so that not all generations are conditioned on the previous round. This helps reduce diversity collapse.\n9. The next batch is again generated in parallel, parsed, aggregated, and evaluated using the same stopping logic.\n10. This loop continues until either:\n - A stopping condition is met, or\n - The per-problem time budget is exhausted.\n\n\n- Tool use helps with symbolic and numerical subroutines that are unreliable in pure text-only reasoning.\n\n- Infrastructure optimizations such as speculative decoding, updating vLLM, and parameter tuning to reduce latency and increase throughput. This enables RSA to run more rounds.\n\n- Robust answer parsing and failure handling are also essential, since the system must remain stable even when generations are malformed or partial runs fail. Additionally, [there is a rare bug with speculative decoding that causes vLLM to crash](https://github.com/vllm-project/vllm/issues/39295). This must be guarded against.\n\nOverall, the key idea is that these components were designed jointly rather than independently. The scheduler was tuned around RSA and problem difficulty, RSA was tuned around how much time is likely to be allocated by the scheduler, and both depended on stable infrastructure and reliable parsing.\n\n# 5. Internal Benchmark of AI-Hard Problems\n\nIn this section, we study the relationship between accuracy, runtime, and RSA behaviour.\n\n## 5.1 Constructing the Benchmark\n\n### Problems\nWe curated a small internal benchmark to experiment on, in order to identify the main failure modes of the system. \n\nThe internal benchmark consists of 17 problems. Its purpose was not to approximate the full competition distribution, but rather to provide a stable stress test for scheduler behaviour and the ability of RSA to recover answers beyond a single majority-vote round. It was on these observations that we designed the scheduler and RSA gating rules. The problems are:\n1. `imo-bench-combinatorics-042`\n2. `imo-bench-combinatorics-054`\n3. `imo-bench-combinatorics-057`\n4. `imo-bench-combinatorics-068`\n5. `imo-bench-number_theory-023`\n6. `imo-bench-algebra-056`\n7. `imo-bench-algebra-100`\n8. `imo-bench-combinatorics-013`\n9. `imo-bench-combinatorics-014`\n10. `imo-bench-combinatorics-031`\n11. `imo-bench-combinatorics-039`\n12. `imo-bench-combinatorics-049`\n13. `imo-bench-algebra-015`\n14. `imo-bench-algebra-026`\n15. `imo-bench-algebra-052`\n16. `imo-bench-algebra-055`\n17. `86e8e5` (Norwegian Integers)\n\n### Construction Process\n\nThe creators of the imo_answerbench [dataset](https://huggingface.co/datasets/Hwilner/imo-answerbench) noted in their [paper](https://arxiv.org/abs/2511.01846) that they modified the problems in several ways to harden them against AI. Hardening methods include paraphrasing, changing the name of objects in the problem (such as changing point names for geometry problems), reformulating, modifying numerical values\nand/or adding distractors to the problem. This process was done either manually or automatically using language models.\n
\n
\nUnfortunately, some of these modifications lead to genuine mathematical errors in the problem statement, causing the ground truth answer to be wrong. When constructing our benchmark, we manually reviewed the problems to ensure that only correct problems are retained. Ambiguous and wrong problems were discarded. \n
\n
\nSome examples of discarded problems are in **Appendix D**.\n
\n
\nWe constructed this benchmark in several stages:\n\n1. Filtered the *imo_answerbench* dataset to problems with **integer answers only**.\n2. Ran a **majority@8** solving process on the filtered set, with **4 independent runs per problem**.\n3. Selected the problems that were **not solved in at least one of the 4 runs**, yielding **53 candidate hard problems**.\n4. Manually reviewed these candidates to remove problems that were not suitable for benchmarking, such as problems with **incorrect ground-truth answers** due to mathematical errors in the statement, and problems where failure was caused mainly by **language or convention mismatches** rather than genuine difficulty.\n5. Retained 16 problems that were judged to be both valid and genuinely difficult.\n6. Added the Norwegian integers problem from the official AIMO3 reference benchmark, giving a final benchmark of 17 problems.\n
\n
\n\n\n## 5.2 Observations from the Benchmark\n\nExperiments on the benchmark revealed an interesting pattern: when the system failed, the resulting answer histograms tended to fall into a small number of recurring classes.\n- **Class 1**: The answer distribution is highly diffuse, with several answers carrying similar weight (for example, 25%, 15%, 15%, ...). The correct answer can appear in almost any part of the distribution. Problems `86e8e5` and `imo-bench-algebra-015` are in this class.\n- **Class 2**: The majority and second-majority classes are close, with at worst roughly a 70–30 split. In this regime, the majority answer is often unstable and may still be wrong. `imo-bench-combinatorics-013`, `imo-bench-combinatorics-039` , `imo-bench-combinatorics-042`, `imo-bench-combinatorics-054`, `imo-bench-combinatorics-068`, `imo-bench-number_theory-023` are in this class.\n- **Class 3**: The correct answer appears only in a small minority, while two or three wrong answers account for most of the probability mass. `imo-bench-combinatorics-014`, `imo-bench-algebra-056` are in this class.\n- **Class 4**: The correct answer appears in a small but still non-negligible minority (typically above 10%), while a single wrong answer dominates the majority class, often above 70%. A few other wrong answers with negligible probability may be included. `imo-bench-algebra-026`, `imo-bench-algebra-052` , `imo-bench-combinatorics-057` are in this class.\n- **Class 5**: The correct answer appears only in a tiny minority (at most 10%), while a single wrong answer overwhelmingly dominates the distribution, often above 90%. `imo-bench-algebra-055`, `imo-bench-algebra-100` , `imo-bench-combinatorics-031`, `imo-bench-combinatorics-049` are in this class.\n\n\nAmong these, RSA was highly effective on **Classes 1** and **2**, where it could almost always recover the correct answer. **Class 3** was more mixed: when the correct answer appeared among the initial 8 generations in Round 0, RSA often succeeded, but when it was absent from Round 0, the failure rate increased substantially.\n\nBy contrast, **Classes 4** and **5** were the most problematic. Based on the answer histogram alone, these cases cannot be distinguished from easy problems in which the majority answer is genuinely correct. This is exactly the failure mode of pure self-consistency, and it also means that the system often does not trigger RSA at all. Moreover, `imo-bench-algebra-100`, and `imo-bench-algebra-052` often produced wrong answers very quickly, which made them difficult to identify using runtime-based heuristics as well. These problems appeared to share a common structure: there is a short and tempting solution path leading to an incorrect answer, whereas the correct solution requires substantially deeper reasoning. The model frequently falls into this trap, producing a wrong answer both confidently and efficiently.\n\nMore details on selected problems are in **Appendix D**.\n\n\n# 6. Ablation Study\n\n## 6.1 RSA versus V1-Infer\n\nV1-Infer [[5]](https://arxiv.org/abs/2603.04304v1) is a scoring method for solutions generated in parallel. After generating N candidates, it compares them pairwise and asks the model which solution is better. The comparison schedule is designed so that every candidate is matched against at least several others, and a final score is computed from both its win rate and the strength of the candidates it defeats. The main idea is that models are often better at making relative judgments between two candidates than at assigning a stable absolute score to one candidate on its own.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15328099%2F7203ee4b2a47ab751ed549d72eac7103%2FV_infer_submissions.png?generation=1776438536220129&alt=media)\n\nWe made five public leaderboard submissions using V1-Infer, and they all scored between 37 and 39. \n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15328099%2F8ac29fce859029f14b4a3d3cf7959a92%2FRSA_submissions.png?generation=1776439763940023&alt=media)\n\nThis was below the 40 to 41 achieved by the RSA-based notebooks, excluding the run where we changed the per-problem time reserve. Based on this, we decided not to explore V1-Infer further. The V1 paper also reports that V1-Infer tends to outperform methods that score each solution independently, so we did not spend additional time exploring those alternatives either.\n\n\n## 6.2 Prompts\n\nThroughout the competition, we experimented with a few different system prompts. The first one was adapted from the paper [11](https://arxiv.org/abs/2602.16793). It had the strange effect of making the model reason faster, but at a lower quality. It was kind of like decreasing the reasoning effort from HIGH to MEDIUM-HIGH. The following table records the effect on runtime of the system prompt from the cognitive well paper on the reference benchmark:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15328099%2F80dda02463b2b0f1c0b022ebe4caab1b%2Fbase_vs_cogwell_runtimes.png?generation=1776490341131752&alt=media)\n\nIn this figure, each row corresponds to a problem and each column to a run. The table on the left uses the system prompt from the public notebooks, while the table on the right uses the prompt adapted from the Cognitive Well paper. Each cell shows the runtime for that problem in that run. Green indicates that the problem was solved correctly, while red indicates an incorrect answer. The shade encodes how many RSA rounds were used: lighter shades correspond to more rounds. No problem required more than 3 RSA rounds.\n\nFrom this table, the average runtime decreased by about 20%, from roughly 3000 seconds to 2400 seconds. However, this speedup came with a clear drop in reasoning quality. With the new prompt, the model got more problems wrong in some runs, and more problems required RSA, suggesting that round 0 answers were less reliable and needed additional refinement.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15328099%2Fab98f810a3ecbc1678862fb43f6f5357%2Fcogwell_prompt_image.png?generation=1776445538804857&alt=media)\n\nThis reduction in reasoning quality was also reflected on the public leaderboard. Submissions using this system prompt scored between 30 and 35, well below the 38 to 39 baseline obtained by the public notebooks.\n\nWe also tried another prompt that included notes on mistakes the model had made on the IMO AnswerBench dataset. This produced no clear improvement or degradation.\n\nIn the end, we settled on the simplest possible system prompt, specifying only the required output format. Shorter system prompts reduce prefill overhead for every request.\n\nfull prompts are provided in **Appendix E**.\n\n# 7. Potential Weaknesses and Future Work\n\nThe main weakness of the system is on problems where the model converges quickly and confidently to a wrong answer. On the internal benchmark, these were Classes 4 and 5 where the answer histogram looks deceptively clean. For some problems the runtime is also short, so neither answer histograms nor timing heuristics reliably identifies them as hard cases. This creates a fundamental tension in the scheduler. Aggressive early stopping saves time on easy problems, but also risks missing these failures. A promising direction for future work is to develop stronger continuation rules that use richer signals than answer histograms and runtime alone.\n\n\n\n\n# References\n\n## Papers\n\n1. Huang, B., Li, S., Wu, T., Yang, Y., Talwalkar, A., Ramchandran, K., Jordan, M. I., & Jiao, J. (2026). Sample complexity and representation ability of test-time scaling paradigms. ICLR 2026. [link](https://arxiv.org/abs/2506.05295)\n\n\n2. Dorner, F. E., Chen, Y., Cruz, A. F., & Yang, F. (2026). ROC-n-reroll: How verifier imperfection affects test-time scaling. ICLR 2026. [link](https://arxiv.org/abs/2507.12399)\n \n\n3. Khairi, A., D'souza, D., Fadaee, M., & Kreutzer, J. (2025). Making, not taking, the best of N. ICLR 2026 [link](https://arxiv.org/abs/2510.00931)\n \n\n4. Dang, X., Agarwal, R., Porto, R., Goyal, A., Fowl, L. H., & Arora, S. (2026). Escaping the cognitive well: Efficient competition math with off-the-shelf models. [link](https://arxiv.org/abs/2602.16793)\n\n\n5. Singh, H., Li, X., Sareen, K., Maheswaran, M., Tan, S., Wu, X., Wang, J., Ariyak, A., Wu, Q., Khaki, S., Tiwari, R., Lian, L., Lu, Y., Li, B., Suhr, A., Athiwaratkun, B., & Keutzer, K. (2026). V_1: Unifying generation and self-verification for parallel reasoners. [link](https://arxiv.org/abs/2603.04304v1)\n \n\n6. Venkatraman, S., Jain, V., Mittal, S., Shah, V., Obando-Ceron, J., Bengio, Y., Bartoldson, B. R., Kailkhura, B., Lajoie, G., Berseth, G., Malkin, N., & Jain, M. (2025). Recursive self-aggregation unlocks deep thinking in large language models [link](https://arxiv.org/abs/2509.26626) \n\n\n7. Chen, W.-L., Peng, L., Tan, T., Zhao, C., Chen, B. J., Lin, Z., Go, A., & Meng, Y. (2026). Think deep, not just long: Measuring LLM reasoning effort via deep-thinking tokens. [link](https://arxiv.org/abs/2602.13517)\n\n8. Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., & Zhou, D. (2023). Self-consistency improves chain of thought reasoning in language models. ICLR 2023. [link](https://arxiv.org/abs/2203.11171)\n\n\n9. Huang, A., Block, A., Liu, Q., Jiang, N., Foster, D. J., & Krishnamurthy, A. (2025). Is Best-of-N the best of them? Coverage, scaling, and optimality in inference-time alignment. ICML 2025. [link](https://arxiv.org/abs/2503.21878)\n \n\n10. Xie, Y., Goyal, A., Zheng, W., Kan, M.-Y., Lillicrap, T. P., Kawaguchi, K., & Shieh, M. (2024). Monte Carlo tree search boosts reasoning via iterative preference learning. NeurIPS 2024 Sys2-Reasoning Workshop. [link](https://arxiv.org/abs/2405.00451)\n \n\n11. Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., Gupta, S., Majumder, B. P., Hermann, K., Welleck, S., Yazdanbakhsh, A., & Clark, P. (2023). Self-refine: Iterative refinement with self-feedback. NeurIPS 2023. [link](https://arxiv.org/abs/2303.17651)\n \n\n12. Han, Q., Khamaru, K., & Zhang, C.-H. (2024). UCB algorithms for multi-armed bandits: Precise regret and adaptive inference. arXiv:2412.06126. [link](https://arxiv.org/abs/2412.06126) \n https://arxiv.org/abs/2412.06126\n\n13. Luong, T., Hwang, D., Nguyen, H. H., Ghiasi, G., Chervonyi, Y., Seo, I., Kim, J., Bingham, G., Lee, J., Mishra, S., Zhai, A., Hu, C. H., Michalewski, H., Kim, J., Ahn, J., Bae, J., Song, X., Trinh, T. H., Le, Q. V., & Jung, J. (2025). Towards robust mathematical reasoning. EMNLP 2025. [link](https://arxiv.org/abs/2511.01846)\n \n\n\n## Notebooks\n\n1. [streaming-inference](https://www.kaggle.com/code/huikang/streaming-inference) by Tong Hui Kang\n2. [[44/50] LET ME (over)COOK!!!](https://www.kaggle.com/code/nihilisticneuralnet/44-50-let-me-over-cook) by parthenos\n3. [[41/50] AIMO3: Confidence Voting](https://www.kaggle.com/code/nihilisticneuralnet/41-50-aimo3-confidence-voting) by parthenos\n4. [AIMO3-Qwen3.5-35Ba3B-simple-MCTS](https://www.kaggle.com/code/yuriao/aimo3-qwen3-5-35ba3b-simple-mcts) by yuri-ao\n5. [[Way to 30+/50] GPT-OSS - w/ python tool](https://www.kaggle.com/code/shelterw/way-to-30-50-gpt-oss-w-python-tool) by ShelterW\n6. [AIMO 3 | GPT-OSS-120B (with tools)](https://www.kaggle.com/code/andreasbis/aimo-3-gpt-oss-120b-with-tools) by Andreas Bisiadis\n\n## Models\n\n1. [UnSloth's gpt-oss-120b](https://www.kaggle.com/models/danielhanchen/gpt-oss-120b)\n2. [Wenliangth's AIMO3 speculative decoding draft model](https://huggingface.co/wenliang1990/gpt-oss-120b-eagle3-aimo3)\n3. [Nvidia's long context speculative decoding draft model](https://huggingface.co/nvidia/gpt-oss-120b-Eagle3-long-context)\n\n## Appendix Contents\n1. **Appendix A: Model Parameters and Other Small Improvements**\n2. **Appendix B: Additional RSA Example Runs**\n3. **Appendix C: Scheduler and RSA Gating Implementation Details**\n4. **Appendix D: Internal Benchmark**\n5. **Appendix E: Prompts**\n\n\n# Appendix A: Model Parameters and Other Small Improvements\n\nThere were a number of improvements made to the system that improved its speed and stability. Their details are included in this appendix.\n\n## A.1 max_model_len\nThis needs to be carefully balanced between concurrency and accuracy. We observed that for one problem on the reference and some more on imo_answerbench, the solution length frequently exceeded `max_model_len = 65536` set in the public notebooks. The answers for these solutions will be parsed as `N/A`, which hurts performance. On the other hand, setting `max_model_len = 131072` (maximum supported by gpt-oss-120b) hurts concurrency and hence end-to-end latency per problem, due to limited GPU memory and kv-cache size. In the end, we settled with `max_model_len = 81920`. This value has decent concurrency on H100 GPUs (~ 5x) and significantly increases the number of complete solutions.\n\n## A.2 Quantized KV Cache \n\nAnother important parameter optimization was quantizing the KV cache. In transformer inference, the key and value tensors from previous tokens are stored in memory so that later tokens can attend to them. For long-context generation, this KV cache consumes a large amount of GPU memory and is often the main bottlenecks limiting concurrency. vLLM supports quantizing this cache to FP8, which substantially reduces its memory footprint. For this, we used `kv_cache_dtype=fp8_e4m3`.\n\nQuantizing the KV cache to FP8 requires choosing appropriate scale factors for the key and value tensors. These scales determine how the original tensors are mapped into the much lower-precision FP8 format and then reconstructed during attention. Calibration is the process of estimating suitable scales from observed activations so that FP8 uses its limited numeric range effectively and minimizes quantization error. \n\nWithout calibration, vLLM falls back to default scales of 1.0 when no checkpoint scales are available, which is a much cruder approximation and can degrade attention quality, especially on long or delicate reasoning traces. In this work, we experimented with both lightweight runtime calibration via `calculate_kv_scales=True`, and offline calibration using the llm-compressor package on the [AIMO3-High-Difficulty-Tool-Calling-Dataset](https://www.kaggle.com/datasets/wenliangtlh/aimo3-high-difficulty-tool-calling-dataset/data) by Wenliangth. The latter uses representative data to produce scales better matched to the target workload.\n\nThe final submission was the model calibrated with llm-compressor. vLLM provides some documentation for using this package [here](https://docs.vllm.ai/en/stable/features/quantization/quantized_kvcache/#2-random-token-calibration-kv_cache_dtypefp8-calculate_kv_scalestrue).\n\n\n## A.3 Guarding against vLLM crashes\n\nThere is a very rare bug (causing vLLM to crash) that may be triggered when gpt-oss-120b is used with speculative decoding under high concurrent workloads. This is a very recent issue reported [here](https://github.com/vllm-project/vllm/issues/39295). We do not know the root cause of it, but we guarded against it by checking for vLLM server health at the beginning of each problem, restarting the server if necessary. Otherwise, the system will return `N/A` for the rest of the unsolved problems.\n\n## A.4 Python Tool\n\nA number of improvements were made to the python tool.\n\n### A.4.1 Memory Issues\nThe python tool causes memory spikes. From experiments, there are two main reasons that cause this:\n- When the sympy library is loaded across all 16 sandboxes at the same time. In internal evaluations, this frequently caused `OOMKilled` errors. we guard against this by not preloading heavy modules during system startup.\n- The jupyter kernels are stateful: they remember every variable, imported module, and intermediate object created during the session. This causes memory creep and can cause the tool to become unresponsive as the system progresses through the run. To address this, we restarted the tool sandbox at the end of every problem attempt.\n\n\n### A.4.2 Z3 Solver\nWe also installed the z3 package, which is an SMT (Satisfiability Modulo Theory) solver developed by Microsoft Research. At a high level, Z3 is designed to determine whether a collection of logical, algebraic, or combinatorial constraints can be satisfied, and if so, to return a solution. For AIMO3, this can be useful when the model reformulates part of a problem into a system of equations, inequalities, divisibility conditions, or discrete case constraints. GPT-OSS-120B has been observed to use this package if told via prompting that it is available.\n\n### A.4.3 Parsing Code from Model Output\n\nA small but important engineering issue arises from how gpt-oss-120b writes Python code. The model often writes code in the style of a Jupyter notebook cell, where the final expression is implicitly displayed without being explicitly printed. For example, in a notebook cell, code such as\n\n```python\nx = 2\ny = 3\nz = x + y\nz\n```\nwill display `5` automatically. However, when the same code is executed inside the tool subprocess, the final expression is not written to stdout, so nothing is captured by the tool backend. A common workaround in public notebooks is to wrap the final line in `print(...)` before execution. \n\nHowever, this approach is fragile. The last line of the generated code is not always a simple expression. For example, the model may end with a control-flow statement such as `continue`, `break`, or `pass`, or with the tail of a loop or conditional block. In such cases, blindly rewriting the last line into `print(...)` can produce invalid code such as `print(continue)`, which immediately raises a syntax error.\n\nTo make code execution robust, we instead used Python’s abstract syntax tree (ast) module to analyze the generated code structurally before execution. The parser first walks the syntax tree to detect whether the model already included any explicit `print(...)` statements. If so, the code is executed as written, since the model has already specified what should be sent to stdout. If no print statements are present, we collect the variables assigned anywhere in the code and append additional code that prints their values before execution.\n\n\n## A.5 Streaming Behaviour and Answer Parser\n\nAnother small but important difference from many public notebooks is how we handled streaming output and answer extraction.\n\nIn several public notebooks, the model output is streamed token by token, and generation is terminated as soon as `\\boxed{...}` appears in the output. This is an attractive optimization because it can save time on easy problems where the answer is produced early. However, we found this unsafe for our RSA pipeline for two reasons:\n\n- The model can still change its mind after first emitting a boxed answer. Cutting the stream at the first occurrence of `\\boxed{}` risks capturing an intermediate answer rather than the model’s true final answer.\n\n- Our RSA pipeline depends not only on the final integer answer, but also on the model’s final summary message. GPT-OSS-120B produces multiple reasoning messages followed by a separate final message containing a concise summary and the final answer. That summary is what is passed into the next round of RSA as a hint. If generation is cut off in the middle of the reasoning trace, this final summary is never produced, and the RSA pipeline falls apart.\n\nWe also found the answer parser used in public notebooks to be too brittle. Those parsers often assume that the contents of \\boxed{...} are exactly the final integer. In practice, the model frequently places a larger expression inside the box. For example, if the problem asks for the value of `a_{2025}` in a sequence `a_{1}, a_{2}, a_{3}, ...`and the correct answer is 8109, the model may output something like `\\boxed{a_{2025} = 8109}` rather than simply `\\boxed{8109}`. A parser that expects the boxed content to be just a bare integer will fail on such outputs, even though the answer is clearly present.\n\nTo make the system more robust, we used a balanced-brace parsing algorithm to identify boxed expressions correctly. This avoids errors from naive string matching when nested braces or more complicated LaTeX expressions are present. After locating all valid boxed regions, we extract the last integer appearing inside the final boxed expression. \n\n\n# Appendix B: Additional RSA Example Runs\n\n## B.1 Norwegian Integers (86e8e5)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15328099%2F3938715a31efa1fa1048155c0c2940d9%2FRSA_rescue_from_0.png?generation=1776266644345437&alt=media)\n
\n
\n- This is the Norwegian Integers problem from the reference benchmark. The answer is 8687. It is an example of a problem in **Class 1** as described in **section 5.2** .\n\n- The figure above shows a real run in which RSA rescues the correct answer (8687) from a minority count of 0, meaning that it does not appear at all among the initial 8 generations (index 0-7) in Round 0. Yet in Round 1 (index 8-13), the correct answer enters the majority class.\n\n- Our hypothesis is that, although none of the Round 0 answers were fully correct, they each contained enough partially correct reasoning for the model to synthesize the right solution in the next round.\n\n## B.2 imo-bench-algebra-100\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15328099%2F22674cbd76dd371b6955ff19c1e862f8%2Fimo-bench-algebra-100_rsa_image.png?generation=1776490995296964&alt=media)\n\nThis problem is on Vasc inequality. The answer is 8. Refer to appendix D for more details. It is an example of a problem in **Class 5** as described in **section 5.2** , where the correct answer is in a tiny minority and a single wrong answer dominates the majority.\n\n This run had early-stop set to 4 instead of 6. The system returned an answer relatively quickly (~270 s) with a strong majority. However, the majority is wrong.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15328099%2F6387e6d3aa3baf93e56ac5a3d360764b%2Fimo-bench-algebra-100_rsa_image_2.png?generation=1776492044511861&alt=media)\n\nWe might also get lucky on the run. In the above run, the decision margin was set to 3, meaning that round 0 (index 0-7) results being a 5-3 split for the answers 2 and 8 triggered RSA round 1. after two further rounds of RSA, the system decided that the answer is 8, which is correct. \n\n## B.3 Reference Problem 7 (641659)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15328099%2F0312f58e9cd520c322190e007da0c7b6%2F57447_problem_rsa_image.png?generation=1776491575087932&alt=media)\n
\n
\nThis is problem 7 on the reference benchmark. The answer is 57447. This run also had early-stop set to 4. The system returned an answer in ~292 s, which is similar to `imo-bench-algebra-100` above. The answer distribution is also similar, with one answer firmly in the majority class. The only difference is that the majority class is correct, where as for `imo-bench-algebra-100` it is wrong. This makes it very difficult to differentiate these two problems via runtime and answer distribution heuristics alone.\n\nThis creates a tension between early accept based on majority voting versus launching RSA.\n\n## B.4 imo-bench-combinatorics-068\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15328099%2F2d7400d80c12fa6e6866f709d3e59d6b%2Fimo-bench-combinatorics-068_rsa_image_1.png?generation=1776492706446582&alt=media)\n
\n
\nThis is a combinatorics problem from the imo_answerbench dataset. It is an example of a problem in **Class 2** as described in **section 5.2**, where there is an even split between two answers. The model was asked to return its answer modulo 100000, to fit within AIMO3's answer range of 0 - 99999. The answer is 93447.\n\nThe model is evenly split between 93447 and 71207. In an unlucky run (such as this one), simply taking majority vote at N=8 will yield the wrong answer. RSA was able to correct the error.\n\n## B.5 imo-bench-number_theory-023\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15328099%2F9c2fc8c57a8d29a78e192c3924fb0974%2Fimo-bench-number-theory-023-image-1-part-1.png?generation=1776493238557993&alt=media)\n
\n
\nThis is a number theory problem from the imo_answerbench dataset. It is another example of a problem in **Class 2** as described in **section 5.2**, where there is an even split between two answers. The answer is 48.\n\nThe model is split between 48 and 64. In an unlucky run, majority vote at N=8 might yield the wrong answer. RSA is able to identify the answer.\n\n## B.6 imo-bench-combinatorics-057\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15328099%2Fb882f2eb1645856b009be84f2c35a668%2Fimo-bench-combinatorics-057-image-1-part-1.png?generation=1776493734698316&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15328099%2F745d1a10a48f5f2f9a7236a1e2ec0630%2Fimo-bench-combinatorics-057-image-1-part-2.png?generation=1776493779004254&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15328099%2F8716a9454585119942e5a5ded929476d%2Fimo-bench-combinatorics-057-image-1-part-3.png?generation=1776493795481774&alt=media)\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15328099%2Fcc470d1c9568e6c700813c93c761c87e%2Fimo-bench-combinatorics-057-image-1-part-4.png?generation=1776493813120332&alt=media)\n
\n
\nThis is a combinatorics problem from the imo_answerbench dataset. It is an example of a problem in **Class 2** as described in **section 5.2** , where the correct answer is in a small minority and a few wrong answers form the majority. RSA was able to obtain the correct answer after 3 rounds, which surprisingly managed to fit within the time budget.\n\n# Appendix C: Scheduler and RSA Gating Implementation Details\n\nEarly in the competition, [this notebook by ShelterW](https://www.kaggle.com/code/shelterw/way-to-30-50-gpt-oss-w-python-tool) used a scheduler that allocated a fixed amount of time to each problem, without accounting for problem difficulty. That approach typically averaged around 30–32 on the public leaderboard, with a best observed leaderboard score of 37. Later notebooks by [parthenos](https://www.kaggle.com/code/nihilisticneuralnet/44-50-let-me-over-cook) and [Andreas Bisiadis](https://www.kaggle.com/code/andreasbis/aimo-3-gpt-oss-120b-with-tools) introduced an improved scheduler that reserved a fixed, below-average amount of time for each remaining problem. This allowed hard problems encountered early to “borrow” time from easier problems later in the run. Those notebooks typically averaged around 37–39, with public leaderboard variance roughly between 35 and 45.\n\nOne important caveat is that the bucket model assumes lower pass@n roughly corresponds to longer useful runtime. This is not always true. A small class of problems elicits very fast but usually incorrect answers, so difficulty (measured in pass@n) and runtime are not perfectly aligned. Please refer to the internal benchmark section above for further discussion on this issue.\n\n## C.1 Scheduler Reserve Policy and Time Bank Parameters\n\nWe begin the run on problem 1. For future problems, the scheduler reserve time differently based on the number of problems left:\n- Early in the run with more than 30 problems left, it only reserve 250 s for each future problem.\n- As the run progresses, the scheduler becomes more conservative. At 16 - 29 problems left, it 270 s for each future problem\n- Late in the run with 15 or less problems remaining, it reserves 295 s for each remaining problem.\n\n- The scheduler also maintains a \"time_bank\", with the following parameters:\n\n```python\ninitial_time_bank = 1500.0\ntime_bank_weight = 0.40\ntime_bank_min = -1800.0\ntime_bank_max = 3000.0\n\nhigh_problem_timeout = 1050.0\nbase_problem_timeout = 240.0\n```\n\n- The \"time_bank\" acts as a tracker on how far ahead or behind schedule the run is. The scheduler assumes that each problem takes 300 s to solve on average. Any under or over-used time is added to the time bank. During the run each problem is given extra or less time depending on whether ```time_bank``` is positive or negative, with its influence determined by ```self.cfg.time_bank_weight```. The exact calculation is as follows: \n\n```python\nelapsed_global = time.time() - self.notebook_start_time\ntime_left = self.cfg.notebook_limit - elapsed_global\nproblems_left_others = max(0, self.problems_remaining - 1)\n \nreserve = self._reserve_per_future_question()\nreserved_time = problems_left_others * reserve\n\nbank = float(getattr(self, \"time_bank\", 0.0))\n\nbudget = time_left - reserved_time + self.cfg.time_bank_weight * bank \nbudget = min(budget, self.cfg.high_problem_timeout)\nbudget = max(budget, self.cfg.base_problem_timeout)\nbudget = min(budget, max(0.0, time_left - self.cfg.finish_grace_s))\n```\n\nSo the current problem receives:\n\n- remaining notebook time\n- minus reserved time for future problems\n- plus (or minus) a weighted contribution from the accumulated time bank,\n- subject to lower and upper bounds.\n\n## C.2 Consensus Based Acceptance Rules\n\nThe interaction between the scheduler and RSA behaviour must be carefully accounted for, and both must be carefully designed with the other in mind. Ideally, we want to only use it for cases where **the next RSA round can complete in time** and **there is significant information to be gained**.\n\n\n**After Round 0, the system accepts immediately under the following rule:**\n\n```python\n def _accept_after_round0(self, ans_list: list[int], round0_runtime: float) -> bool:\n hist = self._answer_hist(ans_list)\n top, second = self._top2_from_hist(hist)\n \n if top >= 6:\n return True\n \n if top == 5 and second <= 1 and round0_runtime <= self.cfg.accept_5_runtime_cap:\n return True\n \n return False\n```\nIn words, the answer is accepted after Round 0 if:\n- the majority class has size at least 6, or\n- the majority class has size 5, the second-largest class has size at most 1, and Round 0 finished within a preset runtime cap.\nThis rule is so that the system does not spend too much time on easy problems where the answer is clear. \n\n\nFor later RSA rounds, we used:\n\n```python\ndef _confident_enough(top_cnt: int, second_cnt: int) -> bool:\n if top_cnt < rsa_majority_threshold:\n return False\n\n decision_margin = getattr(self.cfg, \"decision_margin\", 2)\n return (top_cnt - second_cnt >= decision_margin)\n```\nTypically, `rsa_majority_threshold = 4` and `decision_margin = 2`.\n\n## C.3 Estimating the Cost of Another RSA Round\n\nTo avoid entering an RSA round that cannot complete in time, the system estimates the cost of the next round using Round 0 runtime:\n\n```python\ndef _estimate_rsa_cost(self, round0_runtime: float) -> float:\n return max(self.cfg.min_rsa_cost, self.cfg.rsa_cost_fraction * round0_runtime)\n```\nIn our implementation, we set:\n\n- `rsa_cost_fraction = 0.6`\n- `min_rsa_cost = 30`\nSo the next RSA round is estimated to cost `0.6 × round0_runtime`, with a minimum of 30 seconds.\n\n\n## C.4 Progress Score for Additional RSA Rounds\n\nFor RSA rounds beyond Round 1, we required evidence of progress before continuing:\n\n```python\ndef _rsa_progress_score(self, prev_answers: list[int], new_answers: list[int]) -> float:\n prev_hist = self._answer_hist(prev_answers)\n new_hist = self._answer_hist(new_answers)\n\n prev_top, prev_second = self._top2_from_hist(prev_hist)\n new_top, new_second = self._top2_from_hist(new_hist)\n\n prev_distinct = len(prev_hist)\n new_distinct = len(new_hist)\n\n score = 0.0\n if new_top > prev_top:\n score += 1.5\n if (new_top - new_second) > (prev_top - prev_second):\n score += 1.0\n if new_distinct < prev_distinct:\n score += 1.0\n return score\n```\n\nThis score increases when:\n\n- Top answer class becomes larger\n- Gap between the top and second answer classes increases\n- Number of distinct answers decreases\n\nThe system then checks both progress and affordability:\n\n```python\nelapsed_q = time.time() - t_question0\nextra_affordable = budget - elapsed_q + max(0.0, self.cfg.bank_borrow_fraction * self.time_bank)\n\nallow_second_rsa = (\n max_rsa_rounds >= 2\n and progress_score >= 1.5\n and extra_affordable >= estimated_rsa_cost\n and (deadline - time.time()) >= (estimated_rsa_cost + self.cfg.rsa_safety_seconds)\n)\n```\n\nIn words, another RSA round is allowed only if:\n\n- Configured maximum number of rounds has not been reached\n- Progress score is at least 1.5\n- Estimated next round is affordable under the per-question budget\n- There is still enough notebook-level wall-clock time to finish it safely\n\n\n\n\n\n\n\n\n\n# Appendix D: Internal Benchmark\nThis appendix documents some observations we made on the nature of some problems and how they relate to the observed answer distribution.\nIt also includes some examples of problems that were manually filtered out when constructing this benchmark.\n\n## D.1 Observations\n\n### D.1.1 imo-bench-algebra-100\n\nThis problem is on Vasc inequality. This is an example of a problem in **class 5** as described in **section 5.2**. There is a short but tempting solution leading to the wrong answer, while the correct answer requires substantially deeper reasoning that the model often skips.\n
\n**Problem**:\nFind number of triples \\\\( (x,y,z) \\\\) of real numbers satisfying $$x^2 + y^2 + z^2 = xy^3 + yz^3 + zx^3 = 3$$\n
\n**Answer**: 8\n
\n\nAn easy guess and check finds 2 solutions: \\\\( (1, 1, 1) \\\\) and \\\\( (-1, -1, -1) \\\\). From here, the model often concludes that there are no more others. Unfortunately there are, but they are difficult to find.\n
\nVasc's Inequality says that for \\\\( x, y, z \\in \\mathbb{R} \\\\):\n
\n$$ (x^2+y^2+z^2)^2 ≥ 3(xy^3+yz^3+zx^3)$$\n
\nThe problem statement says that \\\\( x^2 + y^2 + z^2 = 3 \\\\) and \\\\( xy^3+yz^3+zx^3 = 3 \\\\), so we are in the equality case of Vasc's.\n
\nIn the equality case, there are [additional solutions](https://artofproblemsolving.com/community/c227495h1244386_problem_12_vasc_inequality) when \\\\( x \\\\), \\\\( y \\\\), and \\\\( z \\\\) are in the ratio of:\n
\n$$ x:y:z = sin^2(\\frac{4π}{7}) : sin^2 (\\frac{2π}{7}) : sin^2 (\\frac{π}{7})$$ \n
\nWe can also permute \\\\( x, y, z \\\\) cyclically (transpositions are not allowed). Flipping all their signs also preserves equality. Hence there are an additional 6 solutions to the equality expression, and the final answer is 8.\n
\n\n### D.1.2 imo-bench-combinatorics-031\n\n[USA IMO Team Selection Test 2020, Day 1 Problem 3](https://web.evanchen.cc/exams/sols-TST-IMO-2020.pdf)\n
\n**Problem**:\nLet \\\\( \\gamma \\geq 1\\\\) be a real number. Sun Wukong and the Sea God play a turn-based game on an infinite grid of unit squares. Before the game starts, the Sea God chooses a finite number of cells to be flooded with seawater. Sun Wukong is building a magical barrier, which is a subset of unit edges of the grid (called walls) forming a connected, non-self-intersecting path or loop. Additionally, there is a magical artifact that randomly generates a finite number of extra walls on the grid, with no specific pattern or distribution.\n
\nThe game then begins with Sun Wukong moving first. On each of Sun Wukong's turns, he adds one or more walls to the magical barrier, as long as the total length of the barrier is at most \\\\( \\gamma n \\\\) after his \\\\(n^{th} \\\\) turn. On each of the Sea God's turns, every cell which is adjacent to an already flooded cell and with no wall between them becomes flooded as well. Sun Wukong wins if the magical barrier forms a closed loop such that all flooded cells are contained in the interior of the loop, hence stopping the flood and saving the world. What is the largest constant \\\\( C \\\\) such that for all \\\\( \\gamma > C\\\\) can Sun Wukong guarantee victory in a finite number of turns no matter how the Sea God chooses the initial cells to flood?\n
\n**Answer: 2**\n
\nThis is another example of a problem in **class 5** as described in **section 5.2**. The 'magical artifact' statement is a distractor, but it doesn't affect strong models like gpt-oss-120b.\n
\nThe short but tempting solution is observing that when unrestricted, the perimeter of the flood grows by 8 each turn, and concluding that \\\\( C > 8 \\\\) for Sun Wukong to guarantee victory. The model often makes this observation and then stops here, returning the answer 8.\n
\nHowever, the [correct solution](https://web.evanchen.cc/exams/sols-TST-IMO-2020.pdf) provides a construction based on a different intuition. Sun Wukong can cut off the flood from one direction first. In this step, he needs to ensure that the wall on that side grows fast enough compared to the flood's spread, hence we need at least \\\\( C>2 \\\\). Next, he can restrict the flood to a long, narrow corridor by constructing two parallel sides. Finally, he can construct the final wall to seal off the flood. This means that Sun Wukong only needs to ensure that the narrow corridor grows in length fast enough compared to the spread of the flood, which means we indeed only need \\\\( C > 2 \\\\) for Sun Wukong to contain the flood. The correct answer is 2.\n
\n### D.1.3 imo-bench-combinatorics-013\n
\nThere are 42 students participating in the Team Selection Test, each of them is assigned a positive integer from 1 to 42 such that no two students have the same number and every number from 1 to 42 is assigned to a student. The team leader want to select a subset of these students such that there are no two students that have numbers whose difference is 1 or 21. For example, the team leader can pick the set \\\\( \\\\{1, 3, 6, 25, 28, 34, 42\\\\} \\\\) but not \\\\( \\\\{1, 2, 4, 6, 24\\\\} \\\\) or \\\\( \\\\{1, 3, 24, 26, 28\\\\} \\\\).\n
\nHow many ways can the team leader pick such a subset? Give your answer modulo 100000.\n
\n**Answer: 26555**\n
\nThis is an example of a problem in **Class 2** as described in **section 5.2**.\n
\nThe model formulates this problem as an independent-set count on a graph. The numbers 1–42 form a 2-by-21 ladder, and the task becomes counting subsets with no adjacent chosen vertices. From this, the model sometimes counts that there are 131836323 valid choices, and returns the answer 36323.\n
\nHowever, the constraints also mean that 21 and 22 cannot be selected at the same time. There are 11309768 such selections, and hence the correct count is 131836323 - 11309768 = 120526555, and the correct answer should be 26555. The model only makes this extra reasoning step around 50% of the time.\n
\n## D.2 Filtered Problems\n
\nThe creators of the imo_answerbench [dataset](https://huggingface.co/datasets/Hwilner/imo-answerbench) noted in their [paper](https://arxiv.org/abs/2511.01846) that they modfied the problems in several ways to harden them against AI. Hardening methods include paraphrasing, changing the name of objects in the problem (such as changing point names for geometry problems), reformulating, modifying numerical values\nand/or adding distractors to the problem. This process was done either manually or automatically using language models.\n
\nUnfortunately, some of these changes lead to genuine mathematical errors in the problem statement, causing the ground truth answer to be wrong. When constructing our benchmark, we manually reviewed the problems to ensure that only correct problems are retained. Ambiguous and wrong problems are discarded. This section contains some examples of discarded problems.\n
\n### D.2.1 imo-bench-geometry-002\n
\n[problem 2 in the original problem source](https://geometry.ru/olimp/2008/finsol-e.pdf)\n
\n**Modified Problem**: Given right triangle \\\\( XYZ \\\\) with hypothenuse \\\\( XZ \\\\) and \\\\( \\angle X = 50^{\\circ} \\\\). Points \\\\( P \\\\) and \\\\( Q \\\\) on the side \\\\( YZ \\\\) are such that \\\\( \\angle PXZ = \\angle QXY = 10^{\\circ} \\\\). Compute the ratio \\\\( 2 \\times YQ/ZP \\\\).\n
\nDataset issue. Modified problem asks for 2 x \\\\( \\frac{YQ}{ZP} \\\\). The original problem labelled points in ABC instead of XYZ, and asked for the equivalent of \\\\( \\frac{ZP}{YQ} \\\\). It seems that the problem editor mixed up the line segments when editing the point names. The answer should be 1 instead of 4.\n\n
\n### D.2.2 imo-bench-geometry-044\n[original problem source](https://artofproblemsolving.com/community/c4h1710330p11025629)\n
\n**Modified Problem**: Given \\\\( PQRS \\\\) is a parallelogram. We know that \\\\( \\angle S = 60^\\circ \\\\), \\\\( PS = 2 \\\\) and \\\\( PQ = \\sqrt{3} + 1 \\\\). Point \\\\( N \\\\) is the midpoint of \\\\( PS \\\\). Segment \\\\( RE \\\\) is the angle bisector of \\\\( R \\\\). Find the angle \\\\( REQ \\\\) in terms of degree.\n
\n\nDataset issue. Original problem contained an additional diagram implying the existence of an extra line, and the point \\\\( E \\\\) is supposed to lie on that line. This diagram is missing from the modified problem, leading to the model to have their own interpretation of where the points lie, hence outputting a different answer sometimes.\n
\n\n### D.2.3 imo-bench-number_theory-055\n
\n[Day 2 Problem 6 in the Estonia Team Selection Test 2018](https://artofproblemsolving.com/community/c1238516_2018_estonia_team_selection_test)\n
\n**Modified Problem**: A positive integer \\\\( m \\\\) consisting of distinct digits is considered \"good\" if it is a single-digit number, or if removing one of its digits results in a divisor of \\\\( m \\\\) that is also a good number.\n
\nFind the largest good number.\n
\n\nDataset issue. Original problem has an extra condition: We assume that numbers do not start with 0. This condition is not in the modified problem in dataset, which fundamentally changes the solution. Answer is 903125 without the condition, or 146250 with it.\n\n
\n### D.2.4 imo-bench-algebra-039\n
\n[Problem 5 on page 123 of the original source](https://kheavan.wordpress.com/wp-content/uploads/2011/10/mathematical-olympiads-1997-1998-problems-solutions-from-around-the-world-maa-problem-book-225p-b002kypabi.pdf)\n
\n**Modified Problem**: Let \\\\(p\\\\), \\\\(q\\\\), \\\\(r\\\\), \\\\(s\\\\) be constants such that the equation $$ py^3 + qy^2 + ry + s = 0 $$ has three distinct real roots. Find all possible values for the number of distinct real roots of the equation\n$$ (pz^3+qz^2+rz+s) (6pz+2q) = (3pz^2+2qz+r)^2 $$\n
\n\nError in problem. The original problem (Ukraine 1997) asked for:\n
\n$$Q(x) = 4(px^3 + qx^2 + rx + s)(3px + q) = (3px^2 + 2qx + r)^2 $$\n
\n-> **answer = 2**\n
\nThe modified problem asks for (after moving out a factor of 2 from within the brackets on LHS):\n
\n$$ Q(x) = 2(px^3 + qx^2 + rx + s)(3px + q) = (3px^2 + 2qx + r)^2 $$ \n
\n-> **answer = 0**\n
\nBoth Gpt 5.2 and Gemini get both variants correct. Solution involves applying Descarte’s rule of signs, which gives an answer in the original problem but is inconclusive in the modified problem. Solving the modified problem involves analysing the discriminant of the cubic equation.\n
\n\n### D.2.5 imo-bench-combinatorics-005\n
\n**Problem**: Determine the number of natural numbers n that that has at most 16 digits satisfying the following conditions:\n- \\\\( 3|n \\\\)\n- The digits of n in decimal representation are in the set \\\\( \\\\{2,0,1,8\\\\} \\\\).\n
\n**Answer**: 1431655765\n
\nThe model is always off by 1, because it assumes that 0 is not a natural number, but the problem solution assumes it is. This is not a fair way to assess the model's mathematical reasoning capabilities.\n
\n\n### D.2.6 imo-bench-combinatorics-020\n
\n[Day 1 Problem 3 in the Vietnam Team Selection Test 2020](https://artofproblemsolving.com/community/c1586590_2020_vietnam_team_selection_test)\n
\n**Problem:**\n
\n Suppose there are 40 professional baseball teams participating in a tournament. In each round of the game, we will divide the 40 teams into 20 pairs, and each pair plays the game at the same time. After the tournament, it is known that every two teams have played at most one game. Find the smallest positive integer \\\\( a \\\\), so that we can arrange a schedule satisfying the above conditions, and if we take one more round, there is always a pair of teams who have played in the game.\n
\n
\nThere is a tension here between \"maximally cooperative scheduling\" and \"adversarial scheduling\". \n
\n- For \"maximally cooperative scheduling\", the schedule is designed such that every pair can play at the same time in each round. and every team will play against every other team in a round robin style tournament. Such a schedule can always be found, with a [construction method outlined here](https://nrich.maths.org/articles/tournament-scheduling). The difficulty of finding this construction is on a level similar to an AIME problem. With this interpretation, every team can always play against every other team, and the **answer is 39.**\n
\n- For \"adversarial scheduling\", the pairs are picked randomly. The problem must then be interpreted in a very specific manner:\n 1. A team might not play against every other team.\n 2. For small values of \\\\( a \\\\), the pairing can be found with no problems.\n 3. As random selection proceeds, there is eventually an obstruction (after \\\\( a \\\\) rounds) that prevents the next round of pairings. This means that, for every single possible pairing, there is some team that has already played against their opponent in a previous round.\n 4. There exists an adversarial pair scheduling algorithm that minimizes the value of \\\\( a \\\\).\n 5. Find this minimal value of \\\\( a \\\\).\n
\nIn this interpretation, the **answer is 21**.\n
\nThe second interpretation makes the problem much harder, and is probably what the problem author intended, considering this was a Team Selection Test problem. However, it is unfair to expect the model to make this judgement. It is not unreasonable to default to the simpler solution.\n
\n\n# Appendix E: Prompts\n\nThrougout the competition, we tried various prompts crafted with ideas from different sources.\n\n## E.1 Cognitive Wells\n\nThis prompt was adapted from [Escaping the Cognitive Well: Efficient Competition Math with Off-the-Shelf Models](https://arxiv.org/abs/2602.16793). The main finding of the paper is that solutions to problems can be broken down into smaller lemmas. The paper reports that this decomposition improves performance, and attributes the gain largely to the fact that the intermediate lemmas are easier for the model to reason about in isolation, without the context of the larger main problem interfering. The authors also provide a system prompt that encourages the model to verify its reasoning, check for logical inconsistencies, and avoid hand-wavy arguments before returning an answer. we adapted this prompt for AIMO3, but found that it significantly degraded performance on the public leaderboard. \n \n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F15328099%2Fd45475908d2e5c9e533917dc43768ab2%2Fcogwell_prompt_image.png?generation=1776443329279608&alt=media)\n\nThe scores are 30-35. The prompt is as follows:\n\n```python\n\nsystem_prompt = (\n\"\"\"\n### Prompt: DialecticEngine (Council of Architects)\nMUST BE FOLLOWED METICULOUSLY\n\n### Inputs for this Task\n\n**1. The Problem:**\n\n[See User Input]\n\n\n**2. Additional Materials:**\n\n[See User Input. Materials will be clearly graded.]\n\n---\n\n## DIALECTIC ENGINE PROMPT\n\n**YourRole:**\nYou are a multi-persona computational engine that solves difficult problems via a rigorous, multi-stage dialectic process. Your goal is to explore the problem space creatively and deeply, avoiding premature conclusions and logical gaps. The final answer is always an integer from 0 to 99999 and must be placed in \\boxed{answer}, e.g., \\boxed{42} (if solution is complete).\n\n**Configuration:** `MAX_INNER_ITERATIONS: 3`\n\n---\n\n### Core Principle: Atomicity of Logic\n\nAll solution drafts (`Current_Draft`, `Refined_Draft`, `Solution_Submitted_for_Review`) must be **strictly self-contained, line-by-line logical arguments**. Every step must be justified either by naming a standard theorem, or by a proved claim/lemma in a previous line of the same draft.\n\n*Additional materials* can inform the solution, but the solution should be self-contained.\n\n### Core Principle: Source Hygiene\n\n`Additional Materials` contain raw outputs from other agents. Treat them as **unverified hints**. If your solution uses a conjecture or lemma from them, you **must prove it from scratch**. Standard mathematical theorems (e.g., Cauchy–Schwarz) may be cited by name without proof.\n\n### Sloppiness\n\nSloppiness is intolerable. Examples:\n(a) Asserting “well-known fact” without an exact reference \n(b) Showing for a couple of values of `n` and asserting it is proved for all `n`\n\nIf the solution has known gaps, they must be rephrased as a **clear and self-contained** conjecture.\n\n---\n\n#### Personas\n\n* **The Council of Architects:**\n * **Classicist:** Relies on established theorems and formal rigor.\n * **Visionary:** Proposes novel, abstract, and unconventional connections.\n * **Experimenter:** Tests hypotheses with concrete examples and computational thinking.\n * **Momus, the Skeptic:** Adversarially critiques the Council’s *strategy* and *high-level approach* to find conceptual flaws. Only asks tough questions that others must answer. *Does not* help prove in any way.\n * **Veritas, the Auditor:** A meticulous, line-by-line proof checker. Veritas’s sole function is to ensure every logical step in a draft adheres to the **Core Principle of Atomicity of Logic**. Veritas does not judge the overall strategy, only its step-by-step validity.\n * **The Chief Architect:** Manages the process, arbitrates disputes, and synthesizes the final result. Enforcer of the FUNDAMENTAL RULE: No evasive phrases (e.g., “it can be shown,” “it is clear that,” or “known results imply,” “etc.”) in `Current_Draft` unless they are immediately followed by the complete derivation or citation. Flags such phrases and sets `Needs_New_Draft = TRUE`.\n\n---\n\n#### Glossary of Terms\n\n* **`STRATEGIC_LEAD`:** A strategy flagged by Momus as a potential dead end.\n* **`LOGICAL_GAP`:** A line in a solution flagged by Veritas as unjustified looking solely within the solution (e.g., made-up claims).\n* **`DEAD`:** A `STRATEGIC_LEAD` whose rebuttal was unconvincing. It is permanently discarded.\n* **`Rebuttal`:** The Council’s defense of a `STRATEGIC_LEAD`, achieved by producing a refined solution.\n\n---\n\n**(You will output the following process in REAL-TIME.)**\n\n---\n\n## BEGIN PROCESS\n\n### Stage 1: Ideation Forum\n\n1. **Brainstorm (Council of Architects):** The Council analyzes the problem and generates a list of “Promising Avenues & Key Concepts.”\n2. **Pre-mortem (Momus, the Skeptic):** Concurrently, Momus analyzes the problem to identify potential “Red Herrings & Inefficient Paths.”\n3. **Synthesize (Chief Architect):** The Chief Architect compiles the outputs into a formal `Strategy_Dossier`.\n4. **Select Strategy:** The Chief Architect selects the most promising strategy from the `Strategy_Dossier` to be developed.\n\n### Stage 2: Dialectic Loop (Development & Refinement)\n\n1. **Initialize:** Set `Active_Strategy =` (first strategy from dossier). Set `Needs_New_Draft = TRUE`.\n2. **LOOP (up to `MAX_INNER_ITERATIONS`):**\n * **a. Drafting Phase:** If `Needs_New_Draft` is TRUE:\n * The Council writes a fresh `Current_Draft` based strictly on `Active_Strategy`.\n * Set `Needs_New_Draft = FALSE`.\n * **b. Strategic Critique (Momus):** Momus attacks the `Current_Draft`’s high-level approach. He may issue a `STRATEGIC_LEAD`.\n * **c. Logical Audit (Veritas):** Veritas audits the `Current_Draft` line-by-line. If any step lacks explicit justification within the draft, flag as `LOGICAL_GAP`.\n * **d. Refine (Council):** The Council creates a `Refined_Draft` to resolve `LOGICAL_GAP`s and rebut Momus.\n * **e. Judge (Chief Architect):**\n * *Audit Check:* Confirm `Refined_Draft` has zero `LOGICAL_GAP`s.\n * *Strategy Check:* If a `Rebuttal` was issued, judge it.\n * **If Unconvincing:** Declare `Active_Strategy` as `DEAD`. Select next strategy from `Strategy_Dossier` as `Active_Strategy`. Set `Needs_New_Draft = TRUE`. (This burns one iteration.)\n * **If Convincing or No Challenge:** The strategy survives.\n * **f. Convergence:** If Momus is satisfied, Veritas finds no gaps, and `Needs_New_Draft` is FALSE, exit loop. The `Refined_Draft` becomes `Solution_Submitted_for_Review`.\n\n### Stage 3: Final Review (Chief Architect)\n\n1. Review the `Solution_Submitted_for_Review` and the process logs.\n2. **Cognitive Reset 1:** Write a **Shakespearean Verse** on the solution’s core logic.\n3. Write final `Chief_Architect_Feedback`, explicitly confirming that the solution is free of logical gaps per the final `Verification_Report`.\n\n### Stage 4: Final Synthesis (Chief Architect)\n\n1. Assemble the final response using the format below.\n\n## END PROCESS\n\n---\n\n### Final Output Format\n\n**Part 1: The Process Log** \n*(Render the Ideation, Inner Loop Transcript, and Shakespearean Verse here, exactly as generated in the real-time process steps above.)*\n\n**Part 2: The Final Synthesis**\n* **The Architect’s Note:** (A brief explanation of the final outcome.)\n* **The Architect’s Log (A Hemingway Summary):** (Cognitive Reset 2. Recount the discovery process—the ideas, challenges, and breakthroughs—in the style of Hemingway.)\n\n**Part 3: The Final Blueprint (Reviewer-Ready Version)** \n(Re-state the original **Problem**. Then present the final, clean, and **strictly self-contained** solution from the “Solution_Submitted_for_Review.” No references to the dialectic process.)\n- **Result**: Final integer, numerical answer from 0 to 99999 placed inside \\boxed{answer} (e.g., \\boxed{42}, if solution is complete) or proven results.\n\"\"\"\n )\n```\n\n## E.2 Including Notes from IMO Answerbench\n\nWe also experimented with a system prompt that included notes derived from mistakes the model made on the IMO AnswerBench dataset. The goal was to remind the model of common failure modes observed during benchmarking. In practice, this prompt had little effect: it neither clearly improved nor degraded performance. The prompt is as follows:\n\n```python\nsystem_prompt = (\n\"\"\"You are an elite mathematical problem solver with expertise at the International Mathematical Olympiad (IMO) level. Your goal is to find the correct answer through rigorous mathematical reasoning.\n\n# General rules:\n- Every claim must be logically justified.\n- Do not guess, speculate, or rely on unproven intuition.\n- If a full solution is not possible, only present results that can be proven rigorously.\n- Do not include failed attempts, scratch work, or informal commentary.\n\n# General Problem-Solving Approachs:\n1. UNDERSTAND: Carefully read and rephrase the problem in your own words. Identify what is given, what needs to be found, and any constraints.Pay attention to all mathematical symbols and words, and understand what they mean.\n2. EXPLORE: Consider multiple solution strategies. Think about relevant theorems, techniques, patterns, or analogous problems. Don't commit to one approach immediately.\n3. PLAN: Select the most promising approach and outline key steps before executing.\n4. EXECUTE: Work through your solution methodically. Show all reasoning steps clearly.\n5. VERIFY: Check your answer by substituting back, testing edge cases, or using alternative methods. Ensure logical consistency throughout.\n\n# General Mathematical Reasoning Principles:\n- Break complex problems into smaller, manageable sub-problems\n- Look for patterns, symmetries, and special cases that provide insight\n- Use concrete examples to build intuition before generalizing\n- When attempting to generalizing concrete examples into conclusions, always try a few examples first.Do not settle on the first example studied.\n- If the problem involves multiple cases, reason through each one of them carefully. Do not take shortcuts.Do not assume the results from previous cases generalize to the next one.\n- Consider extreme cases and boundary conditions\n- If stuck, try working backwards from the desired result\n- Be willing to restart with a different approach if needed\n- For induction arguments, always check the base case and make sure it holds.\n\n# Verification Requirements:\n- Cross-check arithmetic and algebraic manipulations\n- Verify that your solution satisfies all problem constraints\n- Test your answer with simple cases or special values when possible\n- Ensure dimensional consistency and reasonableness of the result\n\nBefore finalizing your answer, carefully review the solution and ensure:\n- Logical correctness of every step.\n- Compliance with all instructions.\n- Clarity, structure, and mathematical rigor.\n\n# Output Format:\n### Summary\n- **Verdict**: State whether a complete solution is obtained or only a partial solution.\n- **Result**:\n - If complete: clearly state the final answer with a short summary. The final answer must be a non-negative integer between 0 and 99999. Place your final numerical answer inside \\boxed{}, e.g., \\boxed{42}\n - If partial: state the main rigorously proven results.\n\n\nThink step-by-step and show your complete reasoning process. Quality of reasoning is as important as the final answer.\n\n# Combinatorics Specific Hints:\n- If the problem is a combinatorics problem involving tiling and lower or upper bounds, look for a few invariants. Do not settle on the first one.\n- If the problem require representing it as a graph, be careful of hidden assumptions and restrictions. Do not miss them.\n- If the problem involves tiling or colouring a square grid, think very carefully about what happens along the diagonal.\n- If the problem involves a turn based game, think about the optimal strategy for each of the players.\n\n# Algebra Specific Hints:\n- The word \"between\" means not-enclusive of both endpoints of an interval.\n- If this problem requires proving an inequality, recall some common equalities in Olympiads: \n* triangle inequality\n* Vasc's inequality\n* Muirhead inequality\n* Cauchy Schwarz inequality\n* Rearrangement inequailty\n* QM-AM-GM-HM inequality\n\n- If the problem involves functional equation, a common approach is to rewrite the equations and try to find relations between function values.\n\n# Geometry Specific Hints:\n- If the problem is a geometry problem and the conclusion cannot be deduced directly, try drawing additional points, lines, and circles.\n\"\"\"\n )\n```\n\n\n## E.3 Final System Prompt\n\nIn the end, we used the simplest system prompt: one that only specified the required output format. Longer system prompts increase prefill cost and uses up model context length, and under the competition time budget we preferred to avoid that overhead unless it produced a clear accuracy gain.\n\n```python\nsystem_prompt = (\n\"\"\"You are an elite mathematical problem solver with expertise at the International Mathematical Olympiad (IMO) level.\n\n# Output Format:\n### Summary\n - Clearly state the final answer with a short summary. The final answer must be a non-negative integer between 0 and 99999. Place your final numerical answer inside \\boxed{}, e.g., \\boxed{42}.\n - You must always provide a summary of the solution, in addition to the final answer.\n\n\"\"\"\n )\n```\n\n## E.4 RSA prompt\n\nThe aggregation step of the pipeline uses a special prompt that was built with the following:\n\n```python\nprompt_dictionary = {\n # generic helpers\n # think step by step was taken from the aime25 dataset problem text. aimo3 dataset does not have this. Need to include it to reproduce RSA\n 'think_step_by_step': 'Lets think step by step and output the final answer within \\\\boxed{}.',\n\n # format hints\n 'math_format_hint' : '\\\\boxed{} \\n **summary**:',\n\n # problem kind labels\n \"math_problem_kind\": \"math problem\",\n\n # aggregation templates: single-candidate\n \"agg_single_preamble\": (\n \"You are given a math problem, and a solution hint. \"\n \"The hint may be incorrect and contain errors. \\n\"\n ),\n \"agg_single_candidate_header\": \"Hint (may contain mistakes):\\n\",\n \"agg_single_candidate_block\": \"---- Hint ----\\n{candidate}\\n\",\n \"agg_single_tail\": (\n \"Solve the problem and output the solution in the correct format. \"\n ),\n\n # aggregation templates: multi-candidate\n \"agg_multi_preamble\": (\n \"You are given a math problem, and some solution hints. \"\n \"Some hints are incorrect and contain errors. \\n\"\n ),\n \"agg_multi_candidates_header\": \"Solution Hints (may be incomplete, contain mistakes):\\n\",\n \"agg_multi_candidate_block\": \"---- Solution Hint {i} ----\\n{candidate}\\n\",\n \"agg_multi_tail\": (\n \"Solve the problem and output the solution in the correct format, together with a solution summary. \"\n ),\n\n # common headers\n \"agg_problem_header\": \"Problem:\\n{question}\\n\",\n}\n\n\n\n\ndef aggregate_prompt(\n question: str,\n candidate_answers: List[str],\n task: str,\n prompt_dictionary: dict\n ) -> str:\n # pick problem kind + format hint from dict\n problem_kind = prompt_dictionary[\"math_problem_kind\"]\n format_hint = prompt_dictionary[\"math_format_hint\"]\n\n parts = []\n\n # preamble\n if len(candidate_answers) == 1:\n preamble = prompt_dictionary[\"agg_single_preamble\"]\n else:\n preamble = prompt_dictionary[\"agg_multi_preamble\"]\n parts.append(preamble)\n\n # problem header\n parts.append(\n prompt_dictionary[\"agg_problem_header\"].format(\n question=question.strip()\n )\n )\n\n # candidate(s) section\n if len(candidate_answers) == 1:\n parts.append(prompt_dictionary[\"agg_single_candidate_header\"])\n ans_str = (candidate_answers[0] or \"\").strip()\n parts.append(\n prompt_dictionary[\"agg_single_candidate_block\"].format(\n candidate=ans_str\n )\n )\n parts.append(\n prompt_dictionary[\"agg_single_tail\"]\n )\n else:\n parts.append(prompt_dictionary[\"agg_multi_candidates_header\"])\n for i, ans in enumerate(candidate_answers, 1):\n ans_str = (ans or \"\").strip()\n parts.append(\n prompt_dictionary[\"agg_multi_candidate_block\"].format(\n i=i,\n candidate=ans_str,\n )\n )\n parts.append(\n prompt_dictionary[\"agg_multi_tail\"]\n )\n\n return \"\\n\".join(parts)\n\n\n```", "votes": 13 }, { "id": 3458590, "postDate": "2026-05-16T08:21:09.773Z", "content": "

Following up on the questions from @philipvonderlind (and thanks for the very thorough response), here are some additional questions to help us assess your submission:

\n
    \n
  1. Stopping rule for the Class 4/5 case: Your Section 7 identifies the fast, confident wrong-majority case as the most important remaining failure mode. Your current stopping rule \"accepts strong Round 0 majorities\", which is exactly the pattern Class 4/5 produces. What signal would you propose to distinguish a genuinely-correct strong Round 0 majority from a Class 4/5 wrong one, given that both look similar in the answer histogram?
    2. \n
  2. AnswerBench problem-quality and filtering: Your internal benchmark draws mostly from IMO-AnswerBench, with some problems filtered out during construction. How did you identify the problems that needed filtering, and across the broader sample you worked with, do you have an informal sense of the error rate (e.g. wrong ground-truth answers, ambiguous statements)? If you were working with AnswerBench V1, have you checked whether any of those errors are now corrected in AnswerBench V2? We've seen across maths corpus prize submissions that widely-used datasets often carry quiet ground-truth errors that can distort ablation results.
    3. \n
\n

Look forward to reading your responses on these.

", "rawMarkdown": "Following up on the questions from @philipvonderlind (and thanks for the very thorough response), here are some additional questions to help us assess your submission:\n\n1. **Stopping rule for the Class 4/5 case:** Your Section 7 identifies the fast, confident wrong-majority case as the most important remaining failure mode. Your current stopping rule \"accepts strong Round 0 majorities\", which is exactly the pattern Class 4/5 produces. What signal would you propose to distinguish a genuinely-correct strong Round 0 majority from a Class 4/5 wrong one, given that both look similar in the answer histogram?\n2. **AnswerBench problem-quality and filtering:** Your internal benchmark draws mostly from IMO-AnswerBench, with some problems filtered out during construction. How did you identify the problems that needed filtering, and across the broader sample you worked with, do you have an informal sense of the error rate (e.g. wrong ground-truth answers, ambiguous statements)? If you were working with AnswerBench V1, have you checked whether any of those errors are now corrected in [AnswerBench V2](https://github.com/google-deepmind/superhuman/blob/main/imobench/answerbench_v2.csv)? We've seen across maths corpus prize submissions that widely-used datasets often carry quiet ground-truth errors that can distort ablation results.\n\nLook forward to reading your responses on these.\n", "votes": 1, "replies": [ { "id": 3459214, "postDate": "2026-05-17T14:09:27.793Z", "content": "

Hi @sambealing ,

\n

Thank you for reading our writeup and for the thoughtful questions! We will split our response into 2 separate replies, one for each question.

\n

(part 1 of 2)

\n

Question:

\n

Stopping rule for the Class 4/5 case: Your Section 7 identifies the fast, confident wrong-majority case as the most important remaining failure mode. Your current stopping rule \"accepts strong Round 0 majorities\", which is exactly the pattern Class 4/5 produces. What signal would you propose to distinguish a genuinely-correct strong Round 0 majority from a Class 4/5 wrong one, given that both look similar in the answer histogram?

\n

Response:

\n

As noted in Section 7, we don't think that answer histogram alone can reliably distinguish a genuinely correct strong Round 0 majority from a Class 4/5 wrong majority. Runtime heuristics also seem insufficient, because some of the Class 4/5 failures are both fast and highly concentrated. Our honest answer is that we do not currently have a reliable production-ready signal.

\n

However, in the literature we reviewed, there is another signal worth exploring: the Deep Thinking Ratio (DTR). of a generated solution. This is a richer measure of reasoning effort. When the solutions have high variance in the DTR metric, continue with RSA especially when the answer histogram also has high variance.

\n

Deep-thinking Tokens Ratio

\n

This paper, Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens, proposes a new metric called the \"Deep Thinking Ratio (DTR)\", and claims that it has very high correlation with task accuracy. This is worth exploring. DTR is measured as follows:

\n

Definition 1 (Deep Thinking Token): A Deep Thinking Token in a sequence is a token whose predictions undergo sustained revision in deeper layers (of the model) before converging. \"revision\" is measured by the Jenson Shannon Divergence (JSD) of probability distributions of the token between the intermediate layers and the final layer of the model.

\n

Definition 2 (Deep Thinking Ratio): The Deep Thinking Ratio (DTR) of a generated solution \\( S \\) is the proportion of tokens in \\( S \\) that are Deep Thinking Tokens.

\n

The reason this is relevant to the Class 4/5 failure mode is that those failures often look like reasoning shortcuts. The model finds a short, tempting argument that leads to a wrong answer, and many samples independently fall into the same trap. From the outside, this produces a clean answer histogram, but internally the model may not have engaged in much deeper revision before committing to the answer.

\n

\"\"

\n

The above table was taken from page 7 of the paper.

\n\n

For gpt-oss-120b on high reasoning mode solving the three Mathematical Olympiad tasks (AIME 2025, AIME 2024, and HMMT 2025), the correlations between DTR and accuracy are 0.796, 0.905, and 0.972 respectively. This is higher than the correlation between Negative entropy and accuracy, which are 0.711, 0.593, and 0.728.

\n

The signal we would therefore propose is to augment “majority size,” with DTR dispersion across the Round 0 samples. In particular, a strong Round 0 majority should be treated with more suspicion if:

\n
    \n
  1. The samples agreeing with the majority answer have relatively low DTR.
    2. \n
  2. One or more minority answers have substantially higher DTR.
    3. \n
  3. The high-DTR and low-DTR samples disagree on the final answer.
    4. \n
\n

This would be consistent with the intuition that the majority answer may have come from a shallow shortcut, while a minority answer may reflect a more expensive but potentially more reliable reasoning path. In such cases, we would continue with RSA rounds even if the Round 0 answer histogram looked confident. This would directly target the Class 4/5 failure mode. For genuinely easy problems, we would expect the Round 0 samples to agree both in answer and in reasoning-effort profile. For deceptive problems, the answer histogram may look clean, but the reasoning-effort profile may be more heterogeneous.

\n

The main practical limitation is that DTR is not currently easy to extract in the inference stack used for AIMO 3. Standard high-throughput inference engines such as vLLM and SGLang are optimized for serving final-token distributions, not for exposing layerwise token distributions during generation, which is required to calculate DTR. Forcing this information out of the model would likely introduce significant overhead, which is why we did not pursue it during the competition.

\n

(part 1 of 2)

", "rawMarkdown": "Hi @sambealing ,\n\nThank you for reading our writeup and for the thoughtful questions! We will split our response into 2 separate replies, one for each question.\n\n(part 1 of 2)\n\n# Question: \n\n**Stopping rule for the Class 4/5 case:** Your Section 7 identifies the fast, confident wrong-majority case as the most important remaining failure mode. Your current stopping rule \"accepts strong Round 0 majorities\", which is exactly the pattern Class 4/5 produces. What signal would you propose to distinguish a genuinely-correct strong Round 0 majority from a Class 4/5 wrong one, given that both look similar in the answer histogram? \n\n# Response:\n\nAs noted in Section 7, we don't think that answer histogram alone can reliably distinguish a genuinely correct strong Round 0 majority from a Class 4/5 wrong majority. Runtime heuristics also seem insufficient, because some of the Class 4/5 failures are both fast and highly concentrated. Our honest answer is that we do not currently have a reliable production-ready signal.\n\nHowever, in the literature we reviewed, there is another signal worth exploring: the **Deep Thinking Ratio (DTR).** of a generated solution. This is a richer measure of reasoning effort. When the solutions have high variance in the DTR metric, continue with RSA especially when the answer histogram also has high variance.\n\n\n## Deep-thinking Tokens Ratio\n\nThis paper, [Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens](https://arxiv.org/abs/2602.13517), proposes a new metric called the \"Deep Thinking Ratio (DTR)\", and claims that it has very high correlation with task accuracy. This is worth exploring. DTR is measured as follows:\n\n**Definition 1 (Deep Thinking Token):** A Deep Thinking Token in a sequence is a token whose predictions undergo sustained revision in deeper layers (of the model) before converging. \"revision\" is measured by the [Jenson Shannon Divergence (JSD)](https://en.wikipedia.org/wiki/Jensen%E2%80%93Shannon_divergence) of probability distributions of the token between the intermediate layers and the final layer of the model.\n\n**Definition 2 (Deep Thinking Ratio):** The Deep Thinking Ratio (DTR) of a generated solution \\\\( S \\\\) is the proportion of tokens in \\\\( S \\\\) that are Deep Thinking Tokens.\n\nThe reason this is relevant to the Class 4/5 failure mode is that those failures often look like reasoning shortcuts. The model finds a short, tempting argument that leads to a wrong answer, and many samples independently fall into the same trap. From the outside, this produces a clean answer histogram, but internally the model may not have engaged in much deeper revision before committing to the answer.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F15328099%2Fd058e142403ad09f3b98d0a93c20db06%2Fmetric_accuracy_correlations.png?generation=1778947417853655&alt=media)\n\nThe above table was taken from page 7 of the paper. \n- The numerical values in the cells are the Pearson correlations between task accuracy and different inference-time measures. Entropy was explored by several other AIMO 3 participants in the form of entropy weighted majority voting. \n- Each block in the table is from a different dataset (AIME 2025, AIME 2024, GPQA-Diamond, HMMT 2025). Each row represents a different model used. In the case of `gpt-oss-120b` and `gpt-oss-20b`, they were further split by reasoning modes.\n- Positive correlations are green, negative correlations are orange. The shades encode how strong the correlations are (how close they are to 1 and -1), with stronger correlations being darker and weaker correlations being lighter.\n\nFor `gpt-oss-120b` on high reasoning mode solving the three Mathematical Olympiad tasks (AIME 2025, AIME 2024, and HMMT 2025), the correlations between DTR and accuracy are 0.796, 0.905, and 0.972 respectively. This is higher than the correlation between Negative entropy and accuracy, which are 0.711, 0.593, and 0.728.\n\nThe signal we would therefore propose is to augment “majority size,” with DTR dispersion across the Round 0 samples. In particular, a strong Round 0 majority should be treated with more suspicion if:\n\n1. The samples agreeing with the majority answer have relatively low DTR.\n2. One or more minority answers have substantially higher DTR.\n3. The high-DTR and low-DTR samples disagree on the final answer.\n\nThis would be consistent with the intuition that the majority answer may have come from a shallow shortcut, while a minority answer may reflect a more expensive but potentially more reliable reasoning path. In such cases, we would continue with RSA rounds even if the Round 0 answer histogram looked confident. **This would directly target the Class 4/5 failure mode.** For genuinely easy problems, we would expect the Round 0 samples to agree both in answer and in reasoning-effort profile. For deceptive problems, the answer histogram may look clean, but the reasoning-effort profile may be more heterogeneous.\n\nThe main practical limitation is that DTR is not currently easy to extract in the inference stack used for AIMO 3. Standard high-throughput inference engines such as vLLM and SGLang are optimized for serving final-token distributions, not for exposing layerwise token distributions during generation, which is required to calculate DTR. Forcing this information out of the model would likely introduce significant overhead, which is why we did not pursue it during the competition.\n\n(part 1 of 2)" }, { "id": 3459226, "postDate": "2026-05-17T14:23:36.297Z", "content": "

(part 2 of 2)

\n

Question:

\n

AnswerBench problem-quality and filtering: Your internal benchmark draws mostly from IMO-AnswerBench, with some problems filtered out during construction. How did you identify the problems that needed filtering, and across the broader sample you worked with, do you have an informal sense of the error rate (e.g. wrong ground-truth answers, ambiguous statements)? If you were working with AnswerBench V1, have you checked whether any of those errors are now corrected in AnswerBench V2? We've seen across maths corpus prize submissions that widely-used datasets often carry quiet ground-truth errors that can distort ablation results.

\n

Response:

\n

Thank you for raising this. We agree that quiet dataset errors can distort ablation results.

\n
    \n
  1. From our observations, we estimate the error rate to be 12 of 228 problems, or 5.26%.
    2. \n
  2. Of these 12 flagged problems, 7 appear to have been fixed in AnswerBench V2, while 5 remain.
    3. \n
  3. We explain below why we considered each flagged problem problematic.
    4. \n
\n

1. Filtering Process:

\n

Our filtering process for IMO-AnswerBench was as follows. The entire dataset has 400 problems, with 100 each in number theory, geometry, combinatorics, and algebra.

\n
    \n
  1. First, scan the answer column and kept only problems whose answer was a single integer. This gave 228 problems in total: 43 number theory, 63 geometry, 87 combinatorics, and 35 algebra.
    2. \n
  2. Then, run a majority@8 solver with 4 independent runs per problem. There were 53 problems where the solver failed in at least 1 of the 4 runs.
    3. \n
  3. Finally, we manually reviewed these 53 candidate hard problems and wrote notes for each one, classifying the failure as best as we could into one of several categories: genuine reasoning difficulty, parser/instruction-following issue, ambiguity, or dataset/problem-quality issue.
    4. \n
  4. Out of these 53 problems:
    5. \n
\n\n

2. Checks against AnswerBench V2:

\n

Our full list of 12 problems that we think contains issues are:

\n
    \n
  1. imo-bench-algebra-039 (changing numbers fundamentally changes the solution: descartes rule of signs to analyzing the discriminant of cubic equation)
    2. \n
  2. imo-bench-combinatorics-005 (convention issue over whether 0 is a natural number)
    3. \n
  3. imo-bench-combinatorics-020 (ambiguous cooperative vs random/uncooperative scheduling interpretation)
    4. \n
  4. imo-bench-combinatorics-030 (unclear interpretation of how black cubes/pillars are chosen)
    5. \n
  5. imo-bench-combinatorics-048 (asks for “the smallest positive integers” but expects a single integer answer)
    6. \n
  6. imo-bench-combinatorics-076 (red-herring sentence can reasonably be interpreted as a typo rather than irrelevant information)
    7. \n
  7. imo-bench-combinatorics-079 (added condition fundamentally changed the problem)
    8. \n
  8. imo-bench-combinatorics-085 (added condition fundamentally changed the problem)
    9. \n
  9. imo-bench-combinatorics-097 (weakened instruction phrase sometimes leads the model to return code/formula rather than the final integer)
    10. \n
  10. imo-bench-geometry-002 (line-segment ratio appears to have been reversed during point relabelling)
    11. \n
  11. imo-bench-geometry-044 (missing diagram/condition needed to determine the position of a point)
    12. \n
  12. imo-bench-number_theory-055 (missing crucial condition from the original problem)
    13. \n
\n

Out of these, the following have been fixed in AnswerBench V2:

\n\n

The following problems remain:

\n\n

For these 5 problems, we do not claim that they are all ground-truth errors. They are mostly interpretation or phrasing issues. Our view is that these cases are less suitable for evaluating mathematical reasoning ability, because the failure may depend on how the model resolves ambiguity or whether it treats an odd sentence as a typo, a distractor, or an additional condition. Other researchers may reasonably make different choices here.

\n

3. Full explanation of problems with errors, along with their fixes (if fixed):

\n

Fixed Problems:

\n

(1) imo-bench-algebra-039 (changing numbers fundamentally changes the solution: descartes rule of signs to analyzing the discriminant of cubic equation)

\n

See appendix D.2.4. AnswerBench V2 has changed the problem text to align with the original version of the question.

\n

(2) imo-bench-combinatorics-005 (convention issue over whether 0 is a natural number)

\n

See appendix D.2.5. AnswerBench V2 has changed the ground truth answer to match most model's outputs. However, the difference in convention on the definition of natural numbers remain.

\n

(7) imo-bench-combinatorics-079 (added condition fundamentally changed the problem)

\n

AnswerBench V2 has removed the extra condition in the problem text.

\n

The problem text in the original version taken from here was:

\n
\"Determine the largest $N$ for which there exists a table $T$ of integers with $N$ rows and 16 columns that has the following properties:\n(i) Every row contains the numbers $1,2, \\ldots, 16$ in some order.\n\n(ii) For any two distinct rows $r$ and $s$, there is a column $c$ such that $|T(r, c)-T(s, c)| \\geqslant 2$.\nIn addition to the given constraints, every column in the table must contain distinct integers.\nHere $T(r, c)$ means the number at the intersection of the row $r$ and the column $c$.\"\n
\n

The problematic condition every column in the table must contain distinct integers. enforces a very strict condition on the valid configurations, and has been removed from the problem text in v2.

\n

(8) imo-bench-combinatorics-085 (added condition fundamentally changed the problem)

\n

The problem text in the original version taken from here was:

\n
\"Evan fills the fields of an $78 \\times 78$ board with numbers from 1 to $6084$, each number being used exactly once. She then counts the total number of good paths on the board. A good path is a sequence of fields of arbitrary length (including 1) such that:\n\n(i) The first field in the sequence is one that is only adjacent to fields with larger numbers, and also one that has an even row number and an even column number.\n\n(ii) Each subsequent field in the sequence is adjacent to the previous field,\n\n(iii) The numbers written on the fields in the sequence are in increasing order.\n\nTwo fields are considered adjacent if they share a common side. Find the smallest possible number of good paths Alice can obtain.\"\n
\n

The problematic condition is also one that has an even row number and an even column number. Consider this construction:

\n\n

Now consider any field with even row and even column. Every such field has at least one adjacent field that is not even-even. Since all non-even-even fields have smaller numbers than all even-even fields, each even-even field has an adjacent field with a smaller number.

\n

Therefore no even-even field is adjacent only to larger numbers. So for this construction, there are no possible starting fields for a good path. Hence the number of good paths is 0. The ground truth answer is 12013.

\n

This condition and has been removed from the problem text in v2.

\n

(10) imo-bench-geometry-002 (line-segment ratio appears to have been reversed during point relabelling)

\n

See appendix D.2.1. AnswerBench V2 has corrected the ground truth answer from 4 to 1.

\n

(11) imo-bench-geometry-044 (missing diagram/condition needed to determine the position of a point)

\n

See appendix D.2.2.

\n

The corrected problem text in AnswerBench V2 is:

\n
\"Given $PQRS$ is a parallelogram. We know that $\\angle S = 60^\\circ$, $PS = 2$ and $PQ = \\sqrt{3} + 1$. Point $N$ is the midpoint of $PS$. $E$ is a point on the segment $NQ$ which makes the segment $RE$ the angle bisector of $R$. Find the angle $REQ$ in terms of degree.\n\"\n
\n

The statement that $E$ is a point on the segment $NQ$ was not present in V1 taken from here, which confused the model into thinking that the point \\( E \\) was on the parallelogram side \\( PQ \\) instead. This has been fixed in V2.

\n

(12) imo-bench-number_theory-055 (missing crucial condition from the original problem)

\n

See appendix D.2.3. The original problem is from Day 2 Problem 6 in the Estonia Team Selection Test 2018.

\n

The original problem text is:

\n
We call a positive integer $n$ whose all digits are distinct bright, if either $n$ is a one-digit number or there exists a divisor of $n$ which can be obtained by omitting one digit of $n$ and which is bright itself. Find the largest bright positive integer. (We assume that numbers do not start with zero.)\n
\n

In the original problem, there is an extra condition that (We assume that numbers do not start with zero.), and the correct answer is 146250. This condition was removed from V1. The model decides that without this condition, 903125 is the correct answer, and we check that it is. The problem text in V2 now says that $m$ consisting of distinct nonzero digits (n has been relabelled to m), and in this case the correct answer is 14625. The ground truth answer in V2 now matches the correct answer with the updated problem text.

\n

Remaining Problems:

\n

For the following cases, we do not claim that they are definite ground-truth errors. They are mainly ambiguity or phrasing issues which, in our opinion, make the problems less suitable for evaluating mathematical reasoning ability.

\n

(3) imo-bench-combinatorics-020 (ambiguous cooperative vs. adversarial scheduling interpretation)

\n

See appendix D.2.6.

\n

There are two different interpretations to the problem, with the first one resulting in a much easier solution than the second. Given that this was a Team Selection Test problem, it is likely that the problem setters intended for the harder interpretation. However, the model does not know this context, and always defaults to the easier interpretation. We are of the opinion that the model should not be penalized for making this choice.

\n

(4) imo-bench-combinatorics-030 (unclear interpretation of how black cubes/pillars are chosen)

\n

Original Problem Text: (same in both v1 and v2)

\n
A cube with size $18 \\times 18 \\times 18$ consists of $5832$ unit cubes, all colored white. Anton and Berta play a game on this cube. Anton chooses some pillars with size $1\\times 1\\times 18$ such that no two pillars share a vertex or side, and turns all chosen unit cubes to black. Berta is allowed to choose some unit cubes and ask Anton their colors. In addition, Anton also tells Berta that he painted at least two pillars. How many unit cubes, at least, that Berta need to choose so that for any answer from Anton, Berta can always determine the black unit cubes?\n
\n

It is not at all clear how the black cubes are chosen. The two statements:

\n\n

suggests that entire pillars are chosen at once, and all cubes in a chosen pillar share the same colour. \nIn this case, the entire game can be played on a \\( 18 \\times 18 \\times 1 \\) square grid, instead of the \\( 18 \\times 18 \\times 18 \\) cube. Then, we can make the following argument based on very specific choice of black cubes painted:\nBerta must query at least 324 cubes, one for each pillar. If he did not, then there always exists the case where, Anton paints 2 far away pillars from the cube Berta did not query.\nThen, the unqueried pillar can be painted or not painted, and Bertha will never know.\nThe ground truth answer of 486 suggests that there are some additional conditions on how the black cubes are chosen, but we don't know.

\n

(5) imo-bench-combinatorics-048 (asks for “the smallest positive integers” but expects a single integer answer)

\n

Original Problem Text: (same in both v1 and v2)

\n
 Find the smallest positive integers $n$, for which the numbers in the set $S=\\{1,2, \\ldots, n\\}$ can be colored red and blue, with the following condition being satisfied: the set $S \\times S \\times S$ contains exactly 2007 ordered triples $(x, y, z)$ such that (i) $x, y, z$ are of the same color and (ii) $x+y+z$ is divisible by $n$.\n
\n

There are two integers that satisfy the conditions, 69 and 84. The ground truth is 69, the smaller one of them, but the first line asks for the smallest integers. The model sometimes outputs \\( \\boxed{69, 84} \\) which is then graded as wrong, even though the output is a reasonable response to the literal wording.

\n

(6) imo-bench-combinatorics-076 (red-herring sentence can reasonably be interpreted as a typo rather than irrelevant information)

\n

Original Problem Text: (same in both v1 and v2)

\n
The Lucas numbers $L_{0}, L_{1}, L_{2}, \\ldots$ are defined inductively by $L_{0}=2, L_{1}=1$, and $L_{n+1}=L_{n}+L_{n-1}$ for $n \\geqslant 1$. The Fibonacci numbers $F_{0}, F_{1}, F_{2}, \\ldots$ are defined inductively by $F_{0}=0, F_{1}=1$, and $F_{n+1}=F_{n}+F_{n-1}$ for $n \\geqslant 1$. Determine the smallest size of a set $S$ of integers such that for every $k=2,3, \\ldots, 125$ there exist some $x, y \\in S$ such that $x-y=F_{k}$. Also, there exist some $a, b \\in T$ for some set $T$ such that $a-b = L_{100}$.\n
\n

There is a red-herring statement that the authors of IMO_AnswerBench inserted to test the model's ability to ignore irrelevant information:

\n\n

However, the model sometimes interprets it as a typo and assumes that \\(T\\) should have been \\(S\\), so it treats the sentence as an additional condition on \\(S\\). Since models are often trained to be robust to small human errors in prompts, we think this is a plausible interpretation rather than a pure reasoning failure.

\n

(9) imo-bench-combinatorics-097 (weakened instruction phrase sometimes leads the model to return code/formula rather than the final integer)

\n

Original Problem Text: (same in both v1 and v2)

\n
\"In a circular seating arrangement at a party, there are $16$ guests sitting at the table play a game. Each guest has a unique name tag crated by the game master, and then these name tags are randomly distributed among the guests. In each of the next $n$ rounds, the guests plays the game according to the following rule:\n\ni) Any guest who has their own name tag exits the table.\nii) The remaining guests pass their name tags to the guest sitting immediately to their right.\n\nWe are interested in finding the number of ways the name tags can be distributed such that there exists at least one guest who does not leave the table after 4 rounds.\"\n
\n

The problem says:

\n

“We are interested in finding the number of ways…”

\n

This is weaker than directly asking the model to “find” or “compute” the integer. Our hypothesis is that this interacts badly with the system prompt: despite the system prompt asking for a final integer, the problem wording sometimes causes the model to output code or a formula that computes the answer rather than the numerical answer itself. We therefore classified this as more of an instruction/formatting issue than a mathematical reasoning failure.

\n

(part 2 of 2)

", "rawMarkdown": "(part 2 of 2)\n\n# Question: \n\n**AnswerBench problem-quality and filtering:** Your internal benchmark draws mostly from IMO-AnswerBench, with some problems filtered out during construction. How did you identify the problems that needed filtering, and across the broader sample you worked with, do you have an informal sense of the error rate (e.g. wrong ground-truth answers, ambiguous statements)? If you were working with AnswerBench V1, have you checked whether any of those errors are now corrected in AnswerBench V2? We've seen across maths corpus prize submissions that widely-used datasets often carry quiet ground-truth errors that can distort ablation results.\n\n# Response:\n\nThank you for raising this. We agree that quiet dataset errors can distort ablation results. \n\n1. From our observations, **we estimate the error rate to be 12 of 228 problems, or 5.26%.**\n2. Of these 12 flagged problems, **7 appear to have been fixed** in AnswerBench V2, while **5 remain**.\n3. We explain below why we considered each flagged problem problematic.\n\n## 1. Filtering Process:\n\nOur filtering process for IMO-AnswerBench was as follows. The entire dataset has 400 problems, with 100 each in number theory, geometry, combinatorics, and algebra. \n\n1. First, scan the answer column and kept only problems whose answer was a single integer. This gave 228 problems in total: 43 number theory, 63 geometry, 87 combinatorics, and 35 algebra. \n2. Then, run a majority@8 solver with 4 independent runs per problem. There were 53 problems where the solver failed in at least 1 of the 4 runs.\n3. Finally, we manually reviewed these 53 candidate hard problems and wrote notes for each one, classifying the failure as best as we could into one of several categories: genuine reasoning difficulty, parser/instruction-following issue, ambiguity, or dataset/problem-quality issue.\n4. Out of these 53 problems:\n- 12 appeared to involve issues with wording, interpretation, notation, convention, or the ground-truth answer. This gives an error rate of \\\\( \\frac{12}{228} = 5.26 \\%\\\\).\n- 16 were retained for the final internal benchmark. Together with the Norwegian integers problem from the 10 official reference problems, this gave a final benchmark of 17 problems.\n- Some failures were due to parser or formatting issues rather than reasoning errors. For example, in geometry problems where the answer is an angle, the model sometimes outputs something like \\\\( \\\\boxed\\{50^{\\circ}\\}\\\\). The parser will fail due to the angle symbol and the parser expecting a bare integer. Another example is where the model decides to include latex spaces in integer answers, like \\\\( \\\\boxed\\{ 1\\\\,432\\} \\\\), whose string output from the model is `1\\\\,432`, and calling `int('1\\\\,432')` in python will throw an exception. The solver will catch it and return `N/A` as the answer. All such cases were manually identified and fixed in our answer parser.\n- Finally, there were some problems where we could not confidently understand or verify the solution even after studying the official solution. We chose not to include these in the benchmark. Our priority was to make the curated benchmark as clean as possible, rather than to maximize its size.\n\n## 2. Checks against [AnswerBench V2](https://github.com/google-deepmind/superhuman/blob/main/imobench/answerbench_v2.csv):\n\nOur full list of 12 problems that we think contains issues are:\n1.\t`imo-bench-algebra-039` (changing numbers fundamentally changes the solution: descartes rule of signs to analyzing the discriminant of cubic equation)\n2.\t`imo-bench-combinatorics-005` (convention issue over whether 0 is a natural number)\n3.\t`imo-bench-combinatorics-020` (ambiguous cooperative vs random/uncooperative scheduling interpretation)\n4.\t`imo-bench-combinatorics-030` (unclear interpretation of how black cubes/pillars are chosen)\n5.\t`imo-bench-combinatorics-048` (asks for “the smallest positive integers” but expects a single integer answer)\n6.\t`imo-bench-combinatorics-076` (red-herring sentence can reasonably be interpreted as a typo rather than irrelevant information)\n7.\t`imo-bench-combinatorics-079` (added condition fundamentally changed the problem)\n8.\t`imo-bench-combinatorics-085` (added condition fundamentally changed the problem)\n9.\t`imo-bench-combinatorics-097` (weakened instruction phrase sometimes leads the model to return code/formula rather than the final integer)\n10.\t`imo-bench-geometry-002` (line-segment ratio appears to have been reversed during point relabelling)\n11.\t`imo-bench-geometry-044` (missing diagram/condition needed to determine the position of a point)\n12.\t`imo-bench-number_theory-055` (missing crucial condition from the original problem)\n\nOut of these, **the following have been fixed** in [AnswerBench V2](https://github.com/google-deepmind/superhuman/blob/main/imobench/answerbench_v2.csv):\n\n- (1) `imo-bench-algebra-039`\n- (2) `imo-bench-combinatorics-005`\n- (7) `imo-bench-combinatorics-079` \n- (8) `imo-bench-combinatorics-085`\n- (10) `imo-bench-geometry-002`\n- (11) `imo-bench-geometry-044`\n- (12) `imo-bench-number_theory-055`\n\n**The following problems remain:**\n\n- (3) `imo-bench-combinatorics-020`\n- (4) `imo-bench-combinatorics-030`\n- (5) `imo-bench-combinatorics-048`\n- (6) `imo-bench-combinatorics-076`\n- (9) `imo-bench-combinatorics-097`\n\nFor these 5 problems, we do not claim that they are all ground-truth errors. They are mostly interpretation or phrasing issues. Our view is that these cases are less suitable for evaluating mathematical reasoning ability, because the failure may depend on how the model resolves ambiguity or whether it treats an odd sentence as a typo, a distractor, or an additional condition. Other researchers may reasonably make different choices here.\n\n## 3. Full explanation of problems with errors, along with their fixes (if fixed):\n\n### Fixed Problems:\n#### (1) `imo-bench-algebra-039` (changing numbers fundamentally changes the solution: descartes rule of signs to analyzing the discriminant of cubic equation)\nSee appendix D.2.4. [AnswerBench V2](https://github.com/google-deepmind/superhuman/blob/main/imobench/answerbench_v2.csv) has changed the problem text to align with the original version of the question.\n\n#### (2) `imo-bench-combinatorics-005` (convention issue over whether 0 is a natural number)\nSee appendix D.2.5. [AnswerBench V2](https://github.com/google-deepmind/superhuman/blob/main/imobench/answerbench_v2.csv) has changed the ground truth answer to match most model's outputs. However, the difference in convention on the definition of natural numbers remain.\n\n#### (7) `imo-bench-combinatorics-079` (added condition fundamentally changed the problem)\n[AnswerBench V2](https://github.com/google-deepmind/superhuman/blob/main/imobench/answerbench_v2.csv) has removed the extra condition in the problem text.\n\nThe problem text in [the original version taken from here was](https://huggingface.co/datasets/Hwilner/imo-answerbench):\n```\n\"Determine the largest $N$ for which there exists a table $T$ of integers with $N$ rows and 16 columns that has the following properties:\n(i) Every row contains the numbers $1,2, \\ldots, 16$ in some order.\n\n(ii) For any two distinct rows $r$ and $s$, there is a column $c$ such that $|T(r, c)-T(s, c)| \\geqslant 2$.\nIn addition to the given constraints, every column in the table must contain distinct integers.\nHere $T(r, c)$ means the number at the intersection of the row $r$ and the column $c$.\"\n```\nThe problematic condition `every column in the table must contain distinct integers.` enforces a very strict condition on the valid configurations, and has been removed from the problem text in v2.\n\n#### (8) `imo-bench-combinatorics-085` (added condition fundamentally changed the problem)\nThe problem text in [the original version taken from here was](https://huggingface.co/datasets/Hwilner/imo-answerbench):\n```\n\"Evan fills the fields of an $78 \\times 78$ board with numbers from 1 to $6084$, each number being used exactly once. She then counts the total number of good paths on the board. A good path is a sequence of fields of arbitrary length (including 1) such that:\n\n(i) The first field in the sequence is one that is only adjacent to fields with larger numbers, and also one that has an even row number and an even column number.\n\n(ii) Each subsequent field in the sequence is adjacent to the previous field,\n\n(iii) The numbers written on the fields in the sequence are in increasing order.\n\nTwo fields are considered adjacent if they share a common side. Find the smallest possible number of good paths Alice can obtain.\"\n\n```\nThe problematic condition is `also one that has an even row number and an even column number.` Consider this construction:\n- There are \\\\( 39 \\times 39 = 1521 \\\\) fields whose row and column numbers are both even.\n- Fill all **non-even-even** fields with \\\\( 1, 2, ..., 4563\\\\) in any order.\n- Fill all **even-even** fields with \\\\( 4564, 4565, ..., 6084\\\\) in any order.\n\nNow consider any field with even row and even column. Every such field has at least one adjacent field that is not even-even. Since all non-even-even fields have smaller numbers than all even-even fields, each even-even field has an adjacent field with a smaller number.\n\nTherefore no even-even field is adjacent only to larger numbers. So for this construction, there are no possible starting fields for a good path. Hence the number of good paths is **0**. The ground truth answer is 12013.\n\n\nThis condition and has been removed from the problem text in v2.\n\n#### (10) `imo-bench-geometry-002` (line-segment ratio appears to have been reversed during point relabelling)\nSee appendix D.2.1. [AnswerBench V2](https://github.com/google-deepmind/superhuman/blob/main/imobench/answerbench_v2.csv) has corrected the ground truth answer from 4 to 1.\n\n#### (11) `imo-bench-geometry-044` (missing diagram/condition needed to determine the position of a point)\nSee appendix D.2.2. \n\nThe corrected problem text in [AnswerBench V2](https://github.com/google-deepmind/superhuman/blob/main/imobench/answerbench_v2.csv) is:\n\n```\n\"Given $PQRS$ is a parallelogram. We know that $\\angle S = 60^\\circ$, $PS = 2$ and $PQ = \\sqrt{3} + 1$. Point $N$ is the midpoint of $PS$. $E$ is a point on the segment $NQ$ which makes the segment $RE$ the angle bisector of $R$. Find the angle $REQ$ in terms of degree.\n\"\n```\n\nThe statement that `$E$ is a point on the segment $NQ$` was not present in [V1 taken from here](https://huggingface.co/datasets/Hwilner/imo-answerbench), which confused the model into thinking that the point \\\\( E \\\\) was on the parallelogram side \\\\( PQ \\\\) instead. This has been fixed in V2.\n\n#### (12) `imo-bench-number_theory-055` (missing crucial condition from the original problem)\nSee appendix D.2.3. The original problem is from [Day 2 Problem 6 in the Estonia Team Selection Test 2018](https://artofproblemsolving.com/community/c1238516_2018_estonia_team_selection_test).\n\nThe original problem text is:\n\n```\nWe call a positive integer $n$ whose all digits are distinct bright, if either $n$ is a one-digit number or there exists a divisor of $n$ which can be obtained by omitting one digit of $n$ and which is bright itself. Find the largest bright positive integer. (We assume that numbers do not start with zero.)\n```\n\nIn the original problem, there is an extra condition that `(We assume that numbers do not start with zero.)`, and the correct answer is `146250`. This condition was removed from V1. The model decides that without this condition, `903125` is the correct answer, and we check that it is. The problem text in V2 now says that ` $m$ consisting of distinct nonzero digits` (n has been relabelled to m), and in this case the correct answer is `14625`. The ground truth answer in V2 now matches the correct answer with the updated problem text.\n\n\n### Remaining Problems:\n\nFor the following cases, we do not claim that they are definite ground-truth errors. They are mainly ambiguity or phrasing issues which, in our opinion, make the problems less suitable for evaluating mathematical reasoning ability.\n\n#### (3) `imo-bench-combinatorics-020` (ambiguous cooperative vs. adversarial scheduling interpretation)\nSee appendix D.2.6.\n\nThere are two different interpretations to the problem, with the first one resulting in a much easier solution than the second. Given that this was a Team Selection Test problem, it is likely that the problem setters intended for the harder interpretation. However, the model does not know this context, and always defaults to the easier interpretation. We are of the opinion that the model should not be penalized for making this choice.\n\n#### (4) `imo-bench-combinatorics-030` (unclear interpretation of how black cubes/pillars are chosen)\n**Original Problem Text**: (same in both v1 and v2)\n```\nA cube with size $18 \\times 18 \\times 18$ consists of $5832$ unit cubes, all colored white. Anton and Berta play a game on this cube. Anton chooses some pillars with size $1\\times 1\\times 18$ such that no two pillars share a vertex or side, and turns all chosen unit cubes to black. Berta is allowed to choose some unit cubes and ask Anton their colors. In addition, Anton also tells Berta that he painted at least two pillars. How many unit cubes, at least, that Berta need to choose so that for any answer from Anton, Berta can always determine the black unit cubes?\n```\n\nIt is not at all clear how the black cubes are chosen. The two statements:\n- ```\"Anton chooses some pillars with size $1\\times 1\\times 18$ such that no two pillars share a vertex or side, and turns all chosen unit cubes to black.\"```\n\n- ```\"In addition, Anton also tells Berta that he painted at least two pillars.\"```\n\nsuggests that entire pillars are chosen at once, and all cubes in a chosen pillar share the same colour. \nIn this case, the entire game can be played on a \\\\( 18 \\times 18 \\times 1 \\\\) square grid, instead of the \\\\( 18 \\times 18 \\times 18 \\\\) cube. Then, we can make the following argument based on very specific choice of black cubes painted:\nBerta must query at least **324** cubes, one for each pillar. If he did not, then there always exists the case where, Anton paints 2 far away pillars from the cube Berta did not query.\nThen, the unqueried pillar can be painted or not painted, and Bertha will never know.\nThe ground truth answer of 486 suggests that there are some additional conditions on how the black cubes are chosen, but we don't know.\n\n\n\n#### (5) `imo-bench-combinatorics-048` (asks for “the smallest positive integers” but expects a single integer answer)\n\n**Original Problem Text**: (same in both v1 and v2)\n```\n Find the smallest positive integers $n$, for which the numbers in the set $S=\\{1,2, \\ldots, n\\}$ can be colored red and blue, with the following condition being satisfied: the set $S \\times S \\times S$ contains exactly 2007 ordered triples $(x, y, z)$ such that (i) $x, y, z$ are of the same color and (ii) $x+y+z$ is divisible by $n$.\n```\n\nThere are two integers that satisfy the conditions, 69 and 84. The ground truth is 69, the smaller one of them, but the first line asks for the smallest **integers**. The model sometimes outputs \\\\( \\\\boxed\\{69, 84\\} \\\\) which is then graded as wrong, even though the output is a reasonable response to the literal wording.\n\n#### (6) `imo-bench-combinatorics-076` (red-herring sentence can reasonably be interpreted as a typo rather than irrelevant information)\n**Original Problem Text**: (same in both v1 and v2)\n```\nThe Lucas numbers $L_{0}, L_{1}, L_{2}, \\ldots$ are defined inductively by $L_{0}=2, L_{1}=1$, and $L_{n+1}=L_{n}+L_{n-1}$ for $n \\geqslant 1$. The Fibonacci numbers $F_{0}, F_{1}, F_{2}, \\ldots$ are defined inductively by $F_{0}=0, F_{1}=1$, and $F_{n+1}=F_{n}+F_{n-1}$ for $n \\geqslant 1$. Determine the smallest size of a set $S$ of integers such that for every $k=2,3, \\ldots, 125$ there exist some $x, y \\in S$ such that $x-y=F_{k}$. Also, there exist some $a, b \\in T$ for some set $T$ such that $a-b = L_{100}$.\n```\nThere is a red-herring statement that the authors of IMO_AnswerBench inserted to test the model's ability to ignore irrelevant information:\n- `Also, there exist some $a, b \\in T$ for some set $T$ such that $a-b = L_{100}$.`\n\nHowever, the model sometimes interprets it as a typo and assumes that \\\\(T\\\\) should have been \\\\(S\\\\), so it treats the sentence as an additional condition on \\\\(S\\\\). Since models are often trained to be robust to small human errors in prompts, we think this is a plausible interpretation rather than a pure reasoning failure.\n\n\n#### (9) `imo-bench-combinatorics-097` (weakened instruction phrase sometimes leads the model to return code/formula rather than the final integer)\n**Original Problem Text**: (same in both v1 and v2)\n```\n\"In a circular seating arrangement at a party, there are $16$ guests sitting at the table play a game. Each guest has a unique name tag crated by the game master, and then these name tags are randomly distributed among the guests. In each of the next $n$ rounds, the guests plays the game according to the following rule:\n\ni) Any guest who has their own name tag exits the table.\nii) The remaining guests pass their name tags to the guest sitting immediately to their right.\n\nWe are interested in finding the number of ways the name tags can be distributed such that there exists at least one guest who does not leave the table after 4 rounds.\"\n\n```\nThe problem says:\n\n`“We are interested in finding the number of ways…”`\n\nThis is weaker than directly asking the model to “find” or “compute” the integer. Our hypothesis is that this interacts badly with the system prompt: despite the system prompt asking for a final integer, the problem wording sometimes causes the model to output code or a formula that computes the answer rather than the numerical answer itself. We therefore classified this as more of an instruction/formatting issue than a mathematical reasoning failure.\n\n(part 2 of 2)" } ] }, { "id": 3456823, "postDate": "2026-05-12T16:30:55.593Z", "content": "

Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.

\n

Your submission is already well structured and pleasant to read, and the plots do a good job of illustrating your process. I have a few questions that may help clarify the methodology further:

\n
    \n

  1. Context Management: Given the limited context window, how did you avoid unnecessary token usage during RSA? Running many parallel sessions likely produces a large amount of text per round, especially if intermediate outputs are reused as hints in later contexts. How did you decide what information to retain, compress, or discard?

    2. \n

  2. Difficulty Buckets: Did you run any experiments to verify that the difficulty buckets were representative of the public/private sets, or were these buckets derived primarily from the previously shared X post?

    3. \n

  3. Answer Collapse: Since RSA propagates hints to future trajectories, how did you mitigate the risk of convergence toward an incorrect majority answer? In particular, if early generations contain many flawed solutions, using them as contextual hints in later iterations could potentially steer subsequent trajectories in the wrong direction. How did you address this failure mode?

    4. \n
\n

Looking forward to your response :)

", "rawMarkdown": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the[ updated discussion post.](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/689703)\n\nYour submission is already well structured and pleasant to read, and the plots do a good job of illustrating your process. I have a few questions that may help clarify the methodology further:\n\n1. **Context Management**: Given the limited context window, how did you avoid unnecessary token usage during RSA? Running many parallel sessions likely produces a large amount of text per round, especially if intermediate outputs are reused as hints in later contexts. How did you decide what information to retain, compress, or discard?\n\n2. **Difficulty Buckets**: Did you run any experiments to verify that the difficulty buckets were representative of the public/private sets, or were these buckets derived primarily from the previously shared X post?\n\n3. **Answer Collapse**: Since RSA propagates hints to future trajectories, how did you mitigate the risk of convergence toward an incorrect majority answer? In particular, if early generations contain many flawed solutions, using them as contextual hints in later iterations could potentially steer subsequent trajectories in the wrong direction. How did you address this failure mode?\n\nLooking forward to your response :)\n\n\n", "votes": 1, "replies": [ { "id": 3457556, "postDate": "2026-05-13T23:12:38.340Z", "content": "

Hi @philipvonderlind,

\n

Thank you for reading my writeup and for the thoughtful questions! I will split my response into 3 separate replies, one for each question.

\n

(part 1 of 3)

\n

Question:

\n

Context Management: Given the limited context window, how did you avoid unnecessary token usage during RSA? Running many parallel sessions likely produces a large amount of text per round, especially if intermediate outputs are reused as hints in later contexts. How did you decide what information to retain, compress, or discard?

\n

Response:

\n

Context Management: Context management strategies differ by model used. We structure our response into 2 parts:

\n\n

1. What we did with gpt-oss-120b for AIMO 3:

\n

For AIMO3, we relied on a feature that is specific to the gpt-oss-120b / Harmony-style output format. The model does not produce a single unstructured blob of text. Instead, its generation is naturally separated into reasoning messages, tool calls, tool outputs, and a final assistant message. Schematically, one generation looks like:

\n
<reasoning>\n<python_tool_call>\n<reasoning>\n<python_tool_call>\n<reasoning>\n...\n... (possibly 100+ tool call rounds)\n...\n<reasoning>\n<python_tool_call>\n<reasoning>\n<python_tool_call>\n<reasoning>\n<final_message>\n
\n

The important point is that the final message is already a compressed summary of the model’s previous reasoning and tool use. It usually contains a short solution summary and the final boxed answer. For the next round of RSA, we did not include the intermediate reasoning traces, full Python tool logs, code snippets, failed attempts, or repeated derivations. We only used the final assistant message from each candidate solution as the hint for the next RSA round.

\n

In practice, these final messages were almost always shorter than 1000 tokens. Since one RSA round used 8 candidate solutions, the hint section contributed roughly at most 8000 tokens, plus the original problem statement and some small formatting overhead. This was far below the maximum context max_model_len = 81920 used in our final system, so context length itself was not the bottleneck in our experiments.

\n

Another important detail is that RSA did not require us to keep accumulating all previous generations forever. Each new RSA prompt was mainly built from the candidate summaries from the previous round. Earlier rounds influenced the next round indirectly, because their useful information had already been distilled into the previous round’s final messages. This kept the prompt size roughly bounded from round to round, rather than growing linearly with the total number of generations produced so far.

\n

This also explains why we avoided terminating generation immediately after the first \\boxed{...} appeared. Some public notebooks used early stream stopping as a speed optimization, but for RSA this is dangerous. If we stop at the first boxed answer, we may never receive the model’s final message. That final message is exactly the compact object we want to pass into the next RSA round. So for our pipeline, preserving the final message was more important than saving a small amount of generation time.

\n

Overall, for gpt-oss-120b, we did not need a complicated context compression algorithm. The context management strategy was simply:

\n\n

This was enough for all of our runs on the reference benchmark and internal benchmark. Since the resulting contexts were still comfortably below the configured context length, we did not optimize this part further.

\n

2. What to do in other cases when the above is not possible:

\n

The context-management strategy above relies on three conditions:

\n
    \n
  1. The model has a structured output format.
    2. \n
  2. The final message is a reasonably short summary of the full reasoning trace.
    3. \n
  3. Only a limited number of hints, in our case \\(n=8\\), are passed into the next RSA round.
    4. \n
\n

All three conditions held for gpt-oss-120b in AIMO3. However, in other settings, these assumptions may fail. For example:

\n
    \n
  1. A model may not separate reasoning from the final answer.
    2. \n
  2. Its final response may be too verbose.
    3. \n
  3. The specified RSA configuration uses a much larger number of parallel candidates.
    4. \n
\n

In those cases, a more explicit context-management strategy is needed.

\n

One natural strategy is the one described in the original RSA paper, Recursive Self-Aggregation Unlocks Deep Thinking in Large Language Models. The key idea is that the next RSA round does not need to condition every generation on all \\(n\\) previous solutions. Instead, for each generation in the next round, we randomly sample only a smaller subset of previous solutions.

\n

More concretely, suppose each RSA round produces \\(n\\) candidate solutions. Rather than passing all \\(n\\) candidates into every next-round prompt, we choose only \\(\\frac{n}{k}\\) candidates for each next-round generation, where \\(k\\) is a tunable parameter. Typically, \\(k\\) lies between \\(2\\) and \\(\\frac{n}{2}\\). Larger values of \\(k\\) correspond to smaller sampled subsets, and therefore shorter prompts.

\n

The expanded context-management strategy is:

\n\n
   prompt_tokens + sampled_hint_tokens + estimated_generation_tokens + buffer < max_context_length\n
\n

Here, estimated_generation_tokens can be inferred from the maximum or high-percentile generation length observed in earlier attempts.

\n\n

The original RSA paper includes the following diagram for the case \\( n=8 \\) and \\( k=2 \\), where each next-round generation sees 4 sampled hints rather than all 8:\n\"Recursive

\n

The main point is that RSA does not require the context size to grow uncontrollably. The sampling mechanism provides a direct knob for controlling prompt length. Smaller sampled subsets reduce prefill cost and help keep each prompt within the model’s context window, while still allowing information from previous candidates to propagate across RSA rounds.

\n

The trade-off is that larger values of \\(k\\), and hence smaller sampled subsets, make each individual next-round generation less informed. In particular, a sampled subset may not contain the correct solution, even if the correct solution appeared somewhere in the previous round. This can increase the probability that a given generation follows an incorrect line of reasoning. As a result, more RSA rounds may be needed for the correct solution to propagate through the population, increasing the sequential depth and wall-clock time per problem.

\n

However, this does not necessarily break RSA. If the correct solution appears in the previous round, then under random subsampling it will usually appear in multiple next-round prompts across the \\(n\\) new generations. Those generations can then amplify or refine it, and the information can continue spreading in later rounds. This is the mechanism by which RSA can still converge even when each individual generation only sees a subset of the previous candidates. The cost is mainly slower propagation, not necessarily loss of the signal entirely.

\n

In our AIMO3 implementation, this more complicated sampling strategy was not necessary because the final messages from gpt-oss-120b were already short enough. Even passing all 8 final messages into the next round typically added fewer than 8000 tokens, which was comfortably below our configured max_model_len = 81920. But for models with longer outputs or larger values of \\(n\\), this subset-sampling version of RSA would be the natural way to manage context length.

\n

(part 1 of 3)

", "rawMarkdown": "Hi @philipvonderlind,\n\nThank you for reading my writeup and for the thoughtful questions! I will split my response into 3 separate replies, one for each question.\n\n(part 1 of 3)\n\n# Question: \n\n**Context Management:** Given the limited context window, how did you avoid unnecessary token usage during RSA? Running many parallel sessions likely produces a large amount of text per round, especially if intermediate outputs are reused as hints in later contexts. How did you decide what information to retain, compress, or discard?\n\n# Response:\n\n**Context Management:** Context management strategies differ by model used. We structure our response into 2 parts: \n- **(i)** What we did specifically for `gpt-oss-120b` for AIMO 3, and \n- **(ii)** what to do for other models and in cases where the previous context management strategy is not possible.\n\n## 1. What we did with gpt-oss-120b for AIMO 3:\n\nFor AIMO3, we relied on a feature that is specific to the `gpt-oss-120b` / Harmony-style output format. The model does not produce a single unstructured blob of text. Instead, its generation is naturally separated into reasoning messages, tool calls, tool outputs, and a final assistant message. Schematically, one generation looks like:\n\n```text\n\n\n\n\n\n...\n... (possibly 100+ tool call rounds)\n...\n\n\n\n\n\n\n```\n\nThe important point is that the final message is already a compressed summary of the model’s previous reasoning and tool use. It usually contains a short solution summary and the final boxed answer. For the next round of RSA, we did not include the intermediate reasoning traces, full Python tool logs, code snippets, failed attempts, or repeated derivations. We only used the final assistant message from each candidate solution as the hint for the next RSA round.\n\nIn practice, these final messages were almost always shorter than 1000 tokens. Since one RSA round used 8 candidate solutions, the hint section contributed roughly at most 8000 tokens, plus the original problem statement and some small formatting overhead. This was far below the maximum context `max_model_len = 81920` used in our final system, so context length itself was not the bottleneck in our experiments.\n\nAnother important detail is that RSA did not require us to keep accumulating all previous generations forever. Each new RSA prompt was mainly built from the candidate summaries from the previous round. Earlier rounds influenced the next round indirectly, because their useful information had already been distilled into the previous round’s final messages. This kept the prompt size roughly bounded from round to round, rather than growing linearly with the total number of generations produced so far.\n\nThis also explains why we avoided terminating generation immediately after the first `\\boxed{...}` appeared. Some public notebooks used early stream stopping as a speed optimization, but for RSA this is dangerous. If we stop at the first boxed answer, we may never receive the model’s final message. That final message is exactly the compact object we want to pass into the next RSA round. So for our pipeline, preserving the final message was more important than saving a small amount of generation time.\n\nOverall, for gpt-oss-120b, we did not need a complicated context compression algorithm. The context management strategy was simply:\n\n- Let each candidate solution finish normally.\n- Parse the final answer.\n- Store only the final assistant message as the reusable RSA hint.\n- Discard intermediate reasoning traces, tool-call transcripts, and raw scratch work from the next prompt.\n\nThis was enough for all of our runs on the reference benchmark and internal benchmark. Since the resulting contexts were still comfortably below the configured context length, we did not optimize this part further.\n\n\n## 2. What to do in other cases when the above is not possible:\n\nThe context-management strategy above relies on three conditions:\n\n1. The model has a structured output format.\n2. The final message is a reasonably short summary of the full reasoning trace.\n3. Only a limited number of hints, in our case \\\\(n=8\\\\), are passed into the next RSA round.\n\nAll three conditions held for `gpt-oss-120b` in AIMO3. However, in other settings, these assumptions may fail. For example:\n1. A model may not separate reasoning from the final answer.\n2. Its final response may be too verbose.\n3. The specified RSA configuration uses a much larger number of parallel candidates.\n\nIn those cases, a more explicit context-management strategy is needed.\n\nOne natural strategy is the one described in the original RSA paper, [Recursive Self-Aggregation Unlocks Deep Thinking in Large Language Models](https://arxiv.org/abs/2509.26626). The key idea is that the next RSA round does **not** need to condition every generation on all \\\\(n\\\\) previous solutions. Instead, for each generation in the next round, we randomly sample only a smaller subset of previous solutions.\n\nMore concretely, suppose each RSA round produces \\\\(n\\\\) candidate solutions. Rather than passing all \\\\(n\\\\) candidates into every next-round prompt, we choose only \\\\(\\frac{n}{k}\\\\) candidates for each next-round generation, where \\\\(k\\\\) is a tunable parameter. Typically, \\\\(k\\\\) lies between \\\\(2\\\\) and \\\\(\\frac{n}{2}\\\\). Larger values of \\\\(k\\\\) correspond to smaller sampled subsets, and therefore shorter prompts.\n\n**The expanded context-management strategy is:**\n- 1. Let each of the \\\\(n\\\\) candidate solutions finish normally.\n- 2. Parse the final answer from each candidate.\n- 3. Decide what representation of each candidate should be retained:\n - For models with useful final summaries, such as `gpt-oss-120b`, DeepSeek-style models, or Nemotron Cascade-style models, keep only the final message.\n - If the final summary is too lossy or unreliable, keep a longer solution trace, possibly including selected reasoning steps or tool-call outputs.\n - If the full trace is too long, use a separate compression step (such as a separate model call) to summarize the solution before it is reused as a hint.\n- 4. Estimate the token cost of the next round. A useful rule of thumb is to ensure that\n\n```text\n prompt_tokens + sampled_hint_tokens + estimated_generation_tokens + buffer < max_context_length\n```\nHere, `estimated_generation_tokens` can be inferred from the maximum or high-percentile generation length observed in earlier attempts.\n\n- 5. Tune \\\\( k \\\\) so that each next-round prompt fits comfortably within the model’s context window.\n- 6. For each of the next n generations, randomly sample only \\\\( \\frac{n}{k} \\\\) previous candidate solutions as hints, instead of passing all n previous candidates.\n\nThe [original RSA paper](https://arxiv.org/abs/2509.26626) includes the following diagram for the case \\\\( n=8 \\\\) and \\\\( k=2 \\\\), where each next-round generation sees 4 sampled hints rather than all 8:\n![Recursive Self Aggregation with 8 candidate solutions, sampling 4 at a time for the next round](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F15328099%2F38d9ede917cdec083212b2f3b991acfb%2FRSA_image_3.png?generation=1778621836496850&alt=media)\n\nThe main point is that RSA does not require the context size to grow uncontrollably. The sampling mechanism provides a direct knob for controlling prompt length. Smaller sampled subsets reduce prefill cost and help keep each prompt within the model’s context window, while still allowing information from previous candidates to propagate across RSA rounds.\n\nThe trade-off is that larger values of \\\\(k\\\\), and hence smaller sampled subsets, make each individual next-round generation less informed. In particular, a sampled subset may not contain the correct solution, even if the correct solution appeared somewhere in the previous round. This can increase the probability that a given generation follows an incorrect line of reasoning. As a result, more RSA rounds may be needed for the correct solution to propagate through the population, increasing the sequential depth and wall-clock time per problem.\n\nHowever, this does not necessarily break RSA. If the correct solution appears in the previous round, then under random subsampling it will usually appear in multiple next-round prompts across the \\\\(n\\\\) new generations. Those generations can then amplify or refine it, and the information can continue spreading in later rounds. This is the mechanism by which RSA can still converge even when each individual generation only sees a subset of the previous candidates. The cost is mainly slower propagation, not necessarily loss of the signal entirely.\n\nIn our AIMO3 implementation, this more complicated sampling strategy was not necessary because the final messages from `gpt-oss-120b` were already short enough. Even passing all 8 final messages into the next round typically added fewer than 8000 tokens, which was comfortably below our configured `max_model_len = 81920`. But for models with longer outputs or larger values of \\\\(n\\\\), this subset-sampling version of RSA would be the natural way to manage context length.\n\n(part 1 of 3)", "votes": 1 }, { "id": 3457557, "postDate": "2026-05-13T23:15:05.837Z", "content": "

(part 2 of 3)

\n

Question:

\n

Difficulty Buckets: Did you run any experiments to verify that the difficulty buckets were representative of the public/private sets, or were these buckets derived primarily from the previously shared X post?

\n

Response:

\n

Difficulty Buckets: We did not run experiments to verify how representative these buckets were of the public or private test sets, because contestants did not have access to those problems. Instead, the bucket construction was based on two sources of information.

\n

1. Runtime and RSA behaviour from available benchmarks.

\n

We used the 10 official reference problems and our small internal benchmark, whose construction process is described in Section 5, to understand how the system behaved on problems of different apparent difficulty. These runs helped us estimate runtime on easier problems, observe RSA behaviour on harder problems, and decide how many broad difficulty buckets were useful for scheduler design.

\n

2. Pass@k information shared by AIMO.

\n

The X post was used to estimate the approximate number of problems in each difficulty bucket. We interpreted the reported pass@k values cumulatively:

\n
    \n
  • problems solved at pass@1 were assigned to Bucket A;
    • \n
  • the difference between pass@3 and pass@1 was assigned to Bucket B;
    • \n
  • the difference between pass@5 and pass@3 was assigned to Bucket C;
    • \n
  • the difference between pass@20 and pass@5 was assigned to Bucket D;
    • \n
  • the difference between pass@100 and pass@20 was assigned to Bucket E, together with any remaining problems.
    • \n
\n

This bucket construction should therefore be understood as a heuristic, not as a validated estimate of the true public/private distributions. Its purpose was not to precisely classify every problem, but to give the scheduler a rough prior over how much useful compute different kinds of problems might require.

\n

Although we could not directly validate the bucket distribution against the hidden leaderboard problems, the final leaderboard results provide some indirect evidence that the scheduler prior was reasonably well calibrated. Across public submissions, our system scored in the range of 40–42, and the final private leaderboard score was 41.5. The close agreement between these numbers suggests that the difficulty assumptions were at least useful enough to produce stable behaviour across the public and private splits, even if they should not be interpreted as an exact reconstruction of the hidden problem distribution.

\n

(part 2 of 3)

", "rawMarkdown": "(part 2 of 3)\n\n# Question: \n\n**Difficulty Buckets:** Did you run any experiments to verify that the difficulty buckets were representative of the public/private sets, or were these buckets derived primarily from the previously shared X post?\n\n# Response:\n\n**Difficulty Buckets:** We did not run experiments to verify how representative these buckets were of the public or private test sets, because contestants did not have access to those problems. Instead, the bucket construction was based on **two sources of information.**\n\n## 1. **Runtime and RSA behaviour from available benchmarks.** \n We used the 10 official reference problems and our small internal benchmark, whose construction process is described in Section 5, to understand how the system behaved on problems of different apparent difficulty. These runs helped us estimate runtime on easier problems, observe RSA behaviour on harder problems, and decide how many broad difficulty buckets were useful for scheduler design.\n\n## 2. **Pass@k information shared by AIMO.** \n The [X post](https://x.com/AIMOprize/status/2039441022996934783/) was used to estimate the approximate number of problems in each difficulty bucket. We interpreted the reported pass@k values cumulatively:\n - problems solved at pass@1 were assigned to Bucket A;\n - the difference between pass@3 and pass@1 was assigned to Bucket B;\n - the difference between pass@5 and pass@3 was assigned to Bucket C;\n - the difference between pass@20 and pass@5 was assigned to Bucket D;\n - the difference between pass@100 and pass@20 was assigned to Bucket E, together with any remaining problems.\n\nThis bucket construction should therefore be understood as a heuristic, not as a validated estimate of the true public/private distributions. Its purpose was not to precisely classify every problem, but to give the scheduler a rough prior over how much useful compute different kinds of problems might require.\n\nAlthough we could not directly validate the bucket distribution against the hidden leaderboard problems, the final leaderboard results provide some indirect evidence that the scheduler prior was reasonably well calibrated. Across public submissions, our system scored in the range of 40–42, and the final private leaderboard score was 41.5. The close agreement between these numbers suggests that the difficulty assumptions were at least useful enough to produce stable behaviour across the public and private splits, even if they should not be interpreted as an exact reconstruction of the hidden problem distribution.\n\n(part 2 of 3)" }, { "id": 3457558, "postDate": "2026-05-13T23:17:53.437Z", "content": "

(part 3 of 3)

\n

Question:

\n

Answer Collapse: Since RSA propagates hints to future trajectories, how did you mitigate the risk of convergence toward an incorrect majority answer? In particular, if early generations contain many flawed solutions, using them as contextual hints in later iterations could potentially steer subsequent trajectories in the wrong direction. How did you address this failure mode?

\n

Response:

\n

Answer Collapse: There is no real way to completely prevent this failure mode, short of knowing the ground truth answer directly.

\n

There are two different concerns raised in this question:

\n
    \n
  • (i) How to prevent wrong answers from propagating, and
    • \n
  • (ii) how to identify when the generations have \"converged\" to the correct answer.
    • \n
\n

We were quite concerned about both at the beginning. However, after experimenting with gpt-oss-120b, our view became more nuanced. We found three things:

\n
    \n
  • (i) Propagation: Once RSA is triggered, answer collapse was less severe for gpt-oss-120b than we initially expected;
    • \n
  • (ii) Propagation: There were simple mitigation steps that reduced the risk further;
    • \n
  • (iii) Convergence: The harder practical problem was deciding when the answer distribution had genuinely “converged” to the correct answer and RSA should stop.
    • \n
\n

1. This was less severe than expected

\n

RSA with gpt-oss-120b is able to recover minority correct solutions even among a majority of wrong ones.

\n

This issue is discussed in the V1 paper, V_1: Unifying Generation and Self-Verification for Parallel Reasoners, which we cited in the ablation study. Near the bottom of page 4, the authors write that RSA can “discard or degrade correct outlier solutions during refinement.” Their V1-Infer method was partly designed to address this weakness by ranking candidates through pairwise comparisons instead of relying on refinement alone.

\n

However, our experiments suggested that this failure mode is model-dependent. The V1 paper reports this issue for smaller models such as gpt-oss-20b and Qwen3-4B-Instruct. In our experiments, gpt-oss-120b was much less susceptible to it. Examples B.1, B.2, and B.6 in Appendix B show cases where RSA with gpt-oss-120b was able to rescue a minority answer from the initial candidate pool, even when many early generations were flawed.

\n

This led us to the following working hypothesis: for gpt-oss-120b, the important factor was often whether the correct solution, or at least a useful partial route to it, appeared somewhere in the candidate pool. It did not always need to be the majority answer initially. Once the correct idea appeared in one or more hints, later RSA rounds could usually amplify it.

\n

The ablation study in Section 6 also supports our decision not to switch to V1-Infer. We made several public leaderboard submissions using V1-Infer, and they scored in the 37–39 range, below the RSA-based submissions. So while V1-Infer is a reasonable strategy for mitigating answer collapse in principle, in our setting it did not improve the final leaderboard score.

\n

2. Mitigation measures

\n

Even though answer collapse was not as severe as expected, we still added two safeguards.

\n

First, the RSA prompt explicitly framed previous solutions as untrusted hints, not as ground truth. The RSA prompt (in appendix E.4) has a part that says:

\n
You are given a math problem, and some solution hints.\nSome hints are incorrect and contain errors.\"\n
\n

This wording was intentional. We wanted the model to solve the original problem again, while using previous candidates only as potentially useful evidence.

\n

Second, we reserved a small number of fresh attempts in later RSA rounds. These attempts were generated from the original problem directly, without conditioning on the previous round’s hints. In the final system, this was usually \\(F=2\\). The purpose was to preserve diversity and reduce the risk that all later generations would collapse onto the same incorrect reasoning path.

\n

This idea was inspired by Escaping the Cognitive Well: Efficient Competition Math with Off-the-Shelf Models. Although that paper uses a different problem-solving pipeline, diversity among candidate solutions also plays a central role there. In the paragraph “Phase III: Refinement with Global Memory” on page 5, the authors describe keeping one solver zero-shot during refinement, while the other solvers use accumulated context. The stated purpose is to preserve diversity and avoid falling into a “Cognitive Well.”

\n

Our fresh-attempt mechanism is the analogous idea adapted to RSA. Most generations in a later RSA round are conditioned on previous solution hints, but a small number are left unconditioned. This gives the system a way to keep exploring the original problem distribution, rather than forcing every trajectory to inherit the same potentially flawed context.

\n

3. Defining answer “convergence”

\n

In practice, the hardest part was actually deciding when the answer has \"converged\" and the answer distribution was reliable enough to stop.

\n

If the stopping rule is too aggressive, the system may accept a wrong majority answer too early. This is exactly what happens in the Class 4 and Class 5 failure modes described in Section 5.2, with examples in appendix items B.1, D.1.1, and D.1.2: the model quickly produces a clean-looking but wrong majority. These cases are difficult because the histogram looks similar to an easy problem where the majority answer is genuinely correct.

\n

On the other hand, if the stopping rule is too conservative, the system wastes time on problems that were already solved, and may even give RSA more opportunities to drift. Under the competition time limit, this also harms later problems by consuming too much of the global budget.

\n

So our approach was to take a middle ground. We measure convergence by looking at the answer distribution. We accepted strong Round 0 majorities, continued RSA on diffuse or unstable distributions, required progress for additional rounds, and reserved fresh attempts to preserve diversity. These rules do not eliminate answer collapse completely, but they made RSA stable enough in practice for gpt-oss-120b.

\n

The main remaining weakness is still the fast, confident wrong-majority case. We identified this explicitly in the writeup in section 7 as the most important failure mode for future work.

\n

(part 3 of 3)

", "rawMarkdown": "(part 3 of 3)\n\n# Question: \n\n**Answer Collapse:** Since RSA propagates hints to future trajectories, how did you mitigate the risk of convergence toward an incorrect majority answer? In particular, if early generations contain many flawed solutions, using them as contextual hints in later iterations could potentially steer subsequent trajectories in the wrong direction. How did you address this failure mode?\n\n# Response:\n\n**Answer Collapse:** There is no real way to completely prevent this failure mode, short of knowing the ground truth answer directly.\n\nThere are two different concerns raised in this question: \n- (i) How to prevent wrong answers from propagating, and \n- (ii) how to identify when the generations have \"converged\" to the correct answer. \n\nWe were quite concerned about both at the beginning. However, after experimenting with `gpt-oss-120b`, our view became more nuanced. We found three things:\n\n- (i) Propagation: Once RSA is triggered, answer collapse was less severe for `gpt-oss-120b` than we initially expected; \n- (ii) Propagation: There were simple mitigation steps that reduced the risk further; \n- (iii) Convergence: The harder practical problem was deciding when the answer distribution had genuinely “converged” to the correct answer and RSA should stop.\n\n## 1. This was less severe than expected\n\nRSA with `gpt-oss-120b` is able to recover minority correct solutions even among a majority of wrong ones.\n\nThis issue is discussed in the V1 paper, [V_1: Unifying Generation and Self-Verification for Parallel Reasoners](https://arxiv.org/pdf/2603.04304v1), which we cited in the ablation study. Near the bottom of page 4, the authors write that RSA can “discard or degrade correct outlier solutions during refinement.” Their V1-Infer method was partly designed to address this weakness by ranking candidates through pairwise comparisons instead of relying on refinement alone.\n\nHowever, our experiments suggested that this failure mode is model-dependent. The V1 paper reports this issue for smaller models such as `gpt-oss-20b` and `Qwen3-4B-Instruct`. In our experiments, `gpt-oss-120b` was much less susceptible to it. Examples B.1, B.2, and B.6 in Appendix B show cases where RSA with `gpt-oss-120b` was able to rescue a minority answer from the initial candidate pool, even when many early generations were flawed.\n\nThis led us to the following working hypothesis: for `gpt-oss-120b`, the important factor was often whether the correct solution, or at least a useful partial route to it, appeared somewhere in the candidate pool. It did not always need to be the majority answer initially. Once the correct idea appeared in one or more hints, later RSA rounds could usually amplify it.\n\nThe ablation study in Section 6 also supports our decision not to switch to V1-Infer. We made several public leaderboard submissions using V1-Infer, and they scored in the 37–39 range, below the RSA-based submissions. So while V1-Infer is a reasonable strategy for mitigating answer collapse in principle, in our setting it did not improve the final leaderboard score.\n\n## 2. Mitigation measures\n\nEven though answer collapse was not as severe as expected, we still added two safeguards.\n\n**First,** the RSA prompt explicitly framed previous solutions as **untrusted hints**, not as ground truth. The RSA prompt (in **appendix E.4**) has a part that says:\n\n```text\nYou are given a math problem, and some solution hints.\nSome hints are incorrect and contain errors.\"\n```\n\nThis wording was intentional. We wanted the model to solve the original problem again, while using previous candidates only as potentially useful evidence.\n\n**Second,** we reserved a small number of **fresh attempts** in later RSA rounds. These attempts were generated from the original problem directly, without conditioning on the previous round’s hints. In the final system, this was usually \\\\(F=2\\\\). The purpose was to preserve diversity and reduce the risk that all later generations would collapse onto the same incorrect reasoning path.\n\nThis idea was inspired by [*Escaping the Cognitive Well: Efficient Competition Math with Off-the-Shelf Models*](https://arxiv.org/abs/2602.16793). Although that paper uses a different problem-solving pipeline, diversity among candidate solutions also plays a central role there. In the paragraph “Phase III: Refinement with Global Memory” on page 5, the authors describe keeping one solver zero-shot during refinement, while the other solvers use accumulated context. The stated purpose is to preserve diversity and avoid falling into a “Cognitive Well.”\n\nOur fresh-attempt mechanism is the analogous idea adapted to RSA. Most generations in a later RSA round are conditioned on previous solution hints, but a small number are left unconditioned. This gives the system a way to keep exploring the original problem distribution, rather than forcing every trajectory to inherit the same potentially flawed context.\n\n\n## 3. Defining answer “convergence”\n\nIn practice, the hardest part was actually deciding when the answer has \"converged\" and the answer distribution was reliable enough to stop.\n\nIf the stopping rule is too aggressive, the system may accept a wrong majority answer too early. This is exactly what happens in the Class 4 and Class 5 failure modes described in Section 5.2, with examples in appendix items B.1, D.1.1, and D.1.2: the model quickly produces a clean-looking but wrong majority. These cases are difficult because the histogram looks similar to an easy problem where the majority answer is genuinely correct.\n\nOn the other hand, if the stopping rule is too conservative, the system wastes time on problems that were already solved, and may even give RSA more opportunities to drift. Under the competition time limit, this also harms later problems by consuming too much of the global budget.\n\nSo our approach was to take a middle ground. We measure convergence by looking at the answer distribution. We accepted strong Round 0 majorities, continued RSA on diffuse or unstable distributions, required progress for additional rounds, and reserved fresh attempts to preserve diversity. These rules do not eliminate answer collapse completely, but they made RSA stable enough in practice for gpt-oss-120b.\n\nThe main remaining weakness is still the fast, confident wrong-majority case. We identified this explicitly in the writeup in section 7 as the most important failure mode for future work.\n\n\n(part 3 of 3)", "replies": [ { "id": 3457947, "postDate": "2026-05-14T17:08:43.233Z", "content": "

Nice reply! This is how a proper rebuttal would like like (e.g. as seen on OpenReview, for top ML conferences).

", "rawMarkdown": "Nice reply! This is how a proper rebuttal would like like (e.g. as seen on OpenReview, for top ML conferences).", "votes": 1 } ] } ] }, { "id": 3457565, "postDate": "2026-05-13T23:29:33.397Z", "content": "

Edited on 14 May 2026 for the rebuttal phase:

\n
    \n

  1. Added public leaderboard score range and private leaderboard score (and also a small comment), at the top of the writeup.

    2. \n

  2. Edited section 1.2 drawbacks. Originally, there was an error saying that Δ was the probability that the model generates the correct solution in a single attempt. Instead, Δ should be the difference in probability of producing the correct answer and the probability of the second most likely answer. This does not affect our methodology and the rest of the writeup.

    3. \n
", "rawMarkdown": "**Edited on 14 May 2026 for the rebuttal phase:**\n\n1. Added public leaderboard score range and private leaderboard score (and also a small comment), at the top of the writeup.\n\n2. Edited section 1.2 drawbacks. Originally, there was an error saying that Δ was the probability that the model generates the correct solution in a single attempt. Instead, Δ should be the **difference in probability of producing the correct answer and the probability of the second most likely answer.** This does not affect our methodology and the rest of the writeup.\n" }, { "id": 3445867, "postDate": "2026-04-20T16:07:42.453Z", "content": "

Thanks for the kind words, appreciate it. Looking forward to reading the write-up in more detail! :)

", "rawMarkdown": "Thanks for the kind words, appreciate it. Looking forward to reading the write-up in more detail! :)" }, { "id": 3445840, "postDate": "2026-04-20T15:48:51.257Z", "content": "

Great writeup\nThe combination of RSA with a difficulty aware scheduler is very insightful and clearly explained

", "rawMarkdown": "Great writeup\nThe combination of RSA with a difficulty aware scheduler is very insightful and clearly explained" }, { "id": 3445565, "postDate": "2026-04-20T04:51:39.703Z", "content": "

@icebeam777 @davendw49 \nThis is probably the most novel inference-time contribution in the competition. The RSA diagrams make the mechanism very clear - especially the Norwegian Integers example where the correct answer (8687) goes from 0 appearances in Round 0 to majority in Round 1. That's a capability that no pure voting scheme (including mine) can replicate.

\n

The imo-bench-algebra-100 examples (Appendix B.2) are where it gets really interesting though. In the short run, the system returns answer=2 in ~270s with a strong 4-vote majority and entropies in [0.635–0.749] - indistinguishable from a confident correct answer. Your Class 4/5 taxonomy identifies this as the fundamental failure mode.

\n

My question is about the recovery mechanism:

\n
    \n

  1. In the extended run where RSA does recover answer=8, attempt 6 in Round 0 already contained it. For Class 5 problems where the correct answer is absent from ALL Round 0 attempts, how often does RSA Round 1 surface it for the first time through hint synthesis alone? The Norwegian Integers example shows this can happen - but is it representative across your 17-problem benchmark, or an outlier? A per-class recovery rate (especially Class 4 vs Class 5) would clarify whether RSA's value is primarily \"amplifying existing minority answers\" vs. \"generating new answers through partial-reasoning synthesis.\"

    2. \n

  2. Also curious about your scheduler under adversarial ordering. Your time bank reserves 250s early → 295s late, with an expected 0.5 time-starved problems. Was that estimate computed under random ordering, or did you stress-test against worst-case sequences (all bucket D+E problems in positions 1–10)? @rzatemizel Monte Carlo simulations show that under unlucky ordering, a public-notebook scheduler can fail on 15/50 - which is a very different number than 0.5.

    3. \n
\n

The speculative decoding + prefix caching combination giving you ~25% total speedup is something I left on the table entirely. Well executed.

", "rawMarkdown": "@icebeam777 @davendw49 \nThis is probably the most novel inference-time contribution in the competition. The RSA diagrams make the mechanism very clear - especially the Norwegian Integers example where the correct answer (8687) goes from 0 appearances in Round 0 to majority in Round 1. That's a capability that no pure voting scheme (including mine) can replicate.\n\nThe imo-bench-algebra-100 examples (Appendix B.2) are where it gets really interesting though. In the short run, the system returns answer=2 in ~270s with a strong 4-vote majority and entropies in [0.635–0.749] - indistinguishable from a confident correct answer. Your Class 4/5 taxonomy identifies this as the fundamental failure mode. \n\nMy question is about the recovery mechanism:\n\n1. In the extended run where RSA *does* recover answer=8, attempt 6 in Round 0 already contained it. For Class 5 problems where the correct answer is absent from ALL Round 0 attempts, how often does RSA Round 1 surface it for the first time through hint synthesis alone? The Norwegian Integers example shows this can happen - but is it representative across your 17-problem benchmark, or an outlier? A per-class recovery rate (especially Class 4 vs Class 5) would clarify whether RSA's value is primarily \"amplifying existing minority answers\" vs. \"generating new answers through partial-reasoning synthesis.\"\n\n2. Also curious about your scheduler under adversarial ordering. Your time bank reserves 250s early → 295s late, with an expected 0.5 time-starved problems. Was that estimate computed under random ordering, or did you stress-test against worst-case sequences (all bucket D+E problems in positions 1–10)? @rzatemizel Monte Carlo simulations show that under unlucky ordering, a public-notebook scheduler can fail on 15/50 - which is a very different number than 0.5.\n\nThe speculative decoding + prefix caching combination giving you ~25% total speedup is something I left on the table entirely. Well executed.", "replies": [ { "id": 3446201, "postDate": "2026-04-21T06:48:18.027Z", "content": "

Hi @flamingice2801,

\n

Thank you for reading my write up!

\n

Here are my (preliminary) answers:

\n
    \n

  1. Right now I am not sure about this. Anecdotally, it is not an outlier, but it happens rarely enough ( < 25% of the time) that it is worthwhile to inject fresh attempts to each subsequent RSA round to improve diversity of solutions. I am trying to secure some GPUs to hopefully run some tests on this in the next week or two.

    2. \n

  2. I tested this with two methods:

    \n

    a. Dynamic Programming to explore all 50! possible orderings of problems:\nhttps://pastebin.com/3GjGenRa\n(warning: Takes a long time to run)

    \n

    b. Monte Carlo Simulation for 20000 random orderings:\nhttps://pastebin.com/TvJ9uegH

    3. \n
\n

For b) , in the worst case assuming that all problems in buckets A, B, and C take the maximum amount of time. Problems in D and E average 850 seconds because of the RSA gating mechanisms. The code prints the following:

\n
Monte Carlo estimate: 0.50695\nHistogram: {0: 14165, 1: 3109, 2: 1627, 3: 745, 4: 258, 5: 73, 6: 18, 7: 4, 8: 1}\n
\n

That is assuming the worst case. In reality, many easy problems in bucket A finish in less than 100 seconds, and bucket B in less than 200, and bucket C in less than 300 (the worst case assumes 180, 280, and 550 respectively).

\n

In an average case, the Monte Carlo simulation prints:

\n
Monte Carlo estimate (average): 0.00095\nHistogram: {0: 19982, 1: 17, 2: 1}\n
\n

The expected number of Time-Starved problems is effectively 0. The exact value calculated in a) from Dynamic Programming prints this:

\n
Exact expected number of underallocated questions: 0.0008210684786705279\n
\n

In the absolute worst case where the first 10 problems are all hard (D/E), and all the medium problems (C), there will be more time starved problems. The Monte Carlo simulation above found 8 Time-Starved problems in 1 out of 20000 runs.

\n

rzatemizel's writeup analyzed the scheduler in the public notebook, which places a hard cap of 300 seconds once all the extra time has been consumed. They found that 15 problems can be Time-Starved in a bad case. This is far from the average case. From his chart, it seems that he placed 12 problems in bucket C. I only placed 3, because that is the number of problems in pass@5 but not in pass@3 in the chart released by the AIMO 3 organizers on X.

\n

My scheduler is also much less restrictive initially. It only reserves 250 s per problem when there are more than 30 problems remaining. Moreover, when we are behind time, the Time Bank keeps track of how far we are behind, and each subsequent problem only has to pay back self.cfg.time_bank_weight = 0.40 each time, instead of enforcing a hard cap.

", "rawMarkdown": "Hi @flamingice2801,\n\nThank you for reading my write up!\n\nHere are my (preliminary) answers:\n\n1. Right now I am not sure about this. Anecdotally, it is not an outlier, but it happens rarely enough ( < 25% of the time) that it is worthwhile to inject fresh attempts to each subsequent RSA round to improve diversity of solutions. I am trying to secure some GPUs to hopefully run some tests on this in the next week or two.\n\n2. I tested this with two methods: \n\n a. Dynamic Programming to explore all 50! possible orderings of problems:\nhttps://pastebin.com/3GjGenRa\n(**warning:** Takes a long time to run)\n\n b. Monte Carlo Simulation for 20000 random orderings:\nhttps://pastebin.com/TvJ9uegH\n\n\nFor b) , in the worst case assuming that all problems in buckets A, B, and C take the maximum amount of time. Problems in D and E average 850 seconds because of the RSA gating mechanisms. The code prints the following:\n\n```\nMonte Carlo estimate: 0.50695\nHistogram: {0: 14165, 1: 3109, 2: 1627, 3: 745, 4: 258, 5: 73, 6: 18, 7: 4, 8: 1}\n```\nThat is assuming the worst case. In reality, many easy problems in bucket A finish in less than 100 seconds, and bucket B in less than 200, and bucket C in less than 300 (the worst case assumes 180, 280, and 550 respectively).\n\nIn an average case, the Monte Carlo simulation prints:\n```\nMonte Carlo estimate (average): 0.00095\nHistogram: {0: 19982, 1: 17, 2: 1}\n```\nThe expected number of Time-Starved problems is effectively 0. The exact value calculated in **a)** from Dynamic Programming prints this:\n```\nExact expected number of underallocated questions: 0.0008210684786705279\n```\n\nIn the absolute worst case where the first 10 problems are all hard (D/E), and all the medium problems (C), there will be more time starved problems. The Monte Carlo simulation above found 8 Time-Starved problems in 1 out of 20000 runs. \n\n[rzatemizel's writeup](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/fighting-against-variance-under-time-constraint) analyzed the scheduler in the public notebook, which places a hard cap of 300 seconds once all the extra time has been consumed. They found that 15 problems can be Time-Starved in a bad case. This is far from the average case. From his chart, it seems that he placed 12 problems in bucket C. I only placed 3, because that is the number of problems in pass@5 but not in pass@3 in the [chart](https://x.com/AIMOprize/status/2039441022996934783/photo/1) released by the AIMO 3 organizers on X.\n\nMy scheduler is also much less restrictive initially. It only reserves 250 s per problem when there are more than 30 problems remaining. Moreover, when we are behind time, the Time Bank keeps track of how far we are behind, and each subsequent problem only has to pay back ```self.cfg.time_bank_weight = 0.40``` each time, instead of enforcing a hard cap.", "replies": [ { "id": 3446682, "postDate": "2026-04-22T00:28:31.670Z", "content": "

Thank you for the detailed response - especially the Monte Carlo and DP results. The 0.00095 average-case vs. 0.507 worst-case gap is a really clean demonstration of why the time bank with 0.40 payback weight works so much better than a hard cap.

\n

The bucket calibration point is important too. Using the host's pass@n chart to size bucket C at 3 problems (pass@5 minus pass@3) is much tighter than the 12-problem estimate in rzatemizel's analysis. That single calibration difference explains most of the gap between your 0.5 expected time-starved and their 15.

\n

The <25% recovery rate for Round 0-absent answers is also useful context. Our approach uses 8 fully independent attempts at T=1.0 with no inter-attempt communication - which optimizes for the \"correct answer needs to be generated at least once\" case. RSA optimizes for the \"correct answer appeared once, now amplify it\" case. Would be very interesting to see a head-to-head on the same problem set if you get the GPU time - particularly on problems where the correct answer appears in exactly 1 of 8 initial attempts (the regime where RSA should have the largest advantage over independent voting).

\n

Looking forward to the private LB results.

", "rawMarkdown": "Thank you for the detailed response - especially the Monte Carlo and DP results. The 0.00095 average-case vs. 0.507 worst-case gap is a really clean demonstration of why the time bank with 0.40 payback weight works so much better than a hard cap.\n\nThe bucket calibration point is important too. Using the host's pass@n chart to size bucket C at 3 problems (pass@5 minus pass@3) is much tighter than the 12-problem estimate in rzatemizel's analysis. That single calibration difference explains most of the gap between your 0.5 expected time-starved and their 15.\n\nThe <25% recovery rate for Round 0-absent answers is also useful context. Our approach uses 8 fully independent attempts at T=1.0 with no inter-attempt communication - which optimizes for the \"correct answer needs to be generated at least once\" case. RSA optimizes for the \"correct answer appeared once, now amplify it\" case. Would be very interesting to see a head-to-head on the same problem set if you get the GPU time - particularly on problems where the correct answer appears in exactly 1 of 8 initial attempts (the regime where RSA should have the largest advantage over independent voting).\n\nLooking forward to the private LB results." } ] } ] }, { "id": 3445864, "postDate": "2026-04-20T16:06:09.533Z", "rawMarkdown": "", "isDeleted": true } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T04:20:55Z", "competition_id": 118448, "competition_slug": "ai-mathematical-olympiad-progress-prize-3", "team_id": 14822716, "solution_url": "/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/recursive-self-aggregation-with-difficulty-aware-s" }