![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F20310625%2F9cd4f4a65dae84459a93b1846355eaea%2FFlowChart.png?generation=1776521614005070&alt=media\")
\nFigure 1: The Multi-Turn Tool-Integrated Reasoning (TIR) orchestration pipeline, highlighting the decoupling of the Planner (code generation) and the Judge (ranking evaluation).This paper investigates the viability of deploying general-purpose, open-weight Large Language Models (LLMs) in heavily constrained, offline inference environments to solve olympiad-level mathematical problems. Operating under the strict hardware limitations of a single NVIDIA H100 GPU and a 5-hour time budget for 50 problems, we introduce a Python-driven, Multi-Turn Tool-Integrated Reasoning (TIR) orchestration pipeline. Rather than relying on specialized fine-tuning, this study optimizes dynamic resource allocation, context management, and hyperparameter targeting using the Qwen 3.5 35B-A3B Mixture-of-Experts (MoE) architecture. We evaluate the efficacy of zero-shot general-purpose models acting as Process Reward Models (PRMs) and demonstrate the necessity of a ranking-based evaluation layer. Achieving a public score of 14/50 on the AI Mathematical Olympiad (AIMO) Progress Prize 3, our results highlight both the emerging capabilities of generalized reasoning models and their persistent vulnerabilities to logical hallucination when operating without domain-specific mathematical post-training.
\nThe pursuit of artificial mathematical reasoning has recently bifurcated into two primary methodologies: domain-specific fine-tuning on highly curated mathematical corpora—exemplified by systems like AlphaGeometry (Trinh et al., 2024)—and the deployment of massive, generalized reasoning models (such as OpenAI's o1 paradigm). Faced with limited computational resources and a restricted timeline, this study opted to test the upper bounds of the latter approach. Instead of training custom weights or employing proprietary evaluation models, we utilized the AIMO Progress Prize 3 as a platform to test how far general-purpose LLMs can advance on a demanding set of problems using purely inference-time orchestration.
\nOur findings demonstrate that while sophisticated inference-time compute scaling and Tool-Integrated Reasoning (TIR) can yield competitive baselines, the concept of an unmodified, general-purpose model reliably excelling at olympiad-level mathematics without specialized post-training remains, at this point, a summer night's dream.
\nThe foundational requirement of our architecture was an offline pipeline capable of benchmarking general-purpose models without requiring specialized evaluators or math-specific weights. Early iterations of this approach mandated a pivot to Alibaba's Qwen MoE family, specifically transitioning from the FP8 quantized variant of Qwen3-30B-A3B to the newer Qwen3.5-35B-A3B (Qwen Team, 2024; Qwen Team, 2025).
\nThe model's weights ($\\approx 30$ GB) fit comfortably on the Kaggle-provided NVIDIA H100 GPU, allocating sufficient space for the KV Cache required by long-context reasoning chains. The deployment relied on vLLM (Kwon et al., 2023), capitalizing on the model's seamless integration of a \"thinking\" mode toggle and its top-tier ability to adhere to strict formatting constraints without excessive prompt engineering.
The core architecture operates as a Tool-Integrated Reasoning loop driven entirely by Python, building upon recent successes in Program-Aided Language models (Gao et al., 2023; Gou et al., 2023). The reliance on Python is predicated on the observation that general-purpose models are heavily fine-tuned for Software Engineering (SE) tasks, rendering them significantly more capable of producing accurate, executable Python code than symbolic representations like Lean.\n\nFigure 1: The Multi-Turn Tool-Integrated Reasoning (TIR) orchestration pipeline, highlighting the decoupling of the Planner (code generation) and the Judge (ranking evaluation).
The \"heartbeat\" of the loop is the Planner. The system prompts force the Planner to frame problems as sequential computational steps toward a solution. Instead of attempting a one-shot mathematical proof—which requires managing massive internal reasoning chains—the model focuses on identifying the next computational step to test a specific hypothesis.
\nConstraints & Fitness Checks:
\nProhibited Methods: Probabilistic search space traversals (e.g., Monte Carlo simulations) and inefficient libraries (e.g., SciPy for symbolic geometry) are strictly prohibited, as they rarely yield the exact integers required by the AIMO format.
If a candidate script employs prohibited methods, an intricate validation loop performs rigorous fitness checks, and the candidate is immediately discarded.
Whether a candidate ultimately outputs a solution is decided by a majority vote, leveraging the principles of self-consistency (Wang et al., 2022) to filter out anomalous logical errors. The model outputs a JSON payload containing a boolean field indicating if the candidate script is intended to be a final solution. If a majority flags a solution, it is evaluated, submitted, and passed through strict regex checks to ensure it is a valid integer modulo $10^5$.
\nTwo auxiliary roles interact with the problem utilizing the model's reasoning capabilities:
\nThe Classifier: Analyzes the raw problem and outputs a structured rephrasing, primarily consisting of explicit tasks and constraints. This format aligns optimally with the Planner's computational disposition. (Note: The nomenclature \"Classifier\" is a vestige of an earlier iteration intended to classify problem difficulty for dynamic step allocation, an approach abandoned due to model inconsistency).
The Final Verdict: Invoked only when the orchestration loop reaches its absolute limit (due to time constraints or max step exhaustion). Utilizing all evidence gathered in the context window, it synthesizes a final, educated approximation of the integer solution.
State-of-the-art agentic mathematical pipelines rely heavily on evaluation models, specifically Outcome Reward Models (ORMs) and Process Reward Models (PRMs). Initial attempts to prompt our general-purpose Judge to mimic a strict PRM demonstrated that unmodified LLMs struggle to handle this gracefully. The PRM setup tested was heavily adapted from the step-wise evaluation format introduced by Lightman et al. The prompt instructed the LLM to read the generated reasoning (the content inside the <think>} blocks, which was computationally expensive to process) and assign a token classification of VALID, INVALID, or NEUTRAL to each logical step, calculating a final confidence score based on the ratio of valid steps.
The data point that forced a pivot was the high false-positive rate. General-purpose models exhibited severe \"confirmation bias\" when acting as zero-shot PRMs; if an early mathematical step contained a subtle arithmetic error but the subsequent logic flowed consistently from that error, the model would confidently grade the remaining steps as VALID, leading to a high overall score for an incorrect solution. Switching to a Ranking Judge bypassed this step-wise hallucination by forcing the model into a comparative analysis (Solution A vs. Solution B), which proved vastly more reliable zero-shot. Furthermore, as demonstrated in existing research, such evaluators work optimally with specifically fine-tuned models, which fundamentally contradicted our objective of evaluating general-purpose models. Switching to a Ranking Judge bypassed this step-wise hallucination by forcing the model into a comparative analysis (Solution A vs. Solution B), which proved vastly more reliable zero-shot. It is important to clarify that while the Judge internally evaluates a ``ranking'' of the reasoning traces, the extraction layer strictly forces a winner-takes-all selection. Confining the final output to a single optimal candidate, rather than a complex ordered list, significantly mitigated the output-space instability typically caused by high inference temperatures.
Executing all candidates prior to the Judge's evaluation was tested but ultimately scrapped. The general-purpose model exhibited a strong bias, clinging to any candidate that produced a single, clean integer, even during early exploratory steps where the objective was search-space mapping. Furthermore, running a multitude of parallel Python sandboxes severely degraded the computational budget.
\nThe Execute Loop runs the selected candidate and returns the standard output. Extensive error handling is implemented to ensure the model receives full tracebacks explaining execution failures.
\nTo preserve compute, a fast Syntax Reviewer agent was introduced. Before a candidate is fully scrapped, the Reviewer attempts to resolve trivial syntactical errors (e.g., indentation errors, missing imports). This reviewer consistently rescued highly promising mathematical algorithms during reference set evaluations.
\nA critical challenge in the AIMO environment is managing the strict 18,000-second offline execution limit. To prevent infinite loops or geometrically expanding search spaces on a single problem, a custom DynamicTimeController was engineered to regulate the temporal and operational bounds of the TIR loop.
Let $T_{total}$ be the total available environment time, $N$ be the total number of problems, and $T_{safe}$ be a rigid teardown safeguard (230 seconds) reserved for final CSV generation and vLLM spin-down. Before attempting the $i$-th problem, the remaining budget $B_i$ is calculated as:
$$B_i = T_{total} - T_{safe} - \\sum_{j=1}^{i-1} t_j$$
\nwhere $t_j$ is the time spent on previous problems. Rather than enforcing a static limit, the budgeter permits the agent to utilize surplus time saved from earlier, easier problems. For the current problem, we define a dynamic temporal ceiling, $T_{max, i}$, controlled by a relaxation factor $\\gamma$ (where $\\gamma \\approx 1.5$):
\n$$T_{max, i} = \\min \\left( \\gamma \\cdot \\frac{B_i}{N - i + 1}, \\;\\; B_i \\right)$$
\nWithin this ceiling, reasoning depth is regulated by a maximum step count ($S_{max} = 5$). Let $t_{infer}$ be the variable inference time and $t_{exec}$ be the sandbox execution time. We enforce:
\n$$\\sum_{k=1}^{S} (t_{infer, k} + t_{exec, k}) \\leq T_{max, i}$$
\nTo guarantee progression, the Python subprocess sandbox is rigidly capped at 15 seconds per execution. If $\\max(t_{exec, k})$ reaches 15, a TimeoutError is piped into the LLM context, punishing inefficient while loops and forcing the model to re-evaluate its logic.
Due to the absence of weight fine-tuning, the \"training\" equivalent in this methodology was the rigorous optimization of generation parameters and VRAM allocation on the Kaggle H100 GPU.
\nVRAM and Context Constraints: The vLLM engine was instantiated with enforce_eager=True to disable CUDA graph capture, successfully preventing Out-of-Memory (OOM) exceptions when managing a 32,768 native context window alongside max_num_seqs=16. The KV Cache was quantized to FP8, doubling the functional context capacity.
Context Pruning: To prevent overflowing the native window when processing tracebacks, the history limit was capped at 15,000 tokens, and \\texttt{stdout} limits were aggressively truncated. Crucially, to prevent catastrophic forgetting and ``loopy'' context degradation, the heavy \\texttt{} reasoning traces were systematically stripped from memory after each step. The pipeline relied on the Summarizer role to compress the global state of the problem, ensuring the model retained its trajectory without drowning in its own previous reasoning loops.
Decoding Strategies: Following Qwen 3 guidelines, greedy decoding was abandoned for reasoning roles. The Planner utilized $T=0.6$, while analytical roles (Judge, Summarizer) utilized cooler temperatures ($T=0.3$) to maintain determinism. Crucially, presence_penalty was explicitly set to $0.0$ across all agents to prevent the fatal hallucination of JSON structures and mathematical syntax.
The pipeline achieved a public score of 14/50. While modest compared to fine-tuned or heavily scaled proprietary models, this score establishes a robust baseline for inference-only, general-purpose MoE models operating under severe zero-shot constraints.\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F20310625%2F01f0e0c9c2bf95a49d9664e13dbfab82%2FAIMEBenchPlotsQwen3.5.png?generation=1776521574372147&alt=media\")
\nFigure 2: Execution time and cumulative accuracy progression for the Qwen 3.5 35B-A3B agent across the offline 30-problem AIME benchmark, demonstrating the Dynamic Time Controller's budget adherence.\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F20310625%2F02beb69a56ad8eb40eaf9c4928b99e6b%2FAIMEBenchPlotsQwen3.png?generation=1776521596014436&alt=media\")
\nFigure 3: Comparative performance of the Qwen 3 30B-A3B agent under identical hyperparameter and time-budget constraints. (Note the variance in generation stability compared to the 3.5 architecture).
A comparison of Figures 2 and 3 highlights a significant discrepancy in execution time and budget adherence. This variance stems from a combination of the dynamic budget logic, step counts, and the underlying model architectures. During early testing, hard-capping the time per problem often starved earlier problems of setup time. To counter this, the Time Controller allowed early problems to ``borrow'' compute budget against the global 5-hour limit, operating on the premise that a capable model would converge quickly and not exhaust this surplus.
\nBecause Qwen3 struggled more with initial logical paths, it failed to converge to a solution as fast as its newer sibling. Consequently, Qwen3 weaponized the ``borrowed'' early time against itself by exhausting its maximum allowed reasoning steps on earlier problems, leading to the late-stage timeout shown in the benchmark. In contrast, Qwen3.5’s hybrid Gated DeltaNet / Attention architecture allowed for significantly faster token generation and context processing, executing the loops faster and with fewer errors.
\nA genuine limitation of this ablation study is the sequential problem ordering used in the AIME benchmark. Due to strict compute limitations and the impending revocation of Kaggle H100 access at the competition's deadline, the benchmark was hastily executed in its default sequential order, which inherently ramps up in difficulty (1-15, 16-30). This default difficulty ramp interacted poorly with the ``borrowing'' mechanism. Because early problems were easier, Qwen3.5 solved them quickly, whereas Qwen3 struggled significantly from the third problem onwards, exhausting its time before ever encountering the hardest problems. Running this benchmark with a randomized problem order would have undoubtedly isolated the model's baseline compute requirements from the controller's behavioral biases, an oversight that must be addressed in future methodological iterations.
\nTo ensure full reproducibility of the inference pipeline and benchmarking results, the complete\nsource code, vLLM configurations, and environment setups are publicly available on Kaggle.
\nNote on Model Access and Configuration: While the orchestration logic and notebooks are\nfully public, the specific Kaggle dataset containing the model weights used for this submission\nremains private. However, the pipeline is designed to be highly modular. Researchers can\nseamlessly plug in their own model paths and adjust all core vLLM and pipeline hyperparameters\n(e.g., maximum reasoning steps, temperature, and context limits) by simply modifying the\nconfiguration variables located in the third cell of the notebooks.
\nThis study demonstrates that advanced inference orchestration—specifically through dynamic time budgeting, rigorous tool integration, and specialized agent routing—can successfully extract competitive mathematical reasoning capabilities from general-purpose LLMs. However, the persistent reliance on logical hallucinations underscores a critical barrier. Until general-purpose models develop innate, rigorous process-reward architectures natively, achieving Olympiad-level mastery without domain-specific post-training remains a summer night's dream.
\nAnother conclusion that I reached after reading through some of the top write-ups is how important parameter count is when it comes to reducing hallucinations and producing more solid reasoning, which seems trivial knowledge, however, I wrongfully believed that a 35B model that has not been heavily quantized would be less likely to hallucinate, which was a lapse in judgment, evident by the leaderboard, and an oversight I should probably have covered.
\nEarly on, I had tried a heavier model, the results of them I sadly do not posses since it was very early and I hadn't documented them overall, I saw however significantly slower reasoning to the point I could barely do any scaffolding around it, and a big reason why I chose to stay around 30B and in particular go with MoE architectures was that I wanted fast and computationally cheap reasoning, to push a heavier context window and more retries/candidates, since I didn't find a model that I could trust. Furthermore, most of the fine-tuned models from progress prize 2 were 7B models, adding to my conviction that overall ended up not being correct or at least not adapted to the latest SOTA models, and in particular one that shaped the public leaderboard as I was conducting my own experiments.
\nThat model was GPT-OSS-120B which seemed to have very promising out of the box results, making up for it's computational cost and increased quantization in raw reasoning performance . I lack the proper compute to thoroughly test my scaffold on it, since we no longer have access to H100 from the competition, however, I can safely assume my results would have drastically improved, thus my premise for choosing a 35B model was all in all proven either false, or at the very least obsolete.
\nFrieder, S., Bealing, S., Vonderlind, P., Li, S., Nikolaiev, A., Smith, G. C., Buzzard, K., Gowers, T., Liu, P. J., Loh, P.-S., Mackey, L., de Moura, L., Roberts, D., Sculley, D., Tao, T., Balduzzi, D., Coyle, S., Gerko, A., Holbrook, R., Howard, A., & XTX Markets. (2025). AI Mathematical Olympiad - Progress Prize 3. Kaggle. https://kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3
Gao, L., Maman, A., Banerjee, S., Goswami, V., Kamalloo, E., Mackey, L., & Aggarwal, G. (2023). PAL: Program-Aided Language Models. In Proceedings of the 40th International Conference on Machine Learning (ICML).
Gou, Z., Shao, Z., Gong, Y., Yang, Y., Huang, M., Duan, N., Chen, W., et al. (2023). ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving. In Proceedings of the International Conference on Learning Representations (ICLR) 2024. arXiv preprint arXiv:2309.17452.
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP '23).
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Sutskever, I., et al. (2023). Let's Verify Step by Step. arXiv preprint arXiv:2305.20050.
OpenAI. (2024). Learning to Reason with LLMs. OpenAI Research. https://openai.com/index/learning-to-reason-with-llms/
OpenAI. (2025). Introducing GPT-OSS. OpenAI Research.. https://openai.com/index/introducing-gpt-oss/
Qwen Team. (2024). Qwen2.5 Technical Report. arXiv preprint arXiv:2412.15115.
Qwen Team. (2025). Qwen3 Technical Report. arXiv preprint arXiv:2505.09388.
Trinh, T. H., Wu, Y., Le, Q. V., He, H., & Luong, T. (2024). Solving olympiad geometry without human demonstrations. Nature, 625(7995), 476–482.
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022). Self-Consistency Improves Chain of Thought Reasoning in Language Models. In Proceedings of the International Conference on Learning Representations (ICLR) 2023. arXiv preprint arXiv:2203.11171.
I took the opportunity of the Rebuttal phase to make a few targeted updates to my write-up. Following the questions from the reviewers @philipvonderlind @sambealing , I ended up making the following edits eligible for the rebuttal phase of the competition:
\nExpanded Conclusion (Model Selection Post-Mortem)
\nWhat was added: 3 paragraphs in the Conclusion discussing the GPT-OSS 120B meta and the quantization vs. parameter count trade-off.
When: At the opening of the rebuttal phase (prior to reviewer questions).
What it addresses: Acts as a self-reflection acknowledging that my initial premise regarding a 35B model hallucinating less than a heavily quantized 120B model was rendered obsolete by the final leaderboard results.
Expanded Section 3.3 (The Judge & PRM Pivot)
\nWhat was added: Detailed the specific PRM prompt used (adapting Lightman et al., classifying blocks) and clarified that the ranking mechanism uses a \"winner-takes-all\" extraction rather than outputting a volatile list.
When: During the reviewer Q&A phase.
What it addresses: Addresses reviewer questions regarding the exact PRM methodology, the LLM \"confirmation bias\" flaw, and concerns about ranking instability at high inference temperatures.
New Subsection in Section 6 (Ablation Limitations & Budget Overshoots)
\nWhat was added: A dedicated subsection titled \"Ablation Study Limitations and Time-Budget Overshoots\".
When: During the reviewer Q&A phase.
What it addresses: Directly answers why Qwen3 weaponized the \"borrowed\" time budget compared to Qwen3.5 (highlighting the Gated DeltaNet efficiency), and acknowledges the sequential AIME difficulty ramp as a genuine limitation of the ablation study.
Updated Section 5 (Catastrophic Forgetting & Context)
\nWhat was added: Clarified the Context Pruning bullet point to explicitly state that blocks are systematically stripped from memory to prevent context degradation.
When: During the reviewer Q&A phase.
What it addresses: Resolves reviewer questions regarding how the pipeline prevents catastrophic forgetting and \"loopy\" behavior within the 32k character limit.
Adding to the comments by @sambealing below, I have a few additional questions:
\nQwen: In your submission, you write that “early iterations of this approach mandated a pivot to Alibaba's” model. From what I have seen, most other competitors concluded that gpt-oss-120b was the stronger choice under the available hardware budget. What motivated your decision to use a less common model for your submission? Did you find any evidence that supported this choice?
Judge Mechanism: Unlike many other approaches, your judge ranks the generated answers rather than selecting a single final answer directly. This seems like a more difficult task, since comparing subtle differences between multiple answers or reasoning traces is not necessarily straightforward. Did you run any experiments to check whether the produced rankings are stable across multiple runs, given that inference temperature is involved? I would expect ranking instability to have a large impact on problems where the solution attempts are highly diverse.
Eager Inference: Why did you choose to disable CUDA graph execution entirely? Reducing the graph size could also save memory while retaining some throughput benefits. This may also interact with your relatively high N=16, since larger numbers of parallel generations require more memory depending on how many sequences are active at once. It would be interesting to see how these settings affect the overall throughput of your pipeline.
Context Limitation and Catastrophic Forgetting: From my experience, Qwen models are particularly prone to generating very long traces. Since you truncate the trace history at 32k characters, did you observe any catastrophic forgetting of the original task or “loopy” behavior, where the model starts rediscovering previously forgotten context after earlier parts of the trace are removed?
Hi, thank you for taking the time to evaluate my work and for the thoughtful questions. Please find my clarifications below:*
\n1. Qwen Selection & The 120B Meta:\nYou are completely correct regarding the leaderboard meta. My decision to use Qwen 3.5 35B over GPT-OSS-120B was driven by three primary hardware and architectural constraints early in the competition:
\nQuantization vs. Parameter Count Trade-off: I operated under the theoretical premise that a 35B model with less aggressive quantization (FP8) would hallucinate less and maintain better logical consistency than a 120B model heavily quantized to fit on the same H100. This was honestly my own conviction driven by the fact I constantly saw smaller models over-achieving in earlier AIMO competitions.
KV-Cache Headroom: Because my pipeline relied heavily on parallel candidate generation, I needed significant VRAM headroom for the KV-Cache. I also wanted to maintain the ability to push parallel generation AND KV-Cache much beyond what I ended up doing.
Native Reasoning Switch: Qwen offered a very stable native reasoning/think switch and exceptional instruction-following for Python generation out-of-the-box.
\nAs I noted in the recent \"Limitations\" edit to my write-up, that I wrote when the rebuttal phase was announced, my premise regarding the 35B vs. 120B quantization trade-off was ultimately proven obsolete by the final leaderboard, but these were the constraints that locked in my architecture early on.
2. Judge Mechanism & Ranking Instability:
\nI apologize if the terminology in the write-up was ambiguous. While the Judge was prompted to \"rank\" the solutions to force a comparative analysis, it did not output a full, multi-item ranking downstream. The prompt forced an internal comparison of the reasoning traces, but the extraction layer strictly required a \"winner-takes-all\" selection of the single best candidate to proceed. By confining the Judge's final output to a single selection rather than a complex ordered list, I mitigated much of the output-space instability typically caused by high inference temperatures, though I agree that highly diverse candidate pools still introduced variance.
\n3. Eager Inference vs. CUDA Graphs:
\nThis was a stability and initialization trade-off. For the majority of my development cycle, stable vLLM support for the newest Qwen MoE architectures was non-existent. I was relying on nightly wheels up until the final week. Because Kaggle kernels frequently took nearly 20 minutes just to spin up the pipeline and load the model weights, attempting to capture and compile optimized CUDA graphs often resulted in unpredictable OOM errors or timeouts before inference even began. Disabling CUDA graph execution entirely enforce_eager=True was a brute-force measure to guarantee the engine would initialize safely within the Kaggle environment, though I agree that tuning the graph size would have yielded better throughput.
4. Context Limitation & Catastrophic Forgetting:
\nTo prevent catastrophic forgetting within the 32k character limit, I implemented an aggressive context-pruning strategy. Instead of letting the raw context naturally truncate (which causes the \"loopy\" behavior you described), my pipeline systematically stripped out the heavy \\texttt{} reasoning traces from memory after a step was completed, heavily truncated stdout tracebacks, and relied on the Summarizer role to compress the state of the problem. This strict memory management ensured the model retained the \"global state\" of its progress without drowning in its own previous reasoning loops, while losing some of the fine details of it's CoT, which was needed given the limiations of my implementation.
\nI hope my responses were adequate.\nBest Regards!
", "votes": 0, "replies": [] } ] }, { "id": 3457718, "author_name": "Sam Bealing", "author_url": "", "post_date": "2026-05-14T08:22:12.173000", "content": "Hi, thank you for your submission to the Writeup Prize. @philipvonderlind and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.
\naime_2026_II_05 (about 9 problems unattempted). The same hyperparameters and time budget were used as for Qwen3.5-35B-A3B in Figure 2, which mostly stayed within budget. Do you have any data on why the older model consumed so much more compute per problem (e.g. longer reasoning traces, more tool-call retries, judge-rejection loops)?Look forward to reading your responses on these.
", "votes": 0, "replies": [ { "id": 3457816, "author_name": "Christos Lazaridis", "author_url": "", "post_date": "2026-05-14T11:32:04.687000", "content": "Hi, Thank you for the thorough review and the questions. Below I provide my initial response to each one of these.
\n1. ORM/PRM to ranking evaluator pivot: Unfortunately (and this was an honest mistake on my part for not keeping my earlier notebooks), I did not preserve the specific Kaggle notebook of my initial zero-shot PRM experiments once I finalized the Ranking-Based Judge architecture. However, the PRM setup I tested was heavily adapted from the step-wise evaluation format introduced in \"Let's Verify Step by Step\" (Lightman et al., 2023).
\nMy prompt instructed the LLM to read the generated reasoning (the content inside the <think> </think> blocks, which was quite time expensive on it's own) and assign each sentence and latex block a token classification of [VALID], [INVALID], or [NEUTRAL] to each logical step, calculating a final confidence score based on the ratio of valid steps. The data point that forced me to pivot was the high false-positive rate. General-purpose models exhibited severe \"confirmation bias\" when acting as zero-shot PRMs; if an early mathematical step contained a subtle arithmetic error but the subsequent logic flowed consistently from that error, the model would confidently grade the remaining steps as [VALID], leading to a high overall score for an incorrect solution. Switching to a Ranking Judge bypassed this step-wise hallucination by forcing the model into a comparative analysis (Solution A vs. Solution B), which proved vastly more reliable zero-shot.
In general as demonstrated in OpenAI's research, such evaluators work with specific fine-tuned models, which went against my objective of evaluating General Purpose Models
\n2. Time-budget overshoots on Qwen3-30B-A3B: This discrepancy comes down to a combination of my dynamic budget logic, step counts, and the underlying model architectures. During early testing, I observed that hard-capping the time per problem often starved earlier problems of the setup time they needed, leaving excessive, unusable time for later problems (since I strictly limited the maximum number of reasoning steps). To counter this, my Time Controller allowed early problems to \"borrow\" compute budget against the global 5-hour limit. This was developed on the premise that Qwen3.5 would be able to discard the unneeded time by convering to a solution and not eat through it's budget. Because Qwen3 struggled more with initial logical paths it struggled to converge to a solution as fast as it's newer sibling and ate through it's time.
\nConsequently, Qwen3 actively consumed this \"borrowed\" early time by exhausting its maximum allowed reasoning steps for the earlier problems. Given my setup was quite lenient in letting the earlier problems borrow significant time, Qwen3 ended up eating up all of it's time before it reached the later problems. In contrast to Qwen3.5. Furthermore, Qwen3.5’s hybrid Gated DeltaNet / Attention architecture allowed for significantly faster token generation and context processing. Qwen3.5 simply executed the loops faster and with fewer errors, whereas Qwen3 weaponized the \"borrowed\" time against itself, leading to the late-stage timeout shown in the AIME benchmark, since it used the borrowed time on easier problems.
\n3. AIME ordering and difficulty ramp: This is a fantastic and highly insightful observation, as this is a genuine limitation of my ablation study. Given the strict compute limitations and the fact that I was running these benchmarks using Kaggle's finite GPU hours, I hastily executed the AIME benchmark in its default sequential order to generate the ablation plots for the write-up before access to H100 was revoked. Given I was working on my submission up until hours before the deadline, I didn't have the time I wanted to run my ablation studies, so I just ran something as fast as I could in order to have something to show for my write-up.
\nThat said, I completely agree that the default difficulty ramp of AIME interacted poorly with my \"borrowing\" mechanism. Because the early problems were easier, the Qwen 3.5 was able to solve them significantly quicker, while Qwen3 struggled singificantly from the third problem onwards, which means it used up all the time without ever tackling the harder problems. Running this benchmark with a randomized problem order would have undoubtedly isolated the model's baseline compute requirements from the controller's behavioral biases. It is a limitation of my testing methodology that I will surely take into account for future work.
\nI appreciate the time you took to evaluate my work,
\nBest Regards
", "votes": 0, "replies": [] } ] }, { "id": 3445851, "author_name": "Simon Frieder", "author_url": "", "post_date": "2026-04-20T15:58:23.240000", "content": "If you want this writeup to be considered towards the Write-Up Prize, you need to make a comment here: https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/689703
\nYou will need to follow the instructions in that comment, and link to the writeup and also provide some further links to either HuggingFace or Google Colab notebook.
", "votes": 0, "replies": [] }, { "id": 3456915, "author_name": "", "author_url": "", "post_date": "2026-05-12T18:30:47.810000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 7, "messages": [ { "id": 3444830, "postDate": "2026-04-18T14:21:14.267Z", "content": "This paper investigates the viability of deploying general-purpose, open-weight Large Language Models (LLMs) in heavily constrained, offline inference environments to solve olympiad-level mathematical problems. Operating under the strict hardware limitations of a single NVIDIA H100 GPU and a 5-hour time budget for 50 problems, we introduce a Python-driven, Multi-Turn Tool-Integrated Reasoning (TIR) orchestration pipeline. Rather than relying on specialized fine-tuning, this study optimizes dynamic resource allocation, context management, and hyperparameter targeting using the Qwen 3.5 35B-A3B Mixture-of-Experts (MoE) architecture. We evaluate the efficacy of zero-shot general-purpose models acting as Process Reward Models (PRMs) and demonstrate the necessity of a ranking-based evaluation layer. Achieving a public score of 14/50 on the AI Mathematical Olympiad (AIMO) Progress Prize 3, our results highlight both the emerging capabilities of generalized reasoning models and their persistent vulnerabilities to logical hallucination when operating without domain-specific mathematical post-training.
\nThe pursuit of artificial mathematical reasoning has recently bifurcated into two primary methodologies: domain-specific fine-tuning on highly curated mathematical corpora—exemplified by systems like AlphaGeometry (Trinh et al., 2024)—and the deployment of massive, generalized reasoning models (such as OpenAI's o1 paradigm). Faced with limited computational resources and a restricted timeline, this study opted to test the upper bounds of the latter approach. Instead of training custom weights or employing proprietary evaluation models, we utilized the AIMO Progress Prize 3 as a platform to test how far general-purpose LLMs can advance on a demanding set of problems using purely inference-time orchestration.
\nOur findings demonstrate that while sophisticated inference-time compute scaling and Tool-Integrated Reasoning (TIR) can yield competitive baselines, the concept of an unmodified, general-purpose model reliably excelling at olympiad-level mathematics without specialized post-training remains, at this point, a summer night's dream.
\nThe foundational requirement of our architecture was an offline pipeline capable of benchmarking general-purpose models without requiring specialized evaluators or math-specific weights. Early iterations of this approach mandated a pivot to Alibaba's Qwen MoE family, specifically transitioning from the FP8 quantized variant of Qwen3-30B-A3B to the newer Qwen3.5-35B-A3B (Qwen Team, 2024; Qwen Team, 2025).
\nThe model's weights ($\\approx 30$ GB) fit comfortably on the Kaggle-provided NVIDIA H100 GPU, allocating sufficient space for the KV Cache required by long-context reasoning chains. The deployment relied on vLLM (Kwon et al., 2023), capitalizing on the model's seamless integration of a \"thinking\" mode toggle and its top-tier ability to adhere to strict formatting constraints without excessive prompt engineering.
The core architecture operates as a Tool-Integrated Reasoning loop driven entirely by Python, building upon recent successes in Program-Aided Language models (Gao et al., 2023; Gou et al., 2023). The reliance on Python is predicated on the observation that general-purpose models are heavily fine-tuned for Software Engineering (SE) tasks, rendering them significantly more capable of producing accurate, executable Python code than symbolic representations like Lean.\n\nFigure 1: The Multi-Turn Tool-Integrated Reasoning (TIR) orchestration pipeline, highlighting the decoupling of the Planner (code generation) and the Judge (ranking evaluation).
The \"heartbeat\" of the loop is the Planner. The system prompts force the Planner to frame problems as sequential computational steps toward a solution. Instead of attempting a one-shot mathematical proof—which requires managing massive internal reasoning chains—the model focuses on identifying the next computational step to test a specific hypothesis.
\nConstraints & Fitness Checks:
\nProhibited Methods: Probabilistic search space traversals (e.g., Monte Carlo simulations) and inefficient libraries (e.g., SciPy for symbolic geometry) are strictly prohibited, as they rarely yield the exact integers required by the AIMO format.
If a candidate script employs prohibited methods, an intricate validation loop performs rigorous fitness checks, and the candidate is immediately discarded.
Whether a candidate ultimately outputs a solution is decided by a majority vote, leveraging the principles of self-consistency (Wang et al., 2022) to filter out anomalous logical errors. The model outputs a JSON payload containing a boolean field indicating if the candidate script is intended to be a final solution. If a majority flags a solution, it is evaluated, submitted, and passed through strict regex checks to ensure it is a valid integer modulo $10^5$.
\nTwo auxiliary roles interact with the problem utilizing the model's reasoning capabilities:
\nThe Classifier: Analyzes the raw problem and outputs a structured rephrasing, primarily consisting of explicit tasks and constraints. This format aligns optimally with the Planner's computational disposition. (Note: The nomenclature \"Classifier\" is a vestige of an earlier iteration intended to classify problem difficulty for dynamic step allocation, an approach abandoned due to model inconsistency).
The Final Verdict: Invoked only when the orchestration loop reaches its absolute limit (due to time constraints or max step exhaustion). Utilizing all evidence gathered in the context window, it synthesizes a final, educated approximation of the integer solution.
State-of-the-art agentic mathematical pipelines rely heavily on evaluation models, specifically Outcome Reward Models (ORMs) and Process Reward Models (PRMs). Initial attempts to prompt our general-purpose Judge to mimic a strict PRM demonstrated that unmodified LLMs struggle to handle this gracefully. The PRM setup tested was heavily adapted from the step-wise evaluation format introduced by Lightman et al. The prompt instructed the LLM to read the generated reasoning (the content inside the <think>} blocks, which was computationally expensive to process) and assign a token classification of VALID, INVALID, or NEUTRAL to each logical step, calculating a final confidence score based on the ratio of valid steps.
The data point that forced a pivot was the high false-positive rate. General-purpose models exhibited severe \"confirmation bias\" when acting as zero-shot PRMs; if an early mathematical step contained a subtle arithmetic error but the subsequent logic flowed consistently from that error, the model would confidently grade the remaining steps as VALID, leading to a high overall score for an incorrect solution. Switching to a Ranking Judge bypassed this step-wise hallucination by forcing the model into a comparative analysis (Solution A vs. Solution B), which proved vastly more reliable zero-shot. Furthermore, as demonstrated in existing research, such evaluators work optimally with specifically fine-tuned models, which fundamentally contradicted our objective of evaluating general-purpose models. Switching to a Ranking Judge bypassed this step-wise hallucination by forcing the model into a comparative analysis (Solution A vs. Solution B), which proved vastly more reliable zero-shot. It is important to clarify that while the Judge internally evaluates a ``ranking'' of the reasoning traces, the extraction layer strictly forces a winner-takes-all selection. Confining the final output to a single optimal candidate, rather than a complex ordered list, significantly mitigated the output-space instability typically caused by high inference temperatures.
Executing all candidates prior to the Judge's evaluation was tested but ultimately scrapped. The general-purpose model exhibited a strong bias, clinging to any candidate that produced a single, clean integer, even during early exploratory steps where the objective was search-space mapping. Furthermore, running a multitude of parallel Python sandboxes severely degraded the computational budget.
\nThe Execute Loop runs the selected candidate and returns the standard output. Extensive error handling is implemented to ensure the model receives full tracebacks explaining execution failures.
\nTo preserve compute, a fast Syntax Reviewer agent was introduced. Before a candidate is fully scrapped, the Reviewer attempts to resolve trivial syntactical errors (e.g., indentation errors, missing imports). This reviewer consistently rescued highly promising mathematical algorithms during reference set evaluations.
\nA critical challenge in the AIMO environment is managing the strict 18,000-second offline execution limit. To prevent infinite loops or geometrically expanding search spaces on a single problem, a custom DynamicTimeController was engineered to regulate the temporal and operational bounds of the TIR loop.
Let $T_{total}$ be the total available environment time, $N$ be the total number of problems, and $T_{safe}$ be a rigid teardown safeguard (230 seconds) reserved for final CSV generation and vLLM spin-down. Before attempting the $i$-th problem, the remaining budget $B_i$ is calculated as:
$$B_i = T_{total} - T_{safe} - \\sum_{j=1}^{i-1} t_j$$
\nwhere $t_j$ is the time spent on previous problems. Rather than enforcing a static limit, the budgeter permits the agent to utilize surplus time saved from earlier, easier problems. For the current problem, we define a dynamic temporal ceiling, $T_{max, i}$, controlled by a relaxation factor $\\gamma$ (where $\\gamma \\approx 1.5$):
\n$$T_{max, i} = \\min \\left( \\gamma \\cdot \\frac{B_i}{N - i + 1}, \\;\\; B_i \\right)$$
\nWithin this ceiling, reasoning depth is regulated by a maximum step count ($S_{max} = 5$). Let $t_{infer}$ be the variable inference time and $t_{exec}$ be the sandbox execution time. We enforce:
\n$$\\sum_{k=1}^{S} (t_{infer, k} + t_{exec, k}) \\leq T_{max, i}$$
\nTo guarantee progression, the Python subprocess sandbox is rigidly capped at 15 seconds per execution. If $\\max(t_{exec, k})$ reaches 15, a TimeoutError is piped into the LLM context, punishing inefficient while loops and forcing the model to re-evaluate its logic.
Due to the absence of weight fine-tuning, the \"training\" equivalent in this methodology was the rigorous optimization of generation parameters and VRAM allocation on the Kaggle H100 GPU.
\nVRAM and Context Constraints: The vLLM engine was instantiated with enforce_eager=True to disable CUDA graph capture, successfully preventing Out-of-Memory (OOM) exceptions when managing a 32,768 native context window alongside max_num_seqs=16. The KV Cache was quantized to FP8, doubling the functional context capacity.
Context Pruning: To prevent overflowing the native window when processing tracebacks, the history limit was capped at 15,000 tokens, and \\texttt{stdout} limits were aggressively truncated. Crucially, to prevent catastrophic forgetting and ``loopy'' context degradation, the heavy \\texttt{} reasoning traces were systematically stripped from memory after each step. The pipeline relied on the Summarizer role to compress the global state of the problem, ensuring the model retained its trajectory without drowning in its own previous reasoning loops.
Decoding Strategies: Following Qwen 3 guidelines, greedy decoding was abandoned for reasoning roles. The Planner utilized $T=0.6$, while analytical roles (Judge, Summarizer) utilized cooler temperatures ($T=0.3$) to maintain determinism. Crucially, presence_penalty was explicitly set to $0.0$ across all agents to prevent the fatal hallucination of JSON structures and mathematical syntax.
The pipeline achieved a public score of 14/50. While modest compared to fine-tuned or heavily scaled proprietary models, this score establishes a robust baseline for inference-only, general-purpose MoE models operating under severe zero-shot constraints.\n\nFigure 2: Execution time and cumulative accuracy progression for the Qwen 3.5 35B-A3B agent across the offline 30-problem AIME benchmark, demonstrating the Dynamic Time Controller's budget adherence.\n\nFigure 3: Comparative performance of the Qwen 3 30B-A3B agent under identical hyperparameter and time-budget constraints. (Note the variance in generation stability compared to the 3.5 architecture).
A comparison of Figures 2 and 3 highlights a significant discrepancy in execution time and budget adherence. This variance stems from a combination of the dynamic budget logic, step counts, and the underlying model architectures. During early testing, hard-capping the time per problem often starved earlier problems of setup time. To counter this, the Time Controller allowed early problems to ``borrow'' compute budget against the global 5-hour limit, operating on the premise that a capable model would converge quickly and not exhaust this surplus.
\nBecause Qwen3 struggled more with initial logical paths, it failed to converge to a solution as fast as its newer sibling. Consequently, Qwen3 weaponized the ``borrowed'' early time against itself by exhausting its maximum allowed reasoning steps on earlier problems, leading to the late-stage timeout shown in the benchmark. In contrast, Qwen3.5’s hybrid Gated DeltaNet / Attention architecture allowed for significantly faster token generation and context processing, executing the loops faster and with fewer errors.
\nA genuine limitation of this ablation study is the sequential problem ordering used in the AIME benchmark. Due to strict compute limitations and the impending revocation of Kaggle H100 access at the competition's deadline, the benchmark was hastily executed in its default sequential order, which inherently ramps up in difficulty (1-15, 16-30). This default difficulty ramp interacted poorly with the ``borrowing'' mechanism. Because early problems were easier, Qwen3.5 solved them quickly, whereas Qwen3 struggled significantly from the third problem onwards, exhausting its time before ever encountering the hardest problems. Running this benchmark with a randomized problem order would have undoubtedly isolated the model's baseline compute requirements from the controller's behavioral biases, an oversight that must be addressed in future methodological iterations.
\nTo ensure full reproducibility of the inference pipeline and benchmarking results, the complete\nsource code, vLLM configurations, and environment setups are publicly available on Kaggle.
\nNote on Model Access and Configuration: While the orchestration logic and notebooks are\nfully public, the specific Kaggle dataset containing the model weights used for this submission\nremains private. However, the pipeline is designed to be highly modular. Researchers can\nseamlessly plug in their own model paths and adjust all core vLLM and pipeline hyperparameters\n(e.g., maximum reasoning steps, temperature, and context limits) by simply modifying the\nconfiguration variables located in the third cell of the notebooks.
\nThis study demonstrates that advanced inference orchestration—specifically through dynamic time budgeting, rigorous tool integration, and specialized agent routing—can successfully extract competitive mathematical reasoning capabilities from general-purpose LLMs. However, the persistent reliance on logical hallucinations underscores a critical barrier. Until general-purpose models develop innate, rigorous process-reward architectures natively, achieving Olympiad-level mastery without domain-specific post-training remains a summer night's dream.
\nAnother conclusion that I reached after reading through some of the top write-ups is how important parameter count is when it comes to reducing hallucinations and producing more solid reasoning, which seems trivial knowledge, however, I wrongfully believed that a 35B model that has not been heavily quantized would be less likely to hallucinate, which was a lapse in judgment, evident by the leaderboard, and an oversight I should probably have covered.
\nEarly on, I had tried a heavier model, the results of them I sadly do not posses since it was very early and I hadn't documented them overall, I saw however significantly slower reasoning to the point I could barely do any scaffolding around it, and a big reason why I chose to stay around 30B and in particular go with MoE architectures was that I wanted fast and computationally cheap reasoning, to push a heavier context window and more retries/candidates, since I didn't find a model that I could trust. Furthermore, most of the fine-tuned models from progress prize 2 were 7B models, adding to my conviction that overall ended up not being correct or at least not adapted to the latest SOTA models, and in particular one that shaped the public leaderboard as I was conducting my own experiments.
\nThat model was GPT-OSS-120B which seemed to have very promising out of the box results, making up for it's computational cost and increased quantization in raw reasoning performance . I lack the proper compute to thoroughly test my scaffold on it, since we no longer have access to H100 from the competition, however, I can safely assume my results would have drastically improved, thus my premise for choosing a 35B model was all in all proven either false, or at the very least obsolete.
\nFrieder, S., Bealing, S., Vonderlind, P., Li, S., Nikolaiev, A., Smith, G. C., Buzzard, K., Gowers, T., Liu, P. J., Loh, P.-S., Mackey, L., de Moura, L., Roberts, D., Sculley, D., Tao, T., Balduzzi, D., Coyle, S., Gerko, A., Holbrook, R., Howard, A., & XTX Markets. (2025). AI Mathematical Olympiad - Progress Prize 3. Kaggle. https://kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3
Gao, L., Maman, A., Banerjee, S., Goswami, V., Kamalloo, E., Mackey, L., & Aggarwal, G. (2023). PAL: Program-Aided Language Models. In Proceedings of the 40th International Conference on Machine Learning (ICML).
Gou, Z., Shao, Z., Gong, Y., Yang, Y., Huang, M., Duan, N., Chen, W., et al. (2023). ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving. In Proceedings of the International Conference on Learning Representations (ICLR) 2024. arXiv preprint arXiv:2309.17452.
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles (SOSP '23).
Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Sutskever, I., et al. (2023). Let's Verify Step by Step. arXiv preprint arXiv:2305.20050.
OpenAI. (2024). Learning to Reason with LLMs. OpenAI Research. https://openai.com/index/learning-to-reason-with-llms/
OpenAI. (2025). Introducing GPT-OSS. OpenAI Research.. https://openai.com/index/introducing-gpt-oss/
Qwen Team. (2024). Qwen2.5 Technical Report. arXiv preprint arXiv:2412.15115.
Qwen Team. (2025). Qwen3 Technical Report. arXiv preprint arXiv:2505.09388.
Trinh, T. H., Wu, Y., Le, Q. V., He, H., & Luong, T. (2024). Solving olympiad geometry without human demonstrations. Nature, 625(7995), 476–482.
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022). Self-Consistency Improves Chain of Thought Reasoning in Language Models. In Proceedings of the International Conference on Learning Representations (ICLR) 2023. arXiv preprint arXiv:2203.11171.
I took the opportunity of the Rebuttal phase to make a few targeted updates to my write-up. Following the questions from the reviewers @philipvonderlind @sambealing , I ended up making the following edits eligible for the rebuttal phase of the competition:
\nExpanded Conclusion (Model Selection Post-Mortem)
\nWhat was added: 3 paragraphs in the Conclusion discussing the GPT-OSS 120B meta and the quantization vs. parameter count trade-off.
When: At the opening of the rebuttal phase (prior to reviewer questions).
What it addresses: Acts as a self-reflection acknowledging that my initial premise regarding a 35B model hallucinating less than a heavily quantized 120B model was rendered obsolete by the final leaderboard results.
Expanded Section 3.3 (The Judge & PRM Pivot)
\nWhat was added: Detailed the specific PRM prompt used (adapting Lightman et al., classifying blocks) and clarified that the ranking mechanism uses a \"winner-takes-all\" extraction rather than outputting a volatile list.
When: During the reviewer Q&A phase.
What it addresses: Addresses reviewer questions regarding the exact PRM methodology, the LLM \"confirmation bias\" flaw, and concerns about ranking instability at high inference temperatures.
New Subsection in Section 6 (Ablation Limitations & Budget Overshoots)
\nWhat was added: A dedicated subsection titled \"Ablation Study Limitations and Time-Budget Overshoots\".
When: During the reviewer Q&A phase.
What it addresses: Directly answers why Qwen3 weaponized the \"borrowed\" time budget compared to Qwen3.5 (highlighting the Gated DeltaNet efficiency), and acknowledges the sequential AIME difficulty ramp as a genuine limitation of the ablation study.
Updated Section 5 (Catastrophic Forgetting & Context)
\nWhat was added: Clarified the Context Pruning bullet point to explicitly state that blocks are systematically stripped from memory to prevent context degradation.
When: During the reviewer Q&A phase.
What it addresses: Resolves reviewer questions regarding how the pipeline prevents catastrophic forgetting and \"loopy\" behavior within the 32k character limit.
Adding to the comments by @sambealing below, I have a few additional questions:
\nQwen: In your submission, you write that “early iterations of this approach mandated a pivot to Alibaba's” model. From what I have seen, most other competitors concluded that gpt-oss-120b was the stronger choice under the available hardware budget. What motivated your decision to use a less common model for your submission? Did you find any evidence that supported this choice?
Judge Mechanism: Unlike many other approaches, your judge ranks the generated answers rather than selecting a single final answer directly. This seems like a more difficult task, since comparing subtle differences between multiple answers or reasoning traces is not necessarily straightforward. Did you run any experiments to check whether the produced rankings are stable across multiple runs, given that inference temperature is involved? I would expect ranking instability to have a large impact on problems where the solution attempts are highly diverse.
Eager Inference: Why did you choose to disable CUDA graph execution entirely? Reducing the graph size could also save memory while retaining some throughput benefits. This may also interact with your relatively high N=16, since larger numbers of parallel generations require more memory depending on how many sequences are active at once. It would be interesting to see how these settings affect the overall throughput of your pipeline.
Context Limitation and Catastrophic Forgetting: From my experience, Qwen models are particularly prone to generating very long traces. Since you truncate the trace history at 32k characters, did you observe any catastrophic forgetting of the original task or “loopy” behavior, where the model starts rediscovering previously forgotten context after earlier parts of the trace are removed?
Hi, thank you for taking the time to evaluate my work and for the thoughtful questions. Please find my clarifications below:*
\n1. Qwen Selection & The 120B Meta:\nYou are completely correct regarding the leaderboard meta. My decision to use Qwen 3.5 35B over GPT-OSS-120B was driven by three primary hardware and architectural constraints early in the competition:
\nQuantization vs. Parameter Count Trade-off: I operated under the theoretical premise that a 35B model with less aggressive quantization (FP8) would hallucinate less and maintain better logical consistency than a 120B model heavily quantized to fit on the same H100. This was honestly my own conviction driven by the fact I constantly saw smaller models over-achieving in earlier AIMO competitions.
KV-Cache Headroom: Because my pipeline relied heavily on parallel candidate generation, I needed significant VRAM headroom for the KV-Cache. I also wanted to maintain the ability to push parallel generation AND KV-Cache much beyond what I ended up doing.
Native Reasoning Switch: Qwen offered a very stable native reasoning/think switch and exceptional instruction-following for Python generation out-of-the-box.
\nAs I noted in the recent \"Limitations\" edit to my write-up, that I wrote when the rebuttal phase was announced, my premise regarding the 35B vs. 120B quantization trade-off was ultimately proven obsolete by the final leaderboard, but these were the constraints that locked in my architecture early on.
2. Judge Mechanism & Ranking Instability:
\nI apologize if the terminology in the write-up was ambiguous. While the Judge was prompted to \"rank\" the solutions to force a comparative analysis, it did not output a full, multi-item ranking downstream. The prompt forced an internal comparison of the reasoning traces, but the extraction layer strictly required a \"winner-takes-all\" selection of the single best candidate to proceed. By confining the Judge's final output to a single selection rather than a complex ordered list, I mitigated much of the output-space instability typically caused by high inference temperatures, though I agree that highly diverse candidate pools still introduced variance.
\n3. Eager Inference vs. CUDA Graphs:
\nThis was a stability and initialization trade-off. For the majority of my development cycle, stable vLLM support for the newest Qwen MoE architectures was non-existent. I was relying on nightly wheels up until the final week. Because Kaggle kernels frequently took nearly 20 minutes just to spin up the pipeline and load the model weights, attempting to capture and compile optimized CUDA graphs often resulted in unpredictable OOM errors or timeouts before inference even began. Disabling CUDA graph execution entirely enforce_eager=True was a brute-force measure to guarantee the engine would initialize safely within the Kaggle environment, though I agree that tuning the graph size would have yielded better throughput.
4. Context Limitation & Catastrophic Forgetting:
\nTo prevent catastrophic forgetting within the 32k character limit, I implemented an aggressive context-pruning strategy. Instead of letting the raw context naturally truncate (which causes the \"loopy\" behavior you described), my pipeline systematically stripped out the heavy \\texttt{} reasoning traces from memory after a step was completed, heavily truncated stdout tracebacks, and relied on the Summarizer role to compress the state of the problem. This strict memory management ensured the model retained the \"global state\" of its progress without drowning in its own previous reasoning loops, while losing some of the fine details of it's CoT, which was needed given the limiations of my implementation.
\nI hope my responses were adequate.\nBest Regards!
", "rawMarkdown": "Hi, thank you for taking the time to evaluate my work and for the thoughtful questions. Please find my clarifications below:*\n\n**1. Qwen Selection & The 120B Meta:**\nYou are completely correct regarding the leaderboard meta. My decision to use Qwen 3.5 35B over GPT-OSS-120B was driven by three primary hardware and architectural constraints early in the competition:\n\n- **Quantization vs. Parameter Count Trade-off:** I operated under the theoretical premise that a 35B model with less aggressive quantization (FP8) would hallucinate less and maintain better logical consistency than a 120B model heavily quantized to fit on the same H100. This was honestly my own conviction driven by the fact I constantly saw smaller models over-achieving in earlier AIMO competitions.\n \n- **KV-Cache Headroom:** Because my pipeline relied heavily on parallel candidate generation, I needed significant VRAM headroom for the KV-Cache. I also wanted to maintain the ability to push parallel generation AND KV-Cache much beyond what I ended up doing.\n \n- **Native Reasoning Switch:** Qwen offered a very stable native reasoning/think switch and exceptional instruction-following for Python generation out-of-the-box.\n \n As I noted in the recent \"Limitations\" edit to my write-up, that I wrote when the rebuttal phase was announced, my premise regarding the 35B vs. 120B quantization trade-off was ultimately proven obsolete by the final leaderboard, but these were the constraints that locked in my architecture early on.\n \n---\n**2. Judge Mechanism & Ranking Instability:**\n\nI apologize if the terminology in the write-up was ambiguous. While the Judge was prompted to \"rank\" the solutions to force a comparative analysis, it did not output a full, multi-item ranking downstream. The prompt forced an internal comparison of the reasoning traces, but the extraction layer strictly required a \"winner-takes-all\" selection of the single best candidate to proceed. By confining the Judge's final output to a single selection rather than a complex ordered list, I mitigated much of the output-space instability typically caused by high inference temperatures, though I agree that highly diverse candidate pools still introduced variance.\n\n---\n**3. Eager Inference vs. CUDA Graphs:**\n\nThis was a stability and initialization trade-off. For the majority of my development cycle, stable vLLM support for the newest Qwen MoE architectures was non-existent. I was relying on nightly wheels up until the final week. Because Kaggle kernels frequently took nearly 20 minutes just to spin up the pipeline and load the model weights, attempting to capture and compile optimized CUDA graphs often resulted in unpredictable OOM errors or timeouts before inference even began. Disabling CUDA graph execution entirely `enforce_eager=True` was a brute-force measure to guarantee the engine would initialize safely within the Kaggle environment, though I agree that tuning the graph size would have yielded better throughput.\n\n---\n**4. Context Limitation & Catastrophic Forgetting:**\n\nTo prevent catastrophic forgetting within the 32k character limit, I implemented an aggressive context-pruning strategy. Instead of letting the raw context naturally truncate (which causes the \"loopy\" behavior you described), my pipeline systematically stripped out the heavy \\texttt{} reasoning traces from memory after a step was completed, heavily truncated stdout tracebacks, and relied on the Summarizer role to compress the state of the problem. This strict memory management ensured the model retained the \"global state\" of its progress without drowning in its own previous reasoning loops, while losing some of the fine details of it's CoT, which was needed given the limiations of my implementation.\n\n---\n\nI hope my responses were adequate.\nBest Regards!" } ] }, { "id": 3457718, "postDate": "2026-05-14T08:22:12.173Z", "content": "Hi, thank you for your submission to the Writeup Prize. @philipvonderlind and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.
\naime_2026_II_05 (about 9 problems unattempted). The same hyperparameters and time budget were used as for Qwen3.5-35B-A3B in Figure 2, which mostly stayed within budget. Do you have any data on why the older model consumed so much more compute per problem (e.g. longer reasoning traces, more tool-call retries, judge-rejection loops)?Look forward to reading your responses on these.
", "rawMarkdown": "Hi, thank you for your submission to the Writeup Prize. @philipvonderlind and I will post some questions below that help us in assessing your submission more closely, as described in the [updated discussion post](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/689703).\n\n1. **ORM/PRM to ranking evaluator pivot:** You abandoned the PRM approach because \"unmodified LLMs struggle to handle this gracefully\" (§3.3), but don't say much about what specific PRM-style rubric or prompts you tried first. Could you share the PRM setup you tested, and what data you collected to support that the ranking-based evaluator outperformed it?\n2. **Time-budget overshoots on Qwen3-30B-A3B:** Figure 3 shows several problems on the Qwen3-30B-A3B run taking 1000-1400s against the 360s average budget, and the system runs out of time around `aime_2026_II_05` (about 9 problems unattempted). The same hyperparameters and time budget were used as for Qwen3.5-35B-A3B in Figure 2, which mostly stayed within budget. Do you have any data on why the older model consumed so much more compute per problem (e.g. longer reasoning traces, more tool-call retries, judge-rejection loops)?\n3. **AIME ordering and difficulty ramp:** AIME problems come in two sets of 15 in roughly increasing difficulty (1-15 and 16-30), so the time controller effectively sees two difficulty ramps from easy to hard. Do you think this could have impacted the results and if so, did you consider running the benchmark with a randomised problem order to remove this effect?\n\nLook forward to reading your responses on these.\n", "replies": [ { "id": 3457816, "postDate": "2026-05-14T11:32:04.687Z", "content": "Hi, Thank you for the thorough review and the questions. Below I provide my initial response to each one of these.
\n1. ORM/PRM to ranking evaluator pivot: Unfortunately (and this was an honest mistake on my part for not keeping my earlier notebooks), I did not preserve the specific Kaggle notebook of my initial zero-shot PRM experiments once I finalized the Ranking-Based Judge architecture. However, the PRM setup I tested was heavily adapted from the step-wise evaluation format introduced in \"Let's Verify Step by Step\" (Lightman et al., 2023).
\nMy prompt instructed the LLM to read the generated reasoning (the content inside the <think> </think> blocks, which was quite time expensive on it's own) and assign each sentence and latex block a token classification of [VALID], [INVALID], or [NEUTRAL] to each logical step, calculating a final confidence score based on the ratio of valid steps. The data point that forced me to pivot was the high false-positive rate. General-purpose models exhibited severe \"confirmation bias\" when acting as zero-shot PRMs; if an early mathematical step contained a subtle arithmetic error but the subsequent logic flowed consistently from that error, the model would confidently grade the remaining steps as [VALID], leading to a high overall score for an incorrect solution. Switching to a Ranking Judge bypassed this step-wise hallucination by forcing the model into a comparative analysis (Solution A vs. Solution B), which proved vastly more reliable zero-shot.
In general as demonstrated in OpenAI's research, such evaluators work with specific fine-tuned models, which went against my objective of evaluating General Purpose Models
\n2. Time-budget overshoots on Qwen3-30B-A3B: This discrepancy comes down to a combination of my dynamic budget logic, step counts, and the underlying model architectures. During early testing, I observed that hard-capping the time per problem often starved earlier problems of the setup time they needed, leaving excessive, unusable time for later problems (since I strictly limited the maximum number of reasoning steps). To counter this, my Time Controller allowed early problems to \"borrow\" compute budget against the global 5-hour limit. This was developed on the premise that Qwen3.5 would be able to discard the unneeded time by convering to a solution and not eat through it's budget. Because Qwen3 struggled more with initial logical paths it struggled to converge to a solution as fast as it's newer sibling and ate through it's time.
\nConsequently, Qwen3 actively consumed this \"borrowed\" early time by exhausting its maximum allowed reasoning steps for the earlier problems. Given my setup was quite lenient in letting the earlier problems borrow significant time, Qwen3 ended up eating up all of it's time before it reached the later problems. In contrast to Qwen3.5. Furthermore, Qwen3.5’s hybrid Gated DeltaNet / Attention architecture allowed for significantly faster token generation and context processing. Qwen3.5 simply executed the loops faster and with fewer errors, whereas Qwen3 weaponized the \"borrowed\" time against itself, leading to the late-stage timeout shown in the AIME benchmark, since it used the borrowed time on easier problems.
\n3. AIME ordering and difficulty ramp: This is a fantastic and highly insightful observation, as this is a genuine limitation of my ablation study. Given the strict compute limitations and the fact that I was running these benchmarks using Kaggle's finite GPU hours, I hastily executed the AIME benchmark in its default sequential order to generate the ablation plots for the write-up before access to H100 was revoked. Given I was working on my submission up until hours before the deadline, I didn't have the time I wanted to run my ablation studies, so I just ran something as fast as I could in order to have something to show for my write-up.
\nThat said, I completely agree that the default difficulty ramp of AIME interacted poorly with my \"borrowing\" mechanism. Because the early problems were easier, the Qwen 3.5 was able to solve them significantly quicker, while Qwen3 struggled singificantly from the third problem onwards, which means it used up all the time without ever tackling the harder problems. Running this benchmark with a randomized problem order would have undoubtedly isolated the model's baseline compute requirements from the controller's behavioral biases. It is a limitation of my testing methodology that I will surely take into account for future work.
\nI appreciate the time you took to evaluate my work,
\nBest Regards
", "rawMarkdown": "Hi, Thank you for the thorough review and the questions. Below I provide my initial response to each one of these.\n\n**1. ORM/PRM to ranking evaluator pivot:** Unfortunately (and this was an honest mistake on my part for not keeping my earlier notebooks), I did not preserve the specific Kaggle notebook of my initial zero-shot PRM experiments once I finalized the Ranking-Based Judge architecture. However, the PRM setup I tested was heavily adapted from the step-wise evaluation format introduced in _\"Let's Verify Step by Step\"_ (Lightman et al., 2023).\n\nMy prompt instructed the LLM to read the generated reasoning (the content inside the `If you want this writeup to be considered towards the Write-Up Prize, you need to make a comment here: https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/689703
\nYou will need to follow the instructions in that comment, and link to the writeup and also provide some further links to either HuggingFace or Google Colab notebook.
", "rawMarkdown": "If you want this writeup to be considered towards the Write-Up Prize, you need to make a comment here: https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/689703 \n\nYou will need to follow the instructions in that comment, and link to the writeup and also provide some further links to either HuggingFace or Google Colab notebook." }, { "id": 3456915, "postDate": "2026-05-12T18:30:47.810Z", "rawMarkdown": "", "isDeleted": true } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T04:21:07Z", "competition_id": 118448, "competition_slug": "ai-mathematical-olympiad-progress-prize-3", "team_id": 15183920, "solution_url": "/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/pushing-the-limits-of-general-purpose-llms-on-math" }