![\"Host](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F645831%2Fc3afa40a42ea107551dffdbc15c86c9c%2Fhostanalysis2.png?generation=1776744674727446&alt=media\")
We present AMO(Agentic Math Orchestrator), a multi-layer orchestration architecture designed to maximize the inference-time capabilities of frozen open-weight Large Language Models (LLMs) on olympiad-level mathematics. Operating under a strict 5-hour, single-GPU competition environment without internet access, AMO wraps GPT-OSS-120B with parallel reasoning, an asynchronous verification pipeline, real-time trajectory auditing, evidence-weighted answer selection, and topology-aware routing. Rather than fine-tuning the model, we treat mathematical robustness as a systems-engineering challenge. We introduce novel mechanisms including a pre-generation Scout Pool for structural classification, an Adaptive Orchestrator for continuous time-budgeting, dynamic switching between Standard and Agentic Beam Search modes, and a Master Vision conflict-resolution pipeline. We achieved a top public score of 45 using our baseline architecture and 41 using the fully orchestrated AMO version. Through extensive ablations, trace analysis, and local benchmark testing, we document specific LLM failure modes in competitive mathematics—such as extremum bias, execution timeouts, and frame-sensitive divergence—and present our architectural mitigations.
\nThe AI Mathematical Olympiad (AIMO) Progress Prize 3 presents a unique test-time compute challenge: solve 50 original, olympiad-level mathematical problems spanning algebra, combinatorics, geometry, and number theory within a 5-hour wall-clock budget on a single H100 GPU. The environment is strictly constrained—no internet access, no external APIs, and no opportunity for between-submission fine-tuning. Furthermore, the competition scoring heavily penalizes inconsistency; each submission is evaluated twice on the private set, awarding a half-point only if one run is correct. Consequently, mathematical robustness under inference constraints matters as much as the raw capability of the underlying model.
\nThis paper addresses a central challenge: given a powerful, frozen open-weight model (GPT-OSS-120B), how do we build an agentic system that converts high theoretical pass@K capability into reliable, single-submission correctness? The baseline notebooks on OSS120B had shown the high ceiling of it. With the limited time and resources in the competition period, we decided to focus on improving the efficiency and stability of oss120B solvers by creating an agentic dynamic time aware system to maximize the ceiling and floor. Through careful analysis we found many failure points, which could be solved/improved by optimized agentic solutions rather than focusing on a fine tuning to hope for a blackbox solution.
\nThe standard baseline approach to this problem—simply running $N$ parallel Tool-Integrated Reasoning (TIR) solvers and applying majority voting—is highly inefficient. Blind parallel generation wastes massive amounts of compute on \"doomed\" reasoning paths. If a solver misinterprets the problem in Turn 1, the subsequent twenty minutes of generation and Python execution are effectively dead compute. Furthermore, blind majority voting is vulnerable to \"false majorities,\" where multiple solvers anchor onto the same flawed assumption and confidently confirm one another's incorrect answers.
\nTo revolutionize this baseline, we introduce the Agentic Math Orchestrator (AMO). AMO steps away from blind parallel generation, transforming the LLM into a highly structured, asynchronous multi-agent research team. The entire lifecycle of a problem is meticulously managed from end to end:
\nA cornerstone of our architecture is the Agentic Beam Search. Designed to artificially compress the model’s $pass@20$ capability into a smaller compute footprint, this mode pauses parallel solvers mid-flight to conduct adversarial self-evaluations. By pruning mathematically unviable trajectories and applying Maximum Marginal Relevance (MMR) text-clustering to the surviving paths, we force mathematical diversity. This ensures the model searches fundamentally distinct frameworks rather than racing down identical dead ends, drastically increasing the odds of finding the correct solution.
\nTo reliably extract the correct answer from this agentic swarm, we replaced simple majority voting with a Multi-Tiered Evidence Hierarchy. Answers are selected based on rigorous criteria: Resolver agent voting weight, Verifier confidence scores (where adversarial Python scripts attempt to falsify the claim), and \"Code Supremacy\" (awarding bonuses to answers explicitly proven by constraint solvers like z3,ortools, pulp etc. and python tool output verified answer). While we initially explored using token-level entropy dynamics as a primary selection parameter, extensive testing revealed that without a trained calibration model, entropy added noise (especially in python tool based agents)—fluent hallucinations were often indistinguishable from genuine computation. Similarly, early attempts at inter-agent collaboration (such as SwarmRAG or sharing intermediate reasoning bulletins) didn't get into the final submission, after we observed they created echo chambers of correlated errors.
Finally, treating robust inference as a systems-engineering challenge required deep forensic visibility. To build this system, we logged every solver trace including reasoning step, tool call, and token metric as well as the scout/summarizer/verifier/master vision reports from extensive local benchmark runs. We developed a custom Streamlit-based Trace Viewer to replay these multi-turn conversations. This visualizer was instrumental in identifying specific model pathologies—such as infinite loop traps, premature guessing, and format hacking. Insights derived directly from this Trace Viewer birthed our Regex Flaw Registry, a real-time monitoring system that penalizes flawed trajectories and injects mid-flight corrective nudges to steer solvers back on track.
\nIn the following sections, we detail the complete AMO architecture, documenting our dynamic resource scaling, multi-mode solver logic, and the empirical ablations that define our approach to inference-time mathematical orchestration.
\nRecent evaluations published by the competition hosts compared the results of open-weight models (OSS-120B vs. Nemotron-30B) across both public and private sets. While these evaluations were likely conducted without strict inference-time constraints and lacked access to Python execution tools, they provided a critical baseline for the raw reasoning capability of OSS-120B.
\n
The key finding: at pass@100, only 1–2 problems across both sets remained completely unsolvable. At pass@20, a strong open-weight model (class-equivalent to OSS-120B) averaged 43–48 on the private set, with the public and private difficulty distributions showing overlapping 95% confidence intervals. This analysis dictates our fundamental strategy:
\nThe Host's analysis was done on OSS120B models without python tools. So the ceiling of OSS120B with python tool should be higher.
\nThrough forensic trace analysis of our local benchmarks (detailed in Section 3.1), we categorized LLM failures not by mathematical subject (e.g., Geometry vs. Algebra), but by behavioral pathology:
\nimo-bench-algebra-092), where the model clearly overfitted to training data and converged on a known, but incorrect, schema.imo-bench-algebra-055 (a maximum problem), we observed 4 solvers converging on 1055, while only 1 solver correctly found 2079. Under standard majority voting, the system would confidently select the wrong, sub-optimal bound.The baseline notebook provided by the competition using OSS-120B with 8 parallel solvers and simple majority voting already produced scoresranging from 34-44. This proves the model demonstrably gets problems right when it gets the framing correct. Fine-tuning requires datasets, training time, and GPU resources. We had joined the competition fairly late with about 2 months left and the risk-to-reward for finetuning a model to have high scores was low. With limited weekly GPU time and number of days we had, building around OSS120B seemed to be the best option. We had an initial baseline method built with 6 solvers and summarizer-verifier-mastervision-resolver pipeline (not async but once per answer) score 45 in public leaderboard with oss120B-high reasoning model. This demonstrates that 45+ correct answers are achievable, the effort-to-return ratio strongly favors building the best possible amplifier (test-time compute scaling) for a model we trust, rather than training it to know marginally more.
\nTo prevent public leaderboard overfitting, we constructed benchmark_50_v2, a 50-problem local dataset curated from AIMO reference problems and external IMO benchmarks across algebra, combinatorics, geometry, and number theory. All architectural decisions and ablations in this paper are measured against this reliable ground truth, ensuring our improvements generalize to the hidden private set. This dataset is attached along with the writeup. The final submission we made scored 39/50 on this benchmark dataset with 9 minutes to spare. This was the highest score we had among all our inference pipelines built. This was the major reason, we were confident to submit the pipeline which scored only 37 in public leaderboard instead of the 45 which was second on the public leaderboard. This benchmark local run dataset was influential in helping us understand what different failure modes are, how to mitigate them, how to orchestrate time etc. Without these local runs, it was very hard/impossible to build a stable process with the runtime obfuscated in the leaderboard submissions.
Our base engine is GPT-OSS-120B served via vLLM with fp8 weight quantization and fp8_e4m3 KV caching, achieving 96% GPU memory utilization with a 65,536 token context. The model is invoked via the Harmony tool protocol. \nCrucially, to eliminate the multi-second latency of cold-starting Docker/Jupyter instances on every tool call, we pre-warm and maintain a pool of 20 persistent, state-isolated Jupyter kernels.
Standard environments provide math, numpy, and sympy. Through trace analysis, we observed the LLM natively attempting to write formal constraint formulations, only to fail via ImportError. By pre-installing z3-solver, ortools (CP-SAT), pulp, mip, and pyscipopt, we converted silent failures into productive optimization queries. Traces confirmed z3 dominated small-constraint satisfiability, while ortools excelled at large combinatorial enumeration.
We developed a Streamlit application (aimo-solver-trace-viewer) to parse and read the trace logs. This trace viewer was instrumental in understanding how OSS120B solves a prooblem and was instrumental in many of the designs we built. These logs included the raw turns based reasoning at different roles and channels and per turn entropy calculated. Every local run was executed in debug mode, saving the full multi-turn reasoning traces of all solvers and resolvers, including per-turn entropy calculations. This visualizer was instrumental in designing our architecture:
\n_select_answer function blindly extracted numbers from \\boxed{} formats, these hallucinations often bypassed validation. We implemented a real-time Regex Flaw Registry to catch these outputs, penalize the solver's score, and inject mid-flight corrective nudges.We have published the streamlit app public. You can play around with the traces you have generated (by running the submission code attached below) or by using the sample traces attached below for specific questions (Norwegian Problem, imo-bench-combinatorics-014)
\nThe Trace Viewer exposed a critical, systematic vulnerability in standard LLM orchestration: the Extremum Trap.
\nWhen a problem asks for the smallest, largest, minimum, or maximum value, it inherently possesses a multi-modal solution space. A sub-optimal bound (e.g., a configuration that yields 1055 when the true maximum is 2079) is mathematically \"valid\" in the sense that it satisfies all the constraints of the problem.
\nWhen observing our benchmark runs (e.g., imo-bench-algebra-055), we found that standard solvers would quickly find a sub-optimal bound, write a Python script to verify that it satisfies the constraints, receive a True output, and confidently halt. Because it is computationally much easier to find a sub-optimal configuration than the absolute mathematical limit, a majority of parallel solvers will converge on the weaker bound.
Why Verification Fails Here: Even our advanced asynchronous Verifier (Bug Report) agents struggle to falsify these sub-optimal answers. If a solver claims the maximum is 1055, the Verifier writes a script checking if 1055 is achievable. The script returns True, and the Verifier assigns a high Confidence Score to the wrong answer.
This empirical discovery mandated a systemic architectural intervention. We realized that for extremum problems, raw frequency and verification scores are misleading signals. This necessitated the creation of our dedicated Extremum Protocol (detailed in Section 4), which forces the Orchestrator to bypass majority breakouts entirely, intentionally seeking out minority answers that offer tighter, mathematically verified bounds.
\nThe AMO system abandons the standard paradigm of blind, unguided parallel sampling. Instead, it executes as a highly sophisticated, closed-loop pipeline of specialized LLM agents and deterministic Python guardrails. By treating reasoning as a continuous control problem, AMO maximizes test-time compute efficiency.
\nn a 5-hour, 50-problem constrained environment, a static time-per-problem budget creates severe latency spikes; a single hard problem consuming 800+ seconds early in the run will irreversibly starve subsequent problems. The AMO Orchestrator serves as the global brain of the system, replacing static budgets with a closed-loop Bayesian controller.
\n![\"Figure](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F645831%2F699d2691ab16b8282e06a75d1b89cadc%2Forchestrator.jpg?generation=1776744853119000&alt=media\")
Initialization and Global Time Metrics:\nThe Orchestrator initializes a difficulty \"Deck\" modeled on the host's pass@K empirical distribution (27 Easy, 13 Medium, 10 Hard). It maintains a rolling 10-problem adaptive time history for each tier to \"learn\" the specific compute friction of the current test set. At the start of each problem, global time metrics are calculated continuously:
\nTime_Left = 5H - Elapsed - Buffer (120s)Fair_Share = Time_Left / Problems_LeftExpected_Cost = Σ (Deck_Remaining[L] × Rolling_Avg_Time[L])Surplus_Ratio = Time_Left / Expected_CostBayesian Regime Shifting & Resource Binding:\nRelying solely on an LLM's perceived difficulty is dangerous. The Orchestrator cross-references the Scout's difficulty rating (see Sec 4.2) with the Surplus_Ratio to execute Bayesian regime shifts:
Surplus_Ratio > 1.5 or Fair_Share > 360s): Upgrades the problem tier (e.g., Medium $\\rightarrow$ Hard), unlocking the computationally expensive Agentic Beam Search.Surplus_Ratio < 0.7 or Fair_Share < 230s): Engages the Emergency Brake, forcibly downgrading Hard problems to Standard execution to protect the global clock.Based on the calculated Fair_Share, the Orchestrator dynamically binds all execution parameters to prevent clampdowns:
| Fair Share Target | \nMax Initial Solvers | \nEarly Stop Threshold | \nMax Verifier Turns | \nRescue Resolvers | \nPlanners Used? | \nExtremum Force | \n
|---|---|---|---|---|---|---|
| < 240s (Extreme) | \n5 | \n3 | \n5 | \n0 | \nNo | \n1 | \n
| < 280s (Tight) | \n6 | \n4 | \n5 | \n2 | \nNo | \n1 | \n
| < 320s (Moderate) | \n7 | \n4 | \n5 | \n2 | \nYes (N=10) | \n2 | \n
| < 352s (Healthy) | \n8 | \n5 | \n7 | \n2 | \nYes (N=12) | \n3 | \n
| ≥ 352s (Abundant) | \n8 (or Beam) | \n5 | \n8 | \n2 | \nYes (N=16) | \n4 | \n
The Absolute Budget is mechanically clamped: Budget = min(max(fair_share * 2.6 + resolver_bonus, MIN_CAP), MAX_CAP), coupled with a strict global failsafe min(Budget, Time_Left * 0.85) ensuring the system never throws a Kaggle timeout exception.
Post-Solve Empirical Reclassification:\nAfter a problem is solved, the Orchestrator evaluates the actual difficulty to update the Deck. It computes the consensus ratio (e.g., 4/5 solvers agreeing = 80%). If consensus $\\ge 80\\%$, it classifies the problem as Easy regardless of initial predictions. Conversely, if wall-clock time exceeded 800s, it forces a Hard classification. This feedback loop ensures the rolling time averages accurately reflect empirical reality.
\nBefore launching computationally expensive solvers, AMO must map the problem's mathematical topology to inject specific guardrails, identify expected problem level etc.
\n![\"Figure](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F645831%2F94d9e95c3a2384e3b83346e62aa10091%2FScout.jpg?generation=1776744789688538&alt=media\")
The Scout Dispatcher launches 4 parallel threads at a higher temperature (T=0.6) using unique random seeds. The LLM is explicitly instructed not to solve the problem, but to output a JSON schema mapping the domain, extraction operations, large number flags, and timeout risks.
\nConsensus Normalization (Anti-Poisoning):\nBecause LLMs hallucinate, a single flawed scout reading could poison the entire solver pool. We aggregate the 4 JSON blueprints using a Map-Reduce paradigm:
\nis_extremum = True AND the extraction operation is direct, the system activates a global Extremum Override. This disables early-verifier breakouts and forces tight-bound verification during Answer Selection. This makes sure we have checks enabled to make sure we are submitting the best extremum fit verified answer and not just the majority voted.timeout_risk, $+1.5$ for adversarial_structure, $+1.0$ for find_all operations. The average of these scores across valid scouts dictates the final initial routing tier (Easy/Medium/Hard).A fundamental limitation of open-weight LLMs in competitive mathematics is trajectory drift. While a model might possess the mathematical priors to solve a problem (demonstrated by a high pass@100 rate), parallel sampling without intervention wastes vast amounts of compute on \"doomed\" reasoning paths. If a solver commits to a mathematically flawed frame in Turn 1, the subsequent twenty minutes of generation and Python execution become effectively dead compute.
\nFurthermore, we operate under strict hardware limits: a single H100 GPU can practically sustain only 8 to 12 parallel solvers when pushing towards a 65,536 token context limit. For Hard (and some Medium) tier questions where the empirical success rate is pass@20 or even pass@100, simply running 8 parallel solvers at high temperature ($T=1.0$) is mathematically inefficient and relies heavily on luck. To reliably conquer these problems within our allocated 5-hour time budget, we must achieve a higher effective sample size, strictly enforce mathematical diversity, and drastically increase trajectory accuracy—all without exceeding the GPU's memory or the Orchestrator's time allocation.
\nTo solve this, we introduce Agentic Beam Search. Unlike traditional token-level beam search, which evaluates localized token probabilities, Agentic Beam Search operates at the reasoning-trajectory level. Its primary objective is to artificially compress the model's pass@20 capability into a pass@5 compute footprint. It achieves this by spawning a large initial pool of solvers (e.g., N=20), aggressively pruning mathematically unviable trajectories mid-flight, and reallocating the GPU's memory and compute only to diverse, high-potential paths.
\nBecause we operate in a zero-internet inference environment without a pre-trained Process Reward Model (PRM) like GenSelect, we construct our beam evaluation heuristic using a combination of self eval of solvers, empirical execution signals, and text-based clustering.
\n![\"Figure](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F645831%2F9f42bad3d2d2bad5807d06da4e24f3bc%2Fagentic_beam_search.jpg?generation=1776744933303486&alt=media\")
![\"Figure](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F645831%2F1bd97520ea24f9002c3fc1977d3db70d%2Fagentic_beam_self_eval.jpg?generation=1776745869037702&alt=media\")
At designated checkpoints, the solver's execution is paused. Without dropping the existing KV-cache, an adversarial BEAM_EVAL prompt is injected into the Role.ASSISTANT channel. The model is forced to objectively audit its own reasoning trace and output a JSON schema scoring itself out of 5 across five strict criteria:
The model is also instructed to extract a 100-word approach_summary of the current trajectory.
LLM-as-a-Judge mechanisms are notoriously prone to self-confirmation bias; a model will frequently award high scores to fluent but mathematically hallucinated reasoning. To ground the beam score in objective reality, we blend the subjective LLM evaluation with an empirical Python execution score (python_score = successful_calls / total_calls). We also subtract the trace_penalty accumulated by the Regex Flaw Registry (e.g., penalties for outputting infinity or using lazy phrasing). \nThe base node score is calculated as:\n$$Beam_Score = 0.8 \\times \\left(\\frac{self_eval_sum}{25}\\right) + 0.2 \\times python_score - trace_penalty$$
Selecting the top-$K$ nodes based purely on the $Beam_Score$ risks severe mode collapse. If three solvers adopt the same fundamentally flawed (but highly confident) combinatorial approach, keeping all three wastes precious beam width. To combat this, we explicitly require mathematical diversity.
\nDuring the BEAM_EVAL phase, the LLM extracts a 100-word approach_summary of its current trajectory. To measure semantic similarity accurately, we project these summaries into a dense vector space using a lightweight, pre-trained sentence embedding model (e.g., all-MiniLM-L6-v2). We then calculate the cosine distance between the embeddings of all surviving nodes.
Finally, we apply a greedy Maximum Marginal Relevance (MMR) selection algorithm. A candidate's final evaluation score heavily rewards its semantic distance from the approaches of already-selected nodes:\n$$Final_Eval_Score = 0.9 \\times Beam_Score + 0.1 \\times \\min_{selected}(Cosine_Distance)$$\nBy using dense embeddings rather than fragile keyword matching, this guarantees that the surviving beams explore fundamentally distinct mathematical frameworks, dramatically increasing the probability that the true solution path survives the prune.
\nWe deploy this Agentic Beam Search framework in two distinct areas of the AMO architecture, scaling its depth based on the Orchestrator's time budget.
\nOn problems where the Orchestrator detects a \"Moderate\" or \"Healthy\" time budget, framing failures are our primary threat. Before any full solvers are launched, we instantiate a single-turn version of the beam search to generate starting hypotheses.
\nBEAM_EVAL rubric is applied to these 10-12 solvers. Using the MMR text-cosine algorithm, we prune the pool down to 6 to 8 highly diverse, high-quality starting plans.When the Orchestrator calculates a Surplus_Ratio > 1.5 (Time Abundance) or on a Hard problem, it bypasses Standard Mode entirely and deploys the full, multi-turn Agentic Beam Search to find the most diverse best solver traces.
We deploy the Agentic Beam Search framework scaled to the Orchestrator's time budget.
\nWhen routed to Standard Mode (Tight/Moderate budgets), framing failures are our primary threat. Before full solvers are launched, we instantiate a 2-turn version of the beam search to generate starting hypotheses.
\nBEAM_EVAL rubric. This is a mini version of full agentic beam search![\"Figure](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F645831%2F785cdb70a8b23033e181df8fa5364050%2Fstandard_mode_solver.jpg?generation=1776745033197404&alt=media\")
Standard Mode executes 6-8 Tool-Integrated Reasoning (TIR) solvers concurrently. They operate under deterministic real-time guardrails (Section 4.5) and are designed to race toward a verified majority breakout to conserve time for later problems.
\nExecution is governed by deterministic, real-time guardrails:
\nitertools.permutations) instantly injects a Role.ASSISTANT nudge demanding z3 or a soft-timeout loop (if time.time() - t0 > 15: break).ASSISTANT channel, preserving internal monologue continuity.For Hard problems with abundant time, we deploy an Agentic Beam Search to explore full multi-turn reasoning trajectories.
\n![\"Figure](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F645831%2F808b5c456e32aa6b849f7cbd00adf11d%2FBeam_Mode_Solver.jpg?generation=1776745802921454&alt=media\")
ASSISTANT role. The model scores its own trajectory out of 25. Beam Score = 0.8 * self_eval + 0.2 * py_score - trace_penalty\nFinal Eval Score = 0.9 * Beam Score + 0.1 * distance_score\nThe top 16 solvers survive.A major contribution of AMO is the realization that LLMs must be actively \"steered\" during generation. However, standard conversational injection (using Role.USER) breaks the model's internal monologue and induces conversational mode collapse.
Instead, we utilize Context-Preserving Assistant Injections. When a flaw is detected, we format the correction as an internal thought (e.g., \"Wait, I am attempting naïve brute force. I must add a soft-timeout…\") and append it to the vLLM Conversation object as Role.ASSISTANT. Because the existing prompt token IDs are reused, vLLM's prefix caching handles this seamlessly without re-computing attention for the prefix, saving massive compute while forcing the model to adopt the correction as its own logic.
4.5.1 The Regex Flaw Registry\nWe actively parse the real-time text stream against a global registry of behavioral pathologies:
\nitertools, collections.deque, or naïve while loops triggers an instant injection demanding the use of z3 or the insertion of a strict soft-timeout block (if time.time() - t0 > 15: break), directly preventing hard Kaggle kernel timeouts.4.5.2 The Live Guide Checkpoint Algorithm\nTo catch complex mathematical drift, we trigger an LLM-based audit mid-flight.
\nURGENCY score (0.0 to 1.0) and CORRECTION_TEXT. If URGENCY >= 0.90, the correction is injected into the main solver's ASSISTANT channel, demanding a specific, verifiable computational action. AMO does not trust simple majority voting. As solvers find answers, their traces are immediately passed to an asynchronous pipeline. This architecture was inspired by two works: Long-horizon Reasoning Agent for Olympiad-Level Mathematical Problem Solving (Intern-S1-MO) [2] and Gemini 2.5 Pro Capable of Winning Gold at IMO 2025 [1].
\nThe core idea of agentic solution to solving IMO level math problems in [2] was to iteratively reason-verify-summarize it into lemmas, by adding the lemmas created in each steps into a lemma library memory, added in as input into next iteration along with the question. The architecture idea they have used is added below.
\n![\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F645831%2Fd5619890df7ef3f7b559037e3097d71e%2Fintern-s1-MO-agentic-framework.jpg?generation=1776817495224311&alt=media\")
While Intern-S1-MO iteratively reasons-verifies-summarizes by adding lemmas into a library memory (shown above), this multi-step sequential solving is not custom-fit for OSS-120B under severe time starvation. Furthermore, the Gemini 2.5 Pro approach of iteratively verifying until a confident answer is reached is feasible for 6 IMO questions in 5 hours, but not for 50 AIMO questions. Consequently, our Async Summarizer-Verifier-Master Vision pipeline adapts these inspirations into a highly parallelized, AIMO-specific architecture:
\n![\"Figure](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F645831%2Ff7d21b6059b91c5cca94acd3f551814a%2Fsummarizer-verifier-mastervision.jpg?generation=1776745065882961&alt=media\")
TRUST|VERIFY|REJECT.[VERIFIED] lemmas, and sets a Rescue Entry Point.To save time, AMO attempts an Early Breakout if an answer reaches the Majority Vote (5/8 solvers get same answer) OR Majority Threshold total (Total verifier score > 2.4 for an answer across all solvers completed so far). This early breakout saves time. Baseline notebook in AIMO3 had majority vote based breeakout alone, but with our verifier pipeline running async, we can have the verifier score available soon after an answer is generated for a solver too. So in cases where we didn't reach majority and a solver is still running, we wouldn't have to waste time waiting for it complete, if we have 3 verifiers all passing high for the same answer. However, because false majorities exist, this breakout is rigorously gated. We make sure we don't breakout in these 2 specific situations even if we hit majority vote breakouts:
\nThe Evidence Hierarchy Selection:\nIf no breakout occurs, final selection abandons frequency counting for a strict evidence hierarchy:
\nz3/optimizers, and +0.3 for standard empirical Python validation.We have this checked with regex search on the entire python tool outputs in the trace once an answer is foundExtremum Fallback: If an Extremum problem times out before Resolvers finish, AMO overrides raw vote counts entirely. It explicitly scans the pool to select the tightest verified bound (Verifier Score > 0.78) or the bound possessing the highest Code Supremacy, directly neutralizing the LLM Extremum Bias.
\nIn a strictly constrained 5-hour, 50-problem inference environment, mathematical capability is irrelevant if the system runs out of time. One of the most critical vulnerabilities of the baseline architecture was its naive approach to budget allocation, which directly resulted in late-stage problem starvation.
\nTo resolve this, AMO implements a closed-loop, predictive time management system that treats inference time as a dynamically traded commodity, smoothing the allocation curve across the entire problem set.
\nThe standard baseline approach utilizes a naive time_left / questions_left heuristic, often clamped to a fixed maximum (e.g., 900 seconds) for early questions. Because LLMs operating at T>0 can easily get trapped in long, unyielding reasoning paths, difficult problems presented early in the sequence will ruthlessly consume their maximum allowed budget.
As early questions bleed the global clock, the time_left plummets. By the time the system reaches question 20 or 30, the fair_share drops precipitously, triggering a \"cascading starvation\" effect.
![\"Allocated](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F645831%2Fe6236f43cb51d2064bdd1bd6d2638470%2Ftiming_comparison.jpg?generation=1776823605316092&alt=media\")
\nFigure 7: A direct comparison of the Allocated Time Budget across 50 questions during a local benchmark run. Top (Green): AMO Orchestrator. Bottom (Red): Baseline timing method.
The bottom chart (red) in Figure 7 perfectly illustrates this catastrophic failure mode. The baseline system statically allocates 900 seconds for the first few questions. Because actual computation exhausts this budget, the available time crashes. From question 15 onward, the allocation becomes violently erratic, frequently plummeting to 300 seconds. By the final decile (questions 40-50), the system is effectively crippled, lacking the compute time necessary to execute deep reasoning or parallel sampling, regardless of the actual mathematical complexity of those final questions.
\nTo prevent cascading starvation, AMO's Orchestrator abandons static maximums in favor of a forward-looking, deck-aware predictive model.
\nBefore any problem begins, the system relies on an expected difficulty \"Deck\" (initialized to 27 Easy, 13 Medium, 10 Hard based on pass@K priors as explained before) and maintains a 10-problem rolling time history for each tier. Rather than just looking at how much time is left, AMO predicts how much time the entire remaining test set will cost:\n$$Expected_Cost = \\sum_{L \\in {E,M,H}} \\left( Deck_Remaining[L] \\times Rolling_Avg_Time[L] \\right)$$
\nThis allows the Orchestrator to calculate a Surplus_Ratio ($Time_Left / Expected_Cost$). If the Surplus_Ratio < 1.0, the system knows it is running behind schedule relative to the expected difficulty of the remaining questions, even if absolute time appears plentiful.
The Surplus_Ratio enables continuous Bayesian regime shifting, allowing the system to scale its computational footprint up or down dynamically:
Instead of fixed ceilings, the actual Allocated Budget is a dynamic function of the Fair_Share, modulated by the problem tier and scaled via a predefined multiplier (empirically tuned to 2.6) plus a bonus for Resolver agents:\n$$Budget = \\min(\\max(Fair_Share \\times 2.6 + Resolver_Bonus, MIN_CAP), MAX_CAP)$$\nA global failsafe (Time_Left * 0.85) sits above this logic to guarantee the Kaggle kernel will never hit a hard timeout.
A predictive system is only as good as its priors. Because the initial difficulty Deck is an assumption, AMO continuously corrects its own internal models.
\nPost-solve, the Orchestrator executes an Empirical Reclassification. It ignores the Scout's initial difficulty estimate and looks purely at execution reality. If 7 out of 8 solvers independently arrived at the same answer in under 400 seconds, the problem is reclassified as \"Easy,\" regardless of its mathematical domain. If the system had to invoke the Master Vision Agent and consumed 850 seconds, it is reclassified as \"Hard.\"
\nThe system then decrements the corresponding tier in the Deck and updates the Rolling_Avg_Time array. This closed feedback loop allows AMO to \"learn\" the true pace of the unseen test set in real time.
The success of this architecture is visually confirmed in the top chart (green) of Figure 7.
\nUnlike the baseline's erratic crashes, the AMO Orchestrator produces a smooth, undulating allocation curve. By intelligently clamping early-stage over-expenditure and utilizing the difficulty deck to forecast future costs, the system maintains a highly stable budget between 600 and 960 seconds per question.
\nMost importantly, AMO successfully protects the end of the test set. When questions 40 through 50 arrive, the system still possesses ample budget (frequently >800 seconds) to deploy Master Vision, Resolvers, and full Beam Search iterations. By treating compute as a dynamically managed asset rather than a static allowance, AMO ensures that every problem receives the exact level of orchestration it requires, maximizing the probability of single-submission correctness.
\nA persistent challenge in the AIMO competition environment is the extreme volatility of the public leaderboard. Because the test set is small (50 questions) and LLM sampling at T>0 is inherently stochastic, a fragile architecture can occasionally \"get lucky\" and achieve an artificially high score, only to collapse on subsequent runs.
\nRealizing that optimizing purely for the public leaderboard was leading to overfitting and architecture fragility, we halted leaderboard-driven development approximately one month before the deadline. We constructed benchmark_50_v2, a localized dataset of 50 olympiad problems carefully curated from AIMO reference sets and the MATH corpus prize submissions. All subsequent architectural decisions were rigorously evaluated against this stable ground truth. (Note: Because this local benchmark was instituted late in the development cycle, local run data is sparse or unavailable for our earliest architectural iterations).
The table below documents our submission history (only including non-experimental LB submissions to avoid any bias) , tracking both local benchmark accuracy and public leaderboard scores across various architectural iterations.
\n| Method / Architecture | \nRun ID | \nLocal Score (Out of 50) | \nPublic Leaderboard Score | \n
|---|---|---|---|
| AMO-Final-Submission | \n1 | \n38 | \n37 | \n
| AMO-Final-Submission | \n2 | \n39 | \n38 | \n
| AMO-Final-Submission | \n3 | \n39 | \n41 | \n
| AMO-Final-Submission | \n4 | \n39 | \n37 | \n
| SwarmRAG-Async-SVM | \n1 | \n31 | \n39 | \n
| SwarmRAG-Async-SVM | \n2 | \n30 | \n35 | \n
| SwarmRAG-Async-SVM | \n3 | \n37 | \n41 | \n
| SwarmRAG-Async-SVM | \n4 | \n28 | \n38 | \n
| SwarmRAG-Async-SVM | \n5 | \n28 | \n33 | \n
| Standard - SVM | \n1 | \n32 | \n35 | \n
| Standard - SVM | \n2 | \n- | \n45 | \n
| Standard - SVM | \n3 | \n- | \n38 | \n
| Standard - SVM | \n4 | \n- | \n35 | \n
| baseline-with-multiinjection | \n1 | \n- | \n33 | \n
| baseline-with-multiinjection | \n2 | \n- | \n41 | \n
| baseline-with-multiinjection | \n3 | \n- | \n41 | \n
| baseline-with-multiinjection | \n4 | \n- | \n35 | \n
| blackboard-collaborative-bul | \n1 | \n- | \n33 | \n
| blackboard-collaborative-bul | \n2 | \n- | \n35 | \n
| blackboard-collaborative-bul | \n3 | \n- | \n27 | \n
| Scout-CRAG-Arbiter | \n1 | \n- | \n30 | \n
| Scout-CRAG-Arbiter | \n2 | \n- | \n38 | \n
| Scout-CRAG-Arbiter | \n3 | \n- | \n35 | \n
| Scout-CRAG-Arbiter | \n4 | \n- | \n36 | \n
| baseline-notebook | \n1 | \n- | \n39 | \n
At first glance, the public leaderboard data presents a paradox: our simpler Standard - SVM architecture achieved the highest peak score of the competition (45), yet our final, fully orchestrated AMO architecture peaked at 41.
However, a closer analysis of the variance reveals why relying on the Standard - SVM architecture is computationally dangerous. While it achieved a 45 on Run 2, it also scored a 35 on Runs 1 and 4, representing a massive 10-point variance swing. In a competition where the final private score is determined by running the notebook twice and penalizing inconsistency (awarding a half-point if only one run is correct), a highly variant architecture is an unacceptable liability. The 45 was a statistical outlier—a \"lucky\" run where the unstructured solvers happened to avoid timeout traps and frame-divergences. When tested on our stable local benchmark, this same architecture scored 32. Moreover, we were confident with our AMO-Final-Submission as it was more sound in logic as well as stable in local tests. We ran out of time to have few more test runs and experiments on the public leaderboard with this method. One of the lesson we learnt is to start early on AIMO4 to avoid this issue.
The transition to the final AMO architecture was a deliberate engineering decision to trade peak variance for a heavily raised, stable floor. By introducing the Agentic Beam Search, the Async Verifier Pipeline, and the Regex Flaw Registry, we actively constrained the model's stochasticity. The results on our local benchmark demonstrate unprecedented stability: 38, 39, 39, 39. The system became highly predictable, proving that our infrastructure successfully shielded the model from unrecoverable reasoning drifts.
This stability translated directly to the public leaderboard, where the AMO-Final submission consistently grouped in a tight band of 37 to 41, drastically suppressing the 10 point swings seen in earlier iterations. While architectural complexity—such as the overhead of Scouts and Evaluators—consumes inference time and slightly caps the maximum number of problems that can be fully explored, the resulting consistency guarantees the best odds for robust performance on the private evaluation. While it was tempting to go with leaderboard 45 submission, we decided to trust the data. The AMO architecture is superior to standard-SVM pipline and covers a lot of gaps. While the evaluation is done on 50 questions alone and there is a certain luck involved which might favor simpler systems (simplicity is perfection in small samples) we believe the AMO solution is more robust and is the best hedge to any problem and ordering.
\nThe data table also highlights the empirical failure of several collaborative agent patterns, providing the foundation for our ablations (detailed in Section 7):
\nSwarmRAG-Async-SVM and blackboard-collaborative-bul) exhibited extreme volatility. Local scores plummeted as low as 28, and leaderboard scores crashed to 27. Our traces confirmed that these setups actively facilitated \"echo chambers\" of correlated errors.Scout-CRAG-Arbiter runs hovered in the mid-30s. Injecting external problem corpus data during inference caused the model to overfit to the retrieved examples rather than focusing on the actual problem constraints.Ultimately, the empirical data proved that in a compute-constrained environment, isolation, adversarial verification, and structured search (AMO) heavily outperform both blind parallel sampling (baseline) and unverified agentic collaboration (SwarmRAG).
There were many different types of attempts we made on trying to build the best agentic structure for AIMO3. Some of them were novel and forward-looking but ultimately didn't make it into the final AMO to avoid noise, complexity, and lack of time.
\nMotivated by EDIS (arXiv:2602.01288), which suggests that entropy dynamics in token generation reveal reasoning quality, we computed mean, max-window, and tail entropy from raw vLLM top_logprobs.
all-MiniLM-L6-v2. We had 2 RAG systems (one for traces, one for code). The RAG calls were infused as Python modules in the sandbox tools.Following up on the questions from @philipvonderlind, here are some additional questions to help us assess your submission:
\nLook forward to reading your responses on these.
", "votes": 0, "replies": [] }, { "id": 3457751, "author_name": "Philip Vonderlind", "author_url": "", "post_date": "2026-05-14T09:19:50.313000", "content": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.\n
\nI like your use of plots to communicate the proposed architecture. I have a few questions about some of the stages:
\nBenchmark: You mention constructing a local benchmark to test your approach. Could you describe why you chose to create your own benchmark and explain the methodology behind it? There were many Math Corpus Prize submissions that have been widely used by the community, so I would be curious to understand the motivation. In addition, the scores on your local benchmark appear more stable than the public leaderboard results, which seem to have much higher variance. How do you explain this difference? For example, could it be related to always evaluating problems in the same order, differences in difficulty, or other benchmark-specific factors?
Time Management: You mention using the more extensive “Beam Search” generation strategy when there is abundant time remaining. Could this become problematic if many hard problems appear toward the end of the evaluation, after a significant amount of time has already been spent on expensive Beam Search runs earlier? Did you encounter or document such cases during development?
Temperature: Why does your SCOUT stage use a temperature of 0.6? It would be helpful to motivate this choice with ablations showing that a non-default temperature improves performance. For example, the submission by @natnitarach argues that t=1 is already the most diverse and highest-performing setting for gpt-oss-120b.
Scouts: In practice, is the SCOUT phase worth the GPU time it consumes? I would expect it to use a non-negligible portion of the overall budget, which might otherwise be spent on generating additional solution completions. Did you run any experiments confirming that the SCOUT phase improves final performance enough to justify its cost?
Citations and Reproducibility: Please cite the publicly shared notebook, as it appears to serve as the basis for at least parts of your final submission. In addition, please share the code for all ablations and experiments, and document the steps required to reproduce them.
Looking forward to your response :)
", "votes": 0, "replies": [] } ], "comment_count": 2, "messages": [ { "id": 3446759, "postDate": "2026-04-22T04:22:32.973Z", "content": "We present AMO(Agentic Math Orchestrator), a multi-layer orchestration architecture designed to maximize the inference-time capabilities of frozen open-weight Large Language Models (LLMs) on olympiad-level mathematics. Operating under a strict 5-hour, single-GPU competition environment without internet access, AMO wraps GPT-OSS-120B with parallel reasoning, an asynchronous verification pipeline, real-time trajectory auditing, evidence-weighted answer selection, and topology-aware routing. Rather than fine-tuning the model, we treat mathematical robustness as a systems-engineering challenge. We introduce novel mechanisms including a pre-generation Scout Pool for structural classification, an Adaptive Orchestrator for continuous time-budgeting, dynamic switching between Standard and Agentic Beam Search modes, and a Master Vision conflict-resolution pipeline. We achieved a top public score of 45 using our baseline architecture and 41 using the fully orchestrated AMO version. Through extensive ablations, trace analysis, and local benchmark testing, we document specific LLM failure modes in competitive mathematics—such as extremum bias, execution timeouts, and frame-sensitive divergence—and present our architectural mitigations.
\nThe AI Mathematical Olympiad (AIMO) Progress Prize 3 presents a unique test-time compute challenge: solve 50 original, olympiad-level mathematical problems spanning algebra, combinatorics, geometry, and number theory within a 5-hour wall-clock budget on a single H100 GPU. The environment is strictly constrained—no internet access, no external APIs, and no opportunity for between-submission fine-tuning. Furthermore, the competition scoring heavily penalizes inconsistency; each submission is evaluated twice on the private set, awarding a half-point only if one run is correct. Consequently, mathematical robustness under inference constraints matters as much as the raw capability of the underlying model.
\nThis paper addresses a central challenge: given a powerful, frozen open-weight model (GPT-OSS-120B), how do we build an agentic system that converts high theoretical pass@K capability into reliable, single-submission correctness? The baseline notebooks on OSS120B had shown the high ceiling of it. With the limited time and resources in the competition period, we decided to focus on improving the efficiency and stability of oss120B solvers by creating an agentic dynamic time aware system to maximize the ceiling and floor. Through careful analysis we found many failure points, which could be solved/improved by optimized agentic solutions rather than focusing on a fine tuning to hope for a blackbox solution.
\nThe standard baseline approach to this problem—simply running $N$ parallel Tool-Integrated Reasoning (TIR) solvers and applying majority voting—is highly inefficient. Blind parallel generation wastes massive amounts of compute on \"doomed\" reasoning paths. If a solver misinterprets the problem in Turn 1, the subsequent twenty minutes of generation and Python execution are effectively dead compute. Furthermore, blind majority voting is vulnerable to \"false majorities,\" where multiple solvers anchor onto the same flawed assumption and confidently confirm one another's incorrect answers.
\nTo revolutionize this baseline, we introduce the Agentic Math Orchestrator (AMO). AMO steps away from blind parallel generation, transforming the LLM into a highly structured, asynchronous multi-agent research team. The entire lifecycle of a problem is meticulously managed from end to end:
\nA cornerstone of our architecture is the Agentic Beam Search. Designed to artificially compress the model’s $pass@20$ capability into a smaller compute footprint, this mode pauses parallel solvers mid-flight to conduct adversarial self-evaluations. By pruning mathematically unviable trajectories and applying Maximum Marginal Relevance (MMR) text-clustering to the surviving paths, we force mathematical diversity. This ensures the model searches fundamentally distinct frameworks rather than racing down identical dead ends, drastically increasing the odds of finding the correct solution.
\nTo reliably extract the correct answer from this agentic swarm, we replaced simple majority voting with a Multi-Tiered Evidence Hierarchy. Answers are selected based on rigorous criteria: Resolver agent voting weight, Verifier confidence scores (where adversarial Python scripts attempt to falsify the claim), and \"Code Supremacy\" (awarding bonuses to answers explicitly proven by constraint solvers like z3,ortools, pulp etc. and python tool output verified answer). While we initially explored using token-level entropy dynamics as a primary selection parameter, extensive testing revealed that without a trained calibration model, entropy added noise (especially in python tool based agents)—fluent hallucinations were often indistinguishable from genuine computation. Similarly, early attempts at inter-agent collaboration (such as SwarmRAG or sharing intermediate reasoning bulletins) didn't get into the final submission, after we observed they created echo chambers of correlated errors.
Finally, treating robust inference as a systems-engineering challenge required deep forensic visibility. To build this system, we logged every solver trace including reasoning step, tool call, and token metric as well as the scout/summarizer/verifier/master vision reports from extensive local benchmark runs. We developed a custom Streamlit-based Trace Viewer to replay these multi-turn conversations. This visualizer was instrumental in identifying specific model pathologies—such as infinite loop traps, premature guessing, and format hacking. Insights derived directly from this Trace Viewer birthed our Regex Flaw Registry, a real-time monitoring system that penalizes flawed trajectories and injects mid-flight corrective nudges to steer solvers back on track.
\nIn the following sections, we detail the complete AMO architecture, documenting our dynamic resource scaling, multi-mode solver logic, and the empirical ablations that define our approach to inference-time mathematical orchestration.
\nRecent evaluations published by the competition hosts compared the results of open-weight models (OSS-120B vs. Nemotron-30B) across both public and private sets. While these evaluations were likely conducted without strict inference-time constraints and lacked access to Python execution tools, they provided a critical baseline for the raw reasoning capability of OSS-120B.
\n
The key finding: at pass@100, only 1–2 problems across both sets remained completely unsolvable. At pass@20, a strong open-weight model (class-equivalent to OSS-120B) averaged 43–48 on the private set, with the public and private difficulty distributions showing overlapping 95% confidence intervals. This analysis dictates our fundamental strategy:
\nThe Host's analysis was done on OSS120B models without python tools. So the ceiling of OSS120B with python tool should be higher.
\nThrough forensic trace analysis of our local benchmarks (detailed in Section 3.1), we categorized LLM failures not by mathematical subject (e.g., Geometry vs. Algebra), but by behavioral pathology:
\nimo-bench-algebra-092), where the model clearly overfitted to training data and converged on a known, but incorrect, schema.imo-bench-algebra-055 (a maximum problem), we observed 4 solvers converging on 1055, while only 1 solver correctly found 2079. Under standard majority voting, the system would confidently select the wrong, sub-optimal bound.The baseline notebook provided by the competition using OSS-120B with 8 parallel solvers and simple majority voting already produced scoresranging from 34-44. This proves the model demonstrably gets problems right when it gets the framing correct. Fine-tuning requires datasets, training time, and GPU resources. We had joined the competition fairly late with about 2 months left and the risk-to-reward for finetuning a model to have high scores was low. With limited weekly GPU time and number of days we had, building around OSS120B seemed to be the best option. We had an initial baseline method built with 6 solvers and summarizer-verifier-mastervision-resolver pipeline (not async but once per answer) score 45 in public leaderboard with oss120B-high reasoning model. This demonstrates that 45+ correct answers are achievable, the effort-to-return ratio strongly favors building the best possible amplifier (test-time compute scaling) for a model we trust, rather than training it to know marginally more.
\nTo prevent public leaderboard overfitting, we constructed benchmark_50_v2, a 50-problem local dataset curated from AIMO reference problems and external IMO benchmarks across algebra, combinatorics, geometry, and number theory. All architectural decisions and ablations in this paper are measured against this reliable ground truth, ensuring our improvements generalize to the hidden private set. This dataset is attached along with the writeup. The final submission we made scored 39/50 on this benchmark dataset with 9 minutes to spare. This was the highest score we had among all our inference pipelines built. This was the major reason, we were confident to submit the pipeline which scored only 37 in public leaderboard instead of the 45 which was second on the public leaderboard. This benchmark local run dataset was influential in helping us understand what different failure modes are, how to mitigate them, how to orchestrate time etc. Without these local runs, it was very hard/impossible to build a stable process with the runtime obfuscated in the leaderboard submissions.
Our base engine is GPT-OSS-120B served via vLLM with fp8 weight quantization and fp8_e4m3 KV caching, achieving 96% GPU memory utilization with a 65,536 token context. The model is invoked via the Harmony tool protocol. \nCrucially, to eliminate the multi-second latency of cold-starting Docker/Jupyter instances on every tool call, we pre-warm and maintain a pool of 20 persistent, state-isolated Jupyter kernels.
Standard environments provide math, numpy, and sympy. Through trace analysis, we observed the LLM natively attempting to write formal constraint formulations, only to fail via ImportError. By pre-installing z3-solver, ortools (CP-SAT), pulp, mip, and pyscipopt, we converted silent failures into productive optimization queries. Traces confirmed z3 dominated small-constraint satisfiability, while ortools excelled at large combinatorial enumeration.
We developed a Streamlit application (aimo-solver-trace-viewer) to parse and read the trace logs. This trace viewer was instrumental in understanding how OSS120B solves a prooblem and was instrumental in many of the designs we built. These logs included the raw turns based reasoning at different roles and channels and per turn entropy calculated. Every local run was executed in debug mode, saving the full multi-turn reasoning traces of all solvers and resolvers, including per-turn entropy calculations. This visualizer was instrumental in designing our architecture:
\n_select_answer function blindly extracted numbers from \\boxed{} formats, these hallucinations often bypassed validation. We implemented a real-time Regex Flaw Registry to catch these outputs, penalize the solver's score, and inject mid-flight corrective nudges.We have published the streamlit app public. You can play around with the traces you have generated (by running the submission code attached below) or by using the sample traces attached below for specific questions (Norwegian Problem, imo-bench-combinatorics-014)
\nThe Trace Viewer exposed a critical, systematic vulnerability in standard LLM orchestration: the Extremum Trap.
\nWhen a problem asks for the smallest, largest, minimum, or maximum value, it inherently possesses a multi-modal solution space. A sub-optimal bound (e.g., a configuration that yields 1055 when the true maximum is 2079) is mathematically \"valid\" in the sense that it satisfies all the constraints of the problem.
\nWhen observing our benchmark runs (e.g., imo-bench-algebra-055), we found that standard solvers would quickly find a sub-optimal bound, write a Python script to verify that it satisfies the constraints, receive a True output, and confidently halt. Because it is computationally much easier to find a sub-optimal configuration than the absolute mathematical limit, a majority of parallel solvers will converge on the weaker bound.
Why Verification Fails Here: Even our advanced asynchronous Verifier (Bug Report) agents struggle to falsify these sub-optimal answers. If a solver claims the maximum is 1055, the Verifier writes a script checking if 1055 is achievable. The script returns True, and the Verifier assigns a high Confidence Score to the wrong answer.
This empirical discovery mandated a systemic architectural intervention. We realized that for extremum problems, raw frequency and verification scores are misleading signals. This necessitated the creation of our dedicated Extremum Protocol (detailed in Section 4), which forces the Orchestrator to bypass majority breakouts entirely, intentionally seeking out minority answers that offer tighter, mathematically verified bounds.
\nThe AMO system abandons the standard paradigm of blind, unguided parallel sampling. Instead, it executes as a highly sophisticated, closed-loop pipeline of specialized LLM agents and deterministic Python guardrails. By treating reasoning as a continuous control problem, AMO maximizes test-time compute efficiency.
\nn a 5-hour, 50-problem constrained environment, a static time-per-problem budget creates severe latency spikes; a single hard problem consuming 800+ seconds early in the run will irreversibly starve subsequent problems. The AMO Orchestrator serves as the global brain of the system, replacing static budgets with a closed-loop Bayesian controller.
\n
Initialization and Global Time Metrics:\nThe Orchestrator initializes a difficulty \"Deck\" modeled on the host's pass@K empirical distribution (27 Easy, 13 Medium, 10 Hard). It maintains a rolling 10-problem adaptive time history for each tier to \"learn\" the specific compute friction of the current test set. At the start of each problem, global time metrics are calculated continuously:
\nTime_Left = 5H - Elapsed - Buffer (120s)Fair_Share = Time_Left / Problems_LeftExpected_Cost = Σ (Deck_Remaining[L] × Rolling_Avg_Time[L])Surplus_Ratio = Time_Left / Expected_CostBayesian Regime Shifting & Resource Binding:\nRelying solely on an LLM's perceived difficulty is dangerous. The Orchestrator cross-references the Scout's difficulty rating (see Sec 4.2) with the Surplus_Ratio to execute Bayesian regime shifts:
Surplus_Ratio > 1.5 or Fair_Share > 360s): Upgrades the problem tier (e.g., Medium $\\rightarrow$ Hard), unlocking the computationally expensive Agentic Beam Search.Surplus_Ratio < 0.7 or Fair_Share < 230s): Engages the Emergency Brake, forcibly downgrading Hard problems to Standard execution to protect the global clock.Based on the calculated Fair_Share, the Orchestrator dynamically binds all execution parameters to prevent clampdowns:
| Fair Share Target | \nMax Initial Solvers | \nEarly Stop Threshold | \nMax Verifier Turns | \nRescue Resolvers | \nPlanners Used? | \nExtremum Force | \n
|---|---|---|---|---|---|---|
| < 240s (Extreme) | \n5 | \n3 | \n5 | \n0 | \nNo | \n1 | \n
| < 280s (Tight) | \n6 | \n4 | \n5 | \n2 | \nNo | \n1 | \n
| < 320s (Moderate) | \n7 | \n4 | \n5 | \n2 | \nYes (N=10) | \n2 | \n
| < 352s (Healthy) | \n8 | \n5 | \n7 | \n2 | \nYes (N=12) | \n3 | \n
| ≥ 352s (Abundant) | \n8 (or Beam) | \n5 | \n8 | \n2 | \nYes (N=16) | \n4 | \n
The Absolute Budget is mechanically clamped: Budget = min(max(fair_share * 2.6 + resolver_bonus, MIN_CAP), MAX_CAP), coupled with a strict global failsafe min(Budget, Time_Left * 0.85) ensuring the system never throws a Kaggle timeout exception.
Post-Solve Empirical Reclassification:\nAfter a problem is solved, the Orchestrator evaluates the actual difficulty to update the Deck. It computes the consensus ratio (e.g., 4/5 solvers agreeing = 80%). If consensus $\\ge 80\\%$, it classifies the problem as Easy regardless of initial predictions. Conversely, if wall-clock time exceeded 800s, it forces a Hard classification. This feedback loop ensures the rolling time averages accurately reflect empirical reality.
\nBefore launching computationally expensive solvers, AMO must map the problem's mathematical topology to inject specific guardrails, identify expected problem level etc.
\n
The Scout Dispatcher launches 4 parallel threads at a higher temperature (T=0.6) using unique random seeds. The LLM is explicitly instructed not to solve the problem, but to output a JSON schema mapping the domain, extraction operations, large number flags, and timeout risks.
\nConsensus Normalization (Anti-Poisoning):\nBecause LLMs hallucinate, a single flawed scout reading could poison the entire solver pool. We aggregate the 4 JSON blueprints using a Map-Reduce paradigm:
\nis_extremum = True AND the extraction operation is direct, the system activates a global Extremum Override. This disables early-verifier breakouts and forces tight-bound verification during Answer Selection. This makes sure we have checks enabled to make sure we are submitting the best extremum fit verified answer and not just the majority voted.timeout_risk, $+1.5$ for adversarial_structure, $+1.0$ for find_all operations. The average of these scores across valid scouts dictates the final initial routing tier (Easy/Medium/Hard).A fundamental limitation of open-weight LLMs in competitive mathematics is trajectory drift. While a model might possess the mathematical priors to solve a problem (demonstrated by a high pass@100 rate), parallel sampling without intervention wastes vast amounts of compute on \"doomed\" reasoning paths. If a solver commits to a mathematically flawed frame in Turn 1, the subsequent twenty minutes of generation and Python execution become effectively dead compute.
\nFurthermore, we operate under strict hardware limits: a single H100 GPU can practically sustain only 8 to 12 parallel solvers when pushing towards a 65,536 token context limit. For Hard (and some Medium) tier questions where the empirical success rate is pass@20 or even pass@100, simply running 8 parallel solvers at high temperature ($T=1.0$) is mathematically inefficient and relies heavily on luck. To reliably conquer these problems within our allocated 5-hour time budget, we must achieve a higher effective sample size, strictly enforce mathematical diversity, and drastically increase trajectory accuracy—all without exceeding the GPU's memory or the Orchestrator's time allocation.
\nTo solve this, we introduce Agentic Beam Search. Unlike traditional token-level beam search, which evaluates localized token probabilities, Agentic Beam Search operates at the reasoning-trajectory level. Its primary objective is to artificially compress the model's pass@20 capability into a pass@5 compute footprint. It achieves this by spawning a large initial pool of solvers (e.g., N=20), aggressively pruning mathematically unviable trajectories mid-flight, and reallocating the GPU's memory and compute only to diverse, high-potential paths.
\nBecause we operate in a zero-internet inference environment without a pre-trained Process Reward Model (PRM) like GenSelect, we construct our beam evaluation heuristic using a combination of self eval of solvers, empirical execution signals, and text-based clustering.
\n
At designated checkpoints, the solver's execution is paused. Without dropping the existing KV-cache, an adversarial BEAM_EVAL prompt is injected into the Role.ASSISTANT channel. The model is forced to objectively audit its own reasoning trace and output a JSON schema scoring itself out of 5 across five strict criteria:
The model is also instructed to extract a 100-word approach_summary of the current trajectory.
LLM-as-a-Judge mechanisms are notoriously prone to self-confirmation bias; a model will frequently award high scores to fluent but mathematically hallucinated reasoning. To ground the beam score in objective reality, we blend the subjective LLM evaluation with an empirical Python execution score (python_score = successful_calls / total_calls). We also subtract the trace_penalty accumulated by the Regex Flaw Registry (e.g., penalties for outputting infinity or using lazy phrasing). \nThe base node score is calculated as:\n$$Beam_Score = 0.8 \\times \\left(\\frac{self_eval_sum}{25}\\right) + 0.2 \\times python_score - trace_penalty$$
Selecting the top-$K$ nodes based purely on the $Beam_Score$ risks severe mode collapse. If three solvers adopt the same fundamentally flawed (but highly confident) combinatorial approach, keeping all three wastes precious beam width. To combat this, we explicitly require mathematical diversity.
\nDuring the BEAM_EVAL phase, the LLM extracts a 100-word approach_summary of its current trajectory. To measure semantic similarity accurately, we project these summaries into a dense vector space using a lightweight, pre-trained sentence embedding model (e.g., all-MiniLM-L6-v2). We then calculate the cosine distance between the embeddings of all surviving nodes.
Finally, we apply a greedy Maximum Marginal Relevance (MMR) selection algorithm. A candidate's final evaluation score heavily rewards its semantic distance from the approaches of already-selected nodes:\n$$Final_Eval_Score = 0.9 \\times Beam_Score + 0.1 \\times \\min_{selected}(Cosine_Distance)$$\nBy using dense embeddings rather than fragile keyword matching, this guarantees that the surviving beams explore fundamentally distinct mathematical frameworks, dramatically increasing the probability that the true solution path survives the prune.
\nWe deploy this Agentic Beam Search framework in two distinct areas of the AMO architecture, scaling its depth based on the Orchestrator's time budget.
\nOn problems where the Orchestrator detects a \"Moderate\" or \"Healthy\" time budget, framing failures are our primary threat. Before any full solvers are launched, we instantiate a single-turn version of the beam search to generate starting hypotheses.
\nBEAM_EVAL rubric is applied to these 10-12 solvers. Using the MMR text-cosine algorithm, we prune the pool down to 6 to 8 highly diverse, high-quality starting plans.When the Orchestrator calculates a Surplus_Ratio > 1.5 (Time Abundance) or on a Hard problem, it bypasses Standard Mode entirely and deploys the full, multi-turn Agentic Beam Search to find the most diverse best solver traces.
We deploy the Agentic Beam Search framework scaled to the Orchestrator's time budget.
\nWhen routed to Standard Mode (Tight/Moderate budgets), framing failures are our primary threat. Before full solvers are launched, we instantiate a 2-turn version of the beam search to generate starting hypotheses.
\nBEAM_EVAL rubric. This is a mini version of full agentic beam search
Standard Mode executes 6-8 Tool-Integrated Reasoning (TIR) solvers concurrently. They operate under deterministic real-time guardrails (Section 4.5) and are designed to race toward a verified majority breakout to conserve time for later problems.
\nExecution is governed by deterministic, real-time guardrails:
\nitertools.permutations) instantly injects a Role.ASSISTANT nudge demanding z3 or a soft-timeout loop (if time.time() - t0 > 15: break).ASSISTANT channel, preserving internal monologue continuity.For Hard problems with abundant time, we deploy an Agentic Beam Search to explore full multi-turn reasoning trajectories.
\n
ASSISTANT role. The model scores its own trajectory out of 25. Beam Score = 0.8 * self_eval + 0.2 * py_score - trace_penalty\nFinal Eval Score = 0.9 * Beam Score + 0.1 * distance_score\nThe top 16 solvers survive.A major contribution of AMO is the realization that LLMs must be actively \"steered\" during generation. However, standard conversational injection (using Role.USER) breaks the model's internal monologue and induces conversational mode collapse.
Instead, we utilize Context-Preserving Assistant Injections. When a flaw is detected, we format the correction as an internal thought (e.g., \"Wait, I am attempting naïve brute force. I must add a soft-timeout…\") and append it to the vLLM Conversation object as Role.ASSISTANT. Because the existing prompt token IDs are reused, vLLM's prefix caching handles this seamlessly without re-computing attention for the prefix, saving massive compute while forcing the model to adopt the correction as its own logic.
4.5.1 The Regex Flaw Registry\nWe actively parse the real-time text stream against a global registry of behavioral pathologies:
\nitertools, collections.deque, or naïve while loops triggers an instant injection demanding the use of z3 or the insertion of a strict soft-timeout block (if time.time() - t0 > 15: break), directly preventing hard Kaggle kernel timeouts.4.5.2 The Live Guide Checkpoint Algorithm\nTo catch complex mathematical drift, we trigger an LLM-based audit mid-flight.
\nURGENCY score (0.0 to 1.0) and CORRECTION_TEXT. If URGENCY >= 0.90, the correction is injected into the main solver's ASSISTANT channel, demanding a specific, verifiable computational action. AMO does not trust simple majority voting. As solvers find answers, their traces are immediately passed to an asynchronous pipeline. This architecture was inspired by two works: Long-horizon Reasoning Agent for Olympiad-Level Mathematical Problem Solving (Intern-S1-MO) [2] and Gemini 2.5 Pro Capable of Winning Gold at IMO 2025 [1].
\nThe core idea of agentic solution to solving IMO level math problems in [2] was to iteratively reason-verify-summarize it into lemmas, by adding the lemmas created in each steps into a lemma library memory, added in as input into next iteration along with the question. The architecture idea they have used is added below.
\n
While Intern-S1-MO iteratively reasons-verifies-summarizes by adding lemmas into a library memory (shown above), this multi-step sequential solving is not custom-fit for OSS-120B under severe time starvation. Furthermore, the Gemini 2.5 Pro approach of iteratively verifying until a confident answer is reached is feasible for 6 IMO questions in 5 hours, but not for 50 AIMO questions. Consequently, our Async Summarizer-Verifier-Master Vision pipeline adapts these inspirations into a highly parallelized, AIMO-specific architecture:
\n
TRUST|VERIFY|REJECT.[VERIFIED] lemmas, and sets a Rescue Entry Point.To save time, AMO attempts an Early Breakout if an answer reaches the Majority Vote (5/8 solvers get same answer) OR Majority Threshold total (Total verifier score > 2.4 for an answer across all solvers completed so far). This early breakout saves time. Baseline notebook in AIMO3 had majority vote based breeakout alone, but with our verifier pipeline running async, we can have the verifier score available soon after an answer is generated for a solver too. So in cases where we didn't reach majority and a solver is still running, we wouldn't have to waste time waiting for it complete, if we have 3 verifiers all passing high for the same answer. However, because false majorities exist, this breakout is rigorously gated. We make sure we don't breakout in these 2 specific situations even if we hit majority vote breakouts:
\nThe Evidence Hierarchy Selection:\nIf no breakout occurs, final selection abandons frequency counting for a strict evidence hierarchy:
\nz3/optimizers, and +0.3 for standard empirical Python validation.We have this checked with regex search on the entire python tool outputs in the trace once an answer is foundExtremum Fallback: If an Extremum problem times out before Resolvers finish, AMO overrides raw vote counts entirely. It explicitly scans the pool to select the tightest verified bound (Verifier Score > 0.78) or the bound possessing the highest Code Supremacy, directly neutralizing the LLM Extremum Bias.
\nIn a strictly constrained 5-hour, 50-problem inference environment, mathematical capability is irrelevant if the system runs out of time. One of the most critical vulnerabilities of the baseline architecture was its naive approach to budget allocation, which directly resulted in late-stage problem starvation.
\nTo resolve this, AMO implements a closed-loop, predictive time management system that treats inference time as a dynamically traded commodity, smoothing the allocation curve across the entire problem set.
\nThe standard baseline approach utilizes a naive time_left / questions_left heuristic, often clamped to a fixed maximum (e.g., 900 seconds) for early questions. Because LLMs operating at T>0 can easily get trapped in long, unyielding reasoning paths, difficult problems presented early in the sequence will ruthlessly consume their maximum allowed budget.
As early questions bleed the global clock, the time_left plummets. By the time the system reaches question 20 or 30, the fair_share drops precipitously, triggering a \"cascading starvation\" effect.
\nFigure 7: A direct comparison of the Allocated Time Budget across 50 questions during a local benchmark run. Top (Green): AMO Orchestrator. Bottom (Red): Baseline timing method.
The bottom chart (red) in Figure 7 perfectly illustrates this catastrophic failure mode. The baseline system statically allocates 900 seconds for the first few questions. Because actual computation exhausts this budget, the available time crashes. From question 15 onward, the allocation becomes violently erratic, frequently plummeting to 300 seconds. By the final decile (questions 40-50), the system is effectively crippled, lacking the compute time necessary to execute deep reasoning or parallel sampling, regardless of the actual mathematical complexity of those final questions.
\nTo prevent cascading starvation, AMO's Orchestrator abandons static maximums in favor of a forward-looking, deck-aware predictive model.
\nBefore any problem begins, the system relies on an expected difficulty \"Deck\" (initialized to 27 Easy, 13 Medium, 10 Hard based on pass@K priors as explained before) and maintains a 10-problem rolling time history for each tier. Rather than just looking at how much time is left, AMO predicts how much time the entire remaining test set will cost:\n$$Expected_Cost = \\sum_{L \\in {E,M,H}} \\left( Deck_Remaining[L] \\times Rolling_Avg_Time[L] \\right)$$
\nThis allows the Orchestrator to calculate a Surplus_Ratio ($Time_Left / Expected_Cost$). If the Surplus_Ratio < 1.0, the system knows it is running behind schedule relative to the expected difficulty of the remaining questions, even if absolute time appears plentiful.
The Surplus_Ratio enables continuous Bayesian regime shifting, allowing the system to scale its computational footprint up or down dynamically:
Instead of fixed ceilings, the actual Allocated Budget is a dynamic function of the Fair_Share, modulated by the problem tier and scaled via a predefined multiplier (empirically tuned to 2.6) plus a bonus for Resolver agents:\n$$Budget = \\min(\\max(Fair_Share \\times 2.6 + Resolver_Bonus, MIN_CAP), MAX_CAP)$$\nA global failsafe (Time_Left * 0.85) sits above this logic to guarantee the Kaggle kernel will never hit a hard timeout.
A predictive system is only as good as its priors. Because the initial difficulty Deck is an assumption, AMO continuously corrects its own internal models.
\nPost-solve, the Orchestrator executes an Empirical Reclassification. It ignores the Scout's initial difficulty estimate and looks purely at execution reality. If 7 out of 8 solvers independently arrived at the same answer in under 400 seconds, the problem is reclassified as \"Easy,\" regardless of its mathematical domain. If the system had to invoke the Master Vision Agent and consumed 850 seconds, it is reclassified as \"Hard.\"
\nThe system then decrements the corresponding tier in the Deck and updates the Rolling_Avg_Time array. This closed feedback loop allows AMO to \"learn\" the true pace of the unseen test set in real time.
The success of this architecture is visually confirmed in the top chart (green) of Figure 7.
\nUnlike the baseline's erratic crashes, the AMO Orchestrator produces a smooth, undulating allocation curve. By intelligently clamping early-stage over-expenditure and utilizing the difficulty deck to forecast future costs, the system maintains a highly stable budget between 600 and 960 seconds per question.
\nMost importantly, AMO successfully protects the end of the test set. When questions 40 through 50 arrive, the system still possesses ample budget (frequently >800 seconds) to deploy Master Vision, Resolvers, and full Beam Search iterations. By treating compute as a dynamically managed asset rather than a static allowance, AMO ensures that every problem receives the exact level of orchestration it requires, maximizing the probability of single-submission correctness.
\nA persistent challenge in the AIMO competition environment is the extreme volatility of the public leaderboard. Because the test set is small (50 questions) and LLM sampling at T>0 is inherently stochastic, a fragile architecture can occasionally \"get lucky\" and achieve an artificially high score, only to collapse on subsequent runs.
\nRealizing that optimizing purely for the public leaderboard was leading to overfitting and architecture fragility, we halted leaderboard-driven development approximately one month before the deadline. We constructed benchmark_50_v2, a localized dataset of 50 olympiad problems carefully curated from AIMO reference sets and the MATH corpus prize submissions. All subsequent architectural decisions were rigorously evaluated against this stable ground truth. (Note: Because this local benchmark was instituted late in the development cycle, local run data is sparse or unavailable for our earliest architectural iterations).
The table below documents our submission history (only including non-experimental LB submissions to avoid any bias) , tracking both local benchmark accuracy and public leaderboard scores across various architectural iterations.
\n| Method / Architecture | \nRun ID | \nLocal Score (Out of 50) | \nPublic Leaderboard Score | \n
|---|---|---|---|
| AMO-Final-Submission | \n1 | \n38 | \n37 | \n
| AMO-Final-Submission | \n2 | \n39 | \n38 | \n
| AMO-Final-Submission | \n3 | \n39 | \n41 | \n
| AMO-Final-Submission | \n4 | \n39 | \n37 | \n
| SwarmRAG-Async-SVM | \n1 | \n31 | \n39 | \n
| SwarmRAG-Async-SVM | \n2 | \n30 | \n35 | \n
| SwarmRAG-Async-SVM | \n3 | \n37 | \n41 | \n
| SwarmRAG-Async-SVM | \n4 | \n28 | \n38 | \n
| SwarmRAG-Async-SVM | \n5 | \n28 | \n33 | \n
| Standard - SVM | \n1 | \n32 | \n35 | \n
| Standard - SVM | \n2 | \n- | \n45 | \n
| Standard - SVM | \n3 | \n- | \n38 | \n
| Standard - SVM | \n4 | \n- | \n35 | \n
| baseline-with-multiinjection | \n1 | \n- | \n33 | \n
| baseline-with-multiinjection | \n2 | \n- | \n41 | \n
| baseline-with-multiinjection | \n3 | \n- | \n41 | \n
| baseline-with-multiinjection | \n4 | \n- | \n35 | \n
| blackboard-collaborative-bul | \n1 | \n- | \n33 | \n
| blackboard-collaborative-bul | \n2 | \n- | \n35 | \n
| blackboard-collaborative-bul | \n3 | \n- | \n27 | \n
| Scout-CRAG-Arbiter | \n1 | \n- | \n30 | \n
| Scout-CRAG-Arbiter | \n2 | \n- | \n38 | \n
| Scout-CRAG-Arbiter | \n3 | \n- | \n35 | \n
| Scout-CRAG-Arbiter | \n4 | \n- | \n36 | \n
| baseline-notebook | \n1 | \n- | \n39 | \n
At first glance, the public leaderboard data presents a paradox: our simpler Standard - SVM architecture achieved the highest peak score of the competition (45), yet our final, fully orchestrated AMO architecture peaked at 41.
However, a closer analysis of the variance reveals why relying on the Standard - SVM architecture is computationally dangerous. While it achieved a 45 on Run 2, it also scored a 35 on Runs 1 and 4, representing a massive 10-point variance swing. In a competition where the final private score is determined by running the notebook twice and penalizing inconsistency (awarding a half-point if only one run is correct), a highly variant architecture is an unacceptable liability. The 45 was a statistical outlier—a \"lucky\" run where the unstructured solvers happened to avoid timeout traps and frame-divergences. When tested on our stable local benchmark, this same architecture scored 32. Moreover, we were confident with our AMO-Final-Submission as it was more sound in logic as well as stable in local tests. We ran out of time to have few more test runs and experiments on the public leaderboard with this method. One of the lesson we learnt is to start early on AIMO4 to avoid this issue.
The transition to the final AMO architecture was a deliberate engineering decision to trade peak variance for a heavily raised, stable floor. By introducing the Agentic Beam Search, the Async Verifier Pipeline, and the Regex Flaw Registry, we actively constrained the model's stochasticity. The results on our local benchmark demonstrate unprecedented stability: 38, 39, 39, 39. The system became highly predictable, proving that our infrastructure successfully shielded the model from unrecoverable reasoning drifts.
This stability translated directly to the public leaderboard, where the AMO-Final submission consistently grouped in a tight band of 37 to 41, drastically suppressing the 10 point swings seen in earlier iterations. While architectural complexity—such as the overhead of Scouts and Evaluators—consumes inference time and slightly caps the maximum number of problems that can be fully explored, the resulting consistency guarantees the best odds for robust performance on the private evaluation. While it was tempting to go with leaderboard 45 submission, we decided to trust the data. The AMO architecture is superior to standard-SVM pipline and covers a lot of gaps. While the evaluation is done on 50 questions alone and there is a certain luck involved which might favor simpler systems (simplicity is perfection in small samples) we believe the AMO solution is more robust and is the best hedge to any problem and ordering.
\nThe data table also highlights the empirical failure of several collaborative agent patterns, providing the foundation for our ablations (detailed in Section 7):
\nSwarmRAG-Async-SVM and blackboard-collaborative-bul) exhibited extreme volatility. Local scores plummeted as low as 28, and leaderboard scores crashed to 27. Our traces confirmed that these setups actively facilitated \"echo chambers\" of correlated errors.Scout-CRAG-Arbiter runs hovered in the mid-30s. Injecting external problem corpus data during inference caused the model to overfit to the retrieved examples rather than focusing on the actual problem constraints.Ultimately, the empirical data proved that in a compute-constrained environment, isolation, adversarial verification, and structured search (AMO) heavily outperform both blind parallel sampling (baseline) and unverified agentic collaboration (SwarmRAG).
There were many different types of attempts we made on trying to build the best agentic structure for AIMO3. Some of them were novel and forward-looking but ultimately didn't make it into the final AMO to avoid noise, complexity, and lack of time.
\nMotivated by EDIS (arXiv:2602.01288), which suggests that entropy dynamics in token generation reveal reasoning quality, we computed mean, max-window, and tail entropy from raw vLLM top_logprobs.
all-MiniLM-L6-v2. We had 2 RAG systems (one for traces, one for code). The RAG calls were infused as Python modules in the sandbox tools.Following up on the questions from @philipvonderlind, here are some additional questions to help us assess your submission:
\nLook forward to reading your responses on these.
", "rawMarkdown": "Following up on the questions from @philipvonderlind, here are some additional questions to help us assess your submission:\n\n1. **Entropy debunking:** Your Trace Viewer analysis concluded that token-level entropy has only weak correlation to correctness in a Python-tooled environment (\"entropy fluctuations largely reflected the model's confidence in Python syntax rather than mathematical truth\"). Several top public notebooks, including the popular [44/50 notebook](https://www.kaggle.com/code/nihilisticneuralnet/44-50-let-me-over-cook), use entropy weighting in their voting. How do you reconcile your finding with their reliance on it? Is the gain there also within noise, or are they using entropy in a way that is different from what you found not to work?\n2. **Verifier Agent error rates:** Your Verifier Agent is described as \"an adversarial auditor that attempts to computationally falsify the claim over 6-8 turns\". Adversarial verification could either catch wrong answers the solver missed (good) or reject correct answers that look unusual (bad). Do you have data on the Verifier's true-positive rate (correctly rejecting wrong claims) vs false-positive rate (rejecting correct claims)?\n3. **Extremum Protocol invocation:** You introduce the Extremum Protocol to address Extremum Bias as a systematic failure mode. On the 50-problem local benchmark, how often does the protocol actually fire, and on what fraction of those firings does it flip the answer from a wrong majority vote to the correct one?\n\nLook forward to reading your responses on these.\n" }, { "id": 3457751, "postDate": "2026-05-14T09:19:50.313Z", "content": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.\n
\nI like your use of plots to communicate the proposed architecture. I have a few questions about some of the stages:
\nBenchmark: You mention constructing a local benchmark to test your approach. Could you describe why you chose to create your own benchmark and explain the methodology behind it? There were many Math Corpus Prize submissions that have been widely used by the community, so I would be curious to understand the motivation. In addition, the scores on your local benchmark appear more stable than the public leaderboard results, which seem to have much higher variance. How do you explain this difference? For example, could it be related to always evaluating problems in the same order, differences in difficulty, or other benchmark-specific factors?
Time Management: You mention using the more extensive “Beam Search” generation strategy when there is abundant time remaining. Could this become problematic if many hard problems appear toward the end of the evaluation, after a significant amount of time has already been spent on expensive Beam Search runs earlier? Did you encounter or document such cases during development?
Temperature: Why does your SCOUT stage use a temperature of 0.6? It would be helpful to motivate this choice with ablations showing that a non-default temperature improves performance. For example, the submission by @natnitarach argues that t=1 is already the most diverse and highest-performing setting for gpt-oss-120b.
Scouts: In practice, is the SCOUT phase worth the GPU time it consumes? I would expect it to use a non-negligible portion of the overall budget, which might otherwise be spent on generating additional solution completions. Did you run any experiments confirming that the SCOUT phase improves final performance enough to justify its cost?
Citations and Reproducibility: Please cite the publicly shared notebook, as it appears to serve as the basis for at least parts of your final submission. In addition, please share the code for all ablations and experiments, and document the steps required to reproduce them.
Looking forward to your response :)
", "rawMarkdown": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the [updated discussion post.\n](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/689703)\n\nI like your use of plots to communicate the proposed architecture. I have a few questions about some of the stages:\n\n1. **Benchmark**: You mention constructing a local benchmark to test your approach. Could you describe why you chose to create your own benchmark and explain the methodology behind it? There were many Math Corpus Prize submissions that have been widely used by the community, so I would be curious to understand the motivation. In addition, the scores on your local benchmark appear more stable than the public leaderboard results, which seem to have much higher variance. How do you explain this difference? For example, could it be related to always evaluating problems in the same order, differences in difficulty, or other benchmark-specific factors?\n\n2. **Time Management**: You mention using the more extensive “Beam Search” generation strategy when there is abundant time remaining. Could this become problematic if many hard problems appear toward the end of the evaluation, after a significant amount of time has already been spent on expensive Beam Search runs earlier? Did you encounter or document such cases during development?\n\n3. **Temperature**: Why does your SCOUT stage use a temperature of `0.6`? It would be helpful to motivate this choice with ablations showing that a non-default temperature improves performance. For example, the [submission](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/model-capability-dominates) by @natnitarach argues that `t=1` is already the most diverse and highest-performing setting for `gpt-oss-120b`.\n\n4. **Scouts**: In practice, is the SCOUT phase worth the GPU time it consumes? I would expect it to use a non-negligible portion of the overall budget, which might otherwise be spent on generating additional solution completions. Did you run any experiments confirming that the SCOUT phase improves final performance enough to justify its cost?\n\n5. **Citations and Reproducibility**: Please cite the publicly shared notebook, as it appears to serve as the basis for at least parts of your final submission. In addition, please share the code for all ablations and experiments, and document the steps required to reproduce them.\n\nLooking forward to your response :)" } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T04:23:48Z", "competition_id": 118448, "competition_slug": "ai-mathematical-olympiad-progress-prize-3", "team_id": 14799861, "solution_url": "/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/amo-agentic-math-orchestrator-for-olympiad-mathem" }