# Fighting Against Variance Under Compute Constraints

Abstract
This write-up presents a stability-focused agent architecture developed for the third AIMO Progress Prize, designed to make final answer selection more reliable under strict compute constraints. Instead of leaning too heavily on raw majority voting, token-level confidence, or other brittle heuristics, the system uses a structured multi-stage workflow in which a small set of base solver attempts generate candidate solutions, summarization nodes turn those traces into more uniform and auditable representations, and tool-enabled judger nodes evaluate the mathematical content of those summaries to support a more consistent final decision. Speculative decoding was incorporated to help the overall workflow fit within the runtime budget. Overall, the approach prioritizes stability, verification, and controlled consensus, particularly on harder problems where direct confidence signals and simple voting tend to become unreliable.
![](assets/7aa91408c97f8067f46b9f6c.png)
1. Introduction: Vulnerabilities in the Public Baseline
Many participants, myself included, started by analyzing the leading public notebooks[1,2]. While these notebooks provided a strong foundation, they contained some vulnerabilities that severely limited their stability on public leaderboard:
The Dynamic Time Cap:
To survive the strict 5-hour runtime limit, the public notebooks implement a time-budgeting mechanism. If the remaining overall budget runs low, it enforces a strict time cap (e.g., 300 seconds) per problem, which often leads to insufficient time to solve moderate-to-hard problems.
The Confidence Trap:
The baseline relies heavily on average token logprobs to measure model confidence, select the best answer, and break ties. However, logprob-based selection is highly vulnerable because there is no strong relation between logprobs and mathematical correctness (see the plots).
My strategy was to build a system that could bypass these vulnerabilities and seek a more stable overall solution process, even if that did not necessarily guarantee a better score.
2. The Compute Constraint & Monte Carlo Simulations
The 300-second time cap introduced by the public notebook's budgeting logic was problematic because we do not know the order or the difficulty distribution of the private test set.
To build some intuition about the effect of this cap, I ran Monte Carlo simulations modeling notebook behavior over 50 questions of varying difficulty. I approximated question difficulty and required compute time using simple heuristics guided by empirical observations. The results suggested that standard public-style approaches may leave many mathematically complex questions without enough compute to reach consensus before hitting the cap or running out of the 5-hour budget, especially in worst-case scenarios.
![](assets/584ba39702b5e76533b9a047.png)
To ensure every question received adequate time to explore mathematical pathways without hitting the budget cap, I implemented speculative decoding using the eagle3 draft model alongside gpt-oss-120b. Many thanks to
[@wenliangtlh](https://www.kaggle.com/wenliangtlh)
for publicly sharing their speculative model [3].
The Gains:
This provided a massive ~40% speedup on base attempt runs and a ~30% speedup on Judger nodes.
The Trade-offs:
Speculative decoding requires roughly 2x the KV cache storage. Consequently, the maximum vLLM concurrency limit dropped from 6 down to 3 (with a 65k context limit).
This hardware bottleneck forced me to be highly efficient. I strictly limited the base solver to 3 parallel runs and leaned heavily on the Judger nodes. In this architecture, the 3 base attempts act as an initial filter for easier questions, while the 5 parallel Judger nodes are only activated for harder problems requiring consensus.
Note on Concurrency:
Even though I instantiate 5 Judger nodes, I let vLLM manage the concurrency internally. While a strict concurrency limit of 3 at a 65k context could theoretically cause a bottleneck, in practice, not all Judger runs consume the full 65k token limit simultaneously. The asynchronous scheduling handles the queueing efficiently.
With speculative decoding enabled and the 3-Base/5-Judger node architecture, the notebooks consistently completed within the 5 hour submission limit. I experimented with different configurations (varying parallel runs and early consensus stops), but the 3-Base / 5-Judger setup with no early stop provided a decently stable score and became the basis for my final submission.
3. The Logprob Illusion
The second vulnerability was using logprobs to select the correct answer. My analysis showed that logprob sums and averages are fundamentally flawed metrics for evaluating mathematical correctness, especially under tie-breaking situations.
![](assets/0c292af5e2887ccdddb62218.png)
![](assets/ea402643a16ddb1efc497ca7.png)
As the plots above demonstrate, there is a massive density overlap in the average token logprobs and attempt vote weights between correct and incorrect answers. In tie situations, logprobs have almost no discriminative power.
Furthermore, the pass@n plots shared by the hosts [4] suggest that even when the model successfully generates the correct answer, that answer is often not the majority vote on hard questions. This makes logprob aggregation especially weak in the settings that matter most, since summing logprobs is unlikely to recover the correct answer when the dominant traces are already wrong.
Crucially, the log probability of a final answer token is entirely conditioned on the reasoning trace that precedes it. So, intuitively, treating one answer as “better” simply because it has a higher logprob than an answer produced through a different sequence of reasoning is mathematically unsound.
4. Methodology: The Summarizer-Judger Workflow
To replace logprob heuristics, I implemented a three-phase decision workflow.
Phase 1: The Base Solver
The system initiates 3 parallel base attempts (maximizing my concurrency limit). The base solver attempts are essentially the same as what is presented in [2], but with a few notable changes:
Removed _ensure_last_print:
The baseline forced a print() wrapper on the final line of code, which caused numerous Python syntax errors. I simply removed it.
The "42 Trap":
Because the system prompt uses \boxed{42} as an example, the model often outputs "42" prematurely. If a 42 is detected, I let the generation complete rather than stopping early, ensuring the parser captures the actual final \boxed{} notation.
Flexible Regex:
I updated the regex to catch answers formatted as equations (e.g., \boxed{a+b=10}) when they are presented at the end of the traces.
Early Exit: If all 3 base attempts independently arrive at the exact same answer, the system immediately outputs that answer. If they disagree, the pipeline moves to the summarization phase.
Phase 2: The Summarizer Node
Initially, I let the base solvers complete their generation and fed the final answer blocks directly to the evaluating nodes. The base outputs varied widely: some were just \boxed{} answers, while others were long step-by-step essays. From the traces I reviewed, the evaluator showed a noticeable tendency to favor the longest explanation, even when it was wrong.
To address this, I introduced a Summarizer Node. I modified the summarizer prompt to stress objectivity, including the instruction: “Be completely objective, do not whitewash the solution.” This helped make the base traces less stylistically uneven and reduced some of the artificial confidence carried into the next stage, resulting in cleaner, more standardized inputs for the judger nodes. I also tried lowering the temperature with a fixed seed to reduce tone variance further, but that did not produce any clear improvement. In future work, this could be improved with a dedicated summary-standardization step or by using a larger-context model that can process full traces directly.
Phase 3: The Judger Node & Smart Voting
The last step of the system is an ensemble of 5 Judger nodes. Early versions of the Judger functioned as simple multiple-choice selectors. This failed entirely if all base options were wrong and often caused the model to hang indefinitely.
I made two critical adjustments:
Loosened the Prompt:
I stopped forcing the Judger to pick an existing answer. Instead, I asked it to evaluate the mathematical reasoning of the summaries and come up with a final answer.
Tool Enablement:
I also gave the Judger node access to the Jupyter sandbox.
The "Smart Voting" Tie-Break Logic:
The system finalizes the answer using a dynamic hierarchy:
Judger votes have the highest priority in the final decision logic. Once 3 judger nodes converge on the same answer, the run is terminated early.
Total Votes (Combining Base + Judger votes, if Judgers tie).
Lowest Average Token Length (If votes are still tied(rare), shorter, more concise answers generally indicate less hallucination and are favored over long, rambling traces).
5. Prompt Engineering on Summarizer and Judger Nodes
This part was extremely challenging since triggering the summarizer and judger nodes is somewhat arbitrary, depending entirely on whether the base attempts agree. To overcome this, I used execution traces saved from previous experiment runs on 100 randomly selected questions from the validation set presented in [5].
To create a useful test set, I filtered these logs for instances where:
There was no early consensus.
At least one correct answer was presented in the base attempts.
The system had ultimately predicted the wrong answer in the previous run.
Using this curated small subset, I hand-tuned the prompts to elicit behaviors such as varying levels of summarization detail and different degrees of judge involvement.
Metric
Exp 1 (Baseline)
Exp 2 (Sum Temp 0.2)
Exp 3 (Prompt 4)
Exp 4 (Prompt 5)
Exp 5 (Prompt 3)
Exp 6 (Prompt 2)
Overall Success
6 / 7 (85.7%)
5 / 7 (71.4%)
7 / 7 (100%)
5 / 7 (71.4%)
6 / 7 (85.7%)
6 / 7 (85.7%)
Completed Judger Runs
21
21
21
21
21
21
Correct Judgers
18
16
16
16
15
17
Judger Accuracy
85.71%
76.19%
76.19%
76.19%
71.43%
80.95%
To validate these offline results, I further tested the Experiment 1 and Experiment 3 prompts dynamically on a hard reference question (Question 10) using multiple replicas.
Metric
Exp 3 (Prompt 4)
Exp 1
Overall System Success
3 / 8 (37.5%)
5 / 8 (62.5%)
System Success When Correct Ans. was Present
3 / 4 (75.0%)
5 / 5 (100.0%)
Judger Win Rate (When Triggered)
1 / 2 (50.0%)
3 / 3 (100.0%)
Zero-Shot Recoveries
0
2
Based on these results, I chose the Experiment 1 summarizer and judger prompts for my final submission. One interesting observation from this test was the judger’s ability to reason independently in some cases: even when all provided base-attempt summaries were incorrect, the judger node was still sometimes able to derive and assign the correct final answer on its own. Although this was not consistent enough to rely on as a primary mechanism, I share a sample run on reference question 10 below:
![](assets/a1d97a9d5464fb5d489f3cb9.png)
6.Experimental Variants
I tested several variants of the pipeline on a 50-question set that included the 10 host reference questions, a subset of 2025 IMO questions with integer answers (or answers that could be converted into integer form), and 2025 Turkish Mathematical Olympiad questions. I intentionally over-sampled reference question 10, evaluating it 8 times, since it was the hardest problem available and served as a useful stress test for both submission-time runtime estimation and judger behavior under pressure. I share some of the more notable experiments below:
Experiment Name
Early Stop Enabled (Base Attempts)
Early Stop Config (Base Attempts)
Judger Early Stop
Context Length
Local Score
Local Completion Time
Public LB
public-notebook 8 runs with early stop 4
yes
4
N/A
65k
45/50
3h 53m
-
4 base, 3 judger
no
N/A
2
65k
46/50
3h 29m
41
3 base, 3 judger
no
N/A
2
65k
46/50
3h 55m
40
8 base with early stop, 4 then 3 judger
yes
4
2
65k
45/50
4h 32m
-
3 base, 5 judger
no
N/A
3
65k
47/50
3h 48m
42*
3 base, 5 judger (higher context limit)
no
N/A
3
75k
49/50
4h 13m
42
5 base, 5 judger (lower context limit)
no
N/A
3
45k
42/50
3h 36m
-
5 base, 5 judger** (higher context limit)
no
N/A
3
72k
45/50
4h 42m
-
3 base, 8 judger**
no
N/A
5
65k
44/50
4h 11m
-
*Final Submission.
**These experiments were affected by the hard 900s time limit for particular questions. While my intuition suggests these would score higher, they were not suitable for final submission due to the 5-hour total budget limit.
Key Insight: Increasing the number of base attempts did not boost accuracy. My intuition is that by feeding more base attempts into the pipeline, we are simply generating more incorrect solution summaries. When the Judger is presented with too many conflicting, flawed solutions, it suffers from context overload and struggles to isolate the correct logic. Keeping the base attempts low (3) and relying on a higher number of empowered, tool-enabled Judger nodes (5) yields a much more stable system.
7.Limitations and Future Work
Lowering the number of base attempts and relying more heavily on judger nodes appears promising, but the effective vLLM concurrency limit of 3 makes this difficult to scale within the 5 hour runtime budget.
More broadly, making robust estimates requires repeated experiments or a broader question set. Because compute was limited, I often had to rely on single-run evaluations together with manual trace inspection, which makes some of the conclusions less certain.
It was also genuinely difficult to construct a local evaluation set that reflected the style and difficulty of the actual competition questions. For future competitions, it would be very helpful to provide more reference questions that better match the true competition distribution.
Another limitation is that, while a 3-attempt base setup gave decent results, the hosts’ analysis suggests that model pass@3 performance is only around 39/50. In that sense, the base solvers remain the main weakness of the system. Simply scaling the number of base attempts did not improve performance in my experiments, although with more compute there may be ways to make this work through broader multi-stage or higher-level aggregation strategies.
With more compute, one natural next step would be to run multiple instances of the full pipeline in parallel and combine them through a meta-judger or broader majority-voting scheme. vLLM concurrency on a single H100 is a real bottleneck here, but slightly more VRAM, for example on an H200, could allow for significantly more base agents.
What did NOT work for me:
Initiating many runs and killing the unpromising ones (as decided by the LLM).
Feeding direct raw solutions instead of summaries to the Judgers.
Heuristic-based voting instead of Judger nodes (e.g., factoring in trace lengths, tool errors, or logprobs).
Collaboration between base solver nodes.
8. Conclusion
This architecture was designed to address some of the weaknesses of time-capped, logprob-based selection by combining speculative decoding with a Summarizer-Judger workflow. Under the constraints of the competition, this made it possible to build a more structured and, in many cases, more stable decision process than simple majority voting or confidence-based tie-breaking alone.
At the same time, the system is still clearly limited by compute, evaluation quality, and the strength of the underlying base solver. Even so, I think the results suggest that structured multi-stage evaluation is a useful direction for improving the reliability of open-weight mathematical reasoning systems.
Thanks again to the hosts for an excellent competition, and also to the authors of the public notebooks and datasets, whose contributions provided a very strong foundation for experimentation.
References
Andreas Bis.
AIMO-3: GPT-OSS-120B with Tools
. Kaggle Notebook.
[https://www.kaggle.com/code/andreasbis/aimo-3-gpt-oss-120b-with-tools](https://www.kaggle.com/code/andreasbis/aimo-3-gpt-oss-120b-with-tools)
NihilisticNeuralNet.
44/50 — Let Me Over Cook
. Kaggle Notebook.
[https://www.kaggle.com/code/nihilisticneuralnet/44-50-let-me-over-cook](https://www.kaggle.com/code/nihilisticneuralnet/44-50-let-me-over-cook)
Wenliang1990.
gpt-oss-120b-eagle3-aimo3
. Hugging Face.
[https://huggingface.co/wenliang1990/gpt-oss-120b-eagle3-aimo3](https://huggingface.co/wenliang1990/gpt-oss-120b-eagle3-aimo3)
AI Mathematical Olympiad – Progress Prize 3 Discussion #679559
. Kaggle Discussion.
[https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/679559](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/679559)
AI Mathematical Olympiad – Progress Prize 3 Discussion #672507
. Kaggle Discussion.
[https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/672507](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/672507)