# Road to Deterministic & Efficient Inference and Contamination Free Evaluation Abstract The deployment of Large Language Models (LLMs) for complex mathematical reasoning is heavily constrained by available compute and inference latency. In the AI Mathematical Olympiad (AIMO) Progress Prize 3, systems are restricted to a strict 5-hour execution window on a single NVIDIA H100 (80GB VRAM) GPU. While recent research has demonstrated that scaling model size and test-time computation improves absolute accuracy on mathematical benchmarks, the degradation in reasoning efficiency—and its subsequent impact under strict compute constraints—remains underexplored. In this paper, we evaluate the reasoning efficiency of state-of-the-art mid-size models, demonstrating that without optimization, the advantages of high-accuracy models are frequently nullified by time constraints. Furthermore, we analyze severe score variance (non-determinism) observed during evaluation, identifying algorithmic time-management strategies and agentic execution faults as the primary culprits. By implementing strict time-allocation controls, robust state recovery mechanisms, and inference optimizations including Speculative Decoding and FP8 KV-cache quantization, we stabilized leaderboard variance and doubled throughput. Finally, to validate these optimizations, we introduce a novel, contamination-free evaluation methodology, ensuring rigorous benchmarking against post-training data sets. 1 Introduction The AI Mathematical Olympiad (AIMO) Progress Prize 3 introduces a highly constrained execution environment, requiring systems to resolve complex mathematical problems within a strict 5-hour time limit using a single NVIDIA H100 GPU. Previous research aimed at improving the performance of LLMs in mathematical reasoning has predominantly been conducted in environments with unrestricted compute. Consequently, while these methodologies improve absolute accuracy on standard mathematical benchmarks, there is limited study on the associated costs to inference efficiency. When deployed in compute-bound environments, models with higher theoretical accuracy often engage in computational "overthinking," leading to timeouts that negate their capability advantages. Simultaneously, LLM inference within agentic frameworks exhibits significant non-determinism. In AIMO3, the effect of this variance appeared disproportionately severe, with public leaderboard scores fluctuating by up to 10 points (a ~25% variance) across identical code submissions. To address these challenges, this paper presents a comprehensive analysis of inference efficiency and non-determinism in mathematical problem-solving agents. Our core contributions include: 1. Analysis and Mitigation of Non-Determinism: Identifying the root causes of execution variance and proposing algorithmic and system-level solutions to stabilize evaluation scores. 2. Compute-Constrained Efficiency Optimization: Evaluating model efficiency and implementing techniques to maximize throughput and pass rates within a strict hardware budget. 3. Contamination-Free Evaluation Framework: Proposing a robust methodology for evaluating LLMs using high-difficulty, guaranteed zero-leakage datasets. In the spirit of open-source research and reproducibility, we have released our full solution codebase, models, datasets, and reasoning traces. Codes & Model: Final Submission with execution time limitation. [Kaggle Link](https://www.kaggle.com/code/chewkokwahibrainai/lb43-12x80k-k8-nospecial-conf2048-eagle) [Colab Link](https://colab.research.google.com/drive/15EU80AQQ9l7wSAKUxcfPVqf1WbbGrh9v?usp=sharing) Inference code with unrestricted execution time, competitive with SOTA achieve by bigger open model. [Kaggle Link](https://www.kaggle.com/code/chewkokwahibrainai/evaluation-code-with-unrestricted-execution-time?scriptVersionId=313742657) [Colab link](https://colab.research.google.com/drive/1qhXG_PKaRFIcW0tVu8tV5DHTng8YWmQT?usp=sharing) Post-mortem evaluation with FP8 kv cache quantization scaling calibration. Dataset: [Questions converted from non-Integer answer](https://www.kaggle.com/datasets/chewkokwahibrainai/matharena-apex-non-integer) [Contamination Free High Difficulties Evaluation set.](https://www.kaggle.com/datasets/chewkokwahibrainai/hardest-competition-problems-2025) [Model Outputs & Reasoning Traces](https://www.kaggle.com/datasets/chewkokwahibrainai/evaluation-result-and-reasoning-trace) 2. Analysis and Mitigation of LLM Non-Determinism 2.1 Algorithmic Time Management as a Source of Variance Initial observations indicated massive score fluctuations (between 35 and 45 points) on the Public Leaderboard despite using identical deterministic code and sampling parameters. Analysis revealed that this variance was not strictly due to the inherent stochasticity of LLMs, but rather an artifact of the dynamic per-question time management algorithm utilized in the execution pipeline. The specific implementation responsible for this variance is detailed below: high_problem_timeout = 900 base_problem_timeout = 300 elapsed_global = time.time() - self.notebook_start_time time_left = self.cfg.notebook_limit - elapsed_global problems_left_others = max(0, self.problems_remaining - 1) reserved_time = problems_left_others * self.cfg.base_problem_timeout budget = time_left - reserved_time budget = min(budget, self.cfg.high_problem_timeout) budget = max(budget, self.cfg.base_problem_timeout) This heuristic operates on two main principles: it guarantees a minimum baseline execution time (300 seconds) for all problems, and it dynamically allocates any "saved" time from rapidly solved problems to the current active problem, up to a maximum threshold (900 seconds). This greedy allocation strategy creates severe sensitivity to question ordering. We can illustrate this via two extreme simulated scenarios across 50 questions that give rise to the extreme 35- and 45-point variance when the question order changes: To demonstrate this, let us define a mock evaluation set of 50 questions categorized into three difficulty tiers based on the total execution time required to generate an answer: Question Type Total Questions Execution time per question Hard 5 > 900 seconds Intermediate 10 300 to 900 seconds Easy 35 < 300 seconds Pessimistic Scenario (35 Points): ![](assets/101e2be65c40a8418d866d99.png) i) 5 Hard Questions are encountered in the first 5 slots, the system fully exhausts the maximum time allowance per question, accumulating 4,500 seconds of compute without yielding correct answers. ii) Consequently, the remaining 45 questions are bottlenecked by the minimum 300-second allocation. iii) The 10 Intermediate Questions failed to produce a correct answer in 300 seconds. iv) As a result, only 35 points have been scored. Optimistic Scenario (45 Points): ![](assets/290a8c8cfe7a33fb644269cf.png) i) Hard Questions arrive during the last 5 slot. ii) The rest of 45 questions arrived in the earlier slot and all have enough time to finish execution, yielding a score of 45 points. Mitigation: By entirely deprecating the dynamic time management code and enforcing a static compute budget per question, the systemic variance was eliminated, resulting in highly stable, reproducible leaderboard scores within a narrow 1-point margin (39 to 40 points). 2.2.Resolving Execution Faults in Agentic Environments The Tool-Integrated Reasoning environment utilizes a stateful Python Code Interpreter (via Jupyter kernels) to execute intermediate mathematical verifications. Initial implementations of this agentic environment suffered from severe systemic instability, frequently resulting in "Notebook Inference Server Errors" or silent process hangs. These issues scaled exponentially under high parallelization (e.g., managing 16–20 concurrent workers). Through rigorous debugging of the agent loop and sandbox architecture, we identified several race conditions, memory leaks, and uncatchable exceptions. To eliminate these intermittent crashes and guarantee back-to-back stable leaderboard scores, we re-architected the sandbox environment with the following mitigation strategies: Dynamic Port Allocation to Prevent Race Conditions: The baseline architecture manually assigned communication ports (shell, iopub, stdin, hb, control) using a locked incremental counter. Under high-concurrency spawning, this frequently led to port collisions. We eliminated manual port assignment, delegating port resolution entirely to the KernelManager to dynamically bind to available OS ports. This resolved initialization bottlenecks and silent kernel startup failures. Zombie Process Mitigation and Hard Kills: Standard kernel interruptions (interrupt_kernel()) often fail when Python is executing compiled C-extensions (e.g., complex matrix operations in numpy or deep symbolic recursions in sympy). These orphaned processes would continue to consume RAM, eventually causing out-of-memory (OOM) crashes. We implemented a cascading shutdown protocol: if a kernel ignores a standard interrupt after a 1-second grace period, the environment intercepts the process ID (PID) and forces an OS-level SIGKILL. This guarantees the termination of rogue memory hogs. Active Health Monitoring via UUID Injection: Because a kernel can silently enter a deadlocked state without terminating, relying on process status is insufficient. We introduced an active is_healthy() heartbeat protocol. Before any tool execution, the sandbox injects a dynamically generated UUID into a print() statement with a strict 2-second timeout constraint. If the exact UUID is not returned, the kernel is definitively classified as unresponsive. Self-Healing and Graceful State Recovery: To maintain inference continuity without crashing the broader agentic loop, we implemented a self-healing mechanism. If the health check fails—or if a kernel is hard-killed—the sandbox pool automatically discards the dead instance and spawns a fresh Jupyter kernel. Crucially, to prevent the LLM from hallucinating variable availability after a restart, the system automatically injects a state-reset warning back into the LLM's context window: [SYSTEM NOTE: The kernel crashed and was restarted. Previous variables are lost.] This prompts the model to intelligently re-declare necessary variables rather than failing on NameError exceptions. By coupling these low-level sandbox optimizations with strict health thresholds, the agentic execution environment achieved near-perfect reliability, eliminating server-side inference errors within the strict 5-hour Kaggle compute window. 3.Optimizing Model Efficiency 3.1 Model Efficiency Evaluation Under the AIMO3 constraints, end-to-end execution time—defined as the total time required for an agent to generate a reasoning trace, execute the associated code, and produce a correct integer answer—is the primary bottleneck. We shortlisted three high-capability mid-size models for evaluation: GPT-OSS 120B, NVIDIA-Nemotron-3-Super, and Qwen3.5-122B-A10B. Evaluations were conducted using two recent competitive datasets (HMMT February 2026 and AIME February 2026). HMMT 2026 February Candidate Model Attempt per Question Tokens per Question End-to-end Time per Question (seconds) GPT OSS 120B 8 24332 383 NVIDIA-Nemotron-3-Super 120B 8 45880 433 Qwen3.5-122B-A10B 8 25103 411 AIME 2026 February Candidate Model Attempt per Question Tokens per Question End-to-end Time per Question (seconds) GPT OSS 120B 8 20119 204 NVIDIA-Nemotron-3-Super 120B 8 25233 213 Qwen3.5-122B-A10B 8 21225 235 3.2 Throughput Enhancements To maximize the number of valid reasoning paths explored per problem within the time limit, raw inference throughput must be optimized without degrading precision. We implemented Speculative Decoding to accelerate token generation and applied FP8 KV-Cache Quantization. These memory and compute optimizations yielded a 2x increase in token throughput. Crucially, this efficiency gain allowed the system to scale its sampling budget beyond the standard 8 attempts per question. Increasing the number of reasoning trajectories directly correlated with higher, more stable leaderboard performance as evidence by screenshot below: ![](assets/6b1f0a79540dced7e018962a.png) 4. Contamination-Free High-Difficulty Evaluation Dataset A persistent challenge in evaluating mathematical LLMs is the risk of data contamination, where benchmark questions are inadvertently included in the model's pre-training corpus. To ensure highly reliable comparisons, we constructed a guaranteed contamination-free evaluation dataset. Methodology: Temporal Isolation: We restricted candidate questions exclusively to those published after June 2024, which serves as the training data cutoff date for our primary model (GPT-OSS 120B). Format Alignment: Rather than discarding questions with non-integer answers, we manually reformulated them into integer-output constraints, matching the precise stylistic guidelines of AIMO problem setters. This yielded an initial pool of 513 candidate questions. Difficulty Filtering: To isolate high-difficulty reasoning challenges, we benchmarked the dataset using the top-performing public models. A question was retained only if it met one of two strict criteria: The model failed at least once across 8 independent sampling attempts. The model succeeded in all 8 attempts, but the cumulative execution time exceeded 500 seconds, indicating high computational friction. This rigorous filtering process resulted in a final dataset of 68 high-difficulty, verifiable, and zero-contamination mathematical problems. Statistic on Source of Questions ![](assets/b2185d975d5222f8914c73a7.png)