![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F21921286%2F739979ed5b173743d8d8ddc38a3b792c%2Fgrouped_bar_pass1.png?generation=1776742366312428&alt=media\")
\n \n Figure 1: pass@1 accuracy comparison across models and benchmarks (Qwen variants only)\n
AI Mathematical Olympiad - Progress Prize 3
\n\nGimran Abdullin · Rohin Garg · Mohammad Hafez · Jurgis Kemeklis · Marina Lin · Debojoti Das Soumya
\nJason Youm · Melody Yu · Brian Zhang · David Zhang · Sophia Zhang · William Zhao · Lerchen Zhong\n
Link to Google Colab notebook
\nFinal Private Score: 26.5
\nWe present our solution to the AI Mathematical Olympiad - Progress Prize 3 (AIMO-3) competition. Our approach is built around three pillars: systematic data curation and benchmark cleaning to establish reliable evaluation signals, extensive exploration of training and scaffolding techniques, and careful inference optimization to maximize the number of high-quality reasoning attempts within strict time and memory constraints. We find that GPT-OSS-120B, served with EAGLE3 speculative decoding and temperature-optimized sampling, provides a strong foundation that is difficult to improve upon through fine-tuning alone given limited compute. Our most effective improvement over majority voting was using a judge-corrected answer selection strategy that has potential to restore approximately 54% of ambivalent problems. Though we only achieved 26.5 for the private score, there are many meaningful findings, though not incorporated in our final submission, that we discovered and believe could be useful for the community. We release all evaluation datasets, rollout corpora, and reasoning-effort traces publicly to support future work.
\nRecent advances in large language models have produced systems capable of remarkable mathematical reasoning, yet olympiad-level mathematics remains a frontier challenge. Problems from the International Mathematical Olympiad (IMO) require not only deep mathematical knowledge but also multi-step creative reasoning, precise computation, and the ability to identify and avoid subtle errors — capabilities that stress-test even the strongest current models.
\nThe AIMO-3 competition provides a structured setting for evaluating progress on this frontier. Participants are given 50 IMO-level problems and must solve as many as possible within a five-hour wall-clock limit on Kaggle hardware (a single H100 GPU). Answers must be non-negative integers, enabling automatic grading. This setup rewards not only model capability but also careful systems engineering: inference efficiency, answer extraction robustness, and smart use of the compute budget all matter.
\nOur approach treats the problem as three interacting subproblems. First, we invested heavily in data quality — cleaning benchmark datasets, expanding training data through multilingual translation and quality filtering, and generating large rollout corpora to characterize model behavior. Second, we explored a wide range of training and scaffolding techniques, including reinforcement learning, model compression, speculative answer selection via PaCoRe and GenSelect, and introspective reward prediction via ZIP-RC. The majority of these experiments did not yield improvements over a strong baseline, and we report these negative results in detail. Third, we optimized the inference pipeline through EAGLE3 speculative decoding, temperature tuning, and careful vLLM configuration to maximize throughput within the time limit.
\nOur main finding is that GPT-OSS-120B is already a remarkably strong base model for this task, and that the most reliable gains came not from modifying the model but from improving how its outputs are aggregated. Our judge-corrected answer selection strategy, which invokes a second model pass on ambivalent problems, recovered roughly a third of cases where majority voting failed. We release all datasets and evaluation artifacts publicly to support reproducibility and future research.
\nWe established trustworthy evaluation metrics by carefully curating and cleaning a primary benchmark derived from IMO problems. The initial problem collections from public sources contained several categories of errors: incorrect answer keys that did not match official solutions, non-integer representations with floating-point precision issues, data contamination where identical problems appeared with conflicting answers, and inconsistent formatting in answer extraction.
\nOur cleaning pipeline addressed each error systematically. First, answer keys were manually verified against official IMO sources; ambiguous problems were flagged for manual resolution. Second, all numerical answers were standardized to canonical integer representations — removing leading zeros, eliminating floating-point variants, and normalizing special characters. Third, we identified and deduplicated problems appearing in multiple sources with conflicting metadata. Finally, we developed robust answer extraction logic to handle the diverse formatting present across different problem sources.
\nThe resulting cleaned IMO AnswerBench became our standard for all evaluation. This benchmark was used throughout the project for ablation studies, hyperparameter sweeps, and final reference runs that informed submission decisions.
\nBenchmark Specifications:
\nOur training corpus evolved through systematic expansion. We began with using the 400 IMO AnswerBench problems as evaluation in early project phases. As we identified additional data sources and developed effective filtering techniques, we ended with a corpus of 2200 high quality problems for our training set.
\nTo expand beyond English-only training data, we conducted multilingual expansion via translation and quality filtering. Chinese competition mathematics problems from BlueMO were automatically translated to English via GPT-5, then filtered by having GPT-5 solve each translated problem and retaining only those where the model achieved sufficient accuracy across 8 rollouts. This filtering removed problematic translations while also collecting high quality rollouts, providing 219 new problems.
\nTo further enrich the training distribution, we sourced problems from PolyMath and OlympMath-style datasets, applying similar quality filters to retain only problems with clean numerical answers. A subset of the hardest problems was isolated via repeated model filtering, yielding a curated set of approximately 17 very hard problems used to stress-test our training and evaluation pipelines.
\nBeyond dataset construction, we developed several tools to systematically characterize and improve data quality.
\nTopic Labeling and Coverage Analysis. We performed mathematical topic labeling and category assignment across the full training corpus to understand domain coverage. This analysis revealed meaningful gaps: geometry problems with hard numerical answers were scarce despite filtering efforts, and competition series such as RMM (Romanian Master of Mathematics) were entirely absent from our datasets. This coverage analysis informed subsequent data sourcing decisions and helped identify which problem types were likely underrepresented in training.
\nVery Hard Problem Filtering. To stress-test our training and evaluation pipelines, we applied repeated model filtering to isolate the hardest problems in our corpus — those where even strong models rarely succeed. This process yielded a curated set of approximately 17 very hard problems, useful for probing the ceiling of model capability and for validating that training improvements were not limited to easier subsets.
\nComputification Pipeline. We developed a computification pipeline to convert free-form olympiad problems into a more structured format amenable to automated processing and quality checking.
\nDivergence Judge. To automate data quality validation at scale, we implemented a divergence judge that compares two independent model solutions to the same problem and flags cases where the solutions diverge. High divergence indicates potential ambiguity in the problem statement, an incorrect answer key, or a problem that is genuinely underspecified. We achieved 100% judge restore on consistency checks, validating that the judge was reliable enough for automated filtering. This tool was used throughout the project to identify and remove problematic problems from both training and evaluation sets.
\nBeyond the training set itself, we generated diverse reasoning trajectories to enable development and validation of scaffolding strategies. Using OSS-120B as our base model for synthesis, we generated approximately 128 traces per problem across our 2170 training set problems. Each trace was stored with the problem ID, attempt number, correctness, full model output, and token count. This corpus enabled a difficulty indicator of all problems, by the correctness count over all 128 traces (with and without tool calling).
\nWe also created reasoning-effort evaluation datasets corresponding to different computational budgets. These consisted of the same problem set evaluated under three distinct configurations:
\n| Level | \navg@16 | \nbest@16 | \nmaj@16 | \n
|---|---|---|---|
| high | \n72.64% | \n87.98% | \n80.60% | \n
| medium | \n58.37% | \n83.61% | \n72.40% | \n
| low | \n50.68% | \n77.60% | \n60.11% | \n
All three variants were evaluated on the same 366 cleaned IMO AnswerBench validation problems. The generated traces for each reasoning level are publicly available on Hugging Face: high-reasoning traces at aimosprite/high-reasoning-eval, medium-reasoning traces at aimosprite/medium-reasoning-eval, and low-reasoning traces at aimosprite/low-reasoning-eval.
All major datasets are publicly available. The IMO AnswerBench (cleaned) is available on HuggingFace at aimosprite/imo_answerbench_clean_notebooks; the reasoning-effort datasets are on HuggingFace at aimosprite/{low,medium,high}-reasoning-eval. The OSS-120B rollout corpus on our training set of 2170 problems is stored on HuggingFace at aimosprite/training-set.
We selected GPT-OSS-120B as our primary base model for the following reasons. Among the open-weight models we evaluated, GPT-OSS-120B offered the best balance of three critical factors: inference speed, output quality, and solution conciseness. We evaluated several alternative base models including Qwen-3 (30B-A3B, 32B), Qwen-3.5 (9B, 35B, 122B), Nemotron 3 (120B), and others, but each presented tradeoffs unfavorable for the competition constraints.
\nQwen models demonstrated strong reasoning capabilities but generated substantially longer outputs, consuming tokens that were critical under the 5-hour time limit. Nemotron showed promise in terms of output quality but suffered from inference speed limitations that made it difficult to fit multiple parallel generations on an H100 within the time budget. GPT-OSS-120B, by contrast, produced reasonably high-quality solutions with concise lengths and could be efficiently parallelized — we achieved 8 parallel inference sequences on a single H100 GPU using vLLM without memory overflow, enabling effective batching of multiple problems simultaneously.
\n\n \n
\n Figure 1: pass@1 accuracy comparison across models and benchmarks (Qwen variants only)\n
\n ![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F21921286%2F13af66e0651e9d324412372bc7e98ba3%2Fgrouped_bar_pass1%20(1).png?generation=1776742429614431&alt=media\")
\n
\n Figure 2: pass@1 accuracy comparison across models and benchmarks (GPT-OSS-120B vs Qwen variants)\n
\n ![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F21921286%2F2aa19ffb14c77fb1d2b31593a7c0ac26%2Faccuracy_bars.png?generation=1776742641156800&alt=media\")
\n
\n Figure 3: avg@16, maj@16, and best@16 accuracy by benchmark for Qwen3.5-35B-A3B\n
\n ![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F21921286%2F31ae9aa55d34f48305161c2b9eb4f7d5%2Fcot_length_kde.png?generation=1776742717404254&alt=media\")
\n
\n Figure 4: Chain-of-thought length distribution across Qwen model variants\n
\n ![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F21921286%2F587001847762ac2a72d99bb7c81195b5%2Fcot_length_kde%20(1).png?generation=1776742744547805&alt=media\")
\n
\n Figure 5: Chain-of-thought length distribution — GPT-OSS-120B generates substantially shorter solutions than Qwen variants\n
\n ![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F21921286%2F8adae5cfc6e42c79dc3e23ee96f0009d%2Fthroughput_comparison.png?generation=1776742571284085&alt=media\")
\n
\n Figure 6: Inference throughput comparison across Qwen variants\n
\n ![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F21921286%2Fbf18a4faa5f71069f2f154d5c24c95e4%2Fthroughput_comparison%20(1).png?generation=1776742402740995&alt=media\")
\n
\n Figure 7: Inference throughput comparison — GPT-OSS-120B achieves 5,063 tok/s vs 1,699 tok/s for Qwen3.5-122B\n
\n ![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F21921286%2F974efa3ebc4fbac4b8b7dd48725cda36%2Fnemo.png?generation=1776789892384818&alt=media\")
\n
\n Figure 8: Nemotron vs GPT-OSS-120B throughput\n
The choice of GPT-OSS-120B was validated across all subsequent training phases. When we later experimented with distillation and compression, we consistently returned to this base model as our starting point, and no alternative consistently outperformed it on our Pareto frontier of accuracy versus inference latency. Additionally, adding Python tool calling further improved accuracy.
\nBase Model Specifications:
\nTo improve inference efficiency without sacrificing accuracy, we investigated on-policy distillation onto REAP-compressed model variants. REAP (Rank Estimation Aggregation Pruning) is a layer-wise weight pruning technique that removes less-important model parameters while maintaining or even slightly improving performance through weight redistribution.
\nWe applied REAP pruning to GPT-OSS-120B at several compression levels, retaining different percentages of original model weights: 97%, 95%, 93%, 90%, 75%, 50%.
\nThe pruning process works by ranking weights across each transformer layer and removing those with smallest magnitude. Importantly, REAP applies per-layer pruning ratios, allowing flexibility to retain more weights in critical early layers and prune more heavily in later layers.
\n\n ![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F21921286%2F3b52160d64c5223c149e2d3bc690f677%2Freap_ans.png?generation=1776788972805190&alt=media\")
\n
\n Figure 9: Accuracies across REAP compression levels — high compression (REAP-50, REAP-75) causes significant accuracy degradation\n
For the most promising REAP variants (REAP-95, REAP-75), we conducted on-policy distillation using the trained full model as the teacher. Since the directly pruned model had some loss in terms of accuracy, the size and speeds could allow for more sophisticated scaffolds.
\nDistillation Hyperparameters:
\n\n ![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F21921286%2F27694b328d1d64a85e5b8b13bf8f8e94%2Fopsd_eval_kl%20(1).png?generation=1776789007178292&alt=media\")
\n
\n Figure 10: On-policy distillation eval accuracy and KL divergence across training steps — accuracy is highly noisy with no clear improvement trend\n
However, the OPD was not very successful — accuracy was very noisy across training steps. Additionally, limited evaluations showed that the REAP models were not necessarily better in terms of throughput and speed, as shown in the comparison below:
\n| \n | reap95 | \nbase | \n
|---|---|---|
| Wall time | \n1,667s (27.8 min) | \n1,242s (20.7 min) | \n
| Total tokens | \n1,686,623 | \n1,209,490 | \n
| Throughput | \n1,012 tok/s | \n974 tok/s | \n
| Attempts run | \n86 | \n74 | \n
| Completed (not deadline) | \n70 | \n65 | \n
| Deadline-killed | \n16 | \n9 | \n
| Python calls | \n1,751 | \n1,005 | \n
| KV cache | \n316K tokens | \n237K tokens | \n
We conducted exploratory reinforcement learning (RL) experiments to directly optimize model generation length while maintaining correctness. The main technique explored was TOGGLE, introduced in Kimi K2.5 [6]. However, rollout rewards were too noisy to produce a useful training signal, and none of these experiments showed meaningful improvement.
\n\n ![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F21921286%2F181691353f6c679e1af95f23718023df%2Flenrl.png?generation=1776789231560501&alt=media\")
\n
\n Figure 11: RL rollout total and response lengths across training steps — highly noisy with no clear downward trend\n
\n ![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F21921286%2Ffb60bf941926fc184fd2242bcb543cef%2Frewrl.png?generation=1776789249762435&alt=media\")
\n
\n Figure 12: RL training reward curves — noisy signal prevented meaningful optimization\n
After exploring multiple training approaches, we concluded that further fine-tuning was not the most effective use of our resources. GPT-OSS-120B was already optimized for extended reasoning and demonstrated strong baseline accuracy out of the box. Without sufficient compute and time to run the large number of training iterations required to meaningfully shift model behavior, fine-tuning experiments risked being inconclusive or introducing regressions. We therefore shifted our focus toward inference-time improvements: scaffolding, answer selection, and throughput optimization, where we could iterate more quickly and measure impact more reliably.
\nBeyond improving the base model itself, we invested substantial effort into scaffolding strategies. Inspired by GenSelect from AIMO-2 [3], we hypothesized that smarter answer selection could unlock accuracy that majority voting leaves on the table. This was strongly motivated by the distribution of correct rollouts across our evaluation set: a large fraction of problems had the correct answer appearing in only 1–7 out of 16 rollouts, meaning the model demonstrably could solve these problems, but majority voting would consistently select the wrong answer. The gap between best@16 (87.98%) and maj@16 (80.60%) on high-reasoning traces confirmed this — nearly 8% of problems were solvable in principle but lost at aggregation. Scaffolding strategies that could identify the correct rollout among a mixed set therefore represented a high-potential direction for improvement.
\n\n ![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F21921286%2Fd2ea68a432305ae2e7eb31ea12d0bf46%2Frollouts_acc.png?generation=1776789879466833&alt=media\")
\n
\n Figure 13: Distribution of correct rollouts per problem out of 16 — many problems have the correct answer in only a minority of rollouts, motivating smarter answer selection\n
We established majority voting as our primary baseline for answer aggregation. For each problem, we generate K independent reasoning attempts and extract the final numerical answer from each. The simplest scaffold is to merely do a majority vote over all the answers. Additionally, we evaluated scaling the number of rollouts under an early-stopping majority strategy, which we denote \"high_match(d, n)\" below: generate up to d rollouts and stop once n agree. Our baseline configuration of (d=8, n=4) at max sequence length 65,536 tokens completed in 167 seconds. Increasing to (d=11, n=4) with a reduced max sequence length of 52,428 tokens achieved 82.23% accuracy in 163.5 seconds — a difference of under 0.6%, corresponding to roughly 2 problems out of 366. We concluded that scaling raw rollout count yields marginal returns, and that judge-based approaches represent a more promising direction for improving accuracy within a fixed compute budget.
\n\n ![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F21921286%2F585ab940c29b3b369c950c7af8045f05%2Fpure_high_wallclock_pareto.png?generation=1776789286189849&alt=media\")
\n
\n Figure 14: Pareto frontier of early-stopping strategies (accuracy vs wall time) — the recommended (d=8, n=4) configuration sits on the frontier with minimal gains from more rollouts\n
With our initial scaffold, around one in every 20 rollouts would fail to return an answer for one of the following reasons: \\boxed was never output, the boxed content included text, or the rollout hit the token limit. In many of these rollouts, the model had already settled on the correct answer in its chain-of-thought, meaning we were losing valuable information every time we failed to extract an answer.
To address this issue, we introduced a recovery step triggered every time the boxed output contained non-digit characters or when the rollout hit the token limit without boxing an answer. This step employed prefilling, where we appended the token encoding of the string “I must respond with a nonnegative integer answer; I will remove all non-digit characters from my answer. Therefore, my final integer answer is \\\\boxed{“ to the assistant’s last generation turn before sending it back to vLLM to generate the completion. By essentially speaking for the model with these specific formatting instructions, we were able to recover integer answers in 99% of cases, with a substantial fraction matching the correct answer.
Our experiments with prefilling revealed unexpected results:
\n\n ![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F21921286%2F74b0deabe739f4bead5d32e51d07437d%2Fprefill_need.webp?generation=1776790254953872&alt=media\")
\n
\n Figure 15: pass@1 rollout accuracies with and without prefilling\n
\n
\n ![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F21921286%2F45c975a7cb70792d67963794474ec505%2Fprefill_improve.png?generation=1776790267766938&alt=media\")
\n
\n Figure 16: maj@8 accuracy with and without prefilling — prefilling degrades majority vote on harder problems\n
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F21921286%2Fa52d9c4077614e53ba3488dfa24a3bbd%2Fprefill_time.webp?generation=1776790282525235&alt=media\")
\n On Full Rollout Set: Upon aggregating all rollout answers, we found that rollouts using prefill had a lower accuracy overall, likely because rollouts that failed to naturally box an answer were less promising overall. Across 20,000 rollouts on the IMO Answerbench dataset, we found that prefilled answers had an average accuracy of 58.5% compared to the non-prefilled average of 78.4%.
\nMajority Voting Degradation: When we analyzed the effect of introducing prefilled answers into our majority voting selection process, we found that our accuracy did not improve. We even found certain problems where the rollouts that used prefilling all produced the same incorrect answer, which was directly harmful.
\nThus, we abandoned the prefilling strategy as it did not appear to boost the accuracy of majority voting.
\nZero-Overhead Introspection For Adaptive Test-time Compute (ZIP-RC) [5] is a technique for selecting answers based on a fine-tuned model to predict the probability distribution of remaining tokens and correctness (reward) in place of unused tokens. There is zero overhead because one can directly extract the distribution while getting each token.
\nThis involved extracting hidden states and training a head to predict the joint distribution of remaining tokens and correctness reward. We found this approach either overfit on the limited available training data or collapsed to predicting a constant distribution. We subsequently simplified the objective to directly predicting completion and reward with separate heads. Additionally, since the last hidden layer is more suited to predicting the next token, like EAGLE-3 [1], we extracted hidden state layers 2, num_layers // 2, and num_layers - 3, denoted eagle_hidden_0, eagle_hidden_1, and eagle_hidden_2. While completion proportion proved reasonably learnable, reward prediction remained too noisy to be reliable. Given this noise, the downstream scaffolding strategy of optimizing a function trading off predicted reward against remaining tokens did not yield meaningful improvements over the baseline, and we abandoned this direction.
\n ![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F21921286%2Fbed03b7e4f8600c5567aad0fda6a4501%2Fcontinuity%202.png?generation=1776795330248038&alt=media\")
\n
\n Figure 18: ZIP-RC Lite prediction curves across hidden layers — reward prediction (blue) is highly noisy while completion proportion (green) is more learnable\n
We implemented a full version of ZIP-RC with the following procedure:
\n\n ![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F21921286%2F168239322796c4f53ab9f361fe6d34b2%2FScreenshot%202026-04-20%20at%2010.35.49PM.png?generation=1776795357148612&alt=media\")
\n
\n Figure 19: Full ZIP-RC training — predictions remain extremely noisy and fail to converge, consistent with insufficient training data\n
As with the partial variant, the resulting predictions were extremely noisy and inaccurate, likely due to insufficient training data — the original paper trained on 100K rollouts of simple math data, whereas our data was considerably more limited. We discontinued both the full and partial ZIP-RC directions in favor of simpler, more reliable inference-time approaches.
\nTo improve upon majority voting, we explored PaCoRe: Learning to Scale Test-Time Compute with Parallel Coordinated Reasoning [4]. Our implementation operated in two stages: in Part A, the model generates 8 parallel rollouts per problem and skips questions where all rollouts agree; in Part B, for problems with diverse outcomes, a \"PaCoRe prompt\" is constructed showing the model a mix of its own correct and incorrect prior attempts as context, then 8 new rollouts are generated conditioned on those references — the intuition being that exposure to diverse solutions helps the model self-correct through consensus. Once sufficient correct Part B traces accumulated, the pipeline paused to run supervised fine-tuning via Unsloth LoRA on the correct traces, saved a checkpoint, and resumed generation in a continuous improvement loop.
\n\n ![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F21921286%2Fe124ddbd2212c90f1abe37a346b74958%2Fpacore_loss.webp?generation=1776795493682096&alt=media\")
\n
\n Figure 20: PaCoRe SFT training loss across rounds — loss decreases consistently across all rounds\n
\n ![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F21921286%2F7300779f6348fb1a9057cfcd1c5f17b2%2Fpacore_met.webp?generation=1776795498394323&alt=media\")
\n
\n Figure 21: PaCoRe eval metrics across rounds — Part A and Part B accuracy remain flat despite decreasing loss, showing no meaningful improvement over majority voting\n
While SFT loss decreased as expected during training, evaluation results showed no meaningful improvement over standard majority voting. We attribute this to the model already being strong enough that the PaCoRe coordination signal provided little additional useful learning signal within our compute budget, and discontinued this direction.
\nGenSelect [3] shares the same infrastructure as PaCoRe but differs in what it trains the model to do in Part B. Part A is identical — 8 parallel rollouts per problem, graded against ground truth, skipping problems where all answers agree. In Part B, rather than synthesizing a new solution from prior attempts, the model acts as a comparative judge: given 4–8 reference solutions from Part A, it must select the best one rather than re-derive independently. The GenSelect prompt explicitly forbids re-solving and enforces voting discipline — the model must either pick the majority answer or cite concrete errors in every majority solution to justify overriding it. The iterative SFT loop is otherwise identical to PaCoRe: once 512 correct Part B traces accumulate, LoRA fine-tuning is run on the correct traces, the checkpoint is reloaded, and generation resumes.
\n\n ![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F21921286%2Fe80532673604b28f47f94b8f9e2f9b63%2Fsft_loss_1.webp?generation=1776795687807431&alt=media\")
\n
\n Figure 22: GenSelect SFT training loss across rounds — loss decreases across all three rounds\n
\n ![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F21921286%2F2ad7b0dbd82f0e8c85f77ba659a771b6%2Feval_metrics_1.webp?generation=1776795682377312&alt=media\")
\n
\n Figure 23: GenSelect eval metrics across rounds — avg@8, maj@8, and best@8 remain flat and noisy across rounds despite training loss reduction\n
Unfortunately, GenSelect suffered the same fate as PaCoRe — while SFT loss decreased during training, we observed no meaningful accuracy improvement over majority voting at evaluation time. We suspect the base model was already capable enough that neither coordination signal provided sufficient learning gradient within our compute budget.
\nReasking is a simple technique where, after generating an initial reasoning attempt, we prompt the model to reconsider its answer while retaining the full context of its previous response. The hypothesis: allowing the model a second chance to reason might catch errors or find better solutions without starting from scratch.
\n\n ![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F21921286%2F2eb3495b5b699973a1f221e35ff15616%2Ftwice_phase_times.webp?generation=1776796055731471&alt=media\")
\n
\n Figure 24: Phase 1 and Phase 2 reasking inference times — Phase 2 takes 35% of them time Phase 1 takes \n
In practice, we observed a slight accuracy improvement of 2.3%: the model would occasionally, only 3.8% of the time, correct a wrong answer on the second attempt, but this came at the cost of slightly longer inference time. Critically, during a prompt sweep for the Phase 2 reasking prompt, the correction was not always in the right direction — the model would sometimes change a correct answer to an incorrect one, making the net gain marginal. Ultimately, we did not pursue reasking further given the unfavorable accuracy-to-latency.
\nOur final submission combined the standard solver scaffold with a judge pass for answer selection on ambivalent problems. The solver ran 8 parallel attempts per problem, with attempts 0–3 at temperature 0.3 and attempts 4–7 at temperature 1.0 to balance speed and diversity. Each attempt used tool-calling with a persistent Jupyter sandbox for Python execution, running up to 128 turns with strict integer-only answer extraction from \\boxed{}.
\nAfter all 8 attempts completed, answer selection proceeded as follows: if 7 or more attempts agreed, the majority answer was used directly (very rare); otherwise, a judge pass was invoked. The judge received all valid rollouts — stripped of code blocks but preserving reasoning — and was instructed to analyze the problem, assess each solution's reasoning quality, and check for specific failure modes including missing cases, off-by-one errors, wrong model selection, weak bounds, parity errors, one-directional proofs, and overcounting. The judge could also run Python verification code if needed. Critically, the judge was constrained to select only from among the answers already proposed by the 8 rollouts and could not introduce a new answer. If the judge failed or timed out, the system fell back to majority vote.
\nOn ambivalent problems, the problems where majority voting over 8 rollouts yields an incorrect answer, the judge recovered between 33% and 54% of errors across 69 ambivalent problems, depending on the prompt used. However, more capable judge prompts introduced substantially longer inference time, so our final submission used a shorter prompt that balanced recovery rate against wall-clock budget. The full judge prompt is included in the appendix.
\nBeyond the above major scaffolding approaches, we explored several smaller-scale experiments with prompt variants and sampling parameters.
\nFull Prompt Sweeping: We prepended instructions like \"Avoid these common mistakes: [list of typical errors]\", compiling a list of frequent errors our model made by problem type. Attempts to have the model avoid these specific mistakes via prompting showed little effect, likely because the model was already aware of these issues but made mistakes due to reasoning errors rather than knowledge gaps.
\nWe also tried prompts like \"BIFURCATION: Identify three different mathematical approaches with their probabilities of success, then choose the one that is most likely to lead to the correct answer…\", and tried having different prompts across the 8 rollouts. However, there was a lot of noise in these results, and they suggest the original baseline prompt may be the best.
\n\n ![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F21921286%2Fc0f5db9a3424a876443b65f1643a4315%2Fpromptsweep.webp?generation=1776796248212787&alt=media\")
\n
\n Figure 25: Prompt sweep results across 10 variants — accuracy, wall clock time, and response length are largely consistent across prompts, with no clear winner\n
Chinese Thinking Prompts: We also hypothesized that prompting the model in Chinese might leverage different reasoning patterns or potentially generate less tokens (since Chinese language is more compact). We tested the exact same baseline prompt translated to Chinese. However, GPT-OSS-120B would automatically translate the problem back to English and think in English, even when the system prompt specifically said to think in Chinese and the problem was presented in Chinese.
\nWe systematically swept sampling hyperparameters to optimize the accuracy-compute tradeoff:
\nParameters Swept:
\nKey Findings:
\nAcross our scaffolding exploration, we investigated a broad range of techniques — from lightweight prompt variants and sampling sweeps to sophisticated generative selection methods like PaCoRe, GenSelect, and ZIP-RC. The majority of these approaches either failed to meaningfully improve over the majority voting baseline or introduced latency costs that outweighed their accuracy gains.
\nThe clearest findings were largely negative: scaling raw rollout count yields diminishing returns, prefilling degraded majority vote signal, reasking introduced bidirectional errors, and SFT-based coordination methods (PaCoRe, GenSelect) did not produce accuracy gains despite successful loss reduction. ZIP-RC, while theoretically appealing, was fundamentally bottlenecked by insufficient training data.
\nThe most effective scaffolding improvement came from our judge-corrected answer selection strategy. Rather than attempting to improve individual rollout quality, the judge operated post-hoc on the set of candidate answers, selectively overriding majority vote only on ambivalent problems where rollouts disagreed. It is important to note the additional inference cost the judge introduces. This targeted intervention, combined with temperature 0.3 for inference efficiency, constituted our final submission strategy. On AstralBench, the judge approach improved over majority voting from 15/50 to 17/50, demonstrating that even a simple judge prompt applied selectively to uncertain cases can meaningfully recover correct answers that majority voting would otherwise miss.
\nAchieving competitive performance required careful optimization of the inference pipeline to fit within Kaggle's time and memory constraints. This section describes the key inference decisions that shaped our final submission.
\nWe served GPT-OSS-120B using vLLM [7] with fp8 KV cache quantization and prefix caching enabled. The vLLM server was configured with a single GPU (tensor-parallel-size=1), a context window of 65,536 tokens, and up to 8 concurrent sequences. Fitting the model alongside EAGLE3 and a 65k context window required careful tuning: setting VLLM_FLOAT32_MATMUL_PRECISION=medium and VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 were necessary to avoid OOM, leaving only ~175 MiB of GPU memory headroom at peak. SGLang was evaluated as an alternative serving framework but was ruled out due to a kernel crash on GPT-OSS-120B caused by an unsupported hidden size.
We integrated EAGLE3 speculative decoding [1] using the draft model wenliang1990/gpt-oss-120b-eagle3-aimo3. We swept over speculative token counts (nst = 3, 4, 5, 6) across multiple AstralBench runs and found nst = 4 consistently performed best, achieving up to 17/50 on AstralBench in approximately 5–5.5 hours. Increasing to nst = 6 did not improve accuracy. Measured acceptance rates for nst = 4 were 0.724, 0.505, 0.357, and 0.252 at successive token positions — strong at the first speculative token but diminishing at depth, consistent with expectations. The Nvidia speculator was ruled out early due to its use of the full vocabulary size; P-EAGLE performed well on reap90 but OOM'd on the base model.
We also evaluated SpecForge for training a custom speculator on our rollout data. However, disk space constraints on our compute cluster limited training to sequence lengths ≤ 4096, and the finetuned speculator did not produce a measurable speedup. We hypothesize this was partly because SpecForge trains all positions and all speculated positions simultaneously, which may be suboptimal.
\nA key finding was that temperature 0.3 produced approximately 30% faster inference than temperature 1.0 with no accuracy loss, as lower temperature generates shorter solutions. Our final submission used temperature 0.3 for attempts 0–3 and temperature 1.0 for attempts 4–7 to balance speed with diversity. A full sweep of top-p, min-p, and repetition penalty found these parameters had negligible effect on accuracy, and we used defaults for all three.
\nOur final submission ran the base GPT-OSS-120B model with EAGLE3 (nst=4), temperature 0.3/1.0 mixed, fp8 KV cache, prefix caching, and 65k context. The base model completed 10 reference problems in 28 minutes at maj@12 with temperature 0.3 — the clearest timing baseline we measured.
\nThe most impactful decisions in our pipeline were largely about restraint: choosing not to fine-tune a model that was already strong, and focusing instead on inference efficiency and answer selection. Temperature 0.3 alone provided a 30% inference speedup with no accuracy cost, effectively giving us more rollouts within the time budget. EAGLE3 speculative decoding with nst=4 provided additional throughput gains and was the most reliable engineering improvement we found. The judge-corrected answer selection strategy was our most effective algorithmic contribution, recovering approximately 33% of problems where majority voting failed by applying a targeted second-pass verification only to ambivalent cases.
\nData quality also proved critical. Early in the project, we discovered systematic errors in public IMO benchmark datasets — incorrect answer keys, non-integer representations, and contamination — that were silently corrupting our evaluation signal. Cleaning these datasets was unglamorous but essential: without trustworthy evaluation, it is impossible to distinguish real improvements from noise, and several early experiments that appeared promising were later found to be artifacts of the buggy benchmarks.
\nThe most consistent theme in our negative results was that GPT-OSS-120B is already operating near the ceiling of what fine-tuning can easily shift. DPO training reduced loss but did not translate to accuracy gains. PaCoRe and GenSelect both showed healthy SFT loss curves but flat accuracy, suggesting the model was already capable of the coordination behavior being trained in. ZIP-RC was theoretically appealing but fundamentally data-limited — the original paper trained on 100K rollouts of simple arithmetic, and our more limited corpus was insufficient to learn reliable reward predictions. REAP model compression, while lossless at high retention rates, did not provide meaningful throughput gains and added engineering complexity without commensurate benefit.
\nPrompt engineering was similarly disappointing. Sweeping over ten structured prompt variants, common-mistake lists, bifurcation prompts, and Chinese-language prompts produced mostly noise, with the original baseline prompt performing as well or better than any alternative. This suggests the model's errors are primarily reasoning failures rather than knowledge or instruction-following failures — a finding with implications for future work.
\nOur approach is fundamentally constrained by the five-hour time limit, which caps the number of rollouts we can generate and limits the complexity of scaffolding we can apply. With more compute, judge selection based methods, as well as those that train the model to identify its best solution are likely to be more fruitful, as suggested by results from prior AIMO competitions [3]. Our SpecForge experiments were similarly hampered by disk and compute constraints that prevented training on long sequences.
\nThe judge strategy, while effective, adds significant inference overhead and its prompt was only lightly tuned. Early experiments suggested that even the first judge prompt tried recovered up to 54% of ambivalent problems, and that substantially better prompts are likely achievable with more iteration. Fine-tuning the judge specifically for mathematical verification is another promising direction not explored here.
\nWe presented our solution to AIMO-3, through a combination of careful data curation, inference optimization, and judge-corrected answer selection. Our central finding is that for a model as capable as GPT-OSS-120B, the highest-leverage improvements come not from modifying the model weights but from engineering the inference pipeline and answer aggregation strategy. Temperature tuning, speculative decoding, and targeted judge verification collectively allowed us to solve more problems within the competition's strict time constraints than any of our training experiments achieved.
\nWe also contribute a set of cleaned evaluation benchmarks and publicly released rollout datasets that we hope will support future research on olympiad-level mathematical reasoning. Our extensive negative results — across RL, distillation, ZIP-RC, PaCoRe, GenSelect, and prompt engineering — provide a map of approaches that do not easily improve over a strong baseline, which we believe is as valuable to the community as our positive findings.
\n[1] Hu et al. \"EAGLE-3: Enabling Lossless Acceleration of LLM Decoding via Dynamic Draft Model Updating.\" arXiv:2503.01840, 2025. https://arxiv.org/abs/2503.01840
\n[2] Leviathan et al. \"Fast Inference from Transformers via Speculative Decoding.\" ICML 2023. https://arxiv.org/abs/2211.17192
\n[3] Moshkov et al. \"AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset.\" arXiv:2504.16891, 2025. https://arxiv.org/abs/2504.16891
\n[4] Hu et al. \"PaCoRe: Learning to Scale Test-Time Compute with Parallel Coordinated Reasoning.\" arXiv:2601.05593. https://arxiv.org/abs/2601.05593
\n[5] Manvi et al. \"Zero-Overhead Introspection for Adaptive Test-Time Compute.\" arXiv:2512.01457, 2025. https://arxiv.org/abs/2512.01457
\n[6] Kimi Team. \"Kimi K2.5: Visual Agentic Intelligence.\" arXiv:2602.02276, 2026. https://arxiv.org/abs/2602.02276
\n[7] Kwon et al. \"Efficient Memory Management for Large Language Model Serving with PagedAttention.\" SOSP 2023. https://arxiv.org/abs/2309.06180
\n[8] aimosprite/high-reasoning-eval. https://huggingface.co/datasets/aimosprite/high-reasoning-eval
\n[9] aimosprite/medium-reasoning-eval. https://huggingface.co/datasets/aimosprite/medium-reasoning-eval
\n[10] aimosprite/low-reasoning-eval. https://huggingface.co/datasets/aimosprite/low-reasoning-eval
\n[11] aimosprite/imo_answerbench_clean_notebooks. https://huggingface.co/datasets/aimosprite/imo_answerbench_clean_notebooks
\n[12] aimosprite/training-set. https://huggingface.co/datasets/aimosprite/training-set
\nclass CFG:\n system_prompt = (\n 'You are a world-class International Mathematical Olympiad (IMO) competitor. '\n 'The final answer must be a non-negative integer between 0 and 99999. '\n 'You must place the final integer answer inside \\\\boxed{}.'\n )\n\n tool_prompt = (\n 'Use this tool to execute Python code. '\n 'The environment is a stateful Jupyter notebook. '\n 'You must use print() to output results.'\n )\n\n preference_prompt = (\n 'Use `math`, `numpy`, `sympy`, `itertools` and `collections` to solve the problem.'\n )\n\n served_model_name = 'gpt-oss'\n model_path = '/kaggle/input/gpt-oss-120b/transformers/default/1'\n\n kv_cache_dtype = 'fp8_e4m3'\n dtype = 'auto'\n\n high_problem_timeout = 900\n base_problem_timeout = 300\n notebook_limit = 17520\n server_timeout = 180\n session_timeout = 960\n jupyter_timeout = 10\n sandbox_timeout = 5\n stream_interval = 200\n\n context_tokens = 65536\n search_tokens = 1024\n buffer_tokens = 512\n batch_size = 8\n early_stop = 7 # disabled — always run all attempts\n attempts = 8\n workers = 16\n turns = 128\n seed = 42\n\n gpu_memory_utilization = 0.96\n temperature = 1.0\n\n # Judge configuration\n judge_temperature = 0.0\n judge_max_tokens = 65536\n judge_timeout = 300\n judge_system_prompt = (\n 'You are an expert mathematical verifier.\\n'\n 'You have access to a Python environment.\\n'\n 'Use ```python code blocks for code.\\n'\n 'Stop immediately after the closing ``` and wait for execution output.\\n'\n 'Never simulate code output.\\n'\n 'After seeing execution output, continue your reasoning normally.\\n'\n 'Keep your final response in the format requested by the user prompt.\\n'\n )\nYou are an expert mathematician judging {num_solutions}\ncandidate solutions (numbered 0 through {max_idx}) to an\nIMO-level math problem.\n\n### Task ###\n\nPick the most likely correct answer. The majority answer is\noften wrong in specific, predictable ways. Your job is to\ncatch those errors.\n\n### Instructions ###\n\n**1. Read the problem yourself first.**\nIdentify: what is given, what constraints apply, what is\nbeing asked, and what form the answer should take.\n\n**2. Assess each solution's reasoning.**\nQuote the solver's interpretation (1-2 sentences). Mark\nMISUNDERSTOOD if it answers a different question. When\nsolutions disagree, the minority may have found something\nthe majority missed.\n\n**3. Common solver failure modes (check for these, but don't\n limit yourself — there are many other possible mistakes):**\n\n * **Missing cases.** Solver finds one solution family and\n stops. Check: are there other valid constructions,\n additional solution branches, or cases the solver\n declared impossible without proof?\n\n * **Off-by-one.** When the majority answer differs from a\n minority answer by exactly 1, double-check which is\n right — the majority is often the one that's off.\n\n * **Wrong model.** Solver maps the problem to the wrong\n combinatorial structure. Check: ordered vs unordered,\n directed vs undirected, walks vs trails, L1 vs L∞\n metric, integer vs rational parameters.\n\n * **Premature impossibility.** Solver claims something is\n impossible via a hand-wavy argument. Check: did it\n actually verify small cases? A construction may exist.\n\n * **Weak bound assumed tight.** Solver proves an inequality\n but never constructs a matching example. A tighter bound\n from another solution may be correct.\n\n * **Parity / divisibility constraint ignored.** Check if\n the solver missed that some quantity must be even, or\n coprime, or satisfy a mod condition.\n\n * **One-directional proof.** Solver shows A implies B but\n the problem needs B implies A too (or vice versa).\n Especially common with polynomial/functional equations.\n\n * **Overcounting / undercounting.** Wrong sieve, double-\n counted configurations, or missed a symmetry factor.\n\n**4. Verification clues:**\n\n * **Strictly better answer wins.** If one solution finds a\n smaller minimum (with construction) or larger maximum,\n and proves it, prefer it.\n\n * **Counterexamples are heavy evidence.** If any solution\n explicitly demonstrates another answer violates a\n constraint, that's strong.\n\n * **Back-substitute.** Does the answer satisfy ALL original\n conditions?\n\n**5. Python verification.**\nYou have a Python tool. To use it, address recipient=python\nwith your code. Do NOT write ``to=python code...`` as text\nin your analysis — that does not execute.\n\n**6. Rules:**\n * Do NOT re-solve from scratch. Compare and verify.\n * Do NOT be swayed by majority count alone.\n * A confident wrong answer repeated 7 times is still wrong.\n\n**7. Output Format**\n\n* **a.** `<problem_statement_analysis>` 2-3 sentences on what\n the problem asks. `</problem_statement_analysis>`\n\n* **b.** `<per_solution_assessment>` For each solution: quote\n its interpretation, check understanding (UNDERSTOOD /\n MISUNDERSTOOD), note any reasoning concerns.\n `</per_solution_assessment>`\n\n* **c.** Understanding array (1=understood, 0=misunderstood):\n `<understanding>[1, 0, 1, 1, 0, 0, 1, 0]</understanding>`\n\n* **d.** Your best answer:\n `<suggested_answer>INTEGER</suggested_answer>`\n or `<suggested_answer>UNKNOWN</suggested_answer>`\n\n========================================================================\n### Problem ###\n\n{problem}\n\n========================================================================\n### Solutions ###\n\n{solutions}\n| Partition | \ntemp_0.6 | \ntemp_1.0 | \ntemp_1.2 | \n
|---|---|---|---|
| Part 0 | \n37/61 (60.7%) | \n38/61 (62.3%) | \n40/61 (65.6%) | \n
| Part 1 | \n50/61 (82.0%) | \n51/61 (83.6%) | \n51/61 (83.6%) | \n
| Part 2 | \n47/61 (77.0%) | \n46/61 (75.4%) | \n44/59 (74.6%) | \n
| Part 3 | \n53/61 (86.9%) | \n53/61 (86.9%) | \n52/61 (85.2%) | \n
| TOTAL | \n187/244 (76.6%) | \n188/244 (77.0%) | \n187/242 (77.3%) | \n
| Config | \navg@8 | \nbest@8 | \nmaj@8 | \nAvg Tokens | \nAvg Wall Time (s) | \nToken Limit Hits | \n
|---|---|---|---|---|---|---|
| top_p=0.80, min_p=0.00 | \n0.792 | \n0.893 | \n0.844 | \n20,669 | \n347.6 | \n43 | \n
| top_p=0.90, min_p=0.00 | \n0.782 | \n0.887 | \n0.833 | \n19,022 | \n312.1 | \n19 | \n
| top_p=0.90, min_p=0.02 | \n0.792 | \n0.901 | \n0.843 | \n20,295 | \n337.0 | \n31 | \n
| top_p=1.00, min_p=0.00 | \n0.778 | \n0.884 | \n0.817 | \n20,643 | \n335.5 | \n41 | \n
| top_p=1.00, min_p=0.05 | \n0.789 | \n0.895 | \n0.848 | \n20,547 | \n330.5 | \n32 | \n
| top_p=1.00, min_p=0.10 (a) | \n0.785 | \n0.895 | \n0.836 | \n20,248 | \n334.7 | \n35 | \n
| top_p=1.00, min_p=0.10 (b) | \n0.788 | \n0.893 | \n0.831 | \n19,126 | \n302.8 | \n29 | \n
Following up on the questions from @philipvonderlind, here are some additional questions to help us assess your submission:
\nLook forward to reading your responses on these.
", "votes": 0, "replies": [] }, { "id": 3457286, "author_name": "Philip Vonderlind", "author_url": "", "post_date": "2026-05-13T13:35:31.793000", "content": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.
\nYour work is already quite extensive. However, I think there are a few open questions that could make it even stronger:
\nDataset Creation: Since you create your own dataset, it would be helpful to include a diagram showing the different filtering stages and the number of samples discarded at each step. As a reference, you could look at the CrystalMath dataset from the Math Corpus Prize. Additionally, did you run any ablations against a human baseline to evaluate the impact of the filtering stages?
Judge Implementation: The judge system appears to be a crucial component of your submission. It would be useful to include plots and a more detailed explanation of how each judge works, as well as the conditions under which each judge is deployed.
Results & Plots: Some results for the Nemotron models seem to be missing from the plots. If available, please add the corresponding completion results, since the model is mentioned in some sections but not consistently covered elsewhere. Furthermore, the results for some of the REAP compression levels also seem to be missing from the 'REAP Compression Overview' (for the 90% to 97% levels).
RL Setup: Did you also experiment with other RL techniques, such as the commonly used GRPO or even just preference optimization (ORPO, …)? What motivated the choice of TOGGLE instead?
Looking forward to your response :)
", "votes": 0, "replies": [] }, { "id": 3456973, "author_name": "Lerchen Zhong", "author_url": "", "post_date": "2026-05-12T20:55:22.230000", "content": "Edited on 5/12 to include private results
", "votes": 0, "replies": [] }, { "id": 3447047, "author_name": "Simon Frieder", "author_url": "", "post_date": "2026-04-22T16:24:13", "content": "This seems like quite a comprehensive write-up -- I guess you are aiming for a prize, since you didn't make an entry towards the Write-Up Prize? (I don't recall seeing it there; there are still a few hours if you want to submit, FYI.)
", "votes": 0, "replies": [] } ], "comment_count": 4, "messages": [ { "id": 3446562, "postDate": "2026-04-21T19:14:50.577Z", "content": "AI Mathematical Olympiad - Progress Prize 3
\n\nGimran Abdullin · Rohin Garg · Mohammad Hafez · Jurgis Kemeklis · Marina Lin · Debojoti Das Soumya
\nJason Youm · Melody Yu · Brian Zhang · David Zhang · Sophia Zhang · William Zhao · Lerchen Zhong\n
Link to Google Colab notebook
\nFinal Private Score: 26.5
\nWe present our solution to the AI Mathematical Olympiad - Progress Prize 3 (AIMO-3) competition. Our approach is built around three pillars: systematic data curation and benchmark cleaning to establish reliable evaluation signals, extensive exploration of training and scaffolding techniques, and careful inference optimization to maximize the number of high-quality reasoning attempts within strict time and memory constraints. We find that GPT-OSS-120B, served with EAGLE3 speculative decoding and temperature-optimized sampling, provides a strong foundation that is difficult to improve upon through fine-tuning alone given limited compute. Our most effective improvement over majority voting was using a judge-corrected answer selection strategy that has potential to restore approximately 54% of ambivalent problems. Though we only achieved 26.5 for the private score, there are many meaningful findings, though not incorporated in our final submission, that we discovered and believe could be useful for the community. We release all evaluation datasets, rollout corpora, and reasoning-effort traces publicly to support future work.
\nRecent advances in large language models have produced systems capable of remarkable mathematical reasoning, yet olympiad-level mathematics remains a frontier challenge. Problems from the International Mathematical Olympiad (IMO) require not only deep mathematical knowledge but also multi-step creative reasoning, precise computation, and the ability to identify and avoid subtle errors — capabilities that stress-test even the strongest current models.
\nThe AIMO-3 competition provides a structured setting for evaluating progress on this frontier. Participants are given 50 IMO-level problems and must solve as many as possible within a five-hour wall-clock limit on Kaggle hardware (a single H100 GPU). Answers must be non-negative integers, enabling automatic grading. This setup rewards not only model capability but also careful systems engineering: inference efficiency, answer extraction robustness, and smart use of the compute budget all matter.
\nOur approach treats the problem as three interacting subproblems. First, we invested heavily in data quality — cleaning benchmark datasets, expanding training data through multilingual translation and quality filtering, and generating large rollout corpora to characterize model behavior. Second, we explored a wide range of training and scaffolding techniques, including reinforcement learning, model compression, speculative answer selection via PaCoRe and GenSelect, and introspective reward prediction via ZIP-RC. The majority of these experiments did not yield improvements over a strong baseline, and we report these negative results in detail. Third, we optimized the inference pipeline through EAGLE3 speculative decoding, temperature tuning, and careful vLLM configuration to maximize throughput within the time limit.
\nOur main finding is that GPT-OSS-120B is already a remarkably strong base model for this task, and that the most reliable gains came not from modifying the model but from improving how its outputs are aggregated. Our judge-corrected answer selection strategy, which invokes a second model pass on ambivalent problems, recovered roughly a third of cases where majority voting failed. We release all datasets and evaluation artifacts publicly to support reproducibility and future research.
\nWe established trustworthy evaluation metrics by carefully curating and cleaning a primary benchmark derived from IMO problems. The initial problem collections from public sources contained several categories of errors: incorrect answer keys that did not match official solutions, non-integer representations with floating-point precision issues, data contamination where identical problems appeared with conflicting answers, and inconsistent formatting in answer extraction.
\nOur cleaning pipeline addressed each error systematically. First, answer keys were manually verified against official IMO sources; ambiguous problems were flagged for manual resolution. Second, all numerical answers were standardized to canonical integer representations — removing leading zeros, eliminating floating-point variants, and normalizing special characters. Third, we identified and deduplicated problems appearing in multiple sources with conflicting metadata. Finally, we developed robust answer extraction logic to handle the diverse formatting present across different problem sources.
\nThe resulting cleaned IMO AnswerBench became our standard for all evaluation. This benchmark was used throughout the project for ablation studies, hyperparameter sweeps, and final reference runs that informed submission decisions.
\nBenchmark Specifications:
\nOur training corpus evolved through systematic expansion. We began with using the 400 IMO AnswerBench problems as evaluation in early project phases. As we identified additional data sources and developed effective filtering techniques, we ended with a corpus of 2200 high quality problems for our training set.
\nTo expand beyond English-only training data, we conducted multilingual expansion via translation and quality filtering. Chinese competition mathematics problems from BlueMO were automatically translated to English via GPT-5, then filtered by having GPT-5 solve each translated problem and retaining only those where the model achieved sufficient accuracy across 8 rollouts. This filtering removed problematic translations while also collecting high quality rollouts, providing 219 new problems.
\nTo further enrich the training distribution, we sourced problems from PolyMath and OlympMath-style datasets, applying similar quality filters to retain only problems with clean numerical answers. A subset of the hardest problems was isolated via repeated model filtering, yielding a curated set of approximately 17 very hard problems used to stress-test our training and evaluation pipelines.
\nBeyond dataset construction, we developed several tools to systematically characterize and improve data quality.
\nTopic Labeling and Coverage Analysis. We performed mathematical topic labeling and category assignment across the full training corpus to understand domain coverage. This analysis revealed meaningful gaps: geometry problems with hard numerical answers were scarce despite filtering efforts, and competition series such as RMM (Romanian Master of Mathematics) were entirely absent from our datasets. This coverage analysis informed subsequent data sourcing decisions and helped identify which problem types were likely underrepresented in training.
\nVery Hard Problem Filtering. To stress-test our training and evaluation pipelines, we applied repeated model filtering to isolate the hardest problems in our corpus — those where even strong models rarely succeed. This process yielded a curated set of approximately 17 very hard problems, useful for probing the ceiling of model capability and for validating that training improvements were not limited to easier subsets.
\nComputification Pipeline. We developed a computification pipeline to convert free-form olympiad problems into a more structured format amenable to automated processing and quality checking.
\nDivergence Judge. To automate data quality validation at scale, we implemented a divergence judge that compares two independent model solutions to the same problem and flags cases where the solutions diverge. High divergence indicates potential ambiguity in the problem statement, an incorrect answer key, or a problem that is genuinely underspecified. We achieved 100% judge restore on consistency checks, validating that the judge was reliable enough for automated filtering. This tool was used throughout the project to identify and remove problematic problems from both training and evaluation sets.
\nBeyond the training set itself, we generated diverse reasoning trajectories to enable development and validation of scaffolding strategies. Using OSS-120B as our base model for synthesis, we generated approximately 128 traces per problem across our 2170 training set problems. Each trace was stored with the problem ID, attempt number, correctness, full model output, and token count. This corpus enabled a difficulty indicator of all problems, by the correctness count over all 128 traces (with and without tool calling).
\nWe also created reasoning-effort evaluation datasets corresponding to different computational budgets. These consisted of the same problem set evaluated under three distinct configurations:
\n| Level | \navg@16 | \nbest@16 | \nmaj@16 | \n
|---|---|---|---|
| high | \n72.64% | \n87.98% | \n80.60% | \n
| medium | \n58.37% | \n83.61% | \n72.40% | \n
| low | \n50.68% | \n77.60% | \n60.11% | \n
All three variants were evaluated on the same 366 cleaned IMO AnswerBench validation problems. The generated traces for each reasoning level are publicly available on Hugging Face: high-reasoning traces at aimosprite/high-reasoning-eval, medium-reasoning traces at aimosprite/medium-reasoning-eval, and low-reasoning traces at aimosprite/low-reasoning-eval.
All major datasets are publicly available. The IMO AnswerBench (cleaned) is available on HuggingFace at aimosprite/imo_answerbench_clean_notebooks; the reasoning-effort datasets are on HuggingFace at aimosprite/{low,medium,high}-reasoning-eval. The OSS-120B rollout corpus on our training set of 2170 problems is stored on HuggingFace at aimosprite/training-set.
We selected GPT-OSS-120B as our primary base model for the following reasons. Among the open-weight models we evaluated, GPT-OSS-120B offered the best balance of three critical factors: inference speed, output quality, and solution conciseness. We evaluated several alternative base models including Qwen-3 (30B-A3B, 32B), Qwen-3.5 (9B, 35B, 122B), Nemotron 3 (120B), and others, but each presented tradeoffs unfavorable for the competition constraints.
\nQwen models demonstrated strong reasoning capabilities but generated substantially longer outputs, consuming tokens that were critical under the 5-hour time limit. Nemotron showed promise in terms of output quality but suffered from inference speed limitations that made it difficult to fit multiple parallel generations on an H100 within the time budget. GPT-OSS-120B, by contrast, produced reasonably high-quality solutions with concise lengths and could be efficiently parallelized — we achieved 8 parallel inference sequences on a single H100 GPU using vLLM without memory overflow, enabling effective batching of multiple problems simultaneously.
\n\n \n
\n Figure 1: pass@1 accuracy comparison across models and benchmarks (Qwen variants only)\n
\n \n
\n Figure 2: pass@1 accuracy comparison across models and benchmarks (GPT-OSS-120B vs Qwen variants)\n
\n \n
\n Figure 3: avg@16, maj@16, and best@16 accuracy by benchmark for Qwen3.5-35B-A3B\n
\n \n
\n Figure 4: Chain-of-thought length distribution across Qwen model variants\n
\n \n
\n Figure 5: Chain-of-thought length distribution — GPT-OSS-120B generates substantially shorter solutions than Qwen variants\n
\n \n
\n Figure 6: Inference throughput comparison across Qwen variants\n
\n \n
\n Figure 7: Inference throughput comparison — GPT-OSS-120B achieves 5,063 tok/s vs 1,699 tok/s for Qwen3.5-122B\n
\n \n
\n Figure 8: Nemotron vs GPT-OSS-120B throughput\n
The choice of GPT-OSS-120B was validated across all subsequent training phases. When we later experimented with distillation and compression, we consistently returned to this base model as our starting point, and no alternative consistently outperformed it on our Pareto frontier of accuracy versus inference latency. Additionally, adding Python tool calling further improved accuracy.
\nBase Model Specifications:
\nTo improve inference efficiency without sacrificing accuracy, we investigated on-policy distillation onto REAP-compressed model variants. REAP (Rank Estimation Aggregation Pruning) is a layer-wise weight pruning technique that removes less-important model parameters while maintaining or even slightly improving performance through weight redistribution.
\nWe applied REAP pruning to GPT-OSS-120B at several compression levels, retaining different percentages of original model weights: 97%, 95%, 93%, 90%, 75%, 50%.
\nThe pruning process works by ranking weights across each transformer layer and removing those with smallest magnitude. Importantly, REAP applies per-layer pruning ratios, allowing flexibility to retain more weights in critical early layers and prune more heavily in later layers.
\n\n \n
\n Figure 9: Accuracies across REAP compression levels — high compression (REAP-50, REAP-75) causes significant accuracy degradation\n
For the most promising REAP variants (REAP-95, REAP-75), we conducted on-policy distillation using the trained full model as the teacher. Since the directly pruned model had some loss in terms of accuracy, the size and speeds could allow for more sophisticated scaffolds.
\nDistillation Hyperparameters:
\n\n \n
\n Figure 10: On-policy distillation eval accuracy and KL divergence across training steps — accuracy is highly noisy with no clear improvement trend\n
However, the OPD was not very successful — accuracy was very noisy across training steps. Additionally, limited evaluations showed that the REAP models were not necessarily better in terms of throughput and speed, as shown in the comparison below:
\n| \n | reap95 | \nbase | \n
|---|---|---|
| Wall time | \n1,667s (27.8 min) | \n1,242s (20.7 min) | \n
| Total tokens | \n1,686,623 | \n1,209,490 | \n
| Throughput | \n1,012 tok/s | \n974 tok/s | \n
| Attempts run | \n86 | \n74 | \n
| Completed (not deadline) | \n70 | \n65 | \n
| Deadline-killed | \n16 | \n9 | \n
| Python calls | \n1,751 | \n1,005 | \n
| KV cache | \n316K tokens | \n237K tokens | \n
We conducted exploratory reinforcement learning (RL) experiments to directly optimize model generation length while maintaining correctness. The main technique explored was TOGGLE, introduced in Kimi K2.5 [6]. However, rollout rewards were too noisy to produce a useful training signal, and none of these experiments showed meaningful improvement.
\n\n \n
\n Figure 11: RL rollout total and response lengths across training steps — highly noisy with no clear downward trend\n
\n \n
\n Figure 12: RL training reward curves — noisy signal prevented meaningful optimization\n
After exploring multiple training approaches, we concluded that further fine-tuning was not the most effective use of our resources. GPT-OSS-120B was already optimized for extended reasoning and demonstrated strong baseline accuracy out of the box. Without sufficient compute and time to run the large number of training iterations required to meaningfully shift model behavior, fine-tuning experiments risked being inconclusive or introducing regressions. We therefore shifted our focus toward inference-time improvements: scaffolding, answer selection, and throughput optimization, where we could iterate more quickly and measure impact more reliably.
\nBeyond improving the base model itself, we invested substantial effort into scaffolding strategies. Inspired by GenSelect from AIMO-2 [3], we hypothesized that smarter answer selection could unlock accuracy that majority voting leaves on the table. This was strongly motivated by the distribution of correct rollouts across our evaluation set: a large fraction of problems had the correct answer appearing in only 1–7 out of 16 rollouts, meaning the model demonstrably could solve these problems, but majority voting would consistently select the wrong answer. The gap between best@16 (87.98%) and maj@16 (80.60%) on high-reasoning traces confirmed this — nearly 8% of problems were solvable in principle but lost at aggregation. Scaffolding strategies that could identify the correct rollout among a mixed set therefore represented a high-potential direction for improvement.
\n\n \n
\n Figure 13: Distribution of correct rollouts per problem out of 16 — many problems have the correct answer in only a minority of rollouts, motivating smarter answer selection\n
We established majority voting as our primary baseline for answer aggregation. For each problem, we generate K independent reasoning attempts and extract the final numerical answer from each. The simplest scaffold is to merely do a majority vote over all the answers. Additionally, we evaluated scaling the number of rollouts under an early-stopping majority strategy, which we denote \"high_match(d, n)\" below: generate up to d rollouts and stop once n agree. Our baseline configuration of (d=8, n=4) at max sequence length 65,536 tokens completed in 167 seconds. Increasing to (d=11, n=4) with a reduced max sequence length of 52,428 tokens achieved 82.23% accuracy in 163.5 seconds — a difference of under 0.6%, corresponding to roughly 2 problems out of 366. We concluded that scaling raw rollout count yields marginal returns, and that judge-based approaches represent a more promising direction for improving accuracy within a fixed compute budget.
\n\n \n
\n Figure 14: Pareto frontier of early-stopping strategies (accuracy vs wall time) — the recommended (d=8, n=4) configuration sits on the frontier with minimal gains from more rollouts\n
With our initial scaffold, around one in every 20 rollouts would fail to return an answer for one of the following reasons: \\boxed was never output, the boxed content included text, or the rollout hit the token limit. In many of these rollouts, the model had already settled on the correct answer in its chain-of-thought, meaning we were losing valuable information every time we failed to extract an answer.
To address this issue, we introduced a recovery step triggered every time the boxed output contained non-digit characters or when the rollout hit the token limit without boxing an answer. This step employed prefilling, where we appended the token encoding of the string “I must respond with a nonnegative integer answer; I will remove all non-digit characters from my answer. Therefore, my final integer answer is \\\\boxed{“ to the assistant’s last generation turn before sending it back to vLLM to generate the completion. By essentially speaking for the model with these specific formatting instructions, we were able to recover integer answers in 99% of cases, with a substantial fraction matching the correct answer.
Our experiments with prefilling revealed unexpected results:
\n\n \n
\n Figure 15: pass@1 rollout accuracies with and without prefilling\n
\n
\n \n
\n Figure 16: maj@8 accuracy with and without prefilling — prefilling degrades majority vote on harder problems\n
\n On Full Rollout Set: Upon aggregating all rollout answers, we found that rollouts using prefill had a lower accuracy overall, likely because rollouts that failed to naturally box an answer were less promising overall. Across 20,000 rollouts on the IMO Answerbench dataset, we found that prefilled answers had an average accuracy of 58.5% compared to the non-prefilled average of 78.4%.
\nMajority Voting Degradation: When we analyzed the effect of introducing prefilled answers into our majority voting selection process, we found that our accuracy did not improve. We even found certain problems where the rollouts that used prefilling all produced the same incorrect answer, which was directly harmful.
\nThus, we abandoned the prefilling strategy as it did not appear to boost the accuracy of majority voting.
\nZero-Overhead Introspection For Adaptive Test-time Compute (ZIP-RC) [5] is a technique for selecting answers based on a fine-tuned model to predict the probability distribution of remaining tokens and correctness (reward) in place of unused tokens. There is zero overhead because one can directly extract the distribution while getting each token.
\nThis involved extracting hidden states and training a head to predict the joint distribution of remaining tokens and correctness reward. We found this approach either overfit on the limited available training data or collapsed to predicting a constant distribution. We subsequently simplified the objective to directly predicting completion and reward with separate heads. Additionally, since the last hidden layer is more suited to predicting the next token, like EAGLE-3 [1], we extracted hidden state layers 2, num_layers // 2, and num_layers - 3, denoted eagle_hidden_0, eagle_hidden_1, and eagle_hidden_2. While completion proportion proved reasonably learnable, reward prediction remained too noisy to be reliable. Given this noise, the downstream scaffolding strategy of optimizing a function trading off predicted reward against remaining tokens did not yield meaningful improvements over the baseline, and we abandoned this direction.
\n \n
\n Figure 18: ZIP-RC Lite prediction curves across hidden layers — reward prediction (blue) is highly noisy while completion proportion (green) is more learnable\n
We implemented a full version of ZIP-RC with the following procedure:
\n\n \n
\n Figure 19: Full ZIP-RC training — predictions remain extremely noisy and fail to converge, consistent with insufficient training data\n
As with the partial variant, the resulting predictions were extremely noisy and inaccurate, likely due to insufficient training data — the original paper trained on 100K rollouts of simple math data, whereas our data was considerably more limited. We discontinued both the full and partial ZIP-RC directions in favor of simpler, more reliable inference-time approaches.
\nTo improve upon majority voting, we explored PaCoRe: Learning to Scale Test-Time Compute with Parallel Coordinated Reasoning [4]. Our implementation operated in two stages: in Part A, the model generates 8 parallel rollouts per problem and skips questions where all rollouts agree; in Part B, for problems with diverse outcomes, a \"PaCoRe prompt\" is constructed showing the model a mix of its own correct and incorrect prior attempts as context, then 8 new rollouts are generated conditioned on those references — the intuition being that exposure to diverse solutions helps the model self-correct through consensus. Once sufficient correct Part B traces accumulated, the pipeline paused to run supervised fine-tuning via Unsloth LoRA on the correct traces, saved a checkpoint, and resumed generation in a continuous improvement loop.
\n\n \n
\n Figure 20: PaCoRe SFT training loss across rounds — loss decreases consistently across all rounds\n
\n \n
\n Figure 21: PaCoRe eval metrics across rounds — Part A and Part B accuracy remain flat despite decreasing loss, showing no meaningful improvement over majority voting\n
While SFT loss decreased as expected during training, evaluation results showed no meaningful improvement over standard majority voting. We attribute this to the model already being strong enough that the PaCoRe coordination signal provided little additional useful learning signal within our compute budget, and discontinued this direction.
\nGenSelect [3] shares the same infrastructure as PaCoRe but differs in what it trains the model to do in Part B. Part A is identical — 8 parallel rollouts per problem, graded against ground truth, skipping problems where all answers agree. In Part B, rather than synthesizing a new solution from prior attempts, the model acts as a comparative judge: given 4–8 reference solutions from Part A, it must select the best one rather than re-derive independently. The GenSelect prompt explicitly forbids re-solving and enforces voting discipline — the model must either pick the majority answer or cite concrete errors in every majority solution to justify overriding it. The iterative SFT loop is otherwise identical to PaCoRe: once 512 correct Part B traces accumulate, LoRA fine-tuning is run on the correct traces, the checkpoint is reloaded, and generation resumes.
\n\n \n
\n Figure 22: GenSelect SFT training loss across rounds — loss decreases across all three rounds\n
\n \n
\n Figure 23: GenSelect eval metrics across rounds — avg@8, maj@8, and best@8 remain flat and noisy across rounds despite training loss reduction\n
Unfortunately, GenSelect suffered the same fate as PaCoRe — while SFT loss decreased during training, we observed no meaningful accuracy improvement over majority voting at evaluation time. We suspect the base model was already capable enough that neither coordination signal provided sufficient learning gradient within our compute budget.
\nReasking is a simple technique where, after generating an initial reasoning attempt, we prompt the model to reconsider its answer while retaining the full context of its previous response. The hypothesis: allowing the model a second chance to reason might catch errors or find better solutions without starting from scratch.
\n\n \n
\n Figure 24: Phase 1 and Phase 2 reasking inference times — Phase 2 takes 35% of them time Phase 1 takes \n
In practice, we observed a slight accuracy improvement of 2.3%: the model would occasionally, only 3.8% of the time, correct a wrong answer on the second attempt, but this came at the cost of slightly longer inference time. Critically, during a prompt sweep for the Phase 2 reasking prompt, the correction was not always in the right direction — the model would sometimes change a correct answer to an incorrect one, making the net gain marginal. Ultimately, we did not pursue reasking further given the unfavorable accuracy-to-latency.
\nOur final submission combined the standard solver scaffold with a judge pass for answer selection on ambivalent problems. The solver ran 8 parallel attempts per problem, with attempts 0–3 at temperature 0.3 and attempts 4–7 at temperature 1.0 to balance speed and diversity. Each attempt used tool-calling with a persistent Jupyter sandbox for Python execution, running up to 128 turns with strict integer-only answer extraction from \\boxed{}.
\nAfter all 8 attempts completed, answer selection proceeded as follows: if 7 or more attempts agreed, the majority answer was used directly (very rare); otherwise, a judge pass was invoked. The judge received all valid rollouts — stripped of code blocks but preserving reasoning — and was instructed to analyze the problem, assess each solution's reasoning quality, and check for specific failure modes including missing cases, off-by-one errors, wrong model selection, weak bounds, parity errors, one-directional proofs, and overcounting. The judge could also run Python verification code if needed. Critically, the judge was constrained to select only from among the answers already proposed by the 8 rollouts and could not introduce a new answer. If the judge failed or timed out, the system fell back to majority vote.
\nOn ambivalent problems, the problems where majority voting over 8 rollouts yields an incorrect answer, the judge recovered between 33% and 54% of errors across 69 ambivalent problems, depending on the prompt used. However, more capable judge prompts introduced substantially longer inference time, so our final submission used a shorter prompt that balanced recovery rate against wall-clock budget. The full judge prompt is included in the appendix.
\nBeyond the above major scaffolding approaches, we explored several smaller-scale experiments with prompt variants and sampling parameters.
\nFull Prompt Sweeping: We prepended instructions like \"Avoid these common mistakes: [list of typical errors]\", compiling a list of frequent errors our model made by problem type. Attempts to have the model avoid these specific mistakes via prompting showed little effect, likely because the model was already aware of these issues but made mistakes due to reasoning errors rather than knowledge gaps.
\nWe also tried prompts like \"BIFURCATION: Identify three different mathematical approaches with their probabilities of success, then choose the one that is most likely to lead to the correct answer…\", and tried having different prompts across the 8 rollouts. However, there was a lot of noise in these results, and they suggest the original baseline prompt may be the best.
\n\n \n
\n Figure 25: Prompt sweep results across 10 variants — accuracy, wall clock time, and response length are largely consistent across prompts, with no clear winner\n
Chinese Thinking Prompts: We also hypothesized that prompting the model in Chinese might leverage different reasoning patterns or potentially generate less tokens (since Chinese language is more compact). We tested the exact same baseline prompt translated to Chinese. However, GPT-OSS-120B would automatically translate the problem back to English and think in English, even when the system prompt specifically said to think in Chinese and the problem was presented in Chinese.
\nWe systematically swept sampling hyperparameters to optimize the accuracy-compute tradeoff:
\nParameters Swept:
\nKey Findings:
\nAcross our scaffolding exploration, we investigated a broad range of techniques — from lightweight prompt variants and sampling sweeps to sophisticated generative selection methods like PaCoRe, GenSelect, and ZIP-RC. The majority of these approaches either failed to meaningfully improve over the majority voting baseline or introduced latency costs that outweighed their accuracy gains.
\nThe clearest findings were largely negative: scaling raw rollout count yields diminishing returns, prefilling degraded majority vote signal, reasking introduced bidirectional errors, and SFT-based coordination methods (PaCoRe, GenSelect) did not produce accuracy gains despite successful loss reduction. ZIP-RC, while theoretically appealing, was fundamentally bottlenecked by insufficient training data.
\nThe most effective scaffolding improvement came from our judge-corrected answer selection strategy. Rather than attempting to improve individual rollout quality, the judge operated post-hoc on the set of candidate answers, selectively overriding majority vote only on ambivalent problems where rollouts disagreed. It is important to note the additional inference cost the judge introduces. This targeted intervention, combined with temperature 0.3 for inference efficiency, constituted our final submission strategy. On AstralBench, the judge approach improved over majority voting from 15/50 to 17/50, demonstrating that even a simple judge prompt applied selectively to uncertain cases can meaningfully recover correct answers that majority voting would otherwise miss.
\nAchieving competitive performance required careful optimization of the inference pipeline to fit within Kaggle's time and memory constraints. This section describes the key inference decisions that shaped our final submission.
\nWe served GPT-OSS-120B using vLLM [7] with fp8 KV cache quantization and prefix caching enabled. The vLLM server was configured with a single GPU (tensor-parallel-size=1), a context window of 65,536 tokens, and up to 8 concurrent sequences. Fitting the model alongside EAGLE3 and a 65k context window required careful tuning: setting VLLM_FLOAT32_MATMUL_PRECISION=medium and VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1 were necessary to avoid OOM, leaving only ~175 MiB of GPU memory headroom at peak. SGLang was evaluated as an alternative serving framework but was ruled out due to a kernel crash on GPT-OSS-120B caused by an unsupported hidden size.
We integrated EAGLE3 speculative decoding [1] using the draft model wenliang1990/gpt-oss-120b-eagle3-aimo3. We swept over speculative token counts (nst = 3, 4, 5, 6) across multiple AstralBench runs and found nst = 4 consistently performed best, achieving up to 17/50 on AstralBench in approximately 5–5.5 hours. Increasing to nst = 6 did not improve accuracy. Measured acceptance rates for nst = 4 were 0.724, 0.505, 0.357, and 0.252 at successive token positions — strong at the first speculative token but diminishing at depth, consistent with expectations. The Nvidia speculator was ruled out early due to its use of the full vocabulary size; P-EAGLE performed well on reap90 but OOM'd on the base model.
We also evaluated SpecForge for training a custom speculator on our rollout data. However, disk space constraints on our compute cluster limited training to sequence lengths ≤ 4096, and the finetuned speculator did not produce a measurable speedup. We hypothesize this was partly because SpecForge trains all positions and all speculated positions simultaneously, which may be suboptimal.
\nA key finding was that temperature 0.3 produced approximately 30% faster inference than temperature 1.0 with no accuracy loss, as lower temperature generates shorter solutions. Our final submission used temperature 0.3 for attempts 0–3 and temperature 1.0 for attempts 4–7 to balance speed with diversity. A full sweep of top-p, min-p, and repetition penalty found these parameters had negligible effect on accuracy, and we used defaults for all three.
\nOur final submission ran the base GPT-OSS-120B model with EAGLE3 (nst=4), temperature 0.3/1.0 mixed, fp8 KV cache, prefix caching, and 65k context. The base model completed 10 reference problems in 28 minutes at maj@12 with temperature 0.3 — the clearest timing baseline we measured.
\nThe most impactful decisions in our pipeline were largely about restraint: choosing not to fine-tune a model that was already strong, and focusing instead on inference efficiency and answer selection. Temperature 0.3 alone provided a 30% inference speedup with no accuracy cost, effectively giving us more rollouts within the time budget. EAGLE3 speculative decoding with nst=4 provided additional throughput gains and was the most reliable engineering improvement we found. The judge-corrected answer selection strategy was our most effective algorithmic contribution, recovering approximately 33% of problems where majority voting failed by applying a targeted second-pass verification only to ambivalent cases.
\nData quality also proved critical. Early in the project, we discovered systematic errors in public IMO benchmark datasets — incorrect answer keys, non-integer representations, and contamination — that were silently corrupting our evaluation signal. Cleaning these datasets was unglamorous but essential: without trustworthy evaluation, it is impossible to distinguish real improvements from noise, and several early experiments that appeared promising were later found to be artifacts of the buggy benchmarks.
\nThe most consistent theme in our negative results was that GPT-OSS-120B is already operating near the ceiling of what fine-tuning can easily shift. DPO training reduced loss but did not translate to accuracy gains. PaCoRe and GenSelect both showed healthy SFT loss curves but flat accuracy, suggesting the model was already capable of the coordination behavior being trained in. ZIP-RC was theoretically appealing but fundamentally data-limited — the original paper trained on 100K rollouts of simple arithmetic, and our more limited corpus was insufficient to learn reliable reward predictions. REAP model compression, while lossless at high retention rates, did not provide meaningful throughput gains and added engineering complexity without commensurate benefit.
\nPrompt engineering was similarly disappointing. Sweeping over ten structured prompt variants, common-mistake lists, bifurcation prompts, and Chinese-language prompts produced mostly noise, with the original baseline prompt performing as well or better than any alternative. This suggests the model's errors are primarily reasoning failures rather than knowledge or instruction-following failures — a finding with implications for future work.
\nOur approach is fundamentally constrained by the five-hour time limit, which caps the number of rollouts we can generate and limits the complexity of scaffolding we can apply. With more compute, judge selection based methods, as well as those that train the model to identify its best solution are likely to be more fruitful, as suggested by results from prior AIMO competitions [3]. Our SpecForge experiments were similarly hampered by disk and compute constraints that prevented training on long sequences.
\nThe judge strategy, while effective, adds significant inference overhead and its prompt was only lightly tuned. Early experiments suggested that even the first judge prompt tried recovered up to 54% of ambivalent problems, and that substantially better prompts are likely achievable with more iteration. Fine-tuning the judge specifically for mathematical verification is another promising direction not explored here.
\nWe presented our solution to AIMO-3, through a combination of careful data curation, inference optimization, and judge-corrected answer selection. Our central finding is that for a model as capable as GPT-OSS-120B, the highest-leverage improvements come not from modifying the model weights but from engineering the inference pipeline and answer aggregation strategy. Temperature tuning, speculative decoding, and targeted judge verification collectively allowed us to solve more problems within the competition's strict time constraints than any of our training experiments achieved.
\nWe also contribute a set of cleaned evaluation benchmarks and publicly released rollout datasets that we hope will support future research on olympiad-level mathematical reasoning. Our extensive negative results — across RL, distillation, ZIP-RC, PaCoRe, GenSelect, and prompt engineering — provide a map of approaches that do not easily improve over a strong baseline, which we believe is as valuable to the community as our positive findings.
\n[1] Hu et al. \"EAGLE-3: Enabling Lossless Acceleration of LLM Decoding via Dynamic Draft Model Updating.\" arXiv:2503.01840, 2025. https://arxiv.org/abs/2503.01840
\n[2] Leviathan et al. \"Fast Inference from Transformers via Speculative Decoding.\" ICML 2023. https://arxiv.org/abs/2211.17192
\n[3] Moshkov et al. \"AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset.\" arXiv:2504.16891, 2025. https://arxiv.org/abs/2504.16891
\n[4] Hu et al. \"PaCoRe: Learning to Scale Test-Time Compute with Parallel Coordinated Reasoning.\" arXiv:2601.05593. https://arxiv.org/abs/2601.05593
\n[5] Manvi et al. \"Zero-Overhead Introspection for Adaptive Test-Time Compute.\" arXiv:2512.01457, 2025. https://arxiv.org/abs/2512.01457
\n[6] Kimi Team. \"Kimi K2.5: Visual Agentic Intelligence.\" arXiv:2602.02276, 2026. https://arxiv.org/abs/2602.02276
\n[7] Kwon et al. \"Efficient Memory Management for Large Language Model Serving with PagedAttention.\" SOSP 2023. https://arxiv.org/abs/2309.06180
\n[8] aimosprite/high-reasoning-eval. https://huggingface.co/datasets/aimosprite/high-reasoning-eval
\n[9] aimosprite/medium-reasoning-eval. https://huggingface.co/datasets/aimosprite/medium-reasoning-eval
\n[10] aimosprite/low-reasoning-eval. https://huggingface.co/datasets/aimosprite/low-reasoning-eval
\n[11] aimosprite/imo_answerbench_clean_notebooks. https://huggingface.co/datasets/aimosprite/imo_answerbench_clean_notebooks
\n[12] aimosprite/training-set. https://huggingface.co/datasets/aimosprite/training-set
\nclass CFG:\n system_prompt = (\n 'You are a world-class International Mathematical Olympiad (IMO) competitor. '\n 'The final answer must be a non-negative integer between 0 and 99999. '\n 'You must place the final integer answer inside \\\\boxed{}.'\n )\n\n tool_prompt = (\n 'Use this tool to execute Python code. '\n 'The environment is a stateful Jupyter notebook. '\n 'You must use print() to output results.'\n )\n\n preference_prompt = (\n 'Use `math`, `numpy`, `sympy`, `itertools` and `collections` to solve the problem.'\n )\n\n served_model_name = 'gpt-oss'\n model_path = '/kaggle/input/gpt-oss-120b/transformers/default/1'\n\n kv_cache_dtype = 'fp8_e4m3'\n dtype = 'auto'\n\n high_problem_timeout = 900\n base_problem_timeout = 300\n notebook_limit = 17520\n server_timeout = 180\n session_timeout = 960\n jupyter_timeout = 10\n sandbox_timeout = 5\n stream_interval = 200\n\n context_tokens = 65536\n search_tokens = 1024\n buffer_tokens = 512\n batch_size = 8\n early_stop = 7 # disabled — always run all attempts\n attempts = 8\n workers = 16\n turns = 128\n seed = 42\n\n gpu_memory_utilization = 0.96\n temperature = 1.0\n\n # Judge configuration\n judge_temperature = 0.0\n judge_max_tokens = 65536\n judge_timeout = 300\n judge_system_prompt = (\n 'You are an expert mathematical verifier.\\n'\n 'You have access to a Python environment.\\n'\n 'Use ```python code blocks for code.\\n'\n 'Stop immediately after the closing ``` and wait for execution output.\\n'\n 'Never simulate code output.\\n'\n 'After seeing execution output, continue your reasoning normally.\\n'\n 'Keep your final response in the format requested by the user prompt.\\n'\n )\nYou are an expert mathematician judging {num_solutions}\ncandidate solutions (numbered 0 through {max_idx}) to an\nIMO-level math problem.\n\n### Task ###\n\nPick the most likely correct answer. The majority answer is\noften wrong in specific, predictable ways. Your job is to\ncatch those errors.\n\n### Instructions ###\n\n**1. Read the problem yourself first.**\nIdentify: what is given, what constraints apply, what is\nbeing asked, and what form the answer should take.\n\n**2. Assess each solution's reasoning.**\nQuote the solver's interpretation (1-2 sentences). Mark\nMISUNDERSTOOD if it answers a different question. When\nsolutions disagree, the minority may have found something\nthe majority missed.\n\n**3. Common solver failure modes (check for these, but don't\n limit yourself — there are many other possible mistakes):**\n\n * **Missing cases.** Solver finds one solution family and\n stops. Check: are there other valid constructions,\n additional solution branches, or cases the solver\n declared impossible without proof?\n\n * **Off-by-one.** When the majority answer differs from a\n minority answer by exactly 1, double-check which is\n right — the majority is often the one that's off.\n\n * **Wrong model.** Solver maps the problem to the wrong\n combinatorial structure. Check: ordered vs unordered,\n directed vs undirected, walks vs trails, L1 vs L∞\n metric, integer vs rational parameters.\n\n * **Premature impossibility.** Solver claims something is\n impossible via a hand-wavy argument. Check: did it\n actually verify small cases? A construction may exist.\n\n * **Weak bound assumed tight.** Solver proves an inequality\n but never constructs a matching example. A tighter bound\n from another solution may be correct.\n\n * **Parity / divisibility constraint ignored.** Check if\n the solver missed that some quantity must be even, or\n coprime, or satisfy a mod condition.\n\n * **One-directional proof.** Solver shows A implies B but\n the problem needs B implies A too (or vice versa).\n Especially common with polynomial/functional equations.\n\n * **Overcounting / undercounting.** Wrong sieve, double-\n counted configurations, or missed a symmetry factor.\n\n**4. Verification clues:**\n\n * **Strictly better answer wins.** If one solution finds a\n smaller minimum (with construction) or larger maximum,\n and proves it, prefer it.\n\n * **Counterexamples are heavy evidence.** If any solution\n explicitly demonstrates another answer violates a\n constraint, that's strong.\n\n * **Back-substitute.** Does the answer satisfy ALL original\n conditions?\n\n**5. Python verification.**\nYou have a Python tool. To use it, address recipient=python\nwith your code. Do NOT write ``to=python code...`` as text\nin your analysis — that does not execute.\n\n**6. Rules:**\n * Do NOT re-solve from scratch. Compare and verify.\n * Do NOT be swayed by majority count alone.\n * A confident wrong answer repeated 7 times is still wrong.\n\n**7. Output Format**\n\n* **a.** `<problem_statement_analysis>` 2-3 sentences on what\n the problem asks. `</problem_statement_analysis>`\n\n* **b.** `<per_solution_assessment>` For each solution: quote\n its interpretation, check understanding (UNDERSTOOD /\n MISUNDERSTOOD), note any reasoning concerns.\n `</per_solution_assessment>`\n\n* **c.** Understanding array (1=understood, 0=misunderstood):\n `<understanding>[1, 0, 1, 1, 0, 0, 1, 0]</understanding>`\n\n* **d.** Your best answer:\n `<suggested_answer>INTEGER</suggested_answer>`\n or `<suggested_answer>UNKNOWN</suggested_answer>`\n\n========================================================================\n### Problem ###\n\n{problem}\n\n========================================================================\n### Solutions ###\n\n{solutions}\n| Partition | \ntemp_0.6 | \ntemp_1.0 | \ntemp_1.2 | \n
|---|---|---|---|
| Part 0 | \n37/61 (60.7%) | \n38/61 (62.3%) | \n40/61 (65.6%) | \n
| Part 1 | \n50/61 (82.0%) | \n51/61 (83.6%) | \n51/61 (83.6%) | \n
| Part 2 | \n47/61 (77.0%) | \n46/61 (75.4%) | \n44/59 (74.6%) | \n
| Part 3 | \n53/61 (86.9%) | \n53/61 (86.9%) | \n52/61 (85.2%) | \n
| TOTAL | \n187/244 (76.6%) | \n188/244 (77.0%) | \n187/242 (77.3%) | \n
| Config | \navg@8 | \nbest@8 | \nmaj@8 | \nAvg Tokens | \nAvg Wall Time (s) | \nToken Limit Hits | \n
|---|---|---|---|---|---|---|
| top_p=0.80, min_p=0.00 | \n0.792 | \n0.893 | \n0.844 | \n20,669 | \n347.6 | \n43 | \n
| top_p=0.90, min_p=0.00 | \n0.782 | \n0.887 | \n0.833 | \n19,022 | \n312.1 | \n19 | \n
| top_p=0.90, min_p=0.02 | \n0.792 | \n0.901 | \n0.843 | \n20,295 | \n337.0 | \n31 | \n
| top_p=1.00, min_p=0.00 | \n0.778 | \n0.884 | \n0.817 | \n20,643 | \n335.5 | \n41 | \n
| top_p=1.00, min_p=0.05 | \n0.789 | \n0.895 | \n0.848 | \n20,547 | \n330.5 | \n32 | \n
| top_p=1.00, min_p=0.10 (a) | \n0.785 | \n0.895 | \n0.836 | \n20,248 | \n334.7 | \n35 | \n
| top_p=1.00, min_p=0.10 (b) | \n0.788 | \n0.893 | \n0.831 | \n19,126 | \n302.8 | \n29 | \n
AI Mathematical Olympiad - Progress Prize 3
\n\nGimran Abdullin · Rohin Garg · Mohammad Hafez · Jurgis Kemeklis · Marina Lin · Debojoti Das Soumya
\nJason Youm · Melody Yu · Brian Zhang · David Zhang · Sophia Zhang · William Zhao · Lerchen Zhong\n
\n \n
\n Figure 1: pass@1 accuracy comparison across models and benchmarks (Qwen variants only)\n
\n \n
\n Figure 2: pass@1 accuracy comparison across models and benchmarks (GPT-OSS-120B vs Qwen variants)\n
\n \n
\n Figure 3: avg@16, maj@16, and best@16 accuracy by benchmark for Qwen3.5-35B-A3B\n
\n \n
\n Figure 4: Chain-of-thought length distribution across Qwen model variants\n
\n \n
\n Figure 5: Chain-of-thought length distribution — GPT-OSS-120B generates substantially shorter solutions than Qwen variants\n
\n \n
\n Figure 6: Inference throughput comparison across Qwen variants\n
\n \n
\n Figure 7: Inference throughput comparison — GPT-OSS-120B achieves 5,063 tok/s vs 1,699 tok/s for Qwen3.5-122B\n
\n \n
\n Figure 8: Nemotron vs GPT-OSS-120B throughput\n
\n \n
\n Figure 9: Accuracies across REAP compression levels — high compression (REAP-50, REAP-75) causes significant accuracy degradation\n
\n \n
\n Figure 10: On-policy distillation eval accuracy and KL divergence across training steps — accuracy is highly noisy with no clear improvement trend\n
\n \n
\n Figure 11: RL rollout total and response lengths across training steps — highly noisy with no clear downward trend\n
\n \n
\n Figure 12: RL training reward curves — noisy signal prevented meaningful optimization\n
\n \n
\n Figure 13: Distribution of correct rollouts per problem out of 16 — many problems have the correct answer in only a minority of rollouts, motivating smarter answer selection\n
\n \n
\n Figure 14: Pareto frontier of early-stopping strategies (accuracy vs wall time) — the recommended (d=8, n=4) configuration sits on the frontier with minimal gains from more rollouts\n
\n \n
\n Figure 15: pass@1 rollout accuracies with and without prefilling\n
\n
\n \n
\n Figure 16: maj@8 accuracy with and without prefilling — prefilling degrades majority vote on harder problems\n
\n \n \n
\n Figure 18: ZIP-RC Lite prediction curves across hidden layers — reward prediction (blue) is highly noisy while completion proportion (green) is more learnable\n
\n \n
\n Figure 19: Full ZIP-RC training — predictions remain extremely noisy and fail to converge, consistent with insufficient training data\n
\n \n
\n Figure 20: PaCoRe SFT training loss across rounds — loss decreases consistently across all rounds\n
\n \n
\n Figure 21: PaCoRe eval metrics across rounds — Part A and Part B accuracy remain flat despite decreasing loss, showing no meaningful improvement over majority voting\n
\n \n
\n Figure 22: GenSelect SFT training loss across rounds — loss decreases across all three rounds\n
\n \n
\n Figure 23: GenSelect eval metrics across rounds — avg@8, maj@8, and best@8 remain flat and noisy across rounds despite training loss reduction\n
\n \n
\n Figure 24: Phase 1 and Phase 2 reasking inference times — Phase 2 takes 35% of them time Phase 1 takes \n
\n \n
\n Figure 25: Prompt sweep results across 10 variants — accuracy, wall clock time, and response length are largely consistent across prompts, with no clear winner\n
Following up on the questions from @philipvonderlind, here are some additional questions to help us assess your submission:
\nLook forward to reading your responses on these.
", "rawMarkdown": "Following up on the questions from @philipvonderlind, here are some additional questions to help us assess your submission:\n\n1. **Judge accuracy vs time tradeoff:** Could you share more quantitative data on the accuracy-vs-time tradeoff across the judge variants you tested (e.g. recovery rate vs added wall-clock per problem)? That would help us understand which design point was actually shipped and what the marginal cost was of moving to higher-recovery variants.\n2. **EAGLE3 throughput gain:** You report some acceptance rates at successive token positions, but I couldn't see an end-to-end throughput delta in the writeup. What's the token/sec or wall-clock improvement on a representative run, compared to the same setup without speculative decoding?\n\nLook forward to reading your responses on these." }, { "id": 3457286, "postDate": "2026-05-13T13:35:31.793Z", "content": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.
\nYour work is already quite extensive. However, I think there are a few open questions that could make it even stronger:
\nDataset Creation: Since you create your own dataset, it would be helpful to include a diagram showing the different filtering stages and the number of samples discarded at each step. As a reference, you could look at the CrystalMath dataset from the Math Corpus Prize. Additionally, did you run any ablations against a human baseline to evaluate the impact of the filtering stages?
Judge Implementation: The judge system appears to be a crucial component of your submission. It would be useful to include plots and a more detailed explanation of how each judge works, as well as the conditions under which each judge is deployed.
Results & Plots: Some results for the Nemotron models seem to be missing from the plots. If available, please add the corresponding completion results, since the model is mentioned in some sections but not consistently covered elsewhere. Furthermore, the results for some of the REAP compression levels also seem to be missing from the 'REAP Compression Overview' (for the 90% to 97% levels).
RL Setup: Did you also experiment with other RL techniques, such as the commonly used GRPO or even just preference optimization (ORPO, …)? What motivated the choice of TOGGLE instead?
Looking forward to your response :)
", "rawMarkdown": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the [updated discussion post](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/689703).\n\nYour work is already quite extensive. However, I think there are a few open questions that could make it even stronger:\n\n* **Dataset Creation**: Since you create your own dataset, it would be helpful to include a diagram showing the different filtering stages and the number of samples discarded at each step. As a reference, you could look at the [CrystalMath dataset](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/672592) from the Math Corpus Prize. Additionally, did you run any ablations against a human baseline to evaluate the impact of the filtering stages?\n\n* **Judge Implementation**: The judge system appears to be a crucial component of your submission. It would be useful to include plots and a more detailed explanation of how each judge works, as well as the conditions under which each judge is deployed.\n\n* **Results & Plots**: Some results for the Nemotron models seem to be missing from the plots. If available, please add the corresponding completion results, since the model is mentioned in some sections but not consistently covered elsewhere. Furthermore, the results for some of the REAP compression levels also seem to be missing from the 'REAP Compression Overview' (for the 90% to 97% levels).\n\n* **RL Setup**: Did you also experiment with other RL techniques, such as the commonly used GRPO or even just preference optimization (ORPO, ...)? What motivated the choice of TOGGLE instead?\n\n\nLooking forward to your response :)\n\n\n" }, { "id": 3456973, "postDate": "2026-05-12T20:55:22.230Z", "content": "Edited on 5/12 to include private results
", "rawMarkdown": "Edited on 5/12 to include private results" }, { "id": 3447047, "postDate": "2026-04-22T16:24:13Z", "content": "This seems like quite a comprehensive write-up -- I guess you are aiming for a prize, since you didn't make an entry towards the Write-Up Prize? (I don't recall seeing it there; there are still a few hours if you want to submit, FYI.)
", "rawMarkdown": "This seems like quite a comprehensive write-up -- I guess you are aiming for a prize, since you didn't make an entry towards the Write-Up Prize? (I don't recall seeing it there; there are still a few hours if you want to submit, FYI.)" } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T04:23:13Z", "competition_id": 118448, "competition_slug": "ai-mathematical-olympiad-progress-prize-3", "team_id": 14802791, "solution_url": "/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/aimo-3-writeup" }