![\"Figure](\"https://raw.githubusercontent.com/utkarshsharma00/aimo3/main/figures/fig1_leaderboard_trajectory.png\")
Team: flamingice (solo) · Competition: AI Mathematical Olympiad - Progress Prize 3 · Submission for the Writeup Prize
\nWe report a negative-results-dominated investigation of inference-time and training-time interventions for gpt-oss-120b on the AIMO3 competition. Our private-leaderboard submission, final_submission v2, achieved a public-leaderboard mean of 40.78 (σ = 1.20, n = 9 identical runs, best 43) and a private-leaderboard score of 41.5/50 using parallel self-consistency over 8 attempts with a 5-component weighted-entropy voting function and a one-sentence system prompt. We submitted 25 successful runs across 11 distinct notebook configurations over 25 days. With one exception, every architectural addition we attempted - multi-strategy prescriptive prompts, structured 5-stage scaffolding, subject-routed multi-stage arbitration with verifier/GenSelect/adversarial passes, and GRPO LoRA fine-tuning - either fell within run-to-run noise or fell catastrophically below it. The most expensive failure (≈$300 of B200 GPU time on GRPO experiments) is documented with 190 rows of training telemetry and traces to a known Unsloth merged_4bit_forced → vLLM MXFP4 deployment pipeline incompatibility. Our central methodological contribution is the noise floor: with σ = 1.20 across identical runs, any apparent improvement smaller than ≈2.4 points cannot be distinguished from running the same notebook twice. We argue that for inference-constrained math competitions, this noise dominates almost all engineering decisions, and that future participants should establish their own variance baselines before iterating on architecture.
The AIMO3 competition required solving 50 private-set olympiad-level problems with non-negative integer answers in [0, 99999], inside a single Kaggle notebook constrained to ≈5 hours of H100 GPU time. The runtime constraint transforms the optimization target: the goal is not \"maximize per-problem accuracy\" but \"maximize the joint expectation of accuracy and successful completion under a wall-clock cliff.\" This distinction kills nearly every intuition borrowed from offline pass@N experiments.
\ngpt-oss-120b (released in 2025, 117B total parameters with ≈5.1B active per token via mixture-of-experts, native MXFP4 weights) is strong enough on olympiad math that the AIMO3 leaderboard is plausibly bottlenecked by inference orchestration rather than reasoning capacity. Public community notebooks built on gpt-oss-120b were reported to score in the high 30s and low 40s out of 50 with relatively simple self-consistency setups. Against that background, our research question was: starting from a strong base model, where do additional engineering investments actually move the score?
We tested four classes of intervention:
\nEvery one of them either left the score within statistical noise of where we started, or made it dramatically worse. The single intervention that produced a stable mean above the noise floor was a 5-component weighted entropy formula for self-consistency voting, combined with deliberate stripping of every other piece of scaffolding. This paper documents what we tried, why we tried it, and - more importantly - why the things that didn't work didn't work.
\nReproducibility note. All code, prompts, and configurations are reported exactly as run. The full submission history (Appendix C), the GRPO training telemetry CSV (Appendix D), the offline research branch catalog (Appendix H), and the verbatim source of every notebook discussed here are available with this writeup.
\nWe made 25 successful submissions across 11 notebook configurations between competition entry and the April 15 deadline. Figure 1 shows the chronological trajectory.
\n
Three structural features of this trajectory motivate the rest of the paper:
\nThe ramp-and-plateau shape. Scores rise from the high 20s to the high 30s as obvious bugs are removed, then plateau in a narrow band centered around 40 with substantial variance. No single architectural change moves us out of this band on the upside.
\nThe catastrophic dips. Three submissions fall well below the band: GRPO at 13, the adaptive router at 28, and the V11 timeout at 33. Each represents a different failure mode (deployment pipeline corruption, time-budget collapse from per-problem complexity, raw wall-clock overrun). All three are documented in detail below.
\nThe final cluster. The last nine submissions on the right are repeated runs of final_submission v2 - the same notebook, the same seed, the same code - submitted to measure run-to-run variance. They span 39 to 43 with a mean of 40.78 and σ = 1.20.
This last cluster is the load-bearing observation of the entire writeup. We turn to it next.
\nWe submitted final_submission v2 nine times without modification. The scores were:
\n\n39, 40, 43, 40, 42, 40, 41, 41, 41
\n
Mean = 40.778. Median = 41. σ = 1.202. Range = 4 points. 95% CI for the mean = [39.85, 41.70]. Figure 2 shows the distribution.
\n![\"Figure](\"https://raw.githubusercontent.com/utkarshsharma00/aimo3/main/figures/fig2_score_distribution.png\")
This variance has two sources we cannot fully separate. First, the gpt-oss-120b decoder produces non-deterministic output even with a fixed seed, because the parallel attempts are issued asynchronously to a vLLM server whose batching schedule depends on real-time CUDA kernel timing and KV-cache state. Second, Kaggle's grading is deterministic but the scheduling of which problems hit which attempt order is not. Whatever the underlying mechanism, the operationally relevant fact is that nine runs of identical code spanned four points.
\nThe implication for our research methodology is severe and one-directional:
\n\n\nAny single comparison between two notebooks that differs by less than ≈2σ ≈ 2.4 points is statistically indistinguishable from running the same notebook twice.
\n
In retrospect, the most important methodological mistake we made early in the competition was treating single-shot leaderboard scores as ground truth. When aimo3_final v9 returned 37 three times in a row and [15/15] AIME v3 returned 35 and 38, we read those as meaningful differences and chased the apparent winner. They are not meaningful differences. They are draws from the same noisy distribution.
We did not realize this until the final week, when a deliberate variance experiment on the converged notebook produced the n = 9 distribution above. Had we run such an experiment on day one, we would have saved ≈$300 of GPU spend and three weeks of architectural iteration. This is recommendation #1 of the paper: future AIMO3-style participants should establish a noise floor before iterating on architecture.
\nThroughout the competition we maintained an offline evaluation loop against the Kaggle community dataset hard_50_math_problems_set_v6.csv (published by user zfturbo as hard-math-problems-for-aimo-3) - a 50-problem benchmark of olympiad-level questions with known integer answers. Offline evaluation served as a debugging sanity check and as a relative-performance gauge for architectures we were not yet willing to spend a submission slot on.
We authored 15 offline research branches during the competition, organized into five families, each targeting a specific failure class observed in earlier runs:
\nNone of these 15 branches were ever submitted to the Kaggle leaderboard. Of the 15, nine were never executed to completion due to infrastructure and wiring bugs. Two contested-resolver branches ran 50 problems on hard50 but never logged per-problem correctness. Three Branch B/C variants share a single stale truncated run (cumulative 19/29 at problem 30) that we cannot attribute to a specific architecture because the three notebooks have different code but identical saved outputs from one Colab session. One notebook - aimo3_branch_B_multicell - completed a full, inline-graded 50-problem hard50 run. That is the only offline number we can cite with full verification. Full per-branch execution status is catalogued in Appendix H.
Branch B multicell on hard50: 27/50 (54.0%), mean 233 s/problem, total 194 min. Figure 9 shows the per-problem timing (top) and cumulative accuracy (bottom). No problem hit the 900 s cap; the run finished comfortably inside a notional 5-hour budget at 3.24 h total.
\n![\"Figure](\"https://raw.githubusercontent.com/utkarshsharma00/aimo3/main/figures/fig9_branch_b_hard50.png\")
Accuracy begins at 100% after problem 2, collapses to the mid-60s by problem 10, and bleeds out to 54% by problem 50. The architecture looked competitive early and degraded on the harder back half. Since no problem timed out, 27/50 reflects answer quality, not budget exhaustion.
\nInterpretation is constrained. Branch B multicell's elaborate pairwise-arbitration architecture scored 27 on hard50; final_submission v2 scored a mean of 40.78 on AIMO3. If hard50 difficulty were comparable to AIMO3, Branch B would project to ~27 on AIMO3 - 14 points below the simpler approach. But hard50 is not AIMO3: it is community-sourced, intentionally difficult, and may systematically underestimate AIMO3 performance - which is precisely why Branches A3 and C+ were explicitly designed \"not for hard50-only tuning.\" We also never ran a variance experiment on hard50 analogous to our n = 9 experiment, so we cannot say whether 27/50 is itself ±2 points of noise.
Even with those caveats, the directional signal was clear: our only offline-executed elaborate architecture predicted a losing Kaggle score, and the simpler baseline outperformed it. The writeup-prize rubric asks \"did you use any internal benchmarks?\" - yes, but less effectively than we should have. Nine of our 15 research notebooks never finished a run; we never calibrated hard50 against AIMO3; and the one completed offline run told a story that should have killed Branch B's lineage earlier than it did. Future participants: instrument your offline benchmark the way you instrument your leaderboard submissions, get all your research notebooks across the finish line, and run a variance experiment on the offline set before treating any single offline number as signal. Appendix H lists all 15 branches with design intent and execution status.
\nfinal_submission v2final_submission v2 is intentionally lean. It does not contain a 5-stage prompt, a router, a verifier, an adversarial pass, GenSelect, multi-strategy prescriptive prompts, or fine-tuned weights. It contains four things. First, the base model: danielhanchen/gpt-oss-120b served by local vLLM (KV-cache fp8_e4m3, gpu_memory_utilization 0.96, max_model_len 65,536). Second, a one-sentence system prompt: \"You are a world-class IMO competitor. The final answer must be a non-negative integer between 0 and 99999. Place your final answer inside \\boxed{}.\" The model is invoked through the openai-harmony format with ReasoningEffort.HIGH, the gpt-oss native reasoning control. Third, tool-augmented parallel self-consistency: 8 attempts dispatched in parallel against a pool of 16 long-lived Jupyter sandboxes (math, numpy, sympy, mpmath, itertools, collections; mpmath.mp.dps = 64), up to 128 harmony turns each, T = 1.0, min_p = 0.02, no top_p, per-attempt seed (global_seed + attempt_index)². Streams are inspected character-by-character; an attempt short-circuits the moment a \\boxed{...} appears. Fourth, 5-component weighted-entropy voting (Section 5.2); when 4 attempts agree, remaining attempts are cancelled.
The time budget is computed adaptively per problem: min(max(time_remaining − 270 × problems_remaining, 270), 900) seconds, with notebook_limit = 17400 (4h50m, leaving safety margin inside Kaggle's 5h cliff). Appendix A contains the full configuration.
Inverse-entropy voting is a known technique: weight each attempt by 1/H, sum weights per answer, return the highest-weighted answer. The novelty in final_submission v2 is that the entropy itself is a five-part composite over the attempt's logprob trajectory, rather than a single mean. Reading from the source:
weighted_entropy = (\n 0.30 * mean_entropy\n + 0.40 * position_weighted_entropy # exponential decay 0.995, recent tokens dominate\n + 0.20 * std_dev_of_entropy # variance penalty for inconsistent confidence\n + 0.30 * 3.0 * high_entropy_ratio # fraction of tokens with H > 2.0 bits, scaled ×3\n - 0.10 * (max_low_entropy_streak / n) # bonus for sustained confident stretches\n)\nThe intuition: position weighting with decay 0.995 means tokens near the final boxed answer count exponentially more than tokens early in the chain of thought - a solution uncertain during exploration but confident at conclusion outranks one that is confident throughout but flinches at the end. The variance penalty punishes oscillating confidence (\"the model kept changing its mind\"). The high-entropy ratio counts tokens where the next-token distribution exceeds 2 bits of Shannon entropy, and the ×3 multiplier inside a 30% weighting makes this term dominant when an attempt has many uncertain stretches. The streak bonus rewards sustained runs of < 1 bit entropy as a marker of \"the model is on a roll.\"
\nThese weights and thresholds are not from a paper. They were tuned by hand on intermediate submissions in this competition. The core idea of inverse-entropy weighted voting was informed by pawanmali's \"Chasing 47/50\" notebook, which demonstrated entropy-based confidence weighting for self-consistency on AIMO3. We do not claim they are optimal - only that this composite produced a stable mean of 40.78 in final_submission v2, while simpler \"mean entropy\" voting in eight other notebooks we submitted produced means in the 33–37 region. Section 6 unpacks that comparison.
final_submission v2For the avoidance of doubt: final_submission v2 does not contain the 5-stage UNDERSTAND/EXPLORE/PLAN/EXECUTE/VERIFY scaffold (tested in [15/15] AIME v3, scored 35 and 38), the 8 prescriptive strategy prompts from the aimo3_final family (five submissions, mean ~36.6), the verifier/GenSelect/adversarial scaffolding from the same family, the subject-classifier routed prompts from aimo3_adaptive v4 (scored 34), the full adaptive router with posterior voting (scored 28), the increased attempt count from base_submission v2 (scored 33, timed out), or the GRPO fine-tuned weights from the v4_grpo submission (scored 13, merge format mismatch). Figures 7 and 8 visualize the same information at the feature and score levels respectively.
Figure 8 collects every intervention we tried as a horizontal bar chart, compared against the baseline of 39 (the first run of aimo3_final v3, which is the earliest non-debugging submission still in our history).
![\"Figure](\"https://raw.githubusercontent.com/utkarshsharma00/aimo3/main/figures/fig8_ablation_waterfall.png\")
Three regions: inside the noise band - early aimo3_final iterations, the multi-strategy prompt family, and the 5-stage scaffold sit within ±2σ of the 39 baseline and are indistinguishable from starting fresh (the strategy-prompts family was tested in five submissions and never broke out). Below the noise band - the structured-scaffold-plus-classifier branch (−5), V11 timeout (−6), the router with GenSelect/adversarial pass (−11), and GRPO (−26); all real failures with identifiable causes that Section 7 dissects. Above the noise band - only final_submission v2 at +1.78, with the lower bound of its 95% CI (39.85) itself above baseline. It is the only intervention in the entire competition with statistically defensible upward movement.
Figure 7 makes the same point feature-by-feature. Each row is a notebook, each column a feature; checkmarks indicate presence.
\n![\"Figure](\"https://raw.githubusercontent.com/utkarshsharma00/aimo3/main/figures/fig7_feature_comparison.png\")
final_submission v2 (top row) has fewer features than every other notebook except the most basic, and is the only one with the 5-component weighted entropy. Notebooks with strategy prompts cluster around 36–37; notebooks with verifier/GenSelect/adversarial scaffolding cluster around 28–37; the router is at 28; GRPO at 13. Adding architecture moved us downward; subtracting it and replacing the voting function moved us upward. The honest causal claim we can make is that stripping the scaffolding AND replacing the voting function with 5-component weighted entropy together produced a defensible improvement above noise. We cannot say which change was load-bearing on its own - that would require an isolated ablation we did not run before the deadline.
The aimo3_final v7/v9/v22 family launches eight parallel attempts, each with a different prescriptive system prompt instructing the model to use a specific mathematical technique - analytic, computational, small-cases, work-backwards, algebraic, modular, geometric, random-testing-and-verification. The hypothesis was that prompt-induced strategy diversity would produce a more diverse candidate pool for self-consistency voting to exploit, extending the DIPPER intuition that framing diversity beats temperature diversity alone.
It failed because gpt-oss-120b at ReasoningEffort.HIGH complies with prescribed techniques even when they are poorly suited - a geometry problem given a \"modular arithmetic\" prompt produced attempts that contorted themselves toward modular methods and failed. The base model already selects appropriate techniques at T = 1.0; external prescription overrides this selection in the wrong direction about as often as the right one. Across five submissions, the mean was approximately 36.6 - Δ ≈ −2.4 vs. baseline, just inside the lower noise edge but directionally negative. The methodological distinction we carried forward: prescribing solution content hurts, prescribing reasoning form (Section 7.2) is roughly neutral.
[15/15] AIME v3, scored 35 and 38)The [15/15] AIME 2026 I 120b in 20mins notebook v3 uses the full 5-stage UNDERSTAND → EXPLORE → PLAN → EXECUTE → VERIFY scaffold popular in CoT prompting research, with a verbose ~1500-character system prompt detailing reasoning principles and verification requirements. The hypothesis was that even a strong base model would benefit from being forced to plan before executing.
The two submissions scored 35 and 38 - squarely inside the ±2σ noise band of baseline. We cannot reject the null hypothesis that the scaffold has zero effect on gpt-oss-120b. A plausible explanation is that ReasoningEffort.HIGH already imposes a reasoning structure that the prompt-level scaffold duplicates without adding signal. Technique transfer from smaller-model prompting research does not automatically apply to gpt-oss-120b at high reasoning effort.
AIMO3_Adaptive_Router_Confidence_Vote_v1, scored 28)A multi-stage system: a subject classifier routed each problem to one of four prompt families (geometry, number theory, combinatorics, algebra); a \"quick stage\" issued 2 cheap attempts (T = 0.55 and 0.80, max 2200 tokens, 24 turns); if those agreed with high posterior confidence, return immediately, otherwise an \"explore stage\" issued more attempts at higher temperature with longer budgets; weak candidates were routed through verifier, GenSelect, and adversarial passes; high_problem_timeout = 840 s per problem. The hypothesis was the standard one: easy problems should consume little budget, hard problems more; an adaptive system should outperform a flat-budget system under heterogeneous difficulty.
It failed structurally. The first problem consumed 840 s. The second consumed approximately 825 s. The router had burned 28% of the entire 5-hour notebook budget on two problems, and the remaining 48 problems shared the rest. Many timed out or returned 0. Final score: 28 - fully twelve points below baseline, the second-largest single-intervention drop in the competition. The router's adaptive logic correctly identified problems 1 and 2 as hard and allocated them more time; it then found nothing useful in the extra time. The correct behavior under those conditions is to give up early and conserve budget for problems where extra time might actually help - we did not implement that. A v2 with reduced timeouts reached 19/30 on an offline slice but we did not submit it; the converged final_submission v2 was already outperforming it. Adaptive complexity is only valuable when the marginal value of extra time on a hard problem is positive. For hard olympiad problems in the AIMO3 distribution against gpt-oss-120b, that marginal value appears to be approximately zero past 5–6 minutes.
base_submission v2, scored 33)This is the team's fork of the public AIMO 3 | GPT-OSS-120B (with tools) notebook (\"the 42-notebook\" in community discussion), with three modifications: attempts raised from 8 to 12, early_stop raised from 4 to 5, and a \"Be concise. Stop immediately after \\boxed{}\" instruction added. The notebook also uses the stock OpenAI gpt-oss-120b weights rather than Daniel Hanchen's variant. Hypothesis: more samples → stronger voting → higher score, up to the point where the time budget binds.
It exceeded the 5-hour budget. Twelve attempts × 50 problems with a verbose chain-of-thought trajectory pushed total runtime past the cliff; problems processed after timeout returned 0. Final score: 33. The experiment is confounded because we changed three variables at once and cannot cleanly attribute the −6 to \"more attempts\" alone. Operational lesson: if you push N upward, run a wall-clock experiment first, and budget for the slowest model variant.
\nThis is the largest and most expensive negative result in the writeup, and the most generally useful for the community. We approached it in two phases.
\nPhase A: TRL GRPOTrainer (collapsed at step 16, $101.73 sunk). Our first attempt used Hugging Face TRL's GRPOTrainer with Unsloth on a RunPod B200 ($5.03/hr). Configuration: LR 5e-6, max_grad_norm 1.0, β = 0.0, 8 generations per problem, use_vllm=False (because TRL's vLLM integration crashed at startup against gpt-oss-120b). Each step took ≈76 minutes. The training collapsed:
Figure 6 shows the trajectory.
\n![\"Figure](\"https://raw.githubusercontent.com/utkarshsharma00/aimo3/main/figures/fig6_trl_collapse.png\")
The diagnosis is straightforward: LR 5e-6 was too aggressive for an MoE model at 117B parameters, and max_grad_norm 1.0 was too loose to clip the gradient explosions that began once the policy started drifting from the reference.
Phase B: Custom vLLM + Unsloth loop (v6) - trained stably, deployed catastrophically. We rebuilt the training loop from scratch to avoid TRL's vLLM integration: each cycle launched vLLM with the current adapter, generated 8 completions per problem for 5 problems, killed vLLM to free GPU memory, applied a GRPO update via Unsloth, saved the adapter, restarted. We tightened the configuration substantially (LR 2e-6 with cosine decay, max_grad_norm 0.3, Dr.GRPO advantage normalization, DAPO-style overlong filtering, binary reward, β = 0, off-policy generation with vLLM on the base model and adapter trained separately; see Appendix B for the full config).
\nWe have telemetry for 190 problems across 38 cycles (cycles 62–99, problems 310–499). The training was stable - zero loss collapses, loss values mostly in [0, 0.3], gradient norms cleanly bounded by the 0.3 clip. Figure 3 shows the per-cycle reward trajectory:
\n![\"Figure](\"https://raw.githubusercontent.com/utkarshsharma00/aimo3/main/figures/fig3_grpo_reward_trajectory.png\")
The reward signal tells the second half of the story. The overall mean reward across all 190 problems was +0.033 - barely above zero, after 38 cycles of training. Only 4 problems out of 190 (2.1%) achieved reward 1.0 (all 8 generations correct): problems 329, 333, 379, 468. Three of those four had near-zero completion length, suggesting trivially short correct answers. Figure 5 shows the full reward distribution:
\n![\"Figure](\"https://raw.githubusercontent.com/utkarshsharma00/aimo3/main/figures/fig5_grpo_reward_histogram.png\")
Forty-seven percent of training problems received net negative reward. Eighteen percent received exactly zero (all 8 generations wrong, or all 8 right) - these are filtered from the gradient by Dr.GRPO's advantage normalization, contributing nothing. Thirty-three percent were partial wins. Only 2% were complete wins. This is the hardest possible shape for GRPO: the model is below 50% on the training distribution, and the reward signal is sparse and noisy. Even a stable training loop cannot extract much policy improvement from this.
\nThe deployment failure. We merged the trained adapter using Unsloth's merged_4bit_forced method. The submission scored 13/50 - a 27-point drop from baseline. The root cause is a quantization format mismatch: gpt-oss-120b's native weight format is MXFP4 (E2M1 with E8M0 block scaling), but merged_4bit_forced produces BitsAndBytes NF4. vLLM routes the merged weights through its BitsAndBytes loader rather than the MXFP4 kernels, producing severely degraded output. The fix - which we discovered post-mortem from Unsloth documentation and vLLM GitHub issues #19361 and #21932 - is to merge with merged_16bit or save with save_method=\"mxfp4\". A community member elsewhere in the AIMO3 forum reportedly spent $500 on SFT+RL with a similar pipeline and scored 36, attributing it to the same issue. We spent approximately $300 total on GRPO across all phases (Figure 4): TRL crash $102, v1–v5 debugging $100, v6 $62, idle $40.
![\"Figure](\"https://raw.githubusercontent.com/utkarshsharma00/aimo3/main/figures/fig4_grpo_cost.png\")
The actionable takeaway: never use merged_4bit_forced to deploy a fine-tuned gpt-oss-120b LoRA to vLLM. Use merged_16bit, save_method=\"mxfp4\", or NVIDIA's QAT pipeline.
We benchmarked exclusively against the open-weight setting because that is what AIMO3's hardware budget allows. The relevant SOTA points are:
\ngpt-oss-120b for AIMO3: Reported in the high 30s to low 40s with simple self-consistency setups.The host shared pass@N analysis on X showing gpt-oss-120b achieves pass@8 ≈ 44 and pass@20 ≈ 45.5 on the private set (95% CI [43, 48]), indicating that the correct answer appears in at least one of 8 attempts for approximately 44 of 50 problems. Our mean of 40.78 implies we correctly select the right answer from the pool approximately 93% of the time (40.78/44), with the remaining ~3 problems per run representing pure selection loss.
\nThe structural shift from AIMO2 to AIMO3 is striking. AIMO2 was won by a team that invested heavily in training-time compute and produced a 14B-parameter model that scored 34. AIMO3 allows H100 hardware, which makes gpt-oss-120b runnable, and the unmodified base model with a sensible voting function reaches 40.78 - six points above the AIMO2 winner with zero training-time investment. Base model capability dominates the AIMO3 landscape in a way it did not dominate AIMO2. Our negative GRPO result is consistent with this: training-time interventions face an uphill battle against a base model that is already near the ceiling of the inference-constrained accuracy distribution.
We did not benchmark against commercial models (GPT-4, Claude, Gemini) because the AIMO3 rules require self-contained Kaggle notebook execution and commercial API calls are not permitted in the competition pipeline.
\nThe competition produced eight findings we believe will help future participants more than another point on the leaderboard would.
\nEstablish the noise floor first. Submit the same notebook 5–10 times before iterating on architecture. σ = 1.20 means any 1–2 point \"improvement\" is invisible to single-shot comparison. Every other recommendation depends on this one.
Per-problem time budgets are absolute. The 5-hour cliff is hard-enforced. Any architecture that can spend more time on a problem must have a hard cap well below what the budget would technically allow - hard problems do not pay you back for extra time.
Strategy-prompt diversity hurts a strong base model. When gpt-oss-120b at ReasoningEffort.HIGH is told to use a specific technique, it complies even when the technique is wrong. Sampling-temperature diversity at T = 1.0 produces enough variance for self-consistency voting without prescriptive prompts.
Confidence estimation matters more than sample count. Our single defensible improvement above noise came from changing the voting function, not from generating more samples. Treat the voting function as a first-class optimization target.
merged_4bit_forced is a trap for gpt-oss-120b. The Unsloth → vLLM pipeline silently produces a BitsAndBytes-quantized model that vLLM cannot run efficiently. Use merged_16bit or save_method=\"mxfp4\".
Stable GRPO training is necessary but not sufficient. Our v6 loop trained stably for 38 cycles with no collapses, but the deployed model scored 13 because of the merge format issue, and the reward signal during training (mean +0.033) was too sparse for meaningful policy improvement.
Instrument your offline benchmark like your leaderboard. We ran 15 offline research branches but only one completed a verifiable end-to-end run. We never calibrated hard50 difficulty against AIMO3. Subtraction beats addition: every architectural addition we tested was neutral or harmful.
Document the negative results. We invested ~$300 of B200 GPU time and three weeks of architectural iteration in directions that did not work. If this writeup helps even one future team avoid the merged_4bit_forced trap or the strategy-prompt-diversity dead end, it has paid for itself many times over.
We did not win AIMO3. Our private-leaderboard score of 41.5/50 and rank 828th / 4,138 (public-leaderboard best score of 43/50 and rank 86th / 4,138) falls squarely within our predicted 95% CI of [39.85, 41.70], validating the variance methodology, well above naive baselines but below the top of the leaderboard. What we contribute to the community is not a higher score but a documented, statistically defensible characterization of the AIMO3 inference problem with gpt-oss-120b: the base model is strong enough that most engineering interventions are dominated by run-to-run noise; the few interventions that move the score appreciably move it downward; and the single defensible upward move we found was a more careful confidence-estimation function for self-consistency voting. The expensive failure modes - strategy-prompt prescription, multi-stage routing, and most importantly GRPO with broken deployment - are all reproducible from the configurations and telemetry in the appendices. Every score in this paper is from an actual public-leaderboard submission, every configuration is from actual source code, and every causal claim is hedged to the strength of the evidence we have.
If you take one thing from this writeup, take this: before you spend a week building a router, submit the same notebook many times and look at the variance.
\nOur selected submission (final_submission v2) scored 41.5/50 on the private leaderboard (rank 828 / 4,138). Under the double-run scoring rule - where each problem receives 1.0 if both runs are correct, 0.5 if one is correct, and 0.0 if neither is - this score is consistent with a per-run accuracy of approximately 41–42, directly in line with our public-LB mean of 40.78 (σ=1.20, n=9).
The private score falls within our predicted 95% confidence interval of [39.85, 41.70], confirming that the variance methodology in Section 3 generalizes from public to private evaluation. The noise floor finding - that σ=1.20 across identical runs means any sub-2.4-point \"improvement\" is statistically indistinguishable from noise - is validated by this out-of-sample result.
\nOur rank shifted from 86th (public, based on best single submission of 43) to 828th (private, based on the selected submission's double-run average). This rank compression is itself evidence for the noise-floor thesis: teams whose public rank was inflated by a single lucky run regressed toward their true mean on the private set, while our pre-registered variance estimate correctly predicted our private performance.
\nfinal_submission v2This is the verbatim configuration from the CFG class in the submitted notebook.
| Parameter | \nValue | \n
|---|---|
| Base model | \n/kaggle/input/models/danielhanchen/gpt-oss-120b/transformers/default/1 | \n
| Served model name (vLLM) | \ngpt-oss | \n
| Quantization | \nNative MXFP4 | \n
| Tensor parallel size | \n1 | \n
| gpu_memory_utilization | \n0.96 | \n
| kv_cache_dtype | \nfp8_e4m3 | \n
| dtype | \nauto | \n
| context_tokens (max_model_len) | \n65,536 | \n
| batch_size (max_num_seqs) | \n256 | \n
| stream_interval | \n200 | \n
| enable_prefix_caching | \nTrue | \n
| async_scheduling | \nTrue | \n
| System prompt | \n\"You are a world-class IMO competitor. The final answer must be a non-negative integer between 0 and 99999. Place your final answer inside \\boxed{}.\" | \n
| Tool prompt | \n\"Use this tool to execute Python code. The environment is a stateful Jupyter notebook. Always use print() to output results.\" | \n
| Preference prompt | \n\"You have access to math, numpy, sympy, and mpmath.\" | \n
| Reasoning effort | \nReasoningEffort.HIGH (set on harmony SystemContent) | \n
| Tool format | \nopenai-harmony Conversation API | \n
| Available libraries in sandbox | \nmath, numpy, sympy, mpmath, itertools, collections; mpmath.mp.dps = 64 | \n
| temperature | \n1.0 | \n
| min_p | \n0.02 | \n
| top_p | \nnot set (intentionally - top_p = 0.9 was tested and caused a regression in earlier iterations) | \n
| top_logprobs | \n5 | \n
| attempts | \n8 | \n
| workers (parallel sandboxes) | \n16 | \n
| early_stop | \n4 (cancel remaining attempts when 4 agree) | \n
| turns (max harmony turns per attempt) | \n128 | \n
| seed (global) | \n42 | \n
| per-attempt seed | \n(seed + attempt_index) ** 2 | \n
| buffer_tokens | \n512 | \n
search_tokens (streaming \\boxed{} window) | \n32 | \n
| notebook_limit | \n17,400 s (4h50m) | \n
| base_problem_timeout | \n270 s | \n
| high_problem_timeout | \n900 s | \n
| server_timeout | \n180 s | \n
| session_timeout | \n960 s | \n
| jupyter_timeout | \n6 s | \n
| sandbox_timeout | \n3 s | \n
| per-problem budget formula | \nmin(max(time_remaining − 270 × (problems_remaining − 1), 270), 900) | \n
def _compute_weighted_entropy(self, logprobs_buffer):\n \"\"\"5-component weighted entropy.\"\"\"\n if not logprobs_buffer:\n return float('inf')\n\n entropies = []\n for top_lp in logprobs_buffer:\n if not isinstance(top_lp, dict) or not top_lp:\n continue\n h = 0.0\n for _, lp in top_lp.items():\n p = math.exp(lp)\n if p > 0:\n h -= p * math.log2(p)\n entropies.append(h)\n\n if not entropies:\n return float('inf')\n\n n = len(entropies)\n mean_ent = sum(entropies) / n\n\n # Variance penalty\n if n > 1:\n variance = sum((e - mean_ent) ** 2 for e in entropies) / (n - 1)\n std_dev = math.sqrt(variance)\n else:\n std_dev = 0.0\n\n # Position-weighted (recent tokens dominate, decay 0.995)\n decay = 0.995\n weights = [decay ** (n - 1 - i) for i in range(n)]\n total_weight = sum(weights)\n position_weighted = (\n sum(w * e for w, e in zip(weights, entropies)) / total_weight\n if total_weight > 0 else mean_ent\n )\n\n # High-entropy ratio (tokens with H > 2.0 bits)\n high_ent_count = sum(1 for e in entropies if e > 2.0)\n high_ent_ratio = high_ent_count / n\n\n # Low-entropy streak bonus (sustained < 1.0 bit)\n max_streak = 0\n current_streak = 0\n for e in entropies:\n if e < 1.0:\n current_streak += 1\n max_streak = max(max_streak, current_streak)\n else:\n current_streak = 0\n streak_bonus = -0.1 * (max_streak / n) if n > 0 else 0.0\n\n weighted = (\n 0.3 * mean_ent\n + 0.4 * position_weighted\n + 0.2 * std_dev\n + 0.3 * high_ent_ratio * 3.0\n + streak_bonus\n )\n return max(weighted, 1e-9)\n\n\ndef _select_answer(self, detailed_results):\n \"\"\"Inverse-entropy weighted voting.\"\"\"\n answer_weights = defaultdict(float)\n answer_votes = defaultdict(int)\n for r in detailed_results:\n a = r['Answer']\n e = r['Entropy']\n if a is not None:\n answer_weights[a] += 1.0 / max(e, 1e-9)\n answer_votes[a] += 1\n if not answer_weights:\n return 0\n scored = sorted(\n [{'answer': a, 'votes': answer_votes[a], 'score': w}\n for a, w in answer_weights.items()],\n key=lambda x: x['score'], reverse=True\n )\n return scored[0]['answer']\ndef _scan_for_answer(self, text):\n pattern = r'\\\\boxed\\s*\\{\\s*([0-9,]+)\\s*\\}'\n matches = re.findall(pattern, text)\n if matches:\n try:\n val = int(matches[-1].replace(',', ''))\n if 0 <= val <= 99999:\n return val\n except ValueError:\n pass\n pattern = r'final\\s+answer\\s+is\\s*([0-9,]+)'\n matches = re.findall(pattern, text, re.IGNORECASE)\n if matches:\n try:\n val = int(matches[-1].replace(',', ''))\n if 0 <= val <= 99999:\n return val\n except ValueError:\n pass\n return None\nThe streaming inspection happens inside _process_attempt: after each chunk of streamed text, the last search_tokens = 32 tokens worth of text are scanned for \\boxed{...}, and if found, the attempt short-circuits without consuming additional turns.
The 5-component weighted entropy formula builds on the multi-component entropy scoring framework from pawanmali's \"Chasing 47/50\" notebook, which demonstrated that position-weighted entropy with variance and streak components substantially outperforms simple mean entropy on AIMO3. Our contribution is the specific tuning of weights and thresholds across intermediate submissions; the structural design of which components to include follows pawanmali's framework.
\n| Parameter | \nValue | \n
|---|---|
| Hardware | \nRunPod B200 (1 GPU, 191.5 GB VRAM) | \n
| Cost rate | \n$5.03/hr | \n
| Base model | \ndanielhanchen/gpt-oss-120b loaded via Unsloth with load_in_4bit=True | \n
| Quantization during training | \nBitsAndBytes NF4 (Unsloth converts MXFP4 → NF4 on load) | \n
| Architecture | \nOff-policy: vLLM serves base model; Unsloth trains LoRA adapter separately; vLLM restarted between cycles | \n
| Learning rate | \n2e-6 with cosine decay | \n
| max_grad_norm | \n0.3 | \n
| Generations per problem | \n8 | \n
| Temperature (rollouts) | \n1.0 | \n
| top_p (rollouts) | \n0.98 | \n
| max_completion_tokens | \n6,144 | \n
| Advantage | \nDr.GRPO (mean subtraction, no std division) | \n
| Overlong filtering | \nDAPO-style (truncated completions removed from gradient) | \n
| KL coefficient (β) | \n0.0 | \n
| Reward function | \nBinary: 1.0 if extracted boxed answer matches ground truth, 0.0 otherwise | \n
| Problems per cycle | \n5 | \n
| Cycles captured in telemetry | \n38 (cycles 62–99) | \n
| Problems captured in telemetry | \n190 (problems 310–499) | \n
| Mean cycle time | \n~1,170 s (gen ~720s + train ~440s) | \n
| Cost in v6 logged segment | \n$61.87 | \n
| Total GRPO project cost (all phases) | \n≈$300 | \n
| Merge method used (the bug) | \nmerged_4bit_forced → produces BnB NF4 | \n
| Merge method that should have been used | \nmerged_16bit or save_method=\"mxfp4\" | \n
| Submission score | \n13/50 | \n
| Parameter | \nValue | \n
|---|---|
| Framework | \nHugging Face TRL GRPOTrainer + Unsloth | \n
| use_vllm | \nFalse (TRL+vLLM crashed at startup) | \n
| Learning rate | \n5e-6 | \n
| max_grad_norm | \n1.0 | \n
| β (KL coefficient) | \n0.0 | \n
| Generations per problem | \n8 | \n
| Steps completed | \n16 (model destroyed at step 16) | \n
| Time per step | \n≈76 minutes | \n
| Cost | \n$101.73 | \n
| Loss at step 1 | \n0.0513 | \n
| KL at steps 8–9 | \n87 | \n
| Gradient norm at steps 8–9 | \n133 | \n
| Reward at step 13 | \n0.5625 (peak) | \n
| Loss at step 16 | \n19,751,736 | \n
| Parameter | \nValue | \n
|---|---|
| Method | \nQLoRA, rank 128 | \n
| Training data | \n63,959 tool-integrated reasoning (TIR) traces | \n
| Steps | \n1,000 | \n
| Final loss | \n0.69 (plateau) | \n
| Uploaded model | \nflamingice2801/gpt-oss-120b-aimo3-tir-sft (Kaggle Models) | \n
| Submitted to AIMO3? | \nNo - abandoned after GRPO deployment failure exposed the merge format issue | \n
Twenty-five successful submissions plus three early debug parquet uploads (score 0) and one notebook exception. Listed in chronological order (oldest first).
\n| # | \nDays ago at writeup | \nNotebook | \nVersion | \nScore | \nPhase | \n
|---|---|---|---|---|---|
| 1 | \n24 | \naimo3_final | \nv3 | \n39 | \nIteration | \n
| 2 | \n23 | \naimo3_final | \nv4_tweaked_temp&model | \n11 | \nDebugging | \n
| 3 | \n22 | \naimo3_final | \nv5_tweaked_temp | \n0 | \nDebugging | \n
| 4 | \n21 | \naimo3_final | \nv7 | \n36 | \nIteration | \n
| 5 | \n20 | \naimo3_final | \nv9 | \n37 | \nIteration | \n
| 6 | \n19 | \naimo3_final | \nv22 | \n36 | \nIteration | \n
| 7 | \n18 | \nbase_submission | \nv2 | \n33 | \nFailure (V11 timeout) | \n
| 8 | \n17 | \n[15/15] AIME 2026 I 120b in 20mins | \nv4_grpo | \n13 | \nFailure (GRPO) | \n
| 9 | \n16 | \n[15/15] AIME 2026 I 120b in 20mins | \nv3 | \n38 | \nIteration | \n
| 10 | \n15 | \naimo3_final | \nv3 (re-run) | \n34 | \nIteration | \n
| 11 | \n14 | \nAIMO3_Adaptive_Router_Confidence_Vote_v1 | \nv1 | \n28 | \nFailure (router) | \n
| 12 | \n13 | \n[15/15] AIME 2026 I 120b in 20mins | \nv3 (re-run) | \n35 | \nIteration | \n
| 13 | \n12 | \naimo3_final | \nv9 (re-run) | \n37 | \nIteration | \n
| 14 | \n11 | \naimo3_final | \nv9 (re-run) | \n37 | \nIteration | \n
| 15 | \n10 | \naimo3_adaptive | \n_v4 | \n34 | \nFailure | \n
| 16 | \n9 | \nfinal_submission | \nv2 | \n39 | \nFinal (run 1) | \n
| 17 | \n8 | \naimo_tester | \nv2 | \n34 | \nIteration | \n
| 18 | \n7 | \nfinal_submission | \nv2 | \n40 | \nFinal (run 2) | \n
| 19 | \n6 | \nfinal_submission | \nv2 | \n43 | \nFinal (run 3) | \n
| 20 | \n4 | \nfinal_submission | \nv2 | \n40 | \nFinal (run 4) | \n
| 21 | \n3 | \nfinal_submission | \nv2 | \n42 | \nFinal (run 5) | \n
| 22 | \n3 | \nfinal_submission | \nv2 | \n40 | \nFinal (run 6) | \n
| 23 | \n2 | \nfinal_submission | \nv2 | \n41 | \nFinal (run 7) | \n
| 24 | \n1 | \nfinal_submission | \nv2 | \n41 | \nFinal (run 8) | \n
| 25 | \n0 | \nfinal_submission | \nv2 | \n41 | \nFinal (run 9) | \n
final_submission v2 distribution: {39, 40, 40, 40, 41, 41, 41, 42, 43}, n = 9, mean 40.778, median 41, σ = 1.202, 95% CI [39.85, 41.70], best 43.
The full per-problem telemetry CSV (190 rows × 14 columns) is provided as a supplementary artifact. Summary statistics:
\n| Statistic | \nValue | \n
|---|---|
| Total rows | \n190 | \n
| Cycle range | \n62 – 99 (38 cycles) | \n
| Problem range | \n310 – 499 | \n
| Generations per problem | \n8 (constant) | \n
| Mean generation time per cycle | \n720 s | \n
| Mean training time per cycle | \n440 s | \n
| Mean cycle time | \n1,170 s | \n
| Cost at cycle 62 | \n$2.34 | \n
| Cost at cycle 99 | \n$61.87 | \n
| Mean reward (all 190 problems) | \n+0.033 | \n
| Problems with reward < 0 | \n90 (47.4%) | \n
| Problems with reward = 0 (filtered) | \n34 (17.9%) | \n
| Problems with 0 < reward < 1 | \n62 (32.6%) | \n
| Problems with reward = 1.0 (all 8 correct) | \n4 (2.1%) | \n
| Problem IDs achieving reward = 1.0 | \n329, 333, 379, 468 | \n
| Loss range | \n[0.0, ~0.7] | \n
| Loss collapses | \n0 | \n
| Mean completion length | \n~3,500 tokens | \n
| Max completion length observed | \n~6,400 tokens (binding against 6,144 cap) | \n
The full CSV columns are: cycle, problem_idx, gen_time_sec, train_time_sec, total_sec, reward_mean, reward_std, num_correct, num_total, loss, completion_len_mean, clipped_ratio, gpu_mem, cost.
The 9-run variance experiment for final_submission v2 produced scores {39, 40, 40, 40, 41, 41, 41, 42, 43}. Descriptive statistics:
| Statistic | \nValue | \n
|---|---|
| n | \n9 | \n
| Sum | \n367 | \n
| Mean | \n40.778 | \n
| Median | \n41 | \n
| Mode | \n40 and 41 (each appear 3 times) | \n
| Min | \n39 | \n
| Max | \n43 | \n
| Range | \n4 | \n
| Sample variance s² | \n1.444 | \n
| Sample standard deviation s | \n1.202 | \n
| Standard error of the mean | \n0.401 | \n
| t-critical (df=8, 95%) | \n2.306 | \n
| 95% CI for mean | \n[39.854, 41.702] | \n
| Coefficient of variation | \n2.95% | \n
Under the assumption that each problem is solved with effective probability p_eff after voting, the score X ~ Binomial(50, p_eff). From mean = 40.778, p_eff ≈ 0.816, predicting a theoretical Binomial standard deviation of √(50 · 0.816 · 0.184) ≈ 2.74. Our observed σ = 1.20 is meaningfully lower than the Binomial prediction, suggesting positive correlation between problems within a single run (i.e., when the model is \"having a good run,\" it tends to solve more problems than independent sampling would predict). Plausible mechanisms include vLLM batching state, KV-cache hit patterns, and GPU thermal/scheduling effects that affect attempts within a single notebook run more than across runs.
\nFor two independent runs from the same configuration with σ = 1.20:
\nA 3-point single-shot difference between two notebooks therefore has approximately a 1-in-14 chance of being pure noise. A 4-point difference reaches conventional statistical significance (p < 0.05) only with this much variance assumed. Differences smaller than 3 points should not be acted on without replication.
\nThe public Kaggle notebook AIMO 3 | GPT-OSS-120B (with tools) (\"the 42-notebook\" in community discussion) is reported to score 42/50 by its authors with default settings. We never submitted the unmodified 42-notebook. Our base_submission v2 is a fork of it with three modifications:
attempts raised from 8 to 12early_stop raised from 4 to 5\"Be concise. Stop immediately after \\boxed{}\" added to the preference promptbase_submission v2 scored 33 - six points below baseline - primarily due to the wall-clock overrun caused by modification #1. Because we changed three variables at once, we cannot attribute the −6 to \"more attempts\" alone, and we cannot use this submission as a measurement of the unmodified 42-notebook's performance in our environment. The 42-notebook's score in our hands is, strictly speaking, unknown. We list this as a methodological limitation rather than a result.
If we were starting AIMO3 again with what we know now, in priority order:
\ngpt-oss-120b. Specifically search for \"MXFP4\" and \"merged_4bit\". This would have saved the ≈$300 GRPO sunk cost.Fifteen offline research branches were authored during the competition to explore architectures that we were not willing to spend Kaggle submission slots on. None of the 15 were submitted to the AIMO3 leaderboard. Their target benchmark was the community-sourced hard_50_math_problems_set_v6.csv dataset (Kaggle dataset zfturbo/hard-math-problems-for-aimo-3). This appendix documents each branch with its design intent (taken verbatim from the top-of-notebook docstring or markdown cell where available), its execution status, and what the preserved notebook outputs actually verify.
| Branch file | \nExecution status | \nVerifiable score | \nMean time/problem | \n
|---|---|---|---|
aimo3_branch_a_contested_resolver | \nAuthored, never executed to completion | \n- | \n- | \n
aimo3_branch_a_contested_resolver_rebuilt | \nAuthored, never executed to completion | \n- | \n- | \n
aimo3-contested-resolver | \nRan 50 problems; no correctness logging | \nCannot verify | \n216 s | \n
aimo3_contested_resolver_conservative (A2) | \nRan 50 problems; no correctness logging | \nCannot verify | \n216 s | \n
aimo3_A3_1_clean_multicell | \nAuthored, never executed to completion | \n- | \n- | \n
aimo3_A3_2_candidategen_multicell | \nAuthored, never executed to completion | \n- | \n- | \n
aimo3_A3_2_patch_multicell | \nAuthored, never executed to completion | \n- | \n- | \n
aimo3_branch_B_multicell | \nComplete 50-problem run with inline grading | \n27/50 (54.0%) | \n233 s | \n
aimo3_branch_B_fixed | \nShares stale truncated run with C_final | \n19/29 (ambiguous) | \n218 s | \n
aimo3_branch_B_updated | \nShares stale truncated run with C_final | \n19/29 (ambiguous) | \n218 s | \n
aimo3_branch_C_final | \nShares stale truncated run with B_fixed/updated | \n19/29 (ambiguous) | \n218 s | \n
aimo3_branch_C_fixed2 | \nAuthored, never executed to completion | \n- | \n- | \n
aimo3_branch_C_c2 | \nAuthored, never executed to completion | \n- | \n- | \n
aimo3_branch_C_trust_patch | \nAuthored, never executed to completion | \n- | \n- | \n
aimo3_branch_C_plus_final (C+) | \nAuthored, never executed to completion | \n- | \n- | \n
Critical caveat on the 19/29 row. Three notebooks (B_fixed, B_updated, C_final) have different code but identical saved outputs - same per-problem trace, same cumulative markers, same trace zip timestamp 20260409_092715. The outputs come from a single Colab run that was saved into three different notebook files during iteration. We cannot attribute 19/29 to any specific architecture and therefore do not cite it as a score for any named branch in the main paper. The run terminated at problem 30 - we do not know whether termination was voluntary (user stopped) or involuntary (Colab session died).
The only offline number that survives full provenance audit is Branch B multicell: 27/50 on hard50. All per-problem [EVAL] id=<X> pred=<Y> actual=<Z> correct=<bool> cumulative=<N>/<M> lines are preserved inline in the notebook output cells.
Files: aimo3_branch_a_contested_resolver, aimo3_branch_a_contested_resolver_rebuilt, aimo3-contested-resolver
Design intent (verbatim from the notebook docstring):
\n\n\n\"This notebook starts from the stable
\nfinal_submissionbaseline rather than the experimental branches. The baseline kept the proven ingredients that were strongest in prior AIMO3 experiments: simple prompts, T=1.0 generation, 8 attempts, weighted entropy, 65k context, and no large bundles of interacting heuristics.\"Motivation: hard50 analysis showed two failure classes - rerankable (correct answer appears in the candidate pool but a plurality of wrong attempts shipped it) and exploration (correct answer never generated). Branch A targets only the first class.
\n
Key changes from final_submission:
support DESC → inverse-entropy DESC → mean entropy ASC → Python error rate ASC.rerank = log1p(avg_inv_entropy) − 0.15 × py_err_rate − 0.03 × log1p(mean_py_calls). Constants drawn from hard50 failure taxonomies (66 vs 69, 570 vs 285, 24 vs 25, 2000 vs 1999, 105 vs 106) and explicitly labeled \"heuristic and intentionally mild; not claimed to be optimal.\"Execution status: The two aimo3_branch_a_* variants have zero outputs (never ran end-to-end). aimo3-contested-resolver ran 50 problems on hard50 but only logged answer=X per problem, never pred=X actual=Y correct=bool, so correctness is not inline-verifiable. Mean time 215.9 s/problem, max 710 s.
File: aimo3_contested_resolver_conservative
Design intent (verbatim):
\n\n\n\"The first Branch A run clarified four things: (1) Candidate tables are useful. They make it obvious when the correct answer is already in the pool. (2) The resolver is NOT reliable enough to hard-override the pipeline. In several cases the rerank or stage-1 winner was better than the resolver. (3) Single-cluster early agreement is still dangerous. Some hard problems produced 4 quick agreements on one wrong answer with no serious alternative exploration. (4) The reranker is better used to nominate a challenger than to become the final answer.\"
\n
Key changes from Branch A:
\nsafe_to_ship: requires single-cluster challenge to have passed when applicable.safe_weight_share = 0.70 - explicitly labeled \"heuristic, not hidden-set tuned, not paper-derived optima.\"Execution status: Same as aimo3-contested-resolver - ran 50 problems on hard50 but without per-problem correctness logging. The outputs are byte-identical to the earlier contested-resolver run (same output hash), suggesting this notebook saved the same run's outputs without re-executing.
Files: aimo3_A3_1_clean_multicell, aimo3_A3_2_candidategen_multicell, aimo3_A3_2_patch_multicell
Design intent (verbatim from markdown cell):
\n\n\n\"Preserve the stable baseline generator while making post-generation decisions much more conservative and closure-aware. The design is intended for AIMO3 itself, not for hard50-only optimization.\"
\n
Key changes from Branch A2:
\n\\boxed{} - collect all boxed integers seen in an attempt and use the LAST one; record box instability signals (multiple boxed values, box_changed flag, last-box position). Targets premature-boxing / double-counting / off-by-one failures.Constants explicitly labeled: \"not hard50-fit coefficients. They are conservative priors chosen from observed failure classes.\"
\nExecution status: All three A3 notebooks have zero outputs. Never executed end-to-end.
\nFiles: aimo3_branch_B_fixed, aimo3_branch_B_updated, aimo3_branch_B_multicell
Design intent (verbatim from markdown cell):
\n\n\n\"Preserve the stable AIMO3 generator while making post-generation arbitration stricter and more faithful to the candidate pool. This branch is intended for AIMO3-level problems, not hard50-only tuning.\"
\n
Key changes:
\nExecution status - this is the one verified data point:
\naimo3_branch_B_multicell: Completed a full 50-problem hard50 run. 27 correct, 23 wrong, 27/50 = 54.0%. Total wall-clock: 194.4 min. Mean time per problem: 233.3 s, median 188 s, max 659 s, min 27 s. 10 problems took ≥ 400 s. Zero problems hit the 900 s cap - the 27/50 is not a timeout artifact, it reflects actual answer quality on the hard50 distribution. All per-problem [EVAL] lines with pred, actual, correct, and cumulative are preserved inline in the notebook outputs.aimo3_branch_B_fixed, aimo3_branch_B_updated: Share stale identical outputs with aimo3_branch_C_final from a Colab run with trace timestamp 20260409_092715, truncated at problem 30, cumulative 19/29 (65.5%). Different code bodies, same saved outputs - we cannot attribute the 19/29 to any specific architecture.Figure 9 in the main paper is based on the Branch B multicell run.
\nFiles: aimo3_branch_C_final, aimo3_branch_C_c2, aimo3_branch_C_trust_patch, aimo3_branch_C_fixed2
Design intent (verbatim from CFG docstring):
\n\n\"Branch C keeps the stable Branch B generator and pairwise-centered arbitration, but adds four portable ideas motivated by the AIMO2 analysis and the recent Branch B traces…\"
\n
Key changes from Branch B:
\nExecution status: C_final shares stale outputs with B_fixed/B_updated (see H.5 caveat). C_c2, C_trust_patch, and C_fixed2 have zero outputs - never executed end-to-end. No new verifiable score.
File: aimo3_branch_C_plus_final
Design intent (verbatim from CFG docstring):
\n\n\"Design constraints: This branch is intended for AIMO3-level problems, not hard50-only tuning. Pairwise remains the main discriminator in binary contests. Audit is weakened relative to Branch B when pairwise + pressure already align. Tail problems are protected by a larger reserve and by lighter verification spending on already-settled problems.\"
\n
Key changes: Inherits the full Branch C controller + code rescue + repair + provenance weighting, then adds:
\nThis is the most elaborate architecture in the 15-branch catalog.
\nExecution status: Zero outputs. Authored but never executed to completion.
\nOf 15 offline research branches authored over the course of the competition:
\nbranch_a variants, all three A3 notebooks, C_c2, C_fixed2, C_trust_patch, C_plus_final)In other words, 60% of our offline research architecture never produced a single hard50 data point. The ones that did were either non-graded (40% of the remaining six) or attribution-ambiguous (50% of the remaining six). We learned a lot from reading the traces and tuning heuristics, but we did not run a clean offline benchmark loop, and the result was that the only number we can defend in this paper is the one Branch B multicell saved inline. The methodological finding in Section 4 stands on that single data point plus the nine \"intended for AIMO3, not hard50\" design-intent statements preserved verbatim above.
\nThe full verbatim docstrings and markdown cells quoted in this appendix are reproduced from the actual .ipynb files and can be verified byte-for-byte against the notebook sources supplied with this writeup.
For consistency in this writeup, the team's notebook history has been mapped to canonical names. The mapping is as follows:
\n| Canonical name (used in writeup) | \nKaggle notebook name & version | \nSubmitted? | \nScore(s) | \n
|---|---|---|---|
| Final submission | \nfinal_submission v2 | \nYes (×9) | \n39, 40, 40, 40, 41, 41, 41, 42, 43 | \n
| 42-notebook fork (V11) | \nbase_submission v2 | \nYes (×1) | \n33 | \n
| GRPO submission | \n[15/15] AIME 2026 I 120b in 20mins v4_grpo | \nYes (×1) | \n13 | \n
| Adaptive router | \nAIMO3_Adaptive_Router_Confidence_Vote_v1 | \nYes (×1) | \n28 | \n
| 5-stage AIME scaffold | \n[15/15] AIME 2026 I 120b in 20mins v3 | \nYes (×2) | \n35, 38 | \n
| Strategy-prompt family | \naimo3_final v7 / v9 / v22 | \nYes (×5) | \n36, 37, 37, 37, 36 | \n
| UNDERSTAND/EXPLORE adaptive | \naimo3_adaptive _v4 | \nYes (×1) | \n34 | \n
| Multi-temperature tester | \naimo_tester v2 | \nYes (×1) | \n34 | \n
| Early iteration | \naimo3_final v3 | \nYes (×2) | \n39, 34 | \n
| Debug / temp tweaks | \naimo3_final v4_tweaked_temp&model, v5_tweaked_temp | \nYes (×2) | \n11, 0 | \n
The name final_submission v2 is preserved as-is per team preference. Submissions earlier than 25 days before writeup (3 × submission.parquet uploads at score 0) and one notebook exception (aimo_check v4) are excluded from analysis.
End of writeup. All scores are from the public AIMO3 leaderboard. Configurations are reproduced from the verbatim source code of each submitted notebook. The GRPO telemetry CSV (190 rows) is provided as a supplementary artifact alongside this writeup.
\nS. Frieder et al., \"AI Mathematical Olympiad - Progress Prize 3,\" Kaggle Competition, 2025. https://kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3
I. Moshkov et al., \"AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning Dataset,\" arXiv:2504.16891, 2025.
P. Mali, \"Scaling Inference-Time Compute for Mathematical Reasoning,\" AIMO3 Writeup, Kaggle, 2026. https://www.kaggle.com/code/pawanmali/chasing-47-50-aimo3-journey-of-60-experiments
N. Nitarach, \"Model Capability Dominates: Inference-Time Optimization Lessons from AIMO 3,\" arXiv:2603.27844, 2026.
X. Wang et al., \"Self-Consistency Improves Chain of Thought Reasoning in Language Models,\" ICLR, 2023.
W. Kwon et al., \"Efficient Memory Management for Large Language Model Serving with PagedAttention,\" SOSP, 2023.
OpenAI, \"gpt-oss-120b Model Card,\" 2025. https://huggingface.co/openai/gpt-oss-120b
Following up on the questions from @philipvonderlind (and thanks for the very thorough response), here are some additional questions to help us assess your submission:
\nLook forward to reading your responses on these.
", "votes": 0, "replies": [ { "id": 3458923, "author_name": "Utkarsh Sharma", "author_url": "", "post_date": "2026-05-16T23:59:20.207000", "content": "Thank you @sambealing - excellent questions.
\n1. Sensitivity of the 5-component formula
\nTo answer the direct question first: the weights were not tuned against the public leaderboard. The 5-component structure was adopted from pawanmali's framework, and the specific constants (0.30/0.40/0.20/0.30/−0.10, decay 0.995, thresholds 2.0 and 1.0) were set during development and held fixed across all final_submission v2 runs. They were not iterated based on leaderboard feedback - the same formula was used for all 9 variance-measurement runs. However, since there was no separate held-out validation set for the weights either, the question of whether the formula's gain survives perturbation is still worth testing.
\nTo address this, I built a post-hoc perturbation analysis using 1,500 synthetic problem scenarios calibrated to the AIMO3 difficulty distribution (35% easy with 6-2/7-1 vote splits, 20% medium with 5-3, 10% hard where the correct answer is minority 3-5, 20% exact ties at 4-4, 15% three-way splits at 3-3-2). Entropy profiles were modeled on observed competition patterns: confident solvers (low entropy, especially near conclusion), uncertain solvers (high entropy with spikes), and exploring-then-converging solvers (high early, low late - the profile position weighting is designed to reward). I verified that the sensitivity formula produces identical outputs to the submission code on 4 test sequences before running the analysis.
\nThe script is public at:\nhttps://github.com/utkarshsharma00/aimo3/blob/main/sensitivity_analysis.py
\nKey finding: the exact weights barely matter. What matters is using entropy weighting at all.
\nThe 5-component formula scores 92.5% on the synthetic benchmark. Plain majority vote scores 73.1% - a 19.4-point gap. But all entropy-weighted variants fall within a 3.1-point spread:
\n| Method | \nAccuracy | \nΔ vs default | \n
|---|---|---|
| No std_dev penalty | \n93.9% | \n+1.4% | \n
| Position-weighted only | \n92.7% | \n+0.2% | \n
| Double position weight | \n92.6% | \n+0.1% | \n
| 5-comp (our default) | \n92.5% | \nbaseline | \n
| No streak bonus | \n92.4% | \n−0.1% | \n
| Half position weight | \n92.4% | \n−0.1% | \n
| Mean + position only | \n92.1% | \n−0.5% | \n
| No high_ent_ratio | \n92.1% | \n−0.5% | \n
| Mean entropy only | \n90.8% | \n−1.7% | \n
| Plain majority vote | \n73.1% | \n−19.4% | \n
Even the worst entropy variant (mean entropy only, 90.8%) beats majority vote (73.1%) by 17.7 points. The 3.1pp spread between entropy variants is dwarfed by the 19.4pp gap from majority vote.
\nUnder individual weight perturbation (each weight scaled by 0.0×, 0.5×, 0.75×, 1.25×, 1.5×, 2.0× independently - 30 perturbations total), the maximum answer flip rate was 1.4% (21/1500 scenarios). Total accuracy range across all 30 perturbations: [92.1%, 93.9%] - a spread of just 1.8 percentage points. Most perturbations caused 0-2 flips out of 1500 (≤0.2%).
\nUnder Monte Carlo random perturbation (200 trials, all 5 weights simultaneously perturbed by random factors in [0.5, 1.5]), mean accuracy change was +0.08% with a standard deviation of 0.38% and a range of [−0.33%, +1.73%]. Mean flip rate was 0.2%, max 1.7%. No perturbation caused more than 1.73% accuracy change in either direction.
\nWhere entropy voting concentrates its value (by scenario type):
\n| Scenario type | \n5-comp | \nMajority | \nΔ | \n
|---|---|---|---|
| Easy (6-2 split) | \n100.0% | \n100.0% | \n+0.0% | \n
| Medium (5-3 split) | \n100.0% | \n100.0% | \n+0.0% | \n
| Hard (3-5 minority) | \n21.7% | \n0.0% | \n+21.7% | \n
| Tied (4-4 split) | \n100.0% | \n48.6% | \n+51.4% | \n
| Three-way (3-3-2) | \n100.0% | \n49.5% | \n+50.5% | \n
On uncontested problems (easy/medium), entropy weighting is irrelevant - majority vote already picks the right answer. The entire value concentrates in contested cases: ties, three-way splits, and minority-correct scenarios. This is consistent with our competition experience: on ~35 of 50 problems, all 8 attempts agree and the voting function doesn't matter. On the ~15 contested problems, entropy weighting breaks ties toward the more confident trace.
\nAn unexpected finding: The analysis suggests our formula may be slightly over-engineered. \"No std_dev penalty\" (removing the variance component entirely) scored +1.4% better than the full 5-component formula, and \"position-weighted only\" (dropping all 4 other components) scored +0.2% better. The std_dev component may introduce noise rather than signal. This is a finding we did not expect and would not have discovered without this post-hoc analysis.
\nImportant caveats: This analysis uses synthetic scenarios, not a replay of actual competition logprobs. The entropy profiles are modeled on observed patterns but make a key assumption: that correct traces tend to have lower position-weighted entropy near the conclusion than incorrect traces. This is the intuition behind position weighting (a solver that reached the correct answer through valid reasoning should be confident at the end), and it's supported by multiple independent findings: pawanmali's multi-run evidence (3-6 point advantage for position-weighted over mean entropy across ~8 runs), Potrhino's empirical result that tail entropy (last 2048 tokens) is the strongest single indicator of incorrect reasoning paths on gpt-oss-120b, outperforming global mean entropy (§4.5.1.3 of their writeup), and the DeepConf framework (Fu et al., 2025), which achieves 99.9% on AIME 2025 by filtering traces using token-level confidence. Conversely, rzatemizel's density analysis shows that mean entropy has near-zero discriminative power - which is precisely why we use position weighting rather than mean entropy. The assumption remains unproven on real gpt-oss-120b logprobs for our specific formula, but the directional evidence across these independent analyses is consistent. A proper validation would require logging per-attempt logprobs during competition runs and replaying them with perturbed weights - data I did not collect. The structural conclusion (any entropy weighting >> majority vote; exact weights don't matter much; position weighting > mean entropy) is robust within this simulation framework.
\n2. Offline branches worth resurrecting
\nIf I had more time, I would prioritize Branch C (controller + code rescue + provenance-aware arbitration).
\nThe design addresses a failure class that few submitted approaches directly targeted: what to do when the attempt pool is fragmented (multiple incompatible answer clusters with no majority) or broken (most attempts failed to produce valid answers). AMO's Master Vision pipeline (davidacad10 et al.) is the closest parallel - both detect the absence of consensus and launch a resolution mechanism. Branch C differs in using a deterministic state machine rather than an LLM agent for the detection step.
\nConvergence/divergence state controller. Detects whether the pool has converged (stop early, save budget), fragmented (multiple clusters, need resolution), or broken (most attempts returned no answer). This is strictly better than flat early-stop thresholds because it adapts compute to the observed pool state rather than applying a fixed rule.
\nTargeted code rescue. When text-based reasoning fails, switches to a tool-backed exact computation path. This directly addresses what IceBeam777's RSA writeup calls \"Class 4/5\" problems - where the model converges confidently to a wrong answer and no amount of resampling fixes it.
\nProvenance-aware weighting. Candidates supported by multiple evidence modes (main pool + code rescue + challenge pass) receive higher weight than single-source answers. This is a richer signal than entropy alone.
\nWhat makes Branch C architecturally interesting is how it relates to other competitors' independently developed approaches:
\nvs. ippeiogawa's Blackboard. The Blackboard shares tool I/O between threads during generation, allowing threads to build on each other's computational discoveries. Branch C's code rescue is complementary - instead of sharing intermediate computations between threads, it launches an entirely separate computation mode when text-based reasoning fails. Both address threads getting stuck in wrong reasoning frames, but from different directions: Blackboard prevents getting stuck by sharing information; code rescue recovers after getting stuck by switching modality.
\nvs. rzatemizel's Judger. The 3-base/5-judger architecture evaluates completed reasoning traces to select the best answer. Branch C's provenance-aware arbitration is similar in spirit - both use evidence quality rather than raw vote count to weight answers. The key difference is that rzatemizel's judger is a separate LLM call that evaluates the full reasoning trace, while Branch C's provenance weighting is a lightweight, deterministic function of which evidence modes support each answer - far less compute cost but less nuanced information about reasoning quality.
\nvs. AMO's Master Vision (davidacad10 et al.). AMO introduces a Master Vision agent that synthesizes findings from competing solvers when no majority exists - essentially a conflict-resolution pipeline. Branch C's convergence controller serves the same role but as a deterministic state machine rather than an LLM agent, avoiding the latency and hallucination risk of another LLM call while sacrificing the flexibility of natural-language synthesis.
\nThe reason Branch C didn't ship was infrastructure: wiring bugs in the multi-stage pipeline prevented end-to-end execution on Colab. With another week of debugging time, I believe the state controller alone would have been worth integrating - it addresses time allocation more intelligently than any flat threshold. But in a time-constrained competition, architectural ambition must be balanced against execution reliability. Our final submission won not because it was the best architecture we designed, but because it was the best one we shipped.
\nHappy to provide additional detail on any of these points.
", "votes": 0, "replies": [ { "id": 3463165, "author_name": "Utkarsh Sharma", "author_url": "", "post_date": "2026-05-26T05:01:58.780000", "content": "Note: After posting this analysis, the 2nd-place solution writeup 'Vote-dominant Ensembling and a Sharper System Prompt' by @varianceofx was published. Their \"tail-windowed entropy\" (Improvement 3) independently converges on the same design as our position weighting - using entropy over only the last 256 tokens rather than the full trace, with the same reasoning that early-trace entropy reflects exploration, not answer confidence. Their ensemble design (Improvement 5) also independently confirms our sensitivity finding: they weight vote_share at 2.0 and confidence at only 0.3, concluding that \"entropy is best used as a tiebreaker, not as the main criterion.\" Two independent teams arriving at the same architectural conclusion strengthens the case that position/tail entropy > mean entropy, and that exact weights are secondary to using entropy weighting at all.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3457396, "author_name": "Utkarsh Sharma", "author_url": "", "post_date": "2026-05-13T17:44:36.593000", "content": "Edited on May 13th 2026 for the rebuttal phase:
\nChanges made:
\nNo architectural claims, experimental results, or methodological conclusions were altered. The private LB result validates the variance methodology described in Section 3 - our pre-registered 95% CI correctly predicted the out-of-sample private score.
", "votes": 0, "replies": [] }, { "id": 3456831, "author_name": "Philip Vonderlind", "author_url": "", "post_date": "2026-05-12T16:43:16.203000", "content": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.
\nI have a few questions regarding some of the ideas proposed and implemented in your writeup:
\nGRPO: Other contestants, notably @outliar, have reported successful gains using GRPO. You mention that the observed improvements in your pipeline may be attributable to noise, which could certainly also apply to other submissions. How would you explain the GRPO gains reported by other contestants in comparison to your own training results?
Local Testing: Most of your ablations appear to rely on single public leaderboard experiments. Did you also attempt to verify these changes locally, for example, using one of the datasets from the Math Corpus Prize entries? Reporting score distributions across repeated runs, along with variance estimates, would make the results easier for others to interpret.
Entropy: Did you verify that entropy-based voting actually correlates with more mathematically correct traces? Some other writeups have reported that model confidence does not necessarily correspond to trace correctness.
Looking forward to your response :)
", "votes": 0, "replies": [ { "id": 3456863, "author_name": "Utkarsh Sharma", "author_url": "", "post_date": "2026-05-12T17:24:06.790000", "content": "Hi @philipvonderlind - thank you for the detailed questions. Happy to address each one.
\n1. GRPO: Reconciling our failure with outliar's reported gains
\nThe critical distinction between our GRPO attempt and outliar's is the deployment path, not the training itself. Our GRPO v6 loop trained stably for 38 cycles with zero loss collapses - the training worked. The failure was at deployment: we used Unsloth's merged_4bit_forced to merge the adapter, which produces BitsAndBytes NF4 weights. gpt-oss-120b's native format is MXFP4. vLLM routes NF4 weights through a different (degraded) kernel path, producing the 13/50 catastrophic score. outliar avoided this by using save_method=\"mxfp4\" - the correct deployment path.
However, even when GRPO deployment succeeds, the gains appear modest relative to inference-only optimization. outliar's final fine-tuned model (SFT + GRPO + ORPO + MXFP4 bake) achieved μ=38.57 (σ=1.67, n=35) - which is 2.2 points below our inference-only μ=40.78 (σ=1.20, n=9). Their pre-ORPO baseline was μ=36.6, so the total fine-tuning gain was approximately +2 points. Meanwhile, Inoichan et al. (\"Pushing the Limits: Post-Training High-Capability Models under Strict Inference\" writeup, 10 authors, 32 H100s) ran full-parameter SFT for 10 epochs on 10.37 billion tokens and improved pass@1 by only +0.77 percentage points - while actually reducing coverage (Pass@16) from 92.28% to 90.88%. Their RL (GRPO variant) showed a similar pattern: +1.0 pass@1 but −2.11 coverage.
\nSo our position is nuanced. We do NOT claim GRPO cannot help - outliar's +2 is real. We claim that (a) the deployment pipeline is a documented trap that cost us $300 and that future participants should be aware of, and (b) even when fine-tuning succeeds, the gains are small relative to inference-only optimization on a base model this strong, and coverage (the model's solvable problem space) may actually shrink with training. The latter finding from Inoichan et al. is the strongest independent evidence supporting our thesis.
\n2. Local Testing: Variance estimates and offline verification
\nYou're right that many of our architectural ablations (strategy prompts, 5-stage scaffold, router, etc.) relied on single or few-shot public leaderboard comparisons. This is the most significant methodological limitation of our writeup, and we acknowledge it.
\nHowever, we did two things that go beyond single-shot LB comparison:
\nThe n=9 variance experiment. We submitted final_submission v2 nine times without modification, producing scores {39, 40, 43, 40, 42, 40, 41, 41, 41} with μ=40.78 and σ=1.20. This established our noise floor: any Δ < 2σ ≈ 2.4 points is indistinguishable from running the same notebook twice. This measurement is what allows us to claim the strategy-prompt family (5 submissions, mean ~36.6) is directionally negative (Δ ≈ −4.2, outside the ±2σ band) rather than noise.
Offline evaluation on hard50. We ran 15 offline research branches against hard_50_math_problems_set_v6.csv (zfturbo's dataset from the Math Corpus Prize). One branch - aimo3_branch_B_multicell - completed a full 50-problem run with inline grading: 27/50 (54.0%), mean 233s/problem. This is the only verified offline number we cite, and we document honestly that 9 of our 15 offline branches never executed to completion. The offline work is detailed in Section 4 and Appendix H of the writeup.
\nWe did not use the offline benchmark for per-variant ablation (e.g., running each of our 11 configurations on hard50 multiple times) - which would have been the methodologically stronger approach. The reason was practical: each hard50 run took ~3.5 hours on Colab, and we prioritized leaderboard submissions over systematic offline testing. In hindsight, this was the wrong tradeoff - our Lesson #1 in Section 9 is exactly this: \"Establish the noise floor first. Submit the same notebook 5-10 times before iterating on architecture.\"
\nFor comparison, sebastiangil00's writeup (\"Three Substrate Traps on gpt-oss-120b-MXFP4\") tested 16 variants on a 199-problem validation set with McNemar paired tests and Benjamini-Hochberg correction - a substantially more rigorous methodology than ours. We consider their statistical approach the gold standard for future competitions.
\n3. Entropy: Does it actually correlate with correctness?
\nThis is the hardest question, and the honest answer is: we don't have conclusive evidence that our specific entropy formula correlates with trace correctness in isolation.
\nWhat we know:
\nMean entropy does not discriminate. rzatemizel's density plots (\"Fighting Variance\" writeup) show correct answers have mean entropy 0.7053 vs wrong answers 0.7503 - a gap of 0.045 on almost completely overlapping distributions. Schwyter (arXiv:2603.27844) also found that model confidence does not correspond to trace correctness, as you reference. We agree with this finding for mean entropy.
\nOur formula is structurally different from mean entropy. The 5-component weighted entropy uses position weighting (40% of the score, exponential decay 0.995 favoring tokens near \\boxed{}), variance penalty, high-entropy ratio, and low-entropy streak bonus. The key intuition is that a solution can be uncertain during exploration but should be confident at the conclusion - position weighting captures this, while mean entropy averages over the entire trace and loses the signal.
rzatemizel confirmed they did not test position-weighted variants. In their reply to our comment on their writeup, they stated: \"I did not check position-weighted entropy or other similar variants, because I lost confidence in logprobs fairly early.\" So the density - overlap finding applies to mean entropy specifically, not necessarily to our position-weighted composite.
\npawanmali provides the strongest supporting evidence. In their reply to our comment, they confirmed: \"mean entropy consistently underperformed position-weighted entropy by 3-6 points across ~8 different runs where that was the primary variable.\" This is multi-run directional evidence that position-weighted entropy captures something that mean entropy does not.
\nOur own comparison is confounded. We changed the voting function AND stripped the scaffolding simultaneously, so we cannot isolate which change was load-bearing. We state this explicitly in the writeup (Section 6, final paragraph). The honest answer is: we don't know whether the entropy formula or the scaffold removal (or both) drove the improvement from ~36.6 to 40.78.
\nThe bottom line: mean entropy is probably not a useful discriminator (confirmed by rzatemizel and Schwyter). Position-weighted entropy may retain discriminative power at the conclusion of the reasoning trace, supported by pawanmali's multi-run evidence but not conclusively proven by our own data. We consider this an open empirical question that deserves a controlled ablation we were unable to run before the deadline.
\nHappy to provide additional detail on any of these points.
", "votes": 0, "replies": [] } ] }, { "id": 3454392, "author_name": "Utkarsh Sharma", "author_url": "", "post_date": "2026-05-07T03:06:55.633000", "content": "Reproducibility Package
\nModel: danielhanchen/gpt-oss-120b (HuggingFace, unmodified - no fine-tuning, no LoRA, no weight patches)\nhttps://huggingface.co/danielhanchen/gpt-oss-120b
\nKaggle submission notebook (final_submission v2):\nhttps://www.kaggle.com/code/flamingice2801/final-submission
\nGoogle Colab notebook (Kaggle-agnostic):\nhttps://colab.research.google.com/github/utkarshsharma00/aimo3/blob/main/colab_inference.ipynb
\nStandalone inference script:\nhttps://github.com/utkarshsharma00/aimo3/blob/main/run_inference.py
\nFigure generation script (reproduces all 9 figures):\nhttps://github.com/utkarshsharma00/aimo3/blob/main/generate_figures.py
\nFull repository:\nhttps://github.com/utkarshsharma00/aimo3
\nThe entire contribution is inference-side - no custom model artifacts. The Colab notebook pulls the stock model from HuggingFace and contains the full pipeline (solver, 5-component entropy voting, sandbox pool, streaming parser, adaptive time management). Requires an A100/H100 80GB runtime.
", "votes": 0, "replies": [] }, { "id": 3446698, "author_name": "Utkarsh Sharma", "author_url": "", "post_date": "2026-04-22T00:58:51.433000", "content": "Note added after reviewing concurrent writeups (April 21, 2026)
\nSeveral independent writeups have now been published that converge on the same central finding documented here. Schwyter (arXiv:2603.27844, 23 experiments across 3 models) found that model capability dominates all prompt-level optimizations by 4×. @merkiraz (11 controlled submissions) found every modification preserved or degraded performance. @kolenkovskin (20+ dead ends across 28 submissions) independently concluded that \"subtraction beats addition.\" @potrhino (7 controlled ablation runs on a 75-problem validation set) documented positive contributions from TIR and dynamic time management but found that the remaining gap is compute-constrained. This convergence across 5+ independent teams - each arriving at similar negative conclusions through different experimental paths - is itself evidence that the finding is robust.
\nRegarding selection loss: the host's pass@N analysis shows pass@8 ≈ 44 on the private set, meaning the correct answer appears in at least one of our 8 attempts for ~44 of 50 problems. Our mean of 40.78 implies we correctly select the right answer from the pool ~93% of the time. The remaining ~3 problems per run where we have the correct answer but select wrong represent the selection loss that our 5-component entropy voting addresses but does not fully close. @rzatemizel density-overlap analysis suggests that mean entropy has near-zero discriminative power between correct and incorrect answers; whether our position-weighted variant retains discriminative power where mean entropy does not remains an open empirical question.
", "votes": 0, "replies": [] } ], "comment_count": 8, "messages": [ { "id": 3444258, "postDate": "2026-04-17T14:45:38.060Z", "content": "Team: flamingice (solo) · Competition: AI Mathematical Olympiad - Progress Prize 3 · Submission for the Writeup Prize
\nWe report a negative-results-dominated investigation of inference-time and training-time interventions for gpt-oss-120b on the AIMO3 competition. Our private-leaderboard submission, final_submission v2, achieved a public-leaderboard mean of 40.78 (σ = 1.20, n = 9 identical runs, best 43) and a private-leaderboard score of 41.5/50 using parallel self-consistency over 8 attempts with a 5-component weighted-entropy voting function and a one-sentence system prompt. We submitted 25 successful runs across 11 distinct notebook configurations over 25 days. With one exception, every architectural addition we attempted - multi-strategy prescriptive prompts, structured 5-stage scaffolding, subject-routed multi-stage arbitration with verifier/GenSelect/adversarial passes, and GRPO LoRA fine-tuning - either fell within run-to-run noise or fell catastrophically below it. The most expensive failure (≈$300 of B200 GPU time on GRPO experiments) is documented with 190 rows of training telemetry and traces to a known Unsloth merged_4bit_forced → vLLM MXFP4 deployment pipeline incompatibility. Our central methodological contribution is the noise floor: with σ = 1.20 across identical runs, any apparent improvement smaller than ≈2.4 points cannot be distinguished from running the same notebook twice. We argue that for inference-constrained math competitions, this noise dominates almost all engineering decisions, and that future participants should establish their own variance baselines before iterating on architecture.
The AIMO3 competition required solving 50 private-set olympiad-level problems with non-negative integer answers in [0, 99999], inside a single Kaggle notebook constrained to ≈5 hours of H100 GPU time. The runtime constraint transforms the optimization target: the goal is not \"maximize per-problem accuracy\" but \"maximize the joint expectation of accuracy and successful completion under a wall-clock cliff.\" This distinction kills nearly every intuition borrowed from offline pass@N experiments.
\ngpt-oss-120b (released in 2025, 117B total parameters with ≈5.1B active per token via mixture-of-experts, native MXFP4 weights) is strong enough on olympiad math that the AIMO3 leaderboard is plausibly bottlenecked by inference orchestration rather than reasoning capacity. Public community notebooks built on gpt-oss-120b were reported to score in the high 30s and low 40s out of 50 with relatively simple self-consistency setups. Against that background, our research question was: starting from a strong base model, where do additional engineering investments actually move the score?
We tested four classes of intervention:
\nEvery one of them either left the score within statistical noise of where we started, or made it dramatically worse. The single intervention that produced a stable mean above the noise floor was a 5-component weighted entropy formula for self-consistency voting, combined with deliberate stripping of every other piece of scaffolding. This paper documents what we tried, why we tried it, and - more importantly - why the things that didn't work didn't work.
\nReproducibility note. All code, prompts, and configurations are reported exactly as run. The full submission history (Appendix C), the GRPO training telemetry CSV (Appendix D), the offline research branch catalog (Appendix H), and the verbatim source of every notebook discussed here are available with this writeup.
\nWe made 25 successful submissions across 11 notebook configurations between competition entry and the April 15 deadline. Figure 1 shows the chronological trajectory.
\n
Three structural features of this trajectory motivate the rest of the paper:
\nThe ramp-and-plateau shape. Scores rise from the high 20s to the high 30s as obvious bugs are removed, then plateau in a narrow band centered around 40 with substantial variance. No single architectural change moves us out of this band on the upside.
\nThe catastrophic dips. Three submissions fall well below the band: GRPO at 13, the adaptive router at 28, and the V11 timeout at 33. Each represents a different failure mode (deployment pipeline corruption, time-budget collapse from per-problem complexity, raw wall-clock overrun). All three are documented in detail below.
\nThe final cluster. The last nine submissions on the right are repeated runs of final_submission v2 - the same notebook, the same seed, the same code - submitted to measure run-to-run variance. They span 39 to 43 with a mean of 40.78 and σ = 1.20.
This last cluster is the load-bearing observation of the entire writeup. We turn to it next.
\nWe submitted final_submission v2 nine times without modification. The scores were:
\n\n39, 40, 43, 40, 42, 40, 41, 41, 41
\n
Mean = 40.778. Median = 41. σ = 1.202. Range = 4 points. 95% CI for the mean = [39.85, 41.70]. Figure 2 shows the distribution.
\n
This variance has two sources we cannot fully separate. First, the gpt-oss-120b decoder produces non-deterministic output even with a fixed seed, because the parallel attempts are issued asynchronously to a vLLM server whose batching schedule depends on real-time CUDA kernel timing and KV-cache state. Second, Kaggle's grading is deterministic but the scheduling of which problems hit which attempt order is not. Whatever the underlying mechanism, the operationally relevant fact is that nine runs of identical code spanned four points.
\nThe implication for our research methodology is severe and one-directional:
\n\n\nAny single comparison between two notebooks that differs by less than ≈2σ ≈ 2.4 points is statistically indistinguishable from running the same notebook twice.
\n
In retrospect, the most important methodological mistake we made early in the competition was treating single-shot leaderboard scores as ground truth. When aimo3_final v9 returned 37 three times in a row and [15/15] AIME v3 returned 35 and 38, we read those as meaningful differences and chased the apparent winner. They are not meaningful differences. They are draws from the same noisy distribution.
We did not realize this until the final week, when a deliberate variance experiment on the converged notebook produced the n = 9 distribution above. Had we run such an experiment on day one, we would have saved ≈$300 of GPU spend and three weeks of architectural iteration. This is recommendation #1 of the paper: future AIMO3-style participants should establish a noise floor before iterating on architecture.
\nThroughout the competition we maintained an offline evaluation loop against the Kaggle community dataset hard_50_math_problems_set_v6.csv (published by user zfturbo as hard-math-problems-for-aimo-3) - a 50-problem benchmark of olympiad-level questions with known integer answers. Offline evaluation served as a debugging sanity check and as a relative-performance gauge for architectures we were not yet willing to spend a submission slot on.
We authored 15 offline research branches during the competition, organized into five families, each targeting a specific failure class observed in earlier runs:
\nNone of these 15 branches were ever submitted to the Kaggle leaderboard. Of the 15, nine were never executed to completion due to infrastructure and wiring bugs. Two contested-resolver branches ran 50 problems on hard50 but never logged per-problem correctness. Three Branch B/C variants share a single stale truncated run (cumulative 19/29 at problem 30) that we cannot attribute to a specific architecture because the three notebooks have different code but identical saved outputs from one Colab session. One notebook - aimo3_branch_B_multicell - completed a full, inline-graded 50-problem hard50 run. That is the only offline number we can cite with full verification. Full per-branch execution status is catalogued in Appendix H.
Branch B multicell on hard50: 27/50 (54.0%), mean 233 s/problem, total 194 min. Figure 9 shows the per-problem timing (top) and cumulative accuracy (bottom). No problem hit the 900 s cap; the run finished comfortably inside a notional 5-hour budget at 3.24 h total.
\n
Accuracy begins at 100% after problem 2, collapses to the mid-60s by problem 10, and bleeds out to 54% by problem 50. The architecture looked competitive early and degraded on the harder back half. Since no problem timed out, 27/50 reflects answer quality, not budget exhaustion.
\nInterpretation is constrained. Branch B multicell's elaborate pairwise-arbitration architecture scored 27 on hard50; final_submission v2 scored a mean of 40.78 on AIMO3. If hard50 difficulty were comparable to AIMO3, Branch B would project to ~27 on AIMO3 - 14 points below the simpler approach. But hard50 is not AIMO3: it is community-sourced, intentionally difficult, and may systematically underestimate AIMO3 performance - which is precisely why Branches A3 and C+ were explicitly designed \"not for hard50-only tuning.\" We also never ran a variance experiment on hard50 analogous to our n = 9 experiment, so we cannot say whether 27/50 is itself ±2 points of noise.
Even with those caveats, the directional signal was clear: our only offline-executed elaborate architecture predicted a losing Kaggle score, and the simpler baseline outperformed it. The writeup-prize rubric asks \"did you use any internal benchmarks?\" - yes, but less effectively than we should have. Nine of our 15 research notebooks never finished a run; we never calibrated hard50 against AIMO3; and the one completed offline run told a story that should have killed Branch B's lineage earlier than it did. Future participants: instrument your offline benchmark the way you instrument your leaderboard submissions, get all your research notebooks across the finish line, and run a variance experiment on the offline set before treating any single offline number as signal. Appendix H lists all 15 branches with design intent and execution status.
\nfinal_submission v2final_submission v2 is intentionally lean. It does not contain a 5-stage prompt, a router, a verifier, an adversarial pass, GenSelect, multi-strategy prescriptive prompts, or fine-tuned weights. It contains four things. First, the base model: danielhanchen/gpt-oss-120b served by local vLLM (KV-cache fp8_e4m3, gpu_memory_utilization 0.96, max_model_len 65,536). Second, a one-sentence system prompt: \"You are a world-class IMO competitor. The final answer must be a non-negative integer between 0 and 99999. Place your final answer inside \\boxed{}.\" The model is invoked through the openai-harmony format with ReasoningEffort.HIGH, the gpt-oss native reasoning control. Third, tool-augmented parallel self-consistency: 8 attempts dispatched in parallel against a pool of 16 long-lived Jupyter sandboxes (math, numpy, sympy, mpmath, itertools, collections; mpmath.mp.dps = 64), up to 128 harmony turns each, T = 1.0, min_p = 0.02, no top_p, per-attempt seed (global_seed + attempt_index)². Streams are inspected character-by-character; an attempt short-circuits the moment a \\boxed{...} appears. Fourth, 5-component weighted-entropy voting (Section 5.2); when 4 attempts agree, remaining attempts are cancelled.
The time budget is computed adaptively per problem: min(max(time_remaining − 270 × problems_remaining, 270), 900) seconds, with notebook_limit = 17400 (4h50m, leaving safety margin inside Kaggle's 5h cliff). Appendix A contains the full configuration.
Inverse-entropy voting is a known technique: weight each attempt by 1/H, sum weights per answer, return the highest-weighted answer. The novelty in final_submission v2 is that the entropy itself is a five-part composite over the attempt's logprob trajectory, rather than a single mean. Reading from the source:
weighted_entropy = (\n 0.30 * mean_entropy\n + 0.40 * position_weighted_entropy # exponential decay 0.995, recent tokens dominate\n + 0.20 * std_dev_of_entropy # variance penalty for inconsistent confidence\n + 0.30 * 3.0 * high_entropy_ratio # fraction of tokens with H > 2.0 bits, scaled ×3\n - 0.10 * (max_low_entropy_streak / n) # bonus for sustained confident stretches\n)\nThe intuition: position weighting with decay 0.995 means tokens near the final boxed answer count exponentially more than tokens early in the chain of thought - a solution uncertain during exploration but confident at conclusion outranks one that is confident throughout but flinches at the end. The variance penalty punishes oscillating confidence (\"the model kept changing its mind\"). The high-entropy ratio counts tokens where the next-token distribution exceeds 2 bits of Shannon entropy, and the ×3 multiplier inside a 30% weighting makes this term dominant when an attempt has many uncertain stretches. The streak bonus rewards sustained runs of < 1 bit entropy as a marker of \"the model is on a roll.\"
\nThese weights and thresholds are not from a paper. They were tuned by hand on intermediate submissions in this competition. The core idea of inverse-entropy weighted voting was informed by pawanmali's \"Chasing 47/50\" notebook, which demonstrated entropy-based confidence weighting for self-consistency on AIMO3. We do not claim they are optimal - only that this composite produced a stable mean of 40.78 in final_submission v2, while simpler \"mean entropy\" voting in eight other notebooks we submitted produced means in the 33–37 region. Section 6 unpacks that comparison.
final_submission v2For the avoidance of doubt: final_submission v2 does not contain the 5-stage UNDERSTAND/EXPLORE/PLAN/EXECUTE/VERIFY scaffold (tested in [15/15] AIME v3, scored 35 and 38), the 8 prescriptive strategy prompts from the aimo3_final family (five submissions, mean ~36.6), the verifier/GenSelect/adversarial scaffolding from the same family, the subject-classifier routed prompts from aimo3_adaptive v4 (scored 34), the full adaptive router with posterior voting (scored 28), the increased attempt count from base_submission v2 (scored 33, timed out), or the GRPO fine-tuned weights from the v4_grpo submission (scored 13, merge format mismatch). Figures 7 and 8 visualize the same information at the feature and score levels respectively.
Figure 8 collects every intervention we tried as a horizontal bar chart, compared against the baseline of 39 (the first run of aimo3_final v3, which is the earliest non-debugging submission still in our history).
Three regions: inside the noise band - early aimo3_final iterations, the multi-strategy prompt family, and the 5-stage scaffold sit within ±2σ of the 39 baseline and are indistinguishable from starting fresh (the strategy-prompts family was tested in five submissions and never broke out). Below the noise band - the structured-scaffold-plus-classifier branch (−5), V11 timeout (−6), the router with GenSelect/adversarial pass (−11), and GRPO (−26); all real failures with identifiable causes that Section 7 dissects. Above the noise band - only final_submission v2 at +1.78, with the lower bound of its 95% CI (39.85) itself above baseline. It is the only intervention in the entire competition with statistically defensible upward movement.
Figure 7 makes the same point feature-by-feature. Each row is a notebook, each column a feature; checkmarks indicate presence.
\n
final_submission v2 (top row) has fewer features than every other notebook except the most basic, and is the only one with the 5-component weighted entropy. Notebooks with strategy prompts cluster around 36–37; notebooks with verifier/GenSelect/adversarial scaffolding cluster around 28–37; the router is at 28; GRPO at 13. Adding architecture moved us downward; subtracting it and replacing the voting function moved us upward. The honest causal claim we can make is that stripping the scaffolding AND replacing the voting function with 5-component weighted entropy together produced a defensible improvement above noise. We cannot say which change was load-bearing on its own - that would require an isolated ablation we did not run before the deadline.
The aimo3_final v7/v9/v22 family launches eight parallel attempts, each with a different prescriptive system prompt instructing the model to use a specific mathematical technique - analytic, computational, small-cases, work-backwards, algebraic, modular, geometric, random-testing-and-verification. The hypothesis was that prompt-induced strategy diversity would produce a more diverse candidate pool for self-consistency voting to exploit, extending the DIPPER intuition that framing diversity beats temperature diversity alone.
It failed because gpt-oss-120b at ReasoningEffort.HIGH complies with prescribed techniques even when they are poorly suited - a geometry problem given a \"modular arithmetic\" prompt produced attempts that contorted themselves toward modular methods and failed. The base model already selects appropriate techniques at T = 1.0; external prescription overrides this selection in the wrong direction about as often as the right one. Across five submissions, the mean was approximately 36.6 - Δ ≈ −2.4 vs. baseline, just inside the lower noise edge but directionally negative. The methodological distinction we carried forward: prescribing solution content hurts, prescribing reasoning form (Section 7.2) is roughly neutral.
[15/15] AIME v3, scored 35 and 38)The [15/15] AIME 2026 I 120b in 20mins notebook v3 uses the full 5-stage UNDERSTAND → EXPLORE → PLAN → EXECUTE → VERIFY scaffold popular in CoT prompting research, with a verbose ~1500-character system prompt detailing reasoning principles and verification requirements. The hypothesis was that even a strong base model would benefit from being forced to plan before executing.
The two submissions scored 35 and 38 - squarely inside the ±2σ noise band of baseline. We cannot reject the null hypothesis that the scaffold has zero effect on gpt-oss-120b. A plausible explanation is that ReasoningEffort.HIGH already imposes a reasoning structure that the prompt-level scaffold duplicates without adding signal. Technique transfer from smaller-model prompting research does not automatically apply to gpt-oss-120b at high reasoning effort.
AIMO3_Adaptive_Router_Confidence_Vote_v1, scored 28)A multi-stage system: a subject classifier routed each problem to one of four prompt families (geometry, number theory, combinatorics, algebra); a \"quick stage\" issued 2 cheap attempts (T = 0.55 and 0.80, max 2200 tokens, 24 turns); if those agreed with high posterior confidence, return immediately, otherwise an \"explore stage\" issued more attempts at higher temperature with longer budgets; weak candidates were routed through verifier, GenSelect, and adversarial passes; high_problem_timeout = 840 s per problem. The hypothesis was the standard one: easy problems should consume little budget, hard problems more; an adaptive system should outperform a flat-budget system under heterogeneous difficulty.
It failed structurally. The first problem consumed 840 s. The second consumed approximately 825 s. The router had burned 28% of the entire 5-hour notebook budget on two problems, and the remaining 48 problems shared the rest. Many timed out or returned 0. Final score: 28 - fully twelve points below baseline, the second-largest single-intervention drop in the competition. The router's adaptive logic correctly identified problems 1 and 2 as hard and allocated them more time; it then found nothing useful in the extra time. The correct behavior under those conditions is to give up early and conserve budget for problems where extra time might actually help - we did not implement that. A v2 with reduced timeouts reached 19/30 on an offline slice but we did not submit it; the converged final_submission v2 was already outperforming it. Adaptive complexity is only valuable when the marginal value of extra time on a hard problem is positive. For hard olympiad problems in the AIMO3 distribution against gpt-oss-120b, that marginal value appears to be approximately zero past 5–6 minutes.
base_submission v2, scored 33)This is the team's fork of the public AIMO 3 | GPT-OSS-120B (with tools) notebook (\"the 42-notebook\" in community discussion), with three modifications: attempts raised from 8 to 12, early_stop raised from 4 to 5, and a \"Be concise. Stop immediately after \\boxed{}\" instruction added. The notebook also uses the stock OpenAI gpt-oss-120b weights rather than Daniel Hanchen's variant. Hypothesis: more samples → stronger voting → higher score, up to the point where the time budget binds.
It exceeded the 5-hour budget. Twelve attempts × 50 problems with a verbose chain-of-thought trajectory pushed total runtime past the cliff; problems processed after timeout returned 0. Final score: 33. The experiment is confounded because we changed three variables at once and cannot cleanly attribute the −6 to \"more attempts\" alone. Operational lesson: if you push N upward, run a wall-clock experiment first, and budget for the slowest model variant.
\nThis is the largest and most expensive negative result in the writeup, and the most generally useful for the community. We approached it in two phases.
\nPhase A: TRL GRPOTrainer (collapsed at step 16, $101.73 sunk). Our first attempt used Hugging Face TRL's GRPOTrainer with Unsloth on a RunPod B200 ($5.03/hr). Configuration: LR 5e-6, max_grad_norm 1.0, β = 0.0, 8 generations per problem, use_vllm=False (because TRL's vLLM integration crashed at startup against gpt-oss-120b). Each step took ≈76 minutes. The training collapsed:
Figure 6 shows the trajectory.
\n
The diagnosis is straightforward: LR 5e-6 was too aggressive for an MoE model at 117B parameters, and max_grad_norm 1.0 was too loose to clip the gradient explosions that began once the policy started drifting from the reference.
Phase B: Custom vLLM + Unsloth loop (v6) - trained stably, deployed catastrophically. We rebuilt the training loop from scratch to avoid TRL's vLLM integration: each cycle launched vLLM with the current adapter, generated 8 completions per problem for 5 problems, killed vLLM to free GPU memory, applied a GRPO update via Unsloth, saved the adapter, restarted. We tightened the configuration substantially (LR 2e-6 with cosine decay, max_grad_norm 0.3, Dr.GRPO advantage normalization, DAPO-style overlong filtering, binary reward, β = 0, off-policy generation with vLLM on the base model and adapter trained separately; see Appendix B for the full config).
\nWe have telemetry for 190 problems across 38 cycles (cycles 62–99, problems 310–499). The training was stable - zero loss collapses, loss values mostly in [0, 0.3], gradient norms cleanly bounded by the 0.3 clip. Figure 3 shows the per-cycle reward trajectory:
\n
The reward signal tells the second half of the story. The overall mean reward across all 190 problems was +0.033 - barely above zero, after 38 cycles of training. Only 4 problems out of 190 (2.1%) achieved reward 1.0 (all 8 generations correct): problems 329, 333, 379, 468. Three of those four had near-zero completion length, suggesting trivially short correct answers. Figure 5 shows the full reward distribution:
\n
Forty-seven percent of training problems received net negative reward. Eighteen percent received exactly zero (all 8 generations wrong, or all 8 right) - these are filtered from the gradient by Dr.GRPO's advantage normalization, contributing nothing. Thirty-three percent were partial wins. Only 2% were complete wins. This is the hardest possible shape for GRPO: the model is below 50% on the training distribution, and the reward signal is sparse and noisy. Even a stable training loop cannot extract much policy improvement from this.
\nThe deployment failure. We merged the trained adapter using Unsloth's merged_4bit_forced method. The submission scored 13/50 - a 27-point drop from baseline. The root cause is a quantization format mismatch: gpt-oss-120b's native weight format is MXFP4 (E2M1 with E8M0 block scaling), but merged_4bit_forced produces BitsAndBytes NF4. vLLM routes the merged weights through its BitsAndBytes loader rather than the MXFP4 kernels, producing severely degraded output. The fix - which we discovered post-mortem from Unsloth documentation and vLLM GitHub issues #19361 and #21932 - is to merge with merged_16bit or save with save_method=\"mxfp4\". A community member elsewhere in the AIMO3 forum reportedly spent $500 on SFT+RL with a similar pipeline and scored 36, attributing it to the same issue. We spent approximately $300 total on GRPO across all phases (Figure 4): TRL crash $102, v1–v5 debugging $100, v6 $62, idle $40.
The actionable takeaway: never use merged_4bit_forced to deploy a fine-tuned gpt-oss-120b LoRA to vLLM. Use merged_16bit, save_method=\"mxfp4\", or NVIDIA's QAT pipeline.
We benchmarked exclusively against the open-weight setting because that is what AIMO3's hardware budget allows. The relevant SOTA points are:
\ngpt-oss-120b for AIMO3: Reported in the high 30s to low 40s with simple self-consistency setups.The host shared pass@N analysis on X showing gpt-oss-120b achieves pass@8 ≈ 44 and pass@20 ≈ 45.5 on the private set (95% CI [43, 48]), indicating that the correct answer appears in at least one of 8 attempts for approximately 44 of 50 problems. Our mean of 40.78 implies we correctly select the right answer from the pool approximately 93% of the time (40.78/44), with the remaining ~3 problems per run representing pure selection loss.
\nThe structural shift from AIMO2 to AIMO3 is striking. AIMO2 was won by a team that invested heavily in training-time compute and produced a 14B-parameter model that scored 34. AIMO3 allows H100 hardware, which makes gpt-oss-120b runnable, and the unmodified base model with a sensible voting function reaches 40.78 - six points above the AIMO2 winner with zero training-time investment. Base model capability dominates the AIMO3 landscape in a way it did not dominate AIMO2. Our negative GRPO result is consistent with this: training-time interventions face an uphill battle against a base model that is already near the ceiling of the inference-constrained accuracy distribution.
We did not benchmark against commercial models (GPT-4, Claude, Gemini) because the AIMO3 rules require self-contained Kaggle notebook execution and commercial API calls are not permitted in the competition pipeline.
\nThe competition produced eight findings we believe will help future participants more than another point on the leaderboard would.
\nEstablish the noise floor first. Submit the same notebook 5–10 times before iterating on architecture. σ = 1.20 means any 1–2 point \"improvement\" is invisible to single-shot comparison. Every other recommendation depends on this one.
Per-problem time budgets are absolute. The 5-hour cliff is hard-enforced. Any architecture that can spend more time on a problem must have a hard cap well below what the budget would technically allow - hard problems do not pay you back for extra time.
Strategy-prompt diversity hurts a strong base model. When gpt-oss-120b at ReasoningEffort.HIGH is told to use a specific technique, it complies even when the technique is wrong. Sampling-temperature diversity at T = 1.0 produces enough variance for self-consistency voting without prescriptive prompts.
Confidence estimation matters more than sample count. Our single defensible improvement above noise came from changing the voting function, not from generating more samples. Treat the voting function as a first-class optimization target.
merged_4bit_forced is a trap for gpt-oss-120b. The Unsloth → vLLM pipeline silently produces a BitsAndBytes-quantized model that vLLM cannot run efficiently. Use merged_16bit or save_method=\"mxfp4\".
Stable GRPO training is necessary but not sufficient. Our v6 loop trained stably for 38 cycles with no collapses, but the deployed model scored 13 because of the merge format issue, and the reward signal during training (mean +0.033) was too sparse for meaningful policy improvement.
Instrument your offline benchmark like your leaderboard. We ran 15 offline research branches but only one completed a verifiable end-to-end run. We never calibrated hard50 difficulty against AIMO3. Subtraction beats addition: every architectural addition we tested was neutral or harmful.
Document the negative results. We invested ~$300 of B200 GPU time and three weeks of architectural iteration in directions that did not work. If this writeup helps even one future team avoid the merged_4bit_forced trap or the strategy-prompt-diversity dead end, it has paid for itself many times over.
We did not win AIMO3. Our private-leaderboard score of 41.5/50 and rank 828th / 4,138 (public-leaderboard best score of 43/50 and rank 86th / 4,138) falls squarely within our predicted 95% CI of [39.85, 41.70], validating the variance methodology, well above naive baselines but below the top of the leaderboard. What we contribute to the community is not a higher score but a documented, statistically defensible characterization of the AIMO3 inference problem with gpt-oss-120b: the base model is strong enough that most engineering interventions are dominated by run-to-run noise; the few interventions that move the score appreciably move it downward; and the single defensible upward move we found was a more careful confidence-estimation function for self-consistency voting. The expensive failure modes - strategy-prompt prescription, multi-stage routing, and most importantly GRPO with broken deployment - are all reproducible from the configurations and telemetry in the appendices. Every score in this paper is from an actual public-leaderboard submission, every configuration is from actual source code, and every causal claim is hedged to the strength of the evidence we have.
If you take one thing from this writeup, take this: before you spend a week building a router, submit the same notebook many times and look at the variance.
\nOur selected submission (final_submission v2) scored 41.5/50 on the private leaderboard (rank 828 / 4,138). Under the double-run scoring rule - where each problem receives 1.0 if both runs are correct, 0.5 if one is correct, and 0.0 if neither is - this score is consistent with a per-run accuracy of approximately 41–42, directly in line with our public-LB mean of 40.78 (σ=1.20, n=9).
The private score falls within our predicted 95% confidence interval of [39.85, 41.70], confirming that the variance methodology in Section 3 generalizes from public to private evaluation. The noise floor finding - that σ=1.20 across identical runs means any sub-2.4-point \"improvement\" is statistically indistinguishable from noise - is validated by this out-of-sample result.
\nOur rank shifted from 86th (public, based on best single submission of 43) to 828th (private, based on the selected submission's double-run average). This rank compression is itself evidence for the noise-floor thesis: teams whose public rank was inflated by a single lucky run regressed toward their true mean on the private set, while our pre-registered variance estimate correctly predicted our private performance.
\nfinal_submission v2This is the verbatim configuration from the CFG class in the submitted notebook.
| Parameter | \nValue | \n
|---|---|
| Base model | \n/kaggle/input/models/danielhanchen/gpt-oss-120b/transformers/default/1 | \n
| Served model name (vLLM) | \ngpt-oss | \n
| Quantization | \nNative MXFP4 | \n
| Tensor parallel size | \n1 | \n
| gpu_memory_utilization | \n0.96 | \n
| kv_cache_dtype | \nfp8_e4m3 | \n
| dtype | \nauto | \n
| context_tokens (max_model_len) | \n65,536 | \n
| batch_size (max_num_seqs) | \n256 | \n
| stream_interval | \n200 | \n
| enable_prefix_caching | \nTrue | \n
| async_scheduling | \nTrue | \n
| System prompt | \n\"You are a world-class IMO competitor. The final answer must be a non-negative integer between 0 and 99999. Place your final answer inside \\boxed{}.\" | \n
| Tool prompt | \n\"Use this tool to execute Python code. The environment is a stateful Jupyter notebook. Always use print() to output results.\" | \n
| Preference prompt | \n\"You have access to math, numpy, sympy, and mpmath.\" | \n
| Reasoning effort | \nReasoningEffort.HIGH (set on harmony SystemContent) | \n
| Tool format | \nopenai-harmony Conversation API | \n
| Available libraries in sandbox | \nmath, numpy, sympy, mpmath, itertools, collections; mpmath.mp.dps = 64 | \n
| temperature | \n1.0 | \n
| min_p | \n0.02 | \n
| top_p | \nnot set (intentionally - top_p = 0.9 was tested and caused a regression in earlier iterations) | \n
| top_logprobs | \n5 | \n
| attempts | \n8 | \n
| workers (parallel sandboxes) | \n16 | \n
| early_stop | \n4 (cancel remaining attempts when 4 agree) | \n
| turns (max harmony turns per attempt) | \n128 | \n
| seed (global) | \n42 | \n
| per-attempt seed | \n(seed + attempt_index) ** 2 | \n
| buffer_tokens | \n512 | \n
search_tokens (streaming \\boxed{} window) | \n32 | \n
| notebook_limit | \n17,400 s (4h50m) | \n
| base_problem_timeout | \n270 s | \n
| high_problem_timeout | \n900 s | \n
| server_timeout | \n180 s | \n
| session_timeout | \n960 s | \n
| jupyter_timeout | \n6 s | \n
| sandbox_timeout | \n3 s | \n
| per-problem budget formula | \nmin(max(time_remaining − 270 × (problems_remaining − 1), 270), 900) | \n
def _compute_weighted_entropy(self, logprobs_buffer):\n \"\"\"5-component weighted entropy.\"\"\"\n if not logprobs_buffer:\n return float('inf')\n\n entropies = []\n for top_lp in logprobs_buffer:\n if not isinstance(top_lp, dict) or not top_lp:\n continue\n h = 0.0\n for _, lp in top_lp.items():\n p = math.exp(lp)\n if p > 0:\n h -= p * math.log2(p)\n entropies.append(h)\n\n if not entropies:\n return float('inf')\n\n n = len(entropies)\n mean_ent = sum(entropies) / n\n\n # Variance penalty\n if n > 1:\n variance = sum((e - mean_ent) ** 2 for e in entropies) / (n - 1)\n std_dev = math.sqrt(variance)\n else:\n std_dev = 0.0\n\n # Position-weighted (recent tokens dominate, decay 0.995)\n decay = 0.995\n weights = [decay ** (n - 1 - i) for i in range(n)]\n total_weight = sum(weights)\n position_weighted = (\n sum(w * e for w, e in zip(weights, entropies)) / total_weight\n if total_weight > 0 else mean_ent\n )\n\n # High-entropy ratio (tokens with H > 2.0 bits)\n high_ent_count = sum(1 for e in entropies if e > 2.0)\n high_ent_ratio = high_ent_count / n\n\n # Low-entropy streak bonus (sustained < 1.0 bit)\n max_streak = 0\n current_streak = 0\n for e in entropies:\n if e < 1.0:\n current_streak += 1\n max_streak = max(max_streak, current_streak)\n else:\n current_streak = 0\n streak_bonus = -0.1 * (max_streak / n) if n > 0 else 0.0\n\n weighted = (\n 0.3 * mean_ent\n + 0.4 * position_weighted\n + 0.2 * std_dev\n + 0.3 * high_ent_ratio * 3.0\n + streak_bonus\n )\n return max(weighted, 1e-9)\n\n\ndef _select_answer(self, detailed_results):\n \"\"\"Inverse-entropy weighted voting.\"\"\"\n answer_weights = defaultdict(float)\n answer_votes = defaultdict(int)\n for r in detailed_results:\n a = r['Answer']\n e = r['Entropy']\n if a is not None:\n answer_weights[a] += 1.0 / max(e, 1e-9)\n answer_votes[a] += 1\n if not answer_weights:\n return 0\n scored = sorted(\n [{'answer': a, 'votes': answer_votes[a], 'score': w}\n for a, w in answer_weights.items()],\n key=lambda x: x['score'], reverse=True\n )\n return scored[0]['answer']\ndef _scan_for_answer(self, text):\n pattern = r'\\\\boxed\\s*\\{\\s*([0-9,]+)\\s*\\}'\n matches = re.findall(pattern, text)\n if matches:\n try:\n val = int(matches[-1].replace(',', ''))\n if 0 <= val <= 99999:\n return val\n except ValueError:\n pass\n pattern = r'final\\s+answer\\s+is\\s*([0-9,]+)'\n matches = re.findall(pattern, text, re.IGNORECASE)\n if matches:\n try:\n val = int(matches[-1].replace(',', ''))\n if 0 <= val <= 99999:\n return val\n except ValueError:\n pass\n return None\nThe streaming inspection happens inside _process_attempt: after each chunk of streamed text, the last search_tokens = 32 tokens worth of text are scanned for \\boxed{...}, and if found, the attempt short-circuits without consuming additional turns.
The 5-component weighted entropy formula builds on the multi-component entropy scoring framework from pawanmali's \"Chasing 47/50\" notebook, which demonstrated that position-weighted entropy with variance and streak components substantially outperforms simple mean entropy on AIMO3. Our contribution is the specific tuning of weights and thresholds across intermediate submissions; the structural design of which components to include follows pawanmali's framework.
\n| Parameter | \nValue | \n
|---|---|
| Hardware | \nRunPod B200 (1 GPU, 191.5 GB VRAM) | \n
| Cost rate | \n$5.03/hr | \n
| Base model | \ndanielhanchen/gpt-oss-120b loaded via Unsloth with load_in_4bit=True | \n
| Quantization during training | \nBitsAndBytes NF4 (Unsloth converts MXFP4 → NF4 on load) | \n
| Architecture | \nOff-policy: vLLM serves base model; Unsloth trains LoRA adapter separately; vLLM restarted between cycles | \n
| Learning rate | \n2e-6 with cosine decay | \n
| max_grad_norm | \n0.3 | \n
| Generations per problem | \n8 | \n
| Temperature (rollouts) | \n1.0 | \n
| top_p (rollouts) | \n0.98 | \n
| max_completion_tokens | \n6,144 | \n
| Advantage | \nDr.GRPO (mean subtraction, no std division) | \n
| Overlong filtering | \nDAPO-style (truncated completions removed from gradient) | \n
| KL coefficient (β) | \n0.0 | \n
| Reward function | \nBinary: 1.0 if extracted boxed answer matches ground truth, 0.0 otherwise | \n
| Problems per cycle | \n5 | \n
| Cycles captured in telemetry | \n38 (cycles 62–99) | \n
| Problems captured in telemetry | \n190 (problems 310–499) | \n
| Mean cycle time | \n~1,170 s (gen ~720s + train ~440s) | \n
| Cost in v6 logged segment | \n$61.87 | \n
| Total GRPO project cost (all phases) | \n≈$300 | \n
| Merge method used (the bug) | \nmerged_4bit_forced → produces BnB NF4 | \n
| Merge method that should have been used | \nmerged_16bit or save_method=\"mxfp4\" | \n
| Submission score | \n13/50 | \n
| Parameter | \nValue | \n
|---|---|
| Framework | \nHugging Face TRL GRPOTrainer + Unsloth | \n
| use_vllm | \nFalse (TRL+vLLM crashed at startup) | \n
| Learning rate | \n5e-6 | \n
| max_grad_norm | \n1.0 | \n
| β (KL coefficient) | \n0.0 | \n
| Generations per problem | \n8 | \n
| Steps completed | \n16 (model destroyed at step 16) | \n
| Time per step | \n≈76 minutes | \n
| Cost | \n$101.73 | \n
| Loss at step 1 | \n0.0513 | \n
| KL at steps 8–9 | \n87 | \n
| Gradient norm at steps 8–9 | \n133 | \n
| Reward at step 13 | \n0.5625 (peak) | \n
| Loss at step 16 | \n19,751,736 | \n
| Parameter | \nValue | \n
|---|---|
| Method | \nQLoRA, rank 128 | \n
| Training data | \n63,959 tool-integrated reasoning (TIR) traces | \n
| Steps | \n1,000 | \n
| Final loss | \n0.69 (plateau) | \n
| Uploaded model | \nflamingice2801/gpt-oss-120b-aimo3-tir-sft (Kaggle Models) | \n
| Submitted to AIMO3? | \nNo - abandoned after GRPO deployment failure exposed the merge format issue | \n
Twenty-five successful submissions plus three early debug parquet uploads (score 0) and one notebook exception. Listed in chronological order (oldest first).
\n| # | \nDays ago at writeup | \nNotebook | \nVersion | \nScore | \nPhase | \n
|---|---|---|---|---|---|
| 1 | \n24 | \naimo3_final | \nv3 | \n39 | \nIteration | \n
| 2 | \n23 | \naimo3_final | \nv4_tweaked_temp&model | \n11 | \nDebugging | \n
| 3 | \n22 | \naimo3_final | \nv5_tweaked_temp | \n0 | \nDebugging | \n
| 4 | \n21 | \naimo3_final | \nv7 | \n36 | \nIteration | \n
| 5 | \n20 | \naimo3_final | \nv9 | \n37 | \nIteration | \n
| 6 | \n19 | \naimo3_final | \nv22 | \n36 | \nIteration | \n
| 7 | \n18 | \nbase_submission | \nv2 | \n33 | \nFailure (V11 timeout) | \n
| 8 | \n17 | \n[15/15] AIME 2026 I 120b in 20mins | \nv4_grpo | \n13 | \nFailure (GRPO) | \n
| 9 | \n16 | \n[15/15] AIME 2026 I 120b in 20mins | \nv3 | \n38 | \nIteration | \n
| 10 | \n15 | \naimo3_final | \nv3 (re-run) | \n34 | \nIteration | \n
| 11 | \n14 | \nAIMO3_Adaptive_Router_Confidence_Vote_v1 | \nv1 | \n28 | \nFailure (router) | \n
| 12 | \n13 | \n[15/15] AIME 2026 I 120b in 20mins | \nv3 (re-run) | \n35 | \nIteration | \n
| 13 | \n12 | \naimo3_final | \nv9 (re-run) | \n37 | \nIteration | \n
| 14 | \n11 | \naimo3_final | \nv9 (re-run) | \n37 | \nIteration | \n
| 15 | \n10 | \naimo3_adaptive | \n_v4 | \n34 | \nFailure | \n
| 16 | \n9 | \nfinal_submission | \nv2 | \n39 | \nFinal (run 1) | \n
| 17 | \n8 | \naimo_tester | \nv2 | \n34 | \nIteration | \n
| 18 | \n7 | \nfinal_submission | \nv2 | \n40 | \nFinal (run 2) | \n
| 19 | \n6 | \nfinal_submission | \nv2 | \n43 | \nFinal (run 3) | \n
| 20 | \n4 | \nfinal_submission | \nv2 | \n40 | \nFinal (run 4) | \n
| 21 | \n3 | \nfinal_submission | \nv2 | \n42 | \nFinal (run 5) | \n
| 22 | \n3 | \nfinal_submission | \nv2 | \n40 | \nFinal (run 6) | \n
| 23 | \n2 | \nfinal_submission | \nv2 | \n41 | \nFinal (run 7) | \n
| 24 | \n1 | \nfinal_submission | \nv2 | \n41 | \nFinal (run 8) | \n
| 25 | \n0 | \nfinal_submission | \nv2 | \n41 | \nFinal (run 9) | \n
final_submission v2 distribution: {39, 40, 40, 40, 41, 41, 41, 42, 43}, n = 9, mean 40.778, median 41, σ = 1.202, 95% CI [39.85, 41.70], best 43.
The full per-problem telemetry CSV (190 rows × 14 columns) is provided as a supplementary artifact. Summary statistics:
\n| Statistic | \nValue | \n
|---|---|
| Total rows | \n190 | \n
| Cycle range | \n62 – 99 (38 cycles) | \n
| Problem range | \n310 – 499 | \n
| Generations per problem | \n8 (constant) | \n
| Mean generation time per cycle | \n720 s | \n
| Mean training time per cycle | \n440 s | \n
| Mean cycle time | \n1,170 s | \n
| Cost at cycle 62 | \n$2.34 | \n
| Cost at cycle 99 | \n$61.87 | \n
| Mean reward (all 190 problems) | \n+0.033 | \n
| Problems with reward < 0 | \n90 (47.4%) | \n
| Problems with reward = 0 (filtered) | \n34 (17.9%) | \n
| Problems with 0 < reward < 1 | \n62 (32.6%) | \n
| Problems with reward = 1.0 (all 8 correct) | \n4 (2.1%) | \n
| Problem IDs achieving reward = 1.0 | \n329, 333, 379, 468 | \n
| Loss range | \n[0.0, ~0.7] | \n
| Loss collapses | \n0 | \n
| Mean completion length | \n~3,500 tokens | \n
| Max completion length observed | \n~6,400 tokens (binding against 6,144 cap) | \n
The full CSV columns are: cycle, problem_idx, gen_time_sec, train_time_sec, total_sec, reward_mean, reward_std, num_correct, num_total, loss, completion_len_mean, clipped_ratio, gpu_mem, cost.
The 9-run variance experiment for final_submission v2 produced scores {39, 40, 40, 40, 41, 41, 41, 42, 43}. Descriptive statistics:
| Statistic | \nValue | \n
|---|---|
| n | \n9 | \n
| Sum | \n367 | \n
| Mean | \n40.778 | \n
| Median | \n41 | \n
| Mode | \n40 and 41 (each appear 3 times) | \n
| Min | \n39 | \n
| Max | \n43 | \n
| Range | \n4 | \n
| Sample variance s² | \n1.444 | \n
| Sample standard deviation s | \n1.202 | \n
| Standard error of the mean | \n0.401 | \n
| t-critical (df=8, 95%) | \n2.306 | \n
| 95% CI for mean | \n[39.854, 41.702] | \n
| Coefficient of variation | \n2.95% | \n
Under the assumption that each problem is solved with effective probability p_eff after voting, the score X ~ Binomial(50, p_eff). From mean = 40.778, p_eff ≈ 0.816, predicting a theoretical Binomial standard deviation of √(50 · 0.816 · 0.184) ≈ 2.74. Our observed σ = 1.20 is meaningfully lower than the Binomial prediction, suggesting positive correlation between problems within a single run (i.e., when the model is \"having a good run,\" it tends to solve more problems than independent sampling would predict). Plausible mechanisms include vLLM batching state, KV-cache hit patterns, and GPU thermal/scheduling effects that affect attempts within a single notebook run more than across runs.
\nFor two independent runs from the same configuration with σ = 1.20:
\nA 3-point single-shot difference between two notebooks therefore has approximately a 1-in-14 chance of being pure noise. A 4-point difference reaches conventional statistical significance (p < 0.05) only with this much variance assumed. Differences smaller than 3 points should not be acted on without replication.
\nThe public Kaggle notebook AIMO 3 | GPT-OSS-120B (with tools) (\"the 42-notebook\" in community discussion) is reported to score 42/50 by its authors with default settings. We never submitted the unmodified 42-notebook. Our base_submission v2 is a fork of it with three modifications:
attempts raised from 8 to 12early_stop raised from 4 to 5\"Be concise. Stop immediately after \\boxed{}\" added to the preference promptbase_submission v2 scored 33 - six points below baseline - primarily due to the wall-clock overrun caused by modification #1. Because we changed three variables at once, we cannot attribute the −6 to \"more attempts\" alone, and we cannot use this submission as a measurement of the unmodified 42-notebook's performance in our environment. The 42-notebook's score in our hands is, strictly speaking, unknown. We list this as a methodological limitation rather than a result.
If we were starting AIMO3 again with what we know now, in priority order:
\ngpt-oss-120b. Specifically search for \"MXFP4\" and \"merged_4bit\". This would have saved the ≈$300 GRPO sunk cost.Fifteen offline research branches were authored during the competition to explore architectures that we were not willing to spend Kaggle submission slots on. None of the 15 were submitted to the AIMO3 leaderboard. Their target benchmark was the community-sourced hard_50_math_problems_set_v6.csv dataset (Kaggle dataset zfturbo/hard-math-problems-for-aimo-3). This appendix documents each branch with its design intent (taken verbatim from the top-of-notebook docstring or markdown cell where available), its execution status, and what the preserved notebook outputs actually verify.
| Branch file | \nExecution status | \nVerifiable score | \nMean time/problem | \n
|---|---|---|---|
aimo3_branch_a_contested_resolver | \nAuthored, never executed to completion | \n- | \n- | \n
aimo3_branch_a_contested_resolver_rebuilt | \nAuthored, never executed to completion | \n- | \n- | \n
aimo3-contested-resolver | \nRan 50 problems; no correctness logging | \nCannot verify | \n216 s | \n
aimo3_contested_resolver_conservative (A2) | \nRan 50 problems; no correctness logging | \nCannot verify | \n216 s | \n
aimo3_A3_1_clean_multicell | \nAuthored, never executed to completion | \n- | \n- | \n
aimo3_A3_2_candidategen_multicell | \nAuthored, never executed to completion | \n- | \n- | \n
aimo3_A3_2_patch_multicell | \nAuthored, never executed to completion | \n- | \n- | \n
aimo3_branch_B_multicell | \nComplete 50-problem run with inline grading | \n27/50 (54.0%) | \n233 s | \n
aimo3_branch_B_fixed | \nShares stale truncated run with C_final | \n19/29 (ambiguous) | \n218 s | \n
aimo3_branch_B_updated | \nShares stale truncated run with C_final | \n19/29 (ambiguous) | \n218 s | \n
aimo3_branch_C_final | \nShares stale truncated run with B_fixed/updated | \n19/29 (ambiguous) | \n218 s | \n
aimo3_branch_C_fixed2 | \nAuthored, never executed to completion | \n- | \n- | \n
aimo3_branch_C_c2 | \nAuthored, never executed to completion | \n- | \n- | \n
aimo3_branch_C_trust_patch | \nAuthored, never executed to completion | \n- | \n- | \n
aimo3_branch_C_plus_final (C+) | \nAuthored, never executed to completion | \n- | \n- | \n
Critical caveat on the 19/29 row. Three notebooks (B_fixed, B_updated, C_final) have different code but identical saved outputs - same per-problem trace, same cumulative markers, same trace zip timestamp 20260409_092715. The outputs come from a single Colab run that was saved into three different notebook files during iteration. We cannot attribute 19/29 to any specific architecture and therefore do not cite it as a score for any named branch in the main paper. The run terminated at problem 30 - we do not know whether termination was voluntary (user stopped) or involuntary (Colab session died).
The only offline number that survives full provenance audit is Branch B multicell: 27/50 on hard50. All per-problem [EVAL] id=<X> pred=<Y> actual=<Z> correct=<bool> cumulative=<N>/<M> lines are preserved inline in the notebook output cells.
Files: aimo3_branch_a_contested_resolver, aimo3_branch_a_contested_resolver_rebuilt, aimo3-contested-resolver
Design intent (verbatim from the notebook docstring):
\n\n\n\"This notebook starts from the stable
\nfinal_submissionbaseline rather than the experimental branches. The baseline kept the proven ingredients that were strongest in prior AIMO3 experiments: simple prompts, T=1.0 generation, 8 attempts, weighted entropy, 65k context, and no large bundles of interacting heuristics.\"Motivation: hard50 analysis showed two failure classes - rerankable (correct answer appears in the candidate pool but a plurality of wrong attempts shipped it) and exploration (correct answer never generated). Branch A targets only the first class.
\n
Key changes from final_submission:
support DESC → inverse-entropy DESC → mean entropy ASC → Python error rate ASC.rerank = log1p(avg_inv_entropy) − 0.15 × py_err_rate − 0.03 × log1p(mean_py_calls). Constants drawn from hard50 failure taxonomies (66 vs 69, 570 vs 285, 24 vs 25, 2000 vs 1999, 105 vs 106) and explicitly labeled \"heuristic and intentionally mild; not claimed to be optimal.\"Execution status: The two aimo3_branch_a_* variants have zero outputs (never ran end-to-end). aimo3-contested-resolver ran 50 problems on hard50 but only logged answer=X per problem, never pred=X actual=Y correct=bool, so correctness is not inline-verifiable. Mean time 215.9 s/problem, max 710 s.
File: aimo3_contested_resolver_conservative
Design intent (verbatim):
\n\n\n\"The first Branch A run clarified four things: (1) Candidate tables are useful. They make it obvious when the correct answer is already in the pool. (2) The resolver is NOT reliable enough to hard-override the pipeline. In several cases the rerank or stage-1 winner was better than the resolver. (3) Single-cluster early agreement is still dangerous. Some hard problems produced 4 quick agreements on one wrong answer with no serious alternative exploration. (4) The reranker is better used to nominate a challenger than to become the final answer.\"
\n
Key changes from Branch A:
\nsafe_to_ship: requires single-cluster challenge to have passed when applicable.safe_weight_share = 0.70 - explicitly labeled \"heuristic, not hidden-set tuned, not paper-derived optima.\"Execution status: Same as aimo3-contested-resolver - ran 50 problems on hard50 but without per-problem correctness logging. The outputs are byte-identical to the earlier contested-resolver run (same output hash), suggesting this notebook saved the same run's outputs without re-executing.
Files: aimo3_A3_1_clean_multicell, aimo3_A3_2_candidategen_multicell, aimo3_A3_2_patch_multicell
Design intent (verbatim from markdown cell):
\n\n\n\"Preserve the stable baseline generator while making post-generation decisions much more conservative and closure-aware. The design is intended for AIMO3 itself, not for hard50-only optimization.\"
\n
Key changes from Branch A2:
\n\\boxed{} - collect all boxed integers seen in an attempt and use the LAST one; record box instability signals (multiple boxed values, box_changed flag, last-box position). Targets premature-boxing / double-counting / off-by-one failures.Constants explicitly labeled: \"not hard50-fit coefficients. They are conservative priors chosen from observed failure classes.\"
\nExecution status: All three A3 notebooks have zero outputs. Never executed end-to-end.
\nFiles: aimo3_branch_B_fixed, aimo3_branch_B_updated, aimo3_branch_B_multicell
Design intent (verbatim from markdown cell):
\n\n\n\"Preserve the stable AIMO3 generator while making post-generation arbitration stricter and more faithful to the candidate pool. This branch is intended for AIMO3-level problems, not hard50-only tuning.\"
\n
Key changes:
\nExecution status - this is the one verified data point:
\naimo3_branch_B_multicell: Completed a full 50-problem hard50 run. 27 correct, 23 wrong, 27/50 = 54.0%. Total wall-clock: 194.4 min. Mean time per problem: 233.3 s, median 188 s, max 659 s, min 27 s. 10 problems took ≥ 400 s. Zero problems hit the 900 s cap - the 27/50 is not a timeout artifact, it reflects actual answer quality on the hard50 distribution. All per-problem [EVAL] lines with pred, actual, correct, and cumulative are preserved inline in the notebook outputs.aimo3_branch_B_fixed, aimo3_branch_B_updated: Share stale identical outputs with aimo3_branch_C_final from a Colab run with trace timestamp 20260409_092715, truncated at problem 30, cumulative 19/29 (65.5%). Different code bodies, same saved outputs - we cannot attribute the 19/29 to any specific architecture.Figure 9 in the main paper is based on the Branch B multicell run.
\nFiles: aimo3_branch_C_final, aimo3_branch_C_c2, aimo3_branch_C_trust_patch, aimo3_branch_C_fixed2
Design intent (verbatim from CFG docstring):
\n\n\"Branch C keeps the stable Branch B generator and pairwise-centered arbitration, but adds four portable ideas motivated by the AIMO2 analysis and the recent Branch B traces…\"
\n
Key changes from Branch B:
\nExecution status: C_final shares stale outputs with B_fixed/B_updated (see H.5 caveat). C_c2, C_trust_patch, and C_fixed2 have zero outputs - never executed end-to-end. No new verifiable score.
File: aimo3_branch_C_plus_final
Design intent (verbatim from CFG docstring):
\n\n\"Design constraints: This branch is intended for AIMO3-level problems, not hard50-only tuning. Pairwise remains the main discriminator in binary contests. Audit is weakened relative to Branch B when pairwise + pressure already align. Tail problems are protected by a larger reserve and by lighter verification spending on already-settled problems.\"
\n
Key changes: Inherits the full Branch C controller + code rescue + repair + provenance weighting, then adds:
\nThis is the most elaborate architecture in the 15-branch catalog.
\nExecution status: Zero outputs. Authored but never executed to completion.
\nOf 15 offline research branches authored over the course of the competition:
\nbranch_a variants, all three A3 notebooks, C_c2, C_fixed2, C_trust_patch, C_plus_final)In other words, 60% of our offline research architecture never produced a single hard50 data point. The ones that did were either non-graded (40% of the remaining six) or attribution-ambiguous (50% of the remaining six). We learned a lot from reading the traces and tuning heuristics, but we did not run a clean offline benchmark loop, and the result was that the only number we can defend in this paper is the one Branch B multicell saved inline. The methodological finding in Section 4 stands on that single data point plus the nine \"intended for AIMO3, not hard50\" design-intent statements preserved verbatim above.
\nThe full verbatim docstrings and markdown cells quoted in this appendix are reproduced from the actual .ipynb files and can be verified byte-for-byte against the notebook sources supplied with this writeup.
For consistency in this writeup, the team's notebook history has been mapped to canonical names. The mapping is as follows:
\n| Canonical name (used in writeup) | \nKaggle notebook name & version | \nSubmitted? | \nScore(s) | \n
|---|---|---|---|
| Final submission | \nfinal_submission v2 | \nYes (×9) | \n39, 40, 40, 40, 41, 41, 41, 42, 43 | \n
| 42-notebook fork (V11) | \nbase_submission v2 | \nYes (×1) | \n33 | \n
| GRPO submission | \n[15/15] AIME 2026 I 120b in 20mins v4_grpo | \nYes (×1) | \n13 | \n
| Adaptive router | \nAIMO3_Adaptive_Router_Confidence_Vote_v1 | \nYes (×1) | \n28 | \n
| 5-stage AIME scaffold | \n[15/15] AIME 2026 I 120b in 20mins v3 | \nYes (×2) | \n35, 38 | \n
| Strategy-prompt family | \naimo3_final v7 / v9 / v22 | \nYes (×5) | \n36, 37, 37, 37, 36 | \n
| UNDERSTAND/EXPLORE adaptive | \naimo3_adaptive _v4 | \nYes (×1) | \n34 | \n
| Multi-temperature tester | \naimo_tester v2 | \nYes (×1) | \n34 | \n
| Early iteration | \naimo3_final v3 | \nYes (×2) | \n39, 34 | \n
| Debug / temp tweaks | \naimo3_final v4_tweaked_temp&model, v5_tweaked_temp | \nYes (×2) | \n11, 0 | \n
The name final_submission v2 is preserved as-is per team preference. Submissions earlier than 25 days before writeup (3 × submission.parquet uploads at score 0) and one notebook exception (aimo_check v4) are excluded from analysis.
End of writeup. All scores are from the public AIMO3 leaderboard. Configurations are reproduced from the verbatim source code of each submitted notebook. The GRPO telemetry CSV (190 rows) is provided as a supplementary artifact alongside this writeup.
\nS. Frieder et al., \"AI Mathematical Olympiad - Progress Prize 3,\" Kaggle Competition, 2025. https://kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3
I. Moshkov et al., \"AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning Dataset,\" arXiv:2504.16891, 2025.
P. Mali, \"Scaling Inference-Time Compute for Mathematical Reasoning,\" AIMO3 Writeup, Kaggle, 2026. https://www.kaggle.com/code/pawanmali/chasing-47-50-aimo3-journey-of-60-experiments
N. Nitarach, \"Model Capability Dominates: Inference-Time Optimization Lessons from AIMO 3,\" arXiv:2603.27844, 2026.
X. Wang et al., \"Self-Consistency Improves Chain of Thought Reasoning in Language Models,\" ICLR, 2023.
W. Kwon et al., \"Efficient Memory Management for Large Language Model Serving with PagedAttention,\" SOSP, 2023.
OpenAI, \"gpt-oss-120b Model Card,\" 2025. https://huggingface.co/openai/gpt-oss-120b
Following up on the questions from @philipvonderlind (and thanks for the very thorough response), here are some additional questions to help us assess your submission:
\nLook forward to reading your responses on these.
", "rawMarkdown": "Following up on the questions from @philipvonderlind (and thanks for the very thorough response), here are some additional questions to help us assess your submission:\n\n1. **Sensitivity of the 5-component formula:** Your entropy formula uses specific weights. Given the σ=1.20 noise you measured, are these constants informally tuned to your dev set, or did you do any sensitivity analysis showing the formula's apparent gain survives perturbations? @pawanmali's evidence supports position weighting in principle, but not necessarily these particular weights.\n2. **Offline branches that didn't complete:** You note 9 of 15 offline branches didn't execute to completion. Were there any that you would have liked to resurrect if you had more time?\n\nLook forward to reading your responses on these.", "replies": [ { "id": 3458923, "postDate": "2026-05-16T23:59:20.207Z", "content": "Thank you @sambealing - excellent questions.
\n1. Sensitivity of the 5-component formula
\nTo answer the direct question first: the weights were not tuned against the public leaderboard. The 5-component structure was adopted from pawanmali's framework, and the specific constants (0.30/0.40/0.20/0.30/−0.10, decay 0.995, thresholds 2.0 and 1.0) were set during development and held fixed across all final_submission v2 runs. They were not iterated based on leaderboard feedback - the same formula was used for all 9 variance-measurement runs. However, since there was no separate held-out validation set for the weights either, the question of whether the formula's gain survives perturbation is still worth testing.
\nTo address this, I built a post-hoc perturbation analysis using 1,500 synthetic problem scenarios calibrated to the AIMO3 difficulty distribution (35% easy with 6-2/7-1 vote splits, 20% medium with 5-3, 10% hard where the correct answer is minority 3-5, 20% exact ties at 4-4, 15% three-way splits at 3-3-2). Entropy profiles were modeled on observed competition patterns: confident solvers (low entropy, especially near conclusion), uncertain solvers (high entropy with spikes), and exploring-then-converging solvers (high early, low late - the profile position weighting is designed to reward). I verified that the sensitivity formula produces identical outputs to the submission code on 4 test sequences before running the analysis.
\nThe script is public at:\nhttps://github.com/utkarshsharma00/aimo3/blob/main/sensitivity_analysis.py
\nKey finding: the exact weights barely matter. What matters is using entropy weighting at all.
\nThe 5-component formula scores 92.5% on the synthetic benchmark. Plain majority vote scores 73.1% - a 19.4-point gap. But all entropy-weighted variants fall within a 3.1-point spread:
\n| Method | \nAccuracy | \nΔ vs default | \n
|---|---|---|
| No std_dev penalty | \n93.9% | \n+1.4% | \n
| Position-weighted only | \n92.7% | \n+0.2% | \n
| Double position weight | \n92.6% | \n+0.1% | \n
| 5-comp (our default) | \n92.5% | \nbaseline | \n
| No streak bonus | \n92.4% | \n−0.1% | \n
| Half position weight | \n92.4% | \n−0.1% | \n
| Mean + position only | \n92.1% | \n−0.5% | \n
| No high_ent_ratio | \n92.1% | \n−0.5% | \n
| Mean entropy only | \n90.8% | \n−1.7% | \n
| Plain majority vote | \n73.1% | \n−19.4% | \n
Even the worst entropy variant (mean entropy only, 90.8%) beats majority vote (73.1%) by 17.7 points. The 3.1pp spread between entropy variants is dwarfed by the 19.4pp gap from majority vote.
\nUnder individual weight perturbation (each weight scaled by 0.0×, 0.5×, 0.75×, 1.25×, 1.5×, 2.0× independently - 30 perturbations total), the maximum answer flip rate was 1.4% (21/1500 scenarios). Total accuracy range across all 30 perturbations: [92.1%, 93.9%] - a spread of just 1.8 percentage points. Most perturbations caused 0-2 flips out of 1500 (≤0.2%).
\nUnder Monte Carlo random perturbation (200 trials, all 5 weights simultaneously perturbed by random factors in [0.5, 1.5]), mean accuracy change was +0.08% with a standard deviation of 0.38% and a range of [−0.33%, +1.73%]. Mean flip rate was 0.2%, max 1.7%. No perturbation caused more than 1.73% accuracy change in either direction.
\nWhere entropy voting concentrates its value (by scenario type):
\n| Scenario type | \n5-comp | \nMajority | \nΔ | \n
|---|---|---|---|
| Easy (6-2 split) | \n100.0% | \n100.0% | \n+0.0% | \n
| Medium (5-3 split) | \n100.0% | \n100.0% | \n+0.0% | \n
| Hard (3-5 minority) | \n21.7% | \n0.0% | \n+21.7% | \n
| Tied (4-4 split) | \n100.0% | \n48.6% | \n+51.4% | \n
| Three-way (3-3-2) | \n100.0% | \n49.5% | \n+50.5% | \n
On uncontested problems (easy/medium), entropy weighting is irrelevant - majority vote already picks the right answer. The entire value concentrates in contested cases: ties, three-way splits, and minority-correct scenarios. This is consistent with our competition experience: on ~35 of 50 problems, all 8 attempts agree and the voting function doesn't matter. On the ~15 contested problems, entropy weighting breaks ties toward the more confident trace.
\nAn unexpected finding: The analysis suggests our formula may be slightly over-engineered. \"No std_dev penalty\" (removing the variance component entirely) scored +1.4% better than the full 5-component formula, and \"position-weighted only\" (dropping all 4 other components) scored +0.2% better. The std_dev component may introduce noise rather than signal. This is a finding we did not expect and would not have discovered without this post-hoc analysis.
\nImportant caveats: This analysis uses synthetic scenarios, not a replay of actual competition logprobs. The entropy profiles are modeled on observed patterns but make a key assumption: that correct traces tend to have lower position-weighted entropy near the conclusion than incorrect traces. This is the intuition behind position weighting (a solver that reached the correct answer through valid reasoning should be confident at the end), and it's supported by multiple independent findings: pawanmali's multi-run evidence (3-6 point advantage for position-weighted over mean entropy across ~8 runs), Potrhino's empirical result that tail entropy (last 2048 tokens) is the strongest single indicator of incorrect reasoning paths on gpt-oss-120b, outperforming global mean entropy (§4.5.1.3 of their writeup), and the DeepConf framework (Fu et al., 2025), which achieves 99.9% on AIME 2025 by filtering traces using token-level confidence. Conversely, rzatemizel's density analysis shows that mean entropy has near-zero discriminative power - which is precisely why we use position weighting rather than mean entropy. The assumption remains unproven on real gpt-oss-120b logprobs for our specific formula, but the directional evidence across these independent analyses is consistent. A proper validation would require logging per-attempt logprobs during competition runs and replaying them with perturbed weights - data I did not collect. The structural conclusion (any entropy weighting >> majority vote; exact weights don't matter much; position weighting > mean entropy) is robust within this simulation framework.
\n2. Offline branches worth resurrecting
\nIf I had more time, I would prioritize Branch C (controller + code rescue + provenance-aware arbitration).
\nThe design addresses a failure class that few submitted approaches directly targeted: what to do when the attempt pool is fragmented (multiple incompatible answer clusters with no majority) or broken (most attempts failed to produce valid answers). AMO's Master Vision pipeline (davidacad10 et al.) is the closest parallel - both detect the absence of consensus and launch a resolution mechanism. Branch C differs in using a deterministic state machine rather than an LLM agent for the detection step.
\nConvergence/divergence state controller. Detects whether the pool has converged (stop early, save budget), fragmented (multiple clusters, need resolution), or broken (most attempts returned no answer). This is strictly better than flat early-stop thresholds because it adapts compute to the observed pool state rather than applying a fixed rule.
\nTargeted code rescue. When text-based reasoning fails, switches to a tool-backed exact computation path. This directly addresses what IceBeam777's RSA writeup calls \"Class 4/5\" problems - where the model converges confidently to a wrong answer and no amount of resampling fixes it.
\nProvenance-aware weighting. Candidates supported by multiple evidence modes (main pool + code rescue + challenge pass) receive higher weight than single-source answers. This is a richer signal than entropy alone.
\nWhat makes Branch C architecturally interesting is how it relates to other competitors' independently developed approaches:
\nvs. ippeiogawa's Blackboard. The Blackboard shares tool I/O between threads during generation, allowing threads to build on each other's computational discoveries. Branch C's code rescue is complementary - instead of sharing intermediate computations between threads, it launches an entirely separate computation mode when text-based reasoning fails. Both address threads getting stuck in wrong reasoning frames, but from different directions: Blackboard prevents getting stuck by sharing information; code rescue recovers after getting stuck by switching modality.
\nvs. rzatemizel's Judger. The 3-base/5-judger architecture evaluates completed reasoning traces to select the best answer. Branch C's provenance-aware arbitration is similar in spirit - both use evidence quality rather than raw vote count to weight answers. The key difference is that rzatemizel's judger is a separate LLM call that evaluates the full reasoning trace, while Branch C's provenance weighting is a lightweight, deterministic function of which evidence modes support each answer - far less compute cost but less nuanced information about reasoning quality.
\nvs. AMO's Master Vision (davidacad10 et al.). AMO introduces a Master Vision agent that synthesizes findings from competing solvers when no majority exists - essentially a conflict-resolution pipeline. Branch C's convergence controller serves the same role but as a deterministic state machine rather than an LLM agent, avoiding the latency and hallucination risk of another LLM call while sacrificing the flexibility of natural-language synthesis.
\nThe reason Branch C didn't ship was infrastructure: wiring bugs in the multi-stage pipeline prevented end-to-end execution on Colab. With another week of debugging time, I believe the state controller alone would have been worth integrating - it addresses time allocation more intelligently than any flat threshold. But in a time-constrained competition, architectural ambition must be balanced against execution reliability. Our final submission won not because it was the best architecture we designed, but because it was the best one we shipped.
\nHappy to provide additional detail on any of these points.
", "rawMarkdown": "Thank you @sambealing - excellent questions. \n\n**1. Sensitivity of the 5-component formula**\n\nTo answer the direct question first: the weights were not tuned against the public leaderboard. The 5-component structure was adopted from pawanmali's framework, and the specific constants (0.30/0.40/0.20/0.30/−0.10, decay 0.995, thresholds 2.0 and 1.0) were set during development and held fixed across all final_submission v2 runs. They were not iterated based on leaderboard feedback - the same formula was used for all 9 variance-measurement runs. However, since there was no separate held-out validation set for the weights either, the question of whether the formula's gain survives perturbation is still worth testing.\n\nTo address this, I built a post-hoc perturbation analysis using 1,500 synthetic problem scenarios calibrated to the AIMO3 difficulty distribution (35% easy with 6-2/7-1 vote splits, 20% medium with 5-3, 10% hard where the correct answer is minority 3-5, 20% exact ties at 4-4, 15% three-way splits at 3-3-2). Entropy profiles were modeled on observed competition patterns: confident solvers (low entropy, especially near conclusion), uncertain solvers (high entropy with spikes), and exploring-then-converging solvers (high early, low late - the profile position weighting is designed to reward). I verified that the sensitivity formula produces identical outputs to the submission code on 4 test sequences before running the analysis.\n\nThe script is public at:\nhttps://github.com/utkarshsharma00/aimo3/blob/main/sensitivity_analysis.py\n\n**Key finding: the exact weights barely matter. What matters is using entropy weighting at all.**\n\nThe 5-component formula scores 92.5% on the synthetic benchmark. Plain majority vote scores 73.1% - a 19.4-point gap. But all entropy-weighted variants fall within a 3.1-point spread:\n\n| Method | Accuracy | Δ vs default |\n|-------------------------|----------|-------------|\n| No std_dev penalty | 93.9% | +1.4% |\n| Position-weighted only | 92.7% | +0.2% |\n| Double position weight | 92.6% | +0.1% |\n| **5-comp (our default)**| **92.5%**| **baseline** |\n| No streak bonus | 92.4% | −0.1% |\n| Half position weight | 92.4% | −0.1% |\n| Mean + position only | 92.1% | −0.5% |\n| No high_ent_ratio | 92.1% | −0.5% |\n| Mean entropy only | 90.8% | −1.7% |\n| **Plain majority vote** | **73.1%**| **−19.4%** |\n\nEven the worst entropy variant (mean entropy only, 90.8%) beats majority vote (73.1%) by 17.7 points. The 3.1pp spread between entropy variants is dwarfed by the 19.4pp gap from majority vote.\n\nUnder individual weight perturbation (each weight scaled by 0.0×, 0.5×, 0.75×, 1.25×, 1.5×, 2.0× independently - 30 perturbations total), the maximum answer flip rate was 1.4% (21/1500 scenarios). Total accuracy range across all 30 perturbations: [92.1%, 93.9%] - a spread of just 1.8 percentage points. Most perturbations caused 0-2 flips out of 1500 (≤0.2%).\n\nUnder Monte Carlo random perturbation (200 trials, all 5 weights simultaneously perturbed by random factors in [0.5, 1.5]), mean accuracy change was +0.08% with a standard deviation of 0.38% and a range of [−0.33%, +1.73%]. Mean flip rate was 0.2%, max 1.7%. No perturbation caused more than 1.73% accuracy change in either direction.\n\nWhere entropy voting concentrates its value (by scenario type):\n\n| Scenario type | 5-comp | Majority | Δ |\n|---------------------|---------|----------|---------|\n| Easy (6-2 split) | 100.0% | 100.0% | +0.0% |\n| Medium (5-3 split) | 100.0% | 100.0% | +0.0% |\n| Hard (3-5 minority) | 21.7% | 0.0% | +21.7% |\n| Tied (4-4 split) | 100.0% | 48.6% | +51.4% |\n| Three-way (3-3-2) | 100.0% | 49.5% | +50.5% |\n\nOn uncontested problems (easy/medium), entropy weighting is irrelevant - majority vote already picks the right answer. The entire value concentrates in contested cases: ties, three-way splits, and minority-correct scenarios. This is consistent with our competition experience: on ~35 of 50 problems, all 8 attempts agree and the voting function doesn't matter. On the ~15 contested problems, entropy weighting breaks ties toward the more confident trace.\n\n**An unexpected finding:** The analysis suggests our formula may be slightly over-engineered. \"No std_dev penalty\" (removing the variance component entirely) scored +1.4% better than the full 5-component formula, and \"position-weighted only\" (dropping all 4 other components) scored +0.2% better. The std_dev component may introduce noise rather than signal. This is a finding we did not expect and would not have discovered without this post-hoc analysis.\n\n**Important caveats:** This analysis uses synthetic scenarios, not a replay of actual competition logprobs. The entropy profiles are modeled on observed patterns but make a key assumption: that correct traces tend to have lower position-weighted entropy near the conclusion than incorrect traces. This is the intuition behind position weighting (a solver that reached the correct answer through valid reasoning should be confident at the end), and it's supported by multiple independent findings: pawanmali's multi-run evidence (3-6 point advantage for position-weighted over mean entropy across ~8 runs), Potrhino's empirical result that tail entropy (last 2048 tokens) is the strongest single indicator of incorrect reasoning paths on gpt-oss-120b, outperforming global mean entropy (§4.5.1.3 of their writeup), and the DeepConf framework (Fu et al., 2025), which achieves 99.9% on AIME 2025 by filtering traces using token-level confidence. Conversely, rzatemizel's density analysis shows that mean entropy has near-zero discriminative power - which is precisely why we use position weighting rather than mean entropy. The assumption remains unproven on real gpt-oss-120b logprobs for our specific formula, but the directional evidence across these independent analyses is consistent. A proper validation would require logging per-attempt logprobs during competition runs and replaying them with perturbed weights - data I did not collect. The structural conclusion (any entropy weighting >> majority vote; exact weights don't matter much; position weighting > mean entropy) is robust within this simulation framework.\n\n**2. Offline branches worth resurrecting**\n\nIf I had more time, I would prioritize **Branch C (controller + code rescue + provenance-aware arbitration)**. \n\nThe design addresses a failure class that few submitted approaches directly targeted: what to do when the attempt pool is *fragmented* (multiple incompatible answer clusters with no majority) or *broken* (most attempts failed to produce valid answers). AMO's Master Vision pipeline (davidacad10 et al.) is the closest parallel - both detect the absence of consensus and launch a resolution mechanism. Branch C differs in using a deterministic state machine rather than an LLM agent for the detection step.\n\n**Convergence/divergence state controller.** Detects whether the pool has converged (stop early, save budget), fragmented (multiple clusters, need resolution), or broken (most attempts returned no answer). This is strictly better than flat early-stop thresholds because it adapts compute to the observed pool state rather than applying a fixed rule.\n\n**Targeted code rescue.** When text-based reasoning fails, switches to a tool-backed exact computation path. This directly addresses what IceBeam777's RSA writeup calls \"Class 4/5\" problems - where the model converges confidently to a wrong answer and no amount of resampling fixes it.\n\n**Provenance-aware weighting.** Candidates supported by multiple evidence modes (main pool + code rescue + challenge pass) receive higher weight than single-source answers. This is a richer signal than entropy alone.\n\nWhat makes Branch C architecturally interesting is how it relates to other competitors' independently developed approaches:\n\n*vs. ippeiogawa's Blackboard.* The Blackboard shares tool I/O between threads during generation, allowing threads to build on each other's computational discoveries. Branch C's code rescue is complementary - instead of sharing intermediate computations between threads, it launches an entirely separate computation mode when text-based reasoning fails. Both address threads getting stuck in wrong reasoning frames, but from different directions: Blackboard prevents getting stuck by sharing information; code rescue recovers after getting stuck by switching modality.\n\n*vs. rzatemizel's Judger.* The 3-base/5-judger architecture evaluates completed reasoning traces to select the best answer. Branch C's provenance-aware arbitration is similar in spirit - both use evidence quality rather than raw vote count to weight answers. The key difference is that rzatemizel's judger is a separate LLM call that evaluates the full reasoning trace, while Branch C's provenance weighting is a lightweight, deterministic function of which evidence modes support each answer - far less compute cost but less nuanced information about reasoning quality.\n\n*vs. AMO's Master Vision (davidacad10 et al.).* AMO introduces a Master Vision agent that synthesizes findings from competing solvers when no majority exists - essentially a conflict-resolution pipeline. Branch C's convergence controller serves the same role but as a deterministic state machine rather than an LLM agent, avoiding the latency and hallucination risk of another LLM call while sacrificing the flexibility of natural-language synthesis.\n\nThe reason Branch C didn't ship was infrastructure: wiring bugs in the multi-stage pipeline prevented end-to-end execution on Colab. With another week of debugging time, I believe the state controller alone would have been worth integrating - it addresses time allocation more intelligently than any flat threshold. But in a time-constrained competition, architectural ambition must be balanced against execution reliability. Our final submission won not because it was the best architecture we designed, but because it was the best one we *shipped*.\n\n---\n\nHappy to provide additional detail on any of these points.", "replies": [ { "id": 3463165, "postDate": "2026-05-26T05:01:58.780Z", "content": "Note: After posting this analysis, the 2nd-place solution writeup 'Vote-dominant Ensembling and a Sharper System Prompt' by @varianceofx was published. Their \"tail-windowed entropy\" (Improvement 3) independently converges on the same design as our position weighting - using entropy over only the last 256 tokens rather than the full trace, with the same reasoning that early-trace entropy reflects exploration, not answer confidence. Their ensemble design (Improvement 5) also independently confirms our sensitivity finding: they weight vote_share at 2.0 and confidence at only 0.3, concluding that \"entropy is best used as a tiebreaker, not as the main criterion.\" Two independent teams arriving at the same architectural conclusion strengthens the case that position/tail entropy > mean entropy, and that exact weights are secondary to using entropy weighting at all.
", "rawMarkdown": "Note: After posting this analysis, the 2nd-place solution writeup 'Vote-dominant Ensembling and a Sharper System Prompt' by @varianceofx was published. Their \"tail-windowed entropy\" (Improvement 3) independently converges on the same design as our position weighting - using entropy over only the last 256 tokens rather than the full trace, with the same reasoning that early-trace entropy reflects exploration, not answer confidence. Their ensemble design (Improvement 5) also independently confirms our sensitivity finding: they weight vote_share at 2.0 and confidence at only 0.3, concluding that \"entropy is best used as a tiebreaker, not as the main criterion.\" Two independent teams arriving at the same architectural conclusion strengthens the case that position/tail entropy > mean entropy, and that exact weights are secondary to using entropy weighting at all." } ] } ] }, { "id": 3457396, "postDate": "2026-05-13T17:44:36.593Z", "content": "Edited on May 13th 2026 for the rebuttal phase:
\nChanges made:
\nNo architectural claims, experimental results, or methodological conclusions were altered. The private LB result validates the variance methodology described in Section 3 - our pre-registered 95% CI correctly predicted the out-of-sample private score.
", "rawMarkdown": "**Edited on May 13th 2026 for the rebuttal phase:**\n\nChanges made:\n- Added Section 11 (Private Leaderboard Result): private LB score of 41.5/50, rank 828/4,138, confirming the score falls within our predicted 95% CI of [39.85, 41.70]\n- Updated Abstract and Section 10 to include the private LB score\n- Responded to @philipvonderlind's three reviewer questions (GRPO reconciliation with outliar's results, local testing methodology, entropy correlation evidence) in a separate comment below\n\nNo architectural claims, experimental results, or methodological conclusions were altered. The private LB result validates the variance methodology described in Section 3 - our pre-registered 95% CI correctly predicted the out-of-sample private score." }, { "id": 3456831, "postDate": "2026-05-12T16:43:16.203Z", "content": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.
\nI have a few questions regarding some of the ideas proposed and implemented in your writeup:
\nGRPO: Other contestants, notably @outliar, have reported successful gains using GRPO. You mention that the observed improvements in your pipeline may be attributable to noise, which could certainly also apply to other submissions. How would you explain the GRPO gains reported by other contestants in comparison to your own training results?
Local Testing: Most of your ablations appear to rely on single public leaderboard experiments. Did you also attempt to verify these changes locally, for example, using one of the datasets from the Math Corpus Prize entries? Reporting score distributions across repeated runs, along with variance estimates, would make the results easier for others to interpret.
Entropy: Did you verify that entropy-based voting actually correlates with more mathematically correct traces? Some other writeups have reported that model confidence does not necessarily correspond to trace correctness.
Looking forward to your response :)
", "rawMarkdown": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.\n\nI have a few questions regarding some of the ideas proposed and implemented in your writeup:\n\n1. **GRPO**: Other contestants, notably @outliar, have [reported successful gains](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/four-phase-adaptation-of-a-120b-model-for-tool-aug) using GRPO. You mention that the observed improvements in your pipeline may be attributable to noise, which could certainly also apply to other submissions. How would you explain the GRPO gains reported by other contestants in comparison to your own training results?\n\n2. **Local Testing**: Most of your ablations appear to rely on single public leaderboard experiments. Did you also attempt to verify these changes locally, for example, using one of the datasets from the Math Corpus Prize entries? Reporting score distributions across repeated runs, along with variance estimates, would make the results easier for others to interpret.\n\n3. **Entropy**: Did you verify that entropy-based voting actually correlates with more mathematically correct traces? Some [other writeups](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/model-capability-dominates) have reported that model confidence does not necessarily correspond to trace correctness.\n\nLooking forward to your response :)\n", "replies": [ { "id": 3456863, "postDate": "2026-05-12T17:24:06.790Z", "content": "Hi @philipvonderlind - thank you for the detailed questions. Happy to address each one.
\n1. GRPO: Reconciling our failure with outliar's reported gains
\nThe critical distinction between our GRPO attempt and outliar's is the deployment path, not the training itself. Our GRPO v6 loop trained stably for 38 cycles with zero loss collapses - the training worked. The failure was at deployment: we used Unsloth's merged_4bit_forced to merge the adapter, which produces BitsAndBytes NF4 weights. gpt-oss-120b's native format is MXFP4. vLLM routes NF4 weights through a different (degraded) kernel path, producing the 13/50 catastrophic score. outliar avoided this by using save_method=\"mxfp4\" - the correct deployment path.
However, even when GRPO deployment succeeds, the gains appear modest relative to inference-only optimization. outliar's final fine-tuned model (SFT + GRPO + ORPO + MXFP4 bake) achieved μ=38.57 (σ=1.67, n=35) - which is 2.2 points below our inference-only μ=40.78 (σ=1.20, n=9). Their pre-ORPO baseline was μ=36.6, so the total fine-tuning gain was approximately +2 points. Meanwhile, Inoichan et al. (\"Pushing the Limits: Post-Training High-Capability Models under Strict Inference\" writeup, 10 authors, 32 H100s) ran full-parameter SFT for 10 epochs on 10.37 billion tokens and improved pass@1 by only +0.77 percentage points - while actually reducing coverage (Pass@16) from 92.28% to 90.88%. Their RL (GRPO variant) showed a similar pattern: +1.0 pass@1 but −2.11 coverage.
\nSo our position is nuanced. We do NOT claim GRPO cannot help - outliar's +2 is real. We claim that (a) the deployment pipeline is a documented trap that cost us $300 and that future participants should be aware of, and (b) even when fine-tuning succeeds, the gains are small relative to inference-only optimization on a base model this strong, and coverage (the model's solvable problem space) may actually shrink with training. The latter finding from Inoichan et al. is the strongest independent evidence supporting our thesis.
\n2. Local Testing: Variance estimates and offline verification
\nYou're right that many of our architectural ablations (strategy prompts, 5-stage scaffold, router, etc.) relied on single or few-shot public leaderboard comparisons. This is the most significant methodological limitation of our writeup, and we acknowledge it.
\nHowever, we did two things that go beyond single-shot LB comparison:
\nThe n=9 variance experiment. We submitted final_submission v2 nine times without modification, producing scores {39, 40, 43, 40, 42, 40, 41, 41, 41} with μ=40.78 and σ=1.20. This established our noise floor: any Δ < 2σ ≈ 2.4 points is indistinguishable from running the same notebook twice. This measurement is what allows us to claim the strategy-prompt family (5 submissions, mean ~36.6) is directionally negative (Δ ≈ −4.2, outside the ±2σ band) rather than noise.
Offline evaluation on hard50. We ran 15 offline research branches against hard_50_math_problems_set_v6.csv (zfturbo's dataset from the Math Corpus Prize). One branch - aimo3_branch_B_multicell - completed a full 50-problem run with inline grading: 27/50 (54.0%), mean 233s/problem. This is the only verified offline number we cite, and we document honestly that 9 of our 15 offline branches never executed to completion. The offline work is detailed in Section 4 and Appendix H of the writeup.
\nWe did not use the offline benchmark for per-variant ablation (e.g., running each of our 11 configurations on hard50 multiple times) - which would have been the methodologically stronger approach. The reason was practical: each hard50 run took ~3.5 hours on Colab, and we prioritized leaderboard submissions over systematic offline testing. In hindsight, this was the wrong tradeoff - our Lesson #1 in Section 9 is exactly this: \"Establish the noise floor first. Submit the same notebook 5-10 times before iterating on architecture.\"
\nFor comparison, sebastiangil00's writeup (\"Three Substrate Traps on gpt-oss-120b-MXFP4\") tested 16 variants on a 199-problem validation set with McNemar paired tests and Benjamini-Hochberg correction - a substantially more rigorous methodology than ours. We consider their statistical approach the gold standard for future competitions.
\n3. Entropy: Does it actually correlate with correctness?
\nThis is the hardest question, and the honest answer is: we don't have conclusive evidence that our specific entropy formula correlates with trace correctness in isolation.
\nWhat we know:
\nMean entropy does not discriminate. rzatemizel's density plots (\"Fighting Variance\" writeup) show correct answers have mean entropy 0.7053 vs wrong answers 0.7503 - a gap of 0.045 on almost completely overlapping distributions. Schwyter (arXiv:2603.27844) also found that model confidence does not correspond to trace correctness, as you reference. We agree with this finding for mean entropy.
\nOur formula is structurally different from mean entropy. The 5-component weighted entropy uses position weighting (40% of the score, exponential decay 0.995 favoring tokens near \\boxed{}), variance penalty, high-entropy ratio, and low-entropy streak bonus. The key intuition is that a solution can be uncertain during exploration but should be confident at the conclusion - position weighting captures this, while mean entropy averages over the entire trace and loses the signal.
rzatemizel confirmed they did not test position-weighted variants. In their reply to our comment on their writeup, they stated: \"I did not check position-weighted entropy or other similar variants, because I lost confidence in logprobs fairly early.\" So the density - overlap finding applies to mean entropy specifically, not necessarily to our position-weighted composite.
\npawanmali provides the strongest supporting evidence. In their reply to our comment, they confirmed: \"mean entropy consistently underperformed position-weighted entropy by 3-6 points across ~8 different runs where that was the primary variable.\" This is multi-run directional evidence that position-weighted entropy captures something that mean entropy does not.
\nOur own comparison is confounded. We changed the voting function AND stripped the scaffolding simultaneously, so we cannot isolate which change was load-bearing. We state this explicitly in the writeup (Section 6, final paragraph). The honest answer is: we don't know whether the entropy formula or the scaffold removal (or both) drove the improvement from ~36.6 to 40.78.
\nThe bottom line: mean entropy is probably not a useful discriminator (confirmed by rzatemizel and Schwyter). Position-weighted entropy may retain discriminative power at the conclusion of the reasoning trace, supported by pawanmali's multi-run evidence but not conclusively proven by our own data. We consider this an open empirical question that deserves a controlled ablation we were unable to run before the deadline.
\nHappy to provide additional detail on any of these points.
", "rawMarkdown": "Hi @philipvonderlind - thank you for the detailed questions. Happy to address each one.\n\n**1. GRPO: Reconciling our failure with outliar's reported gains**\n\nThe critical distinction between our GRPO attempt and outliar's is the deployment path, not the training itself. Our GRPO v6 loop trained stably for 38 cycles with zero loss collapses - the training worked. The failure was at deployment: we used Unsloth's `merged_4bit_forced` to merge the adapter, which produces BitsAndBytes NF4 weights. gpt-oss-120b's native format is MXFP4. vLLM routes NF4 weights through a different (degraded) kernel path, producing the 13/50 catastrophic score. outliar avoided this by using `save_method=\"mxfp4\"` - the correct deployment path.\n\nHowever, even when GRPO deployment succeeds, the gains appear modest relative to inference-only optimization. outliar's final fine-tuned model (SFT + GRPO + ORPO + MXFP4 bake) achieved μ=38.57 (σ=1.67, n=35) - which is 2.2 points below our inference-only μ=40.78 (σ=1.20, n=9). Their pre-ORPO baseline was μ=36.6, so the total fine-tuning gain was approximately +2 points. Meanwhile, Inoichan et al. (\"Pushing the Limits: Post-Training High-Capability Models under Strict Inference\" writeup, 10 authors, 32 H100s) ran full-parameter SFT for 10 epochs on 10.37 billion tokens and improved pass@1 by only +0.77 percentage points - while actually *reducing* coverage (Pass@16) from 92.28% to 90.88%. Their RL (GRPO variant) showed a similar pattern: +1.0 pass@1 but −2.11 coverage.\n\nSo our position is nuanced. We do NOT claim GRPO cannot help - outliar's +2 is real. We claim that (a) the deployment pipeline is a documented trap that cost us $300 and that future participants should be aware of, and (b) even when fine-tuning succeeds, the gains are small relative to inference-only optimization on a base model this strong, and coverage (the model's solvable problem space) may actually shrink with training. The latter finding from Inoichan et al. is the strongest independent evidence supporting our thesis.\n\n**2. Local Testing: Variance estimates and offline verification**\n\nYou're right that many of our architectural ablations (strategy prompts, 5-stage scaffold, router, etc.) relied on single or few-shot public leaderboard comparisons. This is the most significant methodological limitation of our writeup, and we acknowledge it.\n\nHowever, we did two things that go beyond single-shot LB comparison:\n\n*The n=9 variance experiment.* We submitted `final_submission v2` nine times without modification, producing scores {39, 40, 43, 40, 42, 40, 41, 41, 41} with μ=40.78 and σ=1.20. This established our noise floor: any Δ < 2σ ≈ 2.4 points is indistinguishable from running the same notebook twice. This measurement is what allows us to claim the strategy-prompt family (5 submissions, mean ~36.6) is directionally negative (Δ ≈ −4.2, outside the ±2σ band) rather than noise.\n\n*Offline evaluation on hard50.* We ran 15 offline research branches against hard_50_math_problems_set_v6.csv (zfturbo's dataset from the Math Corpus Prize). One branch - aimo3_branch_B_multicell - completed a full 50-problem run with inline grading: 27/50 (54.0%), mean 233s/problem. This is the only verified offline number we cite, and we document honestly that 9 of our 15 offline branches never executed to completion. The offline work is detailed in Section 4 and Appendix H of the writeup.\n\nWe did not use the offline benchmark for per-variant ablation (e.g., running each of our 11 configurations on hard50 multiple times) - which would have been the methodologically stronger approach. The reason was practical: each hard50 run took ~3.5 hours on Colab, and we prioritized leaderboard submissions over systematic offline testing. In hindsight, this was the wrong tradeoff - our Lesson #1 in Section 9 is exactly this: \"Establish the noise floor first. Submit the same notebook 5-10 times before iterating on architecture.\"\n\nFor comparison, sebastiangil00's writeup (\"Three Substrate Traps on gpt-oss-120b-MXFP4\") tested 16 variants on a 199-problem validation set with McNemar paired tests and Benjamini-Hochberg correction - a substantially more rigorous methodology than ours. We consider their statistical approach the gold standard for future competitions.\n\n**3. Entropy: Does it actually correlate with correctness?**\n\nThis is the hardest question, and the honest answer is: we don't have conclusive evidence that our specific entropy formula correlates with trace correctness in isolation.\n\nWhat we know:\n\n*Mean entropy does not discriminate.* rzatemizel's density plots (\"Fighting Variance\" writeup) show correct answers have mean entropy 0.7053 vs wrong answers 0.7503 - a gap of 0.045 on almost completely overlapping distributions. Schwyter (arXiv:2603.27844) also found that model confidence does not correspond to trace correctness, as you reference. We agree with this finding for mean entropy.\n\n*Our formula is structurally different from mean entropy.* The 5-component weighted entropy uses position weighting (40% of the score, exponential decay 0.995 favoring tokens near `\\boxed{}`), variance penalty, high-entropy ratio, and low-entropy streak bonus. The key intuition is that a solution can be uncertain during exploration but should be confident at the conclusion - position weighting captures this, while mean entropy averages over the entire trace and loses the signal.\n\n*rzatemizel confirmed they did not test position-weighted variants.* In their reply to our comment on their writeup, they stated: \"I did not check position-weighted entropy or other similar variants, because I lost confidence in logprobs fairly early.\" So the density - overlap finding applies to mean entropy specifically, not necessarily to our position-weighted composite.\n\n*pawanmali provides the strongest supporting evidence.* In their reply to our comment, they confirmed: \"mean entropy consistently underperformed position-weighted entropy by 3-6 points across ~8 different runs where that was the primary variable.\" This is multi-run directional evidence that position-weighted entropy captures something that mean entropy does not.\n\n*Our own comparison is confounded.* We changed the voting function AND stripped the scaffolding simultaneously, so we cannot isolate which change was load-bearing. We state this explicitly in the writeup (Section 6, final paragraph). The honest answer is: we don't know whether the entropy formula or the scaffold removal (or both) drove the improvement from ~36.6 to 40.78.\n\nThe bottom line: mean entropy is probably not a useful discriminator (confirmed by rzatemizel and Schwyter). Position-weighted entropy may retain discriminative power at the conclusion of the reasoning trace, supported by pawanmali's multi-run evidence but not conclusively proven by our own data. We consider this an open empirical question that deserves a controlled ablation we were unable to run before the deadline.\n\n---\n\nHappy to provide additional detail on any of these points." } ] }, { "id": 3454392, "postDate": "2026-05-07T03:06:55.633Z", "content": "Reproducibility Package
\nModel: danielhanchen/gpt-oss-120b (HuggingFace, unmodified - no fine-tuning, no LoRA, no weight patches)\nhttps://huggingface.co/danielhanchen/gpt-oss-120b
\nKaggle submission notebook (final_submission v2):\nhttps://www.kaggle.com/code/flamingice2801/final-submission
\nGoogle Colab notebook (Kaggle-agnostic):\nhttps://colab.research.google.com/github/utkarshsharma00/aimo3/blob/main/colab_inference.ipynb
\nStandalone inference script:\nhttps://github.com/utkarshsharma00/aimo3/blob/main/run_inference.py
\nFigure generation script (reproduces all 9 figures):\nhttps://github.com/utkarshsharma00/aimo3/blob/main/generate_figures.py
\nFull repository:\nhttps://github.com/utkarshsharma00/aimo3
\nThe entire contribution is inference-side - no custom model artifacts. The Colab notebook pulls the stock model from HuggingFace and contains the full pipeline (solver, 5-component entropy voting, sandbox pool, streaming parser, adaptive time management). Requires an A100/H100 80GB runtime.
", "rawMarkdown": "**Reproducibility Package**\n\n**Model:** danielhanchen/gpt-oss-120b (HuggingFace, unmodified - no fine-tuning, no LoRA, no weight patches)\nhttps://huggingface.co/danielhanchen/gpt-oss-120b\n\n**Kaggle submission notebook (final_submission v2):**\nhttps://www.kaggle.com/code/flamingice2801/final-submission\n\n**Google Colab notebook (Kaggle-agnostic):**\nhttps://colab.research.google.com/github/utkarshsharma00/aimo3/blob/main/colab_inference.ipynb\n\n**Standalone inference script:**\nhttps://github.com/utkarshsharma00/aimo3/blob/main/run_inference.py\n\n**Figure generation script (reproduces all 9 figures):**\nhttps://github.com/utkarshsharma00/aimo3/blob/main/generate_figures.py\n\n**Full repository:**\nhttps://github.com/utkarshsharma00/aimo3\n\nThe entire contribution is inference-side - no custom model artifacts. The Colab notebook pulls the stock model from HuggingFace and contains the full pipeline (solver, 5-component entropy voting, sandbox pool, streaming parser, adaptive time management). Requires an A100/H100 80GB runtime.\n" }, { "id": 3446698, "postDate": "2026-04-22T00:58:51.433Z", "content": "Note added after reviewing concurrent writeups (April 21, 2026)
\nSeveral independent writeups have now been published that converge on the same central finding documented here. Schwyter (arXiv:2603.27844, 23 experiments across 3 models) found that model capability dominates all prompt-level optimizations by 4×. @merkiraz (11 controlled submissions) found every modification preserved or degraded performance. @kolenkovskin (20+ dead ends across 28 submissions) independently concluded that \"subtraction beats addition.\" @potrhino (7 controlled ablation runs on a 75-problem validation set) documented positive contributions from TIR and dynamic time management but found that the remaining gap is compute-constrained. This convergence across 5+ independent teams - each arriving at similar negative conclusions through different experimental paths - is itself evidence that the finding is robust.
\nRegarding selection loss: the host's pass@N analysis shows pass@8 ≈ 44 on the private set, meaning the correct answer appears in at least one of our 8 attempts for ~44 of 50 problems. Our mean of 40.78 implies we correctly select the right answer from the pool ~93% of the time. The remaining ~3 problems per run where we have the correct answer but select wrong represent the selection loss that our 5-component entropy voting addresses but does not fully close. @rzatemizel density-overlap analysis suggests that mean entropy has near-zero discriminative power between correct and incorrect answers; whether our position-weighted variant retains discriminative power where mean entropy does not remains an open empirical question.
", "rawMarkdown": "**Note added after reviewing concurrent writeups (April 21, 2026)**\n\nSeveral independent writeups have now been published that converge on the same central finding documented here. Schwyter (arXiv:2603.27844, 23 experiments across 3 models) found that model capability dominates all prompt-level optimizations by 4×. @merkiraz (11 controlled submissions) found every modification preserved or degraded performance. @kolenkovskin (20+ dead ends across 28 submissions) independently concluded that \"subtraction beats addition.\" @potrhino (7 controlled ablation runs on a 75-problem validation set) documented positive contributions from TIR and dynamic time management but found that the remaining gap is compute-constrained. This convergence across 5+ independent teams - each arriving at similar negative conclusions through different experimental paths - is itself evidence that the finding is robust.\n\nRegarding selection loss: the host's pass@N analysis shows pass@8 ≈ 44 on the private set, meaning the correct answer appears in at least one of our 8 attempts for ~44 of 50 problems. Our mean of 40.78 implies we correctly select the right answer from the pool ~93% of the time. The remaining ~3 problems per run where we have the correct answer but select wrong represent the selection loss that our 5-component entropy voting addresses but does not fully close. @rzatemizel density-overlap analysis suggests that mean entropy has near-zero discriminative power between correct and incorrect answers; whether our position-weighted variant retains discriminative power where mean entropy does not remains an open empirical question." } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T04:20:46Z", "competition_id": 118448, "competition_slug": "ai-mathematical-olympiad-progress-prize-3", "team_id": 15409274, "solution_url": "/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/every-improvement-failed-aimo3" }