![\"Model](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F9101740%2F163076f5cbaa7ef6901f0814cbb20eed%2Ffig3_p_vs_score.png?generation=1776421154591925&alt=media\")
Score: 42/50 (best run), 39.3 mean over 21 runs\nModel: gpt-oss-120b (116.8B total / 5.1B active MoE, FP8)\nConfig: N=8, T=1.0, entropy-weighted majority voting, single H100 80GB\nPaper: arXiv:2603.27844 | Code: GitHub
This work tests whether assigning diverse reasoning strategies to different voters improves majority voting on IMO-level problems. It does not. Across 3 models, 23+ experiments, and 50 problems, every prompt-level intervention failed to beat the baseline reliably. High-temperature sampling already decorrelates errors; weaker prompts reduce per-attempt accuracy more than they reduce correlation. Model capability dominates all prompt-level optimizations by 4×: an 8-point gap at equal N=8 vs. ±2-point prompt effects.
\nThe six-point gap between the best run (42) and pass@20 ≈ 45.5 on gpt-oss-120b is not inference-time headroom. It is selection loss. Three training-free verifiers (constraint check, code re-execution, LM re-verification) are tested across 5 seeds; none robustly closes the gap (all 95% CIs overlap zero). The audit locates the failure: 82–86% of the unrecovered correctness loses by a single vote — a rank-2 margin, not an entropy hole. Closing it remains open. The private LB (released after submission) confirms the diagnosis at competition scale: a 4.5-point shake-up between public and private scores on the same configuration sits within 3σ of base-model run-to-run variance. This work scored 40.5/50 private with the code-re-execution verifier submission.
Majority voting with correlated errors reduces to effective sample size:
\n$$ N_{\\text{eff}} = \\frac{N}{1 + (N-1)\\rho} $$
\nAt ρ=0.3, eight attempts become 2.6 effective votes. The natural fix: assign different reasoning prompts (\"Small Cases First\", \"Work Backwards\", \"Classify Then Solve\") so errors decorrelate.
\nHypothesis: diverse prompts -> lower ρ -> higher expected score.\nResult: every mixer configuration scored at or below baseline. Monotonic degradation with diversity.
\n![\"Prompt](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F9101740%2F46a5eb1af7653eeee0d89ea67d0bee77%2Ffig1_diversity_vs_score.png?generation=1776421187900713&alt=media\")
| Configuration | \nN | \nLB Score | \nΔ | \n
|---|---|---|---|
| Baseline (21-run mean) | \n8 | \n39.3 | \n— | \n
| Conservative mix (5+1+1+1) | \n8 | \n40 | \n+0.7 | \n
| Aggressive mix (3+2+2+1) | \n8 | \n40 | \n+0.7 | \n
| Equal mix (2+2+2+2) | \n8 | \n38 | \n−1.3 | \n
| 8× Small Cases | \n8 | \n37 | \n−2.3 | \n
| 8× Work Backwards | \n8 | \n39 | \n−0.3 | \n
| 8× Classify | \n8 | \n36 | \n−3.3 | \n
| 8× Code-first | \n8 | \n41 / 38 / 34 | \nmean 37.7 | \n
| Formalize-First (F-1) | \n8 | \n39 | \n−0.3 | \n
(a) Temperature already decorrelates. At T=1.0, sampling alone provides most achievable diversity. T ablation: T=0.5→38, T=0.8→40, T=1.0→39.3, T=1.2→37. Prompt diversity on top of high T is redundant.
\n(b) Pairwise correlation is already near zero. Using the method-of-moments estimator on all four models (19 computable points), every estimate is negative. Large-N subset (N≥7, 7 points): mean ρ̂ = −0.122. Full sample: −0.348. Wrong answers scatter across distinct values instead of clustering; there is no correlation headroom to exploit.
\n![\"Per-problem](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F9101740%2F2af76da4c788ac96a89825692498a883%2Ffig_rho_analysis.png?generation=1776421395970527&alt=media\")
(c) The critical test (E10). Equal mix (2+2+2+2) at T=0.5, where prompt diversity should help most because stochastic diversity is suppressed, scored 36, worse than T=0.5 alone (38). Diversity does not compensate even under the ideal conditions for it to help.
\n(d) Weaker prompts lower p̂. The mixer reduces ρ but also reduces per-attempt accuracy. Accuracy loss exceeds decorrelation gain.
\nThe cleanest comparison: identical N=8, identical pipeline, different model.
\n| Model | \nParams (Total/Active) | \nN | \np̂ | \nLB Score | \n
|---|---|---|---|---|
gpt-oss-120b | \n116.8B / 5.1B | \n8 | \n0.69 | \n39.3 (21 runs) | \n
gpt-oss-20b | \n20B / 3.6B | \n8 | \n0.61 | \n31.0 (3 runs) | \n
gpt-oss-20b | \n20B / 3.6B | \n32 | \n0.52 | \n26 | \n
gpt-oss-20b | \n20B / 3.6B | \n3 | \n0.54 | \n28.3 | \n
Nemotron-Super-120B-NVFP4 | \n120B / 12B | \n3 | \n0.47 | \n23 | \n
Qwen3.5-35B-A3B | \n35B / 3B | \n16 | \n0.46 | \n23 | \n
Key observations:
\ngpt-oss-20b at N=32 drops to 26 because per-attempt time shrinks and p̂ falls from 0.61 → 0.52.![\"Qwen3.5-35B-A3B](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F9101740%2Fb554c5b24bf7614d84421275d95acb8b%2Ffig2_qwen_ablation.png?generation=1776421348511010&alt=media\")
Qwen3.5-35B-A3B (8 controlled experiments on 10 local problems): nothing improves beyond baseline. Doubling N from 8→16: no gain. Long prompts: −1. Manufacturer-recommended top_k=20 or presence_penalty: crashes or −1.
Nemotron-Super-120B-NVFP4: 23/50, identical to Qwen despite 4× active parameters. HMMT Feb25 benchmark shows NVFP4 ≈ BF16 (95.4% vs 94.7%), so quantization is not the cause; the N=3 vs N=8 budget is.
\nThe negative result replicates across 3 model families.
\nWith μ̂ = 39.3, σ̂ = 1.7 over 21 baseline runs (range 34–42), each submission is a lottery ticket. P(score ≥ 42 | single run) ≈ 5.6%. The mixer reduces μ without improving the tail. Optimal strategy: submit the unmodified baseline repeatedly. All 42 submissions exhausted.
\nPart of the variance is infrastructure noise. Shared-GPU benchmarks can shift by ~6 points from resource contention alone (Anthropic Engineering, 2026). Infra config is treated as a first-class experimental variable: 21-run statistics, 3-model cross-validation, and exact vLLM flags documented in the paper appendices.
\n![\"Submission](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F9101740%2Fc78d8afee1e5d92481118e5c53ec702e%2Ffig4_lottery_ticket.png?generation=1776421428382805&alt=media\")
A host-posted analysis (see competition discussion and @AIMOprize on X) reports gpt-oss-120b at pass@20 ≈ 45.5 on the private set (95% CI [43, 48]) and pass@100 ≈ 49/50.
The 6-point gap between the best run (42) and pass@20 (45.5) is not inference-time headroom; Equation 1 already absorbs p and ρ. It is selection loss: the correct answer is in the N=8 pool but outvoted by a more common wrong one. Majority voting is the cheapest selector. A verifier (code execution against problem constraints, formal substitution, cross-candidate consistency) could capture some of that gap. Three training-free verifiers are tested in Section 8; none robustly closes it. The audit in Section 8 locates the failure: 82–86% of the unrecovered correctness loses by a single vote — a one-vote margin, not an entropy hole.
\nThe scope of the negative result is narrower than the title suggests. Prompt-level optimization does not help. Selection-level optimization remains open. The three naive verifiers tested in Section 8 do not robustly close it, and the failure mode (rank-2 losses by one vote) constrains what a working selector would need to do.
\nThe full 23-experiment ablation on gpt-oss-120b (Figure: ablation waterfall, Section 9) confirms the negative result: no single experiment reliably exceeds baseline. The best single run (E12 Code-first, 41/50) was not replicated — confirmation runs scored 38 and 34 (3-run mean 37.7, below baseline 39.3). The baseline is a local optimum, and the ±2-point spread within the table matches the run-to-run variance later observed at competition scale on the private leaderboard.
| Competition | \nWinner (public) | \nWinner (private) | \nThis Work (public) | \nThis Work (private) | \nN | \nInnovation | \n
|---|---|---|---|---|---|---|
| AIMO-1 (2024) | \n29/50 | \n— | \n— | \n— | \n48 | \nBrute-force voting | \n
| AIMO-2 (2025) | \n34/50 | \n— | \n— | \n— | \n64 | \nCustom 540K training + GenSelect | \n
| AIMO-3 (2026) | \n46* | \n44† | \n42‡ | \n40.5§ | \n8 | \nOff-the-shelf model, zero training | \n
* Public peak at competition close (Apr 15), dropped to 41.5/50 on private re-evaluation.\n† Private-LB top score; team was previously ranked outside the top 2,000 with a public score of 37/50. Single submission, no method change.\n‡ Best of 21 baseline submissions on the public set.\n§ Code re-execution verifier submission (Section 8).
\nPrivate-LB shake-up. The AIMO~3 private leaderboard was released after this work was finalized. The host attributed the resulting shake-up to intrinsic model variation arising from team clustering on gpt-oss-120b: with 4,000+ teams converging on the same base model, the leaderboard became more susceptible to run-to-run variance than to method differences. The 4.5-point gap between the public peak (46) and its private score (41.5) sits within 3σ of the same configuration re-submitted, given the σ̂ ≈ 1.7 measured across 21 baseline runs (Section 5). The shake-up is selection loss observed at competition scale — methodological differences across teams are smaller than the spread induced by re-running the same model. The host noted that \"more runs (10+) might have overcome this variation,\" but the per-submission compute budget made this infeasible for any team.
This work's verifier-aware submission (Code re-execution, λ=2.0) scored 41/50 public and 40.5/50 private. The 0.5-point gap is a single problem and sits inside the inter-seed σ=2.24 reported in Section 8 (Table). The submission-level result is consistent with the External-50 finding: code re-execution does not robustly close selection loss, and per-submission variance subsumes any small effect.
\n![\"AIMO](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F9101740%2F1d8be1bf2a501b733a8eb57fae9e164b%2Ffig6_leaderboard.png?generation=1778669849552583&alt=media\")
The trend across competitions: from high-N to high-p̂. AIMO-1 used N=48 with brute-force voting; AIMO-2 invested 540K training problems with custom GenSelect; this work uses zero training compute on an off-the-shelf model. Model capability first, everything else second.
\nSection 6 identified selection loss as the real headroom. Closing it requires a selector that distinguishes correct from incorrect candidates without ground truth. The established direction is trained verifiers — outcome-supervised, process-supervised, or step-level reward models — but all require annotated training. The cheapest training-free alternatives are tested here, across five seeds on the external 50 of zfturbo (disjoint from the AIMO-3 hidden LB). Implementation: utils/verified_voting.py.
Mechanism. The entropy-weighted vote score S(a) is multiplied by a verifier boost:
$$\\text{final} = \\arg\\max_a \\, S(a) \\cdot \\big(1 + \\lambda \\cdot \\text{verify}(p,a)\\big), \\quad \\lambda = 2.0$$
\nSetting λ=0 recovers entropy-weighted voting. Verification runs only on the top-K=3 most-voted candidates; unanimous votes skip it. Three strategies:
\nstdout matches. Zero LM calls.Results (External 50, n=5 seeds; paired problem-level bootstrap, B=10,000):
\n| Selector | \nMean ± σ | \nΔ | \n95% CI | \np | \n
|---|---|---|---|---|
| Majority vote | \n27.0 ± 2.35 | \n— | \n— | \n— | \n
| Constraint chk. | \n27.2 ± 1.30 | \n+0.2 | \n[−2.8, +3.2] | \n.895 | \n
| Code re-exec. | \n25.0 ± 2.24 | \n−2.0 | \n[−5.6, +1.6] | \n.305 | \n
| LM re-verify | \n29.6 ± 1.82 | \n+2.6 | \n[−0.4, +5.4] | \n.080 | \n
All three 95% CIs overlap zero. Constraint check is indistinguishable from the baseline. Code re-execution costs two problems on average. LM re-verification trends positive (+2.6, p=0.080), but its CI extends to −0.4. None of the three robustly closes selection loss. All four configurations clock 287–299 s per problem; verifier overhead is within the inter-strategy noise floor.
\n![\"Per-problem](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F9101740%2F621d034ed8f104107cb5790e1e45c504%2Ffig8_verified_voting_per_problem.png?generation=1778669727111949&alt=media\")
Selection-loss audit (where the gap actually lives). Each run on the external 50 generates the correct answer for 41.4 problems on average and ships it for only 27.3. The 14-problem gap is selection loss: the right answer is in the attempt pool but loses the vote.
\n| Selector | \nVoted | \nGenerated | \nGap | \nRank-2 | \n
|---|---|---|---|---|
| Constraint chk. | \n27.2 | \n42.6 | \n+15.4 | \n81% | \n
| Code re-exec. | \n25.0 | \n41.0 | \n+16.0 | \n81% | \n
| LM re-verify | \n29.6 | \n40.6 | \n+11.0 | \n86% | \n
| Pooled | \n27.3 | \n41.4 | \n+14.1 | \n82% | \n
82–86% of the unrecovered correctness is a rank-2 loss — the correct answer was the second most common vote, beaten by a single extra agreeing attempt. The bottleneck is a one-vote margin, not an entropy hole.
\nWhy each strategy fails. Constraint check fires only on problems with explicit numeric ranges; on the rest it adds zero signal. Code re-execution depends on a brittle stdout match — the model often computes an intermediate float in the final code block, then applies post-processing (rounding, modular arithmetic, length count) before \\boxed{}. The valid candidate is scored as invalid, and the verifier flips a correct vote to a wrong one (e.g., IMO548: 22030→22031, off by one). LM re-verification has the largest mean effect (+2.6) but the largest spread per problem: across 5 seeds the inter-seed σ=1.82 is the same size as the effect.
Competition LB. The final submission uses Code re-execution with λ=2.0: 41/50 public, 40.5/50 private (Section 7). The 0.5-point public-vs-private gap sits inside the inter-seed σ=2.24 measured on the external set — the submission-level result is consistent with \"no robust effect.\"
\nBottom line. Three naive training-free verifiers do not close selection loss at this scale. The audit locates the failure precisely: 82–86% of generated-but-discarded correct answers lose by a single vote. Closing this gap likely requires either (i) a trained verifier with calibrated per-candidate confidence, or (ii) a selector that breaks rank-2 ties using signal orthogonal to vote count (semantic agreement across attempts, derivation-graph consistency, or external constraint propagation). Selection loss remains open.
\n![\"Complete](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F9101740%2F1ef9f5444c56008304fd863c66563767%2Ffig5_ablation_waterfall.png?generation=1776421555409602&alt=media\")
Use the largest model that fits. Keep temperature high. Stop engineering prompts. Spend remaining budget on a verifier, not a mixer. Six points separate the best run from pass@20. That gap is selection loss. Three training-free verifiers (Constraint, Code re-execution, LM re-verification) do not robustly close it — all three 95% CIs overlap zero across 5 seeds (Section 8). The audit locates the failure: 86% of the unrecovered correctness loses by a single vote — a rank-2 margin, not an entropy hole. Closing this gap is the open problem this work does not solve, and the private-LB shake-up at competition scale (Section 7) confirms the same diagnosis: run-to-run variance of the base model dominates methodological differences across teams.
\ngpt-oss-20b baseline: nguyennguyen599/aimo3-skills-optional-luck-required-gpt-oss-20bThanks to the AIMO Prize Committee and Kaggle for organizing the competition and providing GPU infrastructure. The baseline notebook builds on the inference pipeline by nihilisticneuralnet.
", "comments": [ { "id": 3475439, "author_name": "Chan Kha Vu", "author_url": "", "post_date": "2026-06-20T01:22:13.707000", "content": "Hey, congrats with winning the Writeup Prize! Our team relied on your findings to rule out simple prompt tweaking / hyperparams tuning experiments, and used your recommended simple setting. Thanks for sharing during the competition.
", "votes": 1, "replies": [ { "id": 3475484, "author_name": "Natapong Nitarach (Schwyter)", "author_url": "", "post_date": "2026-06-20T02:09:28.103000", "content": "Thank you so much, and congrats to @chankhavu and @yeoyunsianggeremie 🎉🚀 Really glad it was useful and helped cut out the stuff that doesn't matter so the budget can go to what actually does. Right now I'm cleaning up the equations and numbers so everything is correct and fully reproducible, though I don't expect the main conclusion to shift much now that the Private Score is out. Awesome work on your side too. That vLLM fork getting prefix caching and speculative decoding to play nice together on hybrid-Mamba is seriously wow.
", "votes": 1, "replies": [] } ] }, { "id": 3444171, "author_name": "JM", "author_url": "", "post_date": "2026-04-17T12:04:50.517000", "content": "Nice writeup on the prompt mixing experiments. But for the verifier under what conditions would the stdout be incorrect if the sandbox was set up properly? Is it possible the final Python block wasn’t actually used to produce the answer (e.g. the model continued reasoning afterward)? In that case, your verifier could produce false negatives when stdout doesn’t match the boxed answer. Also, a single run seems insufficient to conclude it outperforms a simple majority vote, it might just catch the rare hallucinations.
", "votes": 1, "replies": [ { "id": 3444178, "author_name": "Natapong Nitarach (Schwyter)", "author_url": "", "post_date": "2026-04-17T12:26:25.917000", "content": "Your analysis is exact. @julianmukaj
\nFalse negatives: You are right. The model frequently computes an intermediate float in the final Python block, but applies final formatting in the subsequent text before generating \\boxed{}. Strict stdout enforcement rejects valid reasoning. This exact mismatch (e.g., IMO548 off-by-1) is documented in Section 8.
Sample size: The writeup explicitly states a single run is insufficient to separate signal from variance. The empirical data proves this: the verifier actually dropped the external set score from 30 to 28. It does not consistently outperform majority voting.
This naive script does not solve the problem. Its purpose is to demonstrate the 6-point Selection Loss gap. The bottleneck is the selector, not the generator.
", "votes": 1, "replies": [] } ] }, { "id": 3457771, "author_name": "Sam Bealing", "author_url": "", "post_date": "2026-05-14T10:11:52.687000", "content": "Following up on the questions from @philipvonderlind, here are some additional questions to help us assess your submission:
\n\\boxed{} answer. But some off-by-one cases could reflect ambiguity in the source problem itself (e.g. boundary inclusive/exclusive, or the 'is 0 positive?' issue discussed in this Kaggle thread). Have you looked at whether any of the verifier-hurt flips are on problems where both answers could be reasonably defended so really it is an issue with the problem statement rather than the verifier?Look forward to reading your responses on these.
", "votes": 0, "replies": [ { "id": 3457923, "author_name": "Natapong Nitarach (Schwyter)", "author_url": "", "post_date": "2026-05-14T16:10:17.980000", "content": "Hi @sambealing. Answers below.
\n1. Next-step verifier direction. I prioritize (ii) rank-2 tie-breaker over (i) trained verifier. Three reasons. First, training-free sidesteps a distribution-shift problem. Public process-reward models exist (e.g., Skywork-o1-Open-PRM-Qwen-2.5-7B), in the process-supervision direction introduced by Lightman et al. (2023) for math reasoning. These PRMs are trained on different model trace distributions; transfer to gpt-oss-120b's CoT + code-execution traces is unverified, and validating that transfer requires the annotated set AIMO-class competitions exist to substitute for. Rank-2 tie-breaking avoids that validation step. Second, §8 shows 82–86% of unrecovered correctness loses by a one-vote margin; a small lift on rank-2 yields linear score gain on the quantity that dominates the gap. A trained verifier helps most in the low-p̂ regime; gpt-oss-120b at p̂ ≈ 0.69 is bottlenecked by tie-breaking, not calibration. Third, an orthogonal signal is composable: it can later feed (i) as a feature.
\n\"Orthogonal enough\" has an operational test. Condition one: Spearman |ρ_s| < 0.3 with vote count v(a), so the signal supplements rather than re-weights majority. Condition two: on the §8 rank-2 subset, s(a) discriminates correct from wrong with AUC ≥ 0.6, clearing the inter-seed noise floor (σ ≈ 1.8–2.3 in §8). Three candidates that would pass: (i) semantic agreement across reasoning paths, where different derivations reaching the same answer carry more evidence than identical derivations agreeing; (ii) constraint propagation beyond numeric ranges, testing whether the answer satisfies every parsed problem constraint; (iii) step-level consistency between the reasoning trace and the boxed answer, which is exactly the failure mode in (2).
\n2. Off-by-one as ambiguous answers. Fair sharpening; I agree with the principle. Off-by-one disagreements can reflect problem-statement ambiguity (boundary inclusion, \"0 ∈ ℕ⁺\", under-specified modular reductions), and a verifier flipping correct -> wrong on such a problem is a benchmark-side issue, not a verifier-side issue.
\nTriggered by your question, I audited the five Code re-execution net-hurt cases on the external 50 (majority-of-seeds criterion, n=5 seeds):
\n| Problem | \nBase -> Verif | \nDataset-author note | \nAttribution | \n
|---|---|---|---|
| IMO1131 | \n0.80 -> 0.00 | \n\"Incorrect answer\" | \nReference label issue | \n
| IMO1477 | \n0.60 -> 0.00 | \n\"Easy task, but the model misinterprets the text\" | \nProblem-text ambiguity (your hypothesis) | \n
| N00490 | \n0.60 -> 0.40 | \n(no specific note) | \nHurts all three verifiers symmetrically; problem-side, not verifier-side | \n
| N01418 | \n0.60 -> 0.20 | \n(no specific note) | \nUnclassified | \n
| N01455 | \n0.60 -> 0.40 | \n(no specific note) | \nUnclassified | \n
Three of five are consistent with the benchmark-side framing you proposed, two are dataset-author-flagged independently of my analysis. Two remain unclassified within the rebuttal window.
\nOn IMO548 specifically: the case in §8 is a single-seed mechanistic illustration of how stdout-match interacts with post-processing. The model's reasoning trace computes a value, then applies an off-by-one correction before \\boxed{}; the verifier captures the pre-correction value and scores the valid candidate as invalid. At the multi-seed majority level (baseline 0.80, verifier 0.60), IMO548 is not in the net-hurt list above. §8 uses it to illustrate the failure mechanism, not as a population-level effect.
\nThe broader §8 conclusion does not change under either attribution: all three 95% CIs overlap zero across 5 seeds and 250 problem-runs, so the empirical statement (training-free verifiers do not robustly close selection loss) holds whether per-problem failures attribute to verifier implementation or to problem ambiguity. The 82–86% rank-2 finding is the audit signal that should guide the next selector, independent of per-case attribution.
\nBenchmark-design corollary: closing selection loss at AIMO-class scale requires either tighter problem statements upstream, or selectors that route ambiguous problems away from deterministic verifiers. The first is a benchmark issue; the second is the orthogonal-signal direction in (1).
", "votes": 0, "replies": [] } ] }, { "id": 3457739, "author_name": "Philip Vonderlind", "author_url": "", "post_date": "2026-05-14T08:51:54.823000", "content": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.
\nWe think the reproducibility of your work is already outstanding, and several other writeups referring to your work is a clear indication of its impact. Below are a few notes and questions I had while reading the writeup:
\nSelection Loss Gap: The pass@n statistics we report are computed without tools, so the real gap may be even larger. This seems to further support your assumptions.
N-Generations: In the section Why It Fails, you report results for several models side by side to compare their performance. However, I did not fully understand why different values of N were used for each model. In my view, this makes the results less directly comparable. I assume this choice was due to hardware constraints and the different inference requirements of each model. What is your perspective on this?
Stability: Did any approach you tried make the notebook perform consistently on the public leaderboard? Other approaches, such as the one by ippeiogawa, appear to “stabilize” submission performance. Given your findings, would you attribute their reported stability primarily to noise, or do you think their approach addresses a different failure mode?
Local Evaluation Dataset: Why did you choose ZFTurbo as the benchmark for local evaluation? Did you also try any of the other Math Corpus Prize submissions or publicly available benchmarks?
Looking forward to your response :)
", "votes": 0, "replies": [ { "id": 3457897, "author_name": "Natapong Nitarach (Schwyter)", "author_url": "", "post_date": "2026-05-14T15:19:44.493000", "content": "Hi @philipvonderlind. Thanks for the read. Answers below.
\n1. Selection Loss Gap. Confirmed: your post explicitly notes \"used no tool calling or other optimisations\" for the pass@k analysis. My pipeline uses sympy / numpy in 8 persistent kernels per attempt, so per-attempt success rate differs from a tool-free baseline at the same N. I do not claim my N=8 ceiling exceeds your pass@20 ≈ 45.5; that comparison mixes two effects (N=8 vs N=20, with-tools vs without). What I do claim is measured directly in §8 on the same pipeline: per-run generated correct = 41.4 / 50 (n=5 seeds), shipped = 27.3, a 14-point within-pipeline gap independent of tool effects. The §6 / §8 selection-loss claim rests on this within-pipeline measurement, not on a comparison to pass@k.
\n2. N-Generations. Yes, the 5-hour single-pod budget is binding; per-attempt latency scales with active parameters and reasoning length. The clean apples-to-apples row in §3 is gpt-oss-120b @ N=8 (39.3) vs gpt-oss-20b @ N=8 (31.0): same N, same pipeline, 8-point gap. The 4× claim rests on this row. The other 20b rows ablate N within a fixed model and show N cannot recover the gap: at N=32 the score drops to 26 because per-attempt budget shrinks and p̂ falls 0.61 → 0.52. Nemotron @ N=3 and Qwen @ N=16 are budget-bounded upper bounds favorable to them, not point estimates of model rank; the §3 table should be read with that scope.
\n3. Stability. My 21 baseline submissions give σ̂ ≈ 1.7 (range 34–42). §7 replicates the diagnosis at competition scale: the 4.5-point public-vs-private gap on the same config sits inside 3σ. Under this null, \"stability\" across N=2–3 submissions is statistically indistinguishable from noise. I have not inspected ippeiogawa's mechanism, so I will not assert \"noise\" without their multi-run data. Prior: if their pipeline addresses a different failure mode (parse-error retry, format validation, sandbox-timeout recovery), it is complementary and does not contradict the selection-loss claim; if not, the consistency should regress to σ̂ over enough runs.
\n4. Local Evaluation Dataset. Honest answer up front: ZFTurbo was the set the community was actively sharing for AIMO-3-class evaluation, and I ran my baseline on it without an exhaustive search of alternatives. Other Math Corpus Prize submissions were not surveyed within the rebuttal window. This is a real limitation, not a justification dressed up after the fact.
\nTwo reasons it still served the §8 purpose. First, the dataset author publicly documented per-problem gpt-oss-120b success rates across 5–20 seeds; the range is 0–100% with mean ≈ 60%, which gives the discrimination headroom selection-loss work needs. Known label issues are flagged on the dataset card (N01223 with a probably-wrong reference; a few too-easy / too-hard items); the inter-seed σ ≈ 1.8–2.3 in Table §8 absorbs that label noise.
\nSecond, post-hoc cross-validation I did not engineer: the dataset author's own note on N01223 reads \"the true answer often appears as the second-most-likely output (TOP2)\". That is the same failure mode my §8 audit measures at scale: 82–86% of unrecovered correctness is rank-2. Two independent analyses converge on rank-2 as the bottleneck, from different angles.
\nHMMT Feb25 and AIME 2025 were considered but ruled out: gpt-oss-120b saturates both above 0.94, leaving no discrimination for selection-loss effects. The §8 result is a within-pipeline negative claim, scoped to AIMO-3-class IMO-level problems on a disjoint reference set; it does not need broader cross-validation to stand.
", "votes": 0, "replies": [] } ] } ], "comment_count": 8, "messages": [ { "id": 3475439, "postDate": "2026-06-20T01:22:13.707Z", "content": "Hey, congrats with winning the Writeup Prize! Our team relied on your findings to rule out simple prompt tweaking / hyperparams tuning experiments, and used your recommended simple setting. Thanks for sharing during the competition.
", "rawMarkdown": "Hey, congrats with winning the Writeup Prize! Our team relied on your findings to rule out simple prompt tweaking / hyperparams tuning experiments, and used your recommended simple setting. Thanks for sharing **during** the competition.", "votes": 1, "replies": [ { "id": 3475484, "postDate": "2026-06-20T02:09:28.103Z", "content": "Thank you so much, and congrats to @chankhavu and @yeoyunsianggeremie 🎉🚀 Really glad it was useful and helped cut out the stuff that doesn't matter so the budget can go to what actually does. Right now I'm cleaning up the equations and numbers so everything is correct and fully reproducible, though I don't expect the main conclusion to shift much now that the Private Score is out. Awesome work on your side too. That vLLM fork getting prefix caching and speculative decoding to play nice together on hybrid-Mamba is seriously wow.
", "rawMarkdown": "Thank you so much, and congrats to @chankhavu and @yeoyunsianggeremie 🎉🚀 Really glad it was useful and helped cut out the stuff that doesn't matter so the budget can go to what actually does. Right now I'm cleaning up the equations and numbers so everything is correct and fully reproducible, though I don't expect the main conclusion to shift much now that the Private Score is out. Awesome work on your side too. That vLLM fork getting prefix caching and speculative decoding to play nice together on hybrid-Mamba is seriously wow.", "votes": 1 } ] }, { "id": 3444135, "postDate": "2026-04-17T10:49:20.840Z", "content": "Score: 42/50 (best run), 39.3 mean over 21 runs\nModel: gpt-oss-120b (116.8B total / 5.1B active MoE, FP8)\nConfig: N=8, T=1.0, entropy-weighted majority voting, single H100 80GB\nPaper: arXiv:2603.27844 | Code: GitHub
This work tests whether assigning diverse reasoning strategies to different voters improves majority voting on IMO-level problems. It does not. Across 3 models, 23+ experiments, and 50 problems, every prompt-level intervention failed to beat the baseline reliably. High-temperature sampling already decorrelates errors; weaker prompts reduce per-attempt accuracy more than they reduce correlation. Model capability dominates all prompt-level optimizations by 4×: an 8-point gap at equal N=8 vs. ±2-point prompt effects.
\nThe six-point gap between the best run (42) and pass@20 ≈ 45.5 on gpt-oss-120b is not inference-time headroom. It is selection loss. Three training-free verifiers (constraint check, code re-execution, LM re-verification) are tested across 5 seeds; none robustly closes the gap (all 95% CIs overlap zero). The audit locates the failure: 82–86% of the unrecovered correctness loses by a single vote — a rank-2 margin, not an entropy hole. Closing it remains open. The private LB (released after submission) confirms the diagnosis at competition scale: a 4.5-point shake-up between public and private scores on the same configuration sits within 3σ of base-model run-to-run variance. This work scored 40.5/50 private with the code-re-execution verifier submission.
Majority voting with correlated errors reduces to effective sample size:
\n$$ N_{\\text{eff}} = \\frac{N}{1 + (N-1)\\rho} $$
\nAt ρ=0.3, eight attempts become 2.6 effective votes. The natural fix: assign different reasoning prompts (\"Small Cases First\", \"Work Backwards\", \"Classify Then Solve\") so errors decorrelate.
\nHypothesis: diverse prompts -> lower ρ -> higher expected score.\nResult: every mixer configuration scored at or below baseline. Monotonic degradation with diversity.
\n
| Configuration | \nN | \nLB Score | \nΔ | \n
|---|---|---|---|
| Baseline (21-run mean) | \n8 | \n39.3 | \n— | \n
| Conservative mix (5+1+1+1) | \n8 | \n40 | \n+0.7 | \n
| Aggressive mix (3+2+2+1) | \n8 | \n40 | \n+0.7 | \n
| Equal mix (2+2+2+2) | \n8 | \n38 | \n−1.3 | \n
| 8× Small Cases | \n8 | \n37 | \n−2.3 | \n
| 8× Work Backwards | \n8 | \n39 | \n−0.3 | \n
| 8× Classify | \n8 | \n36 | \n−3.3 | \n
| 8× Code-first | \n8 | \n41 / 38 / 34 | \nmean 37.7 | \n
| Formalize-First (F-1) | \n8 | \n39 | \n−0.3 | \n
(a) Temperature already decorrelates. At T=1.0, sampling alone provides most achievable diversity. T ablation: T=0.5→38, T=0.8→40, T=1.0→39.3, T=1.2→37. Prompt diversity on top of high T is redundant.
\n(b) Pairwise correlation is already near zero. Using the method-of-moments estimator on all four models (19 computable points), every estimate is negative. Large-N subset (N≥7, 7 points): mean ρ̂ = −0.122. Full sample: −0.348. Wrong answers scatter across distinct values instead of clustering; there is no correlation headroom to exploit.
\n
(c) The critical test (E10). Equal mix (2+2+2+2) at T=0.5, where prompt diversity should help most because stochastic diversity is suppressed, scored 36, worse than T=0.5 alone (38). Diversity does not compensate even under the ideal conditions for it to help.
\n(d) Weaker prompts lower p̂. The mixer reduces ρ but also reduces per-attempt accuracy. Accuracy loss exceeds decorrelation gain.
\nThe cleanest comparison: identical N=8, identical pipeline, different model.
\n| Model | \nParams (Total/Active) | \nN | \np̂ | \nLB Score | \n
|---|---|---|---|---|
gpt-oss-120b | \n116.8B / 5.1B | \n8 | \n0.69 | \n39.3 (21 runs) | \n
gpt-oss-20b | \n20B / 3.6B | \n8 | \n0.61 | \n31.0 (3 runs) | \n
gpt-oss-20b | \n20B / 3.6B | \n32 | \n0.52 | \n26 | \n
gpt-oss-20b | \n20B / 3.6B | \n3 | \n0.54 | \n28.3 | \n
Nemotron-Super-120B-NVFP4 | \n120B / 12B | \n3 | \n0.47 | \n23 | \n
Qwen3.5-35B-A3B | \n35B / 3B | \n16 | \n0.46 | \n23 | \n
Key observations:
\ngpt-oss-20b at N=32 drops to 26 because per-attempt time shrinks and p̂ falls from 0.61 → 0.52.
Qwen3.5-35B-A3B (8 controlled experiments on 10 local problems): nothing improves beyond baseline. Doubling N from 8→16: no gain. Long prompts: −1. Manufacturer-recommended top_k=20 or presence_penalty: crashes or −1.
Nemotron-Super-120B-NVFP4: 23/50, identical to Qwen despite 4× active parameters. HMMT Feb25 benchmark shows NVFP4 ≈ BF16 (95.4% vs 94.7%), so quantization is not the cause; the N=3 vs N=8 budget is.
\nThe negative result replicates across 3 model families.
\nWith μ̂ = 39.3, σ̂ = 1.7 over 21 baseline runs (range 34–42), each submission is a lottery ticket. P(score ≥ 42 | single run) ≈ 5.6%. The mixer reduces μ without improving the tail. Optimal strategy: submit the unmodified baseline repeatedly. All 42 submissions exhausted.
\nPart of the variance is infrastructure noise. Shared-GPU benchmarks can shift by ~6 points from resource contention alone (Anthropic Engineering, 2026). Infra config is treated as a first-class experimental variable: 21-run statistics, 3-model cross-validation, and exact vLLM flags documented in the paper appendices.
\n
A host-posted analysis (see competition discussion and @AIMOprize on X) reports gpt-oss-120b at pass@20 ≈ 45.5 on the private set (95% CI [43, 48]) and pass@100 ≈ 49/50.
The 6-point gap between the best run (42) and pass@20 (45.5) is not inference-time headroom; Equation 1 already absorbs p and ρ. It is selection loss: the correct answer is in the N=8 pool but outvoted by a more common wrong one. Majority voting is the cheapest selector. A verifier (code execution against problem constraints, formal substitution, cross-candidate consistency) could capture some of that gap. Three training-free verifiers are tested in Section 8; none robustly closes it. The audit in Section 8 locates the failure: 82–86% of the unrecovered correctness loses by a single vote — a one-vote margin, not an entropy hole.
\nThe scope of the negative result is narrower than the title suggests. Prompt-level optimization does not help. Selection-level optimization remains open. The three naive verifiers tested in Section 8 do not robustly close it, and the failure mode (rank-2 losses by one vote) constrains what a working selector would need to do.
\nThe full 23-experiment ablation on gpt-oss-120b (Figure: ablation waterfall, Section 9) confirms the negative result: no single experiment reliably exceeds baseline. The best single run (E12 Code-first, 41/50) was not replicated — confirmation runs scored 38 and 34 (3-run mean 37.7, below baseline 39.3). The baseline is a local optimum, and the ±2-point spread within the table matches the run-to-run variance later observed at competition scale on the private leaderboard.
| Competition | \nWinner (public) | \nWinner (private) | \nThis Work (public) | \nThis Work (private) | \nN | \nInnovation | \n
|---|---|---|---|---|---|---|
| AIMO-1 (2024) | \n29/50 | \n— | \n— | \n— | \n48 | \nBrute-force voting | \n
| AIMO-2 (2025) | \n34/50 | \n— | \n— | \n— | \n64 | \nCustom 540K training + GenSelect | \n
| AIMO-3 (2026) | \n46* | \n44† | \n42‡ | \n40.5§ | \n8 | \nOff-the-shelf model, zero training | \n
* Public peak at competition close (Apr 15), dropped to 41.5/50 on private re-evaluation.\n† Private-LB top score; team was previously ranked outside the top 2,000 with a public score of 37/50. Single submission, no method change.\n‡ Best of 21 baseline submissions on the public set.\n§ Code re-execution verifier submission (Section 8).
\nPrivate-LB shake-up. The AIMO~3 private leaderboard was released after this work was finalized. The host attributed the resulting shake-up to intrinsic model variation arising from team clustering on gpt-oss-120b: with 4,000+ teams converging on the same base model, the leaderboard became more susceptible to run-to-run variance than to method differences. The 4.5-point gap between the public peak (46) and its private score (41.5) sits within 3σ of the same configuration re-submitted, given the σ̂ ≈ 1.7 measured across 21 baseline runs (Section 5). The shake-up is selection loss observed at competition scale — methodological differences across teams are smaller than the spread induced by re-running the same model. The host noted that \"more runs (10+) might have overcome this variation,\" but the per-submission compute budget made this infeasible for any team.
This work's verifier-aware submission (Code re-execution, λ=2.0) scored 41/50 public and 40.5/50 private. The 0.5-point gap is a single problem and sits inside the inter-seed σ=2.24 reported in Section 8 (Table). The submission-level result is consistent with the External-50 finding: code re-execution does not robustly close selection loss, and per-submission variance subsumes any small effect.
\n
The trend across competitions: from high-N to high-p̂. AIMO-1 used N=48 with brute-force voting; AIMO-2 invested 540K training problems with custom GenSelect; this work uses zero training compute on an off-the-shelf model. Model capability first, everything else second.
\nSection 6 identified selection loss as the real headroom. Closing it requires a selector that distinguishes correct from incorrect candidates without ground truth. The established direction is trained verifiers — outcome-supervised, process-supervised, or step-level reward models — but all require annotated training. The cheapest training-free alternatives are tested here, across five seeds on the external 50 of zfturbo (disjoint from the AIMO-3 hidden LB). Implementation: utils/verified_voting.py.
Mechanism. The entropy-weighted vote score S(a) is multiplied by a verifier boost:
$$\\text{final} = \\arg\\max_a \\, S(a) \\cdot \\big(1 + \\lambda \\cdot \\text{verify}(p,a)\\big), \\quad \\lambda = 2.0$$
\nSetting λ=0 recovers entropy-weighted voting. Verification runs only on the top-K=3 most-voted candidates; unanimous votes skip it. Three strategies:
\nstdout matches. Zero LM calls.Results (External 50, n=5 seeds; paired problem-level bootstrap, B=10,000):
\n| Selector | \nMean ± σ | \nΔ | \n95% CI | \np | \n
|---|---|---|---|---|
| Majority vote | \n27.0 ± 2.35 | \n— | \n— | \n— | \n
| Constraint chk. | \n27.2 ± 1.30 | \n+0.2 | \n[−2.8, +3.2] | \n.895 | \n
| Code re-exec. | \n25.0 ± 2.24 | \n−2.0 | \n[−5.6, +1.6] | \n.305 | \n
| LM re-verify | \n29.6 ± 1.82 | \n+2.6 | \n[−0.4, +5.4] | \n.080 | \n
All three 95% CIs overlap zero. Constraint check is indistinguishable from the baseline. Code re-execution costs two problems on average. LM re-verification trends positive (+2.6, p=0.080), but its CI extends to −0.4. None of the three robustly closes selection loss. All four configurations clock 287–299 s per problem; verifier overhead is within the inter-strategy noise floor.
\n
Selection-loss audit (where the gap actually lives). Each run on the external 50 generates the correct answer for 41.4 problems on average and ships it for only 27.3. The 14-problem gap is selection loss: the right answer is in the attempt pool but loses the vote.
\n| Selector | \nVoted | \nGenerated | \nGap | \nRank-2 | \n
|---|---|---|---|---|
| Constraint chk. | \n27.2 | \n42.6 | \n+15.4 | \n81% | \n
| Code re-exec. | \n25.0 | \n41.0 | \n+16.0 | \n81% | \n
| LM re-verify | \n29.6 | \n40.6 | \n+11.0 | \n86% | \n
| Pooled | \n27.3 | \n41.4 | \n+14.1 | \n82% | \n
82–86% of the unrecovered correctness is a rank-2 loss — the correct answer was the second most common vote, beaten by a single extra agreeing attempt. The bottleneck is a one-vote margin, not an entropy hole.
\nWhy each strategy fails. Constraint check fires only on problems with explicit numeric ranges; on the rest it adds zero signal. Code re-execution depends on a brittle stdout match — the model often computes an intermediate float in the final code block, then applies post-processing (rounding, modular arithmetic, length count) before \\boxed{}. The valid candidate is scored as invalid, and the verifier flips a correct vote to a wrong one (e.g., IMO548: 22030→22031, off by one). LM re-verification has the largest mean effect (+2.6) but the largest spread per problem: across 5 seeds the inter-seed σ=1.82 is the same size as the effect.
Competition LB. The final submission uses Code re-execution with λ=2.0: 41/50 public, 40.5/50 private (Section 7). The 0.5-point public-vs-private gap sits inside the inter-seed σ=2.24 measured on the external set — the submission-level result is consistent with \"no robust effect.\"
\nBottom line. Three naive training-free verifiers do not close selection loss at this scale. The audit locates the failure precisely: 82–86% of generated-but-discarded correct answers lose by a single vote. Closing this gap likely requires either (i) a trained verifier with calibrated per-candidate confidence, or (ii) a selector that breaks rank-2 ties using signal orthogonal to vote count (semantic agreement across attempts, derivation-graph consistency, or external constraint propagation). Selection loss remains open.
\n
Use the largest model that fits. Keep temperature high. Stop engineering prompts. Spend remaining budget on a verifier, not a mixer. Six points separate the best run from pass@20. That gap is selection loss. Three training-free verifiers (Constraint, Code re-execution, LM re-verification) do not robustly close it — all three 95% CIs overlap zero across 5 seeds (Section 8). The audit locates the failure: 86% of the unrecovered correctness loses by a single vote — a rank-2 margin, not an entropy hole. Closing this gap is the open problem this work does not solve, and the private-LB shake-up at competition scale (Section 7) confirms the same diagnosis: run-to-run variance of the base model dominates methodological differences across teams.
\ngpt-oss-20b baseline: nguyennguyen599/aimo3-skills-optional-luck-required-gpt-oss-20bThanks to the AIMO Prize Committee and Kaggle for organizing the competition and providing GPU infrastructure. The baseline notebook builds on the inference pipeline by nihilisticneuralnet.
", "rawMarkdown": "# Model Capability Dominates: Inference-Time Optimization Lessons from AIMO 3\n\n**Score:** 42/50 (best run), 39.3 mean over 21 runs\n**Model:** `gpt-oss-120b` (116.8B total / 5.1B active MoE, FP8)\n**Config:** N=8, T=1.0, entropy-weighted majority voting, single H100 80GB\n**Paper:** [arXiv:2603.27844](https://arxiv.org/abs/2603.27844) | **Code:** [GitHub](https://github.com/nat-nischw/model-capability-dominates-lessons-aimo3)\n\n---\n\n## TL;DR\n\nThis work tests whether assigning *diverse reasoning strategies* to different voters improves majority voting on IMO-level problems. It does not. Across **3 models, 23+ experiments, and 50 problems**, every prompt-level intervention failed to beat the baseline reliably. High-temperature sampling already decorrelates errors; weaker prompts reduce per-attempt accuracy more than they reduce correlation. **Model capability dominates all prompt-level optimizations by 4×**: an 8-point gap at equal N=8 vs. ±2-point prompt effects.\n\nThe six-point gap between the best run (42) and `pass@20 ≈ 45.5` on `gpt-oss-120b` is not inference-time headroom. It is **selection loss**. Three training-free verifiers (constraint check, code re-execution, LM re-verification) are tested across 5 seeds; **none robustly closes the gap** (all 95% CIs overlap zero). The audit locates the failure: **82–86% of the unrecovered correctness loses by a single vote** — a rank-2 margin, not an entropy hole. Closing it remains open. The private LB (released after submission) confirms the diagnosis at competition scale: a 4.5-point shake-up between public and private scores on the same configuration sits within 3σ of base-model run-to-run variance. This work scored **40.5/50 private** with the code-re-execution verifier submission.\n\n\n\n---\n\n## 1. The Idea That Didn't Work\n\nMajority voting with correlated errors reduces to effective sample size:\n\n$$ N_{\\text{eff}} = \\frac{N}{1 + (N-1)\\rho} $$\n\nAt ρ=0.3, eight attempts become 2.6 effective votes. The natural fix: assign different reasoning prompts (\"Small Cases First\", \"Work Backwards\", \"Classify Then Solve\") so errors decorrelate.\n\n**Hypothesis:** diverse prompts -> lower ρ -> higher expected score.\n**Result:** every mixer configuration scored at or below baseline. Monotonic degradation with diversity.\n\n\n\n| Configuration | N | LB Score | Δ |\n|---|---|---|---|\n| Baseline (21-run mean) | 8 | **39.3** | — |\n| Conservative mix (5+1+1+1) | 8 | 40 | +0.7 |\n| Aggressive mix (3+2+2+1) | 8 | 40 | +0.7 |\n| Equal mix (2+2+2+2) | 8 | 38 | −1.3 |\n| 8× Small Cases | 8 | 37 | −2.3 |\n| 8× Work Backwards | 8 | 39 | −0.3 |\n| 8× Classify | 8 | 36 | −3.3 |\n| 8× Code-first | 8 | 41 / 38 / 34 | mean 37.7 |\n| Formalize-First (F-1) | 8 | 39 | −0.3 |\n\n---\n\n## 2. Why It Fails\n\n**(a) Temperature already decorrelates.** At T=1.0, sampling alone provides most achievable diversity. T ablation: T=0.5→38, T=0.8→40, T=1.0→39.3, T=1.2→37. Prompt diversity on top of high T is redundant.\n\n**(b) Pairwise correlation is already near zero.** Using the method-of-moments estimator on all four models (19 computable points), **every estimate is negative**. Large-N subset (N≥7, 7 points): mean ρ̂ = **−0.122**. Full sample: −0.348. Wrong answers scatter across distinct values instead of clustering; there is no correlation headroom to exploit.\n\n\n\n**(c) The critical test (E10).** Equal mix (2+2+2+2) at T=0.5, where prompt diversity should help most because stochastic diversity is suppressed, scored **36**, worse than T=0.5 alone (38). Diversity does not compensate even under the ideal conditions for it to help.\n\n**(d) Weaker prompts lower p̂.** The mixer reduces ρ but also reduces per-attempt accuracy. Accuracy loss exceeds decorrelation gain.\n\n---\n\n## 3. Model Capability Dominates\n\nThe cleanest comparison: identical N=8, identical pipeline, different model.\n\n| Model | Params (Total/Active) | N | p̂ | LB Score |\n|---|---|---|---|---|\n| `gpt-oss-120b` | 116.8B / 5.1B | 8 | 0.69 | **39.3** (21 runs) |\n| `gpt-oss-20b` | 20B / 3.6B | 8 | 0.61 | 31.0 (3 runs) |\n| `gpt-oss-20b` | 20B / 3.6B | 32 | 0.52 | 26 |\n| `gpt-oss-20b` | 20B / 3.6B | 3 | 0.54 | 28.3 |\n| `Nemotron-Super-120B-NVFP4` | 120B / 12B | 3 | 0.47 | 23 |\n| `Qwen3.5-35B-A3B` | 35B / 3B | 16 | 0.46 | 23 |\n\n**Key observations:**\n- 8-point gap at equal N=8 (39.3 vs 31.0) is **4× larger** than any prompt optimization (±2 points).\n- Scaling N beyond compute budget **backfires**: `gpt-oss-20b` at N=32 drops to 26 because per-attempt time shrinks and p̂ falls from 0.61 → 0.52.\n- The model capability gap persists at every N.\n\n---\n\n## 4. Cross-Model Validation\n\n\n\n**Qwen3.5-35B-A3B** (8 controlled experiments on 10 local problems): nothing improves beyond baseline. Doubling N from 8→16: no gain. Long prompts: −1. Manufacturer-recommended `top_k=20` or `presence_penalty`: crashes or −1.\n\n**Nemotron-Super-120B-NVFP4**: 23/50, identical to Qwen despite 4× active parameters. HMMT Feb25 benchmark shows NVFP4 ≈ BF16 (95.4% vs 94.7%), so quantization is not the cause; the N=3 vs N=8 budget is.\n\nThe negative result replicates across 3 model families.\n\n---\n\n## 5. Submission as Lottery\n\nWith μ̂ = 39.3, σ̂ = 1.7 over 21 baseline runs (range 34–42), each submission is a lottery ticket. P(score ≥ 42 | single run) ≈ 5.6%. The mixer reduces μ without improving the tail. **Optimal strategy: submit the unmodified baseline repeatedly.** All 42 submissions exhausted.\n\nPart of the variance is infrastructure noise. Shared-GPU benchmarks can shift by ~6 points from resource contention alone [(Anthropic Engineering, 2026)](https://www.anthropic.com/engineering/infrastructure-noise). Infra config is treated as a first-class experimental variable: 21-run statistics, 3-model cross-validation, and exact vLLM flags documented in the paper appendices.\n\n\n\n---\n\n## 6. Selection Loss (The Open Direction)\n\nA host-posted analysis (see [competition discussion](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/679559) and [@AIMOprize on X](https://x.com/AIMOprize/status/2039441022996934783)) reports `gpt-oss-120b` at **pass@20 ≈ 45.5** on the private set (95% CI [43, 48]) and **pass@100 ≈ 49/50**.\n\nThe 6-point gap between the best run (42) and pass@20 (45.5) is **not** inference-time headroom; Equation 1 already absorbs p and ρ. It is **selection loss**: the correct answer is in the N=8 pool but outvoted by a more common wrong one. Majority voting is the cheapest selector. A verifier (code execution against problem constraints, formal substitution, cross-candidate consistency) could capture some of that gap. **Three training-free verifiers are tested in Section 8; none robustly closes it.** The audit in Section 8 locates the failure: **82–86% of the unrecovered correctness loses by a single vote** — a one-vote margin, not an entropy hole.\n\n**The scope of the negative result is narrower than the title suggests. Prompt-level optimization does not help. Selection-level optimization remains open. The three naive verifiers tested in Section 8 do not robustly close it, and the failure mode (rank-2 losses by one vote) constrains what a working selector would need to do.**\n\n---\n\n## 7. AIMO Competition Progression (Public → Private LB)\n\nThe full 23-experiment ablation on `gpt-oss-120b` (Figure: ablation waterfall, Section 9) confirms the negative result: **no single experiment reliably exceeds baseline**. The best single run (E12 Code-first, 41/50) was not replicated — confirmation runs scored 38 and 34 (3-run mean 37.7, below baseline 39.3). The baseline is a local optimum, and the ±2-point spread within the table matches the run-to-run variance later observed at competition scale on the private leaderboard.\n\n| Competition | Winner (public) | Winner (private) | This Work (public) | This Work (private) | N | Innovation |\n|---|---|---|---|---|---|---|\n| AIMO-1 (2024) | 29/50 | — | — | — | 48 | Brute-force voting |\n| AIMO-2 (2025) | 34/50 | — | — | — | 64 | Custom 540K training + GenSelect |\n| AIMO-3 (2026) | 46* | **44**† | **42**‡ | **40.5**§ | **8** | Off-the-shelf model, zero training |\n\n\\* Public peak at competition close (Apr 15), dropped to **41.5/50** on private re-evaluation.\n† Private-LB top score; team was previously ranked outside the top 2,000 with a public score of 37/50. Single submission, no method change.\n‡ Best of 21 baseline submissions on the public set.\n§ Code re-execution verifier submission ([Section 8](#8-naive-verifier-aware-voting)).\n\n**Private-LB shake-up.** The AIMO~3 private leaderboard was released after this work was finalized. The host attributed the resulting shake-up to *intrinsic model variation* arising from team clustering on `gpt-oss-120b`: with 4,000+ teams converging on the same base model, the leaderboard became more susceptible to run-to-run variance than to method differences. The 4.5-point gap between the public peak (46) and its private score (41.5) sits within 3σ of the same configuration re-submitted, given the σ̂ ≈ 1.7 measured across 21 baseline runs (Section 5). The shake-up is **selection loss observed at competition scale** — methodological differences across teams are smaller than the spread induced by re-running the same model. The host noted that \"more runs (10+) might have overcome this variation,\" but the per-submission compute budget made this infeasible for any team.\n\nThis work's verifier-aware submission (Code re-execution, λ=2.0) scored 41/50 public and **40.5/50 private**. The 0.5-point gap is a single problem and sits inside the inter-seed σ=2.24 reported in Section 8 (Table). The submission-level result is consistent with the External-50 finding: code re-execution does not robustly close selection loss, and per-submission variance subsumes any small effect.\n\n\n\nThe trend across competitions: **from high-N to high-p̂**. AIMO-1 used N=48 with brute-force voting; AIMO-2 invested 540K training problems with custom GenSelect; this work uses zero training compute on an off-the-shelf model. Model capability first, everything else second.\n\n---\n\n## 8. Naive Verifier-Aware Voting\n\nSection 6 identified selection loss as the real headroom. Closing it requires a selector that distinguishes correct from incorrect candidates without ground truth. The established direction is *trained* verifiers — outcome-supervised, process-supervised, or step-level reward models — but all require annotated training. The cheapest **training-free** alternatives are tested here, across **five seeds** on the external 50 of [zfturbo](https://www.kaggle.com/datasets/zfturbo/hard-math-problems-for-aimo-3) (disjoint from the AIMO-3 hidden LB). Implementation: [`utils/verified_voting.py`](https://github.com/nat-nischw/model-capability-dominates-lessons-aimo3/blob/main/utils/verified_voting.py).\n\n**Mechanism.** The entropy-weighted vote score `S(a)` is multiplied by a verifier boost:\n\n$$\\text{final} = \\arg\\max_a \\, S(a) \\cdot \\big(1 + \\lambda \\cdot \\text{verify}(p,a)\\big), \\quad \\lambda = 2.0$$\n\nSetting λ=0 recovers entropy-weighted voting. Verification runs only on the top-K=3 most-voted candidates; unanimous votes skip it. Three strategies:\n\n1. **Constraint check.** Parse the problem for numeric ranges (\"between 0 and 99999\", \"positive integer\"); reject violators. **Zero LM calls.**\n2. **Code re-execution.** Re-run the final Python block of the highest-entropy attempt with the candidate substituted; check that `stdout` matches. **Zero LM calls.**\n3. **LM re-verification.** Ask the same model \"Is *a* the answer?\"; use $\\log p(\\text{YES}) - \\log p(\\text{NO})$. **One LM call per candidate.**\n\n**Results (External 50, n=5 seeds; paired problem-level bootstrap, B=10,000):**\n\n| Selector | Mean ± σ | Δ | 95% CI | p |\n|---|---|---|---|---|\n| Majority vote | 27.0 ± 2.35 | — | — | — |\n| Constraint chk. | 27.2 ± 1.30 | **+0.2** | [−2.8, +3.2] | .895 |\n| Code re-exec. | 25.0 ± 2.24 | **−2.0** | [−5.6, +1.6] | .305 |\n| LM re-verify | 29.6 ± 1.82 | **+2.6** | [−0.4, +5.4] | .080 |\n\n**All three 95% CIs overlap zero.** Constraint check is indistinguishable from the baseline. Code re-execution costs two problems on average. LM re-verification trends positive (+2.6, p=0.080), but its CI extends to −0.4. None of the three robustly closes selection loss. All four configurations clock 287–299 s per problem; verifier overhead is within the inter-strategy noise floor.\n\n\n\n\n**Selection-loss audit (where the gap actually lives).** Each run on the external 50 generates the correct answer for **41.4 problems on average** and ships it for only **27.3**. The 14-problem gap is selection loss: the right answer is in the attempt pool but loses the vote.\n\n| Selector | Voted | Generated | Gap | Rank-2 |\n|---|---|---|---|---|\n| Constraint chk. | 27.2 | 42.6 | +15.4 | 81% |\n| Code re-exec. | 25.0 | 41.0 | +16.0 | 81% |\n| LM re-verify | 29.6 | 40.6 | +11.0 | 86% |\n| **Pooled** | **27.3** | **41.4** | **+14.1** | **82%** |\n\n**82–86% of the unrecovered correctness is a rank-2 loss** — the correct answer was the second most common vote, beaten by a single extra agreeing attempt. **The bottleneck is a one-vote margin, not an entropy hole.**\n\n**Why each strategy fails.** Constraint check fires only on problems with explicit numeric ranges; on the rest it adds zero signal. Code re-execution depends on a brittle `stdout` match — the model often computes an intermediate float in the final code block, then applies post-processing (rounding, modular arithmetic, length count) before `\\boxed{}`. The valid candidate is scored as invalid, and the verifier flips a correct vote to a wrong one (e.g., IMO548: 22030→22031, off by one). LM re-verification has the largest mean effect (+2.6) but the largest spread per problem: across 5 seeds the inter-seed σ=1.82 is the same size as the effect.\n\n**Competition LB.** The final submission uses Code re-execution with λ=2.0: **41/50 public, 40.5/50 private** (Section 7). The 0.5-point public-vs-private gap sits inside the inter-seed σ=2.24 measured on the external set — the submission-level result is consistent with \"no robust effect.\"\n\n**Bottom line.** Three naive training-free verifiers do not close selection loss at this scale. The audit locates the failure precisely: 82–86% of generated-but-discarded correct answers lose by a single vote. Closing this gap likely requires either (i) a trained verifier with calibrated per-candidate confidence, or (ii) a selector that breaks rank-2 ties using signal orthogonal to vote count (semantic agreement across attempts, derivation-graph consistency, or external constraint propagation). Selection loss remains open.\n\n---\n\n## 9. Conclusion\n\n\n\n**Use the largest model that fits. Keep temperature high. Stop engineering prompts. Spend remaining budget on a verifier, not a mixer.** Six points separate the best run from pass@20. That gap is selection loss. Three training-free verifiers (Constraint, Code re-execution, LM re-verification) do not robustly close it — all three 95% CIs overlap zero across 5 seeds (Section 8). The audit locates the failure: **86% of the unrecovered correctness loses by a single vote** — a rank-2 margin, not an entropy hole. Closing this gap is the open problem this work does not solve, and the private-LB shake-up at competition scale (Section 7) confirms the same diagnosis: run-to-run variance of the base model dominates methodological differences across teams.\n\n---\n\n## Links\n\n- **Paper (arXiv):** [arxiv.org/abs/2603.27844](https://arxiv.org/abs/2603.27844)\n- **Code + Notebooks:** [github.com/nat-nischw/model-capability-dominates-lessons-aimo3](https://github.com/nat-nischw/model-capability-dominates-lessons-aimo3)\n- **Colab Notebook:** https://colab.research.google.com/drive/18BpnXSITnB7cSxkUVp50d1dbvV2hn0jC?usp=sharing\n- **Baseline Notebook:** Built on [nihilisticneuralnet/44-50-let-me-over-cook](https://www.kaggle.com/code/nihilisticneuralnet/44-50-let-me-over-cook)\n- **Public `gpt-oss-20b` baseline:** [nguyennguyen599/aimo3-skills-optional-luck-required-gpt-oss-20b](https://www.kaggle.com/code/nguyennguyen599/aimo3-skills-optional-luck-required-gpt-oss-20b)\n\n## Acknowledgments\n\nThanks to the AIMO Prize Committee and Kaggle for organizing the competition and providing GPU infrastructure. The baseline notebook builds on the inference pipeline by nihilisticneuralnet.\n", "votes": 16 }, { "id": 3444171, "postDate": "2026-04-17T12:04:50.517Z", "content": "Nice writeup on the prompt mixing experiments. But for the verifier under what conditions would the stdout be incorrect if the sandbox was set up properly? Is it possible the final Python block wasn’t actually used to produce the answer (e.g. the model continued reasoning afterward)? In that case, your verifier could produce false negatives when stdout doesn’t match the boxed answer. Also, a single run seems insufficient to conclude it outperforms a simple majority vote, it might just catch the rare hallucinations.
", "rawMarkdown": "Nice writeup on the prompt mixing experiments. But for the verifier under what conditions would the stdout be incorrect if the sandbox was set up properly? Is it possible the final Python block wasn’t actually used to produce the answer (e.g. the model continued reasoning afterward)? In that case, your verifier could produce false negatives when stdout doesn’t match the boxed answer. Also, a single run seems insufficient to conclude it outperforms a simple majority vote, it might just catch the rare hallucinations. ", "votes": 1, "replies": [ { "id": 3444178, "postDate": "2026-04-17T12:26:25.917Z", "content": "Your analysis is exact. @julianmukaj
\nFalse negatives: You are right. The model frequently computes an intermediate float in the final Python block, but applies final formatting in the subsequent text before generating \\boxed{}. Strict stdout enforcement rejects valid reasoning. This exact mismatch (e.g., IMO548 off-by-1) is documented in Section 8.
Sample size: The writeup explicitly states a single run is insufficient to separate signal from variance. The empirical data proves this: the verifier actually dropped the external set score from 30 to 28. It does not consistently outperform majority voting.
This naive script does not solve the problem. Its purpose is to demonstrate the 6-point Selection Loss gap. The bottleneck is the selector, not the generator.
", "rawMarkdown": "Your analysis is exact. @julianmukaj \n\n- False negatives: You are right. The model frequently computes an intermediate float in the final Python block, but applies final formatting in the subsequent text before generating \\boxed{}. Strict stdout enforcement rejects valid reasoning. This exact mismatch (e.g., IMO548 off-by-1) is documented in Section 8.\n\n- Sample size: The writeup explicitly states a single run is insufficient to separate signal from variance. The empirical data proves this: the verifier actually dropped the external set score from 30 to 28. It does not consistently outperform majority voting.\n\nThis naive script does not solve the problem. Its purpose is to demonstrate the 6-point Selection Loss gap. The bottleneck is the selector, not the generator.", "votes": 1 } ] }, { "id": 3457771, "postDate": "2026-05-14T10:11:52.687Z", "content": "Following up on the questions from @philipvonderlind, here are some additional questions to help us assess your submission:
\n\\boxed{} answer. But some off-by-one cases could reflect ambiguity in the source problem itself (e.g. boundary inclusive/exclusive, or the 'is 0 positive?' issue discussed in this Kaggle thread). Have you looked at whether any of the verifier-hurt flips are on problems where both answers could be reasonably defended so really it is an issue with the problem statement rather than the verifier?Look forward to reading your responses on these.
", "rawMarkdown": "Following up on the questions from @philipvonderlind, here are some additional questions to help us assess your submission:\n\n1. **Next-step verifier direction:** You conclude that closing selection loss likely requires either (i) a trained verifier with calibrated per-candidate confidence, or (ii) a rank-2 tie-breaker using signal orthogonal to vote count. Of those two, which would you prioritise, and what would make a candidate signal feel \"orthogonal enough\" to vote count to be worth testing?\n2. **Off-by-one as ambiguous answers:** Your IMO548 case (22030 vs 22031) is framed as a verifier failure where the intermediate code output differs from the post-processed `\\boxed{}` answer. But some off-by-one cases could reflect ambiguity in the source problem itself (e.g. boundary inclusive/exclusive, or the 'is 0 positive?' issue discussed in [this Kaggle thread](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/672872)). Have you looked at whether any of the verifier-hurt flips are on problems where both answers could be reasonably defended so really it is an issue with the problem statement rather than the verifier?\n\nLook forward to reading your responses on these.\n", "replies": [ { "id": 3457923, "postDate": "2026-05-14T16:10:17.980Z", "content": "Hi @sambealing. Answers below.
\n1. Next-step verifier direction. I prioritize (ii) rank-2 tie-breaker over (i) trained verifier. Three reasons. First, training-free sidesteps a distribution-shift problem. Public process-reward models exist (e.g., Skywork-o1-Open-PRM-Qwen-2.5-7B), in the process-supervision direction introduced by Lightman et al. (2023) for math reasoning. These PRMs are trained on different model trace distributions; transfer to gpt-oss-120b's CoT + code-execution traces is unverified, and validating that transfer requires the annotated set AIMO-class competitions exist to substitute for. Rank-2 tie-breaking avoids that validation step. Second, §8 shows 82–86% of unrecovered correctness loses by a one-vote margin; a small lift on rank-2 yields linear score gain on the quantity that dominates the gap. A trained verifier helps most in the low-p̂ regime; gpt-oss-120b at p̂ ≈ 0.69 is bottlenecked by tie-breaking, not calibration. Third, an orthogonal signal is composable: it can later feed (i) as a feature.
\n\"Orthogonal enough\" has an operational test. Condition one: Spearman |ρ_s| < 0.3 with vote count v(a), so the signal supplements rather than re-weights majority. Condition two: on the §8 rank-2 subset, s(a) discriminates correct from wrong with AUC ≥ 0.6, clearing the inter-seed noise floor (σ ≈ 1.8–2.3 in §8). Three candidates that would pass: (i) semantic agreement across reasoning paths, where different derivations reaching the same answer carry more evidence than identical derivations agreeing; (ii) constraint propagation beyond numeric ranges, testing whether the answer satisfies every parsed problem constraint; (iii) step-level consistency between the reasoning trace and the boxed answer, which is exactly the failure mode in (2).
\n2. Off-by-one as ambiguous answers. Fair sharpening; I agree with the principle. Off-by-one disagreements can reflect problem-statement ambiguity (boundary inclusion, \"0 ∈ ℕ⁺\", under-specified modular reductions), and a verifier flipping correct -> wrong on such a problem is a benchmark-side issue, not a verifier-side issue.
\nTriggered by your question, I audited the five Code re-execution net-hurt cases on the external 50 (majority-of-seeds criterion, n=5 seeds):
\n| Problem | \nBase -> Verif | \nDataset-author note | \nAttribution | \n
|---|---|---|---|
| IMO1131 | \n0.80 -> 0.00 | \n\"Incorrect answer\" | \nReference label issue | \n
| IMO1477 | \n0.60 -> 0.00 | \n\"Easy task, but the model misinterprets the text\" | \nProblem-text ambiguity (your hypothesis) | \n
| N00490 | \n0.60 -> 0.40 | \n(no specific note) | \nHurts all three verifiers symmetrically; problem-side, not verifier-side | \n
| N01418 | \n0.60 -> 0.20 | \n(no specific note) | \nUnclassified | \n
| N01455 | \n0.60 -> 0.40 | \n(no specific note) | \nUnclassified | \n
Three of five are consistent with the benchmark-side framing you proposed, two are dataset-author-flagged independently of my analysis. Two remain unclassified within the rebuttal window.
\nOn IMO548 specifically: the case in §8 is a single-seed mechanistic illustration of how stdout-match interacts with post-processing. The model's reasoning trace computes a value, then applies an off-by-one correction before \\boxed{}; the verifier captures the pre-correction value and scores the valid candidate as invalid. At the multi-seed majority level (baseline 0.80, verifier 0.60), IMO548 is not in the net-hurt list above. §8 uses it to illustrate the failure mechanism, not as a population-level effect.
\nThe broader §8 conclusion does not change under either attribution: all three 95% CIs overlap zero across 5 seeds and 250 problem-runs, so the empirical statement (training-free verifiers do not robustly close selection loss) holds whether per-problem failures attribute to verifier implementation or to problem ambiguity. The 82–86% rank-2 finding is the audit signal that should guide the next selector, independent of per-case attribution.
\nBenchmark-design corollary: closing selection loss at AIMO-class scale requires either tighter problem statements upstream, or selectors that route ambiguous problems away from deterministic verifiers. The first is a benchmark issue; the second is the orthogonal-signal direction in (1).
", "rawMarkdown": "Hi @sambealing. Answers below.\n\n**1. Next-step verifier direction.** I prioritize (ii) rank-2 tie-breaker over (i) trained verifier. Three reasons. First, training-free sidesteps a distribution-shift problem. Public process-reward models exist (e.g., [Skywork-o1-Open-PRM-Qwen-2.5-7B](https://huggingface.co/Skywork/Skywork-o1-Open-PRM-Qwen-2.5-7B)), in the process-supervision direction introduced by [Lightman et al. (2023)](https://arxiv.org/abs/2305.20050) for math reasoning. These PRMs are trained on different model trace distributions; transfer to gpt-oss-120b's CoT + code-execution traces is unverified, and validating that transfer requires the annotated set AIMO-class competitions exist to substitute for. Rank-2 tie-breaking avoids that validation step. Second, §8 shows 82–86% of unrecovered correctness loses by a one-vote margin; a small lift on rank-2 yields linear score gain on the quantity that dominates the gap. A trained verifier helps most in the low-p̂ regime; gpt-oss-120b at p̂ ≈ 0.69 is bottlenecked by tie-breaking, not calibration. Third, an orthogonal signal is composable: it can later feed (i) as a feature.\n\n\"Orthogonal enough\" has an operational test. Condition one: Spearman |ρ_s| < 0.3 with vote count v(a), so the signal supplements rather than re-weights majority. Condition two: on the §8 rank-2 subset, s(a) discriminates correct from wrong with AUC ≥ 0.6, clearing the inter-seed noise floor (σ ≈ 1.8–2.3 in §8). Three candidates that would pass: (i) semantic agreement across reasoning paths, where different derivations reaching the same answer carry more evidence than identical derivations agreeing; (ii) constraint propagation beyond numeric ranges, testing whether the answer satisfies every parsed problem constraint; (iii) step-level consistency between the reasoning trace and the boxed answer, which is exactly the failure mode in (2).\n\n**2. Off-by-one as ambiguous answers.** Fair sharpening; I agree with the principle. Off-by-one disagreements can reflect problem-statement ambiguity (boundary inclusion, \"0 ∈ ℕ⁺\", under-specified modular reductions), and a verifier flipping correct -> wrong on such a problem is a benchmark-side issue, not a verifier-side issue.\n\nTriggered by your question, I audited the five Code re-execution net-hurt cases on the external 50 (majority-of-seeds criterion, n=5 seeds):\n\n| Problem | Base -> Verif | Dataset-author note | Attribution |\n|---|---|---|---|\n| IMO1131 | 0.80 -> 0.00 | \"Incorrect answer\" | Reference label issue |\n| IMO1477 | 0.60 -> 0.00 | \"Easy task, but the model misinterprets the text\" | Problem-text ambiguity (your hypothesis) |\n| N00490 | 0.60 -> 0.40 | (no specific note) | Hurts all three verifiers symmetrically; problem-side, not verifier-side |\n| N01418 | 0.60 -> 0.20 | (no specific note) | Unclassified |\n| N01455 | 0.60 -> 0.40 | (no specific note) | Unclassified |\n\nThree of five are consistent with the benchmark-side framing you proposed, two are dataset-author-flagged independently of my analysis. Two remain unclassified within the rebuttal window.\n\nOn IMO548 specifically: the case in §8 is a single-seed mechanistic illustration of how stdout-match interacts with post-processing. The model's reasoning trace computes a value, then applies an off-by-one correction before \\boxed{}; the verifier captures the pre-correction value and scores the valid candidate as invalid. At the multi-seed majority level (baseline 0.80, verifier 0.60), IMO548 is not in the net-hurt list above. §8 uses it to illustrate the failure mechanism, not as a population-level effect.\n\nThe broader §8 conclusion does not change under either attribution: all three 95% CIs overlap zero across 5 seeds and 250 problem-runs, so the empirical statement (training-free verifiers do not robustly close selection loss) holds whether per-problem failures attribute to verifier implementation or to problem ambiguity. The 82–86% rank-2 finding is the audit signal that should guide the next selector, independent of per-case attribution.\n\nBenchmark-design corollary: closing selection loss at AIMO-class scale requires either tighter problem statements upstream, or selectors that route ambiguous problems away from deterministic verifiers. The first is a benchmark issue; the second is the orthogonal-signal direction in (1)." } ] }, { "id": 3457739, "postDate": "2026-05-14T08:51:54.823Z", "content": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.
\nWe think the reproducibility of your work is already outstanding, and several other writeups referring to your work is a clear indication of its impact. Below are a few notes and questions I had while reading the writeup:
\nSelection Loss Gap: The pass@n statistics we report are computed without tools, so the real gap may be even larger. This seems to further support your assumptions.
N-Generations: In the section Why It Fails, you report results for several models side by side to compare their performance. However, I did not fully understand why different values of N were used for each model. In my view, this makes the results less directly comparable. I assume this choice was due to hardware constraints and the different inference requirements of each model. What is your perspective on this?
Stability: Did any approach you tried make the notebook perform consistently on the public leaderboard? Other approaches, such as the one by ippeiogawa, appear to “stabilize” submission performance. Given your findings, would you attribute their reported stability primarily to noise, or do you think their approach addresses a different failure mode?
Local Evaluation Dataset: Why did you choose ZFTurbo as the benchmark for local evaluation? Did you also try any of the other Math Corpus Prize submissions or publicly available benchmarks?
Looking forward to your response :)
", "rawMarkdown": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the [updated discussion post](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/689703).\n\nWe think the reproducibility of your work is already outstanding, and several other writeups referring to your work is a clear indication of its impact. Below are a few notes and questions I had while reading the writeup:\n\n1. **Selection Loss Gap**: The `pass@n` statistics we report are computed without tools, so the real gap may be even larger. This seems to further support your assumptions.\n\n2. **N-Generations**: In the section `Why It Fails`, you report results for several models side by side to compare their performance. However, I did not fully understand why different values of `N` were used for each model. In my view, this makes the results less directly comparable. I assume this choice was due to hardware constraints and the different inference requirements of each model. What is your perspective on this?\n\n3. **Stability**: Did any approach you tried make the notebook perform consistently on the public leaderboard? Other approaches, such as the [one by ippeiogawa](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/aimo3test), appear to “stabilize” submission performance. Given your findings, would you attribute their reported stability primarily to noise, or do you think their approach addresses a different failure mode?\n\n4. **Local Evaluation Dataset**: Why did you choose ZFTurbo as the benchmark for local evaluation? Did you also try any of the other Math Corpus Prize submissions or publicly available benchmarks?\n\nLooking forward to your response :)\n\n\n", "replies": [ { "id": 3457897, "postDate": "2026-05-14T15:19:44.493Z", "content": "Hi @philipvonderlind. Thanks for the read. Answers below.
\n1. Selection Loss Gap. Confirmed: your post explicitly notes \"used no tool calling or other optimisations\" for the pass@k analysis. My pipeline uses sympy / numpy in 8 persistent kernels per attempt, so per-attempt success rate differs from a tool-free baseline at the same N. I do not claim my N=8 ceiling exceeds your pass@20 ≈ 45.5; that comparison mixes two effects (N=8 vs N=20, with-tools vs without). What I do claim is measured directly in §8 on the same pipeline: per-run generated correct = 41.4 / 50 (n=5 seeds), shipped = 27.3, a 14-point within-pipeline gap independent of tool effects. The §6 / §8 selection-loss claim rests on this within-pipeline measurement, not on a comparison to pass@k.
\n2. N-Generations. Yes, the 5-hour single-pod budget is binding; per-attempt latency scales with active parameters and reasoning length. The clean apples-to-apples row in §3 is gpt-oss-120b @ N=8 (39.3) vs gpt-oss-20b @ N=8 (31.0): same N, same pipeline, 8-point gap. The 4× claim rests on this row. The other 20b rows ablate N within a fixed model and show N cannot recover the gap: at N=32 the score drops to 26 because per-attempt budget shrinks and p̂ falls 0.61 → 0.52. Nemotron @ N=3 and Qwen @ N=16 are budget-bounded upper bounds favorable to them, not point estimates of model rank; the §3 table should be read with that scope.
\n3. Stability. My 21 baseline submissions give σ̂ ≈ 1.7 (range 34–42). §7 replicates the diagnosis at competition scale: the 4.5-point public-vs-private gap on the same config sits inside 3σ. Under this null, \"stability\" across N=2–3 submissions is statistically indistinguishable from noise. I have not inspected ippeiogawa's mechanism, so I will not assert \"noise\" without their multi-run data. Prior: if their pipeline addresses a different failure mode (parse-error retry, format validation, sandbox-timeout recovery), it is complementary and does not contradict the selection-loss claim; if not, the consistency should regress to σ̂ over enough runs.
\n4. Local Evaluation Dataset. Honest answer up front: ZFTurbo was the set the community was actively sharing for AIMO-3-class evaluation, and I ran my baseline on it without an exhaustive search of alternatives. Other Math Corpus Prize submissions were not surveyed within the rebuttal window. This is a real limitation, not a justification dressed up after the fact.
\nTwo reasons it still served the §8 purpose. First, the dataset author publicly documented per-problem gpt-oss-120b success rates across 5–20 seeds; the range is 0–100% with mean ≈ 60%, which gives the discrimination headroom selection-loss work needs. Known label issues are flagged on the dataset card (N01223 with a probably-wrong reference; a few too-easy / too-hard items); the inter-seed σ ≈ 1.8–2.3 in Table §8 absorbs that label noise.
\nSecond, post-hoc cross-validation I did not engineer: the dataset author's own note on N01223 reads \"the true answer often appears as the second-most-likely output (TOP2)\". That is the same failure mode my §8 audit measures at scale: 82–86% of unrecovered correctness is rank-2. Two independent analyses converge on rank-2 as the bottleneck, from different angles.
\nHMMT Feb25 and AIME 2025 were considered but ruled out: gpt-oss-120b saturates both above 0.94, leaving no discrimination for selection-loss effects. The §8 result is a within-pipeline negative claim, scoped to AIMO-3-class IMO-level problems on a disjoint reference set; it does not need broader cross-validation to stand.
", "rawMarkdown": "Hi @philipvonderlind. Thanks for the read. Answers below.\n\n**1. Selection Loss Gap.** Confirmed: your post explicitly notes \"used no tool calling or other optimisations\" for the pass@k analysis. My pipeline uses sympy / numpy in 8 persistent kernels per attempt, so per-attempt success rate differs from a tool-free baseline at the same N. I do not claim my N=8 ceiling exceeds your pass@20 ≈ 45.5; that comparison mixes two effects (N=8 vs N=20, with-tools vs without). What I do claim is measured directly in §8 on the same pipeline: per-run generated correct = 41.4 / 50 (n=5 seeds), shipped = 27.3, a 14-point within-pipeline gap independent of tool effects. The §6 / §8 selection-loss claim rests on this within-pipeline measurement, not on a comparison to pass@k.\n\n**2. N-Generations.** Yes, the 5-hour single-pod budget is binding; per-attempt latency scales with active parameters and reasoning length. The clean apples-to-apples row in §3 is gpt-oss-120b @ N=8 (39.3) vs gpt-oss-20b @ N=8 (31.0): same N, same pipeline, 8-point gap. The 4× claim rests on this row. The other 20b rows ablate N within a fixed model and show N cannot recover the gap: at N=32 the score drops to 26 because per-attempt budget shrinks and p̂ falls 0.61 → 0.52. Nemotron @ N=3 and Qwen @ N=16 are budget-bounded upper bounds favorable to them, not point estimates of model rank; the §3 table should be read with that scope.\n\n**3. Stability.** My 21 baseline submissions give σ̂ ≈ 1.7 (range 34–42). §7 replicates the diagnosis at competition scale: the 4.5-point public-vs-private gap on the same config sits inside 3σ. Under this null, \"stability\" across N=2–3 submissions is statistically indistinguishable from noise. I have not inspected ippeiogawa's mechanism, so I will not assert \"noise\" without their multi-run data. Prior: if their pipeline addresses a different failure mode (parse-error retry, format validation, sandbox-timeout recovery), it is complementary and does not contradict the selection-loss claim; if not, the consistency should regress to σ̂ over enough runs.\n\n**4. Local Evaluation Dataset.** Honest answer up front: ZFTurbo was the set the community was actively sharing for AIMO-3-class evaluation, and I ran my baseline on it without an exhaustive search of alternatives. Other Math Corpus Prize submissions were not surveyed within the rebuttal window. This is a real limitation, not a justification dressed up after the fact.\n\nTwo reasons it still served the §8 purpose. First, the dataset author publicly documented per-problem gpt-oss-120b success rates across 5–20 seeds; the range is 0–100% with mean ≈ 60%, which gives the discrimination headroom selection-loss work needs. Known label issues are flagged on the dataset card (N01223 with a probably-wrong reference; a few too-easy / too-hard items); the inter-seed σ ≈ 1.8–2.3 in Table §8 absorbs that label noise.\n\nSecond, post-hoc cross-validation I did not engineer: the dataset author's own note on N01223 reads \"the true answer often appears as the second-most-likely output (TOP2)\". That is the same failure mode my §8 audit measures at scale: 82–86% of unrecovered correctness is rank-2. Two independent analyses converge on rank-2 as the bottleneck, from different angles.\n\nHMMT Feb25 and AIME 2025 were considered but ruled out: gpt-oss-120b saturates both above 0.94, leaving no discrimination for selection-loss effects. The §8 result is a within-pipeline negative claim, scoped to AIMO-3-class IMO-level problems on a disjoint reference set; it does not need broader cross-validation to stand." } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T04:20:26Z", "competition_id": 118448, "competition_slug": "ai-mathematical-olympiad-progress-prize-3", "team_id": 15176235, "solution_url": "/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/model-capability-dominates" }