# Model Capability Dominates: Why Prompt Diversity Fails

Model Capability Dominates: Inference-Time Optimization Lessons from AIMO 3
Score:
42/50 (best run), 39.3 mean over 21 runs
Model:
gpt-oss-120b
(116.8B total / 5.1B active MoE, FP8)
Config:
N=8, T=1.0, entropy-weighted majority voting, single H100 80GB
Paper:
[arXiv:2603.27844](https://arxiv.org/abs/2603.27844)
|
Code:
[GitHub](https://github.com/nat-nischw/model-capability-dominates-lessons-aimo3)
TL;DR
This work tests whether assigning
diverse reasoning strategies
to different voters improves majority voting on IMO-level problems. It does not. Across
3 models, 23+ experiments, and 50 problems
, every prompt-level intervention failed to beat the baseline reliably. High-temperature sampling already decorrelates errors; weaker prompts reduce per-attempt accuracy more than they reduce correlation.
Model capability dominates all prompt-level optimizations by 4×
: an 8-point gap at equal N=8 vs. ±2-point prompt effects.
The six-point gap between the best run (42) and
pass@20 ≈ 45.5
on
gpt-oss-120b
is not inference-time headroom. It is
selection loss
. Three training-free verifiers (constraint check, code re-execution, LM re-verification) are tested across 5 seeds;
none robustly closes the gap
(all 95% CIs overlap zero). The audit locates the failure:
82–86% of the unrecovered correctness loses by a single vote
— a rank-2 margin, not an entropy hole. Closing it remains open. The private LB (released after submission) confirms the diagnosis at competition scale: a 4.5-point shake-up between public and private scores on the same configuration sits within 3σ of base-model run-to-run variance. This work scored
40.5/50 private
with the code-re-execution verifier submission.
![Model capability dominates: p̂ vs score](assets/abbf533f7fe82c0865ff13e3.png)
1. The Idea That Didn't Work
Majority voting with correlated errors reduces to effective sample size:
$$ N_{\text{eff}} = \frac{N}{1 + (N-1)\rho} $$
At ρ=0.3, eight attempts become 2.6 effective votes. The natural fix: assign different reasoning prompts ("Small Cases First", "Work Backwards", "Classify Then Solve") so errors decorrelate.
Hypothesis:
diverse prompts -> lower ρ -> higher expected score.
Result:
every mixer configuration scored at or below baseline. Monotonic degradation with diversity.
![Prompt diversity vs score](assets/ee9cef94900ecceade52b21f.png)
Configuration
N
LB Score
Δ
Baseline (21-run mean)
8
39.3
—
Conservative mix (5+1+1+1)
8
40
+0.7
Aggressive mix (3+2+2+1)
8
40
+0.7
Equal mix (2+2+2+2)
8
38
−1.3
8× Small Cases
8
37
−2.3
8× Work Backwards
8
39
−0.3
8× Classify
8
36
−3.3
8× Code-first
8
41 / 38 / 34
mean 37.7
Formalize-First (F-1)
8
39
−0.3
2. Why It Fails
(a) Temperature already decorrelates.
At T=1.0, sampling alone provides most achievable diversity. T ablation: T=0.5→38, T=0.8→40, T=1.0→39.3, T=1.2→37. Prompt diversity on top of high T is redundant.
(b) Pairwise correlation is already near zero.
Using the method-of-moments estimator on all four models (19 computable points),
every estimate is negative
. Large-N subset (N≥7, 7 points): mean ρ̂ =
−0.122
. Full sample: −0.348. Wrong answers scatter across distinct values instead of clustering; there is no correlation headroom to exploit.
![Per-problem ρ̂ vs p̂ across four models](assets/356600e4e2ac91cb656527f1.png)
(c) The critical test (E10).
Equal mix (2+2+2+2) at T=0.5, where prompt diversity should help most because stochastic diversity is suppressed, scored
36
, worse than T=0.5 alone (38). Diversity does not compensate even under the ideal conditions for it to help.
(d) Weaker prompts lower p̂.
The mixer reduces ρ but also reduces per-attempt accuracy. Accuracy loss exceeds decorrelation gain.
3. Model Capability Dominates
The cleanest comparison: identical N=8, identical pipeline, different model.
Model
Params (Total/Active)
N
p̂
LB Score
gpt-oss-120b
116.8B / 5.1B
8
0.69
39.3
(21 runs)
gpt-oss-20b
20B / 3.6B
8
0.61
31.0 (3 runs)
gpt-oss-20b
20B / 3.6B
32
0.52
26
gpt-oss-20b
20B / 3.6B
3
0.54
28.3
Nemotron-Super-120B-NVFP4
120B / 12B
3
0.47
23
Qwen3.5-35B-A3B
35B / 3B
16
0.46
23
Key observations:
8-point gap at equal N=8 (39.3 vs 31.0) is
4× larger
than any prompt optimization (±2 points).
Scaling N beyond compute budget
backfires
:
gpt-oss-20b
at N=32 drops to 26 because per-attempt time shrinks and p̂ falls from 0.61 → 0.52.
The model capability gap persists at every N.
4. Cross-Model Validation
![Qwen3.5-35B-A3B ablation](assets/f6e1606616b1f414c415a669.png)
Qwen3.5-35B-A3B
(8 controlled experiments on 10 local problems): nothing improves beyond baseline. Doubling N from 8→16: no gain. Long prompts: −1. Manufacturer-recommended
top_k=20
or
presence_penalty
: crashes or −1.
Nemotron-Super-120B-NVFP4
: 23/50, identical to Qwen despite 4× active parameters. HMMT Feb25 benchmark shows NVFP4 ≈ BF16 (95.4% vs 94.7%), so quantization is not the cause; the N=3 vs N=8 budget is.
The negative result replicates across 3 model families.
5. Submission as Lottery
With μ̂ = 39.3, σ̂ = 1.7 over 21 baseline runs (range 34–42), each submission is a lottery ticket. P(score ≥ 42 | single run) ≈ 5.6%. The mixer reduces μ without improving the tail.
Optimal strategy: submit the unmodified baseline repeatedly.
All 42 submissions exhausted.
Part of the variance is infrastructure noise. Shared-GPU benchmarks can shift by ~6 points from resource contention alone
[(Anthropic Engineering, 2026)](https://www.anthropic.com/engineering/infrastructure-noise)
. Infra config is treated as a first-class experimental variable: 21-run statistics, 3-model cross-validation, and exact vLLM flags documented in the paper appendices.
![Submission as lottery ticket](assets/b5bcdd5f3ca83eaf6a86e261.png)
6. Selection Loss (The Open Direction)
A host-posted analysis (see
[competition discussion](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/679559)
and
[@AIMOprize on X](https://x.com/AIMOprize/status/2039441022996934783)
) reports
gpt-oss-120b
at
pass@20 ≈ 45.5
on the private set (95% CI [43, 48]) and
pass@100 ≈ 49/50
.
The 6-point gap between the best run (42) and pass@20 (45.5) is
not
inference-time headroom; Equation 1 already absorbs p and ρ. It is
selection loss
: the correct answer is in the N=8 pool but outvoted by a more common wrong one. Majority voting is the cheapest selector. A verifier (code execution against problem constraints, formal substitution, cross-candidate consistency) could capture some of that gap.
Three training-free verifiers are tested in Section 8; none robustly closes it.
The audit in Section 8 locates the failure:
82–86% of the unrecovered correctness loses by a single vote
— a one-vote margin, not an entropy hole.
The scope of the negative result is narrower than the title suggests. Prompt-level optimization does not help. Selection-level optimization remains open. The three naive verifiers tested in Section 8 do not robustly close it, and the failure mode (rank-2 losses by one vote) constrains what a working selector would need to do.
7. AIMO Competition Progression (Public → Private LB)
The full 23-experiment ablation on
gpt-oss-120b
(Figure: ablation waterfall, Section 9) confirms the negative result:
no single experiment reliably exceeds baseline
. The best single run (E12 Code-first, 41/50) was not replicated — confirmation runs scored 38 and 34 (3-run mean 37.7, below baseline 39.3). The baseline is a local optimum, and the ±2-point spread within the table matches the run-to-run variance later observed at competition scale on the private leaderboard.
Competition
Winner (public)
Winner (private)
This Work (public)
This Work (private)
N
Innovation
AIMO-1 (2024)
29/50
—
—
—
48
Brute-force voting
AIMO-2 (2025)
34/50
—
—
—
64
Custom 540K training + GenSelect
AIMO-3 (2026)
46*
44
†
42
‡
40.5
§
8
Off-the-shelf model, zero training
* Public peak at competition close (Apr 15), dropped to
41.5/50
on private re-evaluation.
† Private-LB top score; team was previously ranked outside the top 2,000 with a public score of 37/50. Single submission, no method change.
‡ Best of 21 baseline submissions on the public set.
§ Code re-execution verifier submission (
[Section 8](#8-naive-verifier-aware-voting)
).
Private-LB shake-up.
The AIMO~3 private leaderboard was released after this work was finalized. The host attributed the resulting shake-up to
intrinsic model variation
arising from team clustering on
gpt-oss-120b
: with 4,000+ teams converging on the same base model, the leaderboard became more susceptible to run-to-run variance than to method differences. The 4.5-point gap between the public peak (46) and its private score (41.5) sits within 3σ of the same configuration re-submitted, given the σ̂ ≈ 1.7 measured across 21 baseline runs (Section 5). The shake-up is
selection loss observed at competition scale
— methodological differences across teams are smaller than the spread induced by re-running the same model. The host noted that "more runs (10+) might have overcome this variation," but the per-submission compute budget made this infeasible for any team.
This work's verifier-aware submission (Code re-execution, λ=2.0) scored 41/50 public and
40.5/50 private
. The 0.5-point gap is a single problem and sits inside the inter-seed σ=2.24 reported in Section 8 (Table). The submission-level result is consistent with the External-50 finding: code re-execution does not robustly close selection loss, and per-submission variance subsumes any small effect.
![AIMO progression (29→34→44 winners, 48→64→8 voters; public peak 46 dropped to 41.5 on private; 44 is private top)](assets/7c2be68bfb550d57ffe5d87c.png)
The trend across competitions:
from high-N to high-p̂
. AIMO-1 used N=48 with brute-force voting; AIMO-2 invested 540K training problems with custom GenSelect; this work uses zero training compute on an off-the-shelf model. Model capability first, everything else second.
8. Naive Verifier-Aware Voting
Section 6 identified selection loss as the real headroom. Closing it requires a selector that distinguishes correct from incorrect candidates without ground truth. The established direction is
trained
verifiers — outcome-supervised, process-supervised, or step-level reward models — but all require annotated training. The cheapest
training-free
alternatives are tested here, across
five seeds
on the external 50 of
[zfturbo](https://www.kaggle.com/datasets/zfturbo/hard-math-problems-for-aimo-3)
(disjoint from the AIMO-3 hidden LB). Implementation:
[utils/verified_voting.py](https://github.com/nat-nischw/model-capability-dominates-lessons-aimo3/blob/main/utils/verified_voting.py)
.
Mechanism.
The entropy-weighted vote score
S(a)
is multiplied by a verifier boost:
$$\text{final} = \arg\max_a \, S(a) \cdot \big(1 + \lambda \cdot \text{verify}(p,a)\big), \quad \lambda = 2.0$$
Setting λ=0 recovers entropy-weighted voting. Verification runs only on the top-K=3 most-voted candidates; unanimous votes skip it. Three strategies:
Constraint check.
Parse the problem for numeric ranges ("between 0 and 99999", "positive integer"); reject violators.
Zero LM calls.
Code re-execution.
Re-run the final Python block of the highest-entropy attempt with the candidate substituted; check that
stdout
matches.
Zero LM calls.
LM re-verification.
Ask the same model "Is
a
the answer?"; use $\log p(\text{YES}) - \log p(\text{NO})$.
One LM call per candidate.
Results (External 50, n=5 seeds; paired problem-level bootstrap, B=10,000):
Selector
Mean ± σ
Δ
95% CI
p
Majority vote
27.0 ± 2.35
—
—
—
Constraint chk.
27.2 ± 1.30
+0.2
[−2.8, +3.2]
.895
Code re-exec.
25.0 ± 2.24
−2.0
[−5.6, +1.6]
.305
LM re-verify
29.6 ± 1.82
+2.6
[−0.4, +5.4]
.080
All three 95% CIs overlap zero.
Constraint check is indistinguishable from the baseline. Code re-execution costs two problems on average. LM re-verification trends positive (+2.6, p=0.080), but its CI extends to −0.4. None of the three robustly closes selection loss. All four configurations clock 287–299 s per problem; verifier overhead is within the inter-strategy noise floor.
![Per-problem majority outcome across 5 seeds (Both correct / Verifier helped / Verifier hurt / Both wrong; net flips: Constraint +3, Code +1, LM re-verify +7)](assets/9ddbdd0d81ad4d5496e4c69f.png)
Selection-loss audit (where the gap actually lives).
Each run on the external 50 generates the correct answer for
41.4 problems on average
and ships it for only
27.3
. The 14-problem gap is selection loss: the right answer is in the attempt pool but loses the vote.
Selector
Voted
Generated
Gap
Rank-2
Constraint chk.
27.2
42.6
+15.4
81%
Code re-exec.
25.0
41.0
+16.0
81%
LM re-verify
29.6
40.6
+11.0
86%
Pooled
27.3
41.4
+14.1
82%
82–86% of the unrecovered correctness is a rank-2 loss
— the correct answer was the second most common vote, beaten by a single extra agreeing attempt.
The bottleneck is a one-vote margin, not an entropy hole.
Why each strategy fails.
Constraint check fires only on problems with explicit numeric ranges; on the rest it adds zero signal. Code re-execution depends on a brittle
stdout
match — the model often computes an intermediate float in the final code block, then applies post-processing (rounding, modular arithmetic, length count) before
\boxed{}
. The valid candidate is scored as invalid, and the verifier flips a correct vote to a wrong one (e.g., IMO548: 22030→22031, off by one). LM re-verification has the largest mean effect (+2.6) but the largest spread per problem: across 5 seeds the inter-seed σ=1.82 is the same size as the effect.
Competition LB.
The final submission uses Code re-execution with λ=2.0:
41/50 public, 40.5/50 private
(Section 7). The 0.5-point public-vs-private gap sits inside the inter-seed σ=2.24 measured on the external set — the submission-level result is consistent with "no robust effect."
Bottom line.
Three naive training-free verifiers do not close selection loss at this scale. The audit locates the failure precisely: 82–86% of generated-but-discarded correct answers lose by a single vote. Closing this gap likely requires either (i) a trained verifier with calibrated per-candidate confidence, or (ii) a selector that breaks rank-2 ties using signal orthogonal to vote count (semantic agreement across attempts, derivation-graph consistency, or external constraint propagation). Selection loss remains open.
9. Conclusion
![Complete ablation](assets/72484b97c2806c84581cb574.png)
Use the largest model that fits. Keep temperature high. Stop engineering prompts. Spend remaining budget on a verifier, not a mixer.
Six points separate the best run from pass@20. That gap is selection loss. Three training-free verifiers (Constraint, Code re-execution, LM re-verification) do not robustly close it — all three 95% CIs overlap zero across 5 seeds (Section 8). The audit locates the failure:
86% of the unrecovered correctness loses by a single vote
— a rank-2 margin, not an entropy hole. Closing this gap is the open problem this work does not solve, and the private-LB shake-up at competition scale (Section 7) confirms the same diagnosis: run-to-run variance of the base model dominates methodological differences across teams.
Links
Paper (arXiv):
[arxiv.org/abs/2603.27844](https://arxiv.org/abs/2603.27844)
Code + Notebooks:
[github.com/nat-nischw/model-capability-dominates-lessons-aimo3](https://github.com/nat-nischw/model-capability-dominates-lessons-aimo3)
Colab Notebook:
[https://colab.research.google.com/drive/18BpnXSITnB7cSxkUVp50d1dbvV2hn0jC?usp=sharing](https://colab.research.google.com/drive/18BpnXSITnB7cSxkUVp50d1dbvV2hn0jC?usp=sharing)
Baseline Notebook:
Built on
[nihilisticneuralnet/44-50-let-me-over-cook](https://www.kaggle.com/code/nihilisticneuralnet/44-50-let-me-over-cook)
Public
gpt-oss-20b
baseline:
[nguyennguyen599/aimo3-skills-optional-luck-required-gpt-oss-20b](https://www.kaggle.com/code/nguyennguyen599/aimo3-skills-optional-luck-required-gpt-oss-20b)
Acknowledgments
Thanks to the AIMO Prize Committee and Kaggle for organizing the competition and providing GPU infrastructure. The baseline notebook builds on the inference pipeline by nihilisticneuralnet.