# A Practitioner's Plateau on gpt-oss-120b-MXFP4

Three Substrate Traps on gpt-oss-120b-MXFP4 for AIMO3: A Reproducibility Package Accompanied by Sixteen Falsified Consensus Modifications
Author:
Juan Sebastian Gil Pinzon and Kimberley Duran (Kagglers:
sebastiangil00
,
kimberleyduran
; team
Hail Mary
)
Competition:
[AIMO Progress Prize 3](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3)
Final result:
public 42/50, private 39.5/50; final rank 2048 / 3451 teams.
Submission:
[AIMO3 v7 - Winner Fork - Version 12](https://www.kaggle.com/code/sebastiangil00/aimo3-v7-winner-fork)
Base model (HuggingFace):
[unsloth/gpt-oss-120b](https://huggingface.co/unsloth/gpt-oss-120b)
(MXFP4, unmodified — no fine-tuning, no LoRA merge)
Kaggle-agnostic reproduction:
[](https://colab.research.google.com/github/SebastianGilPinzon/aimo3-writeup-public/blob/v1.0-rebuttal/colab_inference.ipynb)
— one-click, CPU-tier compatible for the rule-based detectors; H100 path for full pipeline.
Reproducibility package:
[GitHub v1.0-rebuttal](https://github.com/SebastianGilPinzon/aimo3-writeup-public/tree/v1.0-rebuttal)
(frozen tag; unit tests +
reproduce.sh
+ figure-generator +
reproducibility/COMMIT.txt
with the pinned commit SHA) ·
[Kaggle Dataset](https://www.kaggle.com/datasets/sebastiangil00/aimo3-writeup-artifacts)
Score basis:
public 42/50, private 39.5/50 (−2.5 pp gap = 0.4σ inside the variance-model band of §2.7), rank 2048 / 3451 teams (final private leaderboard, 2026-05).
Abstract
We ship three artifacts for future AIMO entrants on the
gpt-oss-120b-MXFP4 + vLLM 0.11.2
substrate: runnable detectors for three named substrate traps, a sixteen-row falsification catalog, and a one-command Kaggle-agnostic reproduction of our final 42/50 submission. The pinned substrate is
gpt-oss-120b
[1, 2] (MoE expert projections in MXFP4, attention and router in BF16) served through the vLLM 0.11.2 wheel on a single NVIDIA H100 80GB GPU.
Artifact 1: Three named substrate traps, each with a runnable detection script
(§6 and
tests/
). On this substrate, three candidate improvements that appear theoretically sound silently fail in ways that pass type-checks and smoke-tests but destroy the leaderboard score:
LoRA-on-MXFP4 merge-then-requantize collapse
(§6.2,
tests/test_lora_mxfp4_collapse.py
) — the AIMO3 manifestation of the adapter–quantization incompatibility addressed by QA-LoRA [14]. Merging a published AIMO2 LoRA adapter [4] into the MXFP4-quantized base and re-quantizing produces
W′ ≈ W
: the low-magnitude delta
BA
rounds into the same E2M1 [1] representable bin as the base weight. Empirically, the SNR-collapse threshold sits at adapter-to-base ratios ≈2⁻⁴ of the per-block scale; the huikang AIMO2 adapter we tested had per-element delta ratios ≈2⁻⁶ of base scale, well inside the collapsed regime (Fig. 2). Leaderboard impact: 42 → 40.
EAGLE-3 draft-target speculative decoding [15] producing zero-or-degenerate output tokens on vLLM 0.11.2 + MXFP4 + sparse-MoE
(§6.1,
tests/test_eagle3_moe_zero_tokens.py
) — the EAGLE-3 auxiliary draft head assumes dense-layer hidden states; MoE expert dispatch in gpt-oss-120b breaks feature alignment, and the MXFP4 kernel path in vLLM 0.11.2 lacks the fused speculative-verify kernel. The failure surface upstream is heterogeneous: load-time
NotImplementedError
, severe quality collapse, or empty-completion fallback; on our exact substrate the manifestation was empty completions silently parsed to a zero answer. Tracked upstream in vllm-project/vllm (issue IDs in §6.1). Leaderboard impact: 42 → 0.
Sqrt-prior Bayesian vote re-weighting silently inverting correct majorities on a calibrated base model
(§6.3,
tests/test_bayesian_sqrt_inversion.py
) — reweighting the posterior as
π(answer) ∝ √count(answer) · prior(answer)
down-weights the strongest-supported cluster. For a base model with answer-level accuracy >0.5 on easy problems, Condorcet–Jury Theorem conditions [11, 12] favor the plain majority, and our reweighting is
anti-Condorcet
. Leaderboard impact: 42 → 23.
Each detection script returns a boolean verdict in under five minutes on a single NVIDIA H100 80GB. A future competitor can
git clone
the package, run
pytest tests/
, and detect all three traps before burning a submission slot.
Artifact 2: A sixteen-row falsification catalog
(§4, Appendix A1). Over 22 days and ≈180 H100-hours, we tested sixteen candidate modifications to the AIMO3 public-consensus pipeline, drawn from the AIMO2 literature [3, 4], inference-time scaling theory [5, 6], verifier-based selection [7, 8, 9], self-consistency [10], majority-voting theory [11, 12], and quantized fine-tuning [13, 14].
Not one produced a detectable leaderboard improvement over the controlled baseline
, and none survived Benjamini–Hochberg correction at q=0.10 on the 199-problem validation set. The catalog reports, per variant: hypothesis, mechanism, validation-199 effect with Benjamini–Hochberg-adjusted q-value, single-shot leaderboard delta, H100-hours consumed, and the Git commit SHA at which the hypothesis and its falsification criterion were committed
before
the submission that falsified it.
Artifact 3: A reproduction specification and one-command script
(§3) that regenerates our final
submission.parquet
to within two-run seed tolerance on any H100 80GB. The specification pins the vLLM wheel SHA, the gpt-oss-120b weight commit, the Kaggle Docker image digest, a complete
pip freeze
, all prompts verbatim, and the decoding configuration. Our final submission scored 42/50 public and 39.5/50 private — a single realization of the stochastic-sampling distribution analysed in §2.7, with the −2.5 pp gap sitting 0.4σ inside the variance-model band (σ ≈ 6.0 pp on a paired 50-problem gap at measured baseline accuracy p̂ = 0.76 on val-199; 95 % interval ≈ ±12 pp). The observation is consistent with the variance model but not evidence of it; we offer the package as an end-to-end reproducibility target, not as evidence of a substrate ceiling.
We
explicitly do not claim
42 / 50 is the achievable ceiling on this substrate; the released private leaderboard places the top four teams at 44.0/50 (≈+4.5 pp above our private 39.5), and their methods are not publicly disclosed at the time of writing. What we document is a
practitioner's plateau
: the failure envelope of a two-person, inference-only effort that had full access to the public-consensus recipe and 22 days of H100 time to depart from it. We extrapolate tentatively, grounded in the training-heavy wins of Numina [3] and NemoSkills [4], that the path beyond the 44.0 cluster on this substrate likely requires training-data interventions which the quantization regime forecloses without access to full-precision weights.
In one sentence:
this paper is a tool, not a story — three unit tests, sixteen pre-committed falsifications, and a one-command reproduction, assembled so the next AIMO4 entrant does not re-learn what we paid ≈180 H100-hours to learn.
![Figure 3. Submission timeline over the 22-day study: every departure from the day-5 baseline regressed or was neutral.](assets/edc4137ba5b128fba809b03c.png)
Figure 3. Submission timeline over the 22-day study. The day-5 baseline (v7 Winner Fork, 42/50 public) is never exceeded; all 16 subsequent departures regress or are leaderboard-neutral. The final submission (day 22) is byte-identical to the day-5 baseline and scored 42/50 public, 39.5/50 private — a −2.5 pp gap consistent with the variance band of §2.7. (Plot is on the public-LB axis; post-deadline private scores were released 2026-05 and are discussed in §3.6.)
1. Introduction
1.1 Task and scoring
AIMO Progress Prize 3 poses 110 original olympiad-level mathematics problems (50 public, 50 private, 10 reference), with answers restricted to integers in [0, 99999]. Solutions must be produced by an open-weight model running offline on a single NVIDIA H100 80GB GPU within 5 hours of wall-clock time. The scoring metric is penalized accuracy: the submission notebook is executed twice on the private set, and each problem contributes 1.0 if both runs are correct, 0.5 if exactly one is, and 0.0 otherwise. Internet is disabled during inference; only pre-loaded models and datasets are accessible.
The penalized-accuracy rule creates a subtle incentive. A pipeline that is
deterministic in structure but stochastic in sampling
— using time-based seeds so that the two scoring runs draw different samples from the same output distribution — can collect 0.5 partial credit on problems the model solves with probability in the open interval (0, 1). A perfectly deterministic pipeline can only get 1.0 or 0.0. The public-consensus pipeline is stochastic in sampling, and so is ours. When we describe the pipeline below as "reproducible," we mean
reproducible in specification
: given the same model weights, environment, prompts, and decoding hyperparameters, the full distribution over outputs is reproduced; any single run is a draw from that distribution.
1.2 Substrate
The 80GB / 5h envelope tightly constrains the design space. Open-weight models above ≈130B parameters do not fit under any quantization that preserves olympiad-level reasoning; models below ≈14B sacrifice accuracy. The public notebook ecosystem converged on
gpt-oss-120b
[2] in its official release configuration — sparse mixture-of-experts architecture with MoE expert projections quantized to MXFP4 [1] (OCP Microscaling FP4: E2M1 mantissa with a shared E8M0 block scale per 32-element block), while attention projections and the expert-routing network remain in BF16 — served through the vLLM 0.11.2 wheel pinned by the official
gpt-oss
inference distribution, which is the effective version for any Kaggle-offline-compatible deployment. We adopt this substrate unchanged and treat it as a fixed environment rather than a design variable. All claims below are scoped to this specific substrate; we do not generalize beyond it.
1.3 Positioning
Four teams tied at 44.0/50 on the AIMO3 private leaderboard (Exalted Joseph, varianceofx, SKobayak, taku); of these, only Exalted Joseph's
["LET ME (over)COOK!!!" notebook](https://www.kaggle.com/code/theexaltedone/44-50-let-me-over-cook)
[23] is publicly disclosed at the time of writing. The AIMO1 and AIMO2 competitions were each won by training-heavy pipelines: Numina [3] fine-tuned DeepSeekMath-7B on the NuminaMath corpus to win AIMO1, and NemoSkills [4] combined a TIR-trained generator with a separately-trained GenSelect selector to win AIMO2. Both approaches front-load the contribution in
training data and training protocol
, not in inference-time interventions. Against this prior art, the central empirical question we pursue is narrower than "how does one win AIMO3?":
For a small team restricted to inference-time modifications of the public-consensus recipe on the substrate of §1.2, which departures yield leaderboard improvements?
Our answer, below, is that
none of the sixteen departures we tested did
. The best-performing configuration we found is the unmodified baseline. This is not evidence of a true substrate ceiling; it is evidence of a
practitioner's plateau
— the failure envelope of our own 22-day inference-only effort. Training-data interventions (foreclosed for us by the quantization regime, per §6.2) remain a plausible and likely path to higher scores, consistent with Numina and NemoSkills.
Post-deadline corroboration:
the public top scorer (
[Seungjun Lee, 45/50 public](https://www.kaggle.com/code/seungjunlee00/aimo3-public-45-50-public-solution-write-up)
) used no fine-tuning, no EAGLE-3, and no Bayesian re-weighting — independently consistent with our trap diagnoses in §6.
1.4 Method and discipline
Our methodology is
pre-committed journaling
, not formal pre-registration. Every hypothesis in Section 6 was recorded with its falsification criterion in a dated entry of the public journey log (
docs/MASTER.md
, 7,253 lines, Appendix A6)
before
the corresponding submission was pushed; commit timestamps precede leaderboard-result timestamps for all sixteen entries, verifiable via
git log
. This is weaker than trusted-timestamp pre-registration (OSF, ClinicalTrials.gov) but stronger than post-hoc rationalization, and we name the discipline explicitly to avoid overclaiming.
We follow the negative-results reporting posture of Dodge et al. [16] and Lipton & Steinhardt [17]. Each hypothesis is a controlled variant relative to the baseline, tested first on a 199-problem internal validation set (Appendix A3) and, where the validation signal warranted, on the leaderboard. The validation set has limited statistical power: for a McNemar paired test on 199 problems with a discordant-pair proportion in the 10–20 % range typical of our val-199 experiments, the minimum detectable effect at α = 0.05 (two-sided) and 1 − β = 0.80 is approximately
6–9 percentage points
(
MDE ≈ (z_{α/2} + z_β)·√(π_disc / N)
= 2.80·√(π_disc / 199)). Single-submission leaderboard observations on the 50-problem public split carry a binomial 95 % CI of approximately ±6 points. All individual effect sizes reported must be read in this light: a "regression" under ±6 pp LB CI or a val-199 shift under ≈8 pp is statistically consistent with the null of no effect, not evidence of harm. The aggregate claim is stronger than any single component. Formally: under the two-sided null H₀: Δ_v = 0 vs H₁: Δ_v ≠ 0 (effect on validation), we observed zero BH-significant positives at q = 0.10 across the sixteen variants (per-variant McNemar p-values are two-sided), and the sixteen non-positive sign-test outcomes give a one-sided combined p ≈ 3 × 10⁻⁵ against the alternative "any tested inference-only departure improves over the baseline on this substrate". The validation-to-leaderboard transportability caveat still applies: the 199-problem validation distribution is not identical to the 50-problem public split.
Total compute budget expended on the study: ≈180 H100-hours across the sixteen falsifications and thirty reference reproduction runs (≈86 H100-hours on the three Artifact-1 traps alone, reflecting the two 0/50 submissions whose pipelines completed on full 50-problem sets before being diagnosed).
1.5 Contributions and explicit non-contributions
Contributions:
Three substrate-trap detection scripts
(
tests/test_lora_mxfp4_collapse.py
,
tests/test_eagle3_moe_zero_tokens.py
,
tests/test_bayesian_sqrt_inversion.py
), runnable in under five minutes each on an H100, returning a boolean verdict. Each script reproduces the failure mechanism on a minimal input so that a future competitor can detect the trap before committing a submission slot to it (§6, Appendix A2).
A reproduction specification
regenerating our final
submission.parquet
to within two-run seed tolerance on any H100 80GB, pinned to exact library versions, weight hashes, Docker image digest, prompts, and decoding configuration (§3).
A sixteen-row falsification catalog
with per-variant {hypothesis, mechanism, validation-199 effect ± CI, McNemar p and BH-adjusted q, leaderboard delta, H100-hours, commit SHA at which the hypothesis was pre-committed} (§4 summary, Appendix A1 complete).
A substrate-bounded empirical observation
(§7): within the scope of our sixteen inference-only departures, the best-performing configuration on gpt-oss-120b-MXFP4 + vLLM 0.11.2 is the baseline. The path to >44/50 on this substrate was not found by our effort.
Explicit non-contributions:
This is
not a winning solution
. Our score of 42 is tied with the public consensus.
This is
not a proof of an inference-time ceiling
. Four private-LB teams reached 44.0; the failure envelope we document is ours, not the substrate's.
This is
not formally pre-registered
(OSF). Discipline is Git-timestamped journaling.
This is
not an ablation of training-data interventions
. We did not run any successfully; our LoRA attempt is a falsification, not an ablation.
The three findings are
not novel phenomena in general
. Each reduces to a known mechanism from the cited literature. Our contribution is their
AIMO3-specific manifestation
, their
detection script
, and their
leaderboard-verified cost
.
2. Solution Lifecycle
Our submission is functionally byte-identical to the consensus public notebook that scored 42/50 on the AIMO3 public leaderboard throughout March 2026. We describe it here at the component level for reference; the full source is linked in the submission URL above and mirrored in the reproducibility package.
2.1 Pipeline at a glance
Per problem, the pipeline executes: (1) a Harmony-formatted prompt is constructed from the system prompt (§2.2) and the problem statement; (2) eight independent assistant rollouts are dispatched in parallel to a vLLM server (§2.3), each sampling at temperature 1.0 with min-p 0.02 and top-logprobs 5, interleaved with Python execution in a dedicated stateful Jupyter kernel (§2.4); (3) each rollout terminates on
<|return|>
, on reaching the per-attempt token budget, or on an early-stop signal when four rollouts have converged to the same answer (§2.5); (4) the eight (or fewer, post-early-stop) final answers are combined by entropy-weighted majority voting (§2.6). The final answer is written to
submission.parquet
with one row per problem; the problem-serving gateway advances to the next problem.
2.2 Harmony prompt
We use OpenAI's Harmony chat format [2] with
reasoning_effort=HIGH
, the system prompt shown in Appendix A2.1 (verbatim), and two tool namespaces: a stateful Python sandbox (§2.4) and no web access. The system prompt is the public-notebook default — we did not modify it in the final submission. §6.4 documents three prompt variants we tested and falsified.
2.3 Decoding configuration
vLLM 0.11.2 is served locally as an OpenAI-compatible API on
127.0.0.1:8000
, with
--gpu-memory-utilization 0.94
,
--kv-cache-dtype fp8_e4m3
,
--max-model-len 65536
,
--served-model-name gpt-oss
, and the gpt-oss-120b weights auto-detected at
/kaggle/input/gpt-oss-120b/transformers/default/1
. Sampling uses
temperature=1.0
,
min_p=0.02
,
top_k=-1
(unrestricted), and streams tokens with
stream_interval=200
. We additionally request
top_logprobs=5
per generated token, not as a sampling filter but as an
observability
channel feeding the §2.6 entropy-weighted voting rule; the logprobs are read from the streaming response and do not influence token selection. We use
time-based seeds
per rollout — the two-run scoring rule rewards stochastic sampling (§1.1), and a fixed seed forfeits the 0.5-partial-credit EV. The full vLLM command string is in §3.2.
2.4 TIR via stateful Jupyter kernels
Each rollout owns a dedicated
jupyter_client.KernelManager
IPython kernel, pre-imported with
math
,
numpy
,
sympy
,
itertools
,
collections
,
mpmath
(with
mpmath.mp.dps = 64
). When the assistant emits a
<|call|>
to the Python tool namespace, the code is executed inside that kernel, output is captured with a 6-second wall-clock timeout, and re-injected as a
<|message|>
from the
functions.python
role. Kernel state persists across tool calls within a rollout (variable definitions, imports, intermediate computations); kernels are recycled between problems. Sandbox pool size equals the number of concurrent rollouts (
workers=8
). TIR is applied at inference time over the TIR-style-trained gpt-oss-120b checkpoint; we do not claim a novel TIR training contribution.
2.5 Early-stop weighted
A
threading.Event
is shared across the eight rollout workers. After each rollout completes, a shared
collections.Counter
of finalized answers is updated under a
threading.Lock
; if any single answer has accumulated
early_stop=4
or more votes, the event is set, and the remaining in-flight rollouts exit their token-streaming loops on the next iteration. This captures the typical-problem savings documented in §4 (validation-set time reduction of ≈40% with no detectable effect on accuracy). Because
workers == pool_size == attempts == 8
, the sandbox queue is never contended in normal operation and
sandbox_timeout=3s
serves only as a safety net against pathological kernel hangs.
2.6 Entropy-weighted voting
Per rollout we compute the
mean token-level Shannon entropy
across the top-5 logprobs of every generated token, producing a scalar
H
in nats-per-token. The voting rule is:
weight(rollout) = 1 / max(H, 0.3)
score(answer)   = Σ weight(rollout) over rollouts that boxed `answer`
final           = argmax_answer score(answer)
The
max(H, 0.3)
floor caps a single low-entropy rollout's weight at ≈3.33, preventing a confidently-wrong rollout from overriding a higher-entropy consensus. Ties break arbitrarily (Python dict insertion order). No code-execution-result voting, no verifier, no LLM judge. §6.3 documents the falsified sqrt-prior Bayesian alternative.
2.7 Two-run scoring and the variance-reduction argument
We sample stochastically by design. For a problem solved with probability
p ∈ (0, 1)
, expected per-problem score is
p
under
both
deterministic-fixed-seed and stochastic-independent-seed strategies:
p² + 2p(1−p)·0.5 + (1−p)²·0 = p
. Time-based seeds are therefore not an EV play — they are a variance-reduction play. Per-problem variance of the
final
score is
Var_stochastic = 0.5 · p(1−p)
vs
Var_single-run = p(1−p)
: stochastic sampling halves per-problem variance, and under independence across the 50 problems halves the variance of the sum, tightening the realized-score interval around its expectation.
Post-deadline note (added 2026-05-13, rebuttal phase).
The released private leaderboard scored our submission at 39.5/50, a −2.5 pp gap to our 42/50 public. Under the variance model derived above at the measured baseline accuracy
p̂ = 0.76
on val-199, σ(gap) = √(2 · 0.5 · p̂(1 − p̂) / 50) ≈ 6.04 pp on a paired 50-problem split (95 % interval ≈ ±12 pp). The realized gap is 0.41σ inside that interval (two-tailed p ≈ 0.68 against H₀: zero true gap). The host's post-release diagnosis of broad public→private movement as "intrinsic model variation" is independently consistent with this prediction. We frame the gap as
consistent with
the §2.7 variance model, not as evidence of it; a single realization cannot discriminate between substrate-variance, problem-distribution drift, and operator-luck. The pre-registered variance theorem precedes any private observation (Git commit, 2026-04-08, predating the release by five weeks).
3. Reproducibility
This section is the load-bearing reproducibility claim. The specification below, together with the companion repository and Kaggle Dataset, is intended to let a reader regenerate
submission.parquet
to within two-run seed tolerance on any H100 80GB within ≈3 hours of wall-clock time.
3.1 Environment
Component
Pin
GPU
NVIDIA H100 80GB HBM3, compute capability 9.0
CUDA runtime
12.2 (Kaggle default)
NVIDIA driver
≥ 535.86.10
Docker image
gcr.io/kaggle-private-byod/python@sha256:00377cd1b3d470a605bc5b0ceca79969e369644e9b36802242a1c70e627372f9
Python
3.11 (Kaggle default)
vLLM
0.11.2 from official
gpt-oss
wheel (pinned by
kernel_sources: andreasbis/aimo-3-utils
[20]; the public GPT-OSS-120B-with-tools baseline [21] is the inference scaffold our pipeline builds on)
Base model
danielhanchen/gpt-oss-120b/Transformers/default/1
(Kaggle Models) /
[unsloth/gpt-oss-120b](https://huggingface.co/unsloth/gpt-oss-120b)
(HuggingFace mirror); MoE experts in MXFP4, attention/router in BF16
openai-harmony
as pinned in
aimo-3-utils
jupyter-client
as pinned in
aimo-3-utils
The complete
pip freeze
output from the submission environment (≈350 lines) is provided at
reproducibility/pip_freeze.lock
in the companion GitHub repository and sha256-hashed in
reproducibility/pip_freeze.sha256
.
3.2 vLLM server command
python -m vllm.entrypoints.openai.api_server \
  --model /kaggle/input/gpt-oss-120b/transformers/default/1 \
  --served-model-name gpt-oss \
  --dtype auto \
  --quantization mxfp4 \
  --kv-cache-dtype fp8_e4m3 \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.94 \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --host 127.0.0.1 \
  --port 8000 \
  --disable-log-stats
The
--quantization mxfp4
flag is required to engage the MXFP4 kernel path for MoE expert projections on vLLM 0.11.2; without it,
--dtype auto
may silently fall back to a BF16-only path that either OOMs on H100 80GB or produces mismatched numerics against the submission environment.
--trust-remote-code
is required for the gpt-oss architecture on the pinned wheel. The server is spawned as a subprocess and polled for readiness on
http://127.0.0.1:8000/v1/models
with a 300-second timeout (first-invocation Triton compile can take ≈90 s).
3.3 Decoding and pipeline configuration
Parameter
Value
Rationale (§6 for falsified alternatives)
temperature
1.0
§6.8: temperatures 0.5 / 0.7 / 0.8 falsified on val-199
min_p
0.02
§6.9: 0.05 falsified (a public-notebook fork variant we tested)
top_logprobs
5
Required for §2.6 entropy weighting
top_k
-1 (unrestricted)
§6.10: top_k=40 falsified
attempts
(K)
8
§6.11: K=4 / K=12 / K=18 / K=24 falsified
turns
(max TIR rounds)
128
Effectively unbounded; per-rollout token budget is the real cap
context_tokens
65536
Matches vLLM
--max-model-len
buffer_tokens
512
Safety margin between prompt length and
max_tokens
search_tokens
32
Sliding window for stop-phrase detection
workers
8
Concurrent rollouts; matches
attempts
early_stop
4
Majority-threshold for early termination (§2.5)
jupyter_timeout
6 s
Per-TIR-call wall-clock cap
sandbox_timeout
3 s
Queue-acquire timeout for sandbox pool
base_problem_timeout
300 s
Soft per-problem budget
high_problem_timeout
900 s
Problems flagged as high-difficulty (§3.5)
session_timeout
960 s
Hard per-problem wall-clock cap
seed
int(time.time())
per rollout
§1.1 two-run stochasticity
All prompts (system, tool, preference) appear verbatim in Appendix A2; the voting weight formula in §2.6 appears verbatim in the companion repository at
reproducibility/voting.py
.
3.4 One-command reproduction — two modes
On any H100 80GB host with the pinned Docker image, the package supports
two reproduction modes
addressing distinct reviewer needs. A reproducibility reviewer can run either or both:
git clone https://github.com/SebastianGilPinzon/aimo3-writeup-public
cd aimo3-writeup-public

# Mode A (STRICT): fixed seed, byte-identical submission.parquet
REPRODUCE_DETERMINISTIC=1 bash reproduce.sh
python verify.py submission.parquet --mode strict    # asserts sha256 match

# Mode B (FAITHFUL): time-based seeds, mirrors the actual submission
bash reproduce.sh
python verify.py submission.parquet --mode stochastic --n-ref 30
Mode A (strict-determinism).
Sets
PYTHONHASHSEED=42
,
vllm.sampling.seed=42
,
torch.use_deterministic_algorithms(True)
,
CUBLAS_WORKSPACE_CONFIG=:4096:8
, disables
kv_cache_dtype
casting noise, and overrides
seed = 42
per rollout. Produces a
submission.parquet
whose sha256 is published in
reproducibility/strict_mode_sha256.txt
. This is the canonical bitwise-reproducibility target for reviewers.
Mode B (stochastic-faithful).
Uses
seed = int(time.time())
per rollout, matching the actual submission environment and the two-run scoring strategy (§2.7).
verify.py --mode stochastic --n-ref 30
runs a Hoeffding-bound binomial test: for each of the ten reference problems, it asserts that the returned answer falls within the 95% support of the empirical answer distribution across 30 pre-computed reference runs (Appendix A5). A malicious or broken re-implementation producing a fixed wrong answer fails unless that answer appeared in >5% of our 30 runs, which is a non-trivial bar.
Independent of either mode
— and runnable by any reviewer without an H100 —
sha256sum --check reproducibility/artifact_sha256.txt
verifies the SHA256 of the seventeen deterministic files that constitute this writeup package (notebook source, data, reproducibility scripts, tests, figures, config). All seventeen pass on a clean clone in under one second and provide a cryptographic integrity guarantee of the non-submission-parquet portion of the package.
reproduce.sh
executes in four stages: (1) verify environment pins against
reproducibility/environment.lock
; (2) ensure gpt-oss-120b weights are cached (SHA256 verified against
reproducibility/model_weights.sha256
); (3) launch the vLLM server with the §3.2 command string and poll for readiness; (4) run the submission pipeline on the ten reference problems (Kaggle gateway is bypassed locally via a replay harness, see
reproducibility/local_gateway.py
). Expected wall-clock: ≈25 minutes for the 10-problem reference set in either mode; ≈2.5 hours for the 50-problem private-set-equivalent dry run.
3.5 Artifacts
Artifact
Location
Purpose
reproduce.sh
reproducibility/
One-command pipeline (§3.4)
verify.py
reproducibility/
Two-run-tolerance verification
environment.lock
reproducibility/
Pinned versions and digests
pip_freeze.lock
reproducibility/
Complete dependency snapshot
notebook.py
submission/
Full submission source (verbatim from
sebastiangil00/aimo3-v7-winner-fork
Version 12)
tests/test_lora_mxfp4_collapse.py
tests/
§6.2 trap detection
tests/test_eagle3_moe_zero_tokens.py
tests/
§6.1 trap detection
tests/test_bayesian_sqrt_inversion.py
tests/
§6.3 trap detection
expected_hashes.json
reproducibility/
Reference-run answer distributions
reference.csv
data/
10 official AIMO3 reference problems with ground truths
val_199.csv
data/
199-problem internal validation set (construction in Appendix A3)
3.6 Public–private gap as a reproducibility target
Our final submission scored 42/50 public and 39.5/50 private — a gap of −2.5 percentage points. Under the variance model of §2.7, the standard deviation of the paired public–private gap on a 50-problem split at the measured baseline accuracy
p̂ = 0.76
(val-199) is
σ ≈ √(2 · 0.5 · p̂(1 − p̂) / 50) ≈ 6.04 pp
, giving a 95 % gap interval of approximately ±12 pp; the realized −2.5 pp sits 0.41σ inside this interval (two-tailed p ≈ 0.68). We therefore frame the observation as
consistent with
the stochastic-sampling model, not as evidence of it: a single realization cannot discriminate substrate-variance from problem-distribution drift or operator-luck. We do
not
claim this consistency as evidence of a substrate ceiling, of superior calibration, or of low variance per se. We offer it as a concrete end-to-end verification target: any re-implementation of §§3.1–3.4 that scores within approximately ±12 pp of 42 on an independent 50-problem draw is consistent with having faithfully reproduced the pipeline.
3.7 What this specification uniquely pins (a frozen substrate snapshot)
A non-obvious but load-bearing feature of §3:
no public AIMO3 notebook pins all four of
(Docker image digest, vLLM wheel SHA, gpt-oss-120b weight commit, verbatim prompts)
as a set
. Public forks typically pin the model path and one or two hyperparameters. §3.1–3.3 together define a
frozen substrate snapshot
— a dated configuration of the gpt-oss-120b + vLLM 0.11.2 + Kaggle-base-image stack — against which the three trap detection scripts of §6 are scoped. The tests are falsifiable precisely
because
the substrate is pinned: a future drift in any of the four pins invalidates the exact tests and demands either an update or a re-falsification. In this sense §3 is not only a reproduction recipe for a single submission; it is the reference configuration under which §6's substrate claims remain testable going forward.
4. Ablation Catalog: Sixteen Falsifications
4.1 Methodology
Each variant was implemented as a controlled departure from the §2 baseline, modifying one mechanism while holding all others fixed. For each variant we report: (i) a short hypothesis statement, pre-committed in
docs/MASTER.md
with a Git timestamp preceding the submission; (ii) the family (Selection, Generation, Speed, Prompt, Multi-model) to which the departure belongs; (iii) the validation-199 paired-accuracy delta Δ_val relative to the baseline, with 95% Wilson interval; (iv) the submission leaderboard delta Δ_LB on the public 50-problem split (n=1 realization); (v) H100-hours consumed; (vi) the short commit SHA at which the hypothesis was pre-committed. Under BH correction at q=0.10,
zero variants survive
as positive-effect candidates on val-199.
Baseline (§2 pipeline unchanged): val-199 = 151/199 ≈ 0.759; LB = 42/50.
4.2 Summary table
Baseline row is the control; the remaining
sixteen rows are the sixteen falsifications
referenced in the contributions list. Δ_val reports the Wilson 95% CI half-width on a McNemar paired test (199 problems).
q_BH
is the Benjamini–Hochberg-adjusted two-sided q-value against H₀: Δ_val ≤ 0, computed jointly over the sixteen comparisons at FDR = 0.10.
#
Variant (ID)
Family
Hypothesis
Δ_val (pp)
q_BH
Δ_LB (pp)
H100h
Verdict
0
Baseline (§2 pipeline)
—
Canonical public recipe
0.0
—
0 (=42/50)
—
Control
1
Bayesian sqrt-prior vote (v18)
Selection
√-prior over vote counts tightens posterior on well-supported clusters
−9.0 ± 6.0
1.00
−19
(→23/50)
10
Falsified; §6.3
2
EAGLE-3 speculative decoding (v15)
Speed
2–3× throughput at equal accuracy → K=20 at 5h budget
−85 ± 2 (degenerate: 0-token output)
1.00
−42
(→0/50)
18
Falsified; §6.1
3
LoRA fine-tune (huikang adapter, v13)
Generation
AIMO2-winning adapter ports to MXFP4 via merge-then-requant
−0.8 ± 5.6
1.00
−2
(→40/50)
12
Falsified; §6.2
4
GenSelect JUDGE (v20)
Selection
LLM-judge over 8 rollouts > entropy-weighted majority
−3.2 ± 5.8
1.00
−10
(→32/50)
14
Falsified
5
RF classifier over candidate features (v26)
Selection
Random-forest over token/entropy/code-call features predicts correctness
+0.0 train; −1.8 avg on 5 holdouts
1.00
not submitted
22
Falsified (overfit on val)
6
gpt-oss-20B second opinion (v22)
Multi-model
Small-model vote breaks ties on unclear problems
−1.4 ± 5.8
1.00
not submitted
6
Falsified (20B ≈ 1/10 on reference)
7
Code-output voting (v25)
Selection
Vote on Python stdout instead of boxed answer
−4.5 ± 5.7
1.00
not submitted
5
Falsified (GT is intermediate, not stdout)
8
Length-based downweighting (v25-short)
Selection
Short answers → more confident → up-weight
−2.8 ± 5.8
1.00
not submitted
3
Falsified
9
Early-stop=5 (vs baseline 4)
Speed
Tighter consensus threshold before stopping
−1.0 ± 5.9
1.00
not submitted
2
Falsified
10
RETRY on no-consensus (v19 r21)
Generation
Extra rollout when 8 disagree → recovers hard problems
+0.0 ± 5.9
1.00
−10
(→32/50)
8
Falsified (0/7 on val, time-wasted)
11
PRM-weighted voting (v28-prm)
Selection
Process reward model weights per-step quality
−1.0 ± 5.9
1.00
not submitted
11
Falsified
12
Dual-prompt ensemble (v29)
Prompt
Mix two system prompts → diversity increases voting power
−2.2 ± 5.8
1.00
−7
(→35/50)
9
Falsified
13
Evolutionary GA over voting weights (1000 gen)
Selection
GA over (entropy floor, length penalty, code bonus)
+2.0 ± 5.8 train; −1.8 ± 5.0 on 5 random 50/50 splits
1.00
not submitted
26
Falsified (noise fitting)
14
Thought-prefix prompt (huikang-style, v7-thought)
Prompt
Inject "verify by substitution" prefix → self-correction
0.0 ± 5.9
1.00
not submitted
4
Falsified
15
Code-fix retry (ippeiogawa-style, v7-codefix)
Generation
Auto-retry on code error → rescues no-answer rollouts
0.0 ± 5.9
1.00
not submitted
4
Falsified (0/7 rescues; 86e8e5 NOT rescued)
16
min_p=0.05 (public-notebook fork variant, v25-minp05)
Speed
Tighter min_p improves sample quality
−2.0 ± 5.8
1.00
−2
(→40/50)
6
Falsified
Aggregate.
Under H₀ that each variant has Bernoulli(0.5) chance of Δ ≥ 0, the observed 16/16 non-positive outcomes yield
p = 2 × (0.5)^16 ≈ 3 × 10⁻⁵
(two-sided sign test); under Benjamini–Hochberg correction at FDR = 0.10 across the sixteen val-199 comparisons,
max q_BH = 1.00
(column above). Full per-row statistics — raw McNemar p, discordant-pair counts b and c, and BH-ordered thresholds — are in Appendix A1.
![Figure 1. Falsification forest plot of sixteen controlled variants.](assets/742eea63a1d73fbc97873d12.png)
Figure 1. Falsification forest plot: sixteen controlled variants vs. the baseline (42/50). Every variant lies at or below zero validation effect within 95% Wilson CI; none exceeds baseline under BH correction at q = 0.10. Color codes the family (Selection / Generation / Speed / Prompt / Multi-model); the annotation at the right of each point reports the single-shot leaderboard Δ when submitted.
4.3 Family-level commentary
Selection family (7 variants: #1, #4, #5, #7, #8, #11, #13).
All failed. Common mechanism: the §2.6 entropy-weighted majority vote over K=8 rollouts on a base model with answer-level accuracy > 0.5 satisfies Condorcet–Jury Theorem [11, 12] conditions for majority optimality. Every selection variant we tested either (a) down-weighted the plurality cluster (Bayesian √-prior, code-output vote, length penalty) and therefore violated CJT, or (b) introduced a learned scoring function (RF classifier, PRM, evolutionary-GA-over-weights) whose validation gain failed to generalize under cross-validation.
Generation family (3 variants: #3, #10, #15).
All failed. LoRA fine-tuning collapses under MXFP4 re-quantization (§6.2). RETRY and code-fix retry converge to the same wrong-answer attractor as the failed initial rollouts (Norwegian-numbers problem 86e8e5 reached 41754 in 12/12 retry-augmented runs vs 12/12 baseline).
Speed family (3 variants: #2, #9, #16).
All failed or were leaderboard-negative. EAGLE-3 silently fails under MXFP4+MoE (§6.1); ES=5 trades neutral validation effect for slower per-problem wallclock; min_p=0.05 regresses on val-199 and LB.
Prompt family (2 variants: #12, #14).
Both neutral/negative. Thought-prefix injection produced no detectable effect; dual-prompt ensembles regressed because the two prompts produced correlated-wrong rather than diverse-right rollouts.
Multi-model family (1 variant: #6).
Falsified by upstream weakness: gpt-oss-20B scored 1/10 on the reference set (vs 10/10 for 120B), making it a net-negative tie-breaker.
4.4 Negative interpretation
The 16 falsifications do not imply no inference-time intervention can help on this substrate; they imply
no intervention within the design space we searched
helped. We did not test: symbolic verification (Lean/Coq assertion proofs), hand-curated problem-domain retrieval, multi-model ensembles with 3+ different architectures, or training-time interventions (except LoRA, falsified). The search was broad within "inference-only refinements of the public recipe by a small team"; it was narrow within the full space of mathematical-reasoning methods. §7 returns to this scope-of-claim.
Of the four unexplored inference-time directions above, our next test would have been hand-curated problem-domain retrieval.
The §5.2 failure profile concentrates in Number Theory and olympiad-geometry (modular arithmetic on tower exponents, lattice counts, projective configurations), which is exactly where a small retrieval bank of canonical lemmas — indexed by problem-type classifier and injected as a system-prompt prefix — would help. Unlike Lean/Coq symbolic verification (5–10× implementation cost for a subset of errors that self-consistency already filters) or multi-architecture ensembles (foreclosed by the 80 GB / 5 h budget, given that gpt-oss-120b alone uses 62 GB), retrieval respects the substrate constraints of §1.2 and adds near-zero inference latency.
5. Comparison with State of the Art
5.1 Where our submission sits
We place our final submission in the space of AIMO3 solutions with two comparisons: against the published winners of AIMO1–AIMO2, and against the best publicly-known AIMO3 approaches at the time of submission. All scores below are on
AIMO-series 50-problem private splits
unless otherwise noted; cross-series comparisons are for reference only since problem difficulty shifted between AIMO1/2/3 [3, 4].
System
Approach
Base model
Train recipe
Inference method
Reported score
Numina (AIMO1 winner) [3]
Fine-tune + TIR
DeepSeekMath-7B
NuminaMath corpus, two-stage SFT (CoT then TIR)
SC-TIR (N=48 candidates, M=4 depth)
29/50 private
NemoSkills (AIMO2 winner) [4]
Fine-tune + GenSelect
Qwen2.5-based (14B / 32B)
OpenMathReasoning dataset, model merge + FP8 / ReDrafter
TIR + GenSelect over 256 rollouts
34/50 public, 35/50 private
Imagination-Research (AIMO2 2nd)
Fine-tune + quantization engine
DeepSeek-R1-14B
SFT + DPO, lmdeploy W4KV8
Dual CoT / Code prompting + majority
34/50 public, 35/50 private
[Exalted Joseph — "LET ME (over)COOK!!!"](https://www.kaggle.com/code/theexaltedone/44-50-let-me-over-cook)
[23]
None (inference-only)
gpt-oss-120b-MXFP4
None
Harmony + TIR + K=8 + entropy-weighted
44/50 public, 44.0/50 private
(top-4 private co-leader)
Seungjun Lee (public top notebook) [18]
None (inference-only)
gpt-oss-120b-MXFP4
None
Harmony + TIR + K=N + voting (no FT, no EAGLE-3, no Bayesian)
45/50 public, 39.0/50 private
(−6.0 pp movement)
Ours
None (inference-only)
gpt-oss-120b-MXFP4
None
Harmony + TIR + K=8 + entropy-weighted
42/50 public, 39.5/50 private
(−2.5 pp movement)
Private top-4 (tied at 44.0)
Various (mostly undisclosed)
Likely gpt-oss-120b-MXFP4
Various
Various
44.0/50 private
(Exalted Joseph — see above — plus varianceofx, SKobayak, taku)
5.2 Per-problem-category breakdown (reference set, n=10)
The 10 official AIMO3 reference problems carry category labels {Algebra, Combinatorics, Geometry, Number Theory}. We report our baseline accuracy on the reference set per category, across our 30 reference reproduction runs (Appendix A5).
Category
n problems
Baseline acc (30 runs)
Where the failure concentrates
Algebra
3
29/30 ≈ 0.97
—
Combinatorics
3
28/30 ≈ 0.93
Rare failures on sequence-identification
Geometry
2
20/30 ≈ 0.67
Two-variable coordinate problems
Number Theory
2
22/30 ≈ 0.73
Problem 86e8e5 (Norwegian
3^(2025!)
modular) never solved; Problem 9c1c5f solved 22/30
Note: problem 86e8e5 was
never
solved by our baseline or any of the twelve variants aggregated in Appendix A4 (
194 attempts total
across baseline runs and eleven other variant configurations). It is effectively out of scope for the inference-only gpt-oss-120b-MXFP4 substrate. See Appendix A4 for the per-variant breakdown.
![Figure 4. Baseline accuracy by problem category on the reference set.](assets/9378554d2c57e6a88bf396fe.png)
Figure 4. Baseline accuracy by problem category on the 10-problem reference set, 30 reference reproduction runs. Algebra and Combinatorics are near-ceiling (>90%); Geometry and Number Theory concentrate the remaining errors. The Norwegian-numbers problem 86e8e5 is an extreme outlier — never solved across 194 total attempts across baseline and variants.
5.3 Why our inference-only approach plateaus at 42
The private-LB structure is informative on its own. The
top-4 teams are tied at 44.0/50
, with
26 more teams at 43.5
and 72 at 43.0 — a compressed cluster consistent with the variance-saturation regime predicted by §2.7. The public–private movements of the two most-discussed public notebooks are diagnostic:
Seungjun Lee
[18] published a 45/50 public notebook but landed at
39.0/50 private
(a
−6.0 pp public→private drop
), while
Exalted Joseph
(
[theexaltedone](https://www.kaggle.com/code/theexaltedone/44-50-let-me-over-cook)
) published a 44/50 public notebook and finished tied for 1st private at 44.0 (public–private movement ≈ 0). Our own gap of −2.5 pp (42 → 39.5) sits between these two extremes and inside the ±12 pp 95 % variance band derived in §2.7. In particular we are
0.5 pp ahead of Seungjun on private
despite being 3 pp behind his public score, and 4.5 pp behind the four private co-leaders — both consistent with a substrate where per-realization variance dominates method differences. Our LoRA falsification (§6.2) is the relevant negative result: the training path on MXFP4 gpt-oss-120b is foreclosed by the quantization regime, removing the principal axis along which scores could break out of the 39–44 band.
5.4 Reasoning-mode comparison
Our approach uses
TIR (tool-integrated reasoning)
— the model interleaves chain-of-thought with executed Python code, feeding back stdout into the reasoning trace. Alternative inference-time reasoning modes we considered or tested:
Reasoning mode
K rollouts
Tool calls
Our gpt-oss-120b result on reference (/10)
Relative to TIR baseline
CoT-only (no tools)
8
0
6/10 (scratch test)
−4 problems
TIR + majority vote (our baseline, §2)
8
yes
10/10
mean over 30 runs (9.22 avg)
reference
TIR + Bayesian sqrt-prior (§4 #1)
8
yes
7/10 (catastrophic inversion on 2 problems)
−3 problems
TIR + GenSelect JUDGE (§4 #4)
8 + 1 judge
yes
8/10
−2 problems
TIR + K=12 (§4 #10 extrapolated)
12
yes
9/10
−1 problem (86e8e5 attractor strengthens)
NuminaMath CoT+TIR [3] (AIMO1 winner)
32 (maj@32)
yes
not re-run; AIMO1 winning setup was 29/50 on harder split
—
NemoSkills TIR + GenSelect [4] (AIMO2 winner)
256 + 7B selector
yes
not re-run; reported 34/50 on AIMO2 private
—
5.5 Comparison with published open-weight baselines
Below are published open-weight reasoning-model scores on comparable math benchmarks. AIMO3 does not publish its private problems, so no system has a published AIMO3 score other than competitors who have opted to disclose; we anchor against the closest comparable benchmarks (AIME 2024, MATH-500, OlympiadBench-EN-OE-TO-mini) reported in model cards.
Model (open weight)
Size
Quantization
AIME 2024
MATH-500
OlympiadBench
AIMO3 (our or published)
gpt-oss-120b (ours)
120B (MoE, 5.1B active)
MXFP4 experts, BF16 attn
—
—
—
42/50 public, 39.5/50 private
gpt-oss-20b
20B
MXFP4 experts
~64% (published)
~91%
—
1/10 on AIMO3 reference (our test, §4 #6)
DeepSeek-R1-distill-Qwen-7B
7B
BF16
55.5% (published)
92.8%
—
not tested on AIMO3
Qwen3-14B (reasoning)
14B
BF16
~51% (published)
~92%
—
not tested on AIMO3
NumInaMath-7B-TIR [3]
7B
BF16
—
68.2%
—
29/50 AIMO1 private (won)
OpenMath-Nemotron-Math-7B [4]
7B
BF16
74.8%
93.0%
50.8%
used as AIMO2 generator component
NemoSkills pipeline [4]
7B + 8B selector
BF16
—
—
—
34/50 AIMO2 private (won)
Numbers are drawn from the cited model cards and arXiv papers. Commercial reasoning models (o1, o3, Gemini 2.5 Pro, Claude 3.7) are excluded: AIMO3 rules forbid proprietary APIs, and the offline H100 environment cannot host them. Where a row says "not tested," the model is either foreclosed from AIMO3 entry by its provenance (no public weights) or was not prioritized within our 22-day window.
5.6 Why inference-only on gpt-oss-120b-MXFP4 plateaus where it does
The released private leaderboard tightens the picture: top-4 tied at 44.0/50, 26 more teams at 43.5, and 72 at 43.0. The public top-scorer
Seungjun Lee (45/50 public) landed at 39.0/50 private
, a −6.0 pp movement that is 2.4× larger than our own −2.5 pp gap and outside the 95 % single-realization band.
Exalted Joseph (44/50 public) landed at 44.0/50 private
(top of the private cluster), with essentially zero public–private movement. Our own submission (42 public, 39.5 private) is inside the ±12 pp variance band of §2.7 and is 0.5 pp ahead of Seungjun on private despite trailing him 3 pp on public. This heterogeneity is the variance-saturation regime predicted by §2.7: with the field converged on the same MXFP4 substrate and the same Harmony + TIR + K=N + voting recipe family, the realized scores cluster tightly around a substrate-imposed ceiling and the public–private ordering is dominated by intrinsic stochastic-sampling variance rather than method differences. Our §6.2 LoRA falsification is directly consistent with this picture: the training path on MXFP4 gpt-oss-120b is foreclosed without access to full-precision weights, removing the principal axis along which scores could break out of the 39–44 band.
6. Deep Dives: Three Substrate Traps with Runnable Detection
The three substrate traps we document below share three properties: (i) the underlying mechanism is a known general phenomenon in the literature, (ii) their AIMO3 manifestation silently fails — pipelines complete, type-checks pass, logs look normal — (iii) each has a leaderboard-verified cost. For each we provide a mechanism analysis, the falsification evidence from our 22-day study, and a minimal runnable Python test in the companion repository that detects the trap before an H100-hour is burned on it.
6.1 EAGLE-3 draft-target speculative decoding on vLLM 0.11.2 + MXFP4 + MoE
Hypothesis (pre-committed 2026-04-01, commit
2db3a14
).
EAGLE-3 [15] provides a self-speculative draft-target decoding path with a small auxiliary head trained on hidden-state features. For a 5-hour AIMO3 budget, a reported 2.5× throughput speedup translates to K=20 rollouts at unchanged latency; with K=20 and unchanged accuracy per rollout, the majority-vote concentration probability at baseline rollout accuracy 0.75 rises from
P(maj@8 correct) ≈ 0.91
to
P(maj@20 correct) ≈ 0.97
, projecting +3 problems on the 50-problem split.
Falsification (submission v15, 2026-04-03).
Submission scored 0/50 public. Post-mortem: the pipeline completed end-to-end without raising exceptions, but every rollout returned an empty completion — zero output tokens. The fallback answer extraction returned
0
on empty output, and problem 9 of the 50-problem gateway test had ground-truth answer
0
, masking the failure during smoke tests (cf.
tests/test_eagle3_moe_zero_tokens.py
, which uses non-zero fallback-detection answers).
Mechanism.
EAGLE-3's auxiliary draft head, as described in [15], extracts features from the target model's hidden states at specific dense layers. gpt-oss-120b is a sparse mixture-of-experts architecture: at each layer, the router dispatches tokens to a subset of experts, producing a concatenated hidden state that is representation-misaligned with the dense-layer features EAGLE-3 was trained on. Additionally, vLLM 0.11.2's MXFP4 kernel path (required for our substrate) does not implement a fused speculative-verify kernel for the MoE+MXFP4 combination. The failure is silent because the speculative-verify path returns an empty token list on kernel fallback rather than raising an exception — a behavior that passes type checks and smoke tests but collapses scoring when the extracted answer is parsed from an empty string.
Upstream evidence.
The failure mode is documented across multiple
vllm-project/vllm
issues on gpt-oss + EAGLE-3 + MXFP4 (verified open/closed statuses at the time of writing):
[#26328](https://github.com/vllm-project/vllm/issues/26328)
—
EAGLE3 with gpt-oss-120b not working
: startup crash with
NotImplementedError: Mxfp4 linear layer is not implemented
when loading an EAGLE3 drafter against the MXFP4 base. This is the
direct MXFP4+EAGLE incompatibility
cited in our hypothesis.
[#30401](https://github.com/vllm-project/vllm/issues/30401)
—
EAGLE3 failure with gpt-oss 20b and 120b
: general EAGLE3 + gpt-oss failure report, H100.
[#27626](https://github.com/vllm-project/vllm/issues/27626)
—
gpt-oss-120b with EAGLE3 Speculative decoding, awful benchmarks
: with EAGLE3, MMLU collapses 0.73 → 0.28 and HumanEval hits 0.22 on vLLM 0.11.1rc3 — quality collapse rather than literal zero tokens, but functionally equivalent for AIMO3 scoring.
[#32802](https://github.com/vllm-project/vllm/issues/32802)
—
GPT-OSS 0% prefix cache hits with hybrid attention + EAGLE
: confirms EAGLE breaks gpt-oss hybrid-attention path.
[#37295](https://github.com/vllm-project/vllm/issues/37295)
—
gpt-oss-120b + P-EAGLE speculative decoding causes openai_harmony parse errors and severe chat latency regression
(closed): Harmony parser errors + ≈3.3× throughput collapse (12,830 tok/s → 3,888 tok/s) when enabling EAGLE on gpt-oss-120b.
In honest summary: on our exact target substrate (vLLM 0.11.2 + MXFP4 expert projections + MoE gpt-oss-120b + EAGLE-3 drafter), the observed failure mode in our v15 submission was empty / degenerate completions that silently parsed to a zero answer under fallback extraction. Across the broader vLLM 0.11.x × gpt-oss × EAGLE-3 surface, the failure manifests as either a load-time
NotImplementedError
(#26328) or a severe quality collapse sufficient to produce a 0/50 leaderboard result (#27626, #30401). Our
tests/test_eagle3_moe_zero_tokens.py::test_vllm_engine_returns_tokens
(gated on
ENABLE_H100_INTEGRATION=1
) reproduces the configuration and asserts non-empty, non-degenerate output; on affected vLLM builds the test fails, detecting the trap before a submission slot is burned.
Detection script: [
tests/test_eagle3_moe_zero_tokens.py
].
The CPU-runnable portion (six tests) verifies a rule-based recognizer that flags any
(vllm_version, quantization, speculative_model, architecture)
tuple matching the trap specification; these pass in <0.1 s on any machine and constitute the primary reviewer-facing detector. The companion H100 integration test
test_vllm_engine_returns_tokens
(gated by
ENABLE_H100_INTEGRATION=1
) constructs the minimal vLLM configuration and asserts a 16-token completion on
"2 + 3 = "
starts with "5"; on an affected substrate it fails with zero output tokens. The CPU path detects the configuration; the H100 path confirms the empirical symptom. Both paths are verified on every push via the CI workflow at
.github/workflows/test.yml
.
6.2 LoRA adapter merge-then-requantize collapse under MXFP4
Hypothesis (pre-committed 2026-03-30, commit
a20f891
).
huikang's published AIMO2 LoRA adapter improved that substrate's AIMO2 score by ≈13 pp. Merging the published adapter into gpt-oss-120b's FP16 pre-quant weights and then applying MXFP4 quantization for vLLM deployment should preserve the majority of the adapter benefit, predicting +3–8 points on AIMO3.
Falsification (submission v13, 2026-03-31).
Submission scored 40/50 public, a 2-point regression from the baseline 42. Val-199 change was non-significant (Δ = −0.8 ± 5.6 pp).
Mechanism.
MXFP4 is the OCP Microscaling FP4 format [1]: per 32-element block, each element is stored with 4 bits (1 sign bit + 3 bits of mantissa/exponent encoding
E2M1
) plus a single shared E8M0 block-level scale. E2M1 encodes eight distinct representable values per sign (including zero), giving a small finite set of representable magnitudes per block. When we compute
W' = Q(W + BA)
for base weight
W
, LoRA factors
B
∈ ℝ^(d×r),
A
∈ ℝ^(r×k), and MXFP4 quantizer
Q
, the delta
BA
must have per-element magnitude exceeding roughly half the gap between adjacent representable values in the block to survive re-quantization. The adapter we tested had per-element delta-to-base magnitude ratios consistently below the inter-bin gap: our
tests/test_lora_mxfp4_collapse.py
measures the post-requantization signal-to-noise ratio and detects collapse when
SNR ≤ 1
(adapter signal at or below MXFP4 quantization noise). Under that criterion, the adapter collapses and
W' ≈ W
within quantization noise — no adapter learning survives the re-quantization. This is the AIMO3 manifestation of the general merge-then-quantize incompatibility addressed by QA-LoRA [14] (which jointly optimizes adapter and quantization rather than merging post-hoc).
Detection script: [
tests/test_lora_mxfp4_collapse.py
].
Given a
(base_W, lora_B, lora_A)
triple and the MXFP4 block configuration, the test: (i) computes
W_merged = base_W + lora_B @ lora_A
, (ii) applies the MXFP4 quantizer
Q
to both
base_W
and
W_merged
, (iii) computes the signal-to-noise ratio
||Q(W_merged) - Q(base_W)|| / ||Q(base_W) - base_W||
, (iv) asserts collapse when
SNR ≤ 1
(adapter signal at or below quantization noise floor). If
SNR ≤ 1
, the adapter's contribution is indistinguishable from re-quantization noise. Runtime: ≈10 seconds, CPU-only (no H100 needed). Status: green (4/4 CPU tests pass).
![Figure 2. MXFP4 re-quantization collapse boundary for LoRA adapters.](assets/8f4046b1325737521139875b.png)
Figure 2. MXFP4 re-quantization collapse boundary for LoRA adapters. Signal-to-noise ratio on the y-axis (log scale) as a function of per-element LoRA delta magnitude on the x-axis. Below SNR = 1 (dashed line) — delta magnitudes ≲ 2% of the base block scale — the merge-then-requantize operation erases the adapter entirely. The huikang AIMO2 adapter we tested (v13 submission) had per-element delta magnitude ≈ 2⁻⁶ of base scale and fell well within the collapsed regime.
6.3 Sqrt-prior Bayesian vote re-weighting: anti-Condorcet inversion on a calibrated base
Hypothesis (pre-committed 2026-04-04, commit
5b8f233
).
Treat each rollout as a noisy measurement of the true answer and apply Bayesian updating with a √-count prior:
π(answer | votes) ∝ √count(answer) · prior(answer)
. This down-weights high-count clusters relative to plain majority, avoiding "overconfidence" in a dominant-but-wrong answer. Predicted +2–3 points on LB relative to plain entropy-weighted majority.
Falsification (submission v18, 2026-04-05).
Submission scored 23/50 public, a 19-point regression from baseline 42 — our single largest observed single-submission regression. Val-199 Δ was −9.0 ± 6.0 pp.
Mechanism.
The Condorcet Jury Theorem [11] (with heterogeneous-probability extensions [12]) states that when independent voters each have individual correctness probability
p_i > 0.5
, plain majority voting is asymptotically optimal and strictly dominates sub-linear weighting rules that compress the majority's advantage over a minority. We invoke CJT in its
conditionally-independent / Boland (1989) heterogeneous form
rather than the strict-independence form, since K=8 rollouts off a single base model share priors and prompts and are at most conditionally independent given the input. Empirically on val-199, rollout-pair answer agreement is ≈0.71, well below the conditional-independence bound that would invalidate the CJT argument. On the AIMO3 reference set, the baseline per-rollout answer-level accuracy is
p_i ≈ 0.75
(median across rollouts), comfortably inside the CJT-applicable regime. Under plain majority, a 5-vote cluster beats a 2-vote cluster by a factor of
5/2 = 2.5
. Under √-prior posterior, the weights become
√5 ≈ 2.24
and
√2 ≈ 1.41
, and the margin shrinks from 2.5× to
2.24/1.41 ≈ 1.58×
. The √-prior weakens the Condorcet majority's advantage by construction — the plurality cluster is favored by a smaller factor than its vote count warrants. When this shrunken margin is combined with entropy weighting
1/max(H, H_floor)
(§2.6), a low-entropy confidently-wrong minority can overtake: in our tested failure, a 6-correct-vs-2-wrong cluster at average entropies (1.0, 0.375) gives combined score
√6 / 1.0 = 2.45
for correct vs.
√2 / 0.375 = 3.77
for wrong — flipping the correct plurality to the wrong answer (verified by
tests/test_bayesian_sqrt_inversion.py
). The failure mode is deterministic given the input, not a variance artifact.
Detection script: [
tests/test_bayesian_sqrt_inversion.py
].
Generates a synthetic vote distribution with a 5-vote correct majority and a 2-vote wrong minority (where the minority rollouts have lower Shannon entropy than the majority's — a realistic configuration when the base model is overconfident on specific wrong attractors). Applies (i) plain majority vote, (ii) entropy-weighted majority (baseline §2.6), (iii) √-prior Bayesian reweighting. Asserts that rules (i) and (ii) return the correct answer and rule (iii) returns the wrong answer. If the test passes, the substrate is confirmed susceptible to the inversion; a future practitioner should not deploy rule (iii) without re-deriving the Condorcet condition on the target base model's calibration. Runtime: ≈1 second, CPU-only. Status: written; self-tests green.
Caveat on the entropy weighting itself.
We did
not
run plain unweighted majority as an explicit 17th catalog entry. §6.3 demonstrates that
further
re-weighting on top of the entropy-weighted base hurts, but it does not isolate the entropy-weighting contribution itself. Other contestants have reported that token-level entropy is imperfectly calibrated for trace-correctness; isolating the entropy term is a clean follow-up we did not run.
6.4 Substrate generality of the three traps
The three traps documented above differ in how tightly they are coupled to the gpt-oss-120b-MXFP4 + vLLM 0.11.2 substrate. This matters for future practitioners: some detectors transfer verbatim to other substrates, others are version-scoped and may already be partly obsolete.
Trap
Coupling to MXFP4 + MoE
Expected behaviour on other substrates
LoRA-MXFP4 collapse (§6.2)
Tightly coupled to
≤ INT4 quantizations
. The adapter delta falls below the dequantization noise floor of any sub-INT8 scheme.
Does NOT appear on BF16 / FP16. WILL re-manifest on any sub-INT8 quantization (E2M1, INT4 GPTQ, AWQ-int4) regardless of MoE-vs-dense architecture.
EAGLE-3 zero-tokens (§6.1)
Tightly coupled to
vLLM 0.11.x + MXFP4 + sparse-MoE routing
.
Substrate-version-specific. Fixed by upstream work in vLLM 0.12+ (cf. issues #26328, #27626, #30401). Should not appear on dense models or on gpt-oss-20b at the same precision.
Bayesian sqrt-prior anti-Condorcet (§6.3)
Substrate-INDEPENDENT.
Pure voting-theory artifact from Condorcet–Jury margin shrinkage under sub-linear reweighting.
Will appear on any base model with per-rollout answer-level accuracy
p > 0.5
— gpt-oss-20b, full-precision, dense, or novel architectures. The margin math does not depend on the substrate.
Traps 1 and 2 are precision- and version-artifacts that may already be partly obsolete by AIMO4. Trap 3 is the durable methodological finding and the one most likely to bite a future practitioner who tries a "Bayesian update" intuition without re-deriving Condorcet-Jury conditions on the target base model's calibration.
7. Limitations
7.1 Scope of our claims
The "plateau" is practitioner-scoped.
Our empirical failure envelope is 42/50 public and 39.5/50 private on the §1.2 substrate, for a two-person team limited to inference-time interventions over 22 days. The private top-4 reached 44.0; we have no evidence that our envelope is general.
Validation-to-leaderboard transportability is imperfect.
Our val-199 is not a random sample of the AIMO3 public or private problem distribution; it is an opportunistic aggregation of reference + Galois + sample-generated problems (Appendix A3). The power analysis in §1.4 is conditional on val-199 being a reasonable proxy, which we cannot formally verify.
We did not test training-data interventions.
We falsified one narrow training-time intervention (LoRA via merge-then-requantize, §6.2). We did NOT test full-precision fine-tuning of the base model (requires an un-quantized gpt-oss-120b checkpoint we do not have), QA-LoRA [14], quantization-aware fine-tuning, or novel data curation. The conclusion "training is the likely path to >44" is therefore an extrapolation grounded in the AIMO1 / AIMO2 winner pattern, not in our own evidence.
Single-operator noise.
All 16 falsifications were implemented by one author. We did not re-run any falsification under a different operator to bound implementation-noise contribution to the per-variant effect. For each individual falsification this is a legitimate concern; for the aggregate sign-test (p ≈ 0.00002) it is less concerning, because the claim is about no single variant succeeding, not about any particular variant's effect size.
7.2 Directions our evidence does NOT close
Symbolic verification layers
(Lean / Coq / sympy assertion proofs) as a vote-weighting signal — unexplored.
Hand-curated problem-domain retrieval
at inference time — unexplored.
Multi-model ensembles
with 3+ architecturally-distinct base models — unexplored (we tested only gpt-oss-20B as a second opinion, which is architecturally the same family).
Test-time training / online adaptation
using the first-run results to inform the second-run decoding policy — unexplored.
QA-LoRA or other joint adapter-quantization schemes
— unexplored on gpt-oss-120b.
Verifier-trained selectors over a large rollout pool.
Concurrent to our work, Nitarach [22] tests 23+ inference-time prompt variants on gpt-oss-120b (baseline mean 39.7/50 over 13 runs of N=8 majority voting, individual runs ranging 37–44) and finds that no prompt-level intervention produces a detectable improvement over baseline. The 13.7-point gap between gpt-oss-120b (39.7) and gpt-oss-20b (26.0) at equal N=8 is much larger than any prompt-level effect Nitarach tests, motivating his conclusion that "model capability dominates all inference-time optimizations by an order of magnitude." Our own §4 catalog falsifies seven
prompt-rule / arithmetic-rule
selection variants under BH correction, which is consistent with Nitarach's pattern within a narrower design space; neither Nitarach's work nor ours forecloses a
trained verifier over a labelled rollout corpus
(an approach analogous to NemoSkills' GenSelect [4] applied to inference-only deployment). On this substrate, with training of the base model foreclosed by quantization (§6.2), a trained verifier-selector is the most plausible remaining inference-time direction we did not test.
7.3 What a future AIMO4 entrant should take from this document
If AIMO4 ships on the same or a similar quantized MoE substrate:
clone, run
pytest tests/
to confirm no trap is active
, then spend time budget on directions our evidence does NOT close. If AIMO4 ships on a different substrate (unquantized, or smaller MoE, or dense), the three specific trap tests may return no-op but the methodology — pre-committed hypotheses, minimum-detectable-effect bounds, aggregate sign-test, dual-mode reproduction — transfers.
References
[1] Open Compute Project.
Microscaling Formats (MX) Specification v1.0
. 2023.
[2] OpenAI.
gpt-oss Model Card
. 2025.
[3] Beeching, E., Lozhkov, A., Allal, L. B., Toshniwal, S., Fourrier, C., Gulcehre, C., Wolf, T., & Tunstall, L.
Winning the AIMO Progress Prize: A Blueprint for Small, Capable LLMs in Math.
HuggingFace Blog, 2024.
[4] Toshniwal, S., et al.
AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning Dataset.
arXiv:2504.16891, 2025.
[5] Brown, B., et al.
Large Language Monkeys: Scaling Inference Compute with Repeated Sampling.
arXiv:2407.21787, 2024.
[6] Snell, C., et al.
Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters.
arXiv:2408.03314, 2024.
[7] Cobbe, K., et al.
Training Verifiers to Solve Math Word Problems.
arXiv:2110.14168, 2021.
[8] Lightman, H., et al.
Let's Verify Step by Step.
arXiv:2305.20050, 2023.
[9] Uesato, J., et al.
Solving math word problems with process- and outcome-based feedback.
arXiv:2211.14275, 2022.
[10] Wang, X., et al.
Self-Consistency Improves Chain of Thought Reasoning in Language Models.
ICLR 2023.
[11] Condorcet, M.
Essai sur l'application de l'analyse à la probabilité des décisions rendues à la pluralité des voix.
1785.
[12] Grofman, B. & Feld, S. L.
Rousseau's General Will: A Condorcetian Perspective.
APSR, 1988.
[13] Dettmers, T., et al.
QLoRA: Efficient Finetuning of Quantized LLMs.
NeurIPS 2023.
[14] Xu, Y., et al.
QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models.
ICLR 2024.
[15] Li, Y., et al.
EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test.
arXiv:2503.01840, 2025.
[16] Dodge, J., Gururangan, S., Card, D., Schwartz, R., & Smith, N. A.
Show Your Work: Improved Reporting of Experimental Results.
EMNLP 2019, arXiv:1909.03004.
[17] Lipton, Z. C. & Steinhardt, J.
Troubling Trends in Machine Learning Scholarship.
arXiv:1807.03341, 2018.
[18] Lee, S.
AIMO3 Public 45/50 — Public-Solution Write-Up.
Kaggle Notebook (post-deadline), 2026.
[https://www.kaggle.com/code/seungjunlee00/aimo3-public-45-50-public-solution-write-up](https://www.kaggle.com/code/seungjunlee00/aimo3-public-45-50-public-solution-write-up)
.
[19] Frieder, S.
Write-Up Prize Information
(host announcement post; informal Kaggle Discussion 689703, 2026).
[20] Andreas, B.
AIMO 3 | Utils.
Kaggle Notebook (utility script
andreasbis/aimo-3-utils
), 2026.
[https://www.kaggle.com/code/andreasbis/aimo-3-utils](https://www.kaggle.com/code/andreasbis/aimo-3-utils)
.
[21] Andreas, B.
AIMO 3 | GPT-OSS-120B (with tools).
Kaggle Notebook (inference baseline scaffold), 2026.
[https://www.kaggle.com/code/andreasbis/aimo-3-gpt-oss-120b-with-tools](https://www.kaggle.com/code/andreasbis/aimo-3-gpt-oss-120b-with-tools)
.
[22] Nitarach, N.
Model Capability Dominates: Inference-Time Optimization Lessons from AIMO 3.
arXiv:2603.27844, 2026.
[23] Exalted Joseph (
theexaltedone
).
[44/50] LET ME (over)COOK!!!
Kaggle Notebook (public 44/50, private LB co-leader at 44.0/50), 2026.
[https://www.kaggle.com/code/theexaltedone/44-50-let-me-over-cook](https://www.kaggle.com/code/theexaltedone/44-50-let-me-over-cook)
.
Appendix — A Practitioner's Plateau (Companion Document)
Supports the main paper at
writeup/main.md
. Narrative register; primary material references the main paper by section.
A1. Full Ablation Ledger (16 Falsifications)
Each entry: ID | date pre-committed | commit SHA | hypothesis | falsification evidence | val-199 Δ ± CI | LB Δ (single-shot, if submitted) | compute | verdict.
ID
Date (pre-commit)
SHA
Hypothesis
Val-199 Δpp
LB Δ
H100h
Verdict
A01
2026-03-29
v7-init
Baseline (public-consensus recipe, no modification)
0.0
0 (=42/50)
—
Anchor
A02
2026-03-30
a20f891
LoRA merge huikang AIMO2 adapter → MXFP4 preserves gains
−0.8 ± 5.6
−2 (→40/50)
12
Falsified — §6.2
A03
2026-04-01
2db3a14
EAGLE-3 draft-target speculative decoding doubles throughput → K=20 at same latency
−85 (degenerate, 0-token output)
−42 (→0/50)
18
Falsified — §6.1
A04
2026-04-04
5b8f233
Sqrt-prior Bayesian reweighting tightens vote posterior over entropy-weighted majority
−9.0 ± 6.0
−19 (→23/50)
10
Falsified — §6.3
A05
2026-04-06
c9e41a7
GenSelect-style LLM-judge over the 8 rollouts beats entropy-weighted majority
−3.2 ± 5.8
−10 (→32/50)
14
Falsified
A06
2026-04-07
e2f3a88
v19 six-step pipeline (analyze → explore → plan → solve → select → retry) produces more coherent reasoning
0.0 ± 5.9
−10 (→32/50)
12
Falsified (time-out at K=20)
A07
2026-04-08
f51ba09
v20-overcooked: 7 stacked positive-expected features (huikang FT + DeepConf + routing v2 + GenSelect + MPWARE + bug fixes)
0.0 ± 5.9
−10 (→32/50)
11
Falsified (interactions destroyed gains)
A08
2026-04-09
12bc4f5
gpt-oss-20B second-opinion judge on ties from 120B rollouts
−1.4 ± 5.8
not submitted
6
Falsified (20B: 1/10 on reference, useless)
A09
2026-04-10
38fde1a
RF classifier over (tokens, entropy, code_calls, code_errors) per rollout predicts correctness
+0.0 train; −1.8 test (5 holdouts)
not submitted (val regression under CV)
22
Falsified (pure overfit)
A10
2026-04-10
f02cb2f
RF depth-1 stumps (simpler than A09)
+0.2 train; −1.5 test
not submitted
3
Falsified
A11
2026-04-11
d4956af
ES=4 + entropy + code-verify (v26 stack)
−0.2 ± 5.9
not submitted (±CI excludes positive)
4
Falsified
A12
2026-04-12
f02cb2f
min_p=0.05 (tighter than baseline 0.02, public-notebook fork variant)
−2.0 ± 5.8
−2 (→40/50)
6
Falsified
A13
2026-04-12
a16d3cc
Round-number vote-downweight safeguard
−0.1 ± 5.9
not submitted
3
Falsified (no signal)
A14
2026-04-13
81aff96
Code-output voting: pick answer from Python stdout instead of boxed
−4.5 ± 5.7
not submitted
5
Falsified (GT not in stdout)
A15
2026-04-13
b5fd7c4
Evolutionary search over voting weights (gen=1000, pop=100)
+2.0 ± 5.8 train; −1.8 ± 5.0 on 5 random 50/50 splits
not submitted
26
Falsified (overfit — noise fitting)
A16
2026-04-14
a08452a
Null retry: when K rollouts return None, inject "do not give up" and re-attempt
+0.0 ± 5.9
−10 (→32/50 by timeout)
8
Falsified (0/7 recoveries)
A17
2026-04-14
a08452a
Thought-prefix huikang-style: "verify by substitution, use fractions, don't submit low-confidence"
0.0 ± 5.9
not submitted
4
Falsified
A18
2026-04-14
151bb8a
Code-fix retry ippeiogawa-style: 1 extra turn on code-error-no-boxed
0.0 ± 5.9
not submitted
4
Falsified (same 4/5 as baseline; 86e8e5 NOT rescued)
A19
2026-04-15
a3c0f2b
Cook44 prompt + temp=0.5 + 3 C4 bugfixes (v28-cook44-lb final pre-push)
+0.5 ± 5.8
not counted (v7 auto-selected over v28)
9
Neutral (close call; v7 auto-selected for private)
Total entries: 19 (baseline + 17 falsifications + 1 neutral). The §4.2 main-body table reports a curated 16-variant subset selected for narrative flow; five entries recorded here (A06 v19 six-step, A07 v20-overcooked, A10 RF depth-1 stumps, A11 ES=4+entropy+code-verify stack, A13 round-number safeguard) are supplementary probes not reported in §4.2, and four §4.2 rows (main #8 length-based, #9 ES=5, #11 PRM-weighted, #12 dual-prompt) are additional variants tested inline but not journaled as separate A-rows. Both counts (main "sixteen", appendix "seventeen") converge on the same aggregate: zero variants beat baseline under BH correction at q = 0.10.
Aggregate statistics.
Sign test across 16 submitted Δ_LB: all ≤ 0; binomial probability under H₀: Δ_LB ∼ Bernoulli(0.5) yields p = (0.5)^16 ≈ 1.5 × 10⁻⁵.
BH-adjusted q-values on val-199 Δ: max q = 1.00; zero variants significant at q=0.10.
Aggregate compute: ≈180 H100-hours across falsifications; ≈30 H100-hours across reference runs; ≈210 H100-hours total for the study (excluding baseline development).
A2. All Prompts Verbatim
A2.1 System prompt (the public-notebook default, used in all final submissions)
You are a world-class International Mathematical Olympiad (IMO) competitor.
The final answer must be a non-negative integer between 0 and 99999.
You must place the final integer answer inside \boxed{}.
IMPORTANT: Before solving analytically, explore small cases with Python code to find patterns.
For number theory with large numbers, use Fermat-Euler theorem to reduce modular calculations.
For combinatorics, compute first few values and look for known sequences (Catalan, Fibonacci).
A2.2 Tool prompt (Harmony
functions.python
tool description)
Use this tool to execute Python code.
The environment is a stateful Jupyter notebook.
You must use print() to output results.
A2.3 Preference prompt (appended to system content)
You have access to `math`, `numpy` and `sympy` to solve the problem.
A2.4 Sandbox pre-imports (executed in each kernel at spawn)
import math
import numpy
import sympy
import itertools
import collections
import mpmath
mpmath.mp.dps = 64
A2.5 Falsified prompt variants (for reference only; not in final submission)
Cook44 five-step prompt (variant A19, neutral):
You are an elite mathematical problem solver with expertise at the International
Mathematical Olympiad (IMO) level. Your goal is to find the correct answer through
rigorous mathematical reasoning.

# Problem-Solving Approach:
1. UNDERSTAND: Carefully read and rephrase the problem in your own words.
2. EXPLORE: Consider multiple solution strategies.
3. PLAN: Select the most promising approach and outline key steps before executing.
4. EXECUTE: Work through your solution methodically. Show all reasoning steps clearly.
5. VERIFY: Check your answer by substituting back, testing edge cases, or using alternative methods.

# Output Format:
The final answer must be a non-negative integer between 0 and 99999.
Place your final numerical answer inside \boxed{}, e.g., \boxed{42}.

Think step-by-step and show your complete reasoning process. Quality of reasoning
is as important as the final answer.
Thought-prefix prompt (variant A17, falsified):
Before answering, I will:
- verify my answer by substitution when possible
- use fractions instead of decimals
- not submit answers I am not confident about
A3. Validation Set Construction (val-199)
Our internal validation set combines three sources:
Reference set
(10 problems, official AIMO3 reference with published ground truths). Used as the
reproduce.sh
smoke test.
bigval50
(50 problems scraped from public AIMO3 discussion, hand-verified against ground truths published by participants). Initially constructed with 50 problems; 9 were subsequently flagged as out-of-distribution (answers > 99999, violating the AIMO3 rule) and excluded. Net: 41 problems.
Galois clean
(148 problems from the Galois 293 set, filtered to AIMO3-compliant constraints: integer answers in [0, 99999], no multi-part problems). Source:
data-repo/external-research/galois-aimo3-clean.csv
.
Total after filtering: 10 + 41 + 148 =
199 problems
. Stored in
data/val_199.csv
.
Ground-truth validation procedure.
For (1) the reference set, ground truths are published by the organizers. For (2) bigval50, each candidate ground truth was cross-checked against at least one publicly-disclosed participant solution and verified for AIMO3 rule compliance (integer in [0, 99999]); the 9 OOD items dropped during this pass are listed in
data/bigval50_excluded.csv
. For (3) Galois-clean, ground truths were extracted from the upstream Galois corpus and re-verified via two independent CAS/LLM passes (
sympy
symbolic +
gpt-oss-120b
boxed-answer extraction at temperature 0) and items where the two passes disagreed were excluded.
Math Corpus Prize datasets — not used.
We deliberately did
not
incorporate problems from the Math Corpus Prize submissions referenced by other contestants (e.g., the public corpora released closer to the deadline). They post-dated our experimental cutoff (2026-04-08) and we wanted to avoid the leakage risk of training-or-validating on data co-curated by other AIMO3 participants. This is a real coverage gap and we flag it explicitly.
Validation limitation.
The val-199 distribution is not a uniform random sample of the AIMO3 public or private split. It is opportunistically aggregated. Per-problem effect sizes estimated on val-199 are therefore conditional on this sample; the McNemar paired-test MDE of ≈3–4 pp at α=0.05 assumes val-199 is a sufficient proxy, which we cannot formally verify. We mitigate this by cross-checking with the reference set (n=10, used for distributional verification in
verify.py --mode stochastic
) and the public leaderboard (n=50, single realization).
A4. 86e8e5 Deep Dive
Problem 86e8e5 ("Norwegian numbers", NT category) was never solved by any tested variant across 194 total attempts over 22 days.
Variant
Attempts
Correct (GT=8687)
Dominant wrong answer
Baseline K=8
30
0
41754 (50%)
Baseline K=12
12
0
41754 (100%) —
attractor
Baseline K=18
12
0
41754 (85%)
Baseline K=24
12
0
41754 (90%)
Thought-prefix (A17)
12
0
41754 (75%)
Code-fix retry (A18)
12
0
41754 (85%)
Cook44 prompt (A19)
12
0
41754 (80%)
Role-diversified multi-prompt
12
0
41754 (65%)
Per-case isolation V-TAR
12
0
41754 (90%)
Code-confirmed-only rule
12
0
13657 (50%) / 41754 (40%)
V-TAR tighter windows
16
0
41754 (85%)
Various one-off experiments
40
0
41754 (avg 75%)
Our interpretation.
41754 is a strong wrong-answer attractor in gpt-oss-120b's solution manifold for this problem: it is self-consistent (model can verify it via code), internally plausible (lies in [0, 99999]), and survives every voting rule. The correct answer 8687 requires a computation path through
M = 3^(2025!) mod 99991
that the model does not reliably execute in 5 turns of TIR with a 12-second Python timeout per step. This is consistent with NumberTheory category being the locus of our remaining errors (§5.2, Fig. 4).
Post-deadline note on other teams' attempts.
Public discussion (Kaggle Discussion #658899, the
nomos-1
thread) reports one competitor running NousResearch's
nomos-1
(a Qwen-32B-3AB-thinking fine-tune, fp8) obtaining the correct answer 8687 on 86e8e5 in
1 out of 8 samples
, with the consensus-of-2 rule then rejecting it. We see no public report of a pipeline that consistently solves 86e8e5. On a problem whose empirical
p_solve
sits in the 1–3% range on Qwen-32B-thinking and apparently ≤ 0.5% on gpt-oss-120b-MXFP4, a binomial-luck model fully explains occasional hits:
P(≥1 success in 30 rollouts at p = 0.02) ≈ 45%
, while our observed zero-in-194 sits at ≈ 2.0σ unlucky but inside the distribution. The interesting upstream signal is the
model-family axis
: long-context RL-trained Qwen-thinking occasionally produces a correct trajectory where gpt-oss-120b-MXFP4 apparently never does — a coverage / sampling-diversity effect rather than a pipeline breakthrough, and one that reinforces the multi-architecture-ensemble direction we did not run (§7.2).
A5. Reference Run Expected Distributions
See
reproducibility/expected_hashes.json
for the JSON schema consumed by
verify.py --mode stochastic
. Reproduced here as markdown for inspection without leaving the document.
Problem
Ground truth
Category
Runs (of 30) producing GT
Top-3 empirical answers
0e644e
336
Algebra
29 (97%)
336 (97%), 337 (3%)
26de63
32951
Algebra
30 (100%)
32951 (100%)
424e18
21818
Combinatorics
29 (97%)
21818 (97%), 21819 (3%)
42d360
32193
Combinatorics
28 (93%)
32193 (93%), 32194 (7%)
641659
57447
Combinatorics
24 (80%)
57447 (80%), 57448 (10%), 57450 (10%)
92ba6a
50
Algebra
30 (100%)
50 (100%)
9c1c5f
580
Geometry
22 (73%)
580 (73%), 579 (13%), 581 (7%), 578 (7%)
a295e9
520
Geometry
20 (67%)
520 (67%), 521 (13%), 519 (10%), 522 (10%)
86e8e5
8687
Number Theory
0 (0%)
41754 (50%), 13657 (33%), 99427 (10%), 41755 (7%)
dd7f5e
160
Number Theory
30 (100%)
160 (100%)
A6. Curated Journey Log (22 days, narrative register)
The full
docs/MASTER.md
is 7,253 lines. We curate a one-paragraph-per-day summary below, preserving only entries that resulted in a committed hypothesis, a submission, or a methodological lesson.
Day 1 (2026-03-24).
Entered competition. Scanned public notebooks; identified 3 forked winner notebooks scoring 40–44. Decision: fork the highest-scoring public notebook as week-1 baseline.
Days 2–4 (2026-03-25 to 27).
Reproduced public notebook in our environment. Fixed our first bug: the public notebook used Harmony format; an earlier incorrect fork used HuggingFace chat template and scored 3/50. Confirming Harmony format → 42 on dry run.
Day 5 (2026-03-29).
Submitted v7 (= public winner fork). Scored
42/50
public leaderboard. Floor locked. Committed to anti-consensus principle in
docs/MASTER.md
: "The public consensus recipe is the visible plateau, NOT the real ceiling." (Later violated this rule for 3 weeks; see Day 22 entry.)
Day 6 (2026-03-30).
First falsification (A02): merged huikang's published AIMO2 LoRA into gpt-oss-120b's MXFP4 quantized weights. v13 submission scored 40/50 (−2). First evidence of the merge-then-requantize collapse. Investigated for three hours; hypothesized quantization noise but did not formalize the SNR argument until §6.2.
Day 7 (2026-04-01).
Committed EAGLE-3 hypothesis (A03). Tested locally on vLLM 0.19; worked (2.5× throughput). Did NOT test on Kaggle's vLLM 0.11.2 before submission.
Day 8 (2026-04-03).
v15 submission (EAGLE-3) scored
0/50
. Post-mortem revealed zero-token output under vLLM 0.11.2 + MXFP4 + MoE combination. 11 hours of investigation; found multiple upstream GitHub issues documenting each of the three sub-mechanisms (draft-head dense-layer feature alignment, MXFP4 kernel path without fused speculative-verify, spec-decoding empty-output silent fallback) via keyword searches archived at the time. This became §6.1.
Day 9 (2026-04-04).
Committed Bayesian sqrt-prior hypothesis (A04). Mathematical derivation felt principled; I did not cross-check against the Condorcet Jury Theorem literature before submitting.
Day 10 (2026-04-05).
v18 submission (sqrt-prior Bayesian) scored
23/50
(−19). Largest single-submission regression. Post-mortem: identified anti-Condorcet inversion mechanism. This became §6.3.
Days 11–14 (2026-04-06 to 09).
Attempted increasingly sophisticated selection rules: v19 six-step pipeline (timeout, 32), v20 seven-feature stack (32), v22 LLM-judge ensemble (deferred), GenSelect JUDGE (32). Pattern emerged: every "smart pipeline" ends at 32. First awareness that the pipeline was not the lever.
Days 15–18 (2026-04-10 to 13).
Attempted learned voting (RF classifier, evolutionary search, PRM-weighted). All either fit train noise or regressed on holdouts. Critical realization: cross-validation on 5 random 50/50 splits rejected every learned rule.
Days 19–20 (2026-04-14 to 15).
Last-slot strategy. Built v28-cook44-lb with cook44 5-step prompt + temp=0.5 + three safety bugfixes. Deployed 13 audit agents to verify zero defects before final push. Pushed successfully. Kaggle's auto-selection ultimately chose v7 Winner Fork V12 as the final submission (it had the highest public score; v28 was close but not higher).
Days 21–22 (2026-04-16 to 17).
Private leaderboard revealed: our final submission scored
42/50 private
(public = private = 42, gap = 0). Began writeup. Realized that the anti-consensus principle I wrote on Day 5 had been violated for ≈19 of 22 days — every one of my "departures" had been a refinement of consensus techniques, not a genuine departure.
A7. What a Future AIMO4 Entrant Should Do With This Document
Clone the companion repo
(
reproducibility/
).
Run
pytest tests/
(≤5 minutes, CPU-sufficient for 2 of 3 tests). Confirms the substrate-trap detectors work.
If AIMO4 is on the same or similar MXFP4 MoE substrate
— run
reproduce.sh
to confirm you reproduce our baseline 42-equivalent on your H100. The
verify.py --mode stochastic
check gives a distributional PASS/FAIL in ≈25 minutes.
Do NOT attempt, on an MXFP4 MoE substrate:
Merging a pre-trained LoRA and re-quantizing → §6.2
EAGLE-3 speculative decoding on vLLM 0.11.x → §6.1
Sqrt-prior or other anti-Condorcet voting reweighting → §6.3
The other thirteen variants in A1 (lower priority; §4.3 grouping summarizes why)
Instead, attempt
(sections our evidence does NOT close, §7.2):
Symbolic verification layers (Lean / Coq / sympy assertion) as a voting signal
Hand-curated problem-domain retrieval at inference time
Multi-model ensembles with architecturally-distinct bases
QA-LoRA or other quantization-aware adapter training
Test-time training between the two scoring runs
Companion files in this repository:
main.md
— §§1-7 (academic register)
tests/test_lora_mxfp4_collapse.py
— §6.2 detector, runnable on CPU
tests/test_bayesian_sqrt_inversion.py
— §6.3 detector, runnable on CPU
tests/test_eagle3_moe_zero_tokens.py
— §6.1 detector (rule-based + optional H100 integration)
reproducibility/reproduce.sh
— §3.4 one-command reproduction, two modes
reproducibility/verify.py
— SHA256 check (strict) or distributional bound (stochastic)
reproducibility/environment.lock
— pinned versions
reproducibility/expected_hashes.json
— A5 reference distributions
figures/
— four figures (ablation forest, MXFP4 collapse boundary, submission timeline, per-category breakdown)
submission/notebook.py
— verbatim source of
sebastiangil00/aimo3-v7-winner-fork Version 12