# Harmony Format, Dead Ends, and Variance in AIMO3

Solo Agent-Team Approach to AIMO3: Harmony Format Discovery, Negative Results, and Variance Analysis
Submission details:
Leaderboard:
Public LB + Private LB (both final)
Final submission:
[aimo3-v33-fixed Version 2](https://www.kaggle.com/code/kolenkovskin/aimo3-v33-fixed?scriptVersionId=306011227)
Colab notebook (Kaggle-agnostic):
[aimo3-v33-colab](https://colab.research.google.com/drive/1-AHappCM4UvLu3Ud43tYj3opyggyUTge?usp=sharing)
Public LB (best):
39/50 |
Rank:
1518 / 4,138 teams
Private LB (final):
39/50 (matches public score — final submission
aimo3-v33-fixed Version 2
scored 39 on both leaderboards)
Variance:
Same code, same prompts, three runs: 39, 33, 34 on public LB
Team:
Kolenkovskin (solo)
Base model:
[danielhanchen/gpt-oss-120b](https://www.kaggle.com/models/danielhanchen/gpt-oss-120b)
(no fine-tuning in final submission)
Code:
Kaggle notebook linked above (single .ipynb, complete pipeline)
Abstract
We describe a solo competitor's approach to AIMO3 (AI Mathematical Olympiad Progress Prize 3) using GPT-OSS-120B and a multi-agent AI team. Throughout this writeup, "we" refers to the author and the AI agent team used for development.
Our best score of
39/50
(mean ~34, sigma ~3) came from a single discovery: a Harmony message-format error present in all 19 analyzed public notebooks. Adding the correct ChatGPT identity string and a developer message with code-format rules improved score by +2 points — the only statistically meaningful gain across 28 submissions. The same code (v33-fixed) scored 39, 33, and 34 across three runs, demonstrating the variance in this evaluation.
We document 20+ dead ends with empirical evidence, including the finding that all post-hoc selection methods collapse to majority voting on IMO-hard problems where the per-attempt success rate falls below 50%. We provide variance analysis (sigma ≥ 3), a failure taxonomy of GPT-OSS-120B, and three LoRA fine-tuning experiments on an MXFP4-quantized MoE model served with adapters via vLLM. All prompts, configurations, and negative results are included for reproducibility.
1. Introduction
The AI Mathematical Olympiad Progress Prize 3 (AIMO3) challenges participants to solve 50 IMO-level mathematical problems using a single H100 80GB GPU within a 5-hour runtime, with internet disabled and integer answers in [0, 99999]. Scoring uses double-run averaging: each problem is scored 0 or 1 per run, averaged across two independent runs.
We participated as a solo competitor using a multi-agent AI team: Claude Code (Opus) for implementation, Claude Desktop for strategic oversight, Deep Research for literature investigation, Grok for adversarial analysis, and Chrome Extension for community intelligence. This writeup covers 40 days of development across 25+ sessions.
Our core model is GPT-OSS-120B, a Mixture-of-Experts model with 116.8B total parameters (5.1B active per token), quantized to MXFP4 (Microscaling FP4) during post-training. We use self-consistency sampling (K=8) with majority voting and Tool-Integrated Reasoning (TIR) via the Harmony message format.
Key contributions:
Discovery and fix of the Harmony format error present in all public notebooks (+2 points)
Empirical negative-results catalog: 20+ dead ends with quantified regressions
Variance analysis confirming sigma ≥ 3 on identical code (39 and 33 on same submission)
First documented LoRA fine-tuning of an MXFP4-quantized MoE model served with adapters via vLLM
Multi-agent AI team methodology for competition problem-solving
2. Model Understanding: GPT-OSS-120B
2.1 Architecture
GPT-OSS-120B is a Mixture-of-Experts transformer with extreme sparsity:
Total parameters:
116.8B (MoE) + routing layers
Active parameters per token:
5.1B (4.4% utilization)
Experts:
128 per layer, top-4 routing
Quantization:
MXFP4 (e2m1 format, 1 mantissa bit, 8 representable values per sign)
VRAM footprint:
~60.8 GB at gpu_memory_utilization=0.96
The MXFP4 quantization is applied during post-training (not post-hoc), meaning the model was trained with this precision in mind. However, the BF16 router weights combined with MXFP4 expert weights create routing non-determinism: 1,000 identical requests at T=0 produce 80 unique completions.
2.2 Harmony Message Format
GPT-OSS uses a proprietary "Harmony" encoding with 6 message carriers:
#
Carrier
Purpose
Tunable?
1
SystemContent
Model identity and personality
Frozen
2
ReasoningEffort
Controls thinking depth (low/medium/high)
medium
3
ToolNamespaceConfig
Tool definitions (Python interpreter)
Load-bearing
4
DeveloperContent
Developer instructions
+2 points
5
UserContent
Problem statement
Per-problem
6
preference_prompt
Output format preferences
Frozen
Critical discovery (v33, +2 points):
The model expects SystemContent to contain the ChatGPT identity string ("You are ChatGPT, a large language model trained by OpenAI…"). All 19 analyzed public notebooks either omit this or replace it with a custom identity. Additionally, the DeveloperContent carrier (exposed by the openai-harmony library as
.with_instructions()
) accepts code-format rules that tripled the model's Python tool efficiency from 42 to 13 calls per problem, as measured in S12 A/B tests.
Catastrophic failure mode (v32, score=0):
Using
.with_text()
instead of
.with_instructions()
raises
AttributeError
, caught by a silent try/except, causing all 400 inference attempts to return None.
2.3 Output Channels
GPT-OSS produces output in 3 channels:
analysis
(hidden reasoning),
commentary
, and
final
. The host pipeline reads only
final
for answer extraction and routes
recipient=python
blocks to Jupyter kernels. Analysis content is appended to conversation history but never inspected by host code. All 19 analyzed competitor notebooks follow this same pattern.
2.4 Failure Taxonomy on IMO-Level Problems
Type
Frequency
Behavior
Fixable?
E1: Misunderstanding
~54% of errors
Wrong mathematical formalization
LoRA (possibly)
E2: Arithmetic error
~15%
Correct approach, wrong calculation
No (random)
E3: Thread loss
~15%
Starts correctly, diverges mid-solution
Possibly
E4: Wrong approach
~10%
Incorrect strategy from start
LoRA (possibly)
E5: Cannot begin
~6%
Problem too novel for model
No (ceiling)
Source: Prism Math taxonomy (4601 examples) applied to our 12-problem hard set.
![](assets/ee609b84fc2c4f54def1759f.png)
3. Inference Pipeline (Kaggle Submission)
3.1 Architecture
[Problem text] → Harmony encoding → vLLM (GPT-OSS-120B) → TIR loop
                                                              ↓
                                          Jupyter kernel ← code blocks
                                                              ↓
                                          \boxed{answer} extraction
                                                              ↓
                                          K=8 attempts → majority vote
3.2 vLLM Configuration
Parameter
Value
Rationale
gpu_memory_utilization
0.96
Maximum without OOM
max_model_len
65536
Full context for long solutions
max_num_seqs
256
Default; reduction to 8 tested and found neutral
dtype
auto (MXFP4)
Native model precision
kv_cache_dtype
fp8_e4m3fn
Saves VRAM on H100
enforce_eager
True
Avoids CUDA graph issues
port
8000
Local subprocess
3.3 TIR Execution
16 parallel Jupyter kernels via
jupyter_client
Per-kernel timeout: 60 seconds
Code output truncation: first 300 + last 300 characters
Bug spiral detection: abort after 3 consecutive errors (saves ~15% compute on hard problems)
Maximum code executions: not capped (model self-limits at ~6-8)
3.4 Sampling and Selection
K=8
attempts per problem (K=16 tested, no improvement: v28 = 37 = v24)
Temperature:
1.0 (T=0.8 tested in v37, neutral)
Reasoning effort:
medium
Selection:
majority vote with longest-trace tiebreak
Answer extraction:
regex
\boxed{(\d+)}
on final channel output
Extraction accuracy:
100% (verified on 100 test cases)
3.5 Time Management
Total budget: 5 hours (18000 seconds)
Per-problem allocation:
remaining_time / remaining_problems
Startup (vLLM load): ~180 seconds
Average per-problem: ~4-5 minutes (89% budget slack confirmed v25: 30min/290min)
search_tokens=1024 for early
\boxed{}
detection
3.6 Invariants (Breaking Any = Failed Submission)
Dummy
submission.parquet
written before ANY imports (crash protection)
Version stamp printed as first output line
KAGGLE_IS_COMPETITION_RERUN
detection for force-serve mode
try/except in final cell with parquet fallback
ensure_ascii=True
on all
json.dump
calls (cp1252 encoding protection)
Dual-path model/wheels resolution for interactive vs rerun environments
4. Selection Methods: Negative Results
The gap between pass@8 (probability that at least one of 8 attempts is correct) and majority@8 (majority vote accuracy) is large: approximately 80% vs. 40% on our 5-problem measured subset. This coverage-versus-pass-rate (CCPR) gap motivated extensive investigation into selection methods. Every method tested failed to close it.
4.1 Methods Tested
Method
Result
Evidence
Majority voting (baseline)
37/50
v24, v25, v28
IEW entropy weighting
= majority
AUC=0.595 (near-random)
GenSelect (LLM-as-judge)
-3 points
v29=34; Harmony format incompatible
Adaptive routing (difficulty-gated)
-3 points
v29=34; complexity = regression
Self-verification (verify-noDev)
-3 to -5
FPR=75% (self-confirmation bias)
Code trajectory analysis
= majority
Signal vanishes on hard problems
Z3 constraint verification
0/16 distinguish
Regex constraints too weak
ECP evidence mining
-4 points
FPR=37%
ThinkPRM-1.5B (process reward model)
Inverse scoring
Wrong=1.0, correct=0.7 on hard
Longest-trace tiebreak
+1 (tiebreak only)
v44+v45: 6/12 vs majority 5/12
DeepConf (13 methods, K=32)
= majority
0/12 at K=32, all methods
Minority selection
3/3 on hard
But 0/3 on easy — not usable
4.2 Why Selection Fails on Hard Problems
Chen et al. (NeurIPS 2024) prove that when p(correct) < p(wrong) per attempt, increasing K makes majority voting worse, not better. On our hard problems, the model produces the same wrong answer 6–7 out of 8 times. No post-hoc method can recover correctness from a distribution where the correct answer is in the minority.
Generation quality, not selection quality, is the bottleneck.
This overrides our S14–S20 conclusion that "selection = THE bottleneck." On easy problems (pass@1 > 50%), majority voting works. On hard problems (pass@1 < 20%), the model produces the same wrong answer 6–7 out of 8 times — no post-hoc method recovers from that.
4.3 External Confirmation
wenliangtlh independently confirms the CCPR gap: pass@128 = 0.867 vs. maj@128 = 0.453
Top-20 Kaggle teams all use majority voting; no confidence-based selection method appears in the top 20 (per community survey)
Nitarach (arXiv:2603.27844) analysis confirms that approximately 3 borderline problems at p ≈ 0.50 drive all observed leaderboard variance
5. LoRA Fine-tuning Experiments
5.1 Motivation
All AIMO1/2 winners used fine-tuning (NuminaMath, NemoSkills). NemoSkills gained ~4 points over vanilla baselines through fine-tuning [3]. Public notebook ceiling is 44/50; leaderboard leader achieved 46/50 with 113 submissions.
5.2 Technical Challenge: MXFP4 + LoRA
No published results exist for LoRA fine-tuning of MXFP4-quantized MoE models. Key challenges:
Training uses BNB-4bit quantization; inference uses native MXFP4 (different schemes)
All model-merge paths fail (safetensors format mismatch, Marlin kernel incompatibility)
Solution:
vLLM 0.18.1
--enable-lora
serves adapter directly on MXFP4 base (no merge needed)
5.3 Experiments
Run
Data
Format
Examples
seq_len
Loss
Result
v1
Harmony-filtered (easy)
Harmony
987
2048
0.64
5/5 easy, 0/12 hard (= base)
v2
OMR hard (DeepSeek format)
CoT (
<think>
)
2000
2048
0.52
1/12; 5/12 extraction errors
v3
Nemotron-Math-v2 (GPT-OSS traces)
Harmony (converted)
5000
4096
0.55
BNB4/MXFP4 mismatch: code_execs=0
Key finding:
Format matters more than data quality. v2 trained on hard problems in wrong format (CoT) scored worse than v1 trained on easy problems in correct format (Harmony). v3 had correct format but the BNB4→MXFP4 quantization gap caused the model to lose TIR capability entirely (zero code executions).
5.4 Training Configuration (v1, best result)
Framework:
Unsloth + PEFT
Quantization:
BNB-4bit (NF4, double quantization)
LoRA:
r=16, alpha=32, target_modules=[q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj]
Optimizer:
adamw_8bit, lr=1e-4, cosine schedule, warmup_ratio=0.03
Batch:
per_device=2, gradient_accumulation=4
Hardware:
1x H100 80GB SXM (RunPod), ~2h training
Adapter size:
73 MB
5.5 Data Sources
Dataset
Size
Format
Used
nvidia/OpenMathReasoning TIR
1.7M
TIR traces
Filtered to 5000
nvidia/Nemotron-Math-v2
3.5M
GPT-OSS native
Converted 5000
Jackrong/Harmony-dataset
8471
Harmony format
Not used (discovered late)
21 HF AIMO3 finetunes
varies
Unknown
All have empty model cards
6. Variance Analysis
6.1 Empirical Evidence
Our v33 submission scored
39
,
33
, and
34
on identical code across three separate runs (submitted March 24, March 28, and April 7). This 6-point spread (39 vs 33) from the same codebase is the strongest direct evidence of high variance. The mean of our three runs (35.3) is consistent with sigma ~3.
6.2 External Analysis
Nitarach (arXiv 2603.27844) ran 13 identical submissions of the same pipeline and found:
Range: 37–44 (7-point spread)
Mean: 39.7
Estimated sigma: ~1.7
However, our observation (39 vs 33, delta=6) suggests true sigma may be higher (~3), possibly due to:
MXFP4 routing non-determinism (80 unique completions at T=0)
Kaggle H100 hardware variation between runs
~3 borderline problems at p≈0.50 creating bimodal outcomes
6.3 Implications for Evaluation
Delta
Interpretation
< 3
Noise; do not react
3-4
Possibly real; needs 3+ confirmations
> 4
Likely real improvement
This means our 18-submission history contains
one
statistically meaningful finding: v33 (Harmony fix) = +2 over v28 baseline of 37, confirmed by the mechanism (format change has clear causal path). All other deltas are within noise.
6.4 Leaderboard Implications
The leaderboard leader scored 46/50 from 113 entries. Using winner's curse correction (ML estimate), their true mean is likely ~41.8. The public ceiling of 44/50 from unmodified notebooks is also inflated. True ceiling for vanilla GPT-OSS-120B is approximately 36-38/50.
![](assets/f912eb7753a139990b2fcd9a.png)
7. Competitor Analysis
7.1 Public Notebook Forensics
We analyzed 19 public notebooks scoring 36-44 on the leaderboard. Key findings:
Feature
Our v33
Typical competitor
Best competitor
Identity
ChatGPT (correct)
Custom/omitted
Custom/omitted
DeveloperContent
Code-format rules
None
None
K (attempts)
8
8
8-16
Temperature
1.0
0.6-1.0
0.6
gpu_memory_util
0.96
0.90
0.90
max_num_seqs
256
6-8
64
Selection
majority + longest
majority
majority
min_p
0.02
0.02-0.05
0.05
7.2 Comparison with SOTA
Method
Score
Notes
Public baseline (kurianbenoy fork)
36
Unmodified
Best public notebook (ans-verifys)
44
Lucky run; likely mean ~38-40
Leaderboard leader (ippeiogawa)
46
113 entries; true mean ~41.8
Our v33 (final submission)
39 public / 39 private
3-run variance (public): 39, 33, 34 — mean 35.3
AIMO2 winner (NemoSkills, 14B fine-tuned)
34/50
AIMO2, not directly comparable (4x L4 hardware)
DeepSeek-R1 (671B, if available)
N/A
Not applicable (too large)
8. Multi-Agent Workflow
8.1 Agent Roles
Agent
Model
Role
KK (Claude Code)
Claude Opus
Implementation, code audit, testing, memory
KD (Claude Desktop)
Claude Opus
Strategy, Deep Research, coordination
DR (Deep Research)
Claude Opus
Literature investigation (86 reports)
GK (Grok)
grok-4.20
Adversarial analysis, RunPod SSH
KH (Chrome Extension)
Claude Sonnet
Community intelligence, leaderboard
8.2 Communication Infrastructure
Shared filesystem:
theory/
(86 .md files),
prompts/
(45 files),
logs/kk-sessions/
Memory mirror:
.claude-memory-mirror/
(5 files: rules, dead ends, key files, bugs, verified)
Journal:
journal_aimo3.txt
(prepend-only, reverse chronological)
Execution contract:
CLAUDE.md
with mandatory reads and theory-check protocol
8.3 Effective Practices
Bug detection:
Pre-submission code review caught 5 critical bugs in v29 before the Kaggle run
Competitor forensics:
One afternoon analyzing 19 notebooks produced more actionable insight than 86 research investigations
Test framework:
69 local tests running in 0.06 s prevented regression without GPU access
Theory audit:
Classifying all 86 research files, archiving 51, and preserving 35 kept working context focused
8.4 Failure Modes
Research without measurement:
0 of 86 deep-research investigations led to a positive code change
False premise cascade:
One unverified assumption ("our gap = 7 points") propagated through 30+ research files over 20 days
Agent convergence is not evidence:
All three agents agreed that extraction was the bottleneck; the first direct measurement disproved it in under a minute
9. Dead Ends Summary
We tested 20+ approaches beyond the baseline. None improved the score. The full catalog is in Appendix H; here we summarize the key categories:
Category
Methods Tested
Best Result
Lesson
Prompt engineering
3-prompt diversity, minimal prompts
−3 each
Prompts are frozen for reasoning-native models
Sampling
Multi-temp, SC@16
0, 0
Diminishing returns beyond K=8
Selection (LLM-as-judge)
GenSelect via chat API
−3
Harmony format incompatible with judging
Selection (statistical)
IEW, DeepConf (13 methods), LR scoring, Bayesian, CARE
= majority
All collapse to majority on hard problems
Selection (code analysis)
Trajectory, Z3, ECP mining
0 to −4
Signal vanishes on hard
Verification
Self-verify, ThinkPRM-1.5B
−3 to −5, inverse
Self-confirmation bias; PRM inverts on hard
Model diversity
20B dual-model
−1
Same wrong answers as 120B
Prompt diversification
Prescriptive, "solve two ways"
0
Model either knows or doesn't
![](assets/00f8b10a95da90bf6f4eb5aa.png)
10. Lessons Learned
Read competitor code on day 1.
We didn't. That cost 20 days. One afternoon of notebook forensics produced the only +2 gain across 28 submissions. 86 deep-research reports produced zero code changes.
Measure before theorizing.
Three independent agents converged on "extraction is the bottleneck." First measurement: extraction accuracy = 100%. One number killed an entire research track.
Subtraction beats addition.
Every feature we added scored zero or negative. The only improvement came from fixing a bug (Harmony format) and removing harmful features (IEW, GenSelect). Simple majority vote at 37/50 beat our most elaborate pipeline at 34/50.
Ten submissions went to noise.
With sigma ≥ 3, the gap between 33 and 39 is one standard deviation. We needed: delta > 4 plus a mechanistic explanation before reacting. We didn't have that rule until submission 17.
Off-by-one is systematic, not random.
44% of wrong answers differ from gold by ±1 or ±2 — the same boundary confusion (inclusive vs. exclusive ranges, base cases) across different problems. This is a training data artifact, not a pipeline issue.
11. Discussion
11.1 What We Would Do Differently
The optimal strategy for a solo competitor, in hindsight, would be:
Day 1: Fork best public notebook, submit unmodified (baseline)
Day 2: Analyze all public notebooks, find the key diffs
Day 3-5: Fix Harmony format, confirm +2
Day 6-15: LoRA fine-tuning with Jackrong/Harmony-dataset (correct format, hard problems)
Remaining days: Submit LoRA variants, collect variance data
We instead spent 20 days on prompt engineering and selection methods before reading competitor code.
11.2 Open Questions
Does the analysis channel contain usable confidence signals?
Can targeted LoRA on off-by-one examples fix boundary confusion?
What is the true per-problem variance (p_i) for each of the 50 problems?
Would EAGLE3 speculative decoding (40% throughput gain) enable K=12-16 within budget?
11.3 Reproducibility Note
All code is contained in a single Jupyter notebook. All prompts are frozen and documented in Appendix A. The only external dependency beyond standard Kaggle environment is the GPT-OSS-120B model and vLLM wheels, both available as Kaggle datasets.
For Kaggle-agnostic reproducibility, a Google Colab version of the full pipeline is provided:
[aimo3-v33-colab](https://colab.research.google.com/drive/1-AHappCM4UvLu3Ud43tYj3opyggyUTge?usp=sharing)
. It installs the
openai-harmony
library and
vLLM
from public PyPI, downloads the
gpt-oss-120b
weights from Hugging Face, and exposes the same inference harness used in the Kaggle submission (with the
kaggle_evaluation
gateway import guarded by
try/except ImportError
so the notebook runs in any environment with a single H100-class GPU).
References
[1] Nitarach. "Variance Analysis of AIMO3 Submissions." arXiv:2603.27844, 2026.
[2] Chen et al. "When Majority Voting Fails." NeurIPS, 2024.
[3] NVIDIA NemoSkills team. "NemoSkills: AIMO2 Winning Solution." arXiv:2504.16891, 2025.
[4] Prism Math. "Error Taxonomy for Mathematical Reasoning." 4,601 examples, 2025.
[5] Ai. "On Correlated Voting." Corollary 2 (Dawid-Skene limitation), 2025.
Appendix A: Prompts (Verbatim)
All prompts are copied exactly from
aimo3-v33-fixed.ipynb
(CFG class). These produced 39/50.
Any modification causes regression
(confirmed v22, v27).
A.1 System Prompt (Frozen)
You are a world-class International Mathematical Olympiad (IMO) competitor.
The final answer must be a non-negative integer between 0 and 99999.
You must place the final integer answer inside \boxed{}.
Injected into DeveloperContent carrier (not SystemContent). SystemContent carries model identity (A.6).
A.2 Tool/Namespace Prompt (Frozen, Load-Bearing)
Use this tool to execute Python code.
The environment is a stateful Jupyter notebook.
You must use print() to output results.
WARNING:
Removing this caused +9pp execution errors (v27).
A.3 Developer Message (The +2 Discovery)
# PROBLEM-SOLVING PROTOCOL

def solve(problem: str) -> int:
    problem_type = classify(problem)
    answer_analytical = solve_analytically(problem, problem_type)
    answer_numerical = verify_with_python(problem, answer_analytical)
    if answer_analytical != answer_numerical:
        answer_analytical = answer_numerical  # trust Python over reasoning
    return answer_analytical

def solve_analytically(problem, problem_type):
    method_1 = pick_method(problem_type)
    result = attempt(method_1)
    if result is None:
        method_2 = pick_different_method(problem_type, exclude=method_1)
        result = attempt(method_2)
    return result

def verify_with_python(problem, claimed_answer):
    # ALWAYS run. Use INDEPENDENT method from analytical.
    # counting -> enumerate and count
    # optimization -> scipy.optimize or brute force
    # equations -> sympy.solve
    # NEVER recompute same formula
    return execute(write_verification_code(problem))

def on_python_error(code, error):
    # NEVER debug. ALWAYS delete and rewrite from scratch.
    return write_completely_new_code()

# HARD RULES:
# - Final answer inside \\boxed{}
# - Answer: non-negative integer 0-99999
# - Use math, numpy, sympy, scipy, itertools, collections
# - If two methods disagree, run THIRD to break tie
Injected via
DeveloperContent.new().with_instructions()
. Tripled Python tool efficiency: 42 to 13 code calls per problem (S12 A/B test).
A.4 Preference Prompt (Frozen)
Use `math`, `numpy`,`sympy`,`itertools` and `collections` to solve the problem.
A.5 User Prompt Template
Each problem sent as single UserContent message with raw problem text. No framing, no few-shot, no CoT instructions.
A.6 Harmony System Identity (Critical)
SystemContent.new()
    .with_model_identity('You are ChatGPT, a large language model trained by OpenAI.')
    .with_reasoning_effort(reasoning_effort=ReasoningEffort.HIGH)
    .with_tools(tool_config)
This is the +2 discovery.
All 19 competitor notebooks omit or replace this identity. The
openai-harmony
library does NOT auto-inject it.
A.7 Message Assembly
messages = [
    Message.from_role_and_content(Role.SYSTEM, system_content),      # Identity + tools
    Message.from_role_and_content(Role.DEVELOPER, developer_content), # Instructions
    Message.from_role_and_content(Role.USER, problem_text)            # Problem
]
Three messages total. No conversation history between problems.
Appendix B: Complete Version History
![](assets/3d5485070bec2d6f87088236.png)
#
Date
Version
Score
Delta
Key Change
1
2026-03-06
DeepSeek-14B v6
0
—
First submit (RERUN detection broken)
2
2026-03-07
QwQ-32B v7
0
0
Infrastructure works, model too weak
3
2026-03-11
GPT-OSS v19
0
0
6 root-cause fixes, still broken
4
2026-03-12
Fork v3 (baseline)
36
+36
Unmodified kurianbenoy fork
5
2026-03-13
v22
33
−3
3-prompt diversity
6
2026-03-14
v23
36
+3
Minimal prompt + multi-temp
7
2026-03-15
v24
37
+1
batch=256, gpu=0.96, ctx=65536
8
2026-03-16
v25
37
0
Variance confirmation
9
2026-03-18
v28
37
0
SC@16 (= SC@8)
10
2026-03-19
v29
34
−3
GenSelect + adaptive
11
2026-03-20
v30
36
−1
Subtraction (K=8, timeout=10)
12
2026-03-20
v31
34
−3
Align with 42-scorer
13
2026-03-22
v32
0
−37
BUG:
.with_text()
AttributeError — all 400 attempts returned None
14
2026-03-24
v33
39
+2
Harmony fix + developer message
15
2026-03-24
v34
34
−5
PROVE filter + seqs=8
16
2026-03-26
v37b
31
−8
Experimental features
17
2026-03-28
v33 resub #1
33
−6
Same code as row 14; confirms variance
18
~2026-03-31
v35-verify-fresh
34
−5
Self-verification (FPR=75%)
19
~2026-04-01
v31-majority-clean
0
−39
Bug: clean majority rewrite failed
20
~2026-04-02
gptoss-fork-v1.2
34
−5
Fork variant
21
~2026-04-03
v54-mining-longest v1
0
−39
Code mining extraction (broken)
22
~2026-04-04
v54-mining-longest v2
Error
—
Kaggle Error
23
~2026-04-04
v54-code-mining+longest
32
−7
Code mining + longest tiebreak
24
~2026-04-05
v55-eagle3 v2
1
−38
EAGLE3 speculative decoding (VRAM crash)
25
~2026-04-07
v33-fixed resub #2
34
−5
Same code as row 14; third run
26
~2026-04-08
v56-k10-longest
36
−3
K=10 + longest tiebreak
27
~2026-04-10
v57-k12-longest
32
−7
K=12 + longest (over-sampling)
28
~2026-04-11
v58-dsq-lora v3
23
−16
LoRA adapter (BNB4/MXFP4 mismatch)
—
2026-04-15
v33-fixed (final)
34
—
Selected as final submission
Appendix C: Reproducibility Checklist
C.1 Environment
Kaggle GPU:
1x H100 80GB SXM, 5h runtime
Python:
3.12 (Kaggle default)
vLLM:
0.18.1 (installed from Kaggle dataset wheels)
Model:
danielhanchen/gpt-oss-120b (Kaggle model dataset)
Library:
openai-harmony (for Harmony encoding)
C.2 Exact vLLM Arguments
python -m vllm.entrypoints.openai.api_server \
    --seed 42 \
    --model /kaggle/tmp/setup/model \
    --served-model-name gpt-oss \
    --tensor-parallel-size 1 \
    --max-num-seqs 256 \
    --gpu-memory-utilization 0.96 \
    --host 0.0.0.0 \
    --port 8000 \
    --dtype auto \
    --kv-cache-dtype fp8_e4m3 \
    --max-model-len 65536 \
    --stream-interval 200 \
    --max-cudagraph-capture-size 64 \
    --async-scheduling \
    --disable-log-stats \
    --enable-prefix-caching
C.3 Kaggle Dataset Sources
Type
Source ID
Description
competition
118448
aimo-3 competition data
modelInstanceVersion
510391 (model 422384, instance 404485)
danielhanchen/gpt-oss-120b
kernelVersion
289055161
aimo-3-utils (vLLM 0.18.1 wheels + tiktoken encodings)
C.4 Seeds
Global seed: 42
Per-attempt seed:
pow(seed + attempt_index, 2) % (2**31)
Note: MXFP4 routing is non-deterministic regardless of seed
C.5 Critical Code Paths
Function
Cell
Purpose
CFG
(class)
9
All hyperparameters and prompts
AIMO3Template.apply_chat_template()
11
Harmony message encoding
AIMO3Sandbox
12
Jupyter kernel pool management
AIMO3Tool
13
TIR code execution loop
AIMO3Solver.__init__()
14
vLLM server startup + model preload
AIMO3Solver._start_server()
14
vLLM subprocess with exact CLI args
predict()
17
Main entry: K=8 attempts + majority vote
Pre-flight check
16
Developer message rendering verification
All code in single notebook:
aimo3-v33-fixed.ipynb
(19 cells).
Appendix D: Off-by-One Error Analysis
Problem
Gold
Majority
Gold − Majority
Error type
olymp-01629
4950
4949
+1
Boundary inclusive/exclusive
oda-00425
3
4
−1
Count boundary
dapo-15481
2
0
+2
Base case
oda-01612
4
2
+2
Multiplicity
olymp-02392
49
51
−2
Range boundary
dapo-10920
4022
4024
−2
Coefficient error
oda-01183
12
6
+6 (×2)
Factor omission
7/16 (44%) of hard-problem errors are within ±2 of the correct answer.
Appendix E: LoRA Training Logs
E.1 v1 Training (Harmony format, easy data)
Steps: 124, Loss: 1.02 → 0.64
Data: 987 examples, Harmony format, AMC/AIME level
Config: r=16, alpha=32, seq_len=2048, lr=1e-4, 1 epoch
Duration: ~2h on 1x H100 80GB
Result: 5/5 easy (= base), 0/12 hard (= base). No improvement.
Root cause: training examples too short (≤16K chars), output collapsed to ~14 chars
E.2 v2 Training (DeepSeek format, hard data)
Steps: 250, Loss: 0.87 → 0.52
Data: 2000 examples,
<think>
CoT format (DeepSeek-R1 style)
Config: r=16, alpha=32, seq_len=2048, lr=1e-4, 1 epoch
Duration: ~3.8h on 1x H100 80GB
Result: 1/12 correct (= base). 5/12 NoneType extraction errors.
Root cause:
Format mismatch.
Model learned
<think>
format, but inference uses Harmony. Outputs malformed at inference time.
E.3 v3 Training (Nemotron traces, Harmony converted)
Steps: ~625, Loss: 0.91 → 0.55
Data: 5000 examples from nvidia/Nemotron-Math-v2 (GPT-OSS native traces), converted to Harmony
Config: r=16, alpha=32, seq_len=4096, lr=5e-5, 1 epoch
Duration: ~10h on 1x H100 80GB
Result:
BNB4/MXFP4 mismatch.
Training uses BNB-4bit quantization; inference uses native MXFP4. The adapter learned on a different numerical representation. At inference: code_execs=0, turns=1 (model lost TIR capability entirely).
Root cause: Fundamental incompatibility between training-time and inference-time quantization schemes. No known solution short of full model merge (which also fails due to Marlin kernel format).
Appendix F: Experiments Not Run
Experiment
Why It Matters
Why Not Run
Estimated Impact
Multi-run per problem (12+)
Per-problem variance
Max 3 runs available
Diagnostic
Reasoning channel analysis
Hidden confidence signals
Pipeline not instrumented
Unknown
PRM scoring (7B)
Step-level selection
VRAM blocked
+2-4 (literature)
CodeBERT clustering
Approach diversity metric
Not prioritized
Unknown
Adaptive K (K=16-32 on hard)
Budget 89% slack
Not tested in time
+1-2
LoRA on boundary errors
44% of errors = off-by-one
Data not curated
+1-3
Appendix G: Cost and Timeline
Resource
Cost
Time
Kaggle submissions
Free
28 submissions × 5h = 140h compute
RunPod (LoRA training)
~$20
3 training runs × 2-10h
Claude API (multi-agent)
~$200
25+ sessions
Total calendar time
—
40 days (2026-03-06 to 2026-04-15)
Estimated human hours
—
~150h
Appendix H: Dead Ends Catalog (Full Details)
H.1 Prompt Engineering (v22, v27)
Elaborate 3-prompt diversity:
−3 points
(v22=33 vs baseline 36)
Minimal entropy-tuned prompts:
−3 points
(v27=34 vs v24=37)
Lesson:
Prompts are frozen for reasoning-native models. Any change disrupts internal optimization.
H.2 Sampling Parameters (v23, v28)
Multi-temperature [0.5, 0.7, 1.0, 1.3]: neutral (v23=36=baseline)
SC@16 vs SC@8: identical (v28=37=v24)
Lesson:
Diminishing returns beyond K=8 with this model.
H.3 GenSelect — LLM-as-Judge (v29)
Theory: +2-4 points (based on NemoSkills literature)
Reality:
−3 points
(v29=34)
Root cause: GPT-OSS in Harmony format cannot judge solutions via
chat.completions.create()
Lesson:
NemoSkills results (Qwen-based) don't transfer to GPT-OSS architecture.
H.4 Adaptive Complexity Routing (v29)
A0==A1 difficulty classification with multi-track compute allocation
Combined with GenSelect:
−3 total
Lesson:
Complexity introduces regression risk exceeding theoretical gain.
H.5 All Post-Processing Methods (S22)
Code trajectory analysis, code features, early answer filter, number-theoretic properties, Z3 constraints, ECP evidence mining, LR scoring, digit-wise voting, Bayesian selection, CARE SVD
Result:
All reduce to majority on hard problems (30:1 wrong-to-correct ratio is mathematically insurmountable)
H.6 ThinkPRM-1.5B (S22)
1.5B generative PRM, fits alongside 120B (~3GB)
Result:
Inverse scoring on hard problems (wrong=1.0, correct=0.7)
CPU inference: 200-900s per solution (impractical)
H.7 20B Dual-Model Diversity (v46)
Hypothesis: 20B model generates different wrong answers, enabling ensemble
Reality: Same wrong answers as 120B (529, 4949, 0 on test problems)
Score: 4/12 vs 120B 5/12 (regression)
H.8 Prescriptive Prompts Without TIR (v52)
"Solve two ways," "devil's advocate," "approach from number theory"
0/8 gave gold, 0/8 gave new answers
Lesson:
The model either knows or doesn't. No prompt unlocks hidden capability.
H.9 Strategy Diversification (v22, community)
Community analysis recommended this approach; our data contradicts it
v22=−3, v39=−3: every diversity attempt produced a regression