![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F23009828%2F8e917ff52333212a15b90b789c1e5e3a%2Ffig3_error_taxonomy.png?generation=1776779735307120&alt=media\")
Submission details:
\naimo3-v33-fixed Version 2 scored 39 on both leaderboards)We describe a solo competitor's approach to AIMO3 (AI Mathematical Olympiad Progress Prize 3) using GPT-OSS-120B and a multi-agent AI team. Throughout this writeup, \"we\" refers to the author and the AI agent team used for development.
\nOur best score of 39/50 (mean ~34, sigma ~3) came from a single discovery: a Harmony message-format error present in all 19 analyzed public notebooks. Adding the correct ChatGPT identity string and a developer message with code-format rules improved score by +2 points — the only statistically meaningful gain across 28 submissions. The same code (v33-fixed) scored 39, 33, and 34 across three runs, demonstrating the variance in this evaluation.
\nWe document 20+ dead ends with empirical evidence, including the finding that all post-hoc selection methods collapse to majority voting on IMO-hard problems where the per-attempt success rate falls below 50%. We provide variance analysis (sigma ≥ 3), a failure taxonomy of GPT-OSS-120B, and three LoRA fine-tuning experiments on an MXFP4-quantized MoE model served with adapters via vLLM. All prompts, configurations, and negative results are included for reproducibility.
\nThe AI Mathematical Olympiad Progress Prize 3 (AIMO3) challenges participants to solve 50 IMO-level mathematical problems using a single H100 80GB GPU within a 5-hour runtime, with internet disabled and integer answers in [0, 99999]. Scoring uses double-run averaging: each problem is scored 0 or 1 per run, averaged across two independent runs.
\nWe participated as a solo competitor using a multi-agent AI team: Claude Code (Opus) for implementation, Claude Desktop for strategic oversight, Deep Research for literature investigation, Grok for adversarial analysis, and Chrome Extension for community intelligence. This writeup covers 40 days of development across 25+ sessions.
\nOur core model is GPT-OSS-120B, a Mixture-of-Experts model with 116.8B total parameters (5.1B active per token), quantized to MXFP4 (Microscaling FP4) during post-training. We use self-consistency sampling (K=8) with majority voting and Tool-Integrated Reasoning (TIR) via the Harmony message format.
\nKey contributions:
\nGPT-OSS-120B is a Mixture-of-Experts transformer with extreme sparsity:
\nThe MXFP4 quantization is applied during post-training (not post-hoc), meaning the model was trained with this precision in mind. However, the BF16 router weights combined with MXFP4 expert weights create routing non-determinism: 1,000 identical requests at T=0 produce 80 unique completions.
\nGPT-OSS uses a proprietary \"Harmony\" encoding with 6 message carriers:
\n| # | \nCarrier | \nPurpose | \nTunable? | \n
|---|---|---|---|
| 1 | \nSystemContent | \nModel identity and personality | \nFrozen | \n
| 2 | \nReasoningEffort | \nControls thinking depth (low/medium/high) | \nmedium | \n
| 3 | \nToolNamespaceConfig | \nTool definitions (Python interpreter) | \nLoad-bearing | \n
| 4 | \nDeveloperContent | \nDeveloper instructions | \n+2 points | \n
| 5 | \nUserContent | \nProblem statement | \nPer-problem | \n
| 6 | \npreference_prompt | \nOutput format preferences | \nFrozen | \n
Critical discovery (v33, +2 points): The model expects SystemContent to contain the ChatGPT identity string (\"You are ChatGPT, a large language model trained by OpenAI…\"). All 19 analyzed public notebooks either omit this or replace it with a custom identity. Additionally, the DeveloperContent carrier (exposed by the openai-harmony library as .with_instructions()) accepts code-format rules that tripled the model's Python tool efficiency from 42 to 13 calls per problem, as measured in S12 A/B tests.
Catastrophic failure mode (v32, score=0): Using .with_text() instead of .with_instructions() raises AttributeError, caught by a silent try/except, causing all 400 inference attempts to return None.
GPT-OSS produces output in 3 channels: analysis (hidden reasoning), commentary, and final. The host pipeline reads only final for answer extraction and routes recipient=python blocks to Jupyter kernels. Analysis content is appended to conversation history but never inspected by host code. All 19 analyzed competitor notebooks follow this same pattern.
| Type | \nFrequency | \nBehavior | \nFixable? | \n
|---|---|---|---|
| E1: Misunderstanding | \n~54% of errors | \nWrong mathematical formalization | \nLoRA (possibly) | \n
| E2: Arithmetic error | \n~15% | \nCorrect approach, wrong calculation | \nNo (random) | \n
| E3: Thread loss | \n~15% | \nStarts correctly, diverges mid-solution | \nPossibly | \n
| E4: Wrong approach | \n~10% | \nIncorrect strategy from start | \nLoRA (possibly) | \n
| E5: Cannot begin | \n~6% | \nProblem too novel for model | \nNo (ceiling) | \n
Source: Prism Math taxonomy (4601 examples) applied to our 12-problem hard set.
\n
[Problem text] → Harmony encoding → vLLM (GPT-OSS-120B) → TIR loop\n ↓\n Jupyter kernel ← code blocks\n ↓\n \\boxed{answer} extraction\n ↓\n K=8 attempts → majority vote\n| Parameter | \nValue | \nRationale | \n
|---|---|---|
| gpu_memory_utilization | \n0.96 | \nMaximum without OOM | \n
| max_model_len | \n65536 | \nFull context for long solutions | \n
| max_num_seqs | \n256 | \nDefault; reduction to 8 tested and found neutral | \n
| dtype | \nauto (MXFP4) | \nNative model precision | \n
| kv_cache_dtype | \nfp8_e4m3fn | \nSaves VRAM on H100 | \n
| enforce_eager | \nTrue | \nAvoids CUDA graph issues | \n
| port | \n8000 | \nLocal subprocess | \n
jupyter_client\\boxed{(\\d+)} on final channel outputremaining_time / remaining_problems\\boxed{} detectionsubmission.parquet written before ANY imports (crash protection)KAGGLE_IS_COMPETITION_RERUN detection for force-serve modeensure_ascii=True on all json.dump calls (cp1252 encoding protection)The gap between pass@8 (probability that at least one of 8 attempts is correct) and majority@8 (majority vote accuracy) is large: approximately 80% vs. 40% on our 5-problem measured subset. This coverage-versus-pass-rate (CCPR) gap motivated extensive investigation into selection methods. Every method tested failed to close it.
\n| Method | \nResult | \nEvidence | \n
|---|---|---|
| Majority voting (baseline) | \n37/50 | \nv24, v25, v28 | \n
| IEW entropy weighting | \n= majority | \nAUC=0.595 (near-random) | \n
| GenSelect (LLM-as-judge) | \n-3 points | \nv29=34; Harmony format incompatible | \n
| Adaptive routing (difficulty-gated) | \n-3 points | \nv29=34; complexity = regression | \n
| Self-verification (verify-noDev) | \n-3 to -5 | \nFPR=75% (self-confirmation bias) | \n
| Code trajectory analysis | \n= majority | \nSignal vanishes on hard problems | \n
| Z3 constraint verification | \n0/16 distinguish | \nRegex constraints too weak | \n
| ECP evidence mining | \n-4 points | \nFPR=37% | \n
| ThinkPRM-1.5B (process reward model) | \nInverse scoring | \nWrong=1.0, correct=0.7 on hard | \n
| Longest-trace tiebreak | \n+1 (tiebreak only) | \nv44+v45: 6/12 vs majority 5/12 | \n
| DeepConf (13 methods, K=32) | \n= majority | \n0/12 at K=32, all methods | \n
| Minority selection | \n3/3 on hard | \nBut 0/3 on easy — not usable | \n
Chen et al. (NeurIPS 2024) prove that when p(correct) < p(wrong) per attempt, increasing K makes majority voting worse, not better. On our hard problems, the model produces the same wrong answer 6–7 out of 8 times. No post-hoc method can recover correctness from a distribution where the correct answer is in the minority.
\nGeneration quality, not selection quality, is the bottleneck. This overrides our S14–S20 conclusion that \"selection = THE bottleneck.\" On easy problems (pass@1 > 50%), majority voting works. On hard problems (pass@1 < 20%), the model produces the same wrong answer 6–7 out of 8 times — no post-hoc method recovers from that.
\nAll AIMO1/2 winners used fine-tuning (NuminaMath, NemoSkills). NemoSkills gained ~4 points over vanilla baselines through fine-tuning [3]. Public notebook ceiling is 44/50; leaderboard leader achieved 46/50 with 113 submissions.
\nNo published results exist for LoRA fine-tuning of MXFP4-quantized MoE models. Key challenges:
\n--enable-lora serves adapter directly on MXFP4 base (no merge needed)| Run | \nData | \nFormat | \nExamples | \nseq_len | \nLoss | \nResult | \n
|---|---|---|---|---|---|---|
| v1 | \nHarmony-filtered (easy) | \nHarmony | \n987 | \n2048 | \n0.64 | \n5/5 easy, 0/12 hard (= base) | \n
| v2 | \nOMR hard (DeepSeek format) | \nCoT (<think>) | \n2000 | \n2048 | \n0.52 | \n1/12; 5/12 extraction errors | \n
| v3 | \nNemotron-Math-v2 (GPT-OSS traces) | \nHarmony (converted) | \n5000 | \n4096 | \n0.55 | \nBNB4/MXFP4 mismatch: code_execs=0 | \n
Key finding: Format matters more than data quality. v2 trained on hard problems in wrong format (CoT) scored worse than v1 trained on easy problems in correct format (Harmony). v3 had correct format but the BNB4→MXFP4 quantization gap caused the model to lose TIR capability entirely (zero code executions).
\n| Dataset | \nSize | \nFormat | \nUsed | \n
|---|---|---|---|
| nvidia/OpenMathReasoning TIR | \n1.7M | \nTIR traces | \nFiltered to 5000 | \n
| nvidia/Nemotron-Math-v2 | \n3.5M | \nGPT-OSS native | \nConverted 5000 | \n
| Jackrong/Harmony-dataset | \n8471 | \nHarmony format | \nNot used (discovered late) | \n
| 21 HF AIMO3 finetunes | \nvaries | \nUnknown | \nAll have empty model cards | \n
Our v33 submission scored 39, 33, and 34 on identical code across three separate runs (submitted March 24, March 28, and April 7). This 6-point spread (39 vs 33) from the same codebase is the strongest direct evidence of high variance. The mean of our three runs (35.3) is consistent with sigma ~3.
\nNitarach (arXiv 2603.27844) ran 13 identical submissions of the same pipeline and found:
\nHowever, our observation (39 vs 33, delta=6) suggests true sigma may be higher (~3), possibly due to:
\n| Delta | \nInterpretation | \n
|---|---|
| < 3 | \nNoise; do not react | \n
| 3-4 | \nPossibly real; needs 3+ confirmations | \n
| > 4 | \nLikely real improvement | \n
This means our 18-submission history contains one statistically meaningful finding: v33 (Harmony fix) = +2 over v28 baseline of 37, confirmed by the mechanism (format change has clear causal path). All other deltas are within noise.
\nThe leaderboard leader scored 46/50 from 113 entries. Using winner's curse correction (ML estimate), their true mean is likely ~41.8. The public ceiling of 44/50 from unmodified notebooks is also inflated. True ceiling for vanilla GPT-OSS-120B is approximately 36-38/50.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F23009828%2F3f5372dcd902b5038e1b923620324412%2Ffig4_variance.png?generation=1776780061273863&alt=media\")
We analyzed 19 public notebooks scoring 36-44 on the leaderboard. Key findings:
\n| Feature | \nOur v33 | \nTypical competitor | \nBest competitor | \n
|---|---|---|---|
| Identity | \nChatGPT (correct) | \nCustom/omitted | \nCustom/omitted | \n
| DeveloperContent | \nCode-format rules | \nNone | \nNone | \n
| K (attempts) | \n8 | \n8 | \n8-16 | \n
| Temperature | \n1.0 | \n0.6-1.0 | \n0.6 | \n
| gpu_memory_util | \n0.96 | \n0.90 | \n0.90 | \n
| max_num_seqs | \n256 | \n6-8 | \n64 | \n
| Selection | \nmajority + longest | \nmajority | \nmajority | \n
| min_p | \n0.02 | \n0.02-0.05 | \n0.05 | \n
| Method | \nScore | \nNotes | \n
|---|---|---|
| Public baseline (kurianbenoy fork) | \n36 | \nUnmodified | \n
| Best public notebook (ans-verifys) | \n44 | \nLucky run; likely mean ~38-40 | \n
| Leaderboard leader (ippeiogawa) | \n46 | \n113 entries; true mean ~41.8 | \n
| Our v33 (final submission) | \n39 public / 39 private | \n3-run variance (public): 39, 33, 34 — mean 35.3 | \n
| AIMO2 winner (NemoSkills, 14B fine-tuned) | \n34/50 | \nAIMO2, not directly comparable (4x L4 hardware) | \n
| DeepSeek-R1 (671B, if available) | \nN/A | \nNot applicable (too large) | \n
| Agent | \nModel | \nRole | \n
|---|---|---|
| KK (Claude Code) | \nClaude Opus | \nImplementation, code audit, testing, memory | \n
| KD (Claude Desktop) | \nClaude Opus | \nStrategy, Deep Research, coordination | \n
| DR (Deep Research) | \nClaude Opus | \nLiterature investigation (86 reports) | \n
| GK (Grok) | \ngrok-4.20 | \nAdversarial analysis, RunPod SSH | \n
| KH (Chrome Extension) | \nClaude Sonnet | \nCommunity intelligence, leaderboard | \n
theory/ (86 .md files), prompts/ (45 files), logs/kk-sessions/.claude-memory-mirror/ (5 files: rules, dead ends, key files, bugs, verified)journal_aimo3.txt (prepend-only, reverse chronological)CLAUDE.md with mandatory reads and theory-check protocolWe tested 20+ approaches beyond the baseline. None improved the score. The full catalog is in Appendix H; here we summarize the key categories:
\n| Category | \nMethods Tested | \nBest Result | \nLesson | \n
|---|---|---|---|
| Prompt engineering | \n3-prompt diversity, minimal prompts | \n−3 each | \nPrompts are frozen for reasoning-native models | \n
| Sampling | \nMulti-temp, SC@16 | \n0, 0 | \nDiminishing returns beyond K=8 | \n
| Selection (LLM-as-judge) | \nGenSelect via chat API | \n−3 | \nHarmony format incompatible with judging | \n
| Selection (statistical) | \nIEW, DeepConf (13 methods), LR scoring, Bayesian, CARE | \n= majority | \nAll collapse to majority on hard problems | \n
| Selection (code analysis) | \nTrajectory, Z3, ECP mining | \n0 to −4 | \nSignal vanishes on hard | \n
| Verification | \nSelf-verify, ThinkPRM-1.5B | \n−3 to −5, inverse | \nSelf-confirmation bias; PRM inverts on hard | \n
| Model diversity | \n20B dual-model | \n−1 | \nSame wrong answers as 120B | \n
| Prompt diversification | \nPrescriptive, \"solve two ways\" | \n0 | \nModel either knows or doesn't | \n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F23009828%2F4ab94f0ffd6b704aebe83024e63b2773%2Ffig2_ablation.png?generation=1776779929510625&alt=media\")
Read competitor code on day 1. We didn't. That cost 20 days. One afternoon of notebook forensics produced the only +2 gain across 28 submissions. 86 deep-research reports produced zero code changes.
\nMeasure before theorizing. Three independent agents converged on \"extraction is the bottleneck.\" First measurement: extraction accuracy = 100%. One number killed an entire research track.
\nSubtraction beats addition. Every feature we added scored zero or negative. The only improvement came from fixing a bug (Harmony format) and removing harmful features (IEW, GenSelect). Simple majority vote at 37/50 beat our most elaborate pipeline at 34/50.
\nTen submissions went to noise. With sigma ≥ 3, the gap between 33 and 39 is one standard deviation. We needed: delta > 4 plus a mechanistic explanation before reacting. We didn't have that rule until submission 17.
\nOff-by-one is systematic, not random. 44% of wrong answers differ from gold by ±1 or ±2 — the same boundary confusion (inclusive vs. exclusive ranges, base cases) across different problems. This is a training data artifact, not a pipeline issue.
\nThe optimal strategy for a solo competitor, in hindsight, would be:
\nWe instead spent 20 days on prompt engineering and selection methods before reading competitor code.
\nAll code is contained in a single Jupyter notebook. All prompts are frozen and documented in Appendix A. The only external dependency beyond standard Kaggle environment is the GPT-OSS-120B model and vLLM wheels, both available as Kaggle datasets.
\nFor Kaggle-agnostic reproducibility, a Google Colab version of the full pipeline is provided: aimo3-v33-colab. It installs the openai-harmony library and vLLM from public PyPI, downloads the gpt-oss-120b weights from Hugging Face, and exposes the same inference harness used in the Kaggle submission (with the kaggle_evaluation gateway import guarded by try/except ImportError so the notebook runs in any environment with a single H100-class GPU).
[1] Nitarach. \"Variance Analysis of AIMO3 Submissions.\" arXiv:2603.27844, 2026.\n[2] Chen et al. \"When Majority Voting Fails.\" NeurIPS, 2024.\n[3] NVIDIA NemoSkills team. \"NemoSkills: AIMO2 Winning Solution.\" arXiv:2504.16891, 2025.\n[4] Prism Math. \"Error Taxonomy for Mathematical Reasoning.\" 4,601 examples, 2025.\n[5] Ai. \"On Correlated Voting.\" Corollary 2 (Dawid-Skene limitation), 2025.
\nAll prompts are copied exactly from aimo3-v33-fixed.ipynb (CFG class). These produced 39/50. Any modification causes regression (confirmed v22, v27).
You are a world-class International Mathematical Olympiad (IMO) competitor.\nThe final answer must be a non-negative integer between 0 and 99999.\nYou must place the final integer answer inside \\boxed{}.\nInjected into DeveloperContent carrier (not SystemContent). SystemContent carries model identity (A.6).
\nUse this tool to execute Python code.\nThe environment is a stateful Jupyter notebook.\nYou must use print() to output results.\nWARNING: Removing this caused +9pp execution errors (v27).
\n# PROBLEM-SOLVING PROTOCOL\n\ndef solve(problem: str) -> int:\n problem_type = classify(problem)\n answer_analytical = solve_analytically(problem, problem_type)\n answer_numerical = verify_with_python(problem, answer_analytical)\n if answer_analytical != answer_numerical:\n answer_analytical = answer_numerical # trust Python over reasoning\n return answer_analytical\n\ndef solve_analytically(problem, problem_type):\n method_1 = pick_method(problem_type)\n result = attempt(method_1)\n if result is None:\n method_2 = pick_different_method(problem_type, exclude=method_1)\n result = attempt(method_2)\n return result\n\ndef verify_with_python(problem, claimed_answer):\n # ALWAYS run. Use INDEPENDENT method from analytical.\n # counting -> enumerate and count\n # optimization -> scipy.optimize or brute force\n # equations -> sympy.solve\n # NEVER recompute same formula\n return execute(write_verification_code(problem))\n\ndef on_python_error(code, error):\n # NEVER debug. ALWAYS delete and rewrite from scratch.\n return write_completely_new_code()\n\n# HARD RULES:\n# - Final answer inside \\\\boxed{}\n# - Answer: non-negative integer 0-99999\n# - Use math, numpy, sympy, scipy, itertools, collections\n# - If two methods disagree, run THIRD to break tie\nInjected via DeveloperContent.new().with_instructions(). Tripled Python tool efficiency: 42 to 13 code calls per problem (S12 A/B test).
Use `math`, `numpy`,`sympy`,`itertools` and `collections` to solve the problem.\nEach problem sent as single UserContent message with raw problem text. No framing, no few-shot, no CoT instructions.
\nSystemContent.new()\n .with_model_identity('You are ChatGPT, a large language model trained by OpenAI.')\n .with_reasoning_effort(reasoning_effort=ReasoningEffort.HIGH)\n .with_tools(tool_config)\nThis is the +2 discovery. All 19 competitor notebooks omit or replace this identity. The openai-harmony library does NOT auto-inject it.
messages = [\n Message.from_role_and_content(Role.SYSTEM, system_content), # Identity + tools\n Message.from_role_and_content(Role.DEVELOPER, developer_content), # Instructions\n Message.from_role_and_content(Role.USER, problem_text) # Problem\n]\nThree messages total. No conversation history between problems.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F23009828%2Fe92268266332e5124bc6f2e1846937b9%2Ffig1_score_history.png?generation=1776780256053758&alt=media\")
| # | \nDate | \nVersion | \nScore | \nDelta | \nKey Change | \n
|---|---|---|---|---|---|
| 1 | \n2026-03-06 | \nDeepSeek-14B v6 | \n0 | \n— | \nFirst submit (RERUN detection broken) | \n
| 2 | \n2026-03-07 | \nQwQ-32B v7 | \n0 | \n0 | \nInfrastructure works, model too weak | \n
| 3 | \n2026-03-11 | \nGPT-OSS v19 | \n0 | \n0 | \n6 root-cause fixes, still broken | \n
| 4 | \n2026-03-12 | \nFork v3 (baseline) | \n36 | \n+36 | \nUnmodified kurianbenoy fork | \n
| 5 | \n2026-03-13 | \nv22 | \n33 | \n−3 | \n3-prompt diversity | \n
| 6 | \n2026-03-14 | \nv23 | \n36 | \n+3 | \nMinimal prompt + multi-temp | \n
| 7 | \n2026-03-15 | \nv24 | \n37 | \n+1 | \nbatch=256, gpu=0.96, ctx=65536 | \n
| 8 | \n2026-03-16 | \nv25 | \n37 | \n0 | \nVariance confirmation | \n
| 9 | \n2026-03-18 | \nv28 | \n37 | \n0 | \nSC@16 (= SC@8) | \n
| 10 | \n2026-03-19 | \nv29 | \n34 | \n−3 | \nGenSelect + adaptive | \n
| 11 | \n2026-03-20 | \nv30 | \n36 | \n−1 | \nSubtraction (K=8, timeout=10) | \n
| 12 | \n2026-03-20 | \nv31 | \n34 | \n−3 | \nAlign with 42-scorer | \n
| 13 | \n2026-03-22 | \nv32 | \n0 | \n−37 | \nBUG: .with_text() AttributeError — all 400 attempts returned None | \n
| 14 | \n2026-03-24 | \nv33 | \n39 | \n+2 | \nHarmony fix + developer message | \n
| 15 | \n2026-03-24 | \nv34 | \n34 | \n−5 | \nPROVE filter + seqs=8 | \n
| 16 | \n2026-03-26 | \nv37b | \n31 | \n−8 | \nExperimental features | \n
| 17 | \n2026-03-28 | \nv33 resub #1 | \n33 | \n−6 | \nSame code as row 14; confirms variance | \n
| 18 | \n~2026-03-31 | \nv35-verify-fresh | \n34 | \n−5 | \nSelf-verification (FPR=75%) | \n
| 19 | \n~2026-04-01 | \nv31-majority-clean | \n0 | \n−39 | \nBug: clean majority rewrite failed | \n
| 20 | \n~2026-04-02 | \ngptoss-fork-v1.2 | \n34 | \n−5 | \nFork variant | \n
| 21 | \n~2026-04-03 | \nv54-mining-longest v1 | \n0 | \n−39 | \nCode mining extraction (broken) | \n
| 22 | \n~2026-04-04 | \nv54-mining-longest v2 | \nError | \n— | \nKaggle Error | \n
| 23 | \n~2026-04-04 | \nv54-code-mining+longest | \n32 | \n−7 | \nCode mining + longest tiebreak | \n
| 24 | \n~2026-04-05 | \nv55-eagle3 v2 | \n1 | \n−38 | \nEAGLE3 speculative decoding (VRAM crash) | \n
| 25 | \n~2026-04-07 | \nv33-fixed resub #2 | \n34 | \n−5 | \nSame code as row 14; third run | \n
| 26 | \n~2026-04-08 | \nv56-k10-longest | \n36 | \n−3 | \nK=10 + longest tiebreak | \n
| 27 | \n~2026-04-10 | \nv57-k12-longest | \n32 | \n−7 | \nK=12 + longest (over-sampling) | \n
| 28 | \n~2026-04-11 | \nv58-dsq-lora v3 | \n23 | \n−16 | \nLoRA adapter (BNB4/MXFP4 mismatch) | \n
| — | \n2026-04-15 | \nv33-fixed (final) | \n34 | \n— | \nSelected as final submission | \n
python -m vllm.entrypoints.openai.api_server \\\n --seed 42 \\\n --model /kaggle/tmp/setup/model \\\n --served-model-name gpt-oss \\\n --tensor-parallel-size 1 \\\n --max-num-seqs 256 \\\n --gpu-memory-utilization 0.96 \\\n --host 0.0.0.0 \\\n --port 8000 \\\n --dtype auto \\\n --kv-cache-dtype fp8_e4m3 \\\n --max-model-len 65536 \\\n --stream-interval 200 \\\n --max-cudagraph-capture-size 64 \\\n --async-scheduling \\\n --disable-log-stats \\\n --enable-prefix-caching\n| Type | \nSource ID | \nDescription | \n
|---|---|---|
| competition | \n118448 | \naimo-3 competition data | \n
| modelInstanceVersion | \n510391 (model 422384, instance 404485) | \ndanielhanchen/gpt-oss-120b | \n
| kernelVersion | \n289055161 | \naimo-3-utils (vLLM 0.18.1 wheels + tiktoken encodings) | \n
pow(seed + attempt_index, 2) % (2**31)| Function | \nCell | \nPurpose | \n
|---|---|---|
CFG (class) | \n9 | \nAll hyperparameters and prompts | \n
AIMO3Template.apply_chat_template() | \n11 | \nHarmony message encoding | \n
AIMO3Sandbox | \n12 | \nJupyter kernel pool management | \n
AIMO3Tool | \n13 | \nTIR code execution loop | \n
AIMO3Solver.__init__() | \n14 | \nvLLM server startup + model preload | \n
AIMO3Solver._start_server() | \n14 | \nvLLM subprocess with exact CLI args | \n
predict() | \n17 | \nMain entry: K=8 attempts + majority vote | \n
| Pre-flight check | \n16 | \nDeveloper message rendering verification | \n
All code in single notebook: aimo3-v33-fixed.ipynb (19 cells).
| Problem | \nGold | \nMajority | \nGold − Majority | \nError type | \n
|---|---|---|---|---|
| olymp-01629 | \n4950 | \n4949 | \n+1 | \nBoundary inclusive/exclusive | \n
| oda-00425 | \n3 | \n4 | \n−1 | \nCount boundary | \n
| dapo-15481 | \n2 | \n0 | \n+2 | \nBase case | \n
| oda-01612 | \n4 | \n2 | \n+2 | \nMultiplicity | \n
| olymp-02392 | \n49 | \n51 | \n−2 | \nRange boundary | \n
| dapo-10920 | \n4022 | \n4024 | \n−2 | \nCoefficient error | \n
| oda-01183 | \n12 | \n6 | \n+6 (×2) | \nFactor omission | \n
7/16 (44%) of hard-problem errors are within ±2 of the correct answer.
\n<think> CoT format (DeepSeek-R1 style)<think> format, but inference uses Harmony. Outputs malformed at inference time.| Experiment | \nWhy It Matters | \nWhy Not Run | \nEstimated Impact | \n
|---|---|---|---|
| Multi-run per problem (12+) | \nPer-problem variance | \nMax 3 runs available | \nDiagnostic | \n
| Reasoning channel analysis | \nHidden confidence signals | \nPipeline not instrumented | \nUnknown | \n
| PRM scoring (7B) | \nStep-level selection | \nVRAM blocked | \n+2-4 (literature) | \n
| CodeBERT clustering | \nApproach diversity metric | \nNot prioritized | \nUnknown | \n
| Adaptive K (K=16-32 on hard) | \nBudget 89% slack | \nNot tested in time | \n+1-2 | \n
| LoRA on boundary errors | \n44% of errors = off-by-one | \nData not curated | \n+1-3 | \n
| Resource | \nCost | \nTime | \n
|---|---|---|
| Kaggle submissions | \nFree | \n28 submissions × 5h = 140h compute | \n
| RunPod (LoRA training) | \n~$20 | \n3 training runs × 2-10h | \n
| Claude API (multi-agent) | \n~$200 | \n25+ sessions | \n
| Total calendar time | \n— | \n40 days (2026-03-06 to 2026-04-15) | \n
| Estimated human hours | \n— | \n~150h | \n
chat.completions.create()Very detailed and honest write-up. \nEspecially appreciated that you included the failed experiments instead of only the successful parts.
", "votes": 0, "replies": [] }, { "id": 3457782, "author_name": "Philip Vonderlind", "author_url": "", "post_date": "2026-05-14T10:28:25.033000", "content": "Adding to the comments from @sambealing below, I have a few additional questions:
\nArithmetic Errors: You mention that arithmetic errors are not fixable. However, I would argue that they can be addressed through tool calls or formal verification systems such as Lean. If the model has produced a valid reasoning path that is mathematically sound, would you agree that it should be possible to translate it into Python or Lean code for verification? Did you experience any counter-examples in practice?
Answer Recovery: You use a fairly simple regex expression to extract multiple digits from the model's answer. Depending on the implementation, this can fail in several cases. For example, if the model produces an equation, fraction, or additional text inside the boxed environment, the correct answer may not be recovered. Did you observe or document cases where the model produced the correct answer, but the extraction failed, and the problem was therefore marked as unsolved? If so, how would you adapt the extraction logic to recover such cases?
Bottleneck: You claim that “generation quality, not selection quality, is the bottleneck.” However, other contestants argue the opposite, for example @natnitarach in this submission. They argue that selection is the bottleneck, motivated by the publicly available pass@100 statistics on both the public and private sets here. What are your thoughts on this interpretation?
Reproducibility: Is the code needed to reproduce all experiments and notebook configurations available? If so, please share it to improve the reproducibility of your work.
Hi, thank you for your submission to the Writeup Prize. @philipvonderlind and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.
\nLook forward to reading your responses on these.
", "votes": 0, "replies": [] }, { "id": 3446696, "author_name": "", "author_url": "", "post_date": "2026-04-22T00:53:29.913000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 4, "messages": [ { "id": 3446459, "postDate": "2026-04-21T16:46:48.007Z", "content": "Submission details:
\naimo3-v33-fixed Version 2 scored 39 on both leaderboards)We describe a solo competitor's approach to AIMO3 (AI Mathematical Olympiad Progress Prize 3) using GPT-OSS-120B and a multi-agent AI team. Throughout this writeup, \"we\" refers to the author and the AI agent team used for development.
\nOur best score of 39/50 (mean ~34, sigma ~3) came from a single discovery: a Harmony message-format error present in all 19 analyzed public notebooks. Adding the correct ChatGPT identity string and a developer message with code-format rules improved score by +2 points — the only statistically meaningful gain across 28 submissions. The same code (v33-fixed) scored 39, 33, and 34 across three runs, demonstrating the variance in this evaluation.
\nWe document 20+ dead ends with empirical evidence, including the finding that all post-hoc selection methods collapse to majority voting on IMO-hard problems where the per-attempt success rate falls below 50%. We provide variance analysis (sigma ≥ 3), a failure taxonomy of GPT-OSS-120B, and three LoRA fine-tuning experiments on an MXFP4-quantized MoE model served with adapters via vLLM. All prompts, configurations, and negative results are included for reproducibility.
\nThe AI Mathematical Olympiad Progress Prize 3 (AIMO3) challenges participants to solve 50 IMO-level mathematical problems using a single H100 80GB GPU within a 5-hour runtime, with internet disabled and integer answers in [0, 99999]. Scoring uses double-run averaging: each problem is scored 0 or 1 per run, averaged across two independent runs.
\nWe participated as a solo competitor using a multi-agent AI team: Claude Code (Opus) for implementation, Claude Desktop for strategic oversight, Deep Research for literature investigation, Grok for adversarial analysis, and Chrome Extension for community intelligence. This writeup covers 40 days of development across 25+ sessions.
\nOur core model is GPT-OSS-120B, a Mixture-of-Experts model with 116.8B total parameters (5.1B active per token), quantized to MXFP4 (Microscaling FP4) during post-training. We use self-consistency sampling (K=8) with majority voting and Tool-Integrated Reasoning (TIR) via the Harmony message format.
\nKey contributions:
\nGPT-OSS-120B is a Mixture-of-Experts transformer with extreme sparsity:
\nThe MXFP4 quantization is applied during post-training (not post-hoc), meaning the model was trained with this precision in mind. However, the BF16 router weights combined with MXFP4 expert weights create routing non-determinism: 1,000 identical requests at T=0 produce 80 unique completions.
\nGPT-OSS uses a proprietary \"Harmony\" encoding with 6 message carriers:
\n| # | \nCarrier | \nPurpose | \nTunable? | \n
|---|---|---|---|
| 1 | \nSystemContent | \nModel identity and personality | \nFrozen | \n
| 2 | \nReasoningEffort | \nControls thinking depth (low/medium/high) | \nmedium | \n
| 3 | \nToolNamespaceConfig | \nTool definitions (Python interpreter) | \nLoad-bearing | \n
| 4 | \nDeveloperContent | \nDeveloper instructions | \n+2 points | \n
| 5 | \nUserContent | \nProblem statement | \nPer-problem | \n
| 6 | \npreference_prompt | \nOutput format preferences | \nFrozen | \n
Critical discovery (v33, +2 points): The model expects SystemContent to contain the ChatGPT identity string (\"You are ChatGPT, a large language model trained by OpenAI…\"). All 19 analyzed public notebooks either omit this or replace it with a custom identity. Additionally, the DeveloperContent carrier (exposed by the openai-harmony library as .with_instructions()) accepts code-format rules that tripled the model's Python tool efficiency from 42 to 13 calls per problem, as measured in S12 A/B tests.
Catastrophic failure mode (v32, score=0): Using .with_text() instead of .with_instructions() raises AttributeError, caught by a silent try/except, causing all 400 inference attempts to return None.
GPT-OSS produces output in 3 channels: analysis (hidden reasoning), commentary, and final. The host pipeline reads only final for answer extraction and routes recipient=python blocks to Jupyter kernels. Analysis content is appended to conversation history but never inspected by host code. All 19 analyzed competitor notebooks follow this same pattern.
| Type | \nFrequency | \nBehavior | \nFixable? | \n
|---|---|---|---|
| E1: Misunderstanding | \n~54% of errors | \nWrong mathematical formalization | \nLoRA (possibly) | \n
| E2: Arithmetic error | \n~15% | \nCorrect approach, wrong calculation | \nNo (random) | \n
| E3: Thread loss | \n~15% | \nStarts correctly, diverges mid-solution | \nPossibly | \n
| E4: Wrong approach | \n~10% | \nIncorrect strategy from start | \nLoRA (possibly) | \n
| E5: Cannot begin | \n~6% | \nProblem too novel for model | \nNo (ceiling) | \n
Source: Prism Math taxonomy (4601 examples) applied to our 12-problem hard set.
\n
[Problem text] → Harmony encoding → vLLM (GPT-OSS-120B) → TIR loop\n ↓\n Jupyter kernel ← code blocks\n ↓\n \\boxed{answer} extraction\n ↓\n K=8 attempts → majority vote\n| Parameter | \nValue | \nRationale | \n
|---|---|---|
| gpu_memory_utilization | \n0.96 | \nMaximum without OOM | \n
| max_model_len | \n65536 | \nFull context for long solutions | \n
| max_num_seqs | \n256 | \nDefault; reduction to 8 tested and found neutral | \n
| dtype | \nauto (MXFP4) | \nNative model precision | \n
| kv_cache_dtype | \nfp8_e4m3fn | \nSaves VRAM on H100 | \n
| enforce_eager | \nTrue | \nAvoids CUDA graph issues | \n
| port | \n8000 | \nLocal subprocess | \n
jupyter_client\\boxed{(\\d+)} on final channel outputremaining_time / remaining_problems\\boxed{} detectionsubmission.parquet written before ANY imports (crash protection)KAGGLE_IS_COMPETITION_RERUN detection for force-serve modeensure_ascii=True on all json.dump calls (cp1252 encoding protection)The gap between pass@8 (probability that at least one of 8 attempts is correct) and majority@8 (majority vote accuracy) is large: approximately 80% vs. 40% on our 5-problem measured subset. This coverage-versus-pass-rate (CCPR) gap motivated extensive investigation into selection methods. Every method tested failed to close it.
\n| Method | \nResult | \nEvidence | \n
|---|---|---|
| Majority voting (baseline) | \n37/50 | \nv24, v25, v28 | \n
| IEW entropy weighting | \n= majority | \nAUC=0.595 (near-random) | \n
| GenSelect (LLM-as-judge) | \n-3 points | \nv29=34; Harmony format incompatible | \n
| Adaptive routing (difficulty-gated) | \n-3 points | \nv29=34; complexity = regression | \n
| Self-verification (verify-noDev) | \n-3 to -5 | \nFPR=75% (self-confirmation bias) | \n
| Code trajectory analysis | \n= majority | \nSignal vanishes on hard problems | \n
| Z3 constraint verification | \n0/16 distinguish | \nRegex constraints too weak | \n
| ECP evidence mining | \n-4 points | \nFPR=37% | \n
| ThinkPRM-1.5B (process reward model) | \nInverse scoring | \nWrong=1.0, correct=0.7 on hard | \n
| Longest-trace tiebreak | \n+1 (tiebreak only) | \nv44+v45: 6/12 vs majority 5/12 | \n
| DeepConf (13 methods, K=32) | \n= majority | \n0/12 at K=32, all methods | \n
| Minority selection | \n3/3 on hard | \nBut 0/3 on easy — not usable | \n
Chen et al. (NeurIPS 2024) prove that when p(correct) < p(wrong) per attempt, increasing K makes majority voting worse, not better. On our hard problems, the model produces the same wrong answer 6–7 out of 8 times. No post-hoc method can recover correctness from a distribution where the correct answer is in the minority.
\nGeneration quality, not selection quality, is the bottleneck. This overrides our S14–S20 conclusion that \"selection = THE bottleneck.\" On easy problems (pass@1 > 50%), majority voting works. On hard problems (pass@1 < 20%), the model produces the same wrong answer 6–7 out of 8 times — no post-hoc method recovers from that.
\nAll AIMO1/2 winners used fine-tuning (NuminaMath, NemoSkills). NemoSkills gained ~4 points over vanilla baselines through fine-tuning [3]. Public notebook ceiling is 44/50; leaderboard leader achieved 46/50 with 113 submissions.
\nNo published results exist for LoRA fine-tuning of MXFP4-quantized MoE models. Key challenges:
\n--enable-lora serves adapter directly on MXFP4 base (no merge needed)| Run | \nData | \nFormat | \nExamples | \nseq_len | \nLoss | \nResult | \n
|---|---|---|---|---|---|---|
| v1 | \nHarmony-filtered (easy) | \nHarmony | \n987 | \n2048 | \n0.64 | \n5/5 easy, 0/12 hard (= base) | \n
| v2 | \nOMR hard (DeepSeek format) | \nCoT (<think>) | \n2000 | \n2048 | \n0.52 | \n1/12; 5/12 extraction errors | \n
| v3 | \nNemotron-Math-v2 (GPT-OSS traces) | \nHarmony (converted) | \n5000 | \n4096 | \n0.55 | \nBNB4/MXFP4 mismatch: code_execs=0 | \n
Key finding: Format matters more than data quality. v2 trained on hard problems in wrong format (CoT) scored worse than v1 trained on easy problems in correct format (Harmony). v3 had correct format but the BNB4→MXFP4 quantization gap caused the model to lose TIR capability entirely (zero code executions).
\n| Dataset | \nSize | \nFormat | \nUsed | \n
|---|---|---|---|
| nvidia/OpenMathReasoning TIR | \n1.7M | \nTIR traces | \nFiltered to 5000 | \n
| nvidia/Nemotron-Math-v2 | \n3.5M | \nGPT-OSS native | \nConverted 5000 | \n
| Jackrong/Harmony-dataset | \n8471 | \nHarmony format | \nNot used (discovered late) | \n
| 21 HF AIMO3 finetunes | \nvaries | \nUnknown | \nAll have empty model cards | \n
Our v33 submission scored 39, 33, and 34 on identical code across three separate runs (submitted March 24, March 28, and April 7). This 6-point spread (39 vs 33) from the same codebase is the strongest direct evidence of high variance. The mean of our three runs (35.3) is consistent with sigma ~3.
\nNitarach (arXiv 2603.27844) ran 13 identical submissions of the same pipeline and found:
\nHowever, our observation (39 vs 33, delta=6) suggests true sigma may be higher (~3), possibly due to:
\n| Delta | \nInterpretation | \n
|---|---|
| < 3 | \nNoise; do not react | \n
| 3-4 | \nPossibly real; needs 3+ confirmations | \n
| > 4 | \nLikely real improvement | \n
This means our 18-submission history contains one statistically meaningful finding: v33 (Harmony fix) = +2 over v28 baseline of 37, confirmed by the mechanism (format change has clear causal path). All other deltas are within noise.
\nThe leaderboard leader scored 46/50 from 113 entries. Using winner's curse correction (ML estimate), their true mean is likely ~41.8. The public ceiling of 44/50 from unmodified notebooks is also inflated. True ceiling for vanilla GPT-OSS-120B is approximately 36-38/50.
\n
We analyzed 19 public notebooks scoring 36-44 on the leaderboard. Key findings:
\n| Feature | \nOur v33 | \nTypical competitor | \nBest competitor | \n
|---|---|---|---|
| Identity | \nChatGPT (correct) | \nCustom/omitted | \nCustom/omitted | \n
| DeveloperContent | \nCode-format rules | \nNone | \nNone | \n
| K (attempts) | \n8 | \n8 | \n8-16 | \n
| Temperature | \n1.0 | \n0.6-1.0 | \n0.6 | \n
| gpu_memory_util | \n0.96 | \n0.90 | \n0.90 | \n
| max_num_seqs | \n256 | \n6-8 | \n64 | \n
| Selection | \nmajority + longest | \nmajority | \nmajority | \n
| min_p | \n0.02 | \n0.02-0.05 | \n0.05 | \n
| Method | \nScore | \nNotes | \n
|---|---|---|
| Public baseline (kurianbenoy fork) | \n36 | \nUnmodified | \n
| Best public notebook (ans-verifys) | \n44 | \nLucky run; likely mean ~38-40 | \n
| Leaderboard leader (ippeiogawa) | \n46 | \n113 entries; true mean ~41.8 | \n
| Our v33 (final submission) | \n39 public / 39 private | \n3-run variance (public): 39, 33, 34 — mean 35.3 | \n
| AIMO2 winner (NemoSkills, 14B fine-tuned) | \n34/50 | \nAIMO2, not directly comparable (4x L4 hardware) | \n
| DeepSeek-R1 (671B, if available) | \nN/A | \nNot applicable (too large) | \n
| Agent | \nModel | \nRole | \n
|---|---|---|
| KK (Claude Code) | \nClaude Opus | \nImplementation, code audit, testing, memory | \n
| KD (Claude Desktop) | \nClaude Opus | \nStrategy, Deep Research, coordination | \n
| DR (Deep Research) | \nClaude Opus | \nLiterature investigation (86 reports) | \n
| GK (Grok) | \ngrok-4.20 | \nAdversarial analysis, RunPod SSH | \n
| KH (Chrome Extension) | \nClaude Sonnet | \nCommunity intelligence, leaderboard | \n
theory/ (86 .md files), prompts/ (45 files), logs/kk-sessions/.claude-memory-mirror/ (5 files: rules, dead ends, key files, bugs, verified)journal_aimo3.txt (prepend-only, reverse chronological)CLAUDE.md with mandatory reads and theory-check protocolWe tested 20+ approaches beyond the baseline. None improved the score. The full catalog is in Appendix H; here we summarize the key categories:
\n| Category | \nMethods Tested | \nBest Result | \nLesson | \n
|---|---|---|---|
| Prompt engineering | \n3-prompt diversity, minimal prompts | \n−3 each | \nPrompts are frozen for reasoning-native models | \n
| Sampling | \nMulti-temp, SC@16 | \n0, 0 | \nDiminishing returns beyond K=8 | \n
| Selection (LLM-as-judge) | \nGenSelect via chat API | \n−3 | \nHarmony format incompatible with judging | \n
| Selection (statistical) | \nIEW, DeepConf (13 methods), LR scoring, Bayesian, CARE | \n= majority | \nAll collapse to majority on hard problems | \n
| Selection (code analysis) | \nTrajectory, Z3, ECP mining | \n0 to −4 | \nSignal vanishes on hard | \n
| Verification | \nSelf-verify, ThinkPRM-1.5B | \n−3 to −5, inverse | \nSelf-confirmation bias; PRM inverts on hard | \n
| Model diversity | \n20B dual-model | \n−1 | \nSame wrong answers as 120B | \n
| Prompt diversification | \nPrescriptive, \"solve two ways\" | \n0 | \nModel either knows or doesn't | \n
Read competitor code on day 1. We didn't. That cost 20 days. One afternoon of notebook forensics produced the only +2 gain across 28 submissions. 86 deep-research reports produced zero code changes.
\nMeasure before theorizing. Three independent agents converged on \"extraction is the bottleneck.\" First measurement: extraction accuracy = 100%. One number killed an entire research track.
\nSubtraction beats addition. Every feature we added scored zero or negative. The only improvement came from fixing a bug (Harmony format) and removing harmful features (IEW, GenSelect). Simple majority vote at 37/50 beat our most elaborate pipeline at 34/50.
\nTen submissions went to noise. With sigma ≥ 3, the gap between 33 and 39 is one standard deviation. We needed: delta > 4 plus a mechanistic explanation before reacting. We didn't have that rule until submission 17.
\nOff-by-one is systematic, not random. 44% of wrong answers differ from gold by ±1 or ±2 — the same boundary confusion (inclusive vs. exclusive ranges, base cases) across different problems. This is a training data artifact, not a pipeline issue.
\nThe optimal strategy for a solo competitor, in hindsight, would be:
\nWe instead spent 20 days on prompt engineering and selection methods before reading competitor code.
\nAll code is contained in a single Jupyter notebook. All prompts are frozen and documented in Appendix A. The only external dependency beyond standard Kaggle environment is the GPT-OSS-120B model and vLLM wheels, both available as Kaggle datasets.
\nFor Kaggle-agnostic reproducibility, a Google Colab version of the full pipeline is provided: aimo3-v33-colab. It installs the openai-harmony library and vLLM from public PyPI, downloads the gpt-oss-120b weights from Hugging Face, and exposes the same inference harness used in the Kaggle submission (with the kaggle_evaluation gateway import guarded by try/except ImportError so the notebook runs in any environment with a single H100-class GPU).
[1] Nitarach. \"Variance Analysis of AIMO3 Submissions.\" arXiv:2603.27844, 2026.\n[2] Chen et al. \"When Majority Voting Fails.\" NeurIPS, 2024.\n[3] NVIDIA NemoSkills team. \"NemoSkills: AIMO2 Winning Solution.\" arXiv:2504.16891, 2025.\n[4] Prism Math. \"Error Taxonomy for Mathematical Reasoning.\" 4,601 examples, 2025.\n[5] Ai. \"On Correlated Voting.\" Corollary 2 (Dawid-Skene limitation), 2025.
\nAll prompts are copied exactly from aimo3-v33-fixed.ipynb (CFG class). These produced 39/50. Any modification causes regression (confirmed v22, v27).
You are a world-class International Mathematical Olympiad (IMO) competitor.\nThe final answer must be a non-negative integer between 0 and 99999.\nYou must place the final integer answer inside \\boxed{}.\nInjected into DeveloperContent carrier (not SystemContent). SystemContent carries model identity (A.6).
\nUse this tool to execute Python code.\nThe environment is a stateful Jupyter notebook.\nYou must use print() to output results.\nWARNING: Removing this caused +9pp execution errors (v27).
\n# PROBLEM-SOLVING PROTOCOL\n\ndef solve(problem: str) -> int:\n problem_type = classify(problem)\n answer_analytical = solve_analytically(problem, problem_type)\n answer_numerical = verify_with_python(problem, answer_analytical)\n if answer_analytical != answer_numerical:\n answer_analytical = answer_numerical # trust Python over reasoning\n return answer_analytical\n\ndef solve_analytically(problem, problem_type):\n method_1 = pick_method(problem_type)\n result = attempt(method_1)\n if result is None:\n method_2 = pick_different_method(problem_type, exclude=method_1)\n result = attempt(method_2)\n return result\n\ndef verify_with_python(problem, claimed_answer):\n # ALWAYS run. Use INDEPENDENT method from analytical.\n # counting -> enumerate and count\n # optimization -> scipy.optimize or brute force\n # equations -> sympy.solve\n # NEVER recompute same formula\n return execute(write_verification_code(problem))\n\ndef on_python_error(code, error):\n # NEVER debug. ALWAYS delete and rewrite from scratch.\n return write_completely_new_code()\n\n# HARD RULES:\n# - Final answer inside \\\\boxed{}\n# - Answer: non-negative integer 0-99999\n# - Use math, numpy, sympy, scipy, itertools, collections\n# - If two methods disagree, run THIRD to break tie\nInjected via DeveloperContent.new().with_instructions(). Tripled Python tool efficiency: 42 to 13 code calls per problem (S12 A/B test).
Use `math`, `numpy`,`sympy`,`itertools` and `collections` to solve the problem.\nEach problem sent as single UserContent message with raw problem text. No framing, no few-shot, no CoT instructions.
\nSystemContent.new()\n .with_model_identity('You are ChatGPT, a large language model trained by OpenAI.')\n .with_reasoning_effort(reasoning_effort=ReasoningEffort.HIGH)\n .with_tools(tool_config)\nThis is the +2 discovery. All 19 competitor notebooks omit or replace this identity. The openai-harmony library does NOT auto-inject it.
messages = [\n Message.from_role_and_content(Role.SYSTEM, system_content), # Identity + tools\n Message.from_role_and_content(Role.DEVELOPER, developer_content), # Instructions\n Message.from_role_and_content(Role.USER, problem_text) # Problem\n]\nThree messages total. No conversation history between problems.
\n
| # | \nDate | \nVersion | \nScore | \nDelta | \nKey Change | \n
|---|---|---|---|---|---|
| 1 | \n2026-03-06 | \nDeepSeek-14B v6 | \n0 | \n— | \nFirst submit (RERUN detection broken) | \n
| 2 | \n2026-03-07 | \nQwQ-32B v7 | \n0 | \n0 | \nInfrastructure works, model too weak | \n
| 3 | \n2026-03-11 | \nGPT-OSS v19 | \n0 | \n0 | \n6 root-cause fixes, still broken | \n
| 4 | \n2026-03-12 | \nFork v3 (baseline) | \n36 | \n+36 | \nUnmodified kurianbenoy fork | \n
| 5 | \n2026-03-13 | \nv22 | \n33 | \n−3 | \n3-prompt diversity | \n
| 6 | \n2026-03-14 | \nv23 | \n36 | \n+3 | \nMinimal prompt + multi-temp | \n
| 7 | \n2026-03-15 | \nv24 | \n37 | \n+1 | \nbatch=256, gpu=0.96, ctx=65536 | \n
| 8 | \n2026-03-16 | \nv25 | \n37 | \n0 | \nVariance confirmation | \n
| 9 | \n2026-03-18 | \nv28 | \n37 | \n0 | \nSC@16 (= SC@8) | \n
| 10 | \n2026-03-19 | \nv29 | \n34 | \n−3 | \nGenSelect + adaptive | \n
| 11 | \n2026-03-20 | \nv30 | \n36 | \n−1 | \nSubtraction (K=8, timeout=10) | \n
| 12 | \n2026-03-20 | \nv31 | \n34 | \n−3 | \nAlign with 42-scorer | \n
| 13 | \n2026-03-22 | \nv32 | \n0 | \n−37 | \nBUG: .with_text() AttributeError — all 400 attempts returned None | \n
| 14 | \n2026-03-24 | \nv33 | \n39 | \n+2 | \nHarmony fix + developer message | \n
| 15 | \n2026-03-24 | \nv34 | \n34 | \n−5 | \nPROVE filter + seqs=8 | \n
| 16 | \n2026-03-26 | \nv37b | \n31 | \n−8 | \nExperimental features | \n
| 17 | \n2026-03-28 | \nv33 resub #1 | \n33 | \n−6 | \nSame code as row 14; confirms variance | \n
| 18 | \n~2026-03-31 | \nv35-verify-fresh | \n34 | \n−5 | \nSelf-verification (FPR=75%) | \n
| 19 | \n~2026-04-01 | \nv31-majority-clean | \n0 | \n−39 | \nBug: clean majority rewrite failed | \n
| 20 | \n~2026-04-02 | \ngptoss-fork-v1.2 | \n34 | \n−5 | \nFork variant | \n
| 21 | \n~2026-04-03 | \nv54-mining-longest v1 | \n0 | \n−39 | \nCode mining extraction (broken) | \n
| 22 | \n~2026-04-04 | \nv54-mining-longest v2 | \nError | \n— | \nKaggle Error | \n
| 23 | \n~2026-04-04 | \nv54-code-mining+longest | \n32 | \n−7 | \nCode mining + longest tiebreak | \n
| 24 | \n~2026-04-05 | \nv55-eagle3 v2 | \n1 | \n−38 | \nEAGLE3 speculative decoding (VRAM crash) | \n
| 25 | \n~2026-04-07 | \nv33-fixed resub #2 | \n34 | \n−5 | \nSame code as row 14; third run | \n
| 26 | \n~2026-04-08 | \nv56-k10-longest | \n36 | \n−3 | \nK=10 + longest tiebreak | \n
| 27 | \n~2026-04-10 | \nv57-k12-longest | \n32 | \n−7 | \nK=12 + longest (over-sampling) | \n
| 28 | \n~2026-04-11 | \nv58-dsq-lora v3 | \n23 | \n−16 | \nLoRA adapter (BNB4/MXFP4 mismatch) | \n
| — | \n2026-04-15 | \nv33-fixed (final) | \n34 | \n— | \nSelected as final submission | \n
python -m vllm.entrypoints.openai.api_server \\\n --seed 42 \\\n --model /kaggle/tmp/setup/model \\\n --served-model-name gpt-oss \\\n --tensor-parallel-size 1 \\\n --max-num-seqs 256 \\\n --gpu-memory-utilization 0.96 \\\n --host 0.0.0.0 \\\n --port 8000 \\\n --dtype auto \\\n --kv-cache-dtype fp8_e4m3 \\\n --max-model-len 65536 \\\n --stream-interval 200 \\\n --max-cudagraph-capture-size 64 \\\n --async-scheduling \\\n --disable-log-stats \\\n --enable-prefix-caching\n| Type | \nSource ID | \nDescription | \n
|---|---|---|
| competition | \n118448 | \naimo-3 competition data | \n
| modelInstanceVersion | \n510391 (model 422384, instance 404485) | \ndanielhanchen/gpt-oss-120b | \n
| kernelVersion | \n289055161 | \naimo-3-utils (vLLM 0.18.1 wheels + tiktoken encodings) | \n
pow(seed + attempt_index, 2) % (2**31)| Function | \nCell | \nPurpose | \n
|---|---|---|
CFG (class) | \n9 | \nAll hyperparameters and prompts | \n
AIMO3Template.apply_chat_template() | \n11 | \nHarmony message encoding | \n
AIMO3Sandbox | \n12 | \nJupyter kernel pool management | \n
AIMO3Tool | \n13 | \nTIR code execution loop | \n
AIMO3Solver.__init__() | \n14 | \nvLLM server startup + model preload | \n
AIMO3Solver._start_server() | \n14 | \nvLLM subprocess with exact CLI args | \n
predict() | \n17 | \nMain entry: K=8 attempts + majority vote | \n
| Pre-flight check | \n16 | \nDeveloper message rendering verification | \n
All code in single notebook: aimo3-v33-fixed.ipynb (19 cells).
| Problem | \nGold | \nMajority | \nGold − Majority | \nError type | \n
|---|---|---|---|---|
| olymp-01629 | \n4950 | \n4949 | \n+1 | \nBoundary inclusive/exclusive | \n
| oda-00425 | \n3 | \n4 | \n−1 | \nCount boundary | \n
| dapo-15481 | \n2 | \n0 | \n+2 | \nBase case | \n
| oda-01612 | \n4 | \n2 | \n+2 | \nMultiplicity | \n
| olymp-02392 | \n49 | \n51 | \n−2 | \nRange boundary | \n
| dapo-10920 | \n4022 | \n4024 | \n−2 | \nCoefficient error | \n
| oda-01183 | \n12 | \n6 | \n+6 (×2) | \nFactor omission | \n
7/16 (44%) of hard-problem errors are within ±2 of the correct answer.
\n<think> CoT format (DeepSeek-R1 style)<think> format, but inference uses Harmony. Outputs malformed at inference time.| Experiment | \nWhy It Matters | \nWhy Not Run | \nEstimated Impact | \n
|---|---|---|---|
| Multi-run per problem (12+) | \nPer-problem variance | \nMax 3 runs available | \nDiagnostic | \n
| Reasoning channel analysis | \nHidden confidence signals | \nPipeline not instrumented | \nUnknown | \n
| PRM scoring (7B) | \nStep-level selection | \nVRAM blocked | \n+2-4 (literature) | \n
| CodeBERT clustering | \nApproach diversity metric | \nNot prioritized | \nUnknown | \n
| Adaptive K (K=16-32 on hard) | \nBudget 89% slack | \nNot tested in time | \n+1-2 | \n
| LoRA on boundary errors | \n44% of errors = off-by-one | \nData not curated | \n+1-3 | \n
| Resource | \nCost | \nTime | \n
|---|---|---|
| Kaggle submissions | \nFree | \n28 submissions × 5h = 140h compute | \n
| RunPod (LoRA training) | \n~$20 | \n3 training runs × 2-10h | \n
| Claude API (multi-agent) | \n~$200 | \n25+ sessions | \n
| Total calendar time | \n— | \n40 days (2026-03-06 to 2026-04-15) | \n
| Estimated human hours | \n— | \n~150h | \n
chat.completions.create()Very detailed and honest write-up. \nEspecially appreciated that you included the failed experiments instead of only the successful parts.
", "rawMarkdown": "Very detailed and honest write-up. \nEspecially appreciated that you included the failed experiments instead of only the successful parts.\n" }, { "id": 3457782, "postDate": "2026-05-14T10:28:25.033Z", "content": "Adding to the comments from @sambealing below, I have a few additional questions:
\nArithmetic Errors: You mention that arithmetic errors are not fixable. However, I would argue that they can be addressed through tool calls or formal verification systems such as Lean. If the model has produced a valid reasoning path that is mathematically sound, would you agree that it should be possible to translate it into Python or Lean code for verification? Did you experience any counter-examples in practice?
Answer Recovery: You use a fairly simple regex expression to extract multiple digits from the model's answer. Depending on the implementation, this can fail in several cases. For example, if the model produces an equation, fraction, or additional text inside the boxed environment, the correct answer may not be recovered. Did you observe or document cases where the model produced the correct answer, but the extraction failed, and the problem was therefore marked as unsolved? If so, how would you adapt the extraction logic to recover such cases?
Bottleneck: You claim that “generation quality, not selection quality, is the bottleneck.” However, other contestants argue the opposite, for example @natnitarach in this submission. They argue that selection is the bottleneck, motivated by the publicly available pass@100 statistics on both the public and private sets here. What are your thoughts on this interpretation?
Reproducibility: Is the code needed to reproduce all experiments and notebook configurations available? If so, please share it to improve the reproducibility of your work.
Hi, thank you for your submission to the Writeup Prize. @philipvonderlind and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.
\nLook forward to reading your responses on these.
", "rawMarkdown": "Hi, thank you for your submission to the Writeup Prize. @philipvonderlind and I will post some questions below that help us in assessing your submission more closely, as described in the [updated discussion post](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/689703).\n\n1. **Harmony +2 mechanism:** The +2 from v33 came from adding the ChatGPT identity to SystemContent plus the developer-message code rules. Did you have a chance to isolate which of the two (identity vs developer message) actually drove the gain, or are they entangled in the +2?\n2. **Off-by-one: ambiguity vs model error:** Of the 7/16 hard-problem errors within ±2 of gold (Appendix D), did you investigate whether the source problem statements themselves were ambiguous (e.g. the 'is 0 positive?' issue discussed in [this Kaggle thread](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/672872)) versus cases where the question was clear and the model just made a mistake in its solution?\n3. **Variance estimate from n=3:** Your σ ≈ 3 comes from three identical-code runs (39, 33, 34), while the Nitarach n=13 analysis gives σ ≈ 1.7. With only three samples there's a lot of uncertainty on σ itself. Do you think the true σ for this substrate is closer to 3 or 1.7, and what evidence would tip you one way or the other?\n\nLook forward to reading your responses on these.\n" }, { "id": 3446696, "postDate": "2026-04-22T00:53:29.913Z", "rawMarkdown": "", "isDeleted": true } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T04:22:59Z", "competition_id": 118448, "competition_slug": "ai-mathematical-olympiad-progress-prize-3", "team_id": 15309688, "solution_url": "/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/harmony-format-dead-ends-and-variance-in-aimo3" }