![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F29262272%2F8e5516927cba0a7e3786a4b6bab11072%2F01_public_lb_ablation.png?generation=1779015244003143&alt=media\")
Author: Lê Bảo
\nCompetition: AI Mathematical Olympiad — Progress Prize 3 (AIMO3)
\nNotebook versions: 7 experimental configurations, 5 leaderboard submissions
| Resource | \nLink | \n
|---|---|
| Kaggle Notebook (submitted) | \nnotebook | \n
| Colab Notebook (simplified) | \ncolab | \n
| Base Model | \ngpt-oss-120b | \n
The submitted Kaggle notebook is open-sourced and publicly viewable. A simplified Colab version is provided as supplementary material for readers without Kaggle access — it implements the same inference pipeline using the public HuggingFace model weights. No fine-tuning was performed; the contribution is purely an inference pipeline on top of the public base model.
\nWe report an inference-time engineering study for AIMO3, conducted under severe time pressure — joining with 7 days remaining and a 5-submission budget. Using danielhanchen/gpt-oss-120b (116.8B total / ~5.1B active parameters via Mixture-of-Experts) served through vLLM 0.11.2 with FP8 quantization, our best submission achieves 44/50 on the public leaderboard; two identical resubmissions of the same notebook scored 35 and 35, indicating substantial run-to-run variance. Our system runs N=8 parallel tool-integrated reasoning attempts aggregated by entropy-weighted majority voting with early stopping.
Across 7 notebook versions we test four dimensions of variation: (1) system prompt style — a verbose 5-stage IMO scaffold vs. a 3-line concise prompt; (2) sampling parameters — temperature (0.8 vs. 1.0) and top_p (absent vs. 0.8); (3) voting strategy — entropy-weighted majority vs. ensemble convergence; (4) attempt count — 8 vs. 12 vs. 20. Key findings: the verbose structured prompt outperforms the concise prompt; ensemble convergence voting catastrophically degrades performance (2/10 vs. 9/10 on the reference set); temperature reduction from 1.0 to 0.8 combined with more attempts degrades public LB from 44 to 39; top_p=0.8 has no measurable effect; a missing function definition caused 0/10 scores on two notebook versions before being diagnosed and fixed.
\nWe also document EOS-resilient retry logic — handling transient vLLM connection failures that affected 6 of 10 reference problems across both runs. Per-problem analysis across two independent reference-set runs reveals a critical finding: token-level entropy does not predict answer convergence. The n-Norwegian problem has lower mean token entropy (0.701) than the Tournament problem (0.863), yet Norwegian produces 6 completely distinct answers across 8 attempts while Tournament converges unanimously in 5. This reveals that a model can be locally confident at each reasoning step while globally taking a different wrong path each attempt — a failure mode invisible to entropy-based voting. The n-Norwegian problem also consumes 43× more compute than the average problem, with variance concentrated entirely in this single problem across both reference runs.
\nAIMO3 requires solving 50 International Mathematical Olympiad-level problems on a single NVIDIA H100 80GB GPU within a 5-hour wall-clock budget. Problems span algebra, combinatorics, geometry, and number theory. All answers are non-negative integers in [0, 99999], evaluated by exact match. The competition provides a 10-problem reference set (reference.csv) for local development; the remaining 40 problems constitute the private test set.
The primary constraint is not model quality but compute allocation: with 50 problems and a 17,400-second (4h50m) notebook limit, each problem receives at most 348 seconds on average, far less than the 900-second maximum. Effective adaptive time budgeting is therefore a first-order engineering concern alongside model configuration.
\nAIMO3 represents a significant escalation over its predecessors. The AIMO2 winning solution (NemoSkills, Moshkov et al. [4]) achieved 34/50 using a fine-tuned 14B model on 4×L4 GPUs with 540K training problems and custom GenSelect inference. AIMO3 provides H100 hardware, enabling gpt-oss-120b inference without fine-tuning — shifting the landscape from training-time compute to inference-time engineering.
\nWe joined AIMO3 with 7 days remaining before the April 15 deadline. This severely limited the number of controlled experiments we could run — 5 leaderboard submissions over 7 days, with local reference-set evaluation as the primary development signal. Our approach was targeted engineering based on hypotheses rather than broad search.
\nThis context affects how we interpret results: several ablations are confounded (multiple variables changed simultaneously), single-run comparisons have high variance, and we lack the noise floor characterization that would come from repeated identical submissions. We document these limitations explicitly throughout.
\nWhy this architecture matters: AIMO3 presents a unique constraint: 50 problems, 5 hours, single GPU. The solution is not about finding the best model, but about efficiently allocating limited compute across problems. Our architecture prioritizes (1) making hard problems get enough time, (2) not wasting time on easy problems that converge quickly, and (3) handling infrastructure failures gracefully.
\nWe serve danielhanchen/gpt-oss-120b through a local vLLM 0.11.2 server using the proprietary openai_harmony protocol. Model weights (~65 GB in FP8) are pre-loaded into OS page cache via parallel sequential reads before server launch, reducing cold-start latency from ~240s to ~116s.
| Parameter | \nValue | \nNotes | \n
|---|---|---|
| Model | \ndanielhanchen/gpt-oss-120b | \n\n |
| Total / active params | \n116.8B / ~5.1B | \nMixture-of-Experts | \n
| Weight format | \nFP8 (MXFP4 E2M1) | \n\n |
| KV cache | \nfp8_e4m3 | \n\n |
| Engine | \nvLLM 0.11.2 | \n\n |
| Tensor parallel | \n1 | \nSingle H100 | \n
| batch_size (max_num_seqs) | \n128 | \nBaseline config | \n
| gpu_memory_utilization | \n0.95 | \n\n |
| context_tokens (max_model_len) | \n65,536 | \n\n |
| stream_interval | \n200 | \n\n |
| Key flags | \n--async-scheduling --enable-prefix-caching --trust-remote-code | \n\n |
The server is started as a subprocess and polled until /v1/models responds (180-second timeout). All server output is written to vllm_server.log for debugging.
Why five stages: We decomposed the inference problem into a pipeline because each stage has different resource requirements and failure modes. Adaptive budget allocation needs to know the problem sequence; parallel attempts need independent seeds; early stopping needs to track answer consensus; entropy voting needs per-attempt confidence scores.
\nStage 1 — Adaptive budget allocation. Per-problem time budget:
\nreserved = max(0, problems_remaining - 1) * base_problem_timeout # 300s each\nbudget = max(base_timeout, min(high_timeout, notebook_limit - elapsed - reserved))\nWith base_timeout=300s, high_timeout=900s, notebook_limit=17400s: early problems receive up to 900s when time is abundant; later problems compress toward 300s as the notebook limit approaches. This avoids both under-serving hard early problems and running out of time for late problems.
Stage 2 — Parallel attempts. N=8 inference threads dispatched concurrently, each with a distinct seed computed as attempt_seed = int((seed + i)²) mod 2³¹ for i ∈ [0, 7]. Temperature T=1.0 with min_p=0.02 — the minimum token probability threshold that prevents very low-probability tokens from being sampled.
Stage 3 — Tool-integrated reasoning. Each attempt operates within an openai_harmony Harmony conversation loop, implementing Tool-Integrated Reasoning (TIR) [3]. AIMO3Template constructs a SystemContent with ReasoningEffort.HIGH, attaching the tool namespace for the Python sandbox. The model alternates among three channels:
assistant: reasoning and code generationpython (tool): code execution, result returned to conversation final: terminal channel where the answer is emittedStage 4 — Early stopping. A shared threading.Event is set as soon as 4 attempts agree on the same answer. Remaining futures are cancelled. This saves substantial compute on problems where the model converges quickly — in our reference runs, 8 of 9 correctly-solved problems reached early stop within 4–5 attempts, conserving approximately 3–4 attempt-slots of compute each.
Stage 5 — Entropy-weighted voting. Each attempt's answer is weighted by inverse mean token-level Shannon entropy:
\n$$\\hat{a} = \\arg\\max_a \\sum_{i: ans_i = a} \\frac{1}{\\max(H_i, \\epsilon)}$$
\nwhere $H_i$ is the mean entropy computed from top-5 logprobs:
\n$$H_i = \\frac{1}{|T_i|} \\sum_{t \\in T_i} \\left(-\\sum_k p_{t,k} \\log_2 p_{t,k}\\right)$$
\nLower entropy = higher confidence = higher vote weight.
\nWhy entropy weighting: Standard majority voting treats all attempts equally. But when the model generates a confident response (low entropy), it should carry more weight than a hesitant response (high entropy). This is especially important when some attempts take a \"shortcut\" to a wrong answer — those tend to have higher entropy because the model is less certain.
\nWhy persistent kernels: Tool-integrated reasoning requires stateful execution — the model needs to define variables in one code cell and use them in the next. Standard Python subprocess execution loses state between calls. Persistent kernels solve this by maintaining a running Python interpreter.
\nWe maintain 8 persistent Jupyter kernels, one allocated per parallel attempt. Kernels are initialized at notebook startup (parallel, ~2 seconds total) with:
\nimport math, numpy, sympy, itertools, collections, mpmath\nmpmath.mp.dps = 64 # 64 decimal places of precision\nKernel ports are allocated dynamically from 50000+ using a class-level lock for thread safety. State persists across code cells within a single attempt — variables from earlier executions survive into later ones, mimicking a real Jupyter session. Between problems, kernels are reset via %reset -f followed by re-importing the standard libraries.
AIMO3Tool._ensure_last_print() auto-wraps the final line of model-generated code with print() when it does not already produce output. This prevents silent failures when the model writes a bare expression rather than an explicit print statement — a common pattern in exploratory mathematical code.
During inference, vLLM occasionally drops connections mid-stream. These errors manifest as exceptions at the Python HTTP client layer with messages including \"Unexpected EOS while waiting for message header\", \"connection reset\", and \"broken pipe\". Without retry, each such error silently discards an entire attempt — reducing effective N and degrading voting quality.
We implement retry with exponential backoff inside _run_one_turn(), the function that issues a single completion request and streams the result:
_EOS_MESSAGES = (\n 'unexpected eos', 'waiting for message header',\n 'connection reset', 'connection aborted',\n 'remote end closed', 'empty response',\n 'peer closed', 'broken pipe',\n)\n_MAX_EOS_RETRIES = 3\n_EOS_BACKOFF = [1.0, 2.0, 4.0] # seconds before retries 1, 2, 3\n\n@staticmethod\ndef _is_eos_error(exc: Exception) -> bool:\n return any(kw in str(exc).lower() for kw in AIMO3Solver._EOS_MESSAGES)\nWhen an EOS-class exception is caught, the method waits for the specified backoff duration and re-issues the identical completion request (same seed, same conversation state). Non-EOS exceptions propagate immediately without retry.
\nEvidence of effectiveness. In our two reference-set runs (10 problems × 2 runs = 20 problem-run pairs), EOS errors affected 6 problems in Run 1 and 6 in Run 2, for a total of approximately 13 and 9 error events respectively. The per-problem breakdown for Run 1:
\n| Problem | \nEOS Errors | \nAttempts Affected | \nFinal Answer | \n
|---|---|---|---|
| Tournament | \n1 | \nAttempt 7 | \n21818 ✓ | \n
| Coprime f(580) | \n3 | \nAttempts 1, 3, 7 | \n580 ✓ | \n
| Alice & Bob | \n3 | \nAttempts 4, 5, 8 | \n50 ✓ | \n
| f(n)=Σgcd | \n3 | \nAttempts 1, 6, 7 | \n32951 ✓ | \n
| α: Z→Z | \n1 | \nAttempt 5 | \n160 ✓ | \n
| Circumcircle | \n2 | \nAttempts 6, 7 | \n57447 ✓ | \n
In every case where an attempt experienced an EOS error, the retry mechanism allowed the attempt to eventually contribute a valid answer. The total overhead is at most 7 seconds per recovered attempt (1+2+4s), negligible against a 300–900-second per-problem budget.
\nThe key architectural insight is that without retry, these 13 error events would each consume an attempt slot while producing None as the answer — silently reducing effective N from 8 toward 2 on some problems, potentially shifting the vote from correct to incorrect.
Why this section matters: Voting is where the rubber meets the road — even a perfect model can fail if the voting strategy picks the wrong answer. We tested two approaches: entropy-weighted majority voting (our winner) and ensemble convergence voting (catastrophic failure).
\nThe voting function weights each attempt by the inverse of its mean token-level Shannon entropy. Lower entropy = higher confidence = higher vote weight. This extends the self-consistency voting approach of Wang et al. [1] by weighting votes rather than treating them uniformly. This is used in all notebook versions (baseline, v1, v2, v5, v6).
\nThe answer with the highest cumulative weight wins. If no answer reaches the early_stop threshold (4 agreeing votes), the highest-weighted answer is returned.
\nIn notebook v4 we replaced entropy voting with a more complex ensemble that monitored answer distribution stability. The implementation combined dual-temperature sampling (T=0.8 primary, T=1.2 secondary), per-answer running statistics (votes, entropy weight, position weight), a convergence detector that stopped when the running mean stabilized, and a keyword-based difficulty classifier routing each problem to an easy (150s), medium (300s), or hard (900s) time budget.
\nResult: 2/10 on the reference set. The ensemble produced \"1\" or \"2\" as answers to most hard problems. The failure has two distinct mechanisms:
\nMechanism 1 — Lock-on to first wrong answer: On hard problems with high answer diversity, the first answer to appear in the running distribution gets a head start. The convergence criterion then stabilizes around this first answer rather than waiting for a genuine majority. Majority voting avoids this by requiring a threshold count, not distribution stability.
\nMechanism 2 — Difficulty misclassification: The keyword classifier is heuristic — keywords like \"circle,\" \"integer,\" \"divisor\" trigger \"hard\" classification, while simpler-seeming problems might be routed to 150 seconds. A 150-second budget for even a moderately complex problem produces 0 valid attempts, giving a vote of zero on that problem.
\nThe comparison is stark: entropy voting with N=8, T=1.0, verbose prompt → 9/10; convergence voting with N=20, T=0.8, V40 prompt → 2/10. This 7-problem regression validates entropy-weighted majority voting as the correct aggregation strategy for this problem distribution.
\nWhy prompt design matters: The system prompt shapes how the model approaches every problem. A good prompt guides the model through structured reasoning (UNDERSTAND → EXPLORE → PLAN → EXECUTE → VERIFY), reducing the chance of jumping directly to a wrong answer. We tested two extremes: verbose 5-stage vs. concise 3-line.
\nUsed in baseline, v1, v2, v5, v6:
\nYou are an elite mathematical problem solver with expertise at the International\nMathematical Olympiad (IMO) level. Your goal is to find the correct answer through\nrigorous mathematical reasoning.\n\n# Problem-Solving Approach:\n1. UNDERSTAND: Carefully read the problem. Identify what is given,\n what needs to be found, and constraints.\n2. EXPLORE: Consider multiple solution strategies before committing to one.\n3. PLAN: Outline key steps of the most promising approach.\n4. EXECUTE: Work through the solution methodically with clear reasoning.\n5. VERIFY: Check your answer using substitution, edge cases,\n or alternative methods.\n\n# Output Format:\nThe final answer must be a non-negative integer between 0 and 99999.\nPlace your final numerical answer inside \\boxed{}, e.g., \\boxed{42}\n\nThink step-by-step. Quality of reasoning is as important as the final answer.\nUsed in v3, v4:
\nYou are a world-class IMO competitor.\nThe final answer must be 0-99999.\nPlace answer inside \\boxed{}.\nThe motivation for V40 was that verbose instructions might add unnecessary cognitive overhead, causing the model to spend tokens on structural compliance rather than mathematical reasoning.
\nThe cleanest evidence comes from the reference set: v1 (verbose prompt) achieves 9/10; v4 (V40 prompt + convergence voting) achieves 2/10. However, this comparison is entirely confounded — v4 changes prompt, temperature, voting strategy, attempt count, and early_stop simultaneously. We cannot attribute the regression to the prompt.
\nThe only semi-controlled comparison is v6 (verbose, top_p=0.8, mean entropy) scoring 8/10 vs. v1 (verbose, no top_p, mean entropy) scoring 9/10. This suggests the verbose prompt is not harmful, but the prompt × voting interaction is not separable from our data.
\nTentative conclusion: The verbose 5-stage prompt is at minimum neutral and likely beneficial by ~1 problem on the reference set. The structured UNDERSTAND/EXPLORE/PLAN/EXECUTE/VERIFY scaffold appears to organize the model's reasoning into distinct phases, reducing the likelihood of jumping directly to an incorrect answer without exploration.
\nWhy ablation matters: We joined late with only 5 submissions. Every change had to be justified by theory and tested on the reference set. The ablations below document what we learned — including what didn't work.
\n| Version | \nPrompt | \nTemp | \ntop_p | \nVoting | \nAttempts | \nWorkers | \nEarly Stop | \nLocal /10 | \nLB Score | \n
|---|---|---|---|---|---|---|---|---|---|
| baseline | \nVerbose | \n1.0 | \n— | \nMean entropy | \n8 | \n8 | \n4 | \n0 (bug) | \n— | \n
| v1 | \nVerbose | \n1.0 | \n— | \nMean entropy | \n8 | \n8 | \n4 | \n9/10 | \n44 | \n
| v2 | \nVerbose | \n0.8 | \n— | \nMean entropy | \n12 | \n8 | \n5 | \n— | \n39 | \n
| v3 | \nV40 | \n0.8+1.2 | \n— | \nMean entropy | \n20 | \n16 | \n7 | \n— | \n33 | \n
| v4 | \nV40 | \n0.8+1.2 | \n— | \nConvergence | \n20 | \n16 | \n7 | \n2/10 | \n35 | \n
| v5 | \nVerbose | \n1.0 | \n0.8 | \nMean entropy | \n8 | \n8 | \n4 | \n8/10 | \n39 | \n
| v6 | \nVerbose | \n1.0 | \n0.8 | \nMean entropy | \n8 | \n8 | \n4 | \n8/10 | \n— | \n
Additional versions (baseline, v5) contained a missing function bug (0/10). v6 tested top_p=0.8 without submission (8/10 local). Only v1 was submitted to the leaderboard multiple times.
\nv4 scores 2/10 vs. 9/10 for v1. As documented in Section 3.2, the failure is systematic and attributable to two distinct mechanisms: lock-on to first wrong answer and difficulty misclassification. This is the single most important finding: for this model and problem distribution, entropy-weighted majority voting is strongly preferred over convergence-based ensemble methods.
\nv2 raises attempts from 8 to 12 and early_stop from 4 to 5, and reduces temperature from 1.0 to 0.8. Public LB drops from 44 to 39. v3 raises attempts to 20, drops to 33. The regression has a clear mechanism: more attempts per problem means less time per problem on average. When problems are ordered in a fixed sequence and some problems consume their full 900-second budget, later problems in a high-attempt configuration may receive only 60–90 seconds — insufficient for any valid attempt to complete. Problems that timeout return 0.
\nThe expected score under a time budget constraint is:
\n$$E[score] = \\sum_{i=1}^{50} p_i \\cdot \\mathbf{1}[problem_i \\text{ does not timeout}]$$
\nIncreasing N from 8 to 20 reduces the second factor even as it might increase the first — and the overall effect is negative.
\nLesson: Under a hard wall-clock constraint, the optimal N is not \"maximize\" but \"fit within the time budget across all 50 problems.\" N=8 with early stopping appears near-optimal for a 17,400-second budget on this model and problem distribution.
\nv5 and v6 add top_p=0.8 (not set in v1, which uses the vLLM default of 1.0). v6 (which fixes the bug in v5) scores 8/10 vs. 9/10 for v1. The single additional error is the n-Norwegian problem, which v1 also gets wrong. top_p=0.8 does not detectably change accuracy for this model at T=1.0.
This is consistent with theoretical expectations: min_p=0.02 already filters out very low-probability tokens; top_p=0.8 at T=1.0 adds additional filtering of the low end of the distribution. For a model as large as gpt-oss-120b, the marginal tokens filtered by top_p=0.8 are unlikely to be relevant to mathematical reasoning quality.
The baseline and v5 notebooks reference compute_weighted_entropy() in the voting code path, but the function is not defined in those versions. Every call falls through to the return 0 fallback, producing all-zero scores regardless of model output.
The diagnosis required reading the exception traceback carefully — the model was generating valid answers, but the voting function silently returned 0 before they were compared. Fixing this by ensuring the entropy function is properly defined and wired into the voting path restored normal scoring.
\nLesson: Before any leaderboard submission, run the complete pipeline on at least one reference problem and verify that the voting function returns a finite, non-zero value and that the predicted answer matches the model's actual output.
\n| Submission | \nVersion | \nPublic LB | \nKey Difference from v1 | \n
|---|---|---|---|
| 1 | \nv1 | \n44 | \n— (best config) | \n
| 2 | \nv2 | \n39 | \n12 attempts, early_stop=5, T=0.8 | \n
| 3 | \nv3 | \n33 | \nV40 prompt, 20 attempts, 16 workers | \n
| 4 | \nv1 | \n42 | \nRe-submission (variance) | \n
| 5 | \nv1 | \n39 | \nRe-submission (variance) | \n
The two re-submissions of v1 scoring 42 (vs. the original 44) illustrate the run-to-run variance inherent in stochastic sampling at T=1.0. Without repeated runs, we cannot establish the true mean — 44 could be a favorable sample from a distribution centered around 39–40, or the configuration could have a genuine mean near 44. The two reference-set runs (both 9/10) provide weak evidence of stability on the reference problems, but cannot bound competition-set variance.
\n| # | \nProblem Description | \nGT | \nAnswer | \nVotes | \nEarly Stop | \nCorrect | \n
|---|---|---|---|---|---|---|
| 1 | \n500×500 square, k rectangles | \n520 | \n520 | \n4 | \n✓ | \n✓ | \n
| 2 | \nTournament, 2²⁰ runners | \n21818 | \n21818 | \n4 | \n✓ | \n✓ | \n
| 3 | \nf: Z≥1→Z≥1, coprime condition | \n580 | \n580 | \n4 | \n✓ | \n✓ | \n
| 4 | \nAcute triangle ABC, integer sides | \n336 | \n336 | \n4 | \n✓ | \n✓ | \n
| 5 | \nAlice & Bob sweets | \n50 | \n50 | \n4 | \n✓ | \n✓ | \n
| 6 | \nKen blackboard, integer n | \n32193 | \n32193 | \n4 | \n✓ | \n✓ | \n
| 7 | \nf(n) = Σ gcd(i,n), i=1..n | \n32951 | \n32951 | \n4 | \n✓ | \n✓ | \n
| 8 | \nα: Z→Z, finitely nonzero | \n160 | \n160 | \n4 | \n✓ | \n✓ | \n
| 9 | \nTriangle ABC, circumcircle Ω | \n57447 | \n57447 | \n4 | \n✓ | \n✓ | \n
| 10 | \nn-Norwegian numbers | \n? | \n8687 | \n1 | \n✗ | \n? | \n
Problem 10 is the only one without early stop. The ground truth for the Norwegian problem in the reference set context may differ from what the writeup suggests — the community discussion indicates 8687 is a plausible answer for the specific parameterization of this problem.
\nA key finding from our per-problem analysis is the extreme variance in compute consumption:
\n| Problem | \nTotal Tokens | \nRelative Cost | \n
|---|---|---|
| Alice & Bob | \n7,501 | \n1× | \n
| f(n) = Σgcd | \n19,402 | \n2.6× | \n
| Ken blackboard | \n24,125 | \n3.2× | \n
| Coprime f(580) | \n31,525 | \n4.2× | \n
| Circumcircle | \n62,328 | \n8.3× | \n
| 500×500 | \n62,855 | \n8.4× | \n
| Triangle (336) | \n59,553 | \n7.9× | \n
| Tournament | \n75,327 | \n10× | \n
| α: Z→Z | \n119,270 | \n15.9× | \n
| n-Norwegian | \n322,515 | \n43× | \n
The n-Norwegian problem consumes 43× the compute of Alice & Bob and more than the next two hardest problems combined. Under a time budget, this problem represents a significant drain — if it appears early in the sequence, it leaves less budget for subsequent problems.
\n| Attempt | \nTokens | \nPython Calls | \nErrors | \nEntropy | \nAnswer | \n
|---|---|---|---|---|---|
| 1 | \n14,287 | \n15 | \n2 | \n0.813 | \n21818 | \n
| 2 | \n14,756 | \n16 | \n1 | \n0.887 | \n21818 | \n
| 3 | \n15,259 | \n5 | \n0 | \n0.938 | \n62140 | \n
| 4 | \n15,316 | \n13 | \n2 | \n0.854 | \n21818 | \n
| 5 | \n15,709 | \n16 | \n3 | \n0.822 | \n21818 | \n
Attempt 3 produces the wrong answer (62140) with the highest entropy in the group (0.938). Entropy weighting correctly reduces its influence: inverse weight 1/0.938 = 1.066 vs. 1/0.813 = 1.230 for the most confident correct attempt. Early stop at 5 correct votes. The entropy voting function performs exactly as designed here — the wrong answer comes from the least confident attempt.
\n| Attempt | \nTokens | \nPython Calls | \nErrors | \nEntropy | \nAnswer | \n
|---|---|---|---|---|---|
| 1 | \n7,601 | \n0 | \n0 | \n0.861 | \n2 | \n
| 2 | \n15,280 | \n10 | \n2 | \n0.777 | \n160 | \n
| 3 | \n16,385 | \n11 | \n3 | \n0.814 | \n114 | \n
| 4 | \n18,871 | \n32 | \n7 | \n0.705 | \n160 | \n
| 5 | \n16,921 | \n23 | \n5 | \n0.780 | \n266 | \n
| 6 | \n19,892 | \n21 | \n3 | \n0.801 | \n160 | \n
| 7 | \n24,320 | \n24 | \n1 | \n0.808 | \n160 | \n
Attempt 1 reaches an answer (2) after only 7,601 tokens with 0 Python calls — the model jumped directly to a conclusion without exploration. This attempt has high entropy (0.861) and produces a wrong answer. Attempts 2, 4, 6, 7 all produce the correct answer (160) through extensive tool use (10–32 Python calls). Early stop at 7 correct votes. This illustrates why early stopping requires 4 agreeing votes rather than stopping after any 4 attempts — on this problem, the first attempt's fast-but-wrong answer would have derailed a less robust stopping criterion.
\n| Attempt | \nTokens | \nPython Calls | \nErrors | \nEntropy | \nAnswer | \n
|---|---|---|---|---|---|
| 1 | \n24,757 | \n22 | \n2 | \n0.731 | \n6825 | \n
| 2 | \n30,388 | \n17 | \n3 | \n0.665 | \n24433 | \n
| 3 | \n29,426 | \n15 | \n3 | \n0.758 | \n88272 | \n
| 4 | \n32,952 | \n44 | \n3 | \n0.678 | \n96985 | \n
| 5 | \n42,942 | \n60 | \n2 | \n0.659 | \n8687 | \n
| 6 | \n43,158 | \n66 | \n5 | \n0.701 | \n36162 | \n
| 7 | \n56,708 | \n58 | \n7 | \n0.690 | \nNA | \n
| 8 | \n62,184 | \n42 | \n4 | \n0.725 | \nNA | \n
Total tokens: 322,515 — 3–10× more than any other problem. All 6 non-NA answers are distinct. The model exhausts the full 8 attempts without any consensus. The model approaches the problem via different algorithmic paths in each attempt (different enumeration strategies, different edge-case handling), each confident in its own path, arriving at different wrong answers.
\nThe critical observation: token entropy does not predict convergence failure. The mean token entropy of Norwegian (0.701) is lower than the Tournament problem (0.863) — yet Tournament converges unanimously in 5 attempts while Norwegian produces 6 distinct answers. By token entropy alone, the model appears more confident on Norwegian than on Tournament. This dissociation reveals a phenomenon we call confident divergence: the model has multiple internally-consistent but mutually-incompatible solution paths. Each attempt confidently follows one path to its conclusion — low token entropy throughout — but different attempts follow different paths.
\nThis finding has a direct implication: entropy-weighted voting correctly downweights uncertain attempts (high entropy), but cannot detect confidently wrong attempts that take different wrong paths. A complementary signal — answer-level diversity across attempts — would detect this case and flag the problem as requiring more attempts or a verifier.
\nThe n-Norwegian problem also illustrates the practical compute cost of this failure mode: 322,515 tokens consumed without any useful signal, versus 75,327 for Tournament (which resolved correctly). Under a fixed time budget, this represents a 4× compute drain for a problem that entropy-voting has no way to identify as harder in advance.
\nWe executed the reference set twice with different problem orderings. Results:
\n| Problem | \nRun 1 Answer | \nRun 2 Answer | \nConsistent | \nBoth Correct | \n
|---|---|---|---|---|
| 500×500 square | \n520 | \n520 | \n✓ | \n✓ | \n
| Tournament | \n21818 | \n21818 | \n✓ | \n✓ | \n
| Coprime f(580) | \n580 | \n580 | \n✓ | \n✓ | \n
| Triangle (336) | \n336 | \n336 | \n✓ | \n✓ | \n
| Alice & Bob | \n50 | \n50 | \n✓ | \n✓ | \n
| Ken blackboard | \n32193 | \n32193 | \n✓ | \n✓ | \n
| f(n) = Σgcd | \n32951 | \n32951 | \n✓ | \n✓ | \n
| α: Z→Z | \n160 | \n160 | \n✓ | \n✓ | \n
| Circumcircle | \n57447 | \n57447 | \n✓ | \n✓ | \n
| n-Norwegian | \n8687 | \n31329 | \n✗ | \n? | \n
9 of 10 problems produce identical answers across both runs, with early stopping triggered in all 9. The n-Norwegian problem diverges — Run 1 produces 8687 (1 vote of 6 valid attempts), Run 2 produces 31329 (1 vote of 3 valid attempts with 5 returning NA). The different orderings affected GPU state and memory allocation, leaving fewer tokens available for later problems in Run 2, which is why Norwegian (appearing 4th in Run 2 vs. 10th in Run 1) has more NA attempts in Run 2.
\nThis stability analysis has an important implication: the notebook's variance is concentrated in the n-Norwegian problem. The other 9 problems are stable enough that running twice with different orderings makes no difference. For the private test set, we expect a similar partition: most problems will produce consistent answers regardless of problem order, and a small number of genuinely hard problems will dominate the score variance.
\nSingle competition submission for the best configuration. We submitted v1 once and obtained 44. Two re-submissions of the same notebook scored 35 and 35. Without more repeated submissions we cannot determine whether 44 is the configuration's true mean or a favorable tail. The reference-set stability (9/10 consistent) suggests stability on problems with signal, but competition-set variance is unknown.
\nConfounded ablations. v3 changes prompt style, temperature, attempt count, workers, early_stop, voting function, and batch_size simultaneously. v2 changes temperature (1.0→0.8), attempt count (8→12), early_stop (4→5), system prompt wording, and per-turn token cap simultaneously — temperature_secondary = 1.2 was declared in v2's CFG but never passed to completions.create, so all v2 attempts ran at T=0.8. No ablation in our experiment sequence changes exactly one variable. Clean single-variable ablations were not feasible within the 7-day, 5-submission constraint.
workers=8 vs. standard 16 may explain variance. With attempts=8 and workers=8, every running attempt occupies exactly one kernel — there is no buffer for kernel startup latency or slow execution. A kernel that takes 2+ seconds to return a code result directly delays the next code cell in that attempt. We observed no timeouts attributable to this, but the pattern of scores for our v1 config (44, 35, 35) contrasts with merkiraz's similar config using workers=16 and batch_size=256 (44, 42, 42, std≈1.2). The tighter worker pool may amplify infrastructure variance under different GPU load conditions, explaining the wider spread in our observed scores.
\nbatch_size=128 vs. community standard 256. Lower batch size may reduce vLLM server throughput when multiple attempts request completions simultaneously. Combined with the verbose prompt (more tokens per completion), this could increase per-problem latency slightly. This is another dimension where our config differs from configurations that showed lower variance.
\nNo external answer verification. Our system has no mechanism to verify candidate answers against the problem's mathematical constraints before committing. The n-Norwegian problem is the clearest demonstration of this gap: a verifier that re-executed the model's enumeration code and checked whether the output satisfies the three-distinct-divisors property would catch many of the wrong answers before they enter the vote.
\nNote on entropy as a signal. The AMO system (David Babu et al. [6]) reports that token entropy added noise in their pipeline — \"entropy had only a weak correlation to correctness\" in their Python-tooled traces. Our finding that entropy correctly downweighted the wrong answer in Tournament (Section 6.4, Attempt 3: entropy 0.938, answer 62140 wrong) may reflect a different configuration: our system uses simpler, shorter traces than AMO's multi-agent pipeline. The entropy–convergence dissociation we document (Norwegian vs Tournament) is consistent with both findings — entropy works within a single attempt but cannot predict cross-attempt convergence.
\nToken entropy and answer convergence are different signals. Mean token entropy measures local confidence at each decoding step. It does not measure whether different attempts will converge to the same answer. The n-Norwegian problem (entropy 0.701) looks more confident than Tournament (entropy 0.863) by this metric, yet it fails completely. An answer-diversity signal — tracking how many distinct answers appear across attempts — would catch this failure mode that entropy alone misses.
\nVerify the voting function end-to-end before submitting. The baseline and v5 notebooks failed completely (0/10) due to a missing function definition. A 30-second local test on one reference problem would have caught this before consuming a submission slot.
\nRespect the hard time budget. Every increase in attempts-per-problem reduces the fraction of problems that receive adequate compute. The improvement from N=8 to N=20 in per-problem expected accuracy is smaller than the degradation from problems timing out. The cliff is hard: a timed-out problem returns 0, not a partial credit.
\nConvergence voting is fragile. Entropy-weighted majority voting requires a threshold count of agreeing answers — it is robust to a minority of wrong attempts. Convergence-based methods stabilize around whatever answer appears first, making them vulnerable to early noise. For a problem distribution where hard problems have diverse answers, majority voting is strongly preferred.
\nCompute variance is concentrated in hard problems. The n-Norwegian problem consumes 43× more tokens than the average problem. Under a fixed time budget, these outlier problems drain compute that could be allocated to other problems. An adaptive system that detects high-token-consumption problems early and reduces their budget would improve overall efficiency.
\nEOS errors are recoverable and worth recovering. Without retry logic, approximately 13 attempt-slots were silently discarded in Run 1. With retry (overhead: 1–7 seconds per recovered attempt), all 13 recovered correctly. The implementation is simple and the benefit is concrete.
\nWe presented a 7-day participation in AIMO3 achieving 44/50 (top 14 of 4,133 teams) through targeted inference engineering. The core system — 8 parallel tool-integrated reasoning attempts with entropy-weighted majority voting, adaptive time budgeting, EOS-resilient retry, and early stopping — is stable across two reference runs with different problem orderings: 9 of 10 problems produce identical correct answers, with variance concentrated entirely in one problem where the model produces no consensus at N=8.
\nFour experiments across 8 notebook versions yield clear negative results: convergence voting catastrophically degrades performance (2/10 vs. 9/10); increasing attempt count under a fixed time budget hurts public LB score; top_p=0.8 has no detectable effect; and a missing function definition causes complete failure. These negative results are reproducible from the source notebooks provided with this writeup.
\nThe most important empirical finding is the dissociation between token-level entropy and answer convergence. The n-Norwegian problem has lower mean token entropy (0.701) than the Tournament problem (0.863), yet Norwegian fails completely while Tournament converges in 5 attempts. This \"confident divergence\" — where the model is locally confident at each step but globally takes a different wrong path each attempt — is invisible to entropy-weighted voting. Entropy voting correctly handles uncertain attempts; it cannot handle confidently-wrong divergent ones. Answer-level diversity monitoring is the missing signal, and building it into the voting function is the most important direction for future work.
\n[1] Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2023). Self-consistency improves chain of thought reasoning in language models. ICLR 2023. arXiv:2203.11171
\n[2] Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023). Efficient memory management for large language model serving with PagedAttention. SOSP 2023.
\n[3] Gou, Z., Shao, Z., Gong, Y., Yang, Y., Huang, M., Duan, N., Chen, W., & Zhang, T. (2023). ToRA: A tool-integrated reasoning agent for mathematical problem solving. arXiv:2309.17452
\n[4] Moshkov, I., et al. (2025). AIMO-2 winning solution: Building state-of-the-art mathematical reasoning models with OpenMathReasoning dataset. arXiv:2504.16891
\n[5] Frieder, S., et al. (2025). AI Mathematical Olympiad — Progress Prize 3. Kaggle Competition. https://kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3
\n[6] David Babu et al. (2026). AMO: Agentic Math Orchestrator for Olympiad Mathematics. AIMO3 Writeup, Kaggle.
\n[7] Nitarach, N. (2026). Model Capability Dominates: Inference-Time Optimization Lessons from AIMO 3. arXiv:2603.27844
\nclass CFG:\n system_prompt = (\n 'You are an elite mathematical problem solver with expertise at the International '\n 'Mathematical Olympiad (IMO) level. Your goal is to find the correct answer through '\n 'rigorous mathematical reasoning.\\n\\n'\n '# Problem-Solving Approach:\\n'\n '1. UNDERSTAND: Carefully read the problem. Identify what is given, '\n 'what needs to be found, and constraints.\\n'\n '2. EXPLORE: Consider multiple solution strategies before committing to one.\\n'\n '3. PLAN: Outline key steps of the most promising approach.\\n'\n '4. EXECUTE: Work through the solution methodically with clear reasoning.\\n'\n '5. VERIFY: Check your answer using substitution, edge cases, '\n 'or alternative methods.\\n\\n'\n '# Output Format:\\n'\n 'The final answer must be a non-negative integer between 0 and 99999.\\n'\n 'Place your final numerical answer inside \\\\boxed{}, e.g., \\\\boxed{42}\\n\\n'\n 'Think step-by-step. Quality of reasoning is as important as the final answer.'\n )\n tool_prompt = (\n 'Use this tool to execute Python code for:\\n'\n '- Complex or error-prone calculations\\n'\n '- Numerical verification of analytical results\\n'\n '- Testing conjectures or brute-force checking small cases\\n\\n'\n 'The environment is a stateful Jupyter notebook. Code persists between executions.\\n'\n 'Always use print() to display results.\\n\\n'\n 'Code should support your reasoning, not replace it.'\n )\n preference_prompt = (\n 'You have access to `math`, `numpy`, `sympy`, and `mpmath`. '\n 'Use sympy for exact symbolic answers, numpy for numerical work, '\n 'and mpmath for high-precision arithmetic when needed.'\n )\n # Model\n served_model_name = 'gpt-oss'\n model_path = '/kaggle/input/models/danielhanchen/gpt-oss-120b/transformers/default/1'\n # vLLM\n dtype = 'auto'\n kv_cache_dtype = 'fp8_e4m3'\n gpu_memory_utilization = 0.95\n context_tokens = 65536\n batch_size = 128\n stream_interval = 200\n # Timing\n notebook_limit = 17400\n high_problem_timeout = 900\n base_problem_timeout = 300\n server_timeout = 180\n session_timeout = 960\n jupyter_timeout = 8\n sandbox_timeout = 3\n # Solver\n attempts = 8\n turns = 128\n workers = 8\n early_stop = 4\n seed = 42\n # Sampling\n temperature = 1.0\n min_p = 0.02\n top_logprobs = 5\n buffer_tokens = 512\n search_tokens = 32\n| Filename | \nDescription | \nLocal /10 | \nLB | \n
|---|---|---|---|
baseline.ipynb | \nFirst version; compute_weighted_entropy undefined → bug | \n0 | \n— | \n
v1.ipynb | \nFull reference run; verbose prompt; mean entropy voting | \n9/10 | \n44 | \n
v2.ipynb | \n12 attempts, early_stop=5, T=0.8/1.2 dual temperature | \n— | \n39 | \n
v3.ipynb | \nV40 prompt, 20 attempts, 16 workers | \n— | \n33 | \n
v4.ipynb | \nV40 + ensemble convergence voting, 20 attempts | \n2/10 | \n35 | \n
v5.ipynb | \nVerbose, top_p=0.8; bug present | \n0 | \n39 | \n
v6.ipynb | \nVerbose, top_p=0.8, 5-component entropy voting | \n8/10 | \n\n |
v1 is the only version submitted to the leaderboard (scored 44, 35, 35 across three runs).
\npatterns = [\n r'\\\\boxed\\s*\\{\\s*([0-9,]+)\\s*\\}', # \\boxed{N}\n r'\\\\\\\\boxed\\s*\\{\\s*([0-9,]+)\\s*\\}', # \\\\boxed{N} (escaped)\n r'final\\s+answer\\s+is\\s*:?\\s*([0-9,]+)', # \"final answer is N\"\n r'answer\\s*[:=]\\s*([0-9,]+)', # \"answer: N\" or \"answer = N\"\n]\n# Scan last search_tokens=32 chunks of streamed text\n# Use last match (reversed order) to prefer the model's most recent answer\n# Validate: 0 ≤ value ≤ 99999\npython -m vllm.entrypoints.openai.api_server \\\n --seed 42 \\\n --model /kaggle/input/models/danielhanchen/gpt-oss-120b/transformers/default/1 \\\n --served-model-name gpt-oss \\\n --tensor-parallel-size 1 \\\n --max-num-seqs 128 \\\n --gpu-memory-utilization 0.95 \\\n --host 0.0.0.0 --port 8000 \\\n --dtype auto \\\n --kv-cache-dtype fp8_e4m3 \\\n --max-model-len 65536 \\\n --stream-interval 200 \\\n --async-scheduling \\\n --disable-log-stats \\\n --enable-prefix-caching \\\n --trust-remote-code\nNote: --enable-auto-tool-choice and --tool-call-parser pythonic are deliberately excluded. These flags configure vLLM's native tool-call parsing layer, which is incompatible with openai_harmony's custom encoding. Including them causes tool calls to be silently mis-parsed, producing empty responses and infinite reasoning loops.
Following up on the questions from @philipvonderlind, here are some additional questions to help us assess your submission:
\nLook forward to reading your responses on these.
", "votes": 0, "replies": [ { "id": 3459144, "author_name": "Lê Bảo", "author_url": "", "post_date": "2026-05-17T11:38:23.737000", "content": "Hi Sam, thank you for the follow-up questions. These are exactly the two open problems that I think matter most after the writeup: whether compute usage can be turned into an online routing signal, and whether answer-level diversity is useful when the model may simply be unable to solve the problem.
\nYes. I think the evidence in the writeup supports this, but I would use the signal for routing and budget protection, not simply for \"more attempts.\"
\nIn Section 6.3, the Norwegian problem is the clear compute outlier:
\n| Problem | \nTotal tokens | \n
|---|---|
| Alice & Bob | \n7,501 | \n
| f(n) = Σgcd | \n19,402 | \n
| Ken blackboard | \n24,125 | \n
| Coprime f(580) | \n31,525 | \n
| Triangle | \n59,553 | \n
| Circumcircle | \n62,328 | \n
| 500×500 | \n62,855 | \n
| Tournament | \n75,327 | \n
| α: Z→Z | \n119,270 | \n
| n-Norwegian | \n322,515 | \n
Norwegian used about 322k tokens — far more than any other reference problem. It also failed to produce strong answer consensus in the run analyzed in Section 6.4. That means the difficulty signal is not only visible after the full run; it likely starts appearing after the first 1–2 completed attempts.
\nA practical online monitor could track:
\n# Evaluated after first 2 completed attempts\n\nvalid_answers = [a for a in answers if a is not None]\n\nmedian_tokens = median(tokens_per_attempt_so_far)\ndistinct_count = len(set(valid_answers))\nmax_vote = max(Counter(valid_answers).values(), default=0)\nvalid_rate = len(valid_answers) / completed_attempts\n\nhigh_cost = median_tokens > 20_000\nno_consensus = len(valid_answers) >= 2 and distinct_count == len(valid_answers)\n\nis_unstable = high_cost and no_consensus\nThe threshold 20_000 is not meant to be universal — it is a reference-set heuristic. The right value should be calibrated from more runs. The important point is the combination of two signals: high per-attempt token usage and no answer agreement. A problem can be expensive and still worth continuing if the answers are converging. The danger case is expensive and diverse.
I would use the signal as a routing table:
\n| Observed after 1–2 attempts | \nInterpretation | \nAction | \n
|---|---|---|
| Low token usage + agreement | \nEasy, stable | \nEarly stop as soon as threshold is met | \n
| High token usage + agreement emerging | \nHard but converging | \nAllow full budget, continue normally | \n
| High token usage + all answers distinct | \nUnstable / confident divergence | \nSwitch strategy or cap budget | \n
| High NA / invalid rate | \nTimeout or infrastructure risk | \nReduce spending or skip | \n
The most important practical constraint is that the global time budget dominates. Section 5.3 of the writeup shows that increasing attempts under a fixed wall-clock budget can reduce public LB score because later problems get starved. So if a problem is flagged as unstable, I would not automatically give it more attempts. Instead, I would route it to one of two branches:
\nBranch A — enough global time remains
\nUse a targeted intervention rather than another full solve:
\nBranch B — global time is tight
\nCap the budget and move on:
\nSo the correct framing is not:
\n\n\ndetect hard problem → spend more
\n
It is:
\n\n\ndetect expensive + non-convergent problem → either verify differently or stop wasting budget
\n
That is the main engineering lesson I would take from the Norwegian problem.
\nI agree with the concern. If 8 attempts produce many distinct answers, there is no guarantee that attempts 9 or 10 will recover the correct answer. The Norwegian analysis suggests that simply increasing N is likely the wrong intervention.
\nThe reason I still think detecting confident divergence is useful is that it improves calibration even when it does not improve recovery.
\nThe failure mode in the writeup is:
\nAs Section 6.4 documents, Norwegian has lower mean token entropy (0.701) than Tournament (0.863), yet Tournament converges unanimously while Norwegian produces 6 distinct answers. Token entropy and answer convergence are measuring different things:
\n| Signal | \nWhat it measures | \nFailure case | \n
|---|---|---|
| Token entropy | \nLocal confidence during generation | \nModel is confidently following a wrong path | \n
| Answer consensus | \nAgreement across attempts | \nModel produces many incompatible answers | \n
| Token usage | \nCompute cost / difficulty signature | \nModel spends large budget without convergence | \n
Entropy-weighted voting can downweight hesitant attempts. It cannot detect a set of confident but mutually inconsistent attempts. That is why answer-level diversity is a complementary signal.
\nA concrete divergence monitor:
\nvalid_answers = [a for a in answers if a is not None]\nvalid_N = len(valid_answers)\n\nmax_vote_count = max(Counter(valid_answers).values(), default=0)\ndistinct_count = len(set(valid_answers))\n\nconsensus = max_vote_count / valid_N if valid_N else 0\ndiversity = distinct_count / valid_N if valid_N else 1\nmean_entropy = mean(entropies)\n\nconfident_divergence = (\n valid_N >= 4\n and consensus <= 0.25\n and diversity >= 0.75\n and mean_entropy < entropy_threshold # locally confident\n)\nThe thresholds should be calibrated from more runs. The qualitative condition is: low entropy + high diversity = locally confident, globally inconsistent.
\nWhen this is detected, I would not trust the normal entropy-weighted vote and would instead trigger one of the following actions, in order of preference:
\nAction 1 — Candidate verification
\nTake the top-2 or top-3 candidate answers and ask the model to check each against the problem's defining constraints. This is structurally easier than re-deriving from scratch — the model only needs to discriminate between candidates rather than reconstruct a full solution path.
\nAction 2 — Solution-path comparison
\nAsk the model to compare the assumptions behind two conflicting derivations and identify where they diverge. This targets the divergence directly rather than ignoring it.
\nAction 3 — Executable checker
\nWhen the problem structure permits it (number theory, combinatorics), write code to test the defining property on candidate answers. Not always feasible for IMO-level problems, but when available it is a better use of compute than another full reasoning attempt.
\nAction 4 — Budget protection
\nIf verification is not feasible and time is tight, stop spending. Return the current best answer flagged as low-consensus and preserve budget for downstream problems.
\nThis is why detecting confident divergence is useful even when the model cannot reliably recover. It lets the solver distinguish between:
\n| Case | \nExample behavior | \nHow to treat answer | \n
|---|---|---|
| Stable majority | \n4+ attempts agree on same answer | \nHigh confidence | \n
| Uncertain minority | \nOne high-entropy outlier | \nDownweight by entropy | \n
| Confident divergence | \nMany low-entropy distinct answers | \nLow confidence despite entropy | \n
| No valid answers | \nNA / timeouts throughout | \nFallback / budget cap | \n
The Norwegian problem sits in the third category.
\nThe cross-run stability analysis in Section 6.5 supports this interpretation. The same v1 configuration was stable on 9 of 10 reference problems across two independent runs, while Norwegian changed between runs — one run selected the reference answer 8687, another selected a different answer. This suggests the model can sometimes reach the correct answer, but the standard voting signal is not reliable enough to know when it has done so. Detecting divergence would at minimum flag the returned answer as unreliable rather than giving it the same apparent confidence as a stable 4-vote majority.
To summarize:
\nThe practical intervention should be verification, candidate comparison, or budget capping — not blindly increasing attempts.
\nIn hindsight, the voting layer should have used at least four signals together:
\nconfidence = 1 / mean_token_entropy # local confidence per attempt\nconsensus = max_vote_count / valid_N # cross-attempt agreement\ndiversity = distinct_answers / valid_N # cross-attempt disagreement\ncost = tokens_used_so_far # budget pressure\nThe submitted pipeline used entropy and majority voting but did not explicitly track answer diversity or token-cost routing. That is the main change I would make in a next version.
", "votes": 0, "replies": [] } ] }, { "id": 3457719, "author_name": "Philip Vonderlind", "author_url": "", "post_date": "2026-05-14T08:25:41.663000", "content": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.
\nI have a few questions regarding your work:
\nComparison to Public Notebook: Some parts of your submission appear similar to the public notebook shared in the discussion, which many participants used as a starting point. While building on others' work is certainly encouraged, it would be helpful if you could cite the original notebook, include a link to it, and clearly describe which parts you adapted, as well as the motivation behind each change.
Plots: Adding plots to summarize your results, especially the ensemble consensus mechanism, would improve both the readability and presentation of your work. For example, it would be useful to include plots comparing the performance of all ablations on both the public leaderboard and your local evaluation set.
Experiment Construction: In many of your experiments, several changes appear to be introduced at the same time, which makes the results harder to interpret. Do you have results from more isolated experiments where only one component is changed at a time? This would make it easier to attribute score changes to specific modifications and to build on those findings in future work.
Looking forward to your response :)
", "votes": 0, "replies": [ { "id": 3459123, "author_name": "Lê Bảo", "author_url": "", "post_date": "2026-05-17T11:14:31.037000", "content": "Hi Philip, thank you for the detailed questions. I agree with all three concerns: attribution to the public starting point should have been clearer, the results would be easier to understand with plots, and many of my experiments are practical engineering iterations rather than clean single-variable ablations.
\nMy work started from the public HCMUS Group 4 notebook, which I used as the initial runnable baseline. I also used the public AIMO 3 Utils notebook by andreasbis for offline dependency packaging, especially the vllm and openai_harmony wheels.
I should have cited both of these explicitly in the writeup.
\nThe HCMUS baseline already provided the main competition-compatible pipeline:
\ngpt-oss-120b served through vLLM;openai_harmony formatting;So the most accurate description of my work is not “a new pipeline from scratch,” but an inference-time engineering and evaluation study built on top of that public baseline.
\nThe parts I adapted directly were:
\n| Component | \nNotes | \n
|---|---|
vLLM serving of gpt-oss-120b | \nkept the same general serving approach | \n
openai_harmony conversation format | \nkept because it is the expected format for gpt-oss | \n
| persistent Jupyter kernel tool-use | \nkept because stateful tool use is important for multi-step math/code reasoning | \n
| parallel attempts + early stopping | \nkept the same general self-consistency structure | \n
| entropy-weighted voting | \nkept as the main aggregation method | \n
| Kaggle offline dependency setup | \nused the public utility input by andreasbis | \n
My own work was mainly in changing, testing, and analyzing the inference-time configuration around this baseline: prompt variants, attempt count, worker count, batch size, temperature, convergence-voting variants, intended top_p variants, reference-set reruns, public LB submissions, per-problem token analysis, and failure-mode analysis around answer diversity vs. entropy.
The main changes and motivations were:
\n| Version | \nChange | \nMotivation | \nResult / interpretation | \n
|---|---|---|---|
baseline | \nfirst version with missing entropy function bug | \ninitial run | \nfailed due implementation bug, not model quality | \n
v1 | \n8 attempts, entropy voting, verbose prompt | \nstable baseline under time budget | \nbest public LB result | \n
v2 | \n12 attempts, early_stop=5, lower temperature=0.8 | \ntest whether more samples + lower randomness improves reliability | \npublic LB decreased, likely due time-budget pressure and confounded changes | \n
v3 | \nconcise V40 prompt, 20 attempts, 16 workers, larger batch | \ntest whether shorter prompt + more self-consistency improves score | \npublic LB decreased; too expensive under fixed budget | \n
v4 | \nconvergence / ensemble voting family | \ntest whether answer-distribution convergence is better than majority voting | \nstrong negative result locally | \n
v5/v6 | \nintended top_p=0.8 variants | \ntest whether tail-token filtering improves math reliability | \nweak/no clear evidence; in the code, top_p was declared in CFG but not passed to completions.create, so this experiment did not cleanly test top_p | \n
The key lesson is that increasing inference complexity did not automatically help under the 5-hour budget. The best-performing setup was comparatively simple: 8 attempts, early stop at 4, entropy-weighted majority voting.
\nI agree that the writeup would have been much easier to read with plots. Since the writeup has already been submitted, I cannot insert them into the original text, but these are the supplemental plots I prepared after your comment.
\nPublic LB score by submitted notebook version
\nThis summarizes the practical outcome of the submitted configurations: v1, v2, v3, v4, and v5. It also makes clear that these are engineering iterations, not clean one-variable ablations.
Notebook version change matrix
\nThis plot is meant to make the confounding explicit. Most versions change multiple dimensions at once, which is why I should not present them as clean single-variable ablations.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F29262272%2F0b053c7e132e4f3331cfa60ae52f6480%2F05_ablation_change_matrix.png?generation=1779016183515576&alt=media\")
Per-problem compute signature
\nThis shows Norwegian as the clear compute outlier and motivates the token-count difficulty signal.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F29262272%2F56448d59520aa1bb4e3478b4d3f762c3%2F03_problem_token_usage.png?generation=1779015315445959&alt=media\")
Answer diversity vs. consensus
\nThis replaces the table I should have included for the ensemble/consensus discussion. Stable problems cluster around high consensus and low diversity, while Norwegian is the high-diversity / low-consensus outlier.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F29262272%2F4b167e956abcaab2ec95c71b25fd27a1%2F06_consensus_diversity_scatter.png?generation=1779016203985032&alt=media\")
I agree that many experiments changed multiple things at once. This is the biggest limitation of the study.
\nBecause I joined with only 7 days remaining and had a limited submission budget, I treated the notebook versions as practical engineering iterations rather than clean scientific ablations. In hindsight, the writeup should have separated these two categories more explicitly:
\n| Type | \nExamples | \nHow much can be concluded? | \n
|---|---|---|
| repeated same-config check | \nrepeated reference run of v1 | \nuseful for run-to-run stability | \n
| practical engineering iteration | \nv2, v3, v4, v5, v6 | \nuseful negative results, but confounded | \n
| clean one-variable ablation | \nmostly not available | \nlimited by time/submission budget | \n
The closest I have to isolated evidence is the repeated reference-set run of the same v1 configuration discussed in Section 6.5. That checks stability rather than a component change: 9 of 10 reference problems were stable across runs, while the Norwegian problem changed.
For the version comparisons:
\nv2 vs. v1 is not isolated because it changes attempt count, early-stop threshold, and temperature.v3 vs. v1 is not isolated because it changes prompt style, attempts, workers, batch size, and early stopping.v4 should not be interpreted as “only voting changed from v1.” It belongs to the V40 / 20-attempt family and adds convergence / ensemble voting and related routing logic.v5/v6 were intended to test the top_p=0.8 direction, but because top_p was not actually passed into the completion call, this does not support a strong conclusion about top_p.So the safe conclusions are:
\nv1-style entropy-voting setup was the strongest practical configuration I found;top_p, or worker count.If I were repeating the study with more time, I would run a proper ablation grid:
\n| Experiment | \nKeep fixed | \nChange only | \n
|---|---|---|
| prompt ablation | \nN=8, T=1.0, same voting | \nverbose vs concise prompt | \n
| temperature ablation | \nN=8, same prompt/voting | \nT=1.0 vs T=0.8 | \n
| attempt-count ablation | \nsame prompt/voting/T | \nN=8 vs N=12 vs N=20 | \n
| voting ablation | \nsame prompt/N/T | \nmajority vs entropy-weighted vs convergence | \n
| worker/batch ablation | \nsame solver config | \nworkers/batch-size only | \n
That would make attribution much cleaner than the constrained 7-day experiment I was able to run.
", "votes": 0, "replies": [] } ] }, { "id": 3447425, "author_name": "ayushokaay", "author_url": "", "post_date": "2026-04-23T05:29:22.890000", "content": "looks like lot of efforts @heon29 nice work!
", "votes": 0, "replies": [] } ], "comment_count": 5, "messages": [ { "id": 3446856, "postDate": "2026-04-22T08:51:10.923Z", "content": "Author: Lê Bảo
\nCompetition: AI Mathematical Olympiad — Progress Prize 3 (AIMO3)
\nNotebook versions: 7 experimental configurations, 5 leaderboard submissions
| Resource | \nLink | \n
|---|---|
| Kaggle Notebook (submitted) | \nnotebook | \n
| Colab Notebook (simplified) | \ncolab | \n
| Base Model | \ngpt-oss-120b | \n
The submitted Kaggle notebook is open-sourced and publicly viewable. A simplified Colab version is provided as supplementary material for readers without Kaggle access — it implements the same inference pipeline using the public HuggingFace model weights. No fine-tuning was performed; the contribution is purely an inference pipeline on top of the public base model.
\nWe report an inference-time engineering study for AIMO3, conducted under severe time pressure — joining with 7 days remaining and a 5-submission budget. Using danielhanchen/gpt-oss-120b (116.8B total / ~5.1B active parameters via Mixture-of-Experts) served through vLLM 0.11.2 with FP8 quantization, our best submission achieves 44/50 on the public leaderboard; two identical resubmissions of the same notebook scored 35 and 35, indicating substantial run-to-run variance. Our system runs N=8 parallel tool-integrated reasoning attempts aggregated by entropy-weighted majority voting with early stopping.
Across 7 notebook versions we test four dimensions of variation: (1) system prompt style — a verbose 5-stage IMO scaffold vs. a 3-line concise prompt; (2) sampling parameters — temperature (0.8 vs. 1.0) and top_p (absent vs. 0.8); (3) voting strategy — entropy-weighted majority vs. ensemble convergence; (4) attempt count — 8 vs. 12 vs. 20. Key findings: the verbose structured prompt outperforms the concise prompt; ensemble convergence voting catastrophically degrades performance (2/10 vs. 9/10 on the reference set); temperature reduction from 1.0 to 0.8 combined with more attempts degrades public LB from 44 to 39; top_p=0.8 has no measurable effect; a missing function definition caused 0/10 scores on two notebook versions before being diagnosed and fixed.
\nWe also document EOS-resilient retry logic — handling transient vLLM connection failures that affected 6 of 10 reference problems across both runs. Per-problem analysis across two independent reference-set runs reveals a critical finding: token-level entropy does not predict answer convergence. The n-Norwegian problem has lower mean token entropy (0.701) than the Tournament problem (0.863), yet Norwegian produces 6 completely distinct answers across 8 attempts while Tournament converges unanimously in 5. This reveals that a model can be locally confident at each reasoning step while globally taking a different wrong path each attempt — a failure mode invisible to entropy-based voting. The n-Norwegian problem also consumes 43× more compute than the average problem, with variance concentrated entirely in this single problem across both reference runs.
\nAIMO3 requires solving 50 International Mathematical Olympiad-level problems on a single NVIDIA H100 80GB GPU within a 5-hour wall-clock budget. Problems span algebra, combinatorics, geometry, and number theory. All answers are non-negative integers in [0, 99999], evaluated by exact match. The competition provides a 10-problem reference set (reference.csv) for local development; the remaining 40 problems constitute the private test set.
The primary constraint is not model quality but compute allocation: with 50 problems and a 17,400-second (4h50m) notebook limit, each problem receives at most 348 seconds on average, far less than the 900-second maximum. Effective adaptive time budgeting is therefore a first-order engineering concern alongside model configuration.
\nAIMO3 represents a significant escalation over its predecessors. The AIMO2 winning solution (NemoSkills, Moshkov et al. [4]) achieved 34/50 using a fine-tuned 14B model on 4×L4 GPUs with 540K training problems and custom GenSelect inference. AIMO3 provides H100 hardware, enabling gpt-oss-120b inference without fine-tuning — shifting the landscape from training-time compute to inference-time engineering.
\nWe joined AIMO3 with 7 days remaining before the April 15 deadline. This severely limited the number of controlled experiments we could run — 5 leaderboard submissions over 7 days, with local reference-set evaluation as the primary development signal. Our approach was targeted engineering based on hypotheses rather than broad search.
\nThis context affects how we interpret results: several ablations are confounded (multiple variables changed simultaneously), single-run comparisons have high variance, and we lack the noise floor characterization that would come from repeated identical submissions. We document these limitations explicitly throughout.
\nWhy this architecture matters: AIMO3 presents a unique constraint: 50 problems, 5 hours, single GPU. The solution is not about finding the best model, but about efficiently allocating limited compute across problems. Our architecture prioritizes (1) making hard problems get enough time, (2) not wasting time on easy problems that converge quickly, and (3) handling infrastructure failures gracefully.
\nWe serve danielhanchen/gpt-oss-120b through a local vLLM 0.11.2 server using the proprietary openai_harmony protocol. Model weights (~65 GB in FP8) are pre-loaded into OS page cache via parallel sequential reads before server launch, reducing cold-start latency from ~240s to ~116s.
| Parameter | \nValue | \nNotes | \n
|---|---|---|
| Model | \ndanielhanchen/gpt-oss-120b | \n\n |
| Total / active params | \n116.8B / ~5.1B | \nMixture-of-Experts | \n
| Weight format | \nFP8 (MXFP4 E2M1) | \n\n |
| KV cache | \nfp8_e4m3 | \n\n |
| Engine | \nvLLM 0.11.2 | \n\n |
| Tensor parallel | \n1 | \nSingle H100 | \n
| batch_size (max_num_seqs) | \n128 | \nBaseline config | \n
| gpu_memory_utilization | \n0.95 | \n\n |
| context_tokens (max_model_len) | \n65,536 | \n\n |
| stream_interval | \n200 | \n\n |
| Key flags | \n--async-scheduling --enable-prefix-caching --trust-remote-code | \n\n |
The server is started as a subprocess and polled until /v1/models responds (180-second timeout). All server output is written to vllm_server.log for debugging.
Why five stages: We decomposed the inference problem into a pipeline because each stage has different resource requirements and failure modes. Adaptive budget allocation needs to know the problem sequence; parallel attempts need independent seeds; early stopping needs to track answer consensus; entropy voting needs per-attempt confidence scores.
\nStage 1 — Adaptive budget allocation. Per-problem time budget:
\nreserved = max(0, problems_remaining - 1) * base_problem_timeout # 300s each\nbudget = max(base_timeout, min(high_timeout, notebook_limit - elapsed - reserved))\nWith base_timeout=300s, high_timeout=900s, notebook_limit=17400s: early problems receive up to 900s when time is abundant; later problems compress toward 300s as the notebook limit approaches. This avoids both under-serving hard early problems and running out of time for late problems.
Stage 2 — Parallel attempts. N=8 inference threads dispatched concurrently, each with a distinct seed computed as attempt_seed = int((seed + i)²) mod 2³¹ for i ∈ [0, 7]. Temperature T=1.0 with min_p=0.02 — the minimum token probability threshold that prevents very low-probability tokens from being sampled.
Stage 3 — Tool-integrated reasoning. Each attempt operates within an openai_harmony Harmony conversation loop, implementing Tool-Integrated Reasoning (TIR) [3]. AIMO3Template constructs a SystemContent with ReasoningEffort.HIGH, attaching the tool namespace for the Python sandbox. The model alternates among three channels:
assistant: reasoning and code generationpython (tool): code execution, result returned to conversation final: terminal channel where the answer is emittedStage 4 — Early stopping. A shared threading.Event is set as soon as 4 attempts agree on the same answer. Remaining futures are cancelled. This saves substantial compute on problems where the model converges quickly — in our reference runs, 8 of 9 correctly-solved problems reached early stop within 4–5 attempts, conserving approximately 3–4 attempt-slots of compute each.
Stage 5 — Entropy-weighted voting. Each attempt's answer is weighted by inverse mean token-level Shannon entropy:
\n$$\\hat{a} = \\arg\\max_a \\sum_{i: ans_i = a} \\frac{1}{\\max(H_i, \\epsilon)}$$
\nwhere $H_i$ is the mean entropy computed from top-5 logprobs:
\n$$H_i = \\frac{1}{|T_i|} \\sum_{t \\in T_i} \\left(-\\sum_k p_{t,k} \\log_2 p_{t,k}\\right)$$
\nLower entropy = higher confidence = higher vote weight.
\nWhy entropy weighting: Standard majority voting treats all attempts equally. But when the model generates a confident response (low entropy), it should carry more weight than a hesitant response (high entropy). This is especially important when some attempts take a \"shortcut\" to a wrong answer — those tend to have higher entropy because the model is less certain.
\nWhy persistent kernels: Tool-integrated reasoning requires stateful execution — the model needs to define variables in one code cell and use them in the next. Standard Python subprocess execution loses state between calls. Persistent kernels solve this by maintaining a running Python interpreter.
\nWe maintain 8 persistent Jupyter kernels, one allocated per parallel attempt. Kernels are initialized at notebook startup (parallel, ~2 seconds total) with:
\nimport math, numpy, sympy, itertools, collections, mpmath\nmpmath.mp.dps = 64 # 64 decimal places of precision\nKernel ports are allocated dynamically from 50000+ using a class-level lock for thread safety. State persists across code cells within a single attempt — variables from earlier executions survive into later ones, mimicking a real Jupyter session. Between problems, kernels are reset via %reset -f followed by re-importing the standard libraries.
AIMO3Tool._ensure_last_print() auto-wraps the final line of model-generated code with print() when it does not already produce output. This prevents silent failures when the model writes a bare expression rather than an explicit print statement — a common pattern in exploratory mathematical code.
During inference, vLLM occasionally drops connections mid-stream. These errors manifest as exceptions at the Python HTTP client layer with messages including \"Unexpected EOS while waiting for message header\", \"connection reset\", and \"broken pipe\". Without retry, each such error silently discards an entire attempt — reducing effective N and degrading voting quality.
We implement retry with exponential backoff inside _run_one_turn(), the function that issues a single completion request and streams the result:
_EOS_MESSAGES = (\n 'unexpected eos', 'waiting for message header',\n 'connection reset', 'connection aborted',\n 'remote end closed', 'empty response',\n 'peer closed', 'broken pipe',\n)\n_MAX_EOS_RETRIES = 3\n_EOS_BACKOFF = [1.0, 2.0, 4.0] # seconds before retries 1, 2, 3\n\n@staticmethod\ndef _is_eos_error(exc: Exception) -> bool:\n return any(kw in str(exc).lower() for kw in AIMO3Solver._EOS_MESSAGES)\nWhen an EOS-class exception is caught, the method waits for the specified backoff duration and re-issues the identical completion request (same seed, same conversation state). Non-EOS exceptions propagate immediately without retry.
\nEvidence of effectiveness. In our two reference-set runs (10 problems × 2 runs = 20 problem-run pairs), EOS errors affected 6 problems in Run 1 and 6 in Run 2, for a total of approximately 13 and 9 error events respectively. The per-problem breakdown for Run 1:
\n| Problem | \nEOS Errors | \nAttempts Affected | \nFinal Answer | \n
|---|---|---|---|
| Tournament | \n1 | \nAttempt 7 | \n21818 ✓ | \n
| Coprime f(580) | \n3 | \nAttempts 1, 3, 7 | \n580 ✓ | \n
| Alice & Bob | \n3 | \nAttempts 4, 5, 8 | \n50 ✓ | \n
| f(n)=Σgcd | \n3 | \nAttempts 1, 6, 7 | \n32951 ✓ | \n
| α: Z→Z | \n1 | \nAttempt 5 | \n160 ✓ | \n
| Circumcircle | \n2 | \nAttempts 6, 7 | \n57447 ✓ | \n
In every case where an attempt experienced an EOS error, the retry mechanism allowed the attempt to eventually contribute a valid answer. The total overhead is at most 7 seconds per recovered attempt (1+2+4s), negligible against a 300–900-second per-problem budget.
\nThe key architectural insight is that without retry, these 13 error events would each consume an attempt slot while producing None as the answer — silently reducing effective N from 8 toward 2 on some problems, potentially shifting the vote from correct to incorrect.
Why this section matters: Voting is where the rubber meets the road — even a perfect model can fail if the voting strategy picks the wrong answer. We tested two approaches: entropy-weighted majority voting (our winner) and ensemble convergence voting (catastrophic failure).
\nThe voting function weights each attempt by the inverse of its mean token-level Shannon entropy. Lower entropy = higher confidence = higher vote weight. This extends the self-consistency voting approach of Wang et al. [1] by weighting votes rather than treating them uniformly. This is used in all notebook versions (baseline, v1, v2, v5, v6).
\nThe answer with the highest cumulative weight wins. If no answer reaches the early_stop threshold (4 agreeing votes), the highest-weighted answer is returned.
\nIn notebook v4 we replaced entropy voting with a more complex ensemble that monitored answer distribution stability. The implementation combined dual-temperature sampling (T=0.8 primary, T=1.2 secondary), per-answer running statistics (votes, entropy weight, position weight), a convergence detector that stopped when the running mean stabilized, and a keyword-based difficulty classifier routing each problem to an easy (150s), medium (300s), or hard (900s) time budget.
\nResult: 2/10 on the reference set. The ensemble produced \"1\" or \"2\" as answers to most hard problems. The failure has two distinct mechanisms:
\nMechanism 1 — Lock-on to first wrong answer: On hard problems with high answer diversity, the first answer to appear in the running distribution gets a head start. The convergence criterion then stabilizes around this first answer rather than waiting for a genuine majority. Majority voting avoids this by requiring a threshold count, not distribution stability.
\nMechanism 2 — Difficulty misclassification: The keyword classifier is heuristic — keywords like \"circle,\" \"integer,\" \"divisor\" trigger \"hard\" classification, while simpler-seeming problems might be routed to 150 seconds. A 150-second budget for even a moderately complex problem produces 0 valid attempts, giving a vote of zero on that problem.
\nThe comparison is stark: entropy voting with N=8, T=1.0, verbose prompt → 9/10; convergence voting with N=20, T=0.8, V40 prompt → 2/10. This 7-problem regression validates entropy-weighted majority voting as the correct aggregation strategy for this problem distribution.
\nWhy prompt design matters: The system prompt shapes how the model approaches every problem. A good prompt guides the model through structured reasoning (UNDERSTAND → EXPLORE → PLAN → EXECUTE → VERIFY), reducing the chance of jumping directly to a wrong answer. We tested two extremes: verbose 5-stage vs. concise 3-line.
\nUsed in baseline, v1, v2, v5, v6:
\nYou are an elite mathematical problem solver with expertise at the International\nMathematical Olympiad (IMO) level. Your goal is to find the correct answer through\nrigorous mathematical reasoning.\n\n# Problem-Solving Approach:\n1. UNDERSTAND: Carefully read the problem. Identify what is given,\n what needs to be found, and constraints.\n2. EXPLORE: Consider multiple solution strategies before committing to one.\n3. PLAN: Outline key steps of the most promising approach.\n4. EXECUTE: Work through the solution methodically with clear reasoning.\n5. VERIFY: Check your answer using substitution, edge cases,\n or alternative methods.\n\n# Output Format:\nThe final answer must be a non-negative integer between 0 and 99999.\nPlace your final numerical answer inside \\boxed{}, e.g., \\boxed{42}\n\nThink step-by-step. Quality of reasoning is as important as the final answer.\nUsed in v3, v4:
\nYou are a world-class IMO competitor.\nThe final answer must be 0-99999.\nPlace answer inside \\boxed{}.\nThe motivation for V40 was that verbose instructions might add unnecessary cognitive overhead, causing the model to spend tokens on structural compliance rather than mathematical reasoning.
\nThe cleanest evidence comes from the reference set: v1 (verbose prompt) achieves 9/10; v4 (V40 prompt + convergence voting) achieves 2/10. However, this comparison is entirely confounded — v4 changes prompt, temperature, voting strategy, attempt count, and early_stop simultaneously. We cannot attribute the regression to the prompt.
\nThe only semi-controlled comparison is v6 (verbose, top_p=0.8, mean entropy) scoring 8/10 vs. v1 (verbose, no top_p, mean entropy) scoring 9/10. This suggests the verbose prompt is not harmful, but the prompt × voting interaction is not separable from our data.
\nTentative conclusion: The verbose 5-stage prompt is at minimum neutral and likely beneficial by ~1 problem on the reference set. The structured UNDERSTAND/EXPLORE/PLAN/EXECUTE/VERIFY scaffold appears to organize the model's reasoning into distinct phases, reducing the likelihood of jumping directly to an incorrect answer without exploration.
\nWhy ablation matters: We joined late with only 5 submissions. Every change had to be justified by theory and tested on the reference set. The ablations below document what we learned — including what didn't work.
\n| Version | \nPrompt | \nTemp | \ntop_p | \nVoting | \nAttempts | \nWorkers | \nEarly Stop | \nLocal /10 | \nLB Score | \n
|---|---|---|---|---|---|---|---|---|---|
| baseline | \nVerbose | \n1.0 | \n— | \nMean entropy | \n8 | \n8 | \n4 | \n0 (bug) | \n— | \n
| v1 | \nVerbose | \n1.0 | \n— | \nMean entropy | \n8 | \n8 | \n4 | \n9/10 | \n44 | \n
| v2 | \nVerbose | \n0.8 | \n— | \nMean entropy | \n12 | \n8 | \n5 | \n— | \n39 | \n
| v3 | \nV40 | \n0.8+1.2 | \n— | \nMean entropy | \n20 | \n16 | \n7 | \n— | \n33 | \n
| v4 | \nV40 | \n0.8+1.2 | \n— | \nConvergence | \n20 | \n16 | \n7 | \n2/10 | \n35 | \n
| v5 | \nVerbose | \n1.0 | \n0.8 | \nMean entropy | \n8 | \n8 | \n4 | \n8/10 | \n39 | \n
| v6 | \nVerbose | \n1.0 | \n0.8 | \nMean entropy | \n8 | \n8 | \n4 | \n8/10 | \n— | \n
Additional versions (baseline, v5) contained a missing function bug (0/10). v6 tested top_p=0.8 without submission (8/10 local). Only v1 was submitted to the leaderboard multiple times.
\nv4 scores 2/10 vs. 9/10 for v1. As documented in Section 3.2, the failure is systematic and attributable to two distinct mechanisms: lock-on to first wrong answer and difficulty misclassification. This is the single most important finding: for this model and problem distribution, entropy-weighted majority voting is strongly preferred over convergence-based ensemble methods.
\nv2 raises attempts from 8 to 12 and early_stop from 4 to 5, and reduces temperature from 1.0 to 0.8. Public LB drops from 44 to 39. v3 raises attempts to 20, drops to 33. The regression has a clear mechanism: more attempts per problem means less time per problem on average. When problems are ordered in a fixed sequence and some problems consume their full 900-second budget, later problems in a high-attempt configuration may receive only 60–90 seconds — insufficient for any valid attempt to complete. Problems that timeout return 0.
\nThe expected score under a time budget constraint is:
\n$$E[score] = \\sum_{i=1}^{50} p_i \\cdot \\mathbf{1}[problem_i \\text{ does not timeout}]$$
\nIncreasing N from 8 to 20 reduces the second factor even as it might increase the first — and the overall effect is negative.
\nLesson: Under a hard wall-clock constraint, the optimal N is not \"maximize\" but \"fit within the time budget across all 50 problems.\" N=8 with early stopping appears near-optimal for a 17,400-second budget on this model and problem distribution.
\nv5 and v6 add top_p=0.8 (not set in v1, which uses the vLLM default of 1.0). v6 (which fixes the bug in v5) scores 8/10 vs. 9/10 for v1. The single additional error is the n-Norwegian problem, which v1 also gets wrong. top_p=0.8 does not detectably change accuracy for this model at T=1.0.
This is consistent with theoretical expectations: min_p=0.02 already filters out very low-probability tokens; top_p=0.8 at T=1.0 adds additional filtering of the low end of the distribution. For a model as large as gpt-oss-120b, the marginal tokens filtered by top_p=0.8 are unlikely to be relevant to mathematical reasoning quality.
The baseline and v5 notebooks reference compute_weighted_entropy() in the voting code path, but the function is not defined in those versions. Every call falls through to the return 0 fallback, producing all-zero scores regardless of model output.
The diagnosis required reading the exception traceback carefully — the model was generating valid answers, but the voting function silently returned 0 before they were compared. Fixing this by ensuring the entropy function is properly defined and wired into the voting path restored normal scoring.
\nLesson: Before any leaderboard submission, run the complete pipeline on at least one reference problem and verify that the voting function returns a finite, non-zero value and that the predicted answer matches the model's actual output.
\n| Submission | \nVersion | \nPublic LB | \nKey Difference from v1 | \n
|---|---|---|---|
| 1 | \nv1 | \n44 | \n— (best config) | \n
| 2 | \nv2 | \n39 | \n12 attempts, early_stop=5, T=0.8 | \n
| 3 | \nv3 | \n33 | \nV40 prompt, 20 attempts, 16 workers | \n
| 4 | \nv1 | \n42 | \nRe-submission (variance) | \n
| 5 | \nv1 | \n39 | \nRe-submission (variance) | \n
The two re-submissions of v1 scoring 42 (vs. the original 44) illustrate the run-to-run variance inherent in stochastic sampling at T=1.0. Without repeated runs, we cannot establish the true mean — 44 could be a favorable sample from a distribution centered around 39–40, or the configuration could have a genuine mean near 44. The two reference-set runs (both 9/10) provide weak evidence of stability on the reference problems, but cannot bound competition-set variance.
\n| # | \nProblem Description | \nGT | \nAnswer | \nVotes | \nEarly Stop | \nCorrect | \n
|---|---|---|---|---|---|---|
| 1 | \n500×500 square, k rectangles | \n520 | \n520 | \n4 | \n✓ | \n✓ | \n
| 2 | \nTournament, 2²⁰ runners | \n21818 | \n21818 | \n4 | \n✓ | \n✓ | \n
| 3 | \nf: Z≥1→Z≥1, coprime condition | \n580 | \n580 | \n4 | \n✓ | \n✓ | \n
| 4 | \nAcute triangle ABC, integer sides | \n336 | \n336 | \n4 | \n✓ | \n✓ | \n
| 5 | \nAlice & Bob sweets | \n50 | \n50 | \n4 | \n✓ | \n✓ | \n
| 6 | \nKen blackboard, integer n | \n32193 | \n32193 | \n4 | \n✓ | \n✓ | \n
| 7 | \nf(n) = Σ gcd(i,n), i=1..n | \n32951 | \n32951 | \n4 | \n✓ | \n✓ | \n
| 8 | \nα: Z→Z, finitely nonzero | \n160 | \n160 | \n4 | \n✓ | \n✓ | \n
| 9 | \nTriangle ABC, circumcircle Ω | \n57447 | \n57447 | \n4 | \n✓ | \n✓ | \n
| 10 | \nn-Norwegian numbers | \n? | \n8687 | \n1 | \n✗ | \n? | \n
Problem 10 is the only one without early stop. The ground truth for the Norwegian problem in the reference set context may differ from what the writeup suggests — the community discussion indicates 8687 is a plausible answer for the specific parameterization of this problem.
\nA key finding from our per-problem analysis is the extreme variance in compute consumption:
\n| Problem | \nTotal Tokens | \nRelative Cost | \n
|---|---|---|
| Alice & Bob | \n7,501 | \n1× | \n
| f(n) = Σgcd | \n19,402 | \n2.6× | \n
| Ken blackboard | \n24,125 | \n3.2× | \n
| Coprime f(580) | \n31,525 | \n4.2× | \n
| Circumcircle | \n62,328 | \n8.3× | \n
| 500×500 | \n62,855 | \n8.4× | \n
| Triangle (336) | \n59,553 | \n7.9× | \n
| Tournament | \n75,327 | \n10× | \n
| α: Z→Z | \n119,270 | \n15.9× | \n
| n-Norwegian | \n322,515 | \n43× | \n
The n-Norwegian problem consumes 43× the compute of Alice & Bob and more than the next two hardest problems combined. Under a time budget, this problem represents a significant drain — if it appears early in the sequence, it leaves less budget for subsequent problems.
\n| Attempt | \nTokens | \nPython Calls | \nErrors | \nEntropy | \nAnswer | \n
|---|---|---|---|---|---|
| 1 | \n14,287 | \n15 | \n2 | \n0.813 | \n21818 | \n
| 2 | \n14,756 | \n16 | \n1 | \n0.887 | \n21818 | \n
| 3 | \n15,259 | \n5 | \n0 | \n0.938 | \n62140 | \n
| 4 | \n15,316 | \n13 | \n2 | \n0.854 | \n21818 | \n
| 5 | \n15,709 | \n16 | \n3 | \n0.822 | \n21818 | \n
Attempt 3 produces the wrong answer (62140) with the highest entropy in the group (0.938). Entropy weighting correctly reduces its influence: inverse weight 1/0.938 = 1.066 vs. 1/0.813 = 1.230 for the most confident correct attempt. Early stop at 5 correct votes. The entropy voting function performs exactly as designed here — the wrong answer comes from the least confident attempt.
\n| Attempt | \nTokens | \nPython Calls | \nErrors | \nEntropy | \nAnswer | \n
|---|---|---|---|---|---|
| 1 | \n7,601 | \n0 | \n0 | \n0.861 | \n2 | \n
| 2 | \n15,280 | \n10 | \n2 | \n0.777 | \n160 | \n
| 3 | \n16,385 | \n11 | \n3 | \n0.814 | \n114 | \n
| 4 | \n18,871 | \n32 | \n7 | \n0.705 | \n160 | \n
| 5 | \n16,921 | \n23 | \n5 | \n0.780 | \n266 | \n
| 6 | \n19,892 | \n21 | \n3 | \n0.801 | \n160 | \n
| 7 | \n24,320 | \n24 | \n1 | \n0.808 | \n160 | \n
Attempt 1 reaches an answer (2) after only 7,601 tokens with 0 Python calls — the model jumped directly to a conclusion without exploration. This attempt has high entropy (0.861) and produces a wrong answer. Attempts 2, 4, 6, 7 all produce the correct answer (160) through extensive tool use (10–32 Python calls). Early stop at 7 correct votes. This illustrates why early stopping requires 4 agreeing votes rather than stopping after any 4 attempts — on this problem, the first attempt's fast-but-wrong answer would have derailed a less robust stopping criterion.
\n| Attempt | \nTokens | \nPython Calls | \nErrors | \nEntropy | \nAnswer | \n
|---|---|---|---|---|---|
| 1 | \n24,757 | \n22 | \n2 | \n0.731 | \n6825 | \n
| 2 | \n30,388 | \n17 | \n3 | \n0.665 | \n24433 | \n
| 3 | \n29,426 | \n15 | \n3 | \n0.758 | \n88272 | \n
| 4 | \n32,952 | \n44 | \n3 | \n0.678 | \n96985 | \n
| 5 | \n42,942 | \n60 | \n2 | \n0.659 | \n8687 | \n
| 6 | \n43,158 | \n66 | \n5 | \n0.701 | \n36162 | \n
| 7 | \n56,708 | \n58 | \n7 | \n0.690 | \nNA | \n
| 8 | \n62,184 | \n42 | \n4 | \n0.725 | \nNA | \n
Total tokens: 322,515 — 3–10× more than any other problem. All 6 non-NA answers are distinct. The model exhausts the full 8 attempts without any consensus. The model approaches the problem via different algorithmic paths in each attempt (different enumeration strategies, different edge-case handling), each confident in its own path, arriving at different wrong answers.
\nThe critical observation: token entropy does not predict convergence failure. The mean token entropy of Norwegian (0.701) is lower than the Tournament problem (0.863) — yet Tournament converges unanimously in 5 attempts while Norwegian produces 6 distinct answers. By token entropy alone, the model appears more confident on Norwegian than on Tournament. This dissociation reveals a phenomenon we call confident divergence: the model has multiple internally-consistent but mutually-incompatible solution paths. Each attempt confidently follows one path to its conclusion — low token entropy throughout — but different attempts follow different paths.
\nThis finding has a direct implication: entropy-weighted voting correctly downweights uncertain attempts (high entropy), but cannot detect confidently wrong attempts that take different wrong paths. A complementary signal — answer-level diversity across attempts — would detect this case and flag the problem as requiring more attempts or a verifier.
\nThe n-Norwegian problem also illustrates the practical compute cost of this failure mode: 322,515 tokens consumed without any useful signal, versus 75,327 for Tournament (which resolved correctly). Under a fixed time budget, this represents a 4× compute drain for a problem that entropy-voting has no way to identify as harder in advance.
\nWe executed the reference set twice with different problem orderings. Results:
\n| Problem | \nRun 1 Answer | \nRun 2 Answer | \nConsistent | \nBoth Correct | \n
|---|---|---|---|---|
| 500×500 square | \n520 | \n520 | \n✓ | \n✓ | \n
| Tournament | \n21818 | \n21818 | \n✓ | \n✓ | \n
| Coprime f(580) | \n580 | \n580 | \n✓ | \n✓ | \n
| Triangle (336) | \n336 | \n336 | \n✓ | \n✓ | \n
| Alice & Bob | \n50 | \n50 | \n✓ | \n✓ | \n
| Ken blackboard | \n32193 | \n32193 | \n✓ | \n✓ | \n
| f(n) = Σgcd | \n32951 | \n32951 | \n✓ | \n✓ | \n
| α: Z→Z | \n160 | \n160 | \n✓ | \n✓ | \n
| Circumcircle | \n57447 | \n57447 | \n✓ | \n✓ | \n
| n-Norwegian | \n8687 | \n31329 | \n✗ | \n? | \n
9 of 10 problems produce identical answers across both runs, with early stopping triggered in all 9. The n-Norwegian problem diverges — Run 1 produces 8687 (1 vote of 6 valid attempts), Run 2 produces 31329 (1 vote of 3 valid attempts with 5 returning NA). The different orderings affected GPU state and memory allocation, leaving fewer tokens available for later problems in Run 2, which is why Norwegian (appearing 4th in Run 2 vs. 10th in Run 1) has more NA attempts in Run 2.
\nThis stability analysis has an important implication: the notebook's variance is concentrated in the n-Norwegian problem. The other 9 problems are stable enough that running twice with different orderings makes no difference. For the private test set, we expect a similar partition: most problems will produce consistent answers regardless of problem order, and a small number of genuinely hard problems will dominate the score variance.
\nSingle competition submission for the best configuration. We submitted v1 once and obtained 44. Two re-submissions of the same notebook scored 35 and 35. Without more repeated submissions we cannot determine whether 44 is the configuration's true mean or a favorable tail. The reference-set stability (9/10 consistent) suggests stability on problems with signal, but competition-set variance is unknown.
\nConfounded ablations. v3 changes prompt style, temperature, attempt count, workers, early_stop, voting function, and batch_size simultaneously. v2 changes temperature (1.0→0.8), attempt count (8→12), early_stop (4→5), system prompt wording, and per-turn token cap simultaneously — temperature_secondary = 1.2 was declared in v2's CFG but never passed to completions.create, so all v2 attempts ran at T=0.8. No ablation in our experiment sequence changes exactly one variable. Clean single-variable ablations were not feasible within the 7-day, 5-submission constraint.
workers=8 vs. standard 16 may explain variance. With attempts=8 and workers=8, every running attempt occupies exactly one kernel — there is no buffer for kernel startup latency or slow execution. A kernel that takes 2+ seconds to return a code result directly delays the next code cell in that attempt. We observed no timeouts attributable to this, but the pattern of scores for our v1 config (44, 35, 35) contrasts with merkiraz's similar config using workers=16 and batch_size=256 (44, 42, 42, std≈1.2). The tighter worker pool may amplify infrastructure variance under different GPU load conditions, explaining the wider spread in our observed scores.
\nbatch_size=128 vs. community standard 256. Lower batch size may reduce vLLM server throughput when multiple attempts request completions simultaneously. Combined with the verbose prompt (more tokens per completion), this could increase per-problem latency slightly. This is another dimension where our config differs from configurations that showed lower variance.
\nNo external answer verification. Our system has no mechanism to verify candidate answers against the problem's mathematical constraints before committing. The n-Norwegian problem is the clearest demonstration of this gap: a verifier that re-executed the model's enumeration code and checked whether the output satisfies the three-distinct-divisors property would catch many of the wrong answers before they enter the vote.
\nNote on entropy as a signal. The AMO system (David Babu et al. [6]) reports that token entropy added noise in their pipeline — \"entropy had only a weak correlation to correctness\" in their Python-tooled traces. Our finding that entropy correctly downweighted the wrong answer in Tournament (Section 6.4, Attempt 3: entropy 0.938, answer 62140 wrong) may reflect a different configuration: our system uses simpler, shorter traces than AMO's multi-agent pipeline. The entropy–convergence dissociation we document (Norwegian vs Tournament) is consistent with both findings — entropy works within a single attempt but cannot predict cross-attempt convergence.
\nToken entropy and answer convergence are different signals. Mean token entropy measures local confidence at each decoding step. It does not measure whether different attempts will converge to the same answer. The n-Norwegian problem (entropy 0.701) looks more confident than Tournament (entropy 0.863) by this metric, yet it fails completely. An answer-diversity signal — tracking how many distinct answers appear across attempts — would catch this failure mode that entropy alone misses.
\nVerify the voting function end-to-end before submitting. The baseline and v5 notebooks failed completely (0/10) due to a missing function definition. A 30-second local test on one reference problem would have caught this before consuming a submission slot.
\nRespect the hard time budget. Every increase in attempts-per-problem reduces the fraction of problems that receive adequate compute. The improvement from N=8 to N=20 in per-problem expected accuracy is smaller than the degradation from problems timing out. The cliff is hard: a timed-out problem returns 0, not a partial credit.
\nConvergence voting is fragile. Entropy-weighted majority voting requires a threshold count of agreeing answers — it is robust to a minority of wrong attempts. Convergence-based methods stabilize around whatever answer appears first, making them vulnerable to early noise. For a problem distribution where hard problems have diverse answers, majority voting is strongly preferred.
\nCompute variance is concentrated in hard problems. The n-Norwegian problem consumes 43× more tokens than the average problem. Under a fixed time budget, these outlier problems drain compute that could be allocated to other problems. An adaptive system that detects high-token-consumption problems early and reduces their budget would improve overall efficiency.
\nEOS errors are recoverable and worth recovering. Without retry logic, approximately 13 attempt-slots were silently discarded in Run 1. With retry (overhead: 1–7 seconds per recovered attempt), all 13 recovered correctly. The implementation is simple and the benefit is concrete.
\nWe presented a 7-day participation in AIMO3 achieving 44/50 (top 14 of 4,133 teams) through targeted inference engineering. The core system — 8 parallel tool-integrated reasoning attempts with entropy-weighted majority voting, adaptive time budgeting, EOS-resilient retry, and early stopping — is stable across two reference runs with different problem orderings: 9 of 10 problems produce identical correct answers, with variance concentrated entirely in one problem where the model produces no consensus at N=8.
\nFour experiments across 8 notebook versions yield clear negative results: convergence voting catastrophically degrades performance (2/10 vs. 9/10); increasing attempt count under a fixed time budget hurts public LB score; top_p=0.8 has no detectable effect; and a missing function definition causes complete failure. These negative results are reproducible from the source notebooks provided with this writeup.
\nThe most important empirical finding is the dissociation between token-level entropy and answer convergence. The n-Norwegian problem has lower mean token entropy (0.701) than the Tournament problem (0.863), yet Norwegian fails completely while Tournament converges in 5 attempts. This \"confident divergence\" — where the model is locally confident at each step but globally takes a different wrong path each attempt — is invisible to entropy-weighted voting. Entropy voting correctly handles uncertain attempts; it cannot handle confidently-wrong divergent ones. Answer-level diversity monitoring is the missing signal, and building it into the voting function is the most important direction for future work.
\n[1] Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2023). Self-consistency improves chain of thought reasoning in language models. ICLR 2023. arXiv:2203.11171
\n[2] Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023). Efficient memory management for large language model serving with PagedAttention. SOSP 2023.
\n[3] Gou, Z., Shao, Z., Gong, Y., Yang, Y., Huang, M., Duan, N., Chen, W., & Zhang, T. (2023). ToRA: A tool-integrated reasoning agent for mathematical problem solving. arXiv:2309.17452
\n[4] Moshkov, I., et al. (2025). AIMO-2 winning solution: Building state-of-the-art mathematical reasoning models with OpenMathReasoning dataset. arXiv:2504.16891
\n[5] Frieder, S., et al. (2025). AI Mathematical Olympiad — Progress Prize 3. Kaggle Competition. https://kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3
\n[6] David Babu et al. (2026). AMO: Agentic Math Orchestrator for Olympiad Mathematics. AIMO3 Writeup, Kaggle.
\n[7] Nitarach, N. (2026). Model Capability Dominates: Inference-Time Optimization Lessons from AIMO 3. arXiv:2603.27844
\nclass CFG:\n system_prompt = (\n 'You are an elite mathematical problem solver with expertise at the International '\n 'Mathematical Olympiad (IMO) level. Your goal is to find the correct answer through '\n 'rigorous mathematical reasoning.\\n\\n'\n '# Problem-Solving Approach:\\n'\n '1. UNDERSTAND: Carefully read the problem. Identify what is given, '\n 'what needs to be found, and constraints.\\n'\n '2. EXPLORE: Consider multiple solution strategies before committing to one.\\n'\n '3. PLAN: Outline key steps of the most promising approach.\\n'\n '4. EXECUTE: Work through the solution methodically with clear reasoning.\\n'\n '5. VERIFY: Check your answer using substitution, edge cases, '\n 'or alternative methods.\\n\\n'\n '# Output Format:\\n'\n 'The final answer must be a non-negative integer between 0 and 99999.\\n'\n 'Place your final numerical answer inside \\\\boxed{}, e.g., \\\\boxed{42}\\n\\n'\n 'Think step-by-step. Quality of reasoning is as important as the final answer.'\n )\n tool_prompt = (\n 'Use this tool to execute Python code for:\\n'\n '- Complex or error-prone calculations\\n'\n '- Numerical verification of analytical results\\n'\n '- Testing conjectures or brute-force checking small cases\\n\\n'\n 'The environment is a stateful Jupyter notebook. Code persists between executions.\\n'\n 'Always use print() to display results.\\n\\n'\n 'Code should support your reasoning, not replace it.'\n )\n preference_prompt = (\n 'You have access to `math`, `numpy`, `sympy`, and `mpmath`. '\n 'Use sympy for exact symbolic answers, numpy for numerical work, '\n 'and mpmath for high-precision arithmetic when needed.'\n )\n # Model\n served_model_name = 'gpt-oss'\n model_path = '/kaggle/input/models/danielhanchen/gpt-oss-120b/transformers/default/1'\n # vLLM\n dtype = 'auto'\n kv_cache_dtype = 'fp8_e4m3'\n gpu_memory_utilization = 0.95\n context_tokens = 65536\n batch_size = 128\n stream_interval = 200\n # Timing\n notebook_limit = 17400\n high_problem_timeout = 900\n base_problem_timeout = 300\n server_timeout = 180\n session_timeout = 960\n jupyter_timeout = 8\n sandbox_timeout = 3\n # Solver\n attempts = 8\n turns = 128\n workers = 8\n early_stop = 4\n seed = 42\n # Sampling\n temperature = 1.0\n min_p = 0.02\n top_logprobs = 5\n buffer_tokens = 512\n search_tokens = 32\n| Filename | \nDescription | \nLocal /10 | \nLB | \n
|---|---|---|---|
baseline.ipynb | \nFirst version; compute_weighted_entropy undefined → bug | \n0 | \n— | \n
v1.ipynb | \nFull reference run; verbose prompt; mean entropy voting | \n9/10 | \n44 | \n
v2.ipynb | \n12 attempts, early_stop=5, T=0.8/1.2 dual temperature | \n— | \n39 | \n
v3.ipynb | \nV40 prompt, 20 attempts, 16 workers | \n— | \n33 | \n
v4.ipynb | \nV40 + ensemble convergence voting, 20 attempts | \n2/10 | \n35 | \n
v5.ipynb | \nVerbose, top_p=0.8; bug present | \n0 | \n39 | \n
v6.ipynb | \nVerbose, top_p=0.8, 5-component entropy voting | \n8/10 | \n\n |
v1 is the only version submitted to the leaderboard (scored 44, 35, 35 across three runs).
\npatterns = [\n r'\\\\boxed\\s*\\{\\s*([0-9,]+)\\s*\\}', # \\boxed{N}\n r'\\\\\\\\boxed\\s*\\{\\s*([0-9,]+)\\s*\\}', # \\\\boxed{N} (escaped)\n r'final\\s+answer\\s+is\\s*:?\\s*([0-9,]+)', # \"final answer is N\"\n r'answer\\s*[:=]\\s*([0-9,]+)', # \"answer: N\" or \"answer = N\"\n]\n# Scan last search_tokens=32 chunks of streamed text\n# Use last match (reversed order) to prefer the model's most recent answer\n# Validate: 0 ≤ value ≤ 99999\npython -m vllm.entrypoints.openai.api_server \\\n --seed 42 \\\n --model /kaggle/input/models/danielhanchen/gpt-oss-120b/transformers/default/1 \\\n --served-model-name gpt-oss \\\n --tensor-parallel-size 1 \\\n --max-num-seqs 128 \\\n --gpu-memory-utilization 0.95 \\\n --host 0.0.0.0 --port 8000 \\\n --dtype auto \\\n --kv-cache-dtype fp8_e4m3 \\\n --max-model-len 65536 \\\n --stream-interval 200 \\\n --async-scheduling \\\n --disable-log-stats \\\n --enable-prefix-caching \\\n --trust-remote-code\nNote: --enable-auto-tool-choice and --tool-call-parser pythonic are deliberately excluded. These flags configure vLLM's native tool-call parsing layer, which is incompatible with openai_harmony's custom encoding. Including them causes tool calls to be silently mis-parsed, producing empty responses and infinite reasoning loops.
Following up on the questions from @philipvonderlind, here are some additional questions to help us assess your submission:
\nLook forward to reading your responses on these.
", "rawMarkdown": "Following up on the questions from @philipvonderlind, here are some additional questions to help us assess your submission:\n\n1. **Compute signature as difficulty signal:** The Norwegian problem consumes 43× the average compute (322k vs 7.5k tokens) and is also where the cross-run 9/10 stability breaks. Could a running token-count signal after the first 1-2 attempts be used as an early difficulty flag, triggering more budget or extra attempts before the vote commits? What would you propose for this in practice?\n2. **Is detecting confident divergence useful:** You propose answer-level diversity monitoring as the missing signal that entropy can't catch (Norwegian's 6 distinct answers across 8 attempts). But if 8 attempts produce that many distinct answers, it may just indicate the model fundamentally can't solve the problem within budget. Is there reason to believe interventions triggered by this signal (more attempts, verifier injection) would actually recover the correct answer, rather than just generating more wrong answers?\n\nLook forward to reading your responses on these.", "replies": [ { "id": 3459144, "postDate": "2026-05-17T11:38:23.737Z", "content": "Hi Sam, thank you for the follow-up questions. These are exactly the two open problems that I think matter most after the writeup: whether compute usage can be turned into an online routing signal, and whether answer-level diversity is useful when the model may simply be unable to solve the problem.
\nYes. I think the evidence in the writeup supports this, but I would use the signal for routing and budget protection, not simply for \"more attempts.\"
\nIn Section 6.3, the Norwegian problem is the clear compute outlier:
\n| Problem | \nTotal tokens | \n
|---|---|
| Alice & Bob | \n7,501 | \n
| f(n) = Σgcd | \n19,402 | \n
| Ken blackboard | \n24,125 | \n
| Coprime f(580) | \n31,525 | \n
| Triangle | \n59,553 | \n
| Circumcircle | \n62,328 | \n
| 500×500 | \n62,855 | \n
| Tournament | \n75,327 | \n
| α: Z→Z | \n119,270 | \n
| n-Norwegian | \n322,515 | \n
Norwegian used about 322k tokens — far more than any other reference problem. It also failed to produce strong answer consensus in the run analyzed in Section 6.4. That means the difficulty signal is not only visible after the full run; it likely starts appearing after the first 1–2 completed attempts.
\nA practical online monitor could track:
\n# Evaluated after first 2 completed attempts\n\nvalid_answers = [a for a in answers if a is not None]\n\nmedian_tokens = median(tokens_per_attempt_so_far)\ndistinct_count = len(set(valid_answers))\nmax_vote = max(Counter(valid_answers).values(), default=0)\nvalid_rate = len(valid_answers) / completed_attempts\n\nhigh_cost = median_tokens > 20_000\nno_consensus = len(valid_answers) >= 2 and distinct_count == len(valid_answers)\n\nis_unstable = high_cost and no_consensus\nThe threshold 20_000 is not meant to be universal — it is a reference-set heuristic. The right value should be calibrated from more runs. The important point is the combination of two signals: high per-attempt token usage and no answer agreement. A problem can be expensive and still worth continuing if the answers are converging. The danger case is expensive and diverse.
I would use the signal as a routing table:
\n| Observed after 1–2 attempts | \nInterpretation | \nAction | \n
|---|---|---|
| Low token usage + agreement | \nEasy, stable | \nEarly stop as soon as threshold is met | \n
| High token usage + agreement emerging | \nHard but converging | \nAllow full budget, continue normally | \n
| High token usage + all answers distinct | \nUnstable / confident divergence | \nSwitch strategy or cap budget | \n
| High NA / invalid rate | \nTimeout or infrastructure risk | \nReduce spending or skip | \n
The most important practical constraint is that the global time budget dominates. Section 5.3 of the writeup shows that increasing attempts under a fixed wall-clock budget can reduce public LB score because later problems get starved. So if a problem is flagged as unstable, I would not automatically give it more attempts. Instead, I would route it to one of two branches:
\nBranch A — enough global time remains
\nUse a targeted intervention rather than another full solve:
\nBranch B — global time is tight
\nCap the budget and move on:
\nSo the correct framing is not:
\n\n\ndetect hard problem → spend more
\n
It is:
\n\n\ndetect expensive + non-convergent problem → either verify differently or stop wasting budget
\n
That is the main engineering lesson I would take from the Norwegian problem.
\nI agree with the concern. If 8 attempts produce many distinct answers, there is no guarantee that attempts 9 or 10 will recover the correct answer. The Norwegian analysis suggests that simply increasing N is likely the wrong intervention.
\nThe reason I still think detecting confident divergence is useful is that it improves calibration even when it does not improve recovery.
\nThe failure mode in the writeup is:
\nAs Section 6.4 documents, Norwegian has lower mean token entropy (0.701) than Tournament (0.863), yet Tournament converges unanimously while Norwegian produces 6 distinct answers. Token entropy and answer convergence are measuring different things:
\n| Signal | \nWhat it measures | \nFailure case | \n
|---|---|---|
| Token entropy | \nLocal confidence during generation | \nModel is confidently following a wrong path | \n
| Answer consensus | \nAgreement across attempts | \nModel produces many incompatible answers | \n
| Token usage | \nCompute cost / difficulty signature | \nModel spends large budget without convergence | \n
Entropy-weighted voting can downweight hesitant attempts. It cannot detect a set of confident but mutually inconsistent attempts. That is why answer-level diversity is a complementary signal.
\nA concrete divergence monitor:
\nvalid_answers = [a for a in answers if a is not None]\nvalid_N = len(valid_answers)\n\nmax_vote_count = max(Counter(valid_answers).values(), default=0)\ndistinct_count = len(set(valid_answers))\n\nconsensus = max_vote_count / valid_N if valid_N else 0\ndiversity = distinct_count / valid_N if valid_N else 1\nmean_entropy = mean(entropies)\n\nconfident_divergence = (\n valid_N >= 4\n and consensus <= 0.25\n and diversity >= 0.75\n and mean_entropy < entropy_threshold # locally confident\n)\nThe thresholds should be calibrated from more runs. The qualitative condition is: low entropy + high diversity = locally confident, globally inconsistent.
\nWhen this is detected, I would not trust the normal entropy-weighted vote and would instead trigger one of the following actions, in order of preference:
\nAction 1 — Candidate verification
\nTake the top-2 or top-3 candidate answers and ask the model to check each against the problem's defining constraints. This is structurally easier than re-deriving from scratch — the model only needs to discriminate between candidates rather than reconstruct a full solution path.
\nAction 2 — Solution-path comparison
\nAsk the model to compare the assumptions behind two conflicting derivations and identify where they diverge. This targets the divergence directly rather than ignoring it.
\nAction 3 — Executable checker
\nWhen the problem structure permits it (number theory, combinatorics), write code to test the defining property on candidate answers. Not always feasible for IMO-level problems, but when available it is a better use of compute than another full reasoning attempt.
\nAction 4 — Budget protection
\nIf verification is not feasible and time is tight, stop spending. Return the current best answer flagged as low-consensus and preserve budget for downstream problems.
\nThis is why detecting confident divergence is useful even when the model cannot reliably recover. It lets the solver distinguish between:
\n| Case | \nExample behavior | \nHow to treat answer | \n
|---|---|---|
| Stable majority | \n4+ attempts agree on same answer | \nHigh confidence | \n
| Uncertain minority | \nOne high-entropy outlier | \nDownweight by entropy | \n
| Confident divergence | \nMany low-entropy distinct answers | \nLow confidence despite entropy | \n
| No valid answers | \nNA / timeouts throughout | \nFallback / budget cap | \n
The Norwegian problem sits in the third category.
\nThe cross-run stability analysis in Section 6.5 supports this interpretation. The same v1 configuration was stable on 9 of 10 reference problems across two independent runs, while Norwegian changed between runs — one run selected the reference answer 8687, another selected a different answer. This suggests the model can sometimes reach the correct answer, but the standard voting signal is not reliable enough to know when it has done so. Detecting divergence would at minimum flag the returned answer as unreliable rather than giving it the same apparent confidence as a stable 4-vote majority.
To summarize:
\nThe practical intervention should be verification, candidate comparison, or budget capping — not blindly increasing attempts.
\nIn hindsight, the voting layer should have used at least four signals together:
\nconfidence = 1 / mean_token_entropy # local confidence per attempt\nconsensus = max_vote_count / valid_N # cross-attempt agreement\ndiversity = distinct_answers / valid_N # cross-attempt disagreement\ncost = tokens_used_so_far # budget pressure\nThe submitted pipeline used entropy and majority voting but did not explicitly track answer diversity or token-cost routing. That is the main change I would make in a next version.
", "rawMarkdown": "Hi Sam, thank you for the follow-up questions. These are exactly the two open problems that I think matter most after the writeup: whether compute usage can be turned into an online routing signal, and whether answer-level diversity is useful when the model may simply be unable to solve the problem.\n\n### 1. Could a running token-count signal after the first 1–2 attempts be used as an early difficulty flag?\n\nYes. I think the evidence in the writeup supports this, but I would use the signal for **routing and budget protection**, not simply for \"more attempts.\"\n\nIn Section 6.3, the Norwegian problem is the clear compute outlier:\n\n| Problem | Total tokens |\n|---|---:|\n| Alice & Bob | 7,501 |\n| f(n) = Σgcd | 19,402 |\n| Ken blackboard | 24,125 |\n| Coprime f(580) | 31,525 |\n| Triangle | 59,553 |\n| Circumcircle | 62,328 |\n| 500×500 | 62,855 |\n| Tournament | 75,327 |\n| α: Z→Z | 119,270 |\n| n-Norwegian | 322,515 |\n\nNorwegian used about 322k tokens — far more than any other reference problem. It also failed to produce strong answer consensus in the run analyzed in Section 6.4. That means the difficulty signal is not only visible after the full run; it likely starts appearing after the first 1–2 completed attempts.\n\nA practical online monitor could track:\n\n```python\n# Evaluated after first 2 completed attempts\n\nvalid_answers = [a for a in answers if a is not None]\n\nmedian_tokens = median(tokens_per_attempt_so_far)\ndistinct_count = len(set(valid_answers))\nmax_vote = max(Counter(valid_answers).values(), default=0)\nvalid_rate = len(valid_answers) / completed_attempts\n\nhigh_cost = median_tokens > 20_000\nno_consensus = len(valid_answers) >= 2 and distinct_count == len(valid_answers)\n\nis_unstable = high_cost and no_consensus\n```\n\nThe threshold `20_000` is not meant to be universal — it is a reference-set heuristic. The right value should be calibrated from more runs. The important point is the **combination** of two signals: high per-attempt token usage **and** no answer agreement. A problem can be expensive and still worth continuing if the answers are converging. The danger case is expensive **and** diverse.\n\nI would use the signal as a routing table:\n\n| Observed after 1–2 attempts | Interpretation | Action |\n|---|---|---|\n| Low token usage + agreement | Easy, stable | Early stop as soon as threshold is met |\n| High token usage + agreement emerging | Hard but converging | Allow full budget, continue normally |\n| High token usage + all answers distinct | Unstable / confident divergence | Switch strategy or cap budget |\n| High NA / invalid rate | Timeout or infrastructure risk | Reduce spending or skip |\n\nThe most important practical constraint is that the **global time budget dominates**. Section 5.3 of the writeup shows that increasing attempts under a fixed wall-clock budget can reduce public LB score because later problems get starved. So if a problem is flagged as unstable, I would not automatically give it more attempts. Instead, I would route it to one of two branches:\n\n**Branch A — enough global time remains**\n\nUse a targeted intervention rather than another full solve:\n- run a verifier pass on the top candidate answers;\n- ask the model to compare competing derivations and identify where they diverge;\n- run a problem-specific checker if the structure permits it.\n\n**Branch B — global time is tight**\n\nCap the budget and move on:\n- avoid spending 300k+ tokens on a problem with no consensus;\n- preserve time for later problems where early agreement is more likely;\n- return the best available answer, flagged internally as low-consensus.\n\nSo the correct framing is not:\n\n> detect hard problem → spend more\n\nIt is:\n\n> detect expensive + non-convergent problem → either verify differently or stop wasting budget\n\nThat is the main engineering lesson I would take from the Norwegian problem.\n\n---\n\n### 2. Is detecting confident divergence useful if 8 distinct answers may mean the model cannot solve the problem?\n\nI agree with the concern. If 8 attempts produce many distinct answers, there is no guarantee that attempts 9 or 10 will recover the correct answer. The Norwegian analysis suggests that simply increasing N is likely the wrong intervention.\n\nThe reason I still think detecting confident divergence is useful is that it improves **calibration** even when it does not improve **recovery**.\n\nThe failure mode in the writeup is:\n\n- token entropy says: the model seems confident\n- answer diversity says: the attempts do not agree at all\n\nAs Section 6.4 documents, Norwegian has *lower* mean token entropy (0.701) than Tournament (0.863), yet Tournament converges unanimously while Norwegian produces 6 distinct answers. Token entropy and answer convergence are measuring different things:\n\n| Signal | What it measures | Failure case |\n|---|---|---|\n| Token entropy | Local confidence during generation | Model is confidently following a wrong path |\n| Answer consensus | Agreement across attempts | Model produces many incompatible answers |\n| Token usage | Compute cost / difficulty signature | Model spends large budget without convergence |\n\nEntropy-weighted voting can downweight hesitant attempts. It cannot detect a set of confident but mutually inconsistent attempts. That is why answer-level diversity is a complementary signal.\n\nA concrete divergence monitor:\n\n```python\nvalid_answers = [a for a in answers if a is not None]\nvalid_N = len(valid_answers)\n\nmax_vote_count = max(Counter(valid_answers).values(), default=0)\ndistinct_count = len(set(valid_answers))\n\nconsensus = max_vote_count / valid_N if valid_N else 0\ndiversity = distinct_count / valid_N if valid_N else 1\nmean_entropy = mean(entropies)\n\nconfident_divergence = (\n valid_N >= 4\n and consensus <= 0.25\n and diversity >= 0.75\n and mean_entropy < entropy_threshold # locally confident\n)\n```\n\nThe thresholds should be calibrated from more runs. The qualitative condition is: **low entropy + high diversity = locally confident, globally inconsistent**.\n\nWhen this is detected, I would not trust the normal entropy-weighted vote and would instead trigger one of the following actions, in order of preference:\n\n**Action 1 — Candidate verification**\n\nTake the top-2 or top-3 candidate answers and ask the model to check each against the problem's defining constraints. This is structurally easier than re-deriving from scratch — the model only needs to discriminate between candidates rather than reconstruct a full solution path.\n\n**Action 2 — Solution-path comparison**\n\nAsk the model to compare the assumptions behind two conflicting derivations and identify where they diverge. This targets the divergence directly rather than ignoring it.\n\n**Action 3 — Executable checker**\n\nWhen the problem structure permits it (number theory, combinatorics), write code to test the defining property on candidate answers. Not always feasible for IMO-level problems, but when available it is a better use of compute than another full reasoning attempt.\n\n**Action 4 — Budget protection**\n\nIf verification is not feasible and time is tight, stop spending. Return the current best answer flagged as low-consensus and preserve budget for downstream problems.\n\nThis is why detecting confident divergence is useful even when the model cannot reliably recover. It lets the solver distinguish between:\n\n| Case | Example behavior | How to treat answer |\n|---|---|---|\n| Stable majority | 4+ attempts agree on same answer | High confidence |\n| Uncertain minority | One high-entropy outlier | Downweight by entropy |\n| Confident divergence | Many low-entropy distinct answers | Low confidence despite entropy |\n| No valid answers | NA / timeouts throughout | Fallback / budget cap |\n\nThe Norwegian problem sits in the third category.\n\nThe cross-run stability analysis in Section 6.5 supports this interpretation. The same `v1` configuration was stable on 9 of 10 reference problems across two independent runs, while Norwegian changed between runs — one run selected the reference answer 8687, another selected a different answer. This suggests the model can sometimes reach the correct answer, but the standard voting signal is not reliable enough to know when it has done so. Detecting divergence would at minimum flag the returned answer as unreliable rather than giving it the same apparent confidence as a stable 4-vote majority.\n\nTo summarize:\n\n- **If the question is \"would diversity detection guarantee recovery?\"** — no.\n- **If the question is \"would diversity detection make the system better calibrated and more budget-aware?\"** — yes.\n\nThe practical intervention should be verification, candidate comparison, or budget capping — not blindly increasing attempts.\n\nIn hindsight, the voting layer should have used at least four signals together:\n\n```python\nconfidence = 1 / mean_token_entropy # local confidence per attempt\nconsensus = max_vote_count / valid_N # cross-attempt agreement\ndiversity = distinct_answers / valid_N # cross-attempt disagreement\ncost = tokens_used_so_far # budget pressure\n```\n\nThe submitted pipeline used entropy and majority voting but did not explicitly track answer diversity or token-cost routing. That is the main change I would make in a next version." } ] }, { "id": 3457719, "postDate": "2026-05-14T08:25:41.663Z", "content": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.
\nI have a few questions regarding your work:
\nComparison to Public Notebook: Some parts of your submission appear similar to the public notebook shared in the discussion, which many participants used as a starting point. While building on others' work is certainly encouraged, it would be helpful if you could cite the original notebook, include a link to it, and clearly describe which parts you adapted, as well as the motivation behind each change.
Plots: Adding plots to summarize your results, especially the ensemble consensus mechanism, would improve both the readability and presentation of your work. For example, it would be useful to include plots comparing the performance of all ablations on both the public leaderboard and your local evaluation set.
Experiment Construction: In many of your experiments, several changes appear to be introduced at the same time, which makes the results harder to interpret. Do you have results from more isolated experiments where only one component is changed at a time? This would make it easier to attribute score changes to specific modifications and to build on those findings in future work.
Looking forward to your response :)
", "rawMarkdown": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the [updated discussion post](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/689703).\n\nI have a few questions regarding your work:\n\n1. **Comparison to Public Notebook**: Some parts of your submission appear similar to the public notebook shared in the discussion, which many participants used as a starting point. While building on others' work is certainly encouraged, it would be helpful if you could cite the original notebook, include a link to it, and clearly describe which parts you adapted, as well as the motivation behind each change.\n\n2. **Plots**: Adding plots to summarize your results, especially the ensemble consensus mechanism, would improve both the readability and presentation of your work. For example, it would be useful to include plots comparing the performance of all ablations on both the public leaderboard and your local evaluation set.\n\n3. **Experiment Construction**: In many of your experiments, several changes appear to be introduced at the same time, which makes the results harder to interpret. Do you have results from more isolated experiments where only one component is changed at a time? This would make it easier to attribute score changes to specific modifications and to build on those findings in future work.\n\nLooking forward to your response :)\n\n\n", "replies": [ { "id": 3459123, "postDate": "2026-05-17T11:14:31.037Z", "content": "Hi Philip, thank you for the detailed questions. I agree with all three concerns: attribution to the public starting point should have been clearer, the results would be easier to understand with plots, and many of my experiments are practical engineering iterations rather than clean single-variable ablations.
\nMy work started from the public HCMUS Group 4 notebook, which I used as the initial runnable baseline. I also used the public AIMO 3 Utils notebook by andreasbis for offline dependency packaging, especially the vllm and openai_harmony wheels.
I should have cited both of these explicitly in the writeup.
\nThe HCMUS baseline already provided the main competition-compatible pipeline:
\ngpt-oss-120b served through vLLM;openai_harmony formatting;So the most accurate description of my work is not “a new pipeline from scratch,” but an inference-time engineering and evaluation study built on top of that public baseline.
\nThe parts I adapted directly were:
\n| Component | \nNotes | \n
|---|---|
vLLM serving of gpt-oss-120b | \nkept the same general serving approach | \n
openai_harmony conversation format | \nkept because it is the expected format for gpt-oss | \n
| persistent Jupyter kernel tool-use | \nkept because stateful tool use is important for multi-step math/code reasoning | \n
| parallel attempts + early stopping | \nkept the same general self-consistency structure | \n
| entropy-weighted voting | \nkept as the main aggregation method | \n
| Kaggle offline dependency setup | \nused the public utility input by andreasbis | \n
My own work was mainly in changing, testing, and analyzing the inference-time configuration around this baseline: prompt variants, attempt count, worker count, batch size, temperature, convergence-voting variants, intended top_p variants, reference-set reruns, public LB submissions, per-problem token analysis, and failure-mode analysis around answer diversity vs. entropy.
The main changes and motivations were:
\n| Version | \nChange | \nMotivation | \nResult / interpretation | \n
|---|---|---|---|
baseline | \nfirst version with missing entropy function bug | \ninitial run | \nfailed due implementation bug, not model quality | \n
v1 | \n8 attempts, entropy voting, verbose prompt | \nstable baseline under time budget | \nbest public LB result | \n
v2 | \n12 attempts, early_stop=5, lower temperature=0.8 | \ntest whether more samples + lower randomness improves reliability | \npublic LB decreased, likely due time-budget pressure and confounded changes | \n
v3 | \nconcise V40 prompt, 20 attempts, 16 workers, larger batch | \ntest whether shorter prompt + more self-consistency improves score | \npublic LB decreased; too expensive under fixed budget | \n
v4 | \nconvergence / ensemble voting family | \ntest whether answer-distribution convergence is better than majority voting | \nstrong negative result locally | \n
v5/v6 | \nintended top_p=0.8 variants | \ntest whether tail-token filtering improves math reliability | \nweak/no clear evidence; in the code, top_p was declared in CFG but not passed to completions.create, so this experiment did not cleanly test top_p | \n
The key lesson is that increasing inference complexity did not automatically help under the 5-hour budget. The best-performing setup was comparatively simple: 8 attempts, early stop at 4, entropy-weighted majority voting.
\nI agree that the writeup would have been much easier to read with plots. Since the writeup has already been submitted, I cannot insert them into the original text, but these are the supplemental plots I prepared after your comment.
\nPublic LB score by submitted notebook version
\nThis summarizes the practical outcome of the submitted configurations: v1, v2, v3, v4, and v5. It also makes clear that these are engineering iterations, not clean one-variable ablations.
Notebook version change matrix
\nThis plot is meant to make the confounding explicit. Most versions change multiple dimensions at once, which is why I should not present them as clean single-variable ablations.
\n
Per-problem compute signature
\nThis shows Norwegian as the clear compute outlier and motivates the token-count difficulty signal.
\n
Answer diversity vs. consensus
\nThis replaces the table I should have included for the ensemble/consensus discussion. Stable problems cluster around high consensus and low diversity, while Norwegian is the high-diversity / low-consensus outlier.
\n
I agree that many experiments changed multiple things at once. This is the biggest limitation of the study.
\nBecause I joined with only 7 days remaining and had a limited submission budget, I treated the notebook versions as practical engineering iterations rather than clean scientific ablations. In hindsight, the writeup should have separated these two categories more explicitly:
\n| Type | \nExamples | \nHow much can be concluded? | \n
|---|---|---|
| repeated same-config check | \nrepeated reference run of v1 | \nuseful for run-to-run stability | \n
| practical engineering iteration | \nv2, v3, v4, v5, v6 | \nuseful negative results, but confounded | \n
| clean one-variable ablation | \nmostly not available | \nlimited by time/submission budget | \n
The closest I have to isolated evidence is the repeated reference-set run of the same v1 configuration discussed in Section 6.5. That checks stability rather than a component change: 9 of 10 reference problems were stable across runs, while the Norwegian problem changed.
For the version comparisons:
\nv2 vs. v1 is not isolated because it changes attempt count, early-stop threshold, and temperature.v3 vs. v1 is not isolated because it changes prompt style, attempts, workers, batch size, and early stopping.v4 should not be interpreted as “only voting changed from v1.” It belongs to the V40 / 20-attempt family and adds convergence / ensemble voting and related routing logic.v5/v6 were intended to test the top_p=0.8 direction, but because top_p was not actually passed into the completion call, this does not support a strong conclusion about top_p.So the safe conclusions are:
\nv1-style entropy-voting setup was the strongest practical configuration I found;top_p, or worker count.If I were repeating the study with more time, I would run a proper ablation grid:
\n| Experiment | \nKeep fixed | \nChange only | \n
|---|---|---|
| prompt ablation | \nN=8, T=1.0, same voting | \nverbose vs concise prompt | \n
| temperature ablation | \nN=8, same prompt/voting | \nT=1.0 vs T=0.8 | \n
| attempt-count ablation | \nsame prompt/voting/T | \nN=8 vs N=12 vs N=20 | \n
| voting ablation | \nsame prompt/N/T | \nmajority vs entropy-weighted vs convergence | \n
| worker/batch ablation | \nsame solver config | \nworkers/batch-size only | \n
That would make attribution much cleaner than the constrained 7-day experiment I was able to run.
", "rawMarkdown": "Hi Philip, thank you for the detailed questions. I agree with all three concerns: attribution to the public starting point should have been clearer, the results would be easier to understand with plots, and many of my experiments are practical engineering iterations rather than clean single-variable ablations.\n\n### 1. Comparison to the Public Notebook\n\nMy work started from the public [HCMUS Group 4 notebook](https://www.kaggle.com/code/scottford2701/hcmus-group-4?scriptVersionId=310860229), which I used as the initial runnable baseline. I also used the public [AIMO 3 Utils notebook by andreasbis](https://www.kaggle.com/code/andreasbis/aimo-3-utils) for offline dependency packaging, especially the `vllm` and `openai_harmony` wheels.\n\nI should have cited both of these explicitly in the writeup.\n\nThe HCMUS baseline already provided the main competition-compatible pipeline:\n\n- `gpt-oss-120b` served through vLLM;\n- `openai_harmony` formatting;\n- Python tool-use through persistent Jupyter kernels;\n- parallel self-consistency attempts;\n- entropy-weighted majority voting;\n- early stopping;\n- adaptive per-problem budgeting;\n- EOS / connection-error handling;\n- offline Kaggle dependency setup.\n\nSo the most accurate description of my work is not “a new pipeline from scratch,” but an inference-time engineering and evaluation study built on top of that public baseline.\n\nThe parts I adapted directly were:\n\n| Component | Notes |\n|---|---|\n| vLLM serving of `gpt-oss-120b` | kept the same general serving approach |\n| `openai_harmony` conversation format | kept because it is the expected format for `gpt-oss` |\n| persistent Jupyter kernel tool-use | kept because stateful tool use is important for multi-step math/code reasoning |\n| parallel attempts + early stopping | kept the same general self-consistency structure |\n| entropy-weighted voting | kept as the main aggregation method |\n| Kaggle offline dependency setup | used the public utility input by andreasbis |\n\nMy own work was mainly in changing, testing, and analyzing the inference-time configuration around this baseline: prompt variants, attempt count, worker count, batch size, temperature, convergence-voting variants, intended `top_p` variants, reference-set reruns, public LB submissions, per-problem token analysis, and failure-mode analysis around answer diversity vs. entropy.\n\nThe main changes and motivations were:\n\n| Version | Change | Motivation | Result / interpretation |\n|---|---|---|---|\n| `baseline` | first version with missing entropy function bug | initial run | failed due implementation bug, not model quality |\n| `v1` | 8 attempts, entropy voting, verbose prompt | stable baseline under time budget | best public LB result |\n| `v2` | 12 attempts, `early_stop=5`, lower `temperature=0.8` | test whether more samples + lower randomness improves reliability | public LB decreased, likely due time-budget pressure and confounded changes |\n| `v3` | concise V40 prompt, 20 attempts, 16 workers, larger batch | test whether shorter prompt + more self-consistency improves score | public LB decreased; too expensive under fixed budget |\n| `v4` | convergence / ensemble voting family | test whether answer-distribution convergence is better than majority voting | strong negative result locally |\n| `v5/v6` | intended `top_p=0.8` variants | test whether tail-token filtering improves math reliability | weak/no clear evidence; in the code, `top_p` was declared in `CFG` but not passed to `completions.create`, so this experiment did not cleanly test `top_p` |\n\nThe key lesson is that increasing inference complexity did not automatically help under the 5-hour budget. The best-performing setup was comparatively simple: 8 attempts, early stop at 4, entropy-weighted majority voting.\n\n### 2. Plots\n\nI agree that the writeup would have been much easier to read with plots. Since the writeup has already been submitted, I cannot insert them into the original text, but these are the supplemental plots I prepared after your comment.\n\n**Public LB score by submitted notebook version**\n\nThis summarizes the practical outcome of the submitted configurations: `v1`, `v2`, `v3`, `v4`, and `v5`. It also makes clear that these are engineering iterations, not clean one-variable ablations.\n\n\n\n**Notebook version change matrix**\n\nThis plot is meant to make the confounding explicit. Most versions change multiple dimensions at once, which is why I should not present them as clean single-variable ablations.\n\n\n\n**Per-problem compute signature**\n\nThis shows Norwegian as the clear compute outlier and motivates the token-count difficulty signal.\n\n\n\n**Answer diversity vs. consensus**\n\nThis replaces the table I should have included for the ensemble/consensus discussion. Stable problems cluster around high consensus and low diversity, while Norwegian is the high-diversity / low-consensus outlier.\n\n\n\n### 3. Experiment Construction / Isolated Experiments\n\nI agree that many experiments changed multiple things at once. This is the biggest limitation of the study.\n\nBecause I joined with only 7 days remaining and had a limited submission budget, I treated the notebook versions as practical engineering iterations rather than clean scientific ablations. In hindsight, the writeup should have separated these two categories more explicitly:\n\n| Type | Examples | How much can be concluded? |\n|---|---|---|\n| repeated same-config check | repeated reference run of `v1` | useful for run-to-run stability |\n| practical engineering iteration | `v2`, `v3`, `v4`, `v5`, `v6` | useful negative results, but confounded |\n| clean one-variable ablation | mostly not available | limited by time/submission budget |\n\nThe closest I have to isolated evidence is the repeated reference-set run of the same `v1` configuration discussed in Section 6.5. That checks stability rather than a component change: 9 of 10 reference problems were stable across runs, while the Norwegian problem changed.\n\nFor the version comparisons:\n\n- `v2` vs. `v1` is not isolated because it changes attempt count, early-stop threshold, and temperature.\n- `v3` vs. `v1` is not isolated because it changes prompt style, attempts, workers, batch size, and early stopping.\n- `v4` should not be interpreted as “only voting changed from `v1`.” It belongs to the V40 / 20-attempt family and adds convergence / ensemble voting and related routing logic.\n- `v5/v6` were intended to test the `top_p=0.8` direction, but because `top_p` was not actually passed into the completion call, this does not support a strong conclusion about `top_p`.\n\nSo the safe conclusions are:\n\n- the `v1`-style entropy-voting setup was the strongest practical configuration I found;\n- increasing attempts under the fixed wall-clock budget did not reliably improve score;\n- convergence-style voting was a strong negative engineering result in my local test;\n- run-to-run variance was real and concentrated mainly in the Norwegian reference problem;\n- stronger single-variable ablations would be needed to assign precise causal effects to prompt, temperature, `top_p`, or worker count.\n\nIf I were repeating the study with more time, I would run a proper ablation grid:\n\n| Experiment | Keep fixed | Change only |\n|---|---|---|\n| prompt ablation | N=8, T=1.0, same voting | verbose vs concise prompt |\n| temperature ablation | N=8, same prompt/voting | T=1.0 vs T=0.8 |\n| attempt-count ablation | same prompt/voting/T | N=8 vs N=12 vs N=20 |\n| voting ablation | same prompt/N/T | majority vs entropy-weighted vs convergence |\n| worker/batch ablation | same solver config | workers/batch-size only |\n\nThat would make attribution much cleaner than the constrained 7-day experiment I was able to run.\n" } ] }, { "id": 3447425, "postDate": "2026-04-23T05:29:22.890Z", "content": "looks like lot of efforts @heon29 nice work!
", "rawMarkdown": "looks like lot of efforts @heon29 nice work!" } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T04:23:57Z", "competition_id": 118448, "competition_slug": "ai-mathematical-olympiad-progress-prize-3", "team_id": 15572431, "solution_url": "" }