{
"id": 694271,
"title": "Spending Time Wisely, Ignoring Entropy — AI Mathematical Olympiad Progress Prize",
"content": "Author: Ayman Hamed\nCompetition: AI Mathematical Olympiad — Progress Prize 3 (AIMO3)\nPublic Leaderboard: 37/50\nKaggle Notebook (submitted): aimo-39
\nBase Model: gpt-oss-120b
\nAcknowledgments
\nThis solution builds on Siancy's public notebook. The core infrastructure — vLLM serving, Harmony encoding, sandboxed Jupyter tool execution, parallel attempts, early stopping, entropy-weighted voting — is theirs. Thanks also to andreasbis/aimo-3-utils for the offline wheel bundle. My changes (+170, −23 lines from the base) were concentrated in three areas: dynamic time management, dropping entropy-weighted voting in favor of majority vote with centric tie-breaking, and prompt adjustments. These are detailed below, along with the reasoning and empirical observations behind each decision.
\nNote: This writeup documents my personal experience, observations, and hypotheses from this competition. Many of the conclusions here — particularly around entropy, centric selection, and time management — are based on limited experiments and may be wrong or negotiable. I share them in the spirit of open discussion, hoping they spark useful conversation rather than claiming definitive results.
\nOverview
\nThe solver runs gpt-oss-120b on a single H100 via vLLM, with fp8 KV cache, prefix caching, and a 64K context window. For each of the 50 problems, the system issues 6–8 parallel, independently-seeded attempts at temperature=1.0 and reasoning_effort=HIGH. Each attempt has access to a stateful Jupyter sandbox (math, numpy, sympy, itertools, collections) that the model can call mid-generation via Harmony tool-use messages. Output is streamed; as soon as the stream produces the pattern oxed{, the answer is extracted immediately — a deliberately minimal match that avoids relying on the closing }, which many notebooks use but which can fail if the model's output is truncated mid-stream.\nOnce attempts complete, answers are pooled and selected by simple majority vote — no entropy weighting. If no early consensus is reached (fewer than 3 agreements), a single tie-break round of 8 additional attempts is launched, budget permitting. When a numerical tie remains even after tie-breaking, the winner is chosen by centric selection: the answer closest to the weighted median of all votes.\nThe critical differentiator of this solution is dynamic time budgeting: time saved on easy problems flows automatically into hard ones, with the system continuously tracking its own time surplus and adjusting behavior in real time.\nThe full pipeline is illustrated in the flowchart below:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1353753%2F02cfcddfd638ef97458fafd4b4e34485%2Fpipeline_flowchart.svg?generation=1777014369802819&alt=media\")
\nThe pipeline for each problem: time budget → parallel attempts → answer extraction → early stop check → majority vote → tie-break (if budget allows) → centric selection → classify hard/easy and update budget for next problem.
\nWhy I Dropped Entropy-Weighted Voting
\nThe notebook lineage I built on — tracing back to nihilisticneuralnet's 43/50 solution — uses entropy-weighted voting, where each attempt's vote is scaled by 1 / mean_token_entropy. The intuition is that lower entropy signals higher model confidence, and therefore greater likelihood of correctness. Many top solutions in this competition use this approach.\nI investigated entropy weighting thoroughly and decided to drop it in favor of plain majority vote. To be upfront: the base notebook with entropy scored 43/50, while my modified version scored 37/50 — so it is entirely possible that entropy weighting is genuinely helpful and that my other changes (reduced workers, different prompts, the time management overhead itself) are responsible for the score difference. I cannot isolate the effect cleanly. That said, here are the specific observations that led me to question entropy's value — offered as hypotheses, not conclusions.
\nObservation 1: Entropy has very weak separation on hard problems
\nOn a representative hard problem, the entropy values across 8 attempts ranged from 0.644 to 0.709 — roughly a 10% spread. The corresponding weights (1/entropy) ranged from ~1.41 to ~1.55. These weights are nearly identical. On hard problems, where the answer matters most, entropy weighting effectively collapses to plain majority voting.
\nObservation 2: Entropy actively misleads — low entropy favors wrong (or no) answers
\nThis is the strongest example I found. On a hard problem where the correct answer is 8687 (ground truth), here is what the 8 attempts produced:
\n\n\n\n| Attempt | \nPython Calls | \nPython Errors | \nEntropy | \nAnswer | \n
\n\n\n\n| 0 | \n40 | \n5 | \n0.565 | \n<NA\\> | \n
\n\n| 1 | \n24 | \n4 | \n0.602 | \n64836 | \n
\n\n| 2 | \n19 | \n1 | \n0.648 | \n71765 | \n
\n\n| 3 | \n30 | \n3 | \n0.558 | \n<NA\\> | \n
\n\n| 4 | \n42 | \n9 | \n0.582 | \n3107 | \n
\n\n| 5 | \n32 | \n7 | \n0.618 | \n78742 | \n
\n\n| 6 | \n18 | \n0 | \n0.594 | \n23 | \n
\n\n| 7 | \n54 | \n7 | \n0.599 | \n8687 ✅ | \n
\n\nFirst, note that attempts 0 and 3 (entropy 0.565 and 0.558) returned <NA> — the model failed to produce any answer at all. These are not \"confidently wrong\" answers; they are broken runs (likely timeouts or parsing failures). Comparing their entropy against valid answers is not meaningful, so let us set them aside. | \n | \n | \n | \n | \n
\n\n| Among the 6 valid answers only, the entropy ranking tells a clear story: | \n | \n | \n | \n | \n
\n\n| Rank (by entropy, lowest first) | \nEntropy | \nWeight (1/entropy) | \nAnswer | \nCorrect? | \n
\n\n| -------------------------------- | \n--------- | \n------------------- | \n-------- | \n---------- | \n
\n\n| 1st (most \"confident\") | \n0.582 | \n1.718 | \n3107 | \n❌ | \n
\n\n| 2nd | \n0.594 | \n1.684 | \n23 | \n❌ | \n
\n\n| 3rd | \n0.599 | \n1.669 | \n8687 | \n✅ | \n
\n\n| 4th | \n0.602 | \n1.661 | \n64836 | \n❌ | \n
\n\n| 5th | \n0.618 | \n1.616 | \n78742 | \n❌ | \n
\n\n| 6th (least \"confident\") | \n0.648 | \n1.543 | \n71765 | \n❌ | \n
\n\nUnder entropy-weighted voting, the wrong answer 3107 receives the highest weight (1.718) while the correct answer 8687 receives only 1.669 — the third-highest. Entropy actively prefers the wrong answer over the correct one. The two most \"confident\" attempts (3107 and 23) are both wrong; the correct answer sits in the middle of the confidence distribution. | \n | \n | \n | \n | \n
\n\n| Entropy measures token-level linguistic certainty — how confident the model is about its next word. But correctness on Olympiad math depends on logical validity, tool usage, and mathematical reasoning. These are fundamentally different signals. A model can be very \"certain\" at the token level while following a completely wrong mathematical path. | \n | \n | \n | \n | \n
\n\n
\nObservation 3: In the chaos regime, entropy selects the most confident wrong answer
\nWhen all attempts produce different answers (as in the example above — 6 valid answers, 6 different values), entropy weighting simply picks whichever wrong answer was generated most confidently. In this case, entropy would select 3107 over the correct 8687. In this regime, the right move is not better weighting — it is more sampling (tie-break rounds) or cutting losses early.
\nMy conclusion
\nEntropy weighting is not harmful on easy problems (where any voting method works), but it provides no benefit on hard problems (where the outcome matters). The real challenge is not weighting votes better — it is identifying when the model has meaningful signal vs. total disagreement and adapting strategy accordingly. That is what the time management system does.
\nIndependent corroboration
\nIndependently, Lê Bảo's 14th-place writeup documents a related phenomenon they call \"confident divergence.\" Their data shows the n-Norwegian problem has lower mean token entropy (0.701) than the Tournament problem (0.863) — yet Norwegian produces 6 completely distinct wrong answers while Tournament converges unanimously in 5 attempts. Their conclusion: \"entropy voting correctly handles uncertain attempts; it cannot handle confidently-wrong divergent ones.\" They also cite the AMO team (David Babu et al.) finding that entropy had \"only a weak correlation to correctness.\" It is worth noting that Lê Bảo uses entropy-weighted voting and scored 44/50 — so this is not a critique from someone who rejected it, but a limitation acknowledged by someone who kept it.
\nDynamic Time Management
\nThis is the core contribution. Most solutions use a static time budget (e.g., 300s base, 900s max per problem). The problem: easy problems that solve in 60s waste 240s of unused budget, while hard problems that need 800s get cut off at 300s. Over 50 problems, this misallocation adds up.
\nThe budget equation
\nAt the start of each problem, the system computes:
\navailable = time_left − future_need − safety_buffer\nbudget = clamp(available, MIN=200s, MAX=1300s)\n
\nWhere future_need is estimated from the observed hard/easy ratio so far:
\nfuture_need = hard_remaining × hard_avg + easy_remaining × easy_avg\n
\nThe system starts with a prior expectation of 14 hard problems out of 50. After a warmup period (5 problems solved), it switches to using the observed ratio. Hard/easy averages start at 750s and 150s respectively, then blend in actual measurements as data accumulates.
\nThree operating modes
\nThe system operates in one of three modes depending on the time situation:\nNORMAL — surplus time exists. The current problem gets available seconds (clamped to [200, 1300]). Easy problems that solve quickly return their unused time to the global pool, making it available for hard problems later.\nSURVIVAL — future need exceeds available time. The system distributes remaining time equally across remaining problems, no longer being generous with any single problem.\nPANIC — less than 120s per remaining problem. The system reduces to bare-minimum attempts (2–4 instead of 6–8) to ensure every problem gets at least a shot.
\nProblem classification
\nAfter each problem is solved, the system classifies it as \"hard\" or \"easy\" based on two signals: consensus strength (max vote frequency ≤ 2 = hard) and time consumed (> 400s without early stop = hard). This classification feeds back into future budget estimates.
\nTie-Breaking: Centric Selection
\nWhen majority voting produces a clear winner — one answer with strictly more votes than all others — that answer is selected immediately, no further logic needed. Centric selection only activates when there is a genuine tie: multiple answers sharing the maximum vote count. This is an important distinction — on most problems, majority vote decides cleanly and centric selection never runs.\nI use what I call centric selection for tie-breaking, based on a hypothesis that may or may not be correct.
\nThe hypothesis
\nOn hard problems where the model is struggling, the correct answer tends to be somewhere in the \"center\" of the distribution of all attempted answers, rather than at the extremes. Outlier answers are more likely to be the result of a reasoning failure that spiraled in an unusual direction.
\nThe mechanism
\nThe logic in _select_answer() is straightforward:
\nif len(top_answers) == 1:\n final_answer = top_answers[0] # Clear winner — done\nelse:\n # Tie: build weighted list of ALL valid answers (repeated by vote count)\n weighted = []\n for answer, data in sorted_stats:\n weighted.extend([answer] * data['votes'])\n weighted.sort()\n # Find the weighted median\n n = len(weighted)\n if n % 2 == 1:\n reference = weighted[n // 2]\n else:\n reference = (weighted[n // 2 - 1] + weighted[n // 2]) / 2\n # Among tied top answers, pick the one closest to this median\n final_answer = min(\n top_answers,\n key=lambda answer: (abs(answer - reference), answer)\n )\n
\nNote the key design choice: the weighted median is computed from all valid answers (not just the tied ones). This means the single-vote \"noise\" answers contribute to the reference point — they pull the median toward the region of answer space that the model explored most densely. An earlier version used the median of only the tied answers, but that ignores useful distributional information from the non-tied attempts.
\nWorked example
\nSuppose after a tie-break round on a hard problem (ground truth: 8687), the vote distribution is:
\n\n\n\n| Answer | \nVotes | \n
\n\n\n\n| 23 | \n2 | \n
\n\n| 8687 | \n2 | \n
\n\n| 41754 | \n2 | \n
\n\n| 5534 | \n1 | \n
\n\n| 98449 | \n1 | \n
\n\n| Three answers are tied at 2 votes each: 23, 8687, and 41754. Majority vote cannot decide. To break the tie, we build the weighted list by repeating each answer by its vote count: | \n | \n
\n\n
\n[23, 23, 5534, 8687, 8687, 41754, 41754, 98449]\n
\nThis list has 8 elements. The median is the average of the 4th and 5th elements:
\nreference = (8687 + 8687) / 2 = 8687\n
\nNow we measure each tied answer's distance to this reference:
\n\n- |23 − 8687| = 8,664
\n- |8687 − 8687| = 0
\n- |41754 − 8687| = 33,067\n8687 is exactly at the median — distance 0. Centric selection picks 8687, which is the correct answer.\nThe intuition: 23 is an outlier on the low end (likely from a reasoning path that collapsed early), 41754 is an outlier on the high end (likely from a computation that spiraled), and 8687 sits in the center — reachable from multiple reasoning paths. The single-vote answers (5534, 98449) contribute to the weighted median but don't compete in the tie, pulling the reference point toward the region where the model explored most densely.\nThis is the core hypothesis of centric selection: on hard problems where the model is struggling, the correct answer tends to be between the outliers rather than at the extremes. Outlier answers are more likely the result of reasoning failures that diverged in unusual directions.
\n
\nWhy not entropy for tie-breaking?
\nThe argument is the same as above: entropy does not reliably separate correct from incorrect on hard problems. The centric approach at least has a geometric intuition — outlier answers likely stem from divergent reasoning paths, while the correct answer is more likely to be \"reachable\" from multiple reasoning paths, placing it nearer to the center.
\nI want to be honest: this is a hypothesis, not a proven result. It worked in my tests, but I have not done a rigorous ablation to measure its marginal impact vs. random tie-breaking.
\nTie-Break Rounds: When to Spend More Compute
\nThe system permits up to 1 tie-break round of 8 additional attempts, but only under specific conditions:
\n\n- Sufficient problem time remaining (> 300s) — hard problems need 300–600s per attempt; anything less is wasted compute.
\n- Not in PANIC mode — if time is critically low, skip tie-breaking.
\n- Global surplus exists — the estimated time left after serving all remaining problems must justify the cost.
\n- Consensus is genuinely weak — if there is already a 3+ vote leader (even without a tie), there is no need.\nAfter a tie-break round, the system adapts: if consensus is still weak after the extra attempts (
max_freq_after <= 2), it raises the early_stop threshold by 1 — requiring one more agreement before declaring consensus. This prevents the system from locking in on a low-confidence answer after spending extra compute. If the top vote count reaches early_stop + 1, consensus is declared. Otherwise, the system accepts the best available answer via centric selection. \n
\nEmpirical observation
\nIn my test runs, tie-break rounds frequently converted weak signal into correct answers. For example, on one hard problem: Round 1 produced 9 attempts with 8 different answers (pure chaos), but after a tie-break round, the correct answer 8687 emerged with 4 votes — enough to win via majority.
\n\n\n\n| Phase | \nAnswers | \nVotes for correct | \n
\n\n\n\n| Round 1 (8 attempts) | \n8 unique answers | \n1 | \n
\n\n| After tie-break (+8) | \nconsensus forming | \n4 | \n
\n\n| Final result | \n8687 ✅ | \n4 votes | \n
\n\n
\nThis was one of the cases where the extra sampling paid off. But there were also problems where tie-breaking just added more noise. The decision of when to tie-break matters as much as the tie-break itself.
\nPython Execution Metrics: Noisy but Informative (Ignored in Voting)
\nAn interesting side observation: Python execution metrics (number of tool calls, number of errors) do not correlate cleanly with correctness. From the entropy evidence table above:
\n\n- Attempt 7 (54 calls, 7 errors) produced the correct answer
8687 \n- Attempt 6 (18 calls, 0 errors) produced the wrong answer
23\nThis makes sense intuitively: the model may hit errors while exploring the problem space (timeouts, overflows, wrong initial approaches), then recover and find the correct path. Conversely, a clean run might mean the model confidently followed a wrong approach without ever hitting a \"guardrail.\" \n
\nThese metrics are currently not used in voting. Whether they should be incorporated — perhaps as a filter rather than a weight — is an open question.
\nPrompts
\nThe system prompt is deliberately structured but not prescriptive about reasoning strategy:
\nYou are solving a high-level mathematical contest problem (AIMO-style).\n### GOAL\nProduce a correct VERIFIED answer. Elegance is irrelevant.\n### NON-NEGOTIABLE RULES:\n- Never Guess\n- Never compute astronomically large quantities directly\n (n!, a^(b!), 3^(b!), power towers, etc.).\n- If you derive a general rule, formula, or pattern:\n • You MUST justify why it applies in general.\n • You MUST verify it on small or edge cases before trusting it.\n- If a counterexample appears, discard or revise the approach immediately.\n- If multiple answers seem plausible, you MUST disambiguate using structure\n (not intuition, not magnitude, not frequency).\n- For arithmetic functions over huge parameters,\n first reduce to residue classes and prime-power families,\n derive the family formula symbolically,\n then aggregate exactly across all required cases.\n### OUTPUT CONTRACT (STRICT):\n- Output ONLY the final answer in LaTeX boxed form: \\boxed{...}\n- Exactly ONE boxed integer. No extra text.\n- The answer must be an integer in [0, 99999].\n
\nThis prompt sits in a middle ground between extreme minimalism (e.g., \"Solve the following math problem step by step\") and the heavily scaffolded approaches (UNDERSTAND→PLAN→EXECUTE→VERIFY loops) that others tried. The key rules (\"never guess,\" \"verify on small cases,\" \"reduce to residue classes\") are guardrails against common failure modes I observed, not reasoning prescriptions.
\nConfiguration
\n\n\n\n| Parameter | \nValue | \nNotes | \n
\n\n\n\n| Model | \ngpt-oss-120b | \nvLLM, fp8 KV cache, prefix caching | \n
\n\n| reasoning_effort | \nHIGH | \nHarmony system content | \n
\n\n| Context | \n65,536 tokens | \nmax_model_len | \n
\n\n| Temperature | \n1.0 | \n | \n
\n\n| min_p | \n0.02 | \n | \n
\n\n| Attempts per problem | \n6–8 | \n8 default; reduced in PANIC mode | \n
\n\n| Workers | \n6–8 | \nparallel threads per problem | \n
\n\n| Early-stop threshold | \n3 | \nagreements to stop early | \n
\n\n| Voting | \nMajority vote | \nno entropy weighting | \n
\n\n| Tie-breaking | \nCentric selection | \nweighted median proximity | \n
\n\n| Max tie-break rounds | \n1 | \n8 attempts per round | \n
\n\n| Min budget per problem | \n200s | \nfloor | \n
\n\n| Max budget per problem | \n1300s | \nceiling | \n
\n\n| Notebook runtime limit | \n18,000s | \n~5 hours | \n
\n\n| Safety margin | \n5% | \nof remaining time | \n
\n\n
\n| Seed | 42 | per-attempt seed = (42 + i)² |
\nResults and Reproducibility
\nI ran the notebook twice on the public test set:
\n\n\n\n| Run | \nWorkers | \nTie-break rounds | \nPublic Score | \n
\n\n\n\n| 1 | \n8 | \n1 | \n37/50 | \n
\n\n| 2 | \n6 | \n1 | \n36/50 | \n
\n\n| I selected the 37/50 run for submission. The private leaderboard score has not been published yet at the time of writing. | \n | \n | \n | \n
\n\n| Baseline estimate: 36–37 is the reproducible baseline. The 1-point variance between runs comes from stochastic tie-break outcomes on 2–3 hard problems. | \n | \n | \n | \n
\n\n| Stability note: For comparison, Lê Bảo's 14th-place solution (entropy-weighted voting, static 300–900s budget) submitted the same notebook three times and scored 44, 35, 35 — a 9-point spread. My two runs scored 37 and 36 — a 1-point spread. While their peak is higher, the variance is significant: in a competition where you pick one submission, a stable 36–37 may outperform a volatile distribution centered around 38 with tails at 35 and 44. The dynamic time management system likely contributes to this stability by ensuring no problem is starved of compute due to a previous problem overrunning its budget. | \n | \n | \n | \n
\n\n| Upper bound estimate: If the notebook runs twice and the tie-break rounds happen to flip in the right direction on the hard problems, a score of 37–41 feels plausible. This is speculative — the variance on hard problems is real, and I have not done enough runs to estimate the full distribution. | \n | \n | \n | \n
\n\n
\nHardware
\n\n- 1× NVIDIA H100 (Kaggle)
\n- Model weights:
danielhanchen/gpt-oss-120b (transformers format) \n
\n- Wheels: offline bundle from andreasbis/aimo-3-utils
\nWhat I Tried and Discarded
\nI want to briefly mention some things that did not work or that I chose not to pursue. This is the part I'm least confident about — some of these might work with more tuning.\nEntropy-weighted voting: As discussed above. My tests suggested it was at best neutral, at worst slightly misleading on hard problems. But I did not run a clean A/B test across all 50 problems — it is possible the effect is small but positive.\nMore tie-break rounds: I tried MAX_TIE_ROUNDS=2 and 3. Diminishing returns: the second round rarely flipped a problem that the first round didn't. Meanwhile, the extra time consumed could have been spent on subsequent hard problems.\nAggressive early stopping (threshold=2): Occasionally grabbed wrong answers that happened to agree by coincidence. Threshold=3 was the sweet spot.
\nMultiple prompt variants per problem: Running half the attempts with one prompt and half with another. Added complexity without measurably improving accuracy.
\nThe Three Regimes: A Mental Model
\nLooking across many problem runs, I found it useful to think in terms of three distinct regimes:\nStrong consensus — 4+ attempts agree. Any voting method works. Early stopping saves time. This is the easy/medium regime.\nWeak signal — The correct answer appears 2–3 times, surrounded by noise. This is where tie-break rounds and more sampling pay off. The right action is: spend more compute.\nChaos — All attempts produce different answers. No amount of reweighting helps. The right action is: stop early, save compute for later problems, and accept the best guess (or return 0).
\nThe dynamic time management system implicitly handles all three: early stopping for consensus, tie-breaking for weak signal, and PANIC/SURVIVAL modes to avoid blowing the budget on chaos.
\nFinal Thoughts
\nThe main insight I took from this competition is that how you allocate compute across problems matters more than how you aggregate votes within a problem. Entropy weighting, verifier loops, multi-stage pipelines — these all focus on squeezing more signal from a fixed set of attempts. But the bigger lever is ensuring that hard problems get enough attempts in the first place, and that easy problems don't waste time they don't need.\nWhether the specific mechanisms I used (centric selection, the budget equation, the three-mode system) are optimal is an open question. They worked for me, but I suspect there is room for improvement in each one.
\nThis was my deep dive into inference-time engineering for mathematical reasoning, and much of what I've written above reflects intuitions formed during experimentation rather than rigorously controlled results. If any of these observations resonate — or if you have evidence that contradicts them — I'd welcome the discussion.
\nCompetition: AI Mathematical Olympiad — Progress Prize 3
",
"comments": [],
"comment_count": 0,
"messages": [
{
"id": 3447881,
"postDate": "2026-04-24T07:23:58.420Z",
"content": "Author: Ayman Hamed\nCompetition: AI Mathematical Olympiad — Progress Prize 3 (AIMO3)\nPublic Leaderboard: 37/50\nKaggle Notebook (submitted): aimo-39
\nBase Model: gpt-oss-120b
\nAcknowledgments
\nThis solution builds on Siancy's public notebook. The core infrastructure — vLLM serving, Harmony encoding, sandboxed Jupyter tool execution, parallel attempts, early stopping, entropy-weighted voting — is theirs. Thanks also to andreasbis/aimo-3-utils for the offline wheel bundle. My changes (+170, −23 lines from the base) were concentrated in three areas: dynamic time management, dropping entropy-weighted voting in favor of majority vote with centric tie-breaking, and prompt adjustments. These are detailed below, along with the reasoning and empirical observations behind each decision.
\nNote: This writeup documents my personal experience, observations, and hypotheses from this competition. Many of the conclusions here — particularly around entropy, centric selection, and time management — are based on limited experiments and may be wrong or negotiable. I share them in the spirit of open discussion, hoping they spark useful conversation rather than claiming definitive results.
\nOverview
\nThe solver runs gpt-oss-120b on a single H100 via vLLM, with fp8 KV cache, prefix caching, and a 64K context window. For each of the 50 problems, the system issues 6–8 parallel, independently-seeded attempts at temperature=1.0 and reasoning_effort=HIGH. Each attempt has access to a stateful Jupyter sandbox (math, numpy, sympy, itertools, collections) that the model can call mid-generation via Harmony tool-use messages. Output is streamed; as soon as the stream produces the pattern oxed{, the answer is extracted immediately — a deliberately minimal match that avoids relying on the closing }, which many notebooks use but which can fail if the model's output is truncated mid-stream.\nOnce attempts complete, answers are pooled and selected by simple majority vote — no entropy weighting. If no early consensus is reached (fewer than 3 agreements), a single tie-break round of 8 additional attempts is launched, budget permitting. When a numerical tie remains even after tie-breaking, the winner is chosen by centric selection: the answer closest to the weighted median of all votes.\nThe critical differentiator of this solution is dynamic time budgeting: time saved on easy problems flows automatically into hard ones, with the system continuously tracking its own time surplus and adjusting behavior in real time.\nThe full pipeline is illustrated in the flowchart below:\n
\nThe pipeline for each problem: time budget → parallel attempts → answer extraction → early stop check → majority vote → tie-break (if budget allows) → centric selection → classify hard/easy and update budget for next problem.
\nWhy I Dropped Entropy-Weighted Voting
\nThe notebook lineage I built on — tracing back to nihilisticneuralnet's 43/50 solution — uses entropy-weighted voting, where each attempt's vote is scaled by 1 / mean_token_entropy. The intuition is that lower entropy signals higher model confidence, and therefore greater likelihood of correctness. Many top solutions in this competition use this approach.\nI investigated entropy weighting thoroughly and decided to drop it in favor of plain majority vote. To be upfront: the base notebook with entropy scored 43/50, while my modified version scored 37/50 — so it is entirely possible that entropy weighting is genuinely helpful and that my other changes (reduced workers, different prompts, the time management overhead itself) are responsible for the score difference. I cannot isolate the effect cleanly. That said, here are the specific observations that led me to question entropy's value — offered as hypotheses, not conclusions.
\nObservation 1: Entropy has very weak separation on hard problems
\nOn a representative hard problem, the entropy values across 8 attempts ranged from 0.644 to 0.709 — roughly a 10% spread. The corresponding weights (1/entropy) ranged from ~1.41 to ~1.55. These weights are nearly identical. On hard problems, where the answer matters most, entropy weighting effectively collapses to plain majority voting.
\nObservation 2: Entropy actively misleads — low entropy favors wrong (or no) answers
\nThis is the strongest example I found. On a hard problem where the correct answer is 8687 (ground truth), here is what the 8 attempts produced:
\n\n\n\n| Attempt | \nPython Calls | \nPython Errors | \nEntropy | \nAnswer | \n
\n\n\n\n| 0 | \n40 | \n5 | \n0.565 | \n<NA\\> | \n
\n\n| 1 | \n24 | \n4 | \n0.602 | \n64836 | \n
\n\n| 2 | \n19 | \n1 | \n0.648 | \n71765 | \n
\n\n| 3 | \n30 | \n3 | \n0.558 | \n<NA\\> | \n
\n\n| 4 | \n42 | \n9 | \n0.582 | \n3107 | \n
\n\n| 5 | \n32 | \n7 | \n0.618 | \n78742 | \n
\n\n| 6 | \n18 | \n0 | \n0.594 | \n23 | \n
\n\n| 7 | \n54 | \n7 | \n0.599 | \n8687 ✅ | \n
\n\nFirst, note that attempts 0 and 3 (entropy 0.565 and 0.558) returned <NA> — the model failed to produce any answer at all. These are not \"confidently wrong\" answers; they are broken runs (likely timeouts or parsing failures). Comparing their entropy against valid answers is not meaningful, so let us set them aside. | \n | \n | \n | \n | \n
\n\n| Among the 6 valid answers only, the entropy ranking tells a clear story: | \n | \n | \n | \n | \n
\n\n| Rank (by entropy, lowest first) | \nEntropy | \nWeight (1/entropy) | \nAnswer | \nCorrect? | \n
\n\n| -------------------------------- | \n--------- | \n------------------- | \n-------- | \n---------- | \n
\n\n| 1st (most \"confident\") | \n0.582 | \n1.718 | \n3107 | \n❌ | \n
\n\n| 2nd | \n0.594 | \n1.684 | \n23 | \n❌ | \n
\n\n| 3rd | \n0.599 | \n1.669 | \n8687 | \n✅ | \n
\n\n| 4th | \n0.602 | \n1.661 | \n64836 | \n❌ | \n
\n\n| 5th | \n0.618 | \n1.616 | \n78742 | \n❌ | \n
\n\n| 6th (least \"confident\") | \n0.648 | \n1.543 | \n71765 | \n❌ | \n
\n\nUnder entropy-weighted voting, the wrong answer 3107 receives the highest weight (1.718) while the correct answer 8687 receives only 1.669 — the third-highest. Entropy actively prefers the wrong answer over the correct one. The two most \"confident\" attempts (3107 and 23) are both wrong; the correct answer sits in the middle of the confidence distribution. | \n | \n | \n | \n | \n
\n\n| Entropy measures token-level linguistic certainty — how confident the model is about its next word. But correctness on Olympiad math depends on logical validity, tool usage, and mathematical reasoning. These are fundamentally different signals. A model can be very \"certain\" at the token level while following a completely wrong mathematical path. | \n | \n | \n | \n | \n
\n\n
\nObservation 3: In the chaos regime, entropy selects the most confident wrong answer
\nWhen all attempts produce different answers (as in the example above — 6 valid answers, 6 different values), entropy weighting simply picks whichever wrong answer was generated most confidently. In this case, entropy would select 3107 over the correct 8687. In this regime, the right move is not better weighting — it is more sampling (tie-break rounds) or cutting losses early.
\nMy conclusion
\nEntropy weighting is not harmful on easy problems (where any voting method works), but it provides no benefit on hard problems (where the outcome matters). The real challenge is not weighting votes better — it is identifying when the model has meaningful signal vs. total disagreement and adapting strategy accordingly. That is what the time management system does.
\nIndependent corroboration
\nIndependently, Lê Bảo's 14th-place writeup documents a related phenomenon they call \"confident divergence.\" Their data shows the n-Norwegian problem has lower mean token entropy (0.701) than the Tournament problem (0.863) — yet Norwegian produces 6 completely distinct wrong answers while Tournament converges unanimously in 5 attempts. Their conclusion: \"entropy voting correctly handles uncertain attempts; it cannot handle confidently-wrong divergent ones.\" They also cite the AMO team (David Babu et al.) finding that entropy had \"only a weak correlation to correctness.\" It is worth noting that Lê Bảo uses entropy-weighted voting and scored 44/50 — so this is not a critique from someone who rejected it, but a limitation acknowledged by someone who kept it.
\nDynamic Time Management
\nThis is the core contribution. Most solutions use a static time budget (e.g., 300s base, 900s max per problem). The problem: easy problems that solve in 60s waste 240s of unused budget, while hard problems that need 800s get cut off at 300s. Over 50 problems, this misallocation adds up.
\nThe budget equation
\nAt the start of each problem, the system computes:
\navailable = time_left − future_need − safety_buffer\nbudget = clamp(available, MIN=200s, MAX=1300s)\n
\nWhere future_need is estimated from the observed hard/easy ratio so far:
\nfuture_need = hard_remaining × hard_avg + easy_remaining × easy_avg\n
\nThe system starts with a prior expectation of 14 hard problems out of 50. After a warmup period (5 problems solved), it switches to using the observed ratio. Hard/easy averages start at 750s and 150s respectively, then blend in actual measurements as data accumulates.
\nThree operating modes
\nThe system operates in one of three modes depending on the time situation:\nNORMAL — surplus time exists. The current problem gets available seconds (clamped to [200, 1300]). Easy problems that solve quickly return their unused time to the global pool, making it available for hard problems later.\nSURVIVAL — future need exceeds available time. The system distributes remaining time equally across remaining problems, no longer being generous with any single problem.\nPANIC — less than 120s per remaining problem. The system reduces to bare-minimum attempts (2–4 instead of 6–8) to ensure every problem gets at least a shot.
\nProblem classification
\nAfter each problem is solved, the system classifies it as \"hard\" or \"easy\" based on two signals: consensus strength (max vote frequency ≤ 2 = hard) and time consumed (> 400s without early stop = hard). This classification feeds back into future budget estimates.
\nTie-Breaking: Centric Selection
\nWhen majority voting produces a clear winner — one answer with strictly more votes than all others — that answer is selected immediately, no further logic needed. Centric selection only activates when there is a genuine tie: multiple answers sharing the maximum vote count. This is an important distinction — on most problems, majority vote decides cleanly and centric selection never runs.\nI use what I call centric selection for tie-breaking, based on a hypothesis that may or may not be correct.
\nThe hypothesis
\nOn hard problems where the model is struggling, the correct answer tends to be somewhere in the \"center\" of the distribution of all attempted answers, rather than at the extremes. Outlier answers are more likely to be the result of a reasoning failure that spiraled in an unusual direction.
\nThe mechanism
\nThe logic in _select_answer() is straightforward:
\nif len(top_answers) == 1:\n final_answer = top_answers[0] # Clear winner — done\nelse:\n # Tie: build weighted list of ALL valid answers (repeated by vote count)\n weighted = []\n for answer, data in sorted_stats:\n weighted.extend([answer] * data['votes'])\n weighted.sort()\n # Find the weighted median\n n = len(weighted)\n if n % 2 == 1:\n reference = weighted[n // 2]\n else:\n reference = (weighted[n // 2 - 1] + weighted[n // 2]) / 2\n # Among tied top answers, pick the one closest to this median\n final_answer = min(\n top_answers,\n key=lambda answer: (abs(answer - reference), answer)\n )\n
\nNote the key design choice: the weighted median is computed from all valid answers (not just the tied ones). This means the single-vote \"noise\" answers contribute to the reference point — they pull the median toward the region of answer space that the model explored most densely. An earlier version used the median of only the tied answers, but that ignores useful distributional information from the non-tied attempts.
\nWorked example
\nSuppose after a tie-break round on a hard problem (ground truth: 8687), the vote distribution is:
\n\n\n\n| Answer | \nVotes | \n
\n\n\n\n| 23 | \n2 | \n
\n\n| 8687 | \n2 | \n
\n\n| 41754 | \n2 | \n
\n\n| 5534 | \n1 | \n
\n\n| 98449 | \n1 | \n
\n\n| Three answers are tied at 2 votes each: 23, 8687, and 41754. Majority vote cannot decide. To break the tie, we build the weighted list by repeating each answer by its vote count: | \n | \n
\n\n
\n[23, 23, 5534, 8687, 8687, 41754, 41754, 98449]\n
\nThis list has 8 elements. The median is the average of the 4th and 5th elements:
\nreference = (8687 + 8687) / 2 = 8687\n
\nNow we measure each tied answer's distance to this reference:
\n\n- |23 − 8687| = 8,664
\n- |8687 − 8687| = 0
\n- |41754 − 8687| = 33,067\n8687 is exactly at the median — distance 0. Centric selection picks 8687, which is the correct answer.\nThe intuition: 23 is an outlier on the low end (likely from a reasoning path that collapsed early), 41754 is an outlier on the high end (likely from a computation that spiraled), and 8687 sits in the center — reachable from multiple reasoning paths. The single-vote answers (5534, 98449) contribute to the weighted median but don't compete in the tie, pulling the reference point toward the region where the model explored most densely.\nThis is the core hypothesis of centric selection: on hard problems where the model is struggling, the correct answer tends to be between the outliers rather than at the extremes. Outlier answers are more likely the result of reasoning failures that diverged in unusual directions.
\n
\nWhy not entropy for tie-breaking?
\nThe argument is the same as above: entropy does not reliably separate correct from incorrect on hard problems. The centric approach at least has a geometric intuition — outlier answers likely stem from divergent reasoning paths, while the correct answer is more likely to be \"reachable\" from multiple reasoning paths, placing it nearer to the center.
\nI want to be honest: this is a hypothesis, not a proven result. It worked in my tests, but I have not done a rigorous ablation to measure its marginal impact vs. random tie-breaking.
\nTie-Break Rounds: When to Spend More Compute
\nThe system permits up to 1 tie-break round of 8 additional attempts, but only under specific conditions:
\n\n- Sufficient problem time remaining (> 300s) — hard problems need 300–600s per attempt; anything less is wasted compute.
\n- Not in PANIC mode — if time is critically low, skip tie-breaking.
\n- Global surplus exists — the estimated time left after serving all remaining problems must justify the cost.
\n- Consensus is genuinely weak — if there is already a 3+ vote leader (even without a tie), there is no need.\nAfter a tie-break round, the system adapts: if consensus is still weak after the extra attempts (
max_freq_after <= 2), it raises the early_stop threshold by 1 — requiring one more agreement before declaring consensus. This prevents the system from locking in on a low-confidence answer after spending extra compute. If the top vote count reaches early_stop + 1, consensus is declared. Otherwise, the system accepts the best available answer via centric selection. \n
\nEmpirical observation
\nIn my test runs, tie-break rounds frequently converted weak signal into correct answers. For example, on one hard problem: Round 1 produced 9 attempts with 8 different answers (pure chaos), but after a tie-break round, the correct answer 8687 emerged with 4 votes — enough to win via majority.
\n\n\n\n| Phase | \nAnswers | \nVotes for correct | \n
\n\n\n\n| Round 1 (8 attempts) | \n8 unique answers | \n1 | \n
\n\n| After tie-break (+8) | \nconsensus forming | \n4 | \n
\n\n| Final result | \n8687 ✅ | \n4 votes | \n
\n\n
\nThis was one of the cases where the extra sampling paid off. But there were also problems where tie-breaking just added more noise. The decision of when to tie-break matters as much as the tie-break itself.
\nPython Execution Metrics: Noisy but Informative (Ignored in Voting)
\nAn interesting side observation: Python execution metrics (number of tool calls, number of errors) do not correlate cleanly with correctness. From the entropy evidence table above:
\n\n- Attempt 7 (54 calls, 7 errors) produced the correct answer
8687 \n- Attempt 6 (18 calls, 0 errors) produced the wrong answer
23\nThis makes sense intuitively: the model may hit errors while exploring the problem space (timeouts, overflows, wrong initial approaches), then recover and find the correct path. Conversely, a clean run might mean the model confidently followed a wrong approach without ever hitting a \"guardrail.\" \n
\nThese metrics are currently not used in voting. Whether they should be incorporated — perhaps as a filter rather than a weight — is an open question.
\nPrompts
\nThe system prompt is deliberately structured but not prescriptive about reasoning strategy:
\nYou are solving a high-level mathematical contest problem (AIMO-style).\n### GOAL\nProduce a correct VERIFIED answer. Elegance is irrelevant.\n### NON-NEGOTIABLE RULES:\n- Never Guess\n- Never compute astronomically large quantities directly\n (n!, a^(b!), 3^(b!), power towers, etc.).\n- If you derive a general rule, formula, or pattern:\n • You MUST justify why it applies in general.\n • You MUST verify it on small or edge cases before trusting it.\n- If a counterexample appears, discard or revise the approach immediately.\n- If multiple answers seem plausible, you MUST disambiguate using structure\n (not intuition, not magnitude, not frequency).\n- For arithmetic functions over huge parameters,\n first reduce to residue classes and prime-power families,\n derive the family formula symbolically,\n then aggregate exactly across all required cases.\n### OUTPUT CONTRACT (STRICT):\n- Output ONLY the final answer in LaTeX boxed form: \\boxed{...}\n- Exactly ONE boxed integer. No extra text.\n- The answer must be an integer in [0, 99999].\n
\nThis prompt sits in a middle ground between extreme minimalism (e.g., \"Solve the following math problem step by step\") and the heavily scaffolded approaches (UNDERSTAND→PLAN→EXECUTE→VERIFY loops) that others tried. The key rules (\"never guess,\" \"verify on small cases,\" \"reduce to residue classes\") are guardrails against common failure modes I observed, not reasoning prescriptions.
\nConfiguration
\n\n\n\n| Parameter | \nValue | \nNotes | \n
\n\n\n\n| Model | \ngpt-oss-120b | \nvLLM, fp8 KV cache, prefix caching | \n
\n\n| reasoning_effort | \nHIGH | \nHarmony system content | \n
\n\n| Context | \n65,536 tokens | \nmax_model_len | \n
\n\n| Temperature | \n1.0 | \n | \n
\n\n| min_p | \n0.02 | \n | \n
\n\n| Attempts per problem | \n6–8 | \n8 default; reduced in PANIC mode | \n
\n\n| Workers | \n6–8 | \nparallel threads per problem | \n
\n\n| Early-stop threshold | \n3 | \nagreements to stop early | \n
\n\n| Voting | \nMajority vote | \nno entropy weighting | \n
\n\n| Tie-breaking | \nCentric selection | \nweighted median proximity | \n
\n\n| Max tie-break rounds | \n1 | \n8 attempts per round | \n
\n\n| Min budget per problem | \n200s | \nfloor | \n
\n\n| Max budget per problem | \n1300s | \nceiling | \n
\n\n| Notebook runtime limit | \n18,000s | \n~5 hours | \n
\n\n| Safety margin | \n5% | \nof remaining time | \n
\n\n
\n| Seed | 42 | per-attempt seed = (42 + i)² |
\nResults and Reproducibility
\nI ran the notebook twice on the public test set:
\n\n\n\n| Run | \nWorkers | \nTie-break rounds | \nPublic Score | \n
\n\n\n\n| 1 | \n8 | \n1 | \n37/50 | \n
\n\n| 2 | \n6 | \n1 | \n36/50 | \n
\n\n| I selected the 37/50 run for submission. The private leaderboard score has not been published yet at the time of writing. | \n | \n | \n | \n
\n\n| Baseline estimate: 36–37 is the reproducible baseline. The 1-point variance between runs comes from stochastic tie-break outcomes on 2–3 hard problems. | \n | \n | \n | \n
\n\n| Stability note: For comparison, Lê Bảo's 14th-place solution (entropy-weighted voting, static 300–900s budget) submitted the same notebook three times and scored 44, 35, 35 — a 9-point spread. My two runs scored 37 and 36 — a 1-point spread. While their peak is higher, the variance is significant: in a competition where you pick one submission, a stable 36–37 may outperform a volatile distribution centered around 38 with tails at 35 and 44. The dynamic time management system likely contributes to this stability by ensuring no problem is starved of compute due to a previous problem overrunning its budget. | \n | \n | \n | \n
\n\n| Upper bound estimate: If the notebook runs twice and the tie-break rounds happen to flip in the right direction on the hard problems, a score of 37–41 feels plausible. This is speculative — the variance on hard problems is real, and I have not done enough runs to estimate the full distribution. | \n | \n | \n | \n
\n\n
\nHardware
\n\n- 1× NVIDIA H100 (Kaggle)
\n- Model weights:
danielhanchen/gpt-oss-120b (transformers format) \n
\n- Wheels: offline bundle from andreasbis/aimo-3-utils
\nWhat I Tried and Discarded
\nI want to briefly mention some things that did not work or that I chose not to pursue. This is the part I'm least confident about — some of these might work with more tuning.\nEntropy-weighted voting: As discussed above. My tests suggested it was at best neutral, at worst slightly misleading on hard problems. But I did not run a clean A/B test across all 50 problems — it is possible the effect is small but positive.\nMore tie-break rounds: I tried MAX_TIE_ROUNDS=2 and 3. Diminishing returns: the second round rarely flipped a problem that the first round didn't. Meanwhile, the extra time consumed could have been spent on subsequent hard problems.\nAggressive early stopping (threshold=2): Occasionally grabbed wrong answers that happened to agree by coincidence. Threshold=3 was the sweet spot.
\nMultiple prompt variants per problem: Running half the attempts with one prompt and half with another. Added complexity without measurably improving accuracy.
\nThe Three Regimes: A Mental Model
\nLooking across many problem runs, I found it useful to think in terms of three distinct regimes:\nStrong consensus — 4+ attempts agree. Any voting method works. Early stopping saves time. This is the easy/medium regime.\nWeak signal — The correct answer appears 2–3 times, surrounded by noise. This is where tie-break rounds and more sampling pay off. The right action is: spend more compute.\nChaos — All attempts produce different answers. No amount of reweighting helps. The right action is: stop early, save compute for later problems, and accept the best guess (or return 0).
\nThe dynamic time management system implicitly handles all three: early stopping for consensus, tie-breaking for weak signal, and PANIC/SURVIVAL modes to avoid blowing the budget on chaos.
\nFinal Thoughts
\nThe main insight I took from this competition is that how you allocate compute across problems matters more than how you aggregate votes within a problem. Entropy weighting, verifier loops, multi-stage pipelines — these all focus on squeezing more signal from a fixed set of attempts. But the bigger lever is ensuring that hard problems get enough attempts in the first place, and that easy problems don't waste time they don't need.\nWhether the specific mechanisms I used (centric selection, the budget equation, the three-mode system) are optimal is an open question. They worked for me, but I suspect there is room for improvement in each one.
\nThis was my deep dive into inference-time engineering for mathematical reasoning, and much of what I've written above reflects intuitions formed during experimentation rather than rigorously controlled results. If any of these observations resonate — or if you have evidence that contradicts them — I'd welcome the discussion.
\nCompetition: AI Mathematical Olympiad — Progress Prize 3
",
"rawMarkdown": "\nAuthor: Ayman Hamed\n\nCompetition: AI Mathematical Olympiad — Progress Prize 3 (AIMO3)\n\nPublic Leaderboard: 37/50\n\nKaggle Notebook (submitted): [aimo-39](https://www.kaggle.com/code/ayman3000/aimo-39?scriptVersionId=311375491)\n\nBase Model: gpt-oss-120b\n\n---\n\n## Acknowledgments\n\nThis solution builds on [Siancy's public notebook](https://www.kaggle.com/code/shreyansh01m/41-50-same-as-parthenos-s-no-modifications). The core infrastructure — vLLM serving, Harmony encoding, sandboxed Jupyter tool execution, parallel attempts, early stopping, entropy-weighted voting — is theirs. Thanks also to [andreasbis/aimo-3-utils](https://www.kaggle.com/datasets/andreasbis/aimo-3-utils) for the offline wheel bundle. My changes (+170, −23 lines from the base) were concentrated in three areas: **dynamic time management**, **dropping entropy-weighted voting in favor of majority vote with centric tie-breaking**, and **prompt adjustments**. These are detailed below, along with the reasoning and empirical observations behind each decision.\n\n*Note: This writeup documents my personal experience, observations, and hypotheses from this competition. Many of the conclusions here — particularly around entropy, centric selection, and time management — are based on limited experiments and may be wrong or negotiable. I share them in the spirit of open discussion, hoping they spark useful conversation rather than claiming definitive results.*\n\n---\n\n## Overview\n\nThe solver runs `gpt-oss-120b` on a single H100 via vLLM, with fp8 KV cache, prefix caching, and a 64K context window. For each of the 50 problems, the system issues 6–8 parallel, independently-seeded attempts at `temperature=1.0` and `reasoning_effort=HIGH`. Each attempt has access to a stateful Jupyter sandbox (`math`, `numpy`, `sympy`, `itertools`, `collections`) that the model can call mid-generation via Harmony tool-use messages. Output is streamed; as soon as the stream produces the pattern `oxed{`, the answer is extracted immediately — a deliberately minimal match that avoids relying on the closing `}`, which many notebooks use but which can fail if the model's output is truncated mid-stream.\n\nOnce attempts complete, answers are pooled and selected by **simple majority vote** — no entropy weighting. If no early consensus is reached (fewer than 3 agreements), a **single tie-break round** of 8 additional attempts is launched, budget permitting. When a numerical tie remains even after tie-breaking, the winner is chosen by **centric selection**: the answer closest to the weighted median of all votes.\n\nThe critical differentiator of this solution is **dynamic time budgeting**: time saved on easy problems flows automatically into hard ones, with the system continuously tracking its own time surplus and adjusting behavior in real time.\n\nThe full pipeline is illustrated in the flowchart below:\n\n\n\n*The pipeline for each problem: time budget → parallel attempts → answer extraction → early stop check → majority vote → tie-break (if budget allows) → centric selection → classify hard/easy and update budget for next problem.*\n\n---\n\n## Why I Dropped Entropy-Weighted Voting\n\nThe notebook lineage I built on — tracing back to nihilisticneuralnet's 43/50 solution — uses entropy-weighted voting, where each attempt's vote is scaled by `1 / mean_token_entropy`. The intuition is that lower entropy signals higher model confidence, and therefore greater likelihood of correctness. Many top solutions in this competition use this approach.\n\nI investigated entropy weighting thoroughly and decided to drop it in favor of plain majority vote. To be upfront: the base notebook with entropy scored 43/50, while my modified version scored 37/50 — so it is entirely possible that entropy weighting is genuinely helpful and that my other changes (reduced workers, different prompts, the time management overhead itself) are responsible for the score difference. I cannot isolate the effect cleanly. That said, here are the specific observations that led me to question entropy's value — offered as hypotheses, not conclusions.\n\n### Observation 1: Entropy has very weak separation on hard problems\n\nOn a representative hard problem, the entropy values across 8 attempts ranged from 0.644 to 0.709 — roughly a 10% spread. The corresponding weights (`1/entropy`) ranged from ~1.41 to ~1.55. These weights are nearly identical. On hard problems, where the answer matters most, entropy weighting effectively collapses to plain majority voting.\n\n### Observation 2: Entropy actively misleads — low entropy favors wrong (or no) answers\n\nThis is the strongest example I found. On a hard problem where the correct answer is `8687` (ground truth), here is what the 8 attempts produced:\n\n| Attempt | Python Calls | Python Errors | Entropy | Answer |\n|---------|-------------|---------------|---------|--------|\n| 0 | 40 | 5 | 0.565 | \\ |\n| 1 | 24 | 4 | 0.602 | 64836 |\n| 2 | 19 | 1 | 0.648 | 71765 |\n| 3 | 30 | 3 | 0.558 | \\ |\n| 4 | 42 | 9 | 0.582 | 3107 |\n| 5 | 32 | 7 | 0.618 | 78742 |\n| 6 | 18 | 0 | 0.594 | 23 |\n| 7 | 54 | 7 | 0.599 | **8687** ✅ |\n\nFirst, note that attempts 0 and 3 (entropy 0.565 and 0.558) returned `` — the model failed to produce any answer at all. These are not \"confidently wrong\" answers; they are broken runs (likely timeouts or parsing failures). Comparing their entropy against valid answers is not meaningful, so let us set them aside.\n\nAmong the **6 valid answers only**, the entropy ranking tells a clear story:\n\n| Rank (by entropy, lowest first) | Entropy | Weight (1/entropy) | Answer | Correct? |\n|--------------------------------|---------|-------------------|--------|----------|\n| 1st (most \"confident\") | **0.582** | **1.718** | 3107 | ❌ |\n| 2nd | 0.594 | 1.684 | 23 | ❌ |\n| 3rd | 0.599 | 1.669 | **8687** | ✅ |\n| 4th | 0.602 | 1.661 | 64836 | ❌ |\n| 5th | 0.618 | 1.616 | 78742 | ❌ |\n| 6th (least \"confident\") | 0.648 | 1.543 | 71765 | ❌ |\n\nUnder entropy-weighted voting, the wrong answer `3107` receives the highest weight (1.718) while the correct answer `8687` receives only 1.669 — the third-highest. Entropy actively prefers the wrong answer over the correct one. The two most \"confident\" attempts (3107 and 23) are both wrong; the correct answer sits in the middle of the confidence distribution.\n\nEntropy measures token-level linguistic certainty — how confident the model is about its next word. But correctness on Olympiad math depends on logical validity, tool usage, and mathematical reasoning. These are fundamentally different signals. A model can be very \"certain\" at the token level while following a completely wrong mathematical path.\n\n### Observation 3: In the chaos regime, entropy selects the most confident wrong answer\n\nWhen all attempts produce different answers (as in the example above — 6 valid answers, 6 different values), entropy weighting simply picks whichever wrong answer was generated most confidently. In this case, entropy would select `3107` over the correct `8687`. In this regime, the right move is not better *weighting* — it is more *sampling* (tie-break rounds) or cutting losses early.\n\n### My conclusion\n\nEntropy weighting is not harmful on easy problems (where any voting method works), but it provides no benefit on hard problems (where the outcome matters). The real challenge is not weighting votes better — it is identifying *when the model has meaningful signal vs. total disagreement* and adapting strategy accordingly. That is what the time management system does.\n\n### Independent corroboration\n\nIndependently, [Lê Bảo's 14th-place writeup](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/571489) documents a related phenomenon they call \"confident divergence.\" Their data shows the n-Norwegian problem has *lower* mean token entropy (0.701) than the Tournament problem (0.863) — yet Norwegian produces 6 completely distinct wrong answers while Tournament converges unanimously in 5 attempts. Their conclusion: \"entropy voting correctly handles uncertain attempts; it cannot handle confidently-wrong divergent ones.\" They also cite the AMO team (David Babu et al.) finding that entropy had \"only a weak correlation to correctness.\" It is worth noting that Lê Bảo *uses* entropy-weighted voting and scored 44/50 — so this is not a critique from someone who rejected it, but a limitation acknowledged by someone who kept it.\n\n---\n\n## Dynamic Time Management\n\nThis is the core contribution. Most solutions use a static time budget (e.g., 300s base, 900s max per problem). The problem: easy problems that solve in 60s waste 240s of unused budget, while hard problems that need 800s get cut off at 300s. Over 50 problems, this misallocation adds up.\n\n### The budget equation\n\nAt the start of each problem, the system computes:\n\n```\navailable = time_left − future_need − safety_buffer\nbudget = clamp(available, MIN=200s, MAX=1300s)\n```\n\nWhere `future_need` is estimated from the observed hard/easy ratio so far:\n\n```\nfuture_need = hard_remaining × hard_avg + easy_remaining × easy_avg\n```\n\nThe system starts with a prior expectation of 14 hard problems out of 50. After a warmup period (5 problems solved), it switches to using the observed ratio. Hard/easy averages start at 750s and 150s respectively, then blend in actual measurements as data accumulates.\n\n### Three operating modes\n\nThe system operates in one of three modes depending on the time situation:\n\n**NORMAL** — surplus time exists. The current problem gets `available` seconds (clamped to [200, 1300]). Easy problems that solve quickly return their unused time to the global pool, making it available for hard problems later.\n\n**SURVIVAL** — future need exceeds available time. The system distributes remaining time equally across remaining problems, no longer being generous with any single problem.\n\n**PANIC** — less than 120s per remaining problem. The system reduces to bare-minimum attempts (2–4 instead of 6–8) to ensure every problem gets at least a shot.\n\n### Problem classification\n\nAfter each problem is solved, the system classifies it as \"hard\" or \"easy\" based on two signals: consensus strength (max vote frequency ≤ 2 = hard) and time consumed (> 400s without early stop = hard). This classification feeds back into future budget estimates.\n\n---\n\n## Tie-Breaking: Centric Selection\n\nWhen majority voting produces a clear winner — one answer with strictly more votes than all others — that answer is selected immediately, no further logic needed. Centric selection **only activates when there is a genuine tie**: multiple answers sharing the maximum vote count. This is an important distinction — on most problems, majority vote decides cleanly and centric selection never runs.\n\nI use what I call **centric selection** for tie-breaking, based on a hypothesis that may or may not be correct.\n\n### The hypothesis\n\nOn hard problems where the model is struggling, the correct answer tends to be somewhere in the \"center\" of the distribution of all attempted answers, rather than at the extremes. Outlier answers are more likely to be the result of a reasoning failure that spiraled in an unusual direction.\n\n### The mechanism\n\nThe logic in `_select_answer()` is straightforward:\n\n```python\nif len(top_answers) == 1:\n final_answer = top_answers[0] # Clear winner — done\nelse:\n # Tie: build weighted list of ALL valid answers (repeated by vote count)\n weighted = []\n for answer, data in sorted_stats:\n weighted.extend([answer] * data['votes'])\n weighted.sort()\n\n # Find the weighted median\n n = len(weighted)\n if n % 2 == 1:\n reference = weighted[n // 2]\n else:\n reference = (weighted[n // 2 - 1] + weighted[n // 2]) / 2\n\n # Among tied top answers, pick the one closest to this median\n final_answer = min(\n top_answers,\n key=lambda answer: (abs(answer - reference), answer)\n )\n```\n\nNote the key design choice: the weighted median is computed from **all valid answers** (not just the tied ones). This means the single-vote \"noise\" answers contribute to the reference point — they pull the median toward the region of answer space that the model explored most densely. An earlier version used the median of only the tied answers, but that ignores useful distributional information from the non-tied attempts.\n\n### Worked example\n\nSuppose after a tie-break round on a hard problem (ground truth: 8687), the vote distribution is:\n\n| Answer | Votes |\n|--------|-------|\n| 23 | 2 |\n| 8687 | 2 |\n| 41754 | 2 |\n| 5534 | 1 |\n| 98449 | 1 |\n\nThree answers are tied at 2 votes each: 23, 8687, and 41754. Majority vote cannot decide. To break the tie, we build the weighted list by repeating each answer by its vote count:\n\n```\n[23, 23, 5534, 8687, 8687, 41754, 41754, 98449]\n```\n\nThis list has 8 elements. The median is the average of the 4th and 5th elements:\n\n```\nreference = (8687 + 8687) / 2 = 8687\n```\n\nNow we measure each tied answer's distance to this reference:\n\n- |23 − 8687| = **8,664**\n- |8687 − 8687| = **0**\n- |41754 − 8687| = **33,067**\n\n8687 is exactly at the median — distance 0. Centric selection picks **8687**, which is the correct answer.\n\nThe intuition: 23 is an outlier on the low end (likely from a reasoning path that collapsed early), 41754 is an outlier on the high end (likely from a computation that spiraled), and 8687 sits in the center — reachable from multiple reasoning paths. The single-vote answers (5534, 98449) contribute to the weighted median but don't compete in the tie, pulling the reference point toward the region where the model explored most densely.\n\nThis is the core hypothesis of centric selection: on hard problems where the model is struggling, the correct answer tends to be between the outliers rather than at the extremes. Outlier answers are more likely the result of reasoning failures that diverged in unusual directions.\n\n### Why not entropy for tie-breaking?\n\nThe argument is the same as above: entropy does not reliably separate correct from incorrect on hard problems. The centric approach at least has a geometric intuition — outlier answers likely stem from divergent reasoning paths, while the correct answer is more likely to be \"reachable\" from multiple reasoning paths, placing it nearer to the center.\n\n*I want to be honest: this is a hypothesis, not a proven result. It worked in my tests, but I have not done a rigorous ablation to measure its marginal impact vs. random tie-breaking.*\n\n---\n\n## Tie-Break Rounds: When to Spend More Compute\n\nThe system permits up to 1 tie-break round of 8 additional attempts, but only under specific conditions:\n\n- **Sufficient problem time remaining** (> 300s) — hard problems need 300–600s per attempt; anything less is wasted compute.\n- **Not in PANIC mode** — if time is critically low, skip tie-breaking.\n- **Global surplus exists** — the estimated time left after serving all remaining problems must justify the cost.\n- **Consensus is genuinely weak** — if there is already a 3+ vote leader (even without a tie), there is no need.\n\nAfter a tie-break round, the system adapts: if consensus is still weak after the extra attempts (`max_freq_after <= 2`), it raises the `early_stop` threshold by 1 — requiring one more agreement before declaring consensus. This prevents the system from locking in on a low-confidence answer after spending extra compute. If the top vote count reaches `early_stop + 1`, consensus is declared. Otherwise, the system accepts the best available answer via centric selection.\n\n### Empirical observation\n\nIn my test runs, tie-break rounds frequently converted weak signal into correct answers. For example, on one hard problem: Round 1 produced 9 attempts with 8 different answers (pure chaos), but after a tie-break round, the correct answer `8687` emerged with 4 votes — enough to win via majority.\n\n| Phase | Answers | Votes for correct |\n|-------|---------|------------------|\n| Round 1 (8 attempts) | 8 unique answers | 1 |\n| After tie-break (+8) | consensus forming | 4 |\n| Final result | 8687 ✅ | 4 votes |\n\nThis was one of the cases where the extra sampling paid off. But there were also problems where tie-breaking just added more noise. The decision of *when* to tie-break matters as much as the tie-break itself.\n\n---\n\n## Python Execution Metrics: Noisy but Informative (Ignored in Voting)\n\nAn interesting side observation: Python execution metrics (number of tool calls, number of errors) do not correlate cleanly with correctness. From the entropy evidence table above:\n\n- Attempt 7 (54 calls, 7 errors) produced the correct answer `8687`\n- Attempt 6 (18 calls, 0 errors) produced the wrong answer `23`\n\nThis makes sense intuitively: the model may hit errors while exploring the problem space (timeouts, overflows, wrong initial approaches), then recover and find the correct path. Conversely, a clean run might mean the model confidently followed a wrong approach without ever hitting a \"guardrail.\"\n\nThese metrics are currently not used in voting. Whether they *should* be incorporated — perhaps as a filter rather than a weight — is an open question.\n\n---\n\n## Prompts\n\nThe system prompt is deliberately structured but not prescriptive about reasoning strategy:\n\n```\nYou are solving a high-level mathematical contest problem (AIMO-style).\n\n### GOAL\nProduce a correct VERIFIED answer. Elegance is irrelevant.\n\n### NON-NEGOTIABLE RULES:\n- Never Guess\n- Never compute astronomically large quantities directly\n (n!, a^(b!), 3^(b!), power towers, etc.).\n- If you derive a general rule, formula, or pattern:\n • You MUST justify why it applies in general.\n • You MUST verify it on small or edge cases before trusting it.\n- If a counterexample appears, discard or revise the approach immediately.\n- If multiple answers seem plausible, you MUST disambiguate using structure\n (not intuition, not magnitude, not frequency).\n- For arithmetic functions over huge parameters,\n first reduce to residue classes and prime-power families,\n derive the family formula symbolically,\n then aggregate exactly across all required cases.\n\n### OUTPUT CONTRACT (STRICT):\n- Output ONLY the final answer in LaTeX boxed form: \\boxed{...}\n- Exactly ONE boxed integer. No extra text.\n- The answer must be an integer in [0, 99999].\n```\n\nThis prompt sits in a middle ground between extreme minimalism (e.g., \"Solve the following math problem step by step\") and the heavily scaffolded approaches (UNDERSTAND→PLAN→EXECUTE→VERIFY loops) that others tried. The key rules (\"never guess,\" \"verify on small cases,\" \"reduce to residue classes\") are guardrails against common failure modes I observed, not reasoning prescriptions.\n\n---\n\n## Configuration\n\n| Parameter | Value | Notes |\n|-----------|-------|-------|\n| Model | gpt-oss-120b | vLLM, fp8 KV cache, prefix caching |\n| reasoning_effort | HIGH | Harmony system content |\n| Context | 65,536 tokens | max_model_len |\n| Temperature | 1.0 | |\n| min_p | 0.02 | |\n| Attempts per problem | 6–8 | 8 default; reduced in PANIC mode |\n| Workers | 6–8 | parallel threads per problem |\n| Early-stop threshold | 3 | agreements to stop early |\n| Voting | Majority vote | no entropy weighting |\n| Tie-breaking | Centric selection | weighted median proximity |\n| Max tie-break rounds | 1 | 8 attempts per round |\n| Min budget per problem | 200s | floor |\n| Max budget per problem | 1300s | ceiling |\n| Notebook runtime limit | 18,000s | ~5 hours |\n| Safety margin | 5% | of remaining time |\n| Seed | 42 | per-attempt seed = (42 + i)² |\n\n---\n\n## Results and Reproducibility\n\nI ran the notebook twice on the public test set:\n\n| Run | Workers | Tie-break rounds | Public Score |\n|-----|---------|-----------------|-------------|\n| 1 | 8 | 1 | **37/50** |\n| 2 | 6 | 1 | 36/50 |\n\nI selected the 37/50 run for submission. The private leaderboard score has not been published yet at the time of writing.\n\n**Baseline estimate:** 36–37 is the reproducible baseline. The 1-point variance between runs comes from stochastic tie-break outcomes on 2–3 hard problems.\n\n**Stability note:** For comparison, Lê Bảo's 14th-place solution (entropy-weighted voting, static 300–900s budget) submitted the same notebook three times and scored 44, 35, 35 — a 9-point spread. My two runs scored 37 and 36 — a 1-point spread. While their peak is higher, the variance is significant: in a competition where you pick one submission, a stable 36–37 may outperform a volatile distribution centered around 38 with tails at 35 and 44. The dynamic time management system likely contributes to this stability by ensuring no problem is starved of compute due to a previous problem overrunning its budget.\n\n**Upper bound estimate:** If the notebook runs twice and the tie-break rounds happen to flip in the right direction on the hard problems, a score of 37–41 feels plausible. This is speculative — the variance on hard problems is real, and I have not done enough runs to estimate the full distribution.\n\n### Hardware\n\n- 1× NVIDIA H100 (Kaggle)\n- Model weights: `danielhanchen/gpt-oss-120b` (transformers format)\n- Wheels: offline bundle from `andreasbis/aimo-3-utils`\n\n---\n\n## What I Tried and Discarded\n\nI want to briefly mention some things that *did not* work or that I chose not to pursue. This is the part I'm least confident about — some of these might work with more tuning.\n\n**Entropy-weighted voting:** As discussed above. My tests suggested it was at best neutral, at worst slightly misleading on hard problems. But I did not run a clean A/B test across all 50 problems — it is possible the effect is small but positive.\n\n**More tie-break rounds:** I tried `MAX_TIE_ROUNDS=2` and `3`. Diminishing returns: the second round rarely flipped a problem that the first round didn't. Meanwhile, the extra time consumed could have been spent on subsequent hard problems.\n\n**Aggressive early stopping (threshold=2):** Occasionally grabbed wrong answers that happened to agree by coincidence. Threshold=3 was the sweet spot.\n\n**Multiple prompt variants per problem:** Running half the attempts with one prompt and half with another. Added complexity without measurably improving accuracy.\n\n---\n\n## The Three Regimes: A Mental Model\n\nLooking across many problem runs, I found it useful to think in terms of three distinct regimes:\n\n**Strong consensus** — 4+ attempts agree. Any voting method works. Early stopping saves time. This is the easy/medium regime.\n\n**Weak signal** — The correct answer appears 2–3 times, surrounded by noise. This is where tie-break rounds and more sampling pay off. The right action is: spend more compute.\n\n**Chaos** — All attempts produce different answers. No amount of reweighting helps. The right action is: stop early, save compute for later problems, and accept the best guess (or return 0).\n\nThe dynamic time management system implicitly handles all three: early stopping for consensus, tie-breaking for weak signal, and PANIC/SURVIVAL modes to avoid blowing the budget on chaos.\n\n---\n\n## Final Thoughts\n\nThe main insight I took from this competition is that **how you allocate compute across problems matters more than how you aggregate votes within a problem**. Entropy weighting, verifier loops, multi-stage pipelines — these all focus on squeezing more signal from a fixed set of attempts. But the bigger lever is ensuring that hard problems *get enough attempts in the first place*, and that easy problems don't waste time they don't need.\n\nWhether the specific mechanisms I used (centric selection, the budget equation, the three-mode system) are optimal is an open question. They worked for me, but I suspect there is room for improvement in each one.\n\nThis was my deep dive into inference-time engineering for mathematical reasoning, and much of what I've written above reflects intuitions formed during experimentation rather than rigorously controlled results. If any of these observations resonate — or if you have evidence that contradicts them — I'd welcome the discussion.\n\n---\n\n*Competition: [AI Mathematical Olympiad — Progress Prize 3](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3)*\n"
}
],
"complete": true,
"source": "live",
"fetched_at": "2026-10-20T04:25:43Z",
"competition_id": 118448,
"competition_slug": "ai-mathematical-olympiad-progress-prize-3",
"team_id": 14804072,
"solution_url": "/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/spending-time-wisely-ignoring-entropy-ai-mathem"
}