![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F6226045%2Fcbeedc562796501bc8a0c10077a43582%2F01_leaderboard.png?generation=1779134369635658&alt=media\")
Pawan Mali
\nAIMO Progress Prize 3 Competition Writeup - Updated May 2026
\nThis writeup documents three months of inference-time scaling work on the AIMO Progress Prize 3 competition, where the task is to solve 50 hidden IMO-level mathematical problems on a single NVIDIA H100 within a 4.8-hour budget. Across 60+ submitted versions we converged on a configuration we call V143: GPT-OSS-120B with temp=1.0, an 8-attempt parallel-sampling pipeline, early-stopping on 4-vote consensus, a 5-component weighted entropy score for confidence ranking, and a 65K context window with FP8 KV cache. V40 (an earlier iteration of the same config) scored 42/50, and V143 reproduced 41/50 on the hidden test set. After the competition closed we rebuilt the pipeline outside Kaggle with per-attempt instrumentation and rented an H100 80GB to validate against the public 10-problem reference set; that run scored 10/10 in 31 minutes, total cost $5.50, with a full per-attempt CSV preserved. The most surprising finding from the instrumented run: on one reference problem all 8 parallel attempts produced 8 different answers, and the entropy ranking still picked the correct one based on confidence alone. The rest of this writeup catalogues both what worked and the considerably longer list of things that didn't verbose prompts (-9), Bayesian voting (-11), self-refinement (-4), fine-tuning without matching the inference pipeline (-6), bundled \"improvements\" (-6), and a context-length finding that turned out to be confounded with throughput.
AIMO Progress Prize 3 sits at the intersection of two hard problems: mathematical reasoning at IMO level, and inference-time scaling under realistic compute constraints. Fifty hidden problems, answers must be non-negative integers between 0 and 99,999, evaluated on a single H100 with roughly 4.8 hours of wall-clock budget. A $2.2M prize was on offer for 47/50; no team reached it. The winner (ippeiogawa) achieved 46/50; a cluster of teams sat at 44–45.
\nThis writeup documents 60+ submitted versions between January and April 2026. The peak was 42/50 (V40), and we settled on a stable 41/50 baseline (V125, V132, V133, V143). The numerical headline is not the most useful thing here it's that across the same number of versions trying to improve on that baseline, roughly the same number of changes regressed it. Most regressions were not subtle: -6 to -11 points on a single configuration change.
\nThe contributions of this work are three:
\n
Figure 1: Our submissions in the context of the final AIMO3 leaderboard. The top of the leaderboard (gray) clusters between 44 and 46 with the winner ippeiogawa at 46/50. Our peak V40 (42/50, dark green) and stable V143 baseline (41/50, mid green) sit just below the leader cluster, with V148 (40/50) slightly behind. Our two largest regressions V135 (131K context, 35/50) and V142 (Bayesian + VOI voting, 30/50) are highlighted in purple. The shaded band shows the ±3.5-point variance observed across re-runs of V143; the red dashed line marks the 47/50 prize threshold that no team reached.
\nSeveral lines of work inform the design used here. Chain-of-thought prompting (Wei et al., 2022) demonstrated that step-by-step reasoning markedly improves mathematical performance for large models. Self-consistency (Wang et al., 2022) extended that by sampling multiple chains and selecting the majority answer, which is the conceptual foundation of our 8-attempt voting pipeline. More recent test-time-compute work (Snell et al., 2024; Brown et al., 2024) systematically explores how inference budget translates into accuracy gains, and finds that sample-and-vote scales favourably until a diminishing-returns regime. Our experiments are consistent with that, but with the additional finding that the quality of the aggregation function (here, 5-component weighted entropy rather than simple majority) matters at this size.
\nTool-integrated reasoning letting the model emit Python that gets executed in a sandbox has proven particularly effective on mathematical tasks. Our pipeline runs each attempt with access to an isolated Jupyter kernel; the model generates code, the kernel returns results, the model continues. This combines pattern recognition with arithmetic precision.
\nSelf-refinement approaches have shown mixed results. Huang et al. (2023) found that LLMs cannot reliably self-correct reasoning errors without external feedback, and that self-refinement often introduces errors into previously-correct answers. Our V134 experiment is consistent with this: adding a self-refinement step regressed our score by 4 points (within the variance band, but reliably negative across multiple submissions).
\nIn the AIMO competition series specifically, the first progress prize topped out around 29/50, the second edition at roughly 35/50 (with the winning solution introducing a GenSelect-style LLM-as-judge over candidates), and AIMO-3 leaderboard winners reached 46/50. Public AIMO-3 notebooks particularly nihilisticneuralnet (44/50) and huikang established baselines that influenced our search space.
\nWe tested several base models before committing to GPT-OSS-120B. DeepSeek-R1-32B reasoned well but its capacity capped scores around 28–32. Qwen2.5-Math-72B was strong on math but its 32K context limited the length of tool-using reasoning chains, putting it in the 30–35 range. GPT-OSS-120B at 120B parameters and 131K native context consistently scored 38–42 across our pipeline iterations, and was our choice for everything subsequent.
\nThe model is MXFP4-quantized in its public release (openai/gpt-oss-120b), which requires a GPU with compute capability ≥ 8.0 (Ampere or newer). On the H100 the entire 60 GB of weights fits comfortably with a 6 GB FP8 KV cache and an 8 GB safety buffer in 80 GB.
The pipeline has three stages per problem:
\n\\boxed{} answer is produced or the deadline is hit.k=5). At the end of each attempt we summarise the entropy sequence with a 5-component score (§3.3) and store it.1 / max(E, ε) to its extracted answer; the answer with the highest summed weight is selected. Early-stopping triggers when any answer reaches 4 votes, cancelling remaining attempts and freeing the budget for the next problem.For each token t with top-k logprobs we compute Shannon entropy H_t = −Σᵢ pᵢ · log₂(pᵢ). The per-attempt entropy sequence {H₁ … Hₙ} is summarised by a five-component score:
E = 0.3 · mean(H)\n + 0.4 · position_weighted_mean(H, decay=0.995)\n + 0.2 · std(H)\n + 0.9 · |{t : H_t > 2.0}| / n\n − 0.1 · longest_run(H_t < 0.5) / n\nThe components emphasise different aspects of the confidence signal. Position weighting (40% of the score, exponential decay 0.995 favouring recent tokens) reflects an empirical observation: mathematical solutions exhibit exploratory uncertainty early but should converge to confident conclusions, and the final-answer tokens carry most of the signal. The high-entropy-fraction term (r_high, threshold 2.0 bits) penalises attempts with extended uncertain stretches even when the overall mean is moderate. The low-entropy-streak bonus (s_low, threshold 0.5 bits) is a small reward for sustained-confidence regions.
The vote weight for each attempt is 1 / max(E, ε). Lower E means more confident, contributes more weight.
Ablation against a simple-mean-entropy baseline (V136 vs V125) showed a 6-point regression outside the noise band, the largest defensible single-axis result we have.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F6226045%2F6cc0140a58e45cf06cb05109d2be1839%2F02_entropy_components.png?generation=1779134468518859&alt=media\")
Figure 2: The 5-component entropy mechanism in action on real V143 attempt logs. Left panel: all 8 V143 attempts on problem 86e8e5 from the reference reproduction, showing the per-attempt extracted answer and the 5-component entropy score E. Every attempt produced a different answer (50505, 54680, 74763, 8687, 51379, 41754, 93764, 13657), so there was no consensus to vote on; attempt 3 (green) was selected because its E of 0.500 was the lowest, and its answer 8687 was the reference value. Right panel: the per-component decomposition of the winning attempt versus the runner-up (attempt 5, ans=41754, E=0.514). The winning margin of 0.014 is driven by the position-weighted term (40% of the score, favouring confident final-answer tokens) and the standard-deviation term (20%, penalising inconsistent confidence).
\nOur best-performing configuration (V40, 42/50) uses a three-sentence system prompt: identify the model as an IMO competitor, specify the answer range, require the boxed format. Adding structured problem-solving frameworks, verification checklists, or detailed mathematical guidance consistently regressed the score by 4–9 points. We hypothesise three contributing factors: (a) GPT-OSS-120B was trained on extensive mathematical content and already possesses effective problem-solving strategies that elaborate prompts may override; (b) prompt tokens displace solution tokens within a fixed context window; (c) detailed instructions can introduce conflicting guidance.
\nTable 1 summarises our key submissions on the hidden 50-problem competition test set. V40 was our best at 42/50; V125, V132, V133, V143 established a reproducible 41/50 baseline.
\nPost-competition reference reproduction. To address the reproducibility question and to provide per-problem evidence for the analyses in §4.2 and §4.4, we re-ran the V143 configuration on the public 10-problem AIMO3 reference set after the competition closed. The run used a single NVIDIA H100 80GB SXM rented from RunPod (AP-IN-1 data centre, $2.99/hr) with the standalone runner published in our runpod/ directory. Result: 10/10 correct in 1869 seconds (31 minutes), mean wall time 187 s per problem, total spend $5.50. The pipeline generated 1.2 million response tokens, made 1056 Python tool invocations, recorded 202 Python errors (which the voting absorbed without issue), and stored the five entropy components, completion status, and timing for each of 80 attempts.
This is the first end-to-end reproducible evidence we can publish; the hidden 50-problem test set is closed post-competition and cannot be re-scored externally.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F6226045%2F2b1ff8e76545c9412e72dbff773b3df0%2F03_per_version_trajectory.png?generation=1779134504249855&alt=media\")
Figure 3: Score on the hidden 50-problem test set for every submission from V8 to V158 (January to April 2026). The shaded band is the ±3.5-point variance observed across re-runs of the V143 configuration; the dashed line marks the V143 baseline (41/50) and the dotted line marks our V40 ceiling (42/50). Points are coloured by outcome class: V40 (dark green) is our only run above the baseline band; V125 / V132 / V133 / V143 (mid green) sit on the baseline; light green points are above the baseline but inside the band; gray points are within the noise band; light purple are mild regressions (V126 / V127 / V128 / V131 / V134 / V135 / V139 / V141 / V144); dark purple are severe regressions (V124, V142). The orange X markers along the bottom are OOM or crashed runs (14 of them). Only V142 (-11) and V127 (-9) clearly escape the band downward, and no run cleanly clears V40 upward.
\nTable 1: Key Versions on the Hidden Test Set
\n| Version | \nDate | \nScore | \nConfiguration | \n
|---|---|---|---|
| V40 | \nFeb 6 | \n42/50 | \nSimple prompts, temp=1.0, 5-comp entropy (best ever) | \n
| V111 | \nMar 2 | \n40/50 | \n5-comp entropy isolated | \n
| V125 | \nMar 19 | \n41/50 | \ntemp=1.0 + top_p=0.8 | \n
| V132 | \nMar 26 | \n41/50 | \nNo top_p, 5-comp entropy | \n
| V143 | \nApr 6 | \n41/50 | \nStable baseline restored | \n
| V127 | \nMar 22 | \n33/50 | \nVerbose prompts (-9) | \n
| V128 | \nMar 24 | \n35/50 | \nFine-tuned model (-6) | \n
| V131 | \nMar 25 | \n35/50 | \n6 techniques bundled (-6) | \n
| V135 | \nMar 29 | \n35/50 | \n131K context (-6, confounded see §4.2) | \n
| V142 | \nApr 5 | \n30/50 | \nBayesian voting + VOI (-11) | \n
These comparisons are single-run snapshots drawn from production iteration over three months. Identical configurations exhibited ±3-4 point variance across re-runs (see §6), so deltas smaller than roughly 4 points should be treated as suggestive rather than significant. Larger deltas (8+) are confidently outside noise. We did not have time during the competition to run controlled multi-seed ablations; we acknowledge this as a limitation and have since added the per-attempt instrumentation that would enable it.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F6226045%2Fae4766a63f837ac1e27cad798828b85c%2F04_ablation_studies.png?generation=1779134579877758&alt=media\")
Figure 4: Four-panel ablation summary, with every data point labelled by the V-number that produced it. (a) Temperature: V41/V139 at 0.5 scored 34/50, V112 at 0.7 scored 38, V146/V147 at 0.8 scored 40, V125/V143 at 1.0 scored 41. Only temp=0.5 (-7) clears the variance band. (b) Attempt count: V40/V125/V143 at 8 attempts scored 41; V144 at 12 attempts regressed to 35. (c) Voting strategy: V125/V143 (5-component inverse entropy) at 41/50; V136 (simple-mean entropy) at 35; V100/V148 (GenSelect) at 40; V142 (Bayesian + VOI) at 30 the only delta unambiguously outside the variance band. (d) Context length: V125 at 65K scored 41, V135 at 131K scored 35, with the twin axis showing vLLM max concurrency falling from 7.11x to 3.35x at the same max_num_seqs. The 96K intermediate (white circle) was not tested; it is the natural follow-up to disentangle \"capacity hurt\" from \"throughput hurt\" in (d).
Temperature. Higher temperature produced better aggregate scores. temp=1.0 (V40, V125) gave 41–42; temp=0.5 (V41, V139) gave 34. An 8-point gap, comfortably outside the noise band. The mechanism is straightforward: voting benefits from diverse candidate answers, and low temperature causes the 8 attempts to converge on the same answer (sometimes wrong), leaving no correct alternative for the vote to find.
Context length confounded. Doubling the context window from 65K (V125) to 131K (V135) dropped the aggregate score from 41/50 to 35/50. Earlier versions of this writeup framed this as \"context hurts\"; that framing was overreach. The vLLM server logs show that at fixed max_num_seqs=256, max concurrency fell from 7.11x at 65K to 3.35x at 131K effectively halving the parallel attempt pool. For a pipeline whose accuracy comes out of an 8-way vote, halving the pool can on its own cost 4–6 points before any individual problem becomes harder or easier. We cannot from a single aggregate score per submission disentangle \"capacity loss\" from \"throughput loss.\" The honest conclusion: with batch-matched effective parallelism not held constant, we are not measuring context-window effects alone. A controlled comparison (65K @ batch=256 vs 131K @ batch=128, plus an intermediate 96K @ batch=192, with multiple seeds per cell) would resolve the question. We did not run this within the competition window.
Entropy weighting. Replacing the 5-component weighted entropy with simple mean entropy (V136) gave 35/50 against 41/50 baseline (V125), a 6-point regression. The position-weighted term carries most of this: mean entropy treats exploratory uncertainty and answer uncertainty symmetrically, diluting the signal that matters.
\nPrompt complexity. Verbose multi-paragraph prompts with explicit problem-solving frameworks (V127, 33/50) scored 9 points below the same configuration with a three-line system prompt (V40, 42/50). This is one of the larger isolated effects we measured. Distinguishing the three mechanisms hypothesised in §3.4 (override of learned strategies, token displacement, conflicting guidance) would require controlled prompt-length and prompt-content ablations we did not complete.
\nSeveral intuitively appealing approaches produced negative results.
\nThe single-aggregate-score nature of competition submissions made many ablations difficult to interpret. After the competition we added per-attempt CSV logging that captures, for every attempt of every problem: response token count, completion status, timeout flag, the five entropy components individually, Python tool calls and errors, elapsed wall time, and the final voting weight. The instrumented V143 run on the 10-problem reference set produced 80 attempt rows; the patterns visible in this data are:
\nConsensus regimes. Eight of ten reference problems reached complete consensus among the attempts that completed (all completed attempts returned the same answer; early-stop typically triggered after 4 of 8). Two problems (P3, P10) had one outlier answer among the completed attempts; the entropy-weighted majority still selected correctly. One problem 86e8e5, our hardest produced eight different answers from eight attempts, none agreeing. The 5-component entropy correctly identified the single attempt whose final-answer tokens were most confident; that attempt's answer (8687) was the reference value. The margin over the runner-up was 0.014 entropy units.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F6226045%2Fe4bf00065e84be7caacd4770a355085a%2F05_vote_distribution.png?generation=1779134617572453&alt=media\")
Figure 5: Vote-weight distribution per problem from the V143 reference run (10/10 correct, 31 minutes on an H100 80GB SXM). Each panel is one of the ten reference problems with the reference answer in the title; each bar is a distinct answer produced by some attempt, height is the summed inverse-entropy vote weight, count is the number of attempts that produced that answer. Green bars are the selected (correct) answers; purple bars are wrong answers. Eight problems (P1, P2, P4, P5, P7, P8, P9) show a single green bar and unanimous consensus among completed attempts; two problems (P3, P10) have a single purple outlier next to the dominant green; only P6 (86e8e5, highlighted in purple in its title) has eight distinct purple-and-one-green bars, meaning the winning margin came from confidence alone rather than agreement.
\nThroughput band. Figure 6 plots attempt elapsed time against tokens generated. The points cluster on a near-linear band of approximately 100 tokens per second, consistent with vLLM's throughput on a single H100 at this batch size and context length. Hard problems (86e8e5, 641659) generate 3–5× more tokens per attempt and consume the full deadline budget; easy problems finish well under a minute.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F6226045%2F9bdd1e7730e5ff00010a7c57016c50cd%2F06_time_vs_tokens.png?generation=1779134683831582&alt=media\")
Figure 6: V143 per-attempt throughput on the H100 80GB SXM every one of the 80 attempts from the reference reproduction plotted as elapsed wall time vs response tokens generated, coloured by problem. Circles are attempts that produced a \\boxed{} answer; X markers are attempts cancelled by early-stop or terminated at the deadline. The dashed gray line is the fitted throughput band at ~120 tokens/second, consistent with vLLM's reported max concurrency for our 65K-context / fp8_e4m3-KV-cache setup. P6 (brown, top right) is the clear outlier: 8 attempts in the 33-59K token range, all of which ran to the 900-second deadline the same problem that produced the 8 unique answers in Figures 2 and 5.
Limitations of this reproduction. It is a single run, so it does not give us a variance estimate on V143's reference-set score (though the unanimous-vote pattern on 8/10 problems suggests it would be stable). A controlled 65K-vs-131K comparison under matched effective parallelism which would directly answer the question from §4.2 remains future work. The reference set is 10 problems; the hidden test set is 50, and that score cannot be recomputed.
\nThree contributing factors are consistent with our data. First, GPT-OSS-120B was trained on extensive mathematical content and already possesses effective problem-solving strategies; elaborate prompts can override these. Second, verbose prompts displace solution tokens within a fixed context. Third, detailed instructions can introduce conflicting guidance that fights the model's existing dispositions.
\nThe temperature-1.0 win over 0.5 reflects a general principle for ensemble systems: diversity among candidates matters more than the accuracy of any individual candidate, up to a point. Low temperature causes premature convergence; the 8 attempts make the same mistake together. High temperature without a confidence filter would produce noise. The combination temperature 1.0 with min_p=0.02 and inverse-entropy voting gets us both diversity and selection.
\nThe 5-component score substantially outperforms simple averaging because mathematical reasoning has heterogeneous confidence patterns across a response. Early steps are exploratory and naturally uncertain; the final answer should be confident when correct. The position-weighted term captures this; simple averaging dilutes it.
\nThe winner (ippeiogawa) achieved 46/50, four points above us. A cluster of teams sat at 44–45. Public 44/50 notebooks suggest those top configurations use temperature 0.5 combined with simpler entropy calculations the opposite of what works for us. We can't resolve this fully without more controlled experiments, but the most likely explanation is that the optimal temperature depends on the other pipeline components, particularly the entropy weighting scheme. With a simpler aggregation, you may need lower temperature to keep the candidates close to correct, whereas a 5-component score can tolerate higher diversity.
\nA few practical guidelines from this work:
\nThe V143 configuration is the canonical baseline for everything reported here. To reproduce the post-competition 10/10 reference-set result on independent hardware:
\ncd runpod/.bash setup.sh (~35 min): installs vLLM 0.16.0, PyTorch 2.9.1+cu128, openai-harmony, jupyter_client, plus downloads openai/gpt-oss-120b with ignore_patterns=['metal/*','original/*'] to skip 135 GB of unneeded variants.bash start_server.sh in one terminal and wait for Application startup complete.bash run.sh in another terminal. Full 10-problem run completes in ~31 min on an H100 80GB; expect ~50 min on an A100 80GB.Outputs land in /workspace/results/:
submission.csv - id, answer per problemscored.csv - adds the reference answer and a correctness flagper_attempt_log.csv - 80 rows, 16 columns, schema defined in §4.4run_summary.txt - score, wall time, mean time per problemAll sampling is seeded (CFG seed = 42; per-attempt seeds are (42 + i)²). On the same vLLM version and hardware the results are byte-exact reproducible; we verified this by replaying the first five problems in a smoke run before the full reproduction (all five matched bit-for-bit). The V143 configuration as deployed on Kaggle scored 41/50 on the hidden 50-problem test set during the competition; that score cannot be recomputed externally.
Three months and 60+ versions produced one durable insight that I didn't expect going in: simple beats sophisticated, almost everywhere we measured. Three-line prompts beat verbose ones by 9 points. Inverse-entropy voting beat Bayesian aggregation by 11. Standard sampling beat low-temperature settings by 4–8. Adding verification, refinement, or longer context regressed the score. The model already knows how to do mathematics; our job was to sample effectively from it and trust the confidence signal in its own probabilities.
\nThe four-point gap to the top of the leaderboard remains open. Public top solutions appear to use a different temperature/voting pair from ours, suggesting that the optimum is path-dependent on the rest of the pipeline. Closing that gap would likely require either MCTS-style search over solution steps, problem-classification routing to different configurations, or an honest ensemble across multiple model architectures.
\nFor the broader community of practitioners working on inference-time scaling, the catalog of failures here is probably more valuable than the 42/50 headline. We've documented every clever idea we tried that didn't work, with deltas calibrated against the noise band, so the next team can skip what we already tested.
\nThanks to OpenAI and Daniel Hanchen for the GPT-OSS-120B model; to the AIMO Prize Foundation for organising the competition; to the Kaggle community particularly nihilisticneuralnet and huikang whose public notebooks shaped our search space; and to Sam Bealing for the careful review and the questions that pushed me to instrument what I should have instrumented months ago.
\nrunpod_results/results/per_attempt_log.csv in the linked repositoryrunpod/run_v143.pyUpdated post-rebuttal, May 2026.
", "comments": [ { "id": 3459448, "author_name": "Sam Bealing", "author_url": "", "post_date": "2026-05-17T20:29:08.877000", "content": "Apologies for the delay in posting these questions. We're reviewing whether there's a reason this submission was missed from our initial pass; in any case, I've spent the evening reading through your write-up and have the questions below.
\nGiven the late notice, we can give you until Monday, 18th May, 23:59 UTC to respond as a minimum (we'll update if we decide to extend that). Rest assured we'll take the time to give the submission the attention it deserves, and note that we're yet to make a final decision on the winners so you're very much still in consideration here.
\nFor wider context on the rebuttal phase, please see the updated discussion post.
\nLook forward to reading your responses on these.
", "votes": 0, "replies": [ { "id": 3460001, "author_name": "Pawan Rama Mali", "author_url": "", "post_date": "2026-05-18T14:24:56.147000", "content": "Hi Sam,
\nThanks for the careful read, and for the extra time really appreciated. Both questions made me realise I'd been too confident about something I couldn't actually prove from the data. Spent the last day rebuilding the pipeline outside Kaggle so I could re-run with proper instrumentation, and rewriting the parts you flagged.
\nYou're right to push on this. The V125 vs V135 comparison just can't tell us what I claimed it does. When I look at the two server logs side by side, the only thing that's solidly true is that the score dropped from 41 to 35 when I doubled the context. But the same logs show vLLM's max concurrency falling from 7.11× down to 3.35× at the same max_num_seqs. In a pipeline that decides the answer by an 8-way vote, halving the parallel pool could on its own cost four to six points, before any individual problem became newly solvable or newly broken by the longer window.
The three-way split you described of V135 problems that V125 got right, how many timed out, how many voted right with fewer attempts, and how many had at least one correct attempt but the vote signal was too weak I can't give you. We only retained final scores for those runs. So I had been treating the drop as \"context hurt\" when really I had no per-problem evidence to support either side, and your hypothesis (it's mostly throughput, with a balance to be struck) is just as consistent with the numbers.
\nWhat I do have now: I rebuilt the V143 pipeline outside Kaggle this week with per-attempt CSV logging response tokens, completion status, all five entropy components individually, timeout flag, Python tool calls and errors, elapsed wall time, vote weight one row per attempt. Ran it against the 10-problem public reference set on an H100 80GB I rented on RunPod. It scored 10/10 in 31 minutes.
\nIf you'd like, I can run the controlled comparison this enables V143 at 65K vs 96K vs 131K, batch-matched so the effective parallelism is held roughly constant across the cells, three repetitions per cell for variance. That's about ten hours of H100 compute, around $30. It would directly answer the three-way decomposition you sketched, and would also test whether the 96K midpoint dominates either endpoint, which is what your \"balance\" hypothesis implies. Happy to do it either way before the deadline if it helps let me know.
\nI've reflected the honest framing in §4.2 of the new writeup. It now flags the V125/V135 comparison as confounded, lists both mechanisms, and pulls the \"context hurt\" framing back. I also acknowledged that some problems do legitimately need more than 65K tokens, which lines up with what you said you've seen in your own runs.
\nFair point. The version history really was the heart of the writeup and it was just sitting there as a table. I will be updating the writeup with six new figures in the updated version:
\nThe one I think you'll find most useful is a scatter of all 60-odd submissions on a single chart with a ±3.5 variance band drawn around the V143 baseline. What becomes obvious when you plot it that way is how few of our \"improvements\" actually escape the noise band. Most are inside it. Only V142 (Bayesian aggregation, -11) and V127 (verbose prompts, -9) clearly fall below it, and nothing clears V40 upward. Rendering it that way changed how I read our own three months of work.
\nA four-panel ablation grid does the same thing for temperature, attempt count, voting strategy, and context length each panel has the variance band overlaid so the reader can see which deltas are signal and which aren't. The context panel has a twin axis showing vLLM max concurrency next to the score, which makes the confound we just discussed visible without me having to belabour it in text.
\nThe figure I'm happiest about is built from a real per-attempt log: problem 86e8e5 from the reference run. All 8 attempts produced 8 different answers no consensus at all and the 5-component entropy correctly picked the right one purely on confidence, with a margin of 0.014 entropy units over the runner-up. It's the cleanest demonstration I have of the voting actually doing something a simple mean entropy couldn't.
There's also a small-multiples chart with the vote distribution for all ten reference problems (eight unanimous, two with a single outlier, that one 8-way-ambiguous case), and a scatter of per-attempt elapsed time against tokens generated that makes the ~100 tok/s throughput band visible and shows 86e8e5 as a clear outlier with all attempts running to deadline. Plus an H100 memory pie for anyone trying to reproduce.
I haven't updated the writeup yet but I'll do that soon in some time.
\nThanks again for taking the time.
", "votes": 0, "replies": [ { "id": 3461525, "author_name": "Pawan Rama Mali", "author_url": "", "post_date": "2026-05-20T18:35:53.850000", "content": "Please note, the visualizations are also updated. Thanks.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3446694, "author_name": "Utkarsh Sharma", "author_url": "", "post_date": "2026-04-22T00:50:50.033000", "content": "Thank you for this - your notebook was the direct inspiration for our entropy-weighted voting approach, and we've updated our writeup's provenance section to reflect that more clearly.
\nYour finding that \"simple three-line prompts outperform elaborate structured instructions by 9 points\" matches our experience exactly - our multi-strategy prescriptive prompts (8 different technique-specific prompts) scored a mean of 36.6 across 5 submissions, while our one-sentence prompt with the 5-component entropy voting scored 40.78 mean across 9 runs.
\nOne question: you report \"five-component weighted entropy voting surpasses simple averaging by 6 points\" (V136: 35 vs V125: 41). Was that a single-run comparison or multi-run? At σ≈1.7, a 6-point gap from a single run has ~15% chance of being noise. Our own entropy-vs-baseline comparison is confounded because we changed the voting function AND stripped the scaffolding simultaneously - we can't isolate which change was load-bearing.
\nYour 60+ experiments over 2 months is impressive thoroughput for a solo competitor. The companion notebook has been very useful to the community (66 upvotes speaks for itself).
", "votes": 0, "replies": [ { "id": 3446800, "author_name": "Pawan Rama Mali", "author_url": "", "post_date": "2026-04-22T06:13:34.740000", "content": "Hi @utkarshsharma,\nThanks for the kind words, and glad the notebook was useful!\nThe V136 vs V125 comparison was single-run, so you're right that with σ≈1.7, a 6-point gap could partially be noise. However, I observed the same directional trend across multiple adjacent versions - mean entropy consistently underperformed position-weighted entropy by 3-6 points across ~8 different runs where that was the primary variable.\nThe more robust signal was the prompt complexity finding - that held across 15+ versions with very consistent results (verbose prompts never scored above 35, simple prompts rarely scored below 39).\nYour point about confounded changes is exactly why I started testing one variable at a time after V131 - combining 6 \"improvements\" simultaneously caused a 6-point regression, and I couldn't isolate which one was the culprit. Learned that the hard way!\nCongrats on your 40.78 mean - that's solid. Would be curious if you ever isolated the entropy vs scaffolding contribution.
", "votes": 0, "replies": [ { "id": 3448908, "author_name": "Utkarsh Sharma", "author_url": "", "post_date": "2026-04-27T02:05:28.247000", "content": "Really useful data point - \"mean entropy consistently underperformed position-weighted entropy by 3-6 points across ~8 different runs where that was the primary variable\" is the multi-run confirmation I was hoping existed. That's much stronger evidence than either of our single-run comparisons alone.
\nTo answer your question: no, we never isolated entropy vs scaffolding. Our converged submission simultaneously stripped the multi-strategy prompts AND replaced the voting function - we ran out of submission budget before we could test them independently. The honest answer is we don't know which change was load-bearing, and we say so explicitly in the writeup.
\nYour V131 experience (combining 6 \"improvements\" = 6-point regression) is almost identical to our adaptive router experiment - we combined a subject classifier, multi-stage solving, GenSelect, and adversarial verification into one notebook and it scored 28, a 12-point regression. Same lesson learned the same hard way.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3445891, "author_name": "Simon Frieder", "author_url": "", "post_date": "2026-04-20T16:28:51.620000", "content": "Please note that if you want to submit the write-up for the Write-Up prize you need to make a comment under the thread linke below, and follow those instructions:
\nhttps://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/689703
", "votes": 0, "replies": [] }, { "id": 3443552, "author_name": "Spundun Bhatt", "author_url": "", "post_date": "2026-04-16T17:38:40.447000", "content": "Did I miss it or you don't describe position-weighting? Can you describe/link-to position-weighted methods?
", "votes": 0, "replies": [ { "id": 3443573, "author_name": "Pawan Rama Mali", "author_url": "", "post_date": "2026-04-16T18:00:42.463000", "content": "Great question! The position-weighted entropy method works as follows:\n The key insight is that mathematical reasoning shows different confidence patterns at different stages - early tokens often involve uncertain exploration, while final answer tokens should show high confidence if the solution is correct.\n Our 5-component weighted entropy calculation:
\nPawan Mali
\nAIMO Progress Prize 3 Competition Writeup - Updated May 2026
\nThis writeup documents three months of inference-time scaling work on the AIMO Progress Prize 3 competition, where the task is to solve 50 hidden IMO-level mathematical problems on a single NVIDIA H100 within a 4.8-hour budget. Across 60+ submitted versions we converged on a configuration we call V143: GPT-OSS-120B with temp=1.0, an 8-attempt parallel-sampling pipeline, early-stopping on 4-vote consensus, a 5-component weighted entropy score for confidence ranking, and a 65K context window with FP8 KV cache. V40 (an earlier iteration of the same config) scored 42/50, and V143 reproduced 41/50 on the hidden test set. After the competition closed we rebuilt the pipeline outside Kaggle with per-attempt instrumentation and rented an H100 80GB to validate against the public 10-problem reference set; that run scored 10/10 in 31 minutes, total cost $5.50, with a full per-attempt CSV preserved. The most surprising finding from the instrumented run: on one reference problem all 8 parallel attempts produced 8 different answers, and the entropy ranking still picked the correct one based on confidence alone. The rest of this writeup catalogues both what worked and the considerably longer list of things that didn't verbose prompts (-9), Bayesian voting (-11), self-refinement (-4), fine-tuning without matching the inference pipeline (-6), bundled \"improvements\" (-6), and a context-length finding that turned out to be confounded with throughput.
AIMO Progress Prize 3 sits at the intersection of two hard problems: mathematical reasoning at IMO level, and inference-time scaling under realistic compute constraints. Fifty hidden problems, answers must be non-negative integers between 0 and 99,999, evaluated on a single H100 with roughly 4.8 hours of wall-clock budget. A $2.2M prize was on offer for 47/50; no team reached it. The winner (ippeiogawa) achieved 46/50; a cluster of teams sat at 44–45.
\nThis writeup documents 60+ submitted versions between January and April 2026. The peak was 42/50 (V40), and we settled on a stable 41/50 baseline (V125, V132, V133, V143). The numerical headline is not the most useful thing here it's that across the same number of versions trying to improve on that baseline, roughly the same number of changes regressed it. Most regressions were not subtle: -6 to -11 points on a single configuration change.
\nThe contributions of this work are three:
\n
Figure 1: Our submissions in the context of the final AIMO3 leaderboard. The top of the leaderboard (gray) clusters between 44 and 46 with the winner ippeiogawa at 46/50. Our peak V40 (42/50, dark green) and stable V143 baseline (41/50, mid green) sit just below the leader cluster, with V148 (40/50) slightly behind. Our two largest regressions V135 (131K context, 35/50) and V142 (Bayesian + VOI voting, 30/50) are highlighted in purple. The shaded band shows the ±3.5-point variance observed across re-runs of V143; the red dashed line marks the 47/50 prize threshold that no team reached.
\nSeveral lines of work inform the design used here. Chain-of-thought prompting (Wei et al., 2022) demonstrated that step-by-step reasoning markedly improves mathematical performance for large models. Self-consistency (Wang et al., 2022) extended that by sampling multiple chains and selecting the majority answer, which is the conceptual foundation of our 8-attempt voting pipeline. More recent test-time-compute work (Snell et al., 2024; Brown et al., 2024) systematically explores how inference budget translates into accuracy gains, and finds that sample-and-vote scales favourably until a diminishing-returns regime. Our experiments are consistent with that, but with the additional finding that the quality of the aggregation function (here, 5-component weighted entropy rather than simple majority) matters at this size.
\nTool-integrated reasoning letting the model emit Python that gets executed in a sandbox has proven particularly effective on mathematical tasks. Our pipeline runs each attempt with access to an isolated Jupyter kernel; the model generates code, the kernel returns results, the model continues. This combines pattern recognition with arithmetic precision.
\nSelf-refinement approaches have shown mixed results. Huang et al. (2023) found that LLMs cannot reliably self-correct reasoning errors without external feedback, and that self-refinement often introduces errors into previously-correct answers. Our V134 experiment is consistent with this: adding a self-refinement step regressed our score by 4 points (within the variance band, but reliably negative across multiple submissions).
\nIn the AIMO competition series specifically, the first progress prize topped out around 29/50, the second edition at roughly 35/50 (with the winning solution introducing a GenSelect-style LLM-as-judge over candidates), and AIMO-3 leaderboard winners reached 46/50. Public AIMO-3 notebooks particularly nihilisticneuralnet (44/50) and huikang established baselines that influenced our search space.
\nWe tested several base models before committing to GPT-OSS-120B. DeepSeek-R1-32B reasoned well but its capacity capped scores around 28–32. Qwen2.5-Math-72B was strong on math but its 32K context limited the length of tool-using reasoning chains, putting it in the 30–35 range. GPT-OSS-120B at 120B parameters and 131K native context consistently scored 38–42 across our pipeline iterations, and was our choice for everything subsequent.
\nThe model is MXFP4-quantized in its public release (openai/gpt-oss-120b), which requires a GPU with compute capability ≥ 8.0 (Ampere or newer). On the H100 the entire 60 GB of weights fits comfortably with a 6 GB FP8 KV cache and an 8 GB safety buffer in 80 GB.
The pipeline has three stages per problem:
\n\\boxed{} answer is produced or the deadline is hit.k=5). At the end of each attempt we summarise the entropy sequence with a 5-component score (§3.3) and store it.1 / max(E, ε) to its extracted answer; the answer with the highest summed weight is selected. Early-stopping triggers when any answer reaches 4 votes, cancelling remaining attempts and freeing the budget for the next problem.For each token t with top-k logprobs we compute Shannon entropy H_t = −Σᵢ pᵢ · log₂(pᵢ). The per-attempt entropy sequence {H₁ … Hₙ} is summarised by a five-component score:
E = 0.3 · mean(H)\n + 0.4 · position_weighted_mean(H, decay=0.995)\n + 0.2 · std(H)\n + 0.9 · |{t : H_t > 2.0}| / n\n − 0.1 · longest_run(H_t < 0.5) / n\nThe components emphasise different aspects of the confidence signal. Position weighting (40% of the score, exponential decay 0.995 favouring recent tokens) reflects an empirical observation: mathematical solutions exhibit exploratory uncertainty early but should converge to confident conclusions, and the final-answer tokens carry most of the signal. The high-entropy-fraction term (r_high, threshold 2.0 bits) penalises attempts with extended uncertain stretches even when the overall mean is moderate. The low-entropy-streak bonus (s_low, threshold 0.5 bits) is a small reward for sustained-confidence regions.
The vote weight for each attempt is 1 / max(E, ε). Lower E means more confident, contributes more weight.
Ablation against a simple-mean-entropy baseline (V136 vs V125) showed a 6-point regression outside the noise band, the largest defensible single-axis result we have.
\n
Figure 2: The 5-component entropy mechanism in action on real V143 attempt logs. Left panel: all 8 V143 attempts on problem 86e8e5 from the reference reproduction, showing the per-attempt extracted answer and the 5-component entropy score E. Every attempt produced a different answer (50505, 54680, 74763, 8687, 51379, 41754, 93764, 13657), so there was no consensus to vote on; attempt 3 (green) was selected because its E of 0.500 was the lowest, and its answer 8687 was the reference value. Right panel: the per-component decomposition of the winning attempt versus the runner-up (attempt 5, ans=41754, E=0.514). The winning margin of 0.014 is driven by the position-weighted term (40% of the score, favouring confident final-answer tokens) and the standard-deviation term (20%, penalising inconsistent confidence).
\nOur best-performing configuration (V40, 42/50) uses a three-sentence system prompt: identify the model as an IMO competitor, specify the answer range, require the boxed format. Adding structured problem-solving frameworks, verification checklists, or detailed mathematical guidance consistently regressed the score by 4–9 points. We hypothesise three contributing factors: (a) GPT-OSS-120B was trained on extensive mathematical content and already possesses effective problem-solving strategies that elaborate prompts may override; (b) prompt tokens displace solution tokens within a fixed context window; (c) detailed instructions can introduce conflicting guidance.
\nTable 1 summarises our key submissions on the hidden 50-problem competition test set. V40 was our best at 42/50; V125, V132, V133, V143 established a reproducible 41/50 baseline.
\nPost-competition reference reproduction. To address the reproducibility question and to provide per-problem evidence for the analyses in §4.2 and §4.4, we re-ran the V143 configuration on the public 10-problem AIMO3 reference set after the competition closed. The run used a single NVIDIA H100 80GB SXM rented from RunPod (AP-IN-1 data centre, $2.99/hr) with the standalone runner published in our runpod/ directory. Result: 10/10 correct in 1869 seconds (31 minutes), mean wall time 187 s per problem, total spend $5.50. The pipeline generated 1.2 million response tokens, made 1056 Python tool invocations, recorded 202 Python errors (which the voting absorbed without issue), and stored the five entropy components, completion status, and timing for each of 80 attempts.
This is the first end-to-end reproducible evidence we can publish; the hidden 50-problem test set is closed post-competition and cannot be re-scored externally.
\n
Figure 3: Score on the hidden 50-problem test set for every submission from V8 to V158 (January to April 2026). The shaded band is the ±3.5-point variance observed across re-runs of the V143 configuration; the dashed line marks the V143 baseline (41/50) and the dotted line marks our V40 ceiling (42/50). Points are coloured by outcome class: V40 (dark green) is our only run above the baseline band; V125 / V132 / V133 / V143 (mid green) sit on the baseline; light green points are above the baseline but inside the band; gray points are within the noise band; light purple are mild regressions (V126 / V127 / V128 / V131 / V134 / V135 / V139 / V141 / V144); dark purple are severe regressions (V124, V142). The orange X markers along the bottom are OOM or crashed runs (14 of them). Only V142 (-11) and V127 (-9) clearly escape the band downward, and no run cleanly clears V40 upward.
\nTable 1: Key Versions on the Hidden Test Set
\n| Version | \nDate | \nScore | \nConfiguration | \n
|---|---|---|---|
| V40 | \nFeb 6 | \n42/50 | \nSimple prompts, temp=1.0, 5-comp entropy (best ever) | \n
| V111 | \nMar 2 | \n40/50 | \n5-comp entropy isolated | \n
| V125 | \nMar 19 | \n41/50 | \ntemp=1.0 + top_p=0.8 | \n
| V132 | \nMar 26 | \n41/50 | \nNo top_p, 5-comp entropy | \n
| V143 | \nApr 6 | \n41/50 | \nStable baseline restored | \n
| V127 | \nMar 22 | \n33/50 | \nVerbose prompts (-9) | \n
| V128 | \nMar 24 | \n35/50 | \nFine-tuned model (-6) | \n
| V131 | \nMar 25 | \n35/50 | \n6 techniques bundled (-6) | \n
| V135 | \nMar 29 | \n35/50 | \n131K context (-6, confounded see §4.2) | \n
| V142 | \nApr 5 | \n30/50 | \nBayesian voting + VOI (-11) | \n
These comparisons are single-run snapshots drawn from production iteration over three months. Identical configurations exhibited ±3-4 point variance across re-runs (see §6), so deltas smaller than roughly 4 points should be treated as suggestive rather than significant. Larger deltas (8+) are confidently outside noise. We did not have time during the competition to run controlled multi-seed ablations; we acknowledge this as a limitation and have since added the per-attempt instrumentation that would enable it.
\n
Figure 4: Four-panel ablation summary, with every data point labelled by the V-number that produced it. (a) Temperature: V41/V139 at 0.5 scored 34/50, V112 at 0.7 scored 38, V146/V147 at 0.8 scored 40, V125/V143 at 1.0 scored 41. Only temp=0.5 (-7) clears the variance band. (b) Attempt count: V40/V125/V143 at 8 attempts scored 41; V144 at 12 attempts regressed to 35. (c) Voting strategy: V125/V143 (5-component inverse entropy) at 41/50; V136 (simple-mean entropy) at 35; V100/V148 (GenSelect) at 40; V142 (Bayesian + VOI) at 30 the only delta unambiguously outside the variance band. (d) Context length: V125 at 65K scored 41, V135 at 131K scored 35, with the twin axis showing vLLM max concurrency falling from 7.11x to 3.35x at the same max_num_seqs. The 96K intermediate (white circle) was not tested; it is the natural follow-up to disentangle \"capacity hurt\" from \"throughput hurt\" in (d).
Temperature. Higher temperature produced better aggregate scores. temp=1.0 (V40, V125) gave 41–42; temp=0.5 (V41, V139) gave 34. An 8-point gap, comfortably outside the noise band. The mechanism is straightforward: voting benefits from diverse candidate answers, and low temperature causes the 8 attempts to converge on the same answer (sometimes wrong), leaving no correct alternative for the vote to find.
Context length confounded. Doubling the context window from 65K (V125) to 131K (V135) dropped the aggregate score from 41/50 to 35/50. Earlier versions of this writeup framed this as \"context hurts\"; that framing was overreach. The vLLM server logs show that at fixed max_num_seqs=256, max concurrency fell from 7.11x at 65K to 3.35x at 131K effectively halving the parallel attempt pool. For a pipeline whose accuracy comes out of an 8-way vote, halving the pool can on its own cost 4–6 points before any individual problem becomes harder or easier. We cannot from a single aggregate score per submission disentangle \"capacity loss\" from \"throughput loss.\" The honest conclusion: with batch-matched effective parallelism not held constant, we are not measuring context-window effects alone. A controlled comparison (65K @ batch=256 vs 131K @ batch=128, plus an intermediate 96K @ batch=192, with multiple seeds per cell) would resolve the question. We did not run this within the competition window.
Entropy weighting. Replacing the 5-component weighted entropy with simple mean entropy (V136) gave 35/50 against 41/50 baseline (V125), a 6-point regression. The position-weighted term carries most of this: mean entropy treats exploratory uncertainty and answer uncertainty symmetrically, diluting the signal that matters.
\nPrompt complexity. Verbose multi-paragraph prompts with explicit problem-solving frameworks (V127, 33/50) scored 9 points below the same configuration with a three-line system prompt (V40, 42/50). This is one of the larger isolated effects we measured. Distinguishing the three mechanisms hypothesised in §3.4 (override of learned strategies, token displacement, conflicting guidance) would require controlled prompt-length and prompt-content ablations we did not complete.
\nSeveral intuitively appealing approaches produced negative results.
\nThe single-aggregate-score nature of competition submissions made many ablations difficult to interpret. After the competition we added per-attempt CSV logging that captures, for every attempt of every problem: response token count, completion status, timeout flag, the five entropy components individually, Python tool calls and errors, elapsed wall time, and the final voting weight. The instrumented V143 run on the 10-problem reference set produced 80 attempt rows; the patterns visible in this data are:
\nConsensus regimes. Eight of ten reference problems reached complete consensus among the attempts that completed (all completed attempts returned the same answer; early-stop typically triggered after 4 of 8). Two problems (P3, P10) had one outlier answer among the completed attempts; the entropy-weighted majority still selected correctly. One problem 86e8e5, our hardest produced eight different answers from eight attempts, none agreeing. The 5-component entropy correctly identified the single attempt whose final-answer tokens were most confident; that attempt's answer (8687) was the reference value. The margin over the runner-up was 0.014 entropy units.
Figure 5: Vote-weight distribution per problem from the V143 reference run (10/10 correct, 31 minutes on an H100 80GB SXM). Each panel is one of the ten reference problems with the reference answer in the title; each bar is a distinct answer produced by some attempt, height is the summed inverse-entropy vote weight, count is the number of attempts that produced that answer. Green bars are the selected (correct) answers; purple bars are wrong answers. Eight problems (P1, P2, P4, P5, P7, P8, P9) show a single green bar and unanimous consensus among completed attempts; two problems (P3, P10) have a single purple outlier next to the dominant green; only P6 (86e8e5, highlighted in purple in its title) has eight distinct purple-and-one-green bars, meaning the winning margin came from confidence alone rather than agreement.
\nThroughput band. Figure 6 plots attempt elapsed time against tokens generated. The points cluster on a near-linear band of approximately 100 tokens per second, consistent with vLLM's throughput on a single H100 at this batch size and context length. Hard problems (86e8e5, 641659) generate 3–5× more tokens per attempt and consume the full deadline budget; easy problems finish well under a minute.
Figure 6: V143 per-attempt throughput on the H100 80GB SXM every one of the 80 attempts from the reference reproduction plotted as elapsed wall time vs response tokens generated, coloured by problem. Circles are attempts that produced a \\boxed{} answer; X markers are attempts cancelled by early-stop or terminated at the deadline. The dashed gray line is the fitted throughput band at ~120 tokens/second, consistent with vLLM's reported max concurrency for our 65K-context / fp8_e4m3-KV-cache setup. P6 (brown, top right) is the clear outlier: 8 attempts in the 33-59K token range, all of which ran to the 900-second deadline the same problem that produced the 8 unique answers in Figures 2 and 5.
Limitations of this reproduction. It is a single run, so it does not give us a variance estimate on V143's reference-set score (though the unanimous-vote pattern on 8/10 problems suggests it would be stable). A controlled 65K-vs-131K comparison under matched effective parallelism which would directly answer the question from §4.2 remains future work. The reference set is 10 problems; the hidden test set is 50, and that score cannot be recomputed.
\nThree contributing factors are consistent with our data. First, GPT-OSS-120B was trained on extensive mathematical content and already possesses effective problem-solving strategies; elaborate prompts can override these. Second, verbose prompts displace solution tokens within a fixed context. Third, detailed instructions can introduce conflicting guidance that fights the model's existing dispositions.
\nThe temperature-1.0 win over 0.5 reflects a general principle for ensemble systems: diversity among candidates matters more than the accuracy of any individual candidate, up to a point. Low temperature causes premature convergence; the 8 attempts make the same mistake together. High temperature without a confidence filter would produce noise. The combination temperature 1.0 with min_p=0.02 and inverse-entropy voting gets us both diversity and selection.
\nThe 5-component score substantially outperforms simple averaging because mathematical reasoning has heterogeneous confidence patterns across a response. Early steps are exploratory and naturally uncertain; the final answer should be confident when correct. The position-weighted term captures this; simple averaging dilutes it.
\nThe winner (ippeiogawa) achieved 46/50, four points above us. A cluster of teams sat at 44–45. Public 44/50 notebooks suggest those top configurations use temperature 0.5 combined with simpler entropy calculations the opposite of what works for us. We can't resolve this fully without more controlled experiments, but the most likely explanation is that the optimal temperature depends on the other pipeline components, particularly the entropy weighting scheme. With a simpler aggregation, you may need lower temperature to keep the candidates close to correct, whereas a 5-component score can tolerate higher diversity.
\nA few practical guidelines from this work:
\nThe V143 configuration is the canonical baseline for everything reported here. To reproduce the post-competition 10/10 reference-set result on independent hardware:
\ncd runpod/.bash setup.sh (~35 min): installs vLLM 0.16.0, PyTorch 2.9.1+cu128, openai-harmony, jupyter_client, plus downloads openai/gpt-oss-120b with ignore_patterns=['metal/*','original/*'] to skip 135 GB of unneeded variants.bash start_server.sh in one terminal and wait for Application startup complete.bash run.sh in another terminal. Full 10-problem run completes in ~31 min on an H100 80GB; expect ~50 min on an A100 80GB.Outputs land in /workspace/results/:
submission.csv - id, answer per problemscored.csv - adds the reference answer and a correctness flagper_attempt_log.csv - 80 rows, 16 columns, schema defined in §4.4run_summary.txt - score, wall time, mean time per problemAll sampling is seeded (CFG seed = 42; per-attempt seeds are (42 + i)²). On the same vLLM version and hardware the results are byte-exact reproducible; we verified this by replaying the first five problems in a smoke run before the full reproduction (all five matched bit-for-bit). The V143 configuration as deployed on Kaggle scored 41/50 on the hidden 50-problem test set during the competition; that score cannot be recomputed externally.
Three months and 60+ versions produced one durable insight that I didn't expect going in: simple beats sophisticated, almost everywhere we measured. Three-line prompts beat verbose ones by 9 points. Inverse-entropy voting beat Bayesian aggregation by 11. Standard sampling beat low-temperature settings by 4–8. Adding verification, refinement, or longer context regressed the score. The model already knows how to do mathematics; our job was to sample effectively from it and trust the confidence signal in its own probabilities.
\nThe four-point gap to the top of the leaderboard remains open. Public top solutions appear to use a different temperature/voting pair from ours, suggesting that the optimum is path-dependent on the rest of the pipeline. Closing that gap would likely require either MCTS-style search over solution steps, problem-classification routing to different configurations, or an honest ensemble across multiple model architectures.
\nFor the broader community of practitioners working on inference-time scaling, the catalog of failures here is probably more valuable than the 42/50 headline. We've documented every clever idea we tried that didn't work, with deltas calibrated against the noise band, so the next team can skip what we already tested.
\nThanks to OpenAI and Daniel Hanchen for the GPT-OSS-120B model; to the AIMO Prize Foundation for organising the competition; to the Kaggle community particularly nihilisticneuralnet and huikang whose public notebooks shaped our search space; and to Sam Bealing for the careful review and the questions that pushed me to instrument what I should have instrumented months ago.
\nrunpod_results/results/per_attempt_log.csv in the linked repositoryrunpod/run_v143.pyUpdated post-rebuttal, May 2026.
", "rawMarkdown": "# Scaling Inference-Time Compute for Mathematical Reasoning: A Systematic Study of 60+ Experiments on IMO-Level Problems\n\n**Pawan Mali**\n\n*AIMO Progress Prize 3 Competition Writeup - Updated May 2026*\n\n---\n\n## Abstract\n\nThis writeup documents three months of inference-time scaling work on the AIMO Progress Prize 3 competition, where the task is to solve 50 hidden IMO-level mathematical problems on a single NVIDIA H100 within a 4.8-hour budget. Across 60+ submitted versions we converged on a configuration we call V143: GPT-OSS-120B with `temp=1.0`, an 8-attempt parallel-sampling pipeline, early-stopping on 4-vote consensus, a 5-component weighted entropy score for confidence ranking, and a 65K context window with FP8 KV cache. V40 (an earlier iteration of the same config) scored 42/50, and V143 reproduced 41/50 on the hidden test set. After the competition closed we rebuilt the pipeline outside Kaggle with per-attempt instrumentation and rented an H100 80GB to validate against the public 10-problem reference set; that run scored **10/10 in 31 minutes**, total cost $5.50, with a full per-attempt CSV preserved. The most surprising finding from the instrumented run: on one reference problem all 8 parallel attempts produced 8 different answers, and the entropy ranking still picked the correct one based on confidence alone. The rest of this writeup catalogues both what worked and the considerably longer list of things that didn't verbose prompts (-9), Bayesian voting (-11), self-refinement (-4), fine-tuning without matching the inference pipeline (-6), bundled \"improvements\" (-6), and a context-length finding that turned out to be confounded with throughput.\n\n---\n\n## 1. Introduction\n\nAIMO Progress Prize 3 sits at the intersection of two hard problems: mathematical reasoning at IMO level, and inference-time scaling under realistic compute constraints. Fifty hidden problems, answers must be non-negative integers between 0 and 99,999, evaluated on a single H100 with roughly 4.8 hours of wall-clock budget. A $2.2M prize was on offer for 47/50; no team reached it. The winner (ippeiogawa) achieved 46/50; a cluster of teams sat at 44–45.\n\nThis writeup documents 60+ submitted versions between January and April 2026. The peak was 42/50 (V40), and we settled on a stable 41/50 baseline (V125, V132, V133, V143). The numerical headline is not the most useful thing here it's that across the same number of versions trying to improve on that baseline, roughly the same number of changes regressed it. Most regressions were not subtle: -6 to -11 points on a single configuration change.\n\nThe contributions of this work are three:\n\n1. A reproducible 10/10 reference-set result for the V143 configuration, with byte-exact seeding and a per-attempt CSV (16 columns) available alongside this writeup.\n2. A characterisation of what does and doesn't work for inference-time scaling on this kind of task, with deltas calibrated against an observed ±3.5-point single-run variance band so readers can see which findings are signal and which are noise.\n3. An empirical case study of a confidence-only voting decision: a problem where 8 attempts produced 8 different answers, and a 5-component entropy ranking nevertheless picked the correct one.\n\n\n\n*Figure 1: Our submissions in the context of the final AIMO3 leaderboard. The top of the leaderboard (gray) clusters between 44 and 46 with the winner ippeiogawa at 46/50. Our peak V40 (42/50, dark green) and stable V143 baseline (41/50, mid green) sit just below the leader cluster, with V148 (40/50) slightly behind. Our two largest regressions V135 (131K context, 35/50) and V142 (Bayesian + VOI voting, 30/50) are highlighted in purple. The shaded band shows the ±3.5-point variance observed across re-runs of V143; the red dashed line marks the 47/50 prize threshold that no team reached.*\n\n---\n\n## 2. Related Work\n\nSeveral lines of work inform the design used here. **Chain-of-thought prompting** (Wei et al., 2022) demonstrated that step-by-step reasoning markedly improves mathematical performance for large models. **Self-consistency** (Wang et al., 2022) extended that by sampling multiple chains and selecting the majority answer, which is the conceptual foundation of our 8-attempt voting pipeline. More recent test-time-compute work (Snell et al., 2024; Brown et al., 2024) systematically explores how inference budget translates into accuracy gains, and finds that sample-and-vote scales favourably until a diminishing-returns regime. Our experiments are consistent with that, but with the additional finding that the *quality* of the aggregation function (here, 5-component weighted entropy rather than simple majority) matters at this size.\n\n**Tool-integrated reasoning** letting the model emit Python that gets executed in a sandbox has proven particularly effective on mathematical tasks. Our pipeline runs each attempt with access to an isolated Jupyter kernel; the model generates code, the kernel returns results, the model continues. This combines pattern recognition with arithmetic precision.\n\n**Self-refinement** approaches have shown mixed results. Huang et al. (2023) found that LLMs cannot reliably self-correct reasoning errors without external feedback, and that self-refinement often introduces errors into previously-correct answers. Our V134 experiment is consistent with this: adding a self-refinement step regressed our score by 4 points (within the variance band, but reliably negative across multiple submissions).\n\nIn the AIMO competition series specifically, the first progress prize topped out around 29/50, the second edition at roughly 35/50 (with the winning solution introducing a GenSelect-style LLM-as-judge over candidates), and AIMO-3 leaderboard winners reached 46/50. Public AIMO-3 notebooks particularly nihilisticneuralnet (44/50) and huikang established baselines that influenced our search space.\n\n---\n\n## 3. Methodology\n\n### 3.1 Model Selection\n\nWe tested several base models before committing to GPT-OSS-120B. DeepSeek-R1-32B reasoned well but its capacity capped scores around 28–32. Qwen2.5-Math-72B was strong on math but its 32K context limited the length of tool-using reasoning chains, putting it in the 30–35 range. GPT-OSS-120B at 120B parameters and 131K native context consistently scored 38–42 across our pipeline iterations, and was our choice for everything subsequent.\n\nThe model is MXFP4-quantized in its public release (`openai/gpt-oss-120b`), which requires a GPU with compute capability ≥ 8.0 (Ampere or newer). On the H100 the entire 60 GB of weights fits comfortably with a 6 GB FP8 KV cache and an 8 GB safety buffer in 80 GB.\n\n### 3.2 System Architecture\n\nThe pipeline has three stages per problem:\n\n1. **Parallel generation.** Eight attempts are launched concurrently against a vLLM server hosting GPT-OSS-120B. Each attempt has its own Jupyter sandbox; the model is free to emit Python and consume the tool result before continuing. Generation proceeds until a `\\boxed{}` answer is produced or the deadline is hit.\n2. **Per-attempt scoring.** As tokens stream back we accumulate per-token entropy from the top-k logprobs (`k=5`). At the end of each attempt we summarise the entropy sequence with a 5-component score (§3.3) and store it.\n3. **Weighted voting.** Each attempt contributes `1 / max(E, ε)` to its extracted answer; the answer with the highest summed weight is selected. Early-stopping triggers when any answer reaches 4 votes, cancelling remaining attempts and freeing the budget for the next problem.\n\n### 3.3 Five-Component Entropy Voting\n\nFor each token `t` with top-k logprobs we compute Shannon entropy `H_t = −Σᵢ pᵢ · log₂(pᵢ)`. The per-attempt entropy sequence `{H₁ … Hₙ}` is summarised by a five-component score:\n\n```\nE = 0.3 · mean(H)\n + 0.4 · position_weighted_mean(H, decay=0.995)\n + 0.2 · std(H)\n + 0.9 · |{t : H_t > 2.0}| / n\n − 0.1 · longest_run(H_t < 0.5) / n\n```\n\nThe components emphasise different aspects of the confidence signal. Position weighting (40% of the score, exponential decay 0.995 favouring recent tokens) reflects an empirical observation: mathematical solutions exhibit exploratory uncertainty early but should converge to confident conclusions, and the final-answer tokens carry most of the signal. The high-entropy-fraction term (`r_high`, threshold 2.0 bits) penalises attempts with extended uncertain stretches even when the overall mean is moderate. The low-entropy-streak bonus (`s_low`, threshold 0.5 bits) is a small reward for sustained-confidence regions.\n\nThe vote weight for each attempt is `1 / max(E, ε)`. Lower E means more confident, contributes more weight.\n\nAblation against a simple-mean-entropy baseline (V136 vs V125) showed a 6-point regression outside the noise band, the largest defensible single-axis result we have.\n\n\n\n*Figure 2: The 5-component entropy mechanism in action on real V143 attempt logs. Left panel: all 8 V143 attempts on problem 86e8e5 from the reference reproduction, showing the per-attempt extracted answer and the 5-component entropy score E. Every attempt produced a different answer (50505, 54680, 74763, 8687, 51379, 41754, 93764, 13657), so there was no consensus to vote on; attempt 3 (green) was selected because its E of 0.500 was the lowest, and its answer 8687 was the reference value. Right panel: the per-component decomposition of the winning attempt versus the runner-up (attempt 5, ans=41754, E=0.514). The winning margin of 0.014 is driven by the position-weighted term (40% of the score, favouring confident final-answer tokens) and the standard-deviation term (20%, penalising inconsistent confidence).*\n\n### 3.4 Prompting Strategy\n\nOur best-performing configuration (V40, 42/50) uses a three-sentence system prompt: identify the model as an IMO competitor, specify the answer range, require the boxed format. Adding structured problem-solving frameworks, verification checklists, or detailed mathematical guidance consistently regressed the score by 4–9 points. We hypothesise three contributing factors: (a) GPT-OSS-120B was trained on extensive mathematical content and already possesses effective problem-solving strategies that elaborate prompts may override; (b) prompt tokens displace solution tokens within a fixed context window; (c) detailed instructions can introduce conflicting guidance.\n\n---\n\n## 4. Experimental Results\n\n### 4.1 Overall Performance\n\nTable 1 summarises our key submissions on the **hidden 50-problem competition test set**. V40 was our best at 42/50; V125, V132, V133, V143 established a reproducible 41/50 baseline.\n\n**Post-competition reference reproduction.** To address the reproducibility question and to provide per-problem evidence for the analyses in §4.2 and §4.4, we re-ran the V143 configuration on the public 10-problem AIMO3 reference set after the competition closed. The run used a single NVIDIA H100 80GB SXM rented from RunPod (AP-IN-1 data centre, $2.99/hr) with the standalone runner published in our `runpod/` directory. Result: **10/10 correct in 1869 seconds (31 minutes)**, mean wall time 187 s per problem, total spend $5.50. The pipeline generated 1.2 million response tokens, made 1056 Python tool invocations, recorded 202 Python errors (which the voting absorbed without issue), and stored the five entropy components, completion status, and timing for each of 80 attempts.\n\nThis is the first end-to-end reproducible evidence we can publish; the hidden 50-problem test set is closed post-competition and cannot be re-scored externally.\n\n\n\n*Figure 3: Score on the hidden 50-problem test set for every submission from V8 to V158 (January to April 2026). The shaded band is the ±3.5-point variance observed across re-runs of the V143 configuration; the dashed line marks the V143 baseline (41/50) and the dotted line marks our V40 ceiling (42/50). Points are coloured by outcome class: V40 (dark green) is our only run above the baseline band; V125 / V132 / V133 / V143 (mid green) sit on the baseline; light green points are above the baseline but inside the band; gray points are within the noise band; light purple are mild regressions (V126 / V127 / V128 / V131 / V134 / V135 / V139 / V141 / V144); dark purple are severe regressions (V124, V142). The orange X markers along the bottom are OOM or crashed runs (14 of them). Only V142 (-11) and V127 (-9) clearly escape the band downward, and no run cleanly clears V40 upward.*\n\n**Table 1: Key Versions on the Hidden Test Set**\n\n| Version | Date | Score | Configuration |\n|---------|---------|--------|---------------|\n| V40 | Feb 6 | 42/50 | Simple prompts, temp=1.0, 5-comp entropy (best ever) |\n| V111 | Mar 2 | 40/50 | 5-comp entropy isolated |\n| V125 | Mar 19 | 41/50 | temp=1.0 + top_p=0.8 |\n| V132 | Mar 26 | 41/50 | No top_p, 5-comp entropy |\n| V143 | Apr 6 | 41/50 | Stable baseline restored |\n| V127 | Mar 22 | 33/50 | Verbose prompts (-9) |\n| V128 | Mar 24 | 35/50 | Fine-tuned model (-6) |\n| V131 | Mar 25 | 35/50 | 6 techniques bundled (-6) |\n| V135 | Mar 29 | 35/50 | 131K context (-6, confounded see §4.2) |\n| V142 | Apr 5 | 30/50 | Bayesian voting + VOI (-11) |\n\n### 4.2 Ablation Studies (Single-Run Snapshots)\n\nThese comparisons are single-run snapshots drawn from production iteration over three months. Identical configurations exhibited ±3-4 point variance across re-runs (see §6), so deltas smaller than roughly 4 points should be treated as suggestive rather than significant. Larger deltas (8+) are confidently outside noise. We did not have time during the competition to run controlled multi-seed ablations; we acknowledge this as a limitation and have since added the per-attempt instrumentation that would enable it.\n\n\n\n*Figure 4: Four-panel ablation summary, with every data point labelled by the V-number that produced it. (a) Temperature: V41/V139 at 0.5 scored 34/50, V112 at 0.7 scored 38, V146/V147 at 0.8 scored 40, V125/V143 at 1.0 scored 41. Only temp=0.5 (-7) clears the variance band. (b) Attempt count: V40/V125/V143 at 8 attempts scored 41; V144 at 12 attempts regressed to 35. (c) Voting strategy: V125/V143 (5-component inverse entropy) at 41/50; V136 (simple-mean entropy) at 35; V100/V148 (GenSelect) at 40; V142 (Bayesian + VOI) at 30 the only delta unambiguously outside the variance band. (d) Context length: V125 at 65K scored 41, V135 at 131K scored 35, with the twin axis showing vLLM max concurrency falling from 7.11x to 3.35x at the same `max_num_seqs`. The 96K intermediate (white circle) was not tested; it is the natural follow-up to disentangle \"capacity hurt\" from \"throughput hurt\" in (d).*\n\n**Temperature.** Higher temperature produced better aggregate scores. `temp=1.0` (V40, V125) gave 41–42; `temp=0.5` (V41, V139) gave 34. An 8-point gap, comfortably outside the noise band. The mechanism is straightforward: voting benefits from diverse candidate answers, and low temperature causes the 8 attempts to converge on the same answer (sometimes wrong), leaving no correct alternative for the vote to find.\n\n**Context length confounded.** Doubling the context window from 65K (V125) to 131K (V135) dropped the aggregate score from 41/50 to 35/50. Earlier versions of this writeup framed this as \"context hurts\"; that framing was overreach. The vLLM server logs show that at fixed `max_num_seqs=256`, max concurrency fell from 7.11x at 65K to 3.35x at 131K effectively halving the parallel attempt pool. For a pipeline whose accuracy comes out of an 8-way vote, halving the pool can on its own cost 4–6 points before any individual problem becomes harder or easier. We cannot from a single aggregate score per submission disentangle \"capacity loss\" from \"throughput loss.\" The honest conclusion: with batch-matched effective parallelism not held constant, we are not measuring context-window effects alone. A controlled comparison (65K @ batch=256 vs 131K @ batch=128, plus an intermediate 96K @ batch=192, with multiple seeds per cell) would resolve the question. We did not run this within the competition window.\n\n**Entropy weighting.** Replacing the 5-component weighted entropy with simple mean entropy (V136) gave 35/50 against 41/50 baseline (V125), a 6-point regression. The position-weighted term carries most of this: mean entropy treats exploratory uncertainty and answer uncertainty symmetrically, diluting the signal that matters.\n\n**Prompt complexity.** Verbose multi-paragraph prompts with explicit problem-solving frameworks (V127, 33/50) scored 9 points below the same configuration with a three-line system prompt (V40, 42/50). This is one of the larger isolated effects we measured. Distinguishing the three mechanisms hypothesised in §3.4 (override of learned strategies, token displacement, conflicting guidance) would require controlled prompt-length and prompt-content ablations we did not complete.\n\n### 4.3 Failed Experiments\n\nSeveral intuitively appealing approaches produced negative results.\n\n- **Self-refinement.** V134 added a \"review your answer\" step after each attempt. -4 points. The model corrected correct answers into wrong ones with alarming frequency, consistent with Huang et al. 2023.\n- **Answer verification.** V133 asked the model to verify each candidate answer with a CORRECT/WRONG decision. No improvement (matched V132 at 41/50). The model was equally willing to verify wrong answers as correct ones.\n- **Bayesian voting + VOI.** V142 replaced the inverse-entropy aggregation with a Bayesian posterior plus Value-of-Information early-stopping. -11 points, the worst regression we observed. The complex aggregation over-penalised uncertainty in ways uncorrelated with actual correctness.\n- **Fine-tuned model swap.** V128 used huikang/gpt-oss-120b-aimo3 without its matching inference pipeline. -6 points. Fine-tuned weights are optimised for a specific deployment context; swapping in the weights without the rest of the pipeline breaks the optimisation.\n- **Bundled changes.** V131 combined six promising techniques simultaneously. -6 points. We learned to test one variable at a time after this.\n\n### 4.4 Per-Attempt Instrumentation: Real Voting Behaviour\n\nThe single-aggregate-score nature of competition submissions made many ablations difficult to interpret. After the competition we added per-attempt CSV logging that captures, for every attempt of every problem: response token count, completion status, timeout flag, the five entropy components individually, Python tool calls and errors, elapsed wall time, and the final voting weight. The instrumented V143 run on the 10-problem reference set produced 80 attempt rows; the patterns visible in this data are:\n\n**Consensus regimes.** Eight of ten reference problems reached complete consensus among the attempts that completed (all completed attempts returned the same answer; early-stop typically triggered after 4 of 8). Two problems (P3, P10) had one outlier answer among the completed attempts; the entropy-weighted majority still selected correctly. One problem `86e8e5`, our hardest produced eight different answers from eight attempts, none agreeing. The 5-component entropy correctly identified the single attempt whose final-answer tokens were most confident; that attempt's answer (8687) was the reference value. The margin over the runner-up was 0.014 entropy units.\n\n\n\n*Figure 5: Vote-weight distribution per problem from the V143 reference run (10/10 correct, 31 minutes on an H100 80GB SXM). Each panel is one of the ten reference problems with the reference answer in the title; each bar is a distinct answer produced by some attempt, height is the summed inverse-entropy vote weight, count is the number of attempts that produced that answer. Green bars are the selected (correct) answers; purple bars are wrong answers. Eight problems (P1, P2, P4, P5, P7, P8, P9) show a single green bar and unanimous consensus among completed attempts; two problems (P3, P10) have a single purple outlier next to the dominant green; only P6 (86e8e5, highlighted in purple in its title) has eight distinct purple-and-one-green bars, meaning the winning margin came from confidence alone rather than agreement.*\n\n**Throughput band.** Figure 6 plots attempt elapsed time against tokens generated. The points cluster on a near-linear band of approximately 100 tokens per second, consistent with vLLM's throughput on a single H100 at this batch size and context length. Hard problems (`86e8e5`, `641659`) generate 3–5× more tokens per attempt and consume the full deadline budget; easy problems finish well under a minute.\n\n\n\n*Figure 6: V143 per-attempt throughput on the H100 80GB SXM every one of the 80 attempts from the reference reproduction plotted as elapsed wall time vs response tokens generated, coloured by problem. Circles are attempts that produced a `\\boxed{}` answer; X markers are attempts cancelled by early-stop or terminated at the deadline. The dashed gray line is the fitted throughput band at ~120 tokens/second, consistent with vLLM's reported max concurrency for our 65K-context / fp8_e4m3-KV-cache setup. P6 (brown, top right) is the clear outlier: 8 attempts in the 33-59K token range, all of which ran to the 900-second deadline the same problem that produced the 8 unique answers in Figures 2 and 5.*\n\n**Limitations of this reproduction.** It is a single run, so it does not give us a variance estimate on V143's reference-set score (though the unanimous-vote pattern on 8/10 problems suggests it would be stable). A controlled 65K-vs-131K comparison under matched effective parallelism which would directly answer the question from §4.2 remains future work. The reference set is 10 problems; the hidden test set is 50, and that score cannot be recomputed.\n\n---\n\n## 5. Analysis and Discussion\n\n### 5.1 Why Simple Prompts Win\n\nThree contributing factors are consistent with our data. First, GPT-OSS-120B was trained on extensive mathematical content and already possesses effective problem-solving strategies; elaborate prompts can override these. Second, verbose prompts displace solution tokens within a fixed context. Third, detailed instructions can introduce conflicting guidance that fights the model's existing dispositions.\n\n### 5.2 Diversity Drives Voting\n\nThe temperature-1.0 win over 0.5 reflects a general principle for ensemble systems: diversity among candidates matters more than the accuracy of any individual candidate, up to a point. Low temperature causes premature convergence; the 8 attempts make the same mistake together. High temperature without a confidence filter would produce noise. The combination temperature 1.0 with min_p=0.02 and inverse-entropy voting gets us both diversity and selection.\n\n### 5.3 Position-Weighted Confidence\n\nThe 5-component score substantially outperforms simple averaging because mathematical reasoning has heterogeneous confidence patterns across a response. Early steps are exploratory and naturally uncertain; the final answer should be confident when correct. The position-weighted term captures this; simple averaging dilutes it.\n\n### 5.4 The Gap to the Top of the Leaderboard\n\nThe winner (ippeiogawa) achieved 46/50, four points above us. A cluster of teams sat at 44–45. Public 44/50 notebooks suggest those top configurations use temperature 0.5 combined with simpler entropy calculations the opposite of what works for us. We can't resolve this fully without more controlled experiments, but the most likely explanation is that the optimal temperature depends on the other pipeline components, particularly the entropy weighting scheme. With a simpler aggregation, you may need lower temperature to keep the candidates close to correct, whereas a 5-component score can tolerate higher diversity.\n\n---\n\n## 6. Lessons Learned\n\nA few practical guidelines from this work:\n\n- **Test one variable at a time.** V131 bundled six \"improvements\" and regressed -6. The interactions between changes are unpredictable enough that any compound delta is unreadable.\n- **Respect the variance band.** The same configuration produced scores varying by 3–4 points across re-runs on the hidden test set. Reading any single submission as evidence in either direction requires that the delta exceed this band. Most published \"improvements\" in our space don't.\n- **Capable models don't need verbose prompts.** Our biggest measurable wins were from removing things, not adding them.\n- **Match the inference pipeline to the model.** Fine-tuned weights from someone else's pipeline broke our score. The same surely applies in reverse.\n- **Be skeptical of clever ideas.** Bayesian voting, GenSelect, CISC, VOI, self-refinement, answer verification every one of these regressed or didn't help. Simple inverse-entropy weighted voting beat them all.\n- **Instrument early.** The instrumentation we added after the competition would have answered most of the questions our ablation tables couldn't.\n\n---\n\n## 7. Reproducibility\n\nThe V143 configuration is the canonical baseline for everything reported here. To reproduce the post-competition 10/10 reference-set result on independent hardware:\n\n1. Clone the repository and `cd runpod/`.\n2. Provision an NVIDIA H100 80GB or A100 80GB instance with 100 GB of attached storage.\n3. Run `bash setup.sh` (~35 min): installs vLLM 0.16.0, PyTorch 2.9.1+cu128, openai-harmony, jupyter_client, plus downloads `openai/gpt-oss-120b` with `ignore_patterns=['metal/*','original/*']` to skip 135 GB of unneeded variants.\n4. Run `bash start_server.sh` in one terminal and wait for `Application startup complete`.\n5. Run `bash run.sh` in another terminal. Full 10-problem run completes in ~31 min on an H100 80GB; expect ~50 min on an A100 80GB.\n\nOutputs land in `/workspace/results/`:\n\n- `submission.csv` - id, answer per problem\n- `scored.csv` - adds the reference answer and a correctness flag\n- `per_attempt_log.csv` - 80 rows, 16 columns, schema defined in §4.4\n- `run_summary.txt` - score, wall time, mean time per problem\n\nAll sampling is seeded (CFG seed = 42; per-attempt seeds are `(42 + i)²`). On the same vLLM version and hardware the results are byte-exact reproducible; we verified this by replaying the first five problems in a smoke run before the full reproduction (all five matched bit-for-bit). The V143 configuration as deployed on Kaggle scored 41/50 on the hidden 50-problem test set during the competition; that score cannot be recomputed externally.\n\n---\n\n## 8. Conclusion\n\nThree months and 60+ versions produced one durable insight that I didn't expect going in: simple beats sophisticated, almost everywhere we measured. Three-line prompts beat verbose ones by 9 points. Inverse-entropy voting beat Bayesian aggregation by 11. Standard sampling beat low-temperature settings by 4–8. Adding verification, refinement, or longer context regressed the score. The model already knows how to do mathematics; our job was to sample effectively from it and trust the confidence signal in its own probabilities.\n\nThe four-point gap to the top of the leaderboard remains open. Public top solutions appear to use a different temperature/voting pair from ours, suggesting that the optimum is path-dependent on the rest of the pipeline. Closing that gap would likely require either MCTS-style search over solution steps, problem-classification routing to different configurations, or an honest ensemble across multiple model architectures.\n\nFor the broader community of practitioners working on inference-time scaling, the catalog of failures here is probably more valuable than the 42/50 headline. We've documented every clever idea we tried that didn't work, with deltas calibrated against the noise band, so the next team can skip what we already tested.\n\n---\n\n## Acknowledgments\n\nThanks to OpenAI and Daniel Hanchen for the GPT-OSS-120B model; to the AIMO Prize Foundation for organising the competition; to the Kaggle community particularly nihilisticneuralnet and huikang whose public notebooks shaped our search space; and to Sam Bealing for the careful review and the questions that pushed me to instrument what I should have instrumented months ago.\n\n---\n\n## References\n\n1. AIMO Prize Foundation. AI Mathematical Olympiad Progress Prize 3. Kaggle Competition, 2026. https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3\n2. Wei, J., Wang, X., Schuurmans, D., et al. Chain-of-thought prompting elicits reasoning in large language models. *NeurIPS* 35, 2022.\n3. Wang, X., Wei, J., Schuurmans, D., et al. Self-consistency improves chain of thought reasoning in language models. *ICLR*, 2023.\n4. Snell, C., Lee, J., Xu, K., Kumar, A. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. arXiv:2408.03314, 2024.\n5. Brown, B., Juravsky, J., Ehrlich, R., et al. Large language monkeys: Scaling inference compute with repeated sampling. arXiv:2407.21787, 2024.\n6. Huang, J., Chen, X., Mishra, S., et al. Large language models cannot self-correct reasoning yet. *ICLR*, 2024.\n7. Kwon, W., Li, Z., Zhuang, S., et al. Efficient memory management for large language model serving with PagedAttention. *SOSP*, 2023.\n8. Madaan, A., Tandon, N., Gupta, P., et al. Self-Refine: Iterative refinement with self-feedback. *NeurIPS* 36, 2023.\n9. Cobbe, K., Kosaraju, V., Bavarian, M., et al. Training verifiers to solve math word problems. arXiv:2110.14168, 2021.\n10. Hendrycks, D., Burns, C., Kadavath, S., et al. Measuring mathematical problem solving with the MATH dataset. *NeurIPS Datasets and Benchmarks*, 2021.\n11. OpenAI. GPT-OSS-120B. https://huggingface.co/openai/gpt-oss-120b\n12. Toshniwal, S., Du, W., Moshkov, I., et al. OpenMathInstruct-2. arXiv:2410.01560, 2024.\n13. AIMO Prize Foundation. AIMO Progress Prize 2 winning solutions. Numina, 2024.\n14. nihilisticneuralnet. AIMO3 Solution Notebook (44/50). Kaggle, 2026.\n15. huikang. Streaming inference with tool calling for mathematical reasoning. Kaggle, 2026.\n16. datasciencegrad. Stable 42/50 AIMO3 solution. Kaggle, 2026.\n\n---\n\n## Supplementary Materials\n\n- **Per-attempt CSV from the H100 reference reproduction**: see `runpod_results/results/per_attempt_log.csv` in the linked repository\n- **Standalone runner (Kaggle-free)**: `runpod/run_v143.py`\n- **Companion notebook with full version history**: [Chasing 47/50: AIMO3 Journey of 60+ Experiments](https://www.kaggle.com/code/pawanmali/chasing-47-50-aimo3-journey-of-50-experiments)\n- **Original implementation code**: [AIMO3 GPT-OSS-120B](https://www.kaggle.com/code/pawanmali/aimo3-gpt-oss-120b)\n\n---\n\n*Updated post-rebuttal, May 2026.*\n", "votes": 4 }, { "id": 3459448, "postDate": "2026-05-17T20:29:08.877Z", "content": "Apologies for the delay in posting these questions. We're reviewing whether there's a reason this submission was missed from our initial pass; in any case, I've spent the evening reading through your write-up and have the questions below.
\nGiven the late notice, we can give you until Monday, 18th May, 23:59 UTC to respond as a minimum (we'll update if we decide to extend that). Rest assured we'll take the time to give the submission the attention it deserves, and note that we're yet to make a final decision on the winners so you're very much still in consideration here.
\nFor wider context on the rebuttal phase, please see the updated discussion post.
\nLook forward to reading your responses on these.
", "rawMarkdown": "Apologies for the delay in posting these questions. We're reviewing whether there's a reason this submission was missed from our initial pass; in any case, I've spent the evening reading through your write-up and have the questions below.\n\nGiven the late notice, we can give you until Monday, 18th May, 23:59 UTC to respond as a minimum (we'll update if we decide to extend that). Rest assured we'll take the time to give the submission the attention it deserves, and note that we're yet to make a final decision on the winners so you're very much still in consideration here.\n\nFor wider context on the rebuttal phase, please see the [updated discussion post](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/689703).\n\n1. **Context-length finding** §4.2 reports a significant drop off in score (41 to 35) by increasing the size of the context window from 65K to 131K. In our own runs with gpt-oss-120b we've seen some problems that don't reach a correct answer until well past 65K tokens, so we'd expect the longer context to recover some problems even if fewer attempts run in parallel. Did you carry out any runs on external benchmarks to better understand what's going on here. For example, of the V135 problems that V125 got right, do you know how many timed out at 131K vs how many produced fewer attempts that still voted the right way vs how many produced the right answer in at least one attempt but the vote signal was too weak? It would be interesting to better understand whether the takeaway is \"context hurt\" or \"fewer parallel attempts hurt\" - if it is the latter, then there may be some balance to be struck between parallel attempts and longer context.\n2. **Visualising the results:** The 60+ version history is the heart of the writeup but everything is presented in tables rather than actual plots. Do you feel any charts would help bring the writeup to life, and if so are there any you'd want to add?\n\nLook forward to reading your responses on these.", "replies": [ { "id": 3460001, "postDate": "2026-05-18T14:24:56.147Z", "content": "Hi Sam,
\nThanks for the careful read, and for the extra time really appreciated. Both questions made me realise I'd been too confident about something I couldn't actually prove from the data. Spent the last day rebuilding the pipeline outside Kaggle so I could re-run with proper instrumentation, and rewriting the parts you flagged.
\nYou're right to push on this. The V125 vs V135 comparison just can't tell us what I claimed it does. When I look at the two server logs side by side, the only thing that's solidly true is that the score dropped from 41 to 35 when I doubled the context. But the same logs show vLLM's max concurrency falling from 7.11× down to 3.35× at the same max_num_seqs. In a pipeline that decides the answer by an 8-way vote, halving the parallel pool could on its own cost four to six points, before any individual problem became newly solvable or newly broken by the longer window.
The three-way split you described of V135 problems that V125 got right, how many timed out, how many voted right with fewer attempts, and how many had at least one correct attempt but the vote signal was too weak I can't give you. We only retained final scores for those runs. So I had been treating the drop as \"context hurt\" when really I had no per-problem evidence to support either side, and your hypothesis (it's mostly throughput, with a balance to be struck) is just as consistent with the numbers.
\nWhat I do have now: I rebuilt the V143 pipeline outside Kaggle this week with per-attempt CSV logging response tokens, completion status, all five entropy components individually, timeout flag, Python tool calls and errors, elapsed wall time, vote weight one row per attempt. Ran it against the 10-problem public reference set on an H100 80GB I rented on RunPod. It scored 10/10 in 31 minutes.
\nIf you'd like, I can run the controlled comparison this enables V143 at 65K vs 96K vs 131K, batch-matched so the effective parallelism is held roughly constant across the cells, three repetitions per cell for variance. That's about ten hours of H100 compute, around $30. It would directly answer the three-way decomposition you sketched, and would also test whether the 96K midpoint dominates either endpoint, which is what your \"balance\" hypothesis implies. Happy to do it either way before the deadline if it helps let me know.
\nI've reflected the honest framing in §4.2 of the new writeup. It now flags the V125/V135 comparison as confounded, lists both mechanisms, and pulls the \"context hurt\" framing back. I also acknowledged that some problems do legitimately need more than 65K tokens, which lines up with what you said you've seen in your own runs.
\nFair point. The version history really was the heart of the writeup and it was just sitting there as a table. I will be updating the writeup with six new figures in the updated version:
\nThe one I think you'll find most useful is a scatter of all 60-odd submissions on a single chart with a ±3.5 variance band drawn around the V143 baseline. What becomes obvious when you plot it that way is how few of our \"improvements\" actually escape the noise band. Most are inside it. Only V142 (Bayesian aggregation, -11) and V127 (verbose prompts, -9) clearly fall below it, and nothing clears V40 upward. Rendering it that way changed how I read our own three months of work.
\nA four-panel ablation grid does the same thing for temperature, attempt count, voting strategy, and context length each panel has the variance band overlaid so the reader can see which deltas are signal and which aren't. The context panel has a twin axis showing vLLM max concurrency next to the score, which makes the confound we just discussed visible without me having to belabour it in text.
\nThe figure I'm happiest about is built from a real per-attempt log: problem 86e8e5 from the reference run. All 8 attempts produced 8 different answers no consensus at all and the 5-component entropy correctly picked the right one purely on confidence, with a margin of 0.014 entropy units over the runner-up. It's the cleanest demonstration I have of the voting actually doing something a simple mean entropy couldn't.
There's also a small-multiples chart with the vote distribution for all ten reference problems (eight unanimous, two with a single outlier, that one 8-way-ambiguous case), and a scatter of per-attempt elapsed time against tokens generated that makes the ~100 tok/s throughput band visible and shows 86e8e5 as a clear outlier with all attempts running to deadline. Plus an H100 memory pie for anyone trying to reproduce.
I haven't updated the writeup yet but I'll do that soon in some time.
\nThanks again for taking the time.
", "rawMarkdown": "Hi Sam, \n\nThanks for the careful read, and for the extra time really appreciated. Both questions made me realise I'd been too confident about something I couldn't actually prove from the data. Spent the last day rebuilding the pipeline outside Kaggle so I could re-run with proper instrumentation, and rewriting the parts you flagged.\n\n## On the context length question\n\nYou're right to push on this. The V125 vs V135 comparison just can't tell us what I claimed it does. When I look at the two server logs side by side, the only thing that's solidly true is that the score dropped from 41 to 35 when I doubled the context. But the same logs show vLLM's max concurrency falling from 7.11× down to 3.35× at the same `max_num_seqs`. In a pipeline that decides the answer by an 8-way vote, halving the parallel pool could on its own cost four to six points, before any individual problem became newly solvable or newly broken by the longer window.\n\nThe three-way split you described of V135 problems that V125 got right, how many timed out, how many voted right with fewer attempts, and how many had at least one correct attempt but the vote signal was too weak I can't give you. We only retained final scores for those runs. So I had been treating the drop as \"context hurt\" when really I had no per-problem evidence to support either side, and your hypothesis (it's mostly throughput, with a balance to be struck) is just as consistent with the numbers.\n\nWhat I do have now: I rebuilt the V143 pipeline outside Kaggle this week with per-attempt CSV logging response tokens, completion status, all five entropy components individually, timeout flag, Python tool calls and errors, elapsed wall time, vote weight one row per attempt. Ran it against the 10-problem public reference set on an H100 80GB I rented on RunPod. It scored 10/10 in 31 minutes. \n\nIf you'd like, I can run the controlled comparison this enables V143 at 65K vs 96K vs 131K, batch-matched so the effective parallelism is held roughly constant across the cells, three repetitions per cell for variance. That's about ten hours of H100 compute, around $30. It would directly answer the three-way decomposition you sketched, and would also test whether the 96K midpoint dominates either endpoint, which is what your \"balance\" hypothesis implies. Happy to do it either way before the deadline if it helps let me know.\n\nI've reflected the honest framing in §4.2 of the new writeup. It now flags the V125/V135 comparison as confounded, lists both mechanisms, and pulls the \"context hurt\" framing back. I also acknowledged that some problems do legitimately need more than 65K tokens, which lines up with what you said you've seen in your own runs.\n\n## On the visualisations\n\nFair point. The version history really was the heart of the writeup and it was just sitting there as a table. I will be updating the writeup with six new figures in the updated version:\n\nThe one I think you'll find most useful is a scatter of all 60-odd submissions on a single chart with a ±3.5 variance band drawn around the V143 baseline. What becomes obvious when you plot it that way is how few of our \"improvements\" actually escape the noise band. Most are inside it. Only V142 (Bayesian aggregation, -11) and V127 (verbose prompts, -9) clearly fall below it, and nothing clears V40 upward. Rendering it that way changed how I read our own three months of work.\n\nA four-panel ablation grid does the same thing for temperature, attempt count, voting strategy, and context length each panel has the variance band overlaid so the reader can see which deltas are signal and which aren't. The context panel has a twin axis showing vLLM max concurrency next to the score, which makes the confound we just discussed visible without me having to belabour it in text.\n\nThe figure I'm happiest about is built from a real per-attempt log: problem `86e8e5` from the reference run. All 8 attempts produced 8 different answers no consensus at all and the 5-component entropy correctly picked the right one purely on confidence, with a margin of 0.014 entropy units over the runner-up. It's the cleanest demonstration I have of the voting actually doing something a simple mean entropy couldn't.\n\nThere's also a small-multiples chart with the vote distribution for all ten reference problems (eight unanimous, two with a single outlier, that one 8-way-ambiguous case), and a scatter of per-attempt elapsed time against tokens generated that makes the ~100 tok/s throughput band visible and shows `86e8e5` as a clear outlier with all attempts running to deadline. Plus an H100 memory pie for anyone trying to reproduce.\n\nI haven't updated the writeup yet but I'll do that soon in some time. \n\nThanks again for taking the time.", "replies": [ { "id": 3461525, "postDate": "2026-05-20T18:35:53.850Z", "content": "Please note, the visualizations are also updated. Thanks.
", "rawMarkdown": "Please note, the visualizations are also updated. Thanks. " } ] } ] }, { "id": 3446694, "postDate": "2026-04-22T00:50:50.033Z", "content": "Thank you for this - your notebook was the direct inspiration for our entropy-weighted voting approach, and we've updated our writeup's provenance section to reflect that more clearly.
\nYour finding that \"simple three-line prompts outperform elaborate structured instructions by 9 points\" matches our experience exactly - our multi-strategy prescriptive prompts (8 different technique-specific prompts) scored a mean of 36.6 across 5 submissions, while our one-sentence prompt with the 5-component entropy voting scored 40.78 mean across 9 runs.
\nOne question: you report \"five-component weighted entropy voting surpasses simple averaging by 6 points\" (V136: 35 vs V125: 41). Was that a single-run comparison or multi-run? At σ≈1.7, a 6-point gap from a single run has ~15% chance of being noise. Our own entropy-vs-baseline comparison is confounded because we changed the voting function AND stripped the scaffolding simultaneously - we can't isolate which change was load-bearing.
\nYour 60+ experiments over 2 months is impressive thoroughput for a solo competitor. The companion notebook has been very useful to the community (66 upvotes speaks for itself).
", "rawMarkdown": "Thank you for this - your notebook was the direct inspiration for our entropy-weighted voting approach, and we've updated our writeup's provenance section to reflect that more clearly.\n\nYour finding that \"simple three-line prompts outperform elaborate structured instructions by 9 points\" matches our experience exactly - our multi-strategy prescriptive prompts (8 different technique-specific prompts) scored a mean of 36.6 across 5 submissions, while our one-sentence prompt with the 5-component entropy voting scored 40.78 mean across 9 runs.\n\nOne question: you report \"five-component weighted entropy voting surpasses simple averaging by 6 points\" (V136: 35 vs V125: 41). Was that a single-run comparison or multi-run? At σ≈1.7, a 6-point gap from a single run has ~15% chance of being noise. Our own entropy-vs-baseline comparison is confounded because we changed the voting function AND stripped the scaffolding simultaneously - we can't isolate which change was load-bearing.\n\nYour 60+ experiments over 2 months is impressive thoroughput for a solo competitor. The companion notebook has been very useful to the community (66 upvotes speaks for itself).", "replies": [ { "id": 3446800, "postDate": "2026-04-22T06:13:34.740Z", "content": "Hi @utkarshsharma,\nThanks for the kind words, and glad the notebook was useful!\nThe V136 vs V125 comparison was single-run, so you're right that with σ≈1.7, a 6-point gap could partially be noise. However, I observed the same directional trend across multiple adjacent versions - mean entropy consistently underperformed position-weighted entropy by 3-6 points across ~8 different runs where that was the primary variable.\nThe more robust signal was the prompt complexity finding - that held across 15+ versions with very consistent results (verbose prompts never scored above 35, simple prompts rarely scored below 39).\nYour point about confounded changes is exactly why I started testing one variable at a time after V131 - combining 6 \"improvements\" simultaneously caused a 6-point regression, and I couldn't isolate which one was the culprit. Learned that the hard way!\nCongrats on your 40.78 mean - that's solid. Would be curious if you ever isolated the entropy vs scaffolding contribution.
", "rawMarkdown": " Hi @utkarshsharma,\n\nThanks for the kind words, and glad the notebook was useful!\n\nThe V136 vs V125 comparison was single-run, so you're right that with σ≈1.7, a 6-point gap could partially be noise. However, I observed the same directional trend across multiple adjacent versions - mean entropy consistently underperformed position-weighted entropy by 3-6 points across ~8 different runs where that was the primary variable.\n\nThe more robust signal was the prompt complexity finding - that held across 15+ versions with very consistent results (verbose prompts never scored above 35, simple prompts rarely scored below 39).\n\nYour point about confounded changes is exactly why I started testing one variable at a time after V131 - combining 6 \"improvements\" simultaneously caused a 6-point regression, and I couldn't isolate which one was the culprit. Learned that the hard way!\n\nCongrats on your 40.78 mean - that's solid. Would be curious if you ever isolated the entropy vs scaffolding contribution.\n", "replies": [ { "id": 3448908, "postDate": "2026-04-27T02:05:28.247Z", "content": "Really useful data point - \"mean entropy consistently underperformed position-weighted entropy by 3-6 points across ~8 different runs where that was the primary variable\" is the multi-run confirmation I was hoping existed. That's much stronger evidence than either of our single-run comparisons alone.
\nTo answer your question: no, we never isolated entropy vs scaffolding. Our converged submission simultaneously stripped the multi-strategy prompts AND replaced the voting function - we ran out of submission budget before we could test them independently. The honest answer is we don't know which change was load-bearing, and we say so explicitly in the writeup.
\nYour V131 experience (combining 6 \"improvements\" = 6-point regression) is almost identical to our adaptive router experiment - we combined a subject classifier, multi-stage solving, GenSelect, and adversarial verification into one notebook and it scored 28, a 12-point regression. Same lesson learned the same hard way.
", "rawMarkdown": "Really useful data point - \"mean entropy consistently underperformed position-weighted entropy by 3-6 points across ~8 different runs where that was the primary variable\" is the multi-run confirmation I was hoping existed. That's much stronger evidence than either of our single-run comparisons alone.\n\nTo answer your question: no, we never isolated entropy vs scaffolding. Our converged submission simultaneously stripped the multi-strategy prompts AND replaced the voting function - we ran out of submission budget before we could test them independently. The honest answer is we don't know which change was load-bearing, and we say so explicitly in the writeup.\n\nYour V131 experience (combining 6 \"improvements\" = 6-point regression) is almost identical to our adaptive router experiment - we combined a subject classifier, multi-stage solving, GenSelect, and adversarial verification into one notebook and it scored 28, a 12-point regression. Same lesson learned the same hard way." } ] } ] }, { "id": 3445891, "postDate": "2026-04-20T16:28:51.620Z", "content": "Please note that if you want to submit the write-up for the Write-Up prize you need to make a comment under the thread linke below, and follow those instructions:
\nhttps://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/689703
", "rawMarkdown": "Please note that if you want to submit the write-up for the Write-Up prize you need to make a comment under the thread linke below, and follow those instructions:\n\nhttps://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/689703" }, { "id": 3443552, "postDate": "2026-04-16T17:38:40.447Z", "content": "Did I miss it or you don't describe position-weighting? Can you describe/link-to position-weighted methods?
", "rawMarkdown": "Did I miss it or you don't describe position-weighting? Can you describe/link-to position-weighted methods?", "replies": [ { "id": 3443573, "postDate": "2026-04-16T18:00:42.463Z", "content": "Great question! The position-weighted entropy method works as follows:\n The key insight is that mathematical reasoning shows different confidence patterns at different stages - early tokens often involve uncertain exploration, while final answer tokens should show high confidence if the solution is correct.\n Our 5-component weighted entropy calculation:
\n