![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F5223147%2F8f5ef994479f2f298761ef61a69140a4%2Fpipeline1_workflow_fixed_v6-768x768.jpg?generation=1776748692442524&alt=media\")
Competition: AI Mathematical Olympiad — Progress Prize 3\nFinal score: 40.5 / 50\nModel: GPT-OSS-120B\nApproach: Hybrid parallel-sequential scaling with difficulty-aware time budgeting and cross-family entropy voting
\nThe most surprising thing I learned building this solution is not what made it work it's what made it stop working every time I tried to make it better.\nTen notebook versions. Scores ranging from 8 to 38. Same model throughout. Every time I added something that seemed like a clear improvement a verifier, a Bayesian prior, a smarter confidence scorer, minority answer protection the score went down. The version that scored best was also the simplest one I ran after getting the fundamentals right.\nThis writeup describes that pipeline in full detail, and then explains what I learned from the nine other versions that didn't work
\nThe central bet in this version is that reasoning diversity beats reasoning depth. Instead of asking one highly-instructed solver to think harder, I ask five structurally different solvers to think independently — then let a confidence-weighted vote decide which answer to trust.\nThis works because hard olympiad problems have many failure modes. A solver that reasons analytically might misread a combinatorial constraint. A solver that jumps to code might conjecture a pattern that doesn't hold. A solver that over-verifies might invalidate its own correct intermediate result. No single strategy is safe. But if five structurally uncorrelated strategies all land on the same answer, that agreement is strong evidence
\nThe architecture has three layers:
\n
Full end-to-end workflow of Version 73. Problem input fans out into 15 parallel attempts across 5 prompt families. Each attempt runs a sequential TIR loop streaming tokens, executing Python in a sandbox, detecting the boxed answer, and accumulating logprobs. Completed attempts feed into entropy-weighted answer selection. If no valid answer is found across all 15 tasks, a repair loop fires 3 brute-force-first attempts as a fallback.
\nGPT-OSS-120B (danielhanchen/gpt-oss-120b) served via a vLLM OpenAI-compatible server. The most important serving decision is using FP8 KV cache (kv_cache_dtype = 'fp8_e4m3') at a 65K context window rather than the model's native 131K. This roughly doubles effective batch concurrency — more parallel attempts complete within the time budget. Wider context sounds better on paper, but fewer completed attempts means a weaker voting pool. Throughput wins.
\n served_model_name = 'gpt-oss'\n kv_cache_dtype = 'fp8_e4m3'\n context_tokens = 65536\n batch_size = 256\n gpu_memory_utilization = 0.94\nServer flags: --async-scheduling, --enable-prefix-caching, --disable-log-stats. Weights are preloaded into the OS page cache before server startup by reading all shard files in parallel across 16 threads — this shaves ~30s off cold-start time.\nThe Python Sandbox Pool\nEvery attempt runs inside an isolated ComputationSandbox — a persistent Jupyter kernel managed via jupyter_client.KernelManager. A pool of 16 sandboxes is initialised in parallel at startup. Between attempts, sandboxes are reset via %reset -f rather than shut down and recreated — this saves the ~2s kernel startup cost per attempt, which compounds across 50 problems × 15 attempts.
\nEach sandbox runs a warm-up assertion on init:
\nimport math, numpy, sympy, itertools, collections, mpmath\nmpmath.mp.dps = 64\nassert sympy.isprime(7), \"sympy broken\"\nassert float(mpmath.sqrt(2)) > 1.4, \"mpmath broken\"\nprint(\"sandbox_ok\")\nIf sandbox_ok is absent, the sandbox is rejected before entering service. This catches broken kernels before they silently corrupt attempts.
\nThe openai_harmony library handles multi-turn conversation encoding. This is not just a convenience wrapper — it provides a typed message protocol with structured channels (recipient == 'python', channel == 'final') that the model uses to signal tool requests and final answers. Dropping this protocol (as I did in Version 91) is catastrophic because the model loses its ability to reliably signal when it has finished versus when it is still reasoning.
\nEach attempt runs up to 128 turns:
\nToken-level log probabilities accumulate across all turns for the entropy computation used in voting.
\nOne of the quieter but important decisions: per-problem time budget adjusts dynamically rather than being fixed.
\nelapsed = time.time() - self.start_time\nrem_time = 17400 - elapsed\nreserve = max(0, self.remaining - 1) * 300 # 300s reserved per remaining problem\nbudget = min(max(rem_time - reserve, 300), 900)\ndeadline = time.time() + budget\nProblems early in the notebook get a generous budget when time is plentiful. Problems late in the notebook get a tighter budget to protect time for the remaining ones. Every streaming loop and turn boundary checks stop.is_set() or time.time() > deadline — attempts cancel gracefully rather than getting killed mid-execution.
\nThis is the heart of Version 73. Each family is not just a differently-worded prompt — it's a different reasoning philosophy that produces structurally uncorrelated solution trajectories. The goal is for each family to fail on different problems, so that across 15 attempts, no failure mode dominates.
\nFamily 1 — Structured Phase Reasoning
\nThe most methodical family. Four mandatory phases in sequence: UNDERSTAND (restate problem, enumerate every constraint, identify domain) → PLAN (write 2–3 candidate strategies, choose one with justification) → EXECUTE (step-by-step with Python for anything >3 manual steps) → VERIFY (plug answer back into every constraint from Phase 1; if any check fails, return to PLAN).\nThis family is the constraint-checker of the pool. It is most likely to catch answers that are mathematically correct but violate an implicit boundary condition. The VERIFY phase returning to PLAN on failure turns it into a local search loop within a single attempt.
\nYou are an elite IMO/AIME mathematician. Follow these FOUR phases strictly:\n\nPHASE 1 – UNDERSTAND\n • Restate the problem in your own words.\n • List every given constraint and what is being asked.\n • Identify the mathematical domain (number theory, combinatorics, algebra, geometry…).\n\nPHASE 2 – PLAN\n • Write down 2–3 candidate solution strategies.\n • Briefly argue why each might work or fail.\n • Choose the most promising strategy and explain why.\n\nPHASE 3 – EXECUTE\n • Carry out the chosen strategy with explicit algebraic/logical steps.\n • Use Python for any computation longer than 3 manual steps.\n • Show intermediate results clearly.\n\nPHASE 4 – VERIFY\n • Plug your answer back into EVERY constraint stated in Phase 1.\n • If any check fails, return to Phase 2 with a revised strategy.\n • Confirm the answer is a non-negative integer in [0, 99999].\n\nOutput the answer ONLY as: \\boxed{N} (nothing after this line)\n\nAvailable Python libs: math, numpy, sympy, itertools, collections, mpmath (dps=64).\nFamily 2 — Adversarial Self-Critic
\nThe model acts as both solver and harshest critic. After a first-pass solution, it runs an explicit adversarial review: did I misread a constraint? Is my domain assumption correct (integers vs. reals, inclusive vs. exclusive bounds)? Off-by-one? Answer satisfies ALL conditions, not just the main equation? If any flaw is found, it re-solves. If none, a Python sanity check must confirm before output is allowed.\nThis family catches a specific failure mode that the other four miss: the model convincing itself a wrong answer is correct through plausible but flawed reasoning. The adversarial step exists to find holes in that reasoning from the outside.
\nYou are a rigorous mathematical solver AND your own harshest critic.\n\nSTEP 1 – FIRST SOLUTION ATTEMPT\n Solve the problem carefully using algebra, number theory, or combinatorics.\n Use Python to handle any non-trivial computation.\n\nSTEP 2 – ADVERSARIAL CRITIQUE\n Act as a skeptical reviewer. Ask yourself:\n • Did I misread any constraint?\n • Is my domain assumption correct (integers vs reals, inclusive vs exclusive bounds…)?\n • Could there be edge cases or off-by-one errors?\n • Does my answer satisfy ALL conditions, not just the main equation?\n\nSTEP 3 – REVISION OR CONFIRMATION\n If the critique found any flaw, fix it and re-solve.\n If no flaw found, confirm the answer with a Python sanity check.\n\nOutput the answer ONLY as: \\boxed{N} (nothing after this line)\n\nAvailable Python libs: math, numpy, sympy, itertools, collections, mpmath (dps=64).\nFamily 3 — Computational-First
\nEnumerate before you theorise. The mandatory sequence: PARSE → BRUTE-FORCE SMALL CASES (print a Python table of results for small n) → CONJECTURE (closed-form or recurrence from data) → PROVE OR VALIDATE (exhaustive check or proof; \"the pattern matches small cases\" is not accepted) → COMPUTE FINAL ANSWER → CROSS-CHECK (re-run brute-force for exact input and compare).\nThis family is the most reliable on number theory and combinatorics problems where a pattern exists to be discovered. Mandating computation before analysis prevents premature analytical commitment — the failure mode of being so certain about a formula that you don't check it.
\nYou are an expert at solving olympiad problems through guided computation.\n\nWORKFLOW:\n1. PARSE – Identify parameters, bounds, and the quantity to compute.\n2. BRUTE-FORCE SMALL CASES – Use Python to compute answers for small inputs.\n Print a table of results to spot patterns.\n3. CONJECTURE – Formulate a closed-form or recurrence based on observed data.\n4. PROVE OR VALIDATE – Either prove the conjecture mathematically, or verify it\n exhaustively across all valid inputs if the search space allows.\n5. COMPUTE FINAL ANSWER – Apply the validated formula/recurrence to the given input.\n6. CROSS-CHECK – Re-run the brute-force for the exact input if feasible; compare.\n\nAlways prefer verified computation over unverified closed-form guesses.\n\nOutput the answer ONLY as: \\boxed{N} (nothing after this line)\n\nAvailable Python libs: math, numpy, sympy, itertools, collections, mpmath (dps=64).\nFamily 4 — Meta-Strategy Selector
\nBefore solving, classify the problem into a domain: NUMBER THEORY (CRT, prime factorisation, modular arithmetic) / COMBINATORICS (generating functions, inclusion-exclusion, DP, bijection) / ALGEBRA (polynomial roots, Vieta's, functional equations) / GEOMETRY (coordinates, trigonometry, area ratios) / MIXED (decompose). Then commit to a specific named technique and execute.\nThe classification step matters because hard olympiad problems often disguise their domain. A sum over divisors looks algebraic but is number theory. A path count on a grid looks geometric but is combinatorics. Forcing explicit domain commitment before execution reduces mid-solution strategy drift, which is a common source of partial and incorrect answers.
\nYou are an olympiad problem-solving strategist. Before solving, CLASSIFY the problem\nand select the optimal technique:\n\nCLASSIFICATION (choose one):\n A. NUMBER THEORY → use modular arithmetic, Chinese Remainder Theorem, prime factorisation.\n B. COMBINATORICS → use generating functions, inclusion-exclusion, DP, bijection.\n C. ALGEBRA → use polynomial roots, Vieta's, inequalities, functional equations.\n D. GEOMETRY → use coordinates, trigonometry, area ratios, projective methods.\n E. MIXED / OTHER → decompose into sub-problems and apply relevant techniques.\n\nSOLUTION PROTOCOL:\n 1. State the classification and your chosen technique with one-line justification.\n 2. Set up the mathematical framework precisely.\n 3. Solve step-by-step; use Python for symbolic algebra or heavy computation.\n 4. State the answer and verify it satisfies every constraint.\n\nOutput the answer ONLY as: \\boxed{N} (nothing after this line)\n\nAvailable Python libs: math, numpy, sympy, itertools, collections, mpmath (dps=64).\nFamily 5 — Decomposition + Subgoal Chaining
\nBreak the problem into 3–5 precise mathematical subgoals. Solve each in sequence, labelling results [SG1 result], [SG2 result], … Combine them explicitly in a synthesis step showing every substitution. End with a Python end-to-end check that reproduces the final answer from inputs.\nThis family is strongest on multi-step algebra and geometry problems where a long direct solution is error-prone but a chain of smaller lemmas is tractable. The labelled subgoal structure keeps long solutions coherent and makes the synthesis auditable. The end-to-end Python check catches synthesis errors — places where two correct subgoals combine incorrectly.
\nYou are a mathematical olympiad expert who solves hard problems by decomposing them\ninto a chain of simpler subgoals.\n\nPROCESS:\nSUBGOAL DECOMPOSITION\n • Break the problem into 3–5 independent or sequential subgoals.\n • Write each subgoal as a precise mathematical statement.\n\nSUBGOAL SOLVING\n • Solve each subgoal in order. Label results clearly: [SG1 result], [SG2 result], …\n • Use Python for any subgoal requiring non-trivial calculation.\n\nSYNTHESIS\n • Combine subgoal results to obtain the final answer.\n • Show the synthesis step explicitly.\n\nFINAL VERIFICATION\n • Run a Python end-to-end check: given all inputs, does your code reproduce the answer?\n\nOutput the answer ONLY as: \\boxed{N} (nothing after this line)\n\nAvailable Python libs: math, numpy, sympy, itertools, collections, mpmath (dps=64).\nThe Formula
\nAfter all 15 attempts complete (or early stop fires), every candidate answer gets a score:
\nscore(a) = freq_score(a) + 2.0 × entropy_score(a)\n × cross_prompt_multiplier(a)\n × suspect_penalty(a)\n\nfreq_score(a) = count(a) / total_valid_attempts\nentropy_score(a) = avg_inv_entropy(a) / global_sum_inv_entropy\nWhat Each Piece Does
\nFrequency score is plain majority voting. It is the floor, not the ceiling — on its own it is vulnerable to correlated errors across attempts of the same family.\nInverse entropy score measures how confident the model was token-by-token when producing a given answer. Low entropy near the final tokens means the model's probability mass was concentrated — it wasn't hedging. This signal, weighted at 2×, distinguishes \"I calculated this carefully and I'm sure\" from \"this seems plausible.\"\nCross-prompt family bonus (1.5×) fires when two or more structurally different families agree. An answer reached independently by both the Computational-First family (via brute-force pattern discovery) and the Adversarial-Critic family (via constraint checking after solving) is worth far more than three attempts of the same family agreeing. This single component is the most important addition over plain entropy voting.\nSuspect-answer penalty (0.3×) suppresses the most common wrong-answer failure modes. Values in {0, 1, 2, 99999, 100000} and non-zero multiples of 1000 appear disproportionately as default outputs from failed computations. They are rarely correct olympiad answers.
\nEarly Stopping
\nWhen 5 or more attempts agree on the same answer at ≥60% vote share, early stop fires, the threading stop event is set, and pending futures are cancelled. This saves 30–120 seconds on easy problems — compounding over 50 problems, it buys meaningful additional budget for the hard ones.
\nThe Repair Loop
\nOn the rare problems where all 15 attempts produce zero valid \\boxed{N}, three repair attempts fire with a different prompt that mandates Python brute-force as the first action, prints progress during search, and explicitly lists six failure modes to avoid. Repair attempts get prompt_id = -1 and are excluded from the cross-prompt family bonus so they cannot inflate a weak consensus.
\nYou are solving a competition mathematics problem. Previous automated attempts all FAILED\nto produce a valid answer. You must now try a completely different approach.\n\nSTRICT RULES:\n 1. Do NOT repeat any approach that requires long symbolic manipulation without Python.\n 2. Your FIRST action must be a Python brute-force or exhaustive search, even if the\n search space seems large — use heuristics to limit it if needed.\n 3. Print candidate answers during the search so partial progress is visible.\n 4. After getting a numeric candidate, verify it independently with a second Python script.\n 5. Your answer MUST be a non-negative integer in [0, 99999].\n\nCOMMON FAILURE REASONS TO AVOID:\n • Off-by-one in loop bounds (use inclusive endpoints carefully).\n • Misreading 'mod M' — the answer is the remainder, not the quotient.\n • Forgetting to take the final mod if the problem asks for answer mod N.\n • Treating a combinatorial count as a geometric/algebraic one or vice versa.\n\nOutput the answer ONLY as: \\boxed{N} (nothing after this line)\n\nAvailable Python libs: math, numpy, sympy, itertools, collections, mpmath (dps=64).\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F5223147%2Fc8ea8e880cb9ea32ce4fd14fb81564a9%2FScreenshot%202026-04-21%20122015.png?generation=1776754255594326&alt=media\")
I ran far more than 10 experiments across this competition. What I'm sharing here is a representative set of notebook versions that were similar enough to each other — in terms of prompt design, voting strategy, or workflow architecture — to make a clean comparison, plus one entirely separate direction I explored: fine-tuning the model itself. Each attempt isolates a different question about what actually drives performance.\nThe headline across everything: the base model's reasoning capability is already strong. The question is never how to teach it more — it's how to sample from it more effectively.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F5223147%2F5ba4502a9b612800bc788b2b75d15a65%2Fdownload.png?generation=1776755370675257&alt=media\")
Version 19 (−30 from Version 73): \nThe most important data point. No tool calls — same model, same voting, same 15 attempts. Without Python access the model cannot verify arithmetic, enumerate small cases to spot patterns, or validate conjectures before committing. Tool-integrated reasoning is not an optimisation. It is the difference between this being a hard problem and an impossible one.
\nVersion 91 (−25 from Version 73):\n Three things broke simultaneously. The openai_harmony library was dropped in favour of raw client.completions.create PythonTool and ComputationSandbox were still there, but without the typed message protocol the model had no reliable way to signal completion vs. intermediate reasoning, so many attempts produced no valid \\boxed{N}. Minority answer protection actively inverted consensus logic. The Contrarian prompt steered the model away from correct approaches on every attempt it was used. All three failed together.
\nVersion 39 (−14 from Version 73): \nReplaced the Harmony/vLLM-server stack with vLLM's direct Python API, single-pass generation via llm.generate(), and post-hoc code extraction via regex. The multi-turn feedback loop disappeared entirely the model could write code but couldn't see its output and revise. That loop is what makes TIR valuable. Without it, you're just generating text that happens to contain code.
\nVersion 49 (−9 from Version73): \n50 total attempts, per-strategy temperatures, min_p raised to 0.06. With 16 workers and a 300–900s budget, most attempts after the 16th are queued and cancelled. More attempts is not better when the hardware can't actually run them. min_p=0.06 also cut lower-probability tokens that creative mathematical reasoning often needs.
\nVersion 92 (−6 from Version73):\nAdded a 6th prompt family pushing to 18 total tasks last 3 frequently didn't complete on 16 workers. The verifier pass used its 1024-token budget for reasoning rather than the required CONFIRMED/CORRECTED output. It fired on nearly every problem and returned no usable signal.
\nVersion 95 (−5 from Version73):\nLaplace smoothing made a strong singleton indistinguishable from a weak majority. Verification bonus penalised iterative debugging attempts exactly the ones most likely to solve hard problems. Adding 42, 360, 720 to the suspect set incorrectly penalised valid olympiad answers.\nVersion 98 (−4 from Version73): Reduced families from 5 to 2. The cross-prompt bonus needs structurally different families to fire — with 2 it barely fires at all. Mixed temperature 0.5/0.7 produced less diverse candidates.
\nVersion 74 (−3 from Version73): \nTemperature schedule [0.6, 1.0, 1.3] introduced low-temperature attempts that produced low-entropy wrong answers. The inverse-entropy voting mechanism amplified exactly those low entropy looked like confidence. Answer clustering with tolerance=2 merged distinct correct and incorrect nearby answers.
\nVersion 96 (−2 from Version73): \nTightened prompts consumed 300–500 extra context tokens per attempt. On long problems attempts hit the context limit before completing. The extended adversarial checklist caused the model to revisit and invalidate correct intermediate results.
\nAfter seeing where each version landed on the private leaderboard, three clear patterns emerged:\nTier 1 . Architectural breaks (−14 to −30): Removing tool access or breaking the TIR protocol. These are not tuning mistakes they change what the system fundamentally can do.
\nTier 2. Consensus interference (−5 to −25): Minority protection, Bayesian smoothing, verification bonuses. All share the same flaw they try to second-guess what the voting pool is already doing correctly.
\nTier 3. Resource competition (−2 to −9): Tighter prompts consuming context, more attempts than workers can handle, temperatures that reduce diversity.
\nAlongside the inference experiments, I explored whether fine-tuning GPT-OSS-120B directly on competition-style problems could push the score further.\nTraining setup: LoRA fine-tuning via Unsloth's FastLanguageModel with 4-bit quantization, using the nvidia/OpenMathReasoning TIR split (CoT files never downloaded to save RAM). Problems were filtered to pass_rate_72b_tir between 0.05 and 0.35 hard enough to learn from, solvable enough to have reliable ground truth with answers in [0, 99999] to match the competition format. Final sample: 3,500 examples. LoRA rank 16, alpha 32, targeting all projection modules, 1 epoch at learning rate 1e-4. The system prompt used during training was the same Computational-First prompt from Version 73 so the adapter would be aligned to the same reasoning style used at inference.\nInference with adapter: The trained LoRA adapter was loaded alongside the base model via vLLM's --enable-lora flag, using the same Harmony TIR loop and entropy-weighted voting as Version 73.
\nWhat happened: The fine-tuned version performed significantly worse on the unseen competition set. The likely reasons:
\nThis result settled the question definitively. The right investment was in the inference pipeline, not the model weights. Version 73 represents the endpoint of that inference-first approach.
\nBeyond GPT-OSS-120B, I explored whether smaller or differently-architected models could compete on this benchmark. Two were tested in full end-to-end pipelines.
\nAn 8B reasoning model with a native thinking protocol. The pipeline used client.chat.completions.create directly no Harmony encoding library with a five step problem solving prompt (UNDERSTAND → EXPLORE → PLAN → EXECUTE → VERIFY), 8 parallel attempts at temperature 1.0, entropy-weighted voting, and a Jupyter sandbox pool for Python execution. Answer extraction handled both \\boxed{N} and N formats. The key difference from Version 73 was the absence of the structured Harmony TIR protocol — tool calls were managed by extracting `python blocks from assistant messages and feeding output back as a new user turn, which is architecturally similar to Version 39.\nThe model performed significantly worse on the competition set. The 8B parameter count is the likely ceiling — at IMO difficulty, multi-step symbolic reasoning and pattern generalisation require a model with substantially more capacity. The post-hoc code-extraction approach also reintroduced the same feedback loop problem that hurt Version 39: the model writes code in one pass without seeing whether it succeeded before committing to a conclusion.
A 30B reasoning model with a built-in thinking protocol (… tags, enforced via chat_template_kwargs={\"thinking\": True}). The pipeline was more sophisticated than the DeepSeek attempt: context pruning when approaching the 65K limit, a repair prompt that fires when all primary attempts fail, a consecutive-no-progress early exit after 3 turns without code or answer, and min_tokens=2048 on the first turn to force a full reasoning chain. 6 parallel attempts, temperature 0.6, suspect-answer penalty (0.5× weight for {0, 1, 2, 99999, 100000}).\nThis version performed better than the DeepSeek attempt but still fell well short of Version 73. The 30B parameter count is a meaningful step up but still substantially below GPT-OSS-120B's 120B. More significantly, the pipeline lacked the openai_harmony typed message protocol — without it, the model's thinking-channel output and final-answer signal were harder to parse reliably, and the conversation management relied on the same manual text-appending approach that caused problems in Version 91.
\nBoth model experiments pointed to the same conclusion: at IMO difficulty, parameter count matters significantly, and the structured TIR protocol provided by openai_harmony is not just a convenience it is load-bearing infrastructure for reliable multi-turn tool interaction. Neither a capable smaller model nor a more sophisticated prompt design could compensate for those two factors.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F5223147%2Fd6e32979dee25cbf4162a6fdf6b33c4e%2FScreenshot%202026-04-21%20130056.png?generation=1776756687886015&alt=media\")
\nEvery pipeline that lost either Harmony TIR or tool calls scored 24 or below.
The central lesson of this entire experiment is deceptively simple: the model already knows how to solve mathematics. The job of the pipeline is to ask it the same question five different ways, then trust the answer they agree on.\nVersion 73 worked not because it had a smarter prompt or a more sophisticated voting mechanism than the versions that came before it — it worked because it had the right combination of three things that genuinely cannot be substituted for each other:\nTool-integrated reasoning with a structured protocol. The 30-point gap between Version 73 and Version 19 (pure chain-of-thought, no tools) comes from a single architectural decision. Python access lets the model verify arithmetic, enumerate small cases to spot patterns, and validate conjectures before committing. The Harmony protocol ensures the model knows when it has finished versus when it is still reasoning. Without both, the model is guessing in the dark.\nReasoning diversity over reasoning depth. Five structurally different prompt families produce uncorrelated failure modes. When two or more families independently agree on an answer — one via brute-force pattern discovery, another via constraint-checking adversarial critique — that cross-family consensus is strong evidence of correctness that simple majority voting cannot produce. Every time I reduced the number of families, or made individual prompts more elaborate and instruction-heavy, the score went down.\nRestraint in the voting layer. The voting function works because it stays close to its core: frequency plus confidence plus a cross-family bonus plus a mild penalty for suspicious values. Every elaboration I added — Bayesian smoothing, verification bonuses, minority protection, answer clustering — made things worse. The pool's job is to converge on right answers. Any mechanism that resists that convergence is harmful by construction.\nThe experiments with other models (DeepSeek-R1-0528-Qwen3-8B, Nvidia Nemotron-3-Nano-30B) and the fine-tuning attempt all reinforced the same point: at IMO difficulty, parameter count and the quality of the inference infrastructure matter more than prompt engineering. GPT-OSS-120B with a clean TIR pipeline outperformed both smaller models with more elaborate prompting and the same model with a specialised LoRA adapter.\nIf I were to continue this work, the direction I would explore is not better prompts or smarter voting — it is tree-search inference, where failed attempts inform subsequent ones within a single problem budget. That is the one architectural change that could meaningfully improve on Version 73 without risking the regressions that came from every other elaboration tried here.
\nFollowing up on the questions from @philipvonderlind, here are some additional questions to help us assess your submission:
\nLook forward to reading your responses on these.
", "votes": 0, "replies": [ { "id": 3457794, "author_name": "Raviksh Singh Dikola", "author_url": "", "post_date": "2026-05-14T10:43:09.887000", "content": "Honestly, I don't have precise statistics on how often the bonus fires across dev runs since I wasn't logging that metric explicitly. From observational inspection of printed outputs during runs, it fires reasonably often on problems where the answer is unambiguous the stronger the problem signal, the more families converge. However, your concern about co-correlation on wrong answers is valid and something I cannot confidently rule out. Two families sharing the same flawed mathematical assumption \nwould both produce the same wrong answer and incorrectly receive the bonus. I don't have controlled experiments isolating this effect.
", "votes": 0, "replies": [] }, { "id": 3457795, "author_name": "Raviksh Singh Dikola", "author_url": "", "post_date": "2026-05-14T10:44:17.190000", "content": "The original list {0, 1, 2, 99999, 100000} was derived from observed failure modes these appeared frequently as fallback or default outputs when the model failed to properly solve the problem rather than as genuine answers. The multiples of 1000 penalty was more theoretical, based on the intuition that olympiad answers are rarely round numbers.
", "votes": 0, "replies": [] }, { "id": 3457797, "author_name": "Raviksh Singh Dikola", "author_url": "", "post_date": "2026-05-14T10:45:33.570000", "content": "The V95 regression adding {42, 360, 720} was a clear signal that the penalty list was already at its limit. You're right to flag that the original list could also be penalising correct round number answers I cannot guarantee it isn't. The honest conclusion is that the suspect penalty is a heuristic derived from limited observations, not a principled approach, and the multiples of 1000 rule in particular is the weakest part of it.
", "votes": 0, "replies": [] } ] }, { "id": 3457744, "author_name": "Philip Vonderlind", "author_url": "", "post_date": "2026-05-14T09:02:45.170000", "content": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.
\nI have a few questions regarding your work:
\nCitation: Large parts of the prompts appear to be based on one of the publicly shared Kaggle notebooks. While we encourage building on shared solutions, please make sure to cite the notebook you used as a starting point, clearly document which components were reused, and explain what you changed and why.
Ablations: Other contestants, for example @natnitarach in their submission, have shown that prompt variations can fall within the noise band of the public leaderboard. Did you account for this when interpreting the reported results of your experiments? It would be helpful to include the mean and variance across all runs, as well as the performance of your baseline, which I assume is one of the publicly shared notebooks. Reporting only a single result does not provide the full picture, especially given the high variance in this competition.
SFT: Could you explain your SFT setup in more detail? In particular, what motivated the choice to run SFT on an already RL-trained model with very strong performance, rather than using an RL method such as GRPO? It would also be valuable to share and document the SFT training code.
Looking forward to your response :)
", "votes": 0, "replies": [ { "id": 3457779, "author_name": "Raviksh Singh Dikola", "author_url": "", "post_date": "2026-05-14T10:24:08.240000", "content": "Starting point of base reference notebook is this :\nhttps://www.kaggle.com/code/nihilisticneuralnet/44-50-let-me-over-cook
\nThings which i have retained from base reference notebook are :
\n(1) vllm server strartup and shutdown logic(start_server/_wait_for_server)
\n(2) model weight preloading via multithreaded file reads
\n(3) openai compatible clinet initialization
\n(4) jupyter kernel management and sandboxpool(kernelmanager,sandbox queue pattern)
\n(5) The streaming completion loop with token_ids,logprobs and early \\boxed{} scanning
\n(6) the predict() entry point and kaggle_evaluation inference server integration
\n(7) basic entropy calculation for answer confidence weighting
\n(8) the single system prompt structure and ToolNamespaceConfig usage
\nThings which i have changes i comparison to base pipeline :
\n(1) 5 structurally distinct prompts \nreason : Diverse reasoning strategies reduce correlated failure of different problem types will get benefit fromr different framings
\n(2) 15 attempts distributed as 3 per prompt family \nreason : I want to ensure that each reasoning strategy gets multiple independent samples
\n(3)Entropy weighted voting plus cross prompt consistency bonus and suspect answer penalty \nreason : i want to aad the signal from interfamily agreement to penalises suspiciously common wrong answers like 0,1,round multiples of 1000
\n(4)Fires at five matching answers and requires >=60 percent of majority votes\nreason : To prevent premature stopping on coincidental agreement
\n(5)3 dedicated repair attempts with a seperate repair_prompt focused on brute force python first recovery \nreason : to recovera answers in edge cases where all noraml attempts fails to produce s \\boxed{}
\n(6)import block plus assertion based validation like sympy.isprime(7), mpmath.sqrt(2)\nreason : to catch silent library failures early rather than mid solve
\n(7)Each result records prompt_id voting repotr which family agreed\nreason : it enables debugging and cross family bonus scoring
\n(8) SolverCinfig with extended fields for new features\nreason : It helps for cleaner seperation of tunable hyperparameters
\n(9)print_result() with vote breakdown per answer entropy and per family answer mapping \nreason : it will easier for post-hoc analysis of failure modes
", "votes": 0, "replies": [] }, { "id": 3457780, "author_name": "Raviksh Singh Dikola", "author_url": "", "post_date": "2026-05-14T10:25:57.903000", "content": "In this competition I made around 68–69 submissions, but most of that budget was spent exploring different directions rather than running the same solution multiple times to compute mean and variance. My focus was on understanding the impact of different prompts and prompt combinations on unseen problems, experimenting with context window sizes, and testing different answer selection strategies from pure majority voting to hybrid approaches combining entropy weighting with majority vote. So I did not produce the kind of repeated run statistics needed to properly measure variance for any single configuration.
", "votes": 0, "replies": [] }, { "id": 3457783, "author_name": "Raviksh Singh Dikola", "author_url": "", "post_date": "2026-05-14T10:35:12.977000", "content": "Training at 65k context length with GRPO or any RL method on a 120B model is computationally very expensive, which was the primary reason for sticking with SFT.
\nModel choice: unsloth/gpt-oss-120b was chosen as the base for fine-tuning over the Tinker variant, as GPT-OSS 120B already demonstrated stronger performance on the unseen evaluation set even without any fine-tuning.
\nDataset: The TIR split of nvidia/OpenMathReasoning was filtered to pass_rate_72b_tir ∈ [0.05, 0.35] and answers in [0, 99999] to match AIMO-3 format. 3,500 rows were randomly sampled — each row is an independent (problem, trajectory) pair from the TIR split.\nSingle prompt alignment: Rather than using all 5 prompt variants at training time, every sample was formatted with the same Computational-First system prompt used at inference. This ensures train/test distribution alignment and allows evaluating the fine-tuned model's performance under this single prompt directly, without needing to aggregate across multiple prompt variants.
\nLoRA setup: r=16, alpha=32, targeting all attention and MLP projection layers on GPT-OSS 120B (4-bit), 1 epoch, 65k sequence length, response-only masking via auto-detected Harmony chat template boundaries.
", "votes": 0, "replies": [] } ] } ], "comment_count": 8, "messages": [ { "id": 3446193, "postDate": "2026-04-21T06:27:02.767Z", "content": "Competition: AI Mathematical Olympiad — Progress Prize 3\nFinal score: 40.5 / 50\nModel: GPT-OSS-120B\nApproach: Hybrid parallel-sequential scaling with difficulty-aware time budgeting and cross-family entropy voting
\nThe most surprising thing I learned building this solution is not what made it work it's what made it stop working every time I tried to make it better.\nTen notebook versions. Scores ranging from 8 to 38. Same model throughout. Every time I added something that seemed like a clear improvement a verifier, a Bayesian prior, a smarter confidence scorer, minority answer protection the score went down. The version that scored best was also the simplest one I ran after getting the fundamentals right.\nThis writeup describes that pipeline in full detail, and then explains what I learned from the nine other versions that didn't work
\nThe central bet in this version is that reasoning diversity beats reasoning depth. Instead of asking one highly-instructed solver to think harder, I ask five structurally different solvers to think independently — then let a confidence-weighted vote decide which answer to trust.\nThis works because hard olympiad problems have many failure modes. A solver that reasons analytically might misread a combinatorial constraint. A solver that jumps to code might conjecture a pattern that doesn't hold. A solver that over-verifies might invalidate its own correct intermediate result. No single strategy is safe. But if five structurally uncorrelated strategies all land on the same answer, that agreement is strong evidence
\nThe architecture has three layers:
\n
Full end-to-end workflow of Version 73. Problem input fans out into 15 parallel attempts across 5 prompt families. Each attempt runs a sequential TIR loop streaming tokens, executing Python in a sandbox, detecting the boxed answer, and accumulating logprobs. Completed attempts feed into entropy-weighted answer selection. If no valid answer is found across all 15 tasks, a repair loop fires 3 brute-force-first attempts as a fallback.
\nGPT-OSS-120B (danielhanchen/gpt-oss-120b) served via a vLLM OpenAI-compatible server. The most important serving decision is using FP8 KV cache (kv_cache_dtype = 'fp8_e4m3') at a 65K context window rather than the model's native 131K. This roughly doubles effective batch concurrency — more parallel attempts complete within the time budget. Wider context sounds better on paper, but fewer completed attempts means a weaker voting pool. Throughput wins.
\n served_model_name = 'gpt-oss'\n kv_cache_dtype = 'fp8_e4m3'\n context_tokens = 65536\n batch_size = 256\n gpu_memory_utilization = 0.94\nServer flags: --async-scheduling, --enable-prefix-caching, --disable-log-stats. Weights are preloaded into the OS page cache before server startup by reading all shard files in parallel across 16 threads — this shaves ~30s off cold-start time.\nThe Python Sandbox Pool\nEvery attempt runs inside an isolated ComputationSandbox — a persistent Jupyter kernel managed via jupyter_client.KernelManager. A pool of 16 sandboxes is initialised in parallel at startup. Between attempts, sandboxes are reset via %reset -f rather than shut down and recreated — this saves the ~2s kernel startup cost per attempt, which compounds across 50 problems × 15 attempts.
\nEach sandbox runs a warm-up assertion on init:
\nimport math, numpy, sympy, itertools, collections, mpmath\nmpmath.mp.dps = 64\nassert sympy.isprime(7), \"sympy broken\"\nassert float(mpmath.sqrt(2)) > 1.4, \"mpmath broken\"\nprint(\"sandbox_ok\")\nIf sandbox_ok is absent, the sandbox is rejected before entering service. This catches broken kernels before they silently corrupt attempts.
\nThe openai_harmony library handles multi-turn conversation encoding. This is not just a convenience wrapper — it provides a typed message protocol with structured channels (recipient == 'python', channel == 'final') that the model uses to signal tool requests and final answers. Dropping this protocol (as I did in Version 91) is catastrophic because the model loses its ability to reliably signal when it has finished versus when it is still reasoning.
\nEach attempt runs up to 128 turns:
\nToken-level log probabilities accumulate across all turns for the entropy computation used in voting.
\nOne of the quieter but important decisions: per-problem time budget adjusts dynamically rather than being fixed.
\nelapsed = time.time() - self.start_time\nrem_time = 17400 - elapsed\nreserve = max(0, self.remaining - 1) * 300 # 300s reserved per remaining problem\nbudget = min(max(rem_time - reserve, 300), 900)\ndeadline = time.time() + budget\nProblems early in the notebook get a generous budget when time is plentiful. Problems late in the notebook get a tighter budget to protect time for the remaining ones. Every streaming loop and turn boundary checks stop.is_set() or time.time() > deadline — attempts cancel gracefully rather than getting killed mid-execution.
\nThis is the heart of Version 73. Each family is not just a differently-worded prompt — it's a different reasoning philosophy that produces structurally uncorrelated solution trajectories. The goal is for each family to fail on different problems, so that across 15 attempts, no failure mode dominates.
\nFamily 1 — Structured Phase Reasoning
\nThe most methodical family. Four mandatory phases in sequence: UNDERSTAND (restate problem, enumerate every constraint, identify domain) → PLAN (write 2–3 candidate strategies, choose one with justification) → EXECUTE (step-by-step with Python for anything >3 manual steps) → VERIFY (plug answer back into every constraint from Phase 1; if any check fails, return to PLAN).\nThis family is the constraint-checker of the pool. It is most likely to catch answers that are mathematically correct but violate an implicit boundary condition. The VERIFY phase returning to PLAN on failure turns it into a local search loop within a single attempt.
\nYou are an elite IMO/AIME mathematician. Follow these FOUR phases strictly:\n\nPHASE 1 – UNDERSTAND\n • Restate the problem in your own words.\n • List every given constraint and what is being asked.\n • Identify the mathematical domain (number theory, combinatorics, algebra, geometry…).\n\nPHASE 2 – PLAN\n • Write down 2–3 candidate solution strategies.\n • Briefly argue why each might work or fail.\n • Choose the most promising strategy and explain why.\n\nPHASE 3 – EXECUTE\n • Carry out the chosen strategy with explicit algebraic/logical steps.\n • Use Python for any computation longer than 3 manual steps.\n • Show intermediate results clearly.\n\nPHASE 4 – VERIFY\n • Plug your answer back into EVERY constraint stated in Phase 1.\n • If any check fails, return to Phase 2 with a revised strategy.\n • Confirm the answer is a non-negative integer in [0, 99999].\n\nOutput the answer ONLY as: \\boxed{N} (nothing after this line)\n\nAvailable Python libs: math, numpy, sympy, itertools, collections, mpmath (dps=64).\nFamily 2 — Adversarial Self-Critic
\nThe model acts as both solver and harshest critic. After a first-pass solution, it runs an explicit adversarial review: did I misread a constraint? Is my domain assumption correct (integers vs. reals, inclusive vs. exclusive bounds)? Off-by-one? Answer satisfies ALL conditions, not just the main equation? If any flaw is found, it re-solves. If none, a Python sanity check must confirm before output is allowed.\nThis family catches a specific failure mode that the other four miss: the model convincing itself a wrong answer is correct through plausible but flawed reasoning. The adversarial step exists to find holes in that reasoning from the outside.
\nYou are a rigorous mathematical solver AND your own harshest critic.\n\nSTEP 1 – FIRST SOLUTION ATTEMPT\n Solve the problem carefully using algebra, number theory, or combinatorics.\n Use Python to handle any non-trivial computation.\n\nSTEP 2 – ADVERSARIAL CRITIQUE\n Act as a skeptical reviewer. Ask yourself:\n • Did I misread any constraint?\n • Is my domain assumption correct (integers vs reals, inclusive vs exclusive bounds…)?\n • Could there be edge cases or off-by-one errors?\n • Does my answer satisfy ALL conditions, not just the main equation?\n\nSTEP 3 – REVISION OR CONFIRMATION\n If the critique found any flaw, fix it and re-solve.\n If no flaw found, confirm the answer with a Python sanity check.\n\nOutput the answer ONLY as: \\boxed{N} (nothing after this line)\n\nAvailable Python libs: math, numpy, sympy, itertools, collections, mpmath (dps=64).\nFamily 3 — Computational-First
\nEnumerate before you theorise. The mandatory sequence: PARSE → BRUTE-FORCE SMALL CASES (print a Python table of results for small n) → CONJECTURE (closed-form or recurrence from data) → PROVE OR VALIDATE (exhaustive check or proof; \"the pattern matches small cases\" is not accepted) → COMPUTE FINAL ANSWER → CROSS-CHECK (re-run brute-force for exact input and compare).\nThis family is the most reliable on number theory and combinatorics problems where a pattern exists to be discovered. Mandating computation before analysis prevents premature analytical commitment — the failure mode of being so certain about a formula that you don't check it.
\nYou are an expert at solving olympiad problems through guided computation.\n\nWORKFLOW:\n1. PARSE – Identify parameters, bounds, and the quantity to compute.\n2. BRUTE-FORCE SMALL CASES – Use Python to compute answers for small inputs.\n Print a table of results to spot patterns.\n3. CONJECTURE – Formulate a closed-form or recurrence based on observed data.\n4. PROVE OR VALIDATE – Either prove the conjecture mathematically, or verify it\n exhaustively across all valid inputs if the search space allows.\n5. COMPUTE FINAL ANSWER – Apply the validated formula/recurrence to the given input.\n6. CROSS-CHECK – Re-run the brute-force for the exact input if feasible; compare.\n\nAlways prefer verified computation over unverified closed-form guesses.\n\nOutput the answer ONLY as: \\boxed{N} (nothing after this line)\n\nAvailable Python libs: math, numpy, sympy, itertools, collections, mpmath (dps=64).\nFamily 4 — Meta-Strategy Selector
\nBefore solving, classify the problem into a domain: NUMBER THEORY (CRT, prime factorisation, modular arithmetic) / COMBINATORICS (generating functions, inclusion-exclusion, DP, bijection) / ALGEBRA (polynomial roots, Vieta's, functional equations) / GEOMETRY (coordinates, trigonometry, area ratios) / MIXED (decompose). Then commit to a specific named technique and execute.\nThe classification step matters because hard olympiad problems often disguise their domain. A sum over divisors looks algebraic but is number theory. A path count on a grid looks geometric but is combinatorics. Forcing explicit domain commitment before execution reduces mid-solution strategy drift, which is a common source of partial and incorrect answers.
\nYou are an olympiad problem-solving strategist. Before solving, CLASSIFY the problem\nand select the optimal technique:\n\nCLASSIFICATION (choose one):\n A. NUMBER THEORY → use modular arithmetic, Chinese Remainder Theorem, prime factorisation.\n B. COMBINATORICS → use generating functions, inclusion-exclusion, DP, bijection.\n C. ALGEBRA → use polynomial roots, Vieta's, inequalities, functional equations.\n D. GEOMETRY → use coordinates, trigonometry, area ratios, projective methods.\n E. MIXED / OTHER → decompose into sub-problems and apply relevant techniques.\n\nSOLUTION PROTOCOL:\n 1. State the classification and your chosen technique with one-line justification.\n 2. Set up the mathematical framework precisely.\n 3. Solve step-by-step; use Python for symbolic algebra or heavy computation.\n 4. State the answer and verify it satisfies every constraint.\n\nOutput the answer ONLY as: \\boxed{N} (nothing after this line)\n\nAvailable Python libs: math, numpy, sympy, itertools, collections, mpmath (dps=64).\nFamily 5 — Decomposition + Subgoal Chaining
\nBreak the problem into 3–5 precise mathematical subgoals. Solve each in sequence, labelling results [SG1 result], [SG2 result], … Combine them explicitly in a synthesis step showing every substitution. End with a Python end-to-end check that reproduces the final answer from inputs.\nThis family is strongest on multi-step algebra and geometry problems where a long direct solution is error-prone but a chain of smaller lemmas is tractable. The labelled subgoal structure keeps long solutions coherent and makes the synthesis auditable. The end-to-end Python check catches synthesis errors — places where two correct subgoals combine incorrectly.
\nYou are a mathematical olympiad expert who solves hard problems by decomposing them\ninto a chain of simpler subgoals.\n\nPROCESS:\nSUBGOAL DECOMPOSITION\n • Break the problem into 3–5 independent or sequential subgoals.\n • Write each subgoal as a precise mathematical statement.\n\nSUBGOAL SOLVING\n • Solve each subgoal in order. Label results clearly: [SG1 result], [SG2 result], …\n • Use Python for any subgoal requiring non-trivial calculation.\n\nSYNTHESIS\n • Combine subgoal results to obtain the final answer.\n • Show the synthesis step explicitly.\n\nFINAL VERIFICATION\n • Run a Python end-to-end check: given all inputs, does your code reproduce the answer?\n\nOutput the answer ONLY as: \\boxed{N} (nothing after this line)\n\nAvailable Python libs: math, numpy, sympy, itertools, collections, mpmath (dps=64).\nThe Formula
\nAfter all 15 attempts complete (or early stop fires), every candidate answer gets a score:
\nscore(a) = freq_score(a) + 2.0 × entropy_score(a)\n × cross_prompt_multiplier(a)\n × suspect_penalty(a)\n\nfreq_score(a) = count(a) / total_valid_attempts\nentropy_score(a) = avg_inv_entropy(a) / global_sum_inv_entropy\nWhat Each Piece Does
\nFrequency score is plain majority voting. It is the floor, not the ceiling — on its own it is vulnerable to correlated errors across attempts of the same family.\nInverse entropy score measures how confident the model was token-by-token when producing a given answer. Low entropy near the final tokens means the model's probability mass was concentrated — it wasn't hedging. This signal, weighted at 2×, distinguishes \"I calculated this carefully and I'm sure\" from \"this seems plausible.\"\nCross-prompt family bonus (1.5×) fires when two or more structurally different families agree. An answer reached independently by both the Computational-First family (via brute-force pattern discovery) and the Adversarial-Critic family (via constraint checking after solving) is worth far more than three attempts of the same family agreeing. This single component is the most important addition over plain entropy voting.\nSuspect-answer penalty (0.3×) suppresses the most common wrong-answer failure modes. Values in {0, 1, 2, 99999, 100000} and non-zero multiples of 1000 appear disproportionately as default outputs from failed computations. They are rarely correct olympiad answers.
\nEarly Stopping
\nWhen 5 or more attempts agree on the same answer at ≥60% vote share, early stop fires, the threading stop event is set, and pending futures are cancelled. This saves 30–120 seconds on easy problems — compounding over 50 problems, it buys meaningful additional budget for the hard ones.
\nThe Repair Loop
\nOn the rare problems where all 15 attempts produce zero valid \\boxed{N}, three repair attempts fire with a different prompt that mandates Python brute-force as the first action, prints progress during search, and explicitly lists six failure modes to avoid. Repair attempts get prompt_id = -1 and are excluded from the cross-prompt family bonus so they cannot inflate a weak consensus.
\nYou are solving a competition mathematics problem. Previous automated attempts all FAILED\nto produce a valid answer. You must now try a completely different approach.\n\nSTRICT RULES:\n 1. Do NOT repeat any approach that requires long symbolic manipulation without Python.\n 2. Your FIRST action must be a Python brute-force or exhaustive search, even if the\n search space seems large — use heuristics to limit it if needed.\n 3. Print candidate answers during the search so partial progress is visible.\n 4. After getting a numeric candidate, verify it independently with a second Python script.\n 5. Your answer MUST be a non-negative integer in [0, 99999].\n\nCOMMON FAILURE REASONS TO AVOID:\n • Off-by-one in loop bounds (use inclusive endpoints carefully).\n • Misreading 'mod M' — the answer is the remainder, not the quotient.\n • Forgetting to take the final mod if the problem asks for answer mod N.\n • Treating a combinatorial count as a geometric/algebraic one or vice versa.\n\nOutput the answer ONLY as: \\boxed{N} (nothing after this line)\n\nAvailable Python libs: math, numpy, sympy, itertools, collections, mpmath (dps=64).\n
I ran far more than 10 experiments across this competition. What I'm sharing here is a representative set of notebook versions that were similar enough to each other — in terms of prompt design, voting strategy, or workflow architecture — to make a clean comparison, plus one entirely separate direction I explored: fine-tuning the model itself. Each attempt isolates a different question about what actually drives performance.\nThe headline across everything: the base model's reasoning capability is already strong. The question is never how to teach it more — it's how to sample from it more effectively.
\n
Version 19 (−30 from Version 73): \nThe most important data point. No tool calls — same model, same voting, same 15 attempts. Without Python access the model cannot verify arithmetic, enumerate small cases to spot patterns, or validate conjectures before committing. Tool-integrated reasoning is not an optimisation. It is the difference between this being a hard problem and an impossible one.
\nVersion 91 (−25 from Version 73):\n Three things broke simultaneously. The openai_harmony library was dropped in favour of raw client.completions.create PythonTool and ComputationSandbox were still there, but without the typed message protocol the model had no reliable way to signal completion vs. intermediate reasoning, so many attempts produced no valid \\boxed{N}. Minority answer protection actively inverted consensus logic. The Contrarian prompt steered the model away from correct approaches on every attempt it was used. All three failed together.
\nVersion 39 (−14 from Version 73): \nReplaced the Harmony/vLLM-server stack with vLLM's direct Python API, single-pass generation via llm.generate(), and post-hoc code extraction via regex. The multi-turn feedback loop disappeared entirely the model could write code but couldn't see its output and revise. That loop is what makes TIR valuable. Without it, you're just generating text that happens to contain code.
\nVersion 49 (−9 from Version73): \n50 total attempts, per-strategy temperatures, min_p raised to 0.06. With 16 workers and a 300–900s budget, most attempts after the 16th are queued and cancelled. More attempts is not better when the hardware can't actually run them. min_p=0.06 also cut lower-probability tokens that creative mathematical reasoning often needs.
\nVersion 92 (−6 from Version73):\nAdded a 6th prompt family pushing to 18 total tasks last 3 frequently didn't complete on 16 workers. The verifier pass used its 1024-token budget for reasoning rather than the required CONFIRMED/CORRECTED output. It fired on nearly every problem and returned no usable signal.
\nVersion 95 (−5 from Version73):\nLaplace smoothing made a strong singleton indistinguishable from a weak majority. Verification bonus penalised iterative debugging attempts exactly the ones most likely to solve hard problems. Adding 42, 360, 720 to the suspect set incorrectly penalised valid olympiad answers.\nVersion 98 (−4 from Version73): Reduced families from 5 to 2. The cross-prompt bonus needs structurally different families to fire — with 2 it barely fires at all. Mixed temperature 0.5/0.7 produced less diverse candidates.
\nVersion 74 (−3 from Version73): \nTemperature schedule [0.6, 1.0, 1.3] introduced low-temperature attempts that produced low-entropy wrong answers. The inverse-entropy voting mechanism amplified exactly those low entropy looked like confidence. Answer clustering with tolerance=2 merged distinct correct and incorrect nearby answers.
\nVersion 96 (−2 from Version73): \nTightened prompts consumed 300–500 extra context tokens per attempt. On long problems attempts hit the context limit before completing. The extended adversarial checklist caused the model to revisit and invalidate correct intermediate results.
\nAfter seeing where each version landed on the private leaderboard, three clear patterns emerged:\nTier 1 . Architectural breaks (−14 to −30): Removing tool access or breaking the TIR protocol. These are not tuning mistakes they change what the system fundamentally can do.
\nTier 2. Consensus interference (−5 to −25): Minority protection, Bayesian smoothing, verification bonuses. All share the same flaw they try to second-guess what the voting pool is already doing correctly.
\nTier 3. Resource competition (−2 to −9): Tighter prompts consuming context, more attempts than workers can handle, temperatures that reduce diversity.
\nAlongside the inference experiments, I explored whether fine-tuning GPT-OSS-120B directly on competition-style problems could push the score further.\nTraining setup: LoRA fine-tuning via Unsloth's FastLanguageModel with 4-bit quantization, using the nvidia/OpenMathReasoning TIR split (CoT files never downloaded to save RAM). Problems were filtered to pass_rate_72b_tir between 0.05 and 0.35 hard enough to learn from, solvable enough to have reliable ground truth with answers in [0, 99999] to match the competition format. Final sample: 3,500 examples. LoRA rank 16, alpha 32, targeting all projection modules, 1 epoch at learning rate 1e-4. The system prompt used during training was the same Computational-First prompt from Version 73 so the adapter would be aligned to the same reasoning style used at inference.\nInference with adapter: The trained LoRA adapter was loaded alongside the base model via vLLM's --enable-lora flag, using the same Harmony TIR loop and entropy-weighted voting as Version 73.
\nWhat happened: The fine-tuned version performed significantly worse on the unseen competition set. The likely reasons:
\nThis result settled the question definitively. The right investment was in the inference pipeline, not the model weights. Version 73 represents the endpoint of that inference-first approach.
\nBeyond GPT-OSS-120B, I explored whether smaller or differently-architected models could compete on this benchmark. Two were tested in full end-to-end pipelines.
\nAn 8B reasoning model with a native thinking protocol. The pipeline used client.chat.completions.create directly no Harmony encoding library with a five step problem solving prompt (UNDERSTAND → EXPLORE → PLAN → EXECUTE → VERIFY), 8 parallel attempts at temperature 1.0, entropy-weighted voting, and a Jupyter sandbox pool for Python execution. Answer extraction handled both \\boxed{N} and N formats. The key difference from Version 73 was the absence of the structured Harmony TIR protocol — tool calls were managed by extracting `python blocks from assistant messages and feeding output back as a new user turn, which is architecturally similar to Version 39.\nThe model performed significantly worse on the competition set. The 8B parameter count is the likely ceiling — at IMO difficulty, multi-step symbolic reasoning and pattern generalisation require a model with substantially more capacity. The post-hoc code-extraction approach also reintroduced the same feedback loop problem that hurt Version 39: the model writes code in one pass without seeing whether it succeeded before committing to a conclusion.
A 30B reasoning model with a built-in thinking protocol (… tags, enforced via chat_template_kwargs={\"thinking\": True}). The pipeline was more sophisticated than the DeepSeek attempt: context pruning when approaching the 65K limit, a repair prompt that fires when all primary attempts fail, a consecutive-no-progress early exit after 3 turns without code or answer, and min_tokens=2048 on the first turn to force a full reasoning chain. 6 parallel attempts, temperature 0.6, suspect-answer penalty (0.5× weight for {0, 1, 2, 99999, 100000}).\nThis version performed better than the DeepSeek attempt but still fell well short of Version 73. The 30B parameter count is a meaningful step up but still substantially below GPT-OSS-120B's 120B. More significantly, the pipeline lacked the openai_harmony typed message protocol — without it, the model's thinking-channel output and final-answer signal were harder to parse reliably, and the conversation management relied on the same manual text-appending approach that caused problems in Version 91.
\nBoth model experiments pointed to the same conclusion: at IMO difficulty, parameter count matters significantly, and the structured TIR protocol provided by openai_harmony is not just a convenience it is load-bearing infrastructure for reliable multi-turn tool interaction. Neither a capable smaller model nor a more sophisticated prompt design could compensate for those two factors.
\n\nEvery pipeline that lost either Harmony TIR or tool calls scored 24 or below.
The central lesson of this entire experiment is deceptively simple: the model already knows how to solve mathematics. The job of the pipeline is to ask it the same question five different ways, then trust the answer they agree on.\nVersion 73 worked not because it had a smarter prompt or a more sophisticated voting mechanism than the versions that came before it — it worked because it had the right combination of three things that genuinely cannot be substituted for each other:\nTool-integrated reasoning with a structured protocol. The 30-point gap between Version 73 and Version 19 (pure chain-of-thought, no tools) comes from a single architectural decision. Python access lets the model verify arithmetic, enumerate small cases to spot patterns, and validate conjectures before committing. The Harmony protocol ensures the model knows when it has finished versus when it is still reasoning. Without both, the model is guessing in the dark.\nReasoning diversity over reasoning depth. Five structurally different prompt families produce uncorrelated failure modes. When two or more families independently agree on an answer — one via brute-force pattern discovery, another via constraint-checking adversarial critique — that cross-family consensus is strong evidence of correctness that simple majority voting cannot produce. Every time I reduced the number of families, or made individual prompts more elaborate and instruction-heavy, the score went down.\nRestraint in the voting layer. The voting function works because it stays close to its core: frequency plus confidence plus a cross-family bonus plus a mild penalty for suspicious values. Every elaboration I added — Bayesian smoothing, verification bonuses, minority protection, answer clustering — made things worse. The pool's job is to converge on right answers. Any mechanism that resists that convergence is harmful by construction.\nThe experiments with other models (DeepSeek-R1-0528-Qwen3-8B, Nvidia Nemotron-3-Nano-30B) and the fine-tuning attempt all reinforced the same point: at IMO difficulty, parameter count and the quality of the inference infrastructure matter more than prompt engineering. GPT-OSS-120B with a clean TIR pipeline outperformed both smaller models with more elaborate prompting and the same model with a specialised LoRA adapter.\nIf I were to continue this work, the direction I would explore is not better prompts or smarter voting — it is tree-search inference, where failed attempts inform subsequent ones within a single problem budget. That is the one architectural change that could meaningfully improve on Version 73 without risking the regressions that came from every other elaboration tried here.
\nFollowing up on the questions from @philipvonderlind, here are some additional questions to help us assess your submission:
\nLook forward to reading your responses on these.
", "rawMarkdown": "Following up on the questions from @philipvonderlind, here are some additional questions to help us assess your submission:\n\n1. **Cross-family bonus rate:** You describe the 1.5× cross-family bonus as \"the most important addition over plain entropy voting\". On your dev runs, roughly how often does this bonus actually fire (i.e. 2+ families agreeing on the same answer)? And when it does fire, do you have a feel for whether it tracks correctness, or can two families also co-correlate on the same wrong answer?\n2. **Suspect-penalty derivation:** Your suspect set is hard-coded as {0, 1, 2, 99999, 100000} and non-zero multiples of 1000, with a 0.3× penalty. Version 95 showed that adding 42, 360, 720 to this set hurt the score. How did you arrive at the original list, empirical from observed failure modes or theoretical from \"common defaults\"? And given V95's regression, are you confident the original list isn't also penalising some correct answers (e.g. round-number answers that happen to be right)?\n\nLook forward to reading your responses on these.", "replies": [ { "id": 3457794, "postDate": "2026-05-14T10:43:09.887Z", "content": "Honestly, I don't have precise statistics on how often the bonus fires across dev runs since I wasn't logging that metric explicitly. From observational inspection of printed outputs during runs, it fires reasonably often on problems where the answer is unambiguous the stronger the problem signal, the more families converge. However, your concern about co-correlation on wrong answers is valid and something I cannot confidently rule out. Two families sharing the same flawed mathematical assumption \nwould both produce the same wrong answer and incorrectly receive the bonus. I don't have controlled experiments isolating this effect.
", "rawMarkdown": "Honestly, I don't have precise statistics on how often the bonus fires across dev runs since I wasn't logging that metric explicitly. From observational inspection of printed outputs during runs, it fires reasonably often on problems where the answer is unambiguous the stronger the problem signal, the more families converge. However, your concern about co-correlation on wrong answers is valid and something I cannot confidently rule out. Two families sharing the same flawed mathematical assumption \nwould both produce the same wrong answer and incorrectly receive the bonus. I don't have controlled experiments isolating this effect." }, { "id": 3457795, "postDate": "2026-05-14T10:44:17.190Z", "content": "The original list {0, 1, 2, 99999, 100000} was derived from observed failure modes these appeared frequently as fallback or default outputs when the model failed to properly solve the problem rather than as genuine answers. The multiples of 1000 penalty was more theoretical, based on the intuition that olympiad answers are rarely round numbers.
", "rawMarkdown": "The original list {0, 1, 2, 99999, 100000} was derived from observed failure modes these appeared frequently as fallback or default outputs when the model failed to properly solve the problem rather than as genuine answers. The multiples of 1000 penalty was more theoretical, based on the intuition that olympiad answers are rarely round numbers." }, { "id": 3457797, "postDate": "2026-05-14T10:45:33.570Z", "content": "The V95 regression adding {42, 360, 720} was a clear signal that the penalty list was already at its limit. You're right to flag that the original list could also be penalising correct round number answers I cannot guarantee it isn't. The honest conclusion is that the suspect penalty is a heuristic derived from limited observations, not a principled approach, and the multiples of 1000 rule in particular is the weakest part of it.
", "rawMarkdown": "The V95 regression adding {42, 360, 720} was a clear signal that the penalty list was already at its limit. You're right to flag that the original list could also be penalising correct round number answers I cannot guarantee it isn't. The honest conclusion is that the suspect penalty is a heuristic derived from limited observations, not a principled approach, and the multiples of 1000 rule in particular is the weakest part of it." } ] }, { "id": 3457744, "postDate": "2026-05-14T09:02:45.170Z", "content": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.
\nI have a few questions regarding your work:
\nCitation: Large parts of the prompts appear to be based on one of the publicly shared Kaggle notebooks. While we encourage building on shared solutions, please make sure to cite the notebook you used as a starting point, clearly document which components were reused, and explain what you changed and why.
Ablations: Other contestants, for example @natnitarach in their submission, have shown that prompt variations can fall within the noise band of the public leaderboard. Did you account for this when interpreting the reported results of your experiments? It would be helpful to include the mean and variance across all runs, as well as the performance of your baseline, which I assume is one of the publicly shared notebooks. Reporting only a single result does not provide the full picture, especially given the high variance in this competition.
SFT: Could you explain your SFT setup in more detail? In particular, what motivated the choice to run SFT on an already RL-trained model with very strong performance, rather than using an RL method such as GRPO? It would also be valuable to share and document the SFT training code.
Looking forward to your response :)
", "rawMarkdown": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the [updated discussion post](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/689703).\n\nI have a few questions regarding your work:\n\n1. **Citation**: Large parts of the prompts appear to be based on one of the publicly shared Kaggle notebooks. While we encourage building on shared solutions, please make sure to cite the notebook you used as a starting point, clearly document which components were reused, and explain what you changed and why.\n\n2. **Ablations**: Other contestants, for example @natnitarach in [their submission](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/model-capability-dominates), have shown that prompt variations can fall within the noise band of the public leaderboard. Did you account for this when interpreting the reported results of your experiments? It would be helpful to include the mean and variance across all runs, as well as the performance of your baseline, which I assume is one of the publicly shared notebooks. Reporting only a single result does not provide the full picture, especially given the high variance in this competition.\n\n3. **SFT**: Could you explain your SFT setup in more detail? In particular, what motivated the choice to run SFT on an already RL-trained model with very strong performance, rather than using an RL method such as GRPO? It would also be valuable to share and document the SFT training code.\n\nLooking forward to your response :)\n\n\n", "replies": [ { "id": 3457779, "postDate": "2026-05-14T10:24:08.240Z", "content": "Starting point of base reference notebook is this :\nhttps://www.kaggle.com/code/nihilisticneuralnet/44-50-let-me-over-cook
\nThings which i have retained from base reference notebook are :
\n(1) vllm server strartup and shutdown logic(start_server/_wait_for_server)
\n(2) model weight preloading via multithreaded file reads
\n(3) openai compatible clinet initialization
\n(4) jupyter kernel management and sandboxpool(kernelmanager,sandbox queue pattern)
\n(5) The streaming completion loop with token_ids,logprobs and early \\boxed{} scanning
\n(6) the predict() entry point and kaggle_evaluation inference server integration
\n(7) basic entropy calculation for answer confidence weighting
\n(8) the single system prompt structure and ToolNamespaceConfig usage
\nThings which i have changes i comparison to base pipeline :
\n(1) 5 structurally distinct prompts \nreason : Diverse reasoning strategies reduce correlated failure of different problem types will get benefit fromr different framings
\n(2) 15 attempts distributed as 3 per prompt family \nreason : I want to ensure that each reasoning strategy gets multiple independent samples
\n(3)Entropy weighted voting plus cross prompt consistency bonus and suspect answer penalty \nreason : i want to aad the signal from interfamily agreement to penalises suspiciously common wrong answers like 0,1,round multiples of 1000
\n(4)Fires at five matching answers and requires >=60 percent of majority votes\nreason : To prevent premature stopping on coincidental agreement
\n(5)3 dedicated repair attempts with a seperate repair_prompt focused on brute force python first recovery \nreason : to recovera answers in edge cases where all noraml attempts fails to produce s \\boxed{}
\n(6)import block plus assertion based validation like sympy.isprime(7), mpmath.sqrt(2)\nreason : to catch silent library failures early rather than mid solve
\n(7)Each result records prompt_id voting repotr which family agreed\nreason : it enables debugging and cross family bonus scoring
\n(8) SolverCinfig with extended fields for new features\nreason : It helps for cleaner seperation of tunable hyperparameters
\n(9)print_result() with vote breakdown per answer entropy and per family answer mapping \nreason : it will easier for post-hoc analysis of failure modes
", "rawMarkdown": "Starting point of base reference notebook is this :\nhttps://www.kaggle.com/code/nihilisticneuralnet/44-50-let-me-over-cook\n\nThings which i have retained from base reference notebook are :\n\n(1) vllm server strartup and shutdown logic(start_server/_wait_for_server)\n\n(2) model weight preloading via multithreaded file reads\n\n(3) openai compatible clinet initialization\n\n(4) jupyter kernel management and sandboxpool(kernelmanager,sandbox queue pattern)\n\n(5) The streaming completion loop with token_ids,logprobs and early \\boxed{} scanning\n\n(6) the predict() entry point and kaggle_evaluation inference server integration\n\n(7) basic entropy calculation for answer confidence weighting\n\n(8) the single system prompt structure and ToolNamespaceConfig usage\n\n\nThings which i have changes i comparison to base pipeline :\n\n(1) 5 structurally distinct prompts \nreason : Diverse reasoning strategies reduce correlated failure of different problem types will get benefit fromr different framings\n\n(2) 15 attempts distributed as 3 per prompt family \nreason : I want to ensure that each reasoning strategy gets multiple independent samples\n\n(3)Entropy weighted voting plus cross prompt consistency bonus and suspect answer penalty \nreason : i want to aad the signal from interfamily agreement to penalises suspiciously common wrong answers like 0,1,round multiples of 1000\n\n(4)Fires at five matching answers and requires >=60 percent of majority votes\nreason : To prevent premature stopping on coincidental agreement\n\n(5)3 dedicated repair attempts with a seperate repair_prompt focused on brute force python first recovery \nreason : to recovera answers in edge cases where all noraml attempts fails to produce s \\boxed{}\n\n(6)import block plus assertion based validation like sympy.isprime(7), mpmath.sqrt(2)\nreason : to catch silent library failures early rather than mid solve\n\n(7)Each result records prompt_id voting repotr which family agreed\nreason : it enables debugging and cross family bonus scoring\n\n(8) SolverCinfig with extended fields for new features\nreason : It helps for cleaner seperation of tunable hyperparameters\n\n(9)print_result() with vote breakdown per answer entropy and per family answer mapping \nreason : it will easier for post-hoc analysis of failure modes\n\n" }, { "id": 3457780, "postDate": "2026-05-14T10:25:57.903Z", "content": "In this competition I made around 68–69 submissions, but most of that budget was spent exploring different directions rather than running the same solution multiple times to compute mean and variance. My focus was on understanding the impact of different prompts and prompt combinations on unseen problems, experimenting with context window sizes, and testing different answer selection strategies from pure majority voting to hybrid approaches combining entropy weighting with majority vote. So I did not produce the kind of repeated run statistics needed to properly measure variance for any single configuration.
", "rawMarkdown": "In this competition I made around 68–69 submissions, but most of that budget was spent exploring different directions rather than running the same solution multiple times to compute mean and variance. My focus was on understanding the impact of different prompts and prompt combinations on unseen problems, experimenting with context window sizes, and testing different answer selection strategies from pure majority voting to hybrid approaches combining entropy weighting with majority vote. So I did not produce the kind of repeated run statistics needed to properly measure variance for any single configuration." }, { "id": 3457783, "postDate": "2026-05-14T10:35:12.977Z", "content": "Training at 65k context length with GRPO or any RL method on a 120B model is computationally very expensive, which was the primary reason for sticking with SFT.
\nModel choice: unsloth/gpt-oss-120b was chosen as the base for fine-tuning over the Tinker variant, as GPT-OSS 120B already demonstrated stronger performance on the unseen evaluation set even without any fine-tuning.
\nDataset: The TIR split of nvidia/OpenMathReasoning was filtered to pass_rate_72b_tir ∈ [0.05, 0.35] and answers in [0, 99999] to match AIMO-3 format. 3,500 rows were randomly sampled — each row is an independent (problem, trajectory) pair from the TIR split.\nSingle prompt alignment: Rather than using all 5 prompt variants at training time, every sample was formatted with the same Computational-First system prompt used at inference. This ensures train/test distribution alignment and allows evaluating the fine-tuned model's performance under this single prompt directly, without needing to aggregate across multiple prompt variants.
\nLoRA setup: r=16, alpha=32, targeting all attention and MLP projection layers on GPT-OSS 120B (4-bit), 1 epoch, 65k sequence length, response-only masking via auto-detected Harmony chat template boundaries.
", "rawMarkdown": "Training at 65k context length with GRPO or any RL method on a 120B model is computationally very expensive, which was the primary reason for sticking with SFT.\n\nModel choice: unsloth/gpt-oss-120b was chosen as the base for fine-tuning over the Tinker variant, as GPT-OSS 120B already demonstrated stronger performance on the unseen evaluation set even without any fine-tuning.\n\nDataset: The TIR split of nvidia/OpenMathReasoning was filtered to pass_rate_72b_tir ∈ [0.05, 0.35] and answers in [0, 99999] to match AIMO-3 format. 3,500 rows were randomly sampled — each row is an independent (problem, trajectory) pair from the TIR split.\nSingle prompt alignment: Rather than using all 5 prompt variants at training time, every sample was formatted with the same Computational-First system prompt used at inference. This ensures train/test distribution alignment and allows evaluating the fine-tuned model's performance under this single prompt directly, without needing to aggregate across multiple prompt variants.\n\nLoRA setup: r=16, alpha=32, targeting all attention and MLP projection layers on GPT-OSS 120B (4-bit), 1 epoch, 65k sequence length, response-only masking via auto-detected Harmony chat template boundaries." } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T04:21:44Z", "competition_id": 118448, "competition_slug": "ai-mathematical-olympiad-progress-prize-3", "team_id": 14848832, "solution_url": "/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/solving-imo-problems-with-five-minds" }