This solution is built on top of Parthenos's and Andreas' excellent public notebook, which I used as my starting point. The core infrastructure GPT-OSS 120B served via vLLM, the persistent Jupyter sandbox pool, the multi-attempt parallel inference loop, and the entropy-based scoring concept all originate from Parthenos's design. I credit that work as the essential foundation.
\nMy contribution is a set of 7 targeted modifications that improved accuracy and robustness. This write-up documents both what I inherited and what I changed, along with the reasoning behind each decision.
\nThis solution is a pure inference optimization attempt. No training data, no fine-tuning. The task is to get a frontier-scale language model to correctly solve 50 IMO-level mathematical problems within a fixed time budget, with answers in the range [0, 99999].
\nThe pipeline at a high level:
\n\\boxed{}Total inference time: ~4–5 hours for 50 problems on 1× H100 80GB.
\nThe most impactful single change. Parthenos's prompt was a solid 5-step protocol (UNDERSTAND / EXPLORE / PLAN / EXECUTE / VERIFY) which is a significant improvement on Andreas' notebook, but it had no explicit constraint on the answer range.
\nAIMO3 problems frequently require modular reduction. The raw answer might be a 20-digit number, and the problem asks for the last 5 digits. Without an explicit reminder, the model would often solve the problem correctly and then box the raw large integer, scoring zero.
\nSystem Prompt:
\nYou are an elite mathematical problem solver competing at the International Mathematical Olympiad (IMO) level. Your sole objective is to produce the correct non-negative integer answer.\n\n# Mandatory answer constraints:\n\nThe final answer MUST be a non-negative integer in [0, 99999]. If your raw answer exceeds 99999, the problem is asking for something modular (e.g., the last five digits, or the answer mod 10^5). Re-read the problem statement carefully before boxing your answer.\n\n# Problem-Solving Protocol:\n\n1. PARSE: Identify exactly what quantity is being asked for. Write it down explicitly.\n2. EXPLORE: Consider at least two distinct solution strategies before committing. Note relevant theorems, constraints, and symmetries.\n3. PLAN: Choose the most rigorous approach and outline key lemmas.\n4. EXECUTE: Work step by step. Show all algebraic and logical steps clearly.\n5. VERIFY: Before writing \\boxed{}, independently re-derive or numerically confirm the answer. Check it satisfies every constraint. If verification fails, restart.\n\n# Common pitfalls to avoid:\n\n* Off-by-one errors in combinatorics and indexing\n* Forgetting the modular reduction when the answer would otherwise exceed 99999\n* Over-counting or under-counting in enumeration problems\n* Sign errors and missed edge cases in algebraic manipulation\n* Trusting an unverified intermediate result\n\n# Verification checklist (complete before \\boxed{}):\n\n* Does the answer satisfy all constraints stated in the problem?\n* Does it agree with small/simple special cases?\n* Is it in [0, 99999]?\n* Have you confirmed it via an independent route (code, substitution, or counting argument)?\n\n# Output format:\n\nPlace your final numerical answer inside \\boxed{}, e.g., \\boxed{42}.\nNever place an expression, fraction, or inequality inside \\boxed{} — integers only.\n\nThink step-by-step. Quality of reasoning is paramount.\nWhat I added:
\n# Mandatory answer constraints:\nThe final answer MUST be a non-negative integer in [0, 99999].\nIf your raw answer exceeds 99999, the problem is asking for something modular\n(e.g., the last five digits, or the answer mod 10^5). Re-read the problem\nstatement carefully before boxing your answer.\nI also added a pre-boxing verification checklist consisting of 4 explicit questions the model must answer before writing \\boxed{}:
And a common pitfalls list: off-by-one errors, forgotten modular reduction, over/under-counting, sign errors, trusting unverified intermediate results.
\nThis was the highest-leverage change. It addressed a structural failure mode rather than a marginal improvement.
\nParthenos's preference_prompt was a long multi-section description (~200 tokens) appended to the user turn for every problem. I condensed it to a single line:
Preference prompt:
\nAvailable libraries: math, numpy (numerical arrays/linear-algebra), sympy (exact symbolic computation, number theory, polynomial ops), mpmath (arbitrary-precision arithmetic, set to 64 decimal places), itertools, collections.\n\nUse sympy for exact answers; numpy for large-scale numerics; combine both to derive symbolically and verify numerically.\nThis saves ~200 context tokens per attempt which is small in isolation, but it compounds across 8 attempts × 128 turns × 50 problems.
\nBoth my version and base notebook uses logprob entropy to measure how \"confident\" a completion was, for use in the ensemble. But there's an important difference in what entropy is measured over.
\nParthenos: averages entropy over the full token stream.
\nMine: averages entropy only over the last 256 tokens (logprob_tail = 256).
The reasoning: early in a completion, the model is exploring considering multiple approaches, writing \"let me try…\", reconsidering. This is high-entropy by design and doesn't tell you anything about how confident the model is in its final answer. The last 256 tokens are where the model commits: writing the final derivation, boxing the answer. Entropy here is a much cleaner signal.
\nBase _scan_for_answer() had two patterns:
\\boxed{N} — the intended format\"final answer is N\" — a prose fallbackThe prose fallback sounds reasonable, but it fires constantly on intermediate working steps. Phrases like \"the final answer is 42 for the base case, but let me now generalize…\" appear throughout the model's reasoning. During streaming, this committed to wrong answers early.
\nI removed the prose fallback entirely, and restricted extraction to \\boxed{} only, with two robustness improvements:
\\boxed{42,000} → 42000\\boxed{ 42 } → 42\\boxed{} occurrence in the window (most likely to be the definitive answer)This is where my approach differs most meaningfully from Parthenos's.
\nBase _select_answer():
weight = 1.0 / max(entropy, 1e-9)\nanswer_weights[answer] += weight # accumulates 1/entropy per vote\nscore = total_weight # pure entropy sum\nThis is pure entropy weighting. Vote count is tracked and displayed but never used in the score. A single attempt with very low entropy (high confidence) can override a 5-1 vote majority if the confident outlier has entropy near zero.
\nMy _select_answer():
vote_share = votes / total_votes\nconfidence = 1.0 / max(median_entropy, 1e-9)\nconsensus = 1.0 if (votes == max_votes and confidence > 1.0) else 0.0\nscore = vote_share * 2.0 + confidence * 0.3 + consensus * 0.5\nVote share dominates (weight 2.0). Entropy is a tiebreaker (weight 0.3). The consensus bonus (0.5) nudges clear winners that are also confident.
\nThe practical implication: Parthenos's method can be hijacked by a single very confident but wrong attempt. In the voting-ensemble of my solution notebook however, it will be much harder for a confident attempt to overturn the final answer selection. You need both majority votes and reasonable entropy to win. This matters specifically on hard problems where one attempt finds a clever shortcut (low entropy, confident) but gets the wrong answer, while the other 5 struggle through the full derivation (higher entropy) and arrive correctly.
\nA concrete example from my run — the 500×500 rectangles problem:
\n| Answer | \nVotes | \nEntropy | \nParthenos score | \nMy score | \n
|---|---|---|---|---|
| 520 | \n4 | \n0.996 | \n4.016 | \n1.944 | \n
| 519 | \n2 | \n0.928 | \n2.155 | \n0.895 | \n
| 706 | \n1 | \n0.696 | \n1.437 | \n0.717 | \n
Both methods select 520 here. But consider a hypothetical where 706 had entropy 0.05: Parthenos's score would be 1/0.05 = 20.0, overruling the 4-vote majority. Mine would be 0.143 × 2.0 + 20.0 × 0.3 = 6.3 vs 520's 1.944 + 0.3 + 0.5 = 2.744. The confident outlier would still win under my formula in this extreme case, but the threshold for an outlier to override a 4-1 majority is much higher.
\nMinor change: when computing per-answer confidence, I use median entropy across that answer's attempts rather than mean. This makes the score robust to a single outlier attempt that happened to produce very low entropy on a fluke.
\nParthenos's early stopping: halt when 4 attempts give the same answer (early_stop = 4).
My addition: also halt when the current leader's vote margin cannot be overturned even if every remaining attempt votes for second place.
\ndef _can_remaining_change_winner(self, valid_answers, attempts_completed):\n remaining = self.cfg.attempts - attempts_completed\n counts = Counter(valid_answers)\n top_two = counts.most_common(2)\n leader_count = top_two[0][1]\n second_count = top_two[1][1]\n return (second_count + remaining) > leader_count\nIf the leader has 4 votes and second place has 1 with 2 attempts remaining: 1 + 2 = 3 < 4 → stop early. This fires before the early_stop = 4 threshold on many clear-consensus problems, freeing time budget for problems where it's genuinely needed.
The biggest insight from this competition is how much prompt engineering still matters even at 120B scale. The modular arithmetic reminder alone (a single paragraph addition to the system prompt) eliminated an entire class of otherwise-correct solutions that were scoring zero. The model knew how to solve the problems; it just needed to be reminded what format the answer needed to be in.
\nThe second insight is about ensemble design. Entropy-based confidence scoring sounds sophisticated, but it's fragile when used as the primary signal. Vote count is a more robust aggregator across diverse reasoning paths where entropy is best used as a tiebreaker, not as the main criterion.
\n| Item | \nDetail | \n
|---|---|
| GPU | \n1× H100 80GB (Kaggle competition hardware) | \n
| Model | \nGPT-OSS 120B, FP8 KV-cache, 65,536 context | \n
| Attempts per problem | \n8 (parallel, 16 Jupyter kernels) | \n
| Avg time per problem | \n~300–400 seconds | \n
| Total runtime (50 problems) | \n~4–5 hours | \n
| Training | \nNone | \n
Thanks to Parthenos and Andreas' for sharing the base notebook openly, to the AIMO competition organisers, and to the Kaggle community for a great competition.
", "comments": [ { "id": 3463023, "author_name": "Fff343", "author_url": "", "post_date": "2026-05-25T16:39:09.417000", "content": "Why notebook shows private score as 35 in place of 44
", "votes": 1, "replies": [ { "id": 3463127, "author_name": "varianceofx", "author_url": "", "post_date": "2026-05-26T00:46:19.463000", "content": "Apologies, that was version 12. I've changed the link to version 10
", "votes": 0, "replies": [ { "id": 3463275, "author_name": "Fff343", "author_url": "", "post_date": "2026-05-26T13:39:54.023000", "content": "Thanks but how u got two private score when we can submit one version only i am not doubting just curiosity
", "votes": 0, "replies": [] }, { "id": 3463293, "author_name": "varianceofx", "author_url": "", "post_date": "2026-05-26T14:24:01.360000", "content": "The versions refer to different iterations of the same notebook during experimentation. Version 10 was the one selected for final submission rerun with private LB score (44), while later versions (score 35) are public scores that were just my other attempts.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3462749, "author_name": "@kaggleqrdl", "author_url": "", "post_date": "2026-05-24T19:03:12.110000", "content": "Nice! Looks like you made some serious improvements. Well done.
", "votes": 0, "replies": [] }, { "id": 3463694, "author_name": "Durga Kumari", "author_url": "", "post_date": "2026-05-27T17:33:56.187000", "content": "Excellent write-up. The vote dominant ensemble idea and the modular answer reminder in the system prompt are especially insightful.
", "votes": 0, "replies": [] } ], "comment_count": 6, "messages": [ { "id": 3462438, "postDate": "2026-05-23T09:31:05.967Z", "content": "This solution is built on top of Parthenos's and Andreas' excellent public notebook, which I used as my starting point. The core infrastructure GPT-OSS 120B served via vLLM, the persistent Jupyter sandbox pool, the multi-attempt parallel inference loop, and the entropy-based scoring concept all originate from Parthenos's design. I credit that work as the essential foundation.
\nMy contribution is a set of 7 targeted modifications that improved accuracy and robustness. This write-up documents both what I inherited and what I changed, along with the reasoning behind each decision.
\nThis solution is a pure inference optimization attempt. No training data, no fine-tuning. The task is to get a frontier-scale language model to correctly solve 50 IMO-level mathematical problems within a fixed time budget, with answers in the range [0, 99999].
\nThe pipeline at a high level:
\n\\boxed{}Total inference time: ~4–5 hours for 50 problems on 1× H100 80GB.
\nThe most impactful single change. Parthenos's prompt was a solid 5-step protocol (UNDERSTAND / EXPLORE / PLAN / EXECUTE / VERIFY) which is a significant improvement on Andreas' notebook, but it had no explicit constraint on the answer range.
\nAIMO3 problems frequently require modular reduction. The raw answer might be a 20-digit number, and the problem asks for the last 5 digits. Without an explicit reminder, the model would often solve the problem correctly and then box the raw large integer, scoring zero.
\nSystem Prompt:
\nYou are an elite mathematical problem solver competing at the International Mathematical Olympiad (IMO) level. Your sole objective is to produce the correct non-negative integer answer.\n\n# Mandatory answer constraints:\n\nThe final answer MUST be a non-negative integer in [0, 99999]. If your raw answer exceeds 99999, the problem is asking for something modular (e.g., the last five digits, or the answer mod 10^5). Re-read the problem statement carefully before boxing your answer.\n\n# Problem-Solving Protocol:\n\n1. PARSE: Identify exactly what quantity is being asked for. Write it down explicitly.\n2. EXPLORE: Consider at least two distinct solution strategies before committing. Note relevant theorems, constraints, and symmetries.\n3. PLAN: Choose the most rigorous approach and outline key lemmas.\n4. EXECUTE: Work step by step. Show all algebraic and logical steps clearly.\n5. VERIFY: Before writing \\boxed{}, independently re-derive or numerically confirm the answer. Check it satisfies every constraint. If verification fails, restart.\n\n# Common pitfalls to avoid:\n\n* Off-by-one errors in combinatorics and indexing\n* Forgetting the modular reduction when the answer would otherwise exceed 99999\n* Over-counting or under-counting in enumeration problems\n* Sign errors and missed edge cases in algebraic manipulation\n* Trusting an unverified intermediate result\n\n# Verification checklist (complete before \\boxed{}):\n\n* Does the answer satisfy all constraints stated in the problem?\n* Does it agree with small/simple special cases?\n* Is it in [0, 99999]?\n* Have you confirmed it via an independent route (code, substitution, or counting argument)?\n\n# Output format:\n\nPlace your final numerical answer inside \\boxed{}, e.g., \\boxed{42}.\nNever place an expression, fraction, or inequality inside \\boxed{} — integers only.\n\nThink step-by-step. Quality of reasoning is paramount.\nWhat I added:
\n# Mandatory answer constraints:\nThe final answer MUST be a non-negative integer in [0, 99999].\nIf your raw answer exceeds 99999, the problem is asking for something modular\n(e.g., the last five digits, or the answer mod 10^5). Re-read the problem\nstatement carefully before boxing your answer.\nI also added a pre-boxing verification checklist consisting of 4 explicit questions the model must answer before writing \\boxed{}:
And a common pitfalls list: off-by-one errors, forgotten modular reduction, over/under-counting, sign errors, trusting unverified intermediate results.
\nThis was the highest-leverage change. It addressed a structural failure mode rather than a marginal improvement.
\nParthenos's preference_prompt was a long multi-section description (~200 tokens) appended to the user turn for every problem. I condensed it to a single line:
Preference prompt:
\nAvailable libraries: math, numpy (numerical arrays/linear-algebra), sympy (exact symbolic computation, number theory, polynomial ops), mpmath (arbitrary-precision arithmetic, set to 64 decimal places), itertools, collections.\n\nUse sympy for exact answers; numpy for large-scale numerics; combine both to derive symbolically and verify numerically.\nThis saves ~200 context tokens per attempt which is small in isolation, but it compounds across 8 attempts × 128 turns × 50 problems.
\nBoth my version and base notebook uses logprob entropy to measure how \"confident\" a completion was, for use in the ensemble. But there's an important difference in what entropy is measured over.
\nParthenos: averages entropy over the full token stream.
\nMine: averages entropy only over the last 256 tokens (logprob_tail = 256).
The reasoning: early in a completion, the model is exploring considering multiple approaches, writing \"let me try…\", reconsidering. This is high-entropy by design and doesn't tell you anything about how confident the model is in its final answer. The last 256 tokens are where the model commits: writing the final derivation, boxing the answer. Entropy here is a much cleaner signal.
\nBase _scan_for_answer() had two patterns:
\\boxed{N} — the intended format\"final answer is N\" — a prose fallbackThe prose fallback sounds reasonable, but it fires constantly on intermediate working steps. Phrases like \"the final answer is 42 for the base case, but let me now generalize…\" appear throughout the model's reasoning. During streaming, this committed to wrong answers early.
\nI removed the prose fallback entirely, and restricted extraction to \\boxed{} only, with two robustness improvements:
\\boxed{42,000} → 42000\\boxed{ 42 } → 42\\boxed{} occurrence in the window (most likely to be the definitive answer)This is where my approach differs most meaningfully from Parthenos's.
\nBase _select_answer():
weight = 1.0 / max(entropy, 1e-9)\nanswer_weights[answer] += weight # accumulates 1/entropy per vote\nscore = total_weight # pure entropy sum\nThis is pure entropy weighting. Vote count is tracked and displayed but never used in the score. A single attempt with very low entropy (high confidence) can override a 5-1 vote majority if the confident outlier has entropy near zero.
\nMy _select_answer():
vote_share = votes / total_votes\nconfidence = 1.0 / max(median_entropy, 1e-9)\nconsensus = 1.0 if (votes == max_votes and confidence > 1.0) else 0.0\nscore = vote_share * 2.0 + confidence * 0.3 + consensus * 0.5\nVote share dominates (weight 2.0). Entropy is a tiebreaker (weight 0.3). The consensus bonus (0.5) nudges clear winners that are also confident.
\nThe practical implication: Parthenos's method can be hijacked by a single very confident but wrong attempt. In the voting-ensemble of my solution notebook however, it will be much harder for a confident attempt to overturn the final answer selection. You need both majority votes and reasonable entropy to win. This matters specifically on hard problems where one attempt finds a clever shortcut (low entropy, confident) but gets the wrong answer, while the other 5 struggle through the full derivation (higher entropy) and arrive correctly.
\nA concrete example from my run — the 500×500 rectangles problem:
\n| Answer | \nVotes | \nEntropy | \nParthenos score | \nMy score | \n
|---|---|---|---|---|
| 520 | \n4 | \n0.996 | \n4.016 | \n1.944 | \n
| 519 | \n2 | \n0.928 | \n2.155 | \n0.895 | \n
| 706 | \n1 | \n0.696 | \n1.437 | \n0.717 | \n
Both methods select 520 here. But consider a hypothetical where 706 had entropy 0.05: Parthenos's score would be 1/0.05 = 20.0, overruling the 4-vote majority. Mine would be 0.143 × 2.0 + 20.0 × 0.3 = 6.3 vs 520's 1.944 + 0.3 + 0.5 = 2.744. The confident outlier would still win under my formula in this extreme case, but the threshold for an outlier to override a 4-1 majority is much higher.
\nMinor change: when computing per-answer confidence, I use median entropy across that answer's attempts rather than mean. This makes the score robust to a single outlier attempt that happened to produce very low entropy on a fluke.
\nParthenos's early stopping: halt when 4 attempts give the same answer (early_stop = 4).
My addition: also halt when the current leader's vote margin cannot be overturned even if every remaining attempt votes for second place.
\ndef _can_remaining_change_winner(self, valid_answers, attempts_completed):\n remaining = self.cfg.attempts - attempts_completed\n counts = Counter(valid_answers)\n top_two = counts.most_common(2)\n leader_count = top_two[0][1]\n second_count = top_two[1][1]\n return (second_count + remaining) > leader_count\nIf the leader has 4 votes and second place has 1 with 2 attempts remaining: 1 + 2 = 3 < 4 → stop early. This fires before the early_stop = 4 threshold on many clear-consensus problems, freeing time budget for problems where it's genuinely needed.
The biggest insight from this competition is how much prompt engineering still matters even at 120B scale. The modular arithmetic reminder alone (a single paragraph addition to the system prompt) eliminated an entire class of otherwise-correct solutions that were scoring zero. The model knew how to solve the problems; it just needed to be reminded what format the answer needed to be in.
\nThe second insight is about ensemble design. Entropy-based confidence scoring sounds sophisticated, but it's fragile when used as the primary signal. Vote count is a more robust aggregator across diverse reasoning paths where entropy is best used as a tiebreaker, not as the main criterion.
\n| Item | \nDetail | \n
|---|---|
| GPU | \n1× H100 80GB (Kaggle competition hardware) | \n
| Model | \nGPT-OSS 120B, FP8 KV-cache, 65,536 context | \n
| Attempts per problem | \n8 (parallel, 16 Jupyter kernels) | \n
| Avg time per problem | \n~300–400 seconds | \n
| Total runtime (50 problems) | \n~4–5 hours | \n
| Training | \nNone | \n
Thanks to Parthenos and Andreas' for sharing the base notebook openly, to the AIMO competition organisers, and to the Kaggle community for a great competition.
", "rawMarkdown": "## Attribution\n\nThis solution is built on top of Parthenos's and Andreas' excellent public notebook, which I used as my starting point. The core infrastructure GPT-OSS 120B served via vLLM, the persistent Jupyter sandbox pool, the multi-attempt parallel inference loop, and the entropy-based scoring concept all originate from Parthenos's design. I credit that work as the essential foundation.\n\nMy contribution is a set of 7 targeted modifications that improved accuracy and robustness. This write-up documents both what I inherited and what I changed, along with the reasoning behind each decision.\n\n---\n\n## Overview\n\nThis solution is a pure inference optimization attempt. No training data, no fine-tuning. The task is to get a frontier-scale language model to correctly solve 50 IMO-level mathematical problems within a fixed time budget, with answers in the range [0, 99999].\n\nThe pipeline at a high level:\n\n1. Serve GPT-OSS 120B locally via vLLM with FP8 KV-cache\n2. For each problem, run 8 parallel reasoning attempts across 16 persistent Jupyter kernels\n3. Each attempt streams a completion, executes Python tool calls (sympy, numpy, mpmath), and extracts the final answer from `\\boxed{}`\n4. An entropy-weighted majority vote ensemble selects the final answer\n5. A dynamic time budget allocates more time to harder problems\n\nTotal inference time: ~4–5 hours for 50 problems on 1× H100 80GB.\n\n---\n\n## What I Changed (and Why)\n\n### Improvement 1: Sharpened System Prompt\n\nThe most impactful single change. Parthenos's prompt was a solid 5-step protocol (UNDERSTAND / EXPLORE / PLAN / EXECUTE / VERIFY) which is a significant improvement on Andreas' notebook, but it had no explicit constraint on the answer range.\n\nAIMO3 problems frequently require modular reduction. The raw answer might be a 20-digit number, and the problem asks for the last 5 digits. Without an explicit reminder, the model would often solve the problem correctly and then box the raw large integer, scoring zero.\n\n**System Prompt:**\n\n```\nYou are an elite mathematical problem solver competing at the International Mathematical Olympiad (IMO) level. Your sole objective is to produce the correct non-negative integer answer.\n\n# Mandatory answer constraints:\n\nThe final answer MUST be a non-negative integer in [0, 99999]. If your raw answer exceeds 99999, the problem is asking for something modular (e.g., the last five digits, or the answer mod 10^5). Re-read the problem statement carefully before boxing your answer.\n\n# Problem-Solving Protocol:\n\n1. PARSE: Identify exactly what quantity is being asked for. Write it down explicitly.\n2. EXPLORE: Consider at least two distinct solution strategies before committing. Note relevant theorems, constraints, and symmetries.\n3. PLAN: Choose the most rigorous approach and outline key lemmas.\n4. EXECUTE: Work step by step. Show all algebraic and logical steps clearly.\n5. VERIFY: Before writing \\boxed{}, independently re-derive or numerically confirm the answer. Check it satisfies every constraint. If verification fails, restart.\n\n# Common pitfalls to avoid:\n\n* Off-by-one errors in combinatorics and indexing\n* Forgetting the modular reduction when the answer would otherwise exceed 99999\n* Over-counting or under-counting in enumeration problems\n* Sign errors and missed edge cases in algebraic manipulation\n* Trusting an unverified intermediate result\n\n# Verification checklist (complete before \\boxed{}):\n\n* Does the answer satisfy all constraints stated in the problem?\n* Does it agree with small/simple special cases?\n* Is it in [0, 99999]?\n* Have you confirmed it via an independent route (code, substitution, or counting argument)?\n\n# Output format:\n\nPlace your final numerical answer inside \\boxed{}, e.g., \\boxed{42}.\nNever place an expression, fraction, or inequality inside \\boxed{} — integers only.\n\nThink step-by-step. Quality of reasoning is paramount.\n\n```\n\n**What I added:**\n\n```\n# Mandatory answer constraints:\nThe final answer MUST be a non-negative integer in [0, 99999].\nIf your raw answer exceeds 99999, the problem is asking for something modular\n(e.g., the last five digits, or the answer mod 10^5). Re-read the problem\nstatement carefully before boxing your answer.\n```\n\nI also added a pre-boxing verification checklist consisting of 4 explicit questions the model must answer before writing `\\boxed{}`:\n- Does the answer satisfy all constraints in the problem?\n- Does it agree with simple special cases?\n- Is it in [0, 99999]?\n- Have you confirmed it via an independent route?\n\nAnd a common pitfalls list: off-by-one errors, forgotten modular reduction, over/under-counting, sign errors, trusting unverified intermediate results.\n\nThis was the highest-leverage change. It addressed a structural failure mode rather than a marginal improvement.\n\n---\n\n### Improvement 2: Condensed Library Hint\n\nParthenos's `preference_prompt` was a long multi-section description (~200 tokens) appended to the user turn for every problem. I condensed it to a single line:\n\n**Preference prompt:**\n\n```\nAvailable libraries: math, numpy (numerical arrays/linear-algebra), sympy (exact symbolic computation, number theory, polynomial ops), mpmath (arbitrary-precision arithmetic, set to 64 decimal places), itertools, collections.\n\nUse sympy for exact answers; numpy for large-scale numerics; combine both to derive symbolically and verify numerically.\n\n```\n\nThis saves ~200 context tokens per attempt which is small in isolation, but it compounds across 8 attempts × 128 turns × 50 problems.\n\n---\n\n### Improvement 3: Tail-Windowed Entropy\n\nBoth my version and base notebook uses logprob entropy to measure how \"confident\" a completion was, for use in the ensemble. But there's an important difference in *what* entropy is measured over.\n\n**Parthenos:** averages entropy over the full token stream.\n\n**Mine:** averages entropy only over the last 256 tokens (`logprob_tail = 256`).\n\nThe reasoning: early in a completion, the model is exploring considering multiple approaches, writing \"let me try...\", reconsidering. This is high-entropy by design and doesn't tell you anything about how confident the model is in its *final* answer. The last 256 tokens are where the model commits: writing the final derivation, boxing the answer. Entropy here is a much cleaner signal.\n\n---\n\n### Improvement 4: Answer Extraction\n\nBase `_scan_for_answer()` had two patterns:\n1. `\\boxed{N}` — the intended format\n2. `\"final answer is N\"` — a prose fallback\n\nThe prose fallback sounds reasonable, but it fires constantly on intermediate working steps. Phrases like \"the final answer is 42 for the base case, but let me now generalize...\" appear throughout the model's reasoning. During streaming, this committed to wrong answers early.\n\nI removed the prose fallback entirely, and restricted extraction to `\\boxed{}` only, with two robustness improvements:\n- **Comma handling**: `\\boxed{42,000}` → 42000\n- **Whitespace handling**: `\\boxed{ 42 }` → 42\n- **Last match wins**: returns the final `\\boxed{}` occurrence in the window (most likely to be the definitive answer)\n\n---\n\n### Improvement 5: Ensemble Selection\n\nThis is where my approach differs most meaningfully from Parthenos's.\n\n**Base `_select_answer()`:**\n```python\nweight = 1.0 / max(entropy, 1e-9)\nanswer_weights[answer] += weight # accumulates 1/entropy per vote\nscore = total_weight # pure entropy sum\n```\n\nThis is **pure entropy weighting**. Vote count is tracked and displayed but never used in the score. A single attempt with very low entropy (high confidence) can override a 5-1 vote majority if the confident outlier has entropy near zero.\n\n**My `_select_answer()`:**\n```python\nvote_share = votes / total_votes\nconfidence = 1.0 / max(median_entropy, 1e-9)\nconsensus = 1.0 if (votes == max_votes and confidence > 1.0) else 0.0\nscore = vote_share * 2.0 + confidence * 0.3 + consensus * 0.5\n```\n\nVote share dominates (weight 2.0). Entropy is a tiebreaker (weight 0.3). The consensus bonus (0.5) nudges clear winners that are also confident.\n\nThe practical implication: Parthenos's method can be hijacked by a single very confident but wrong attempt. In the voting-ensemble of my solution notebook however, it will be much harder for a confident attempt to overturn the final answer selection. You need both majority votes and reasonable entropy to win. This matters specifically on hard problems where one attempt finds a clever shortcut (low entropy, confident) but gets the wrong answer, while the other 5 struggle through the full derivation (higher entropy) and arrive correctly.\n\nA concrete example from my run — the 500×500 rectangles problem:\n\n| Answer | Votes | Entropy | Parthenos score | My score |\n|--------|-------|---------|-----------------|----------|\n| 520 | 4 | 0.996 | 4.016 | 1.944 |\n| 519 | 2 | 0.928 | 2.155 | 0.895 |\n| 706 | 1 | 0.696 | 1.437 | 0.717 |\n\nBoth methods select 520 here. But consider a hypothetical where 706 had entropy 0.05: Parthenos's score would be 1/0.05 = 20.0, overruling the 4-vote majority. Mine would be 0.143 × 2.0 + 20.0 × 0.3 = 6.3 vs 520's 1.944 + 0.3 + 0.5 = 2.744. The confident outlier would still win under my formula in this extreme case, but the threshold for an outlier to override a 4-1 majority is much higher.\n\n---\n\n### Improvement 6: Median Entropy (Robustness)\n\nMinor change: when computing per-answer confidence, I use median entropy across that answer's attempts rather than mean. This makes the score robust to a single outlier attempt that happened to produce very low entropy on a fluke.\n\n---\n\n### Improvement 7: Dynamic Uncatchable-Leader Early Stop\n\nParthenos's early stopping: halt when 4 attempts give the same answer (`early_stop = 4`).\n\nMy addition: also halt when the current leader's vote margin cannot be overturned even if every remaining attempt votes for second place.\n\n```python\ndef _can_remaining_change_winner(self, valid_answers, attempts_completed):\n remaining = self.cfg.attempts - attempts_completed\n counts = Counter(valid_answers)\n top_two = counts.most_common(2)\n leader_count = top_two[0][1]\n second_count = top_two[1][1]\n return (second_count + remaining) > leader_count\n```\n\nIf the leader has 4 votes and second place has 1 with 2 attempts remaining: `1 + 2 = 3 < 4` → stop early. This fires before the `early_stop = 4` threshold on many clear-consensus problems, freeing time budget for problems where it's genuinely needed.\n\n---\n\n## What Didn't Work / What I Didn't Change\n\n- **Temperature and min_p**: left unchanged at 1.0 and 0.02. Tried lower temperatures briefly but diversity suffers.\n- **More attempts**: going beyond 8 hits diminishing returns against the time budget.\n- **Fine-tuning**: no training was performed. The base GPT-OSS 120B was strong enough that inference engineering yielded more gain per hour than fine-tuning in the available time.\n- **Multiple system prompts**: tried running different system prompts across attempts (e.g., one \"algebraic\" and one \"computational\" persona). No measurable improvement, added complexity.\n\n---\n\n## What I Learned\n\nThe biggest insight from this competition is how much prompt engineering still matters even at 120B scale. The modular arithmetic reminder alone (a single paragraph addition to the system prompt) eliminated an entire class of otherwise-correct solutions that were scoring zero. The model knew how to solve the problems; it just needed to be reminded what format the answer needed to be in.\n\nThe second insight is about ensemble design. Entropy-based confidence scoring sounds sophisticated, but it's fragile when used as the primary signal. Vote count is a more robust aggregator across diverse reasoning paths where entropy is best used as a tiebreaker, not as the main criterion.\n\n---\n\n## Hardware & Runtime\n\n| Item | Detail |\n|------|--------|\n| GPU | 1× H100 80GB (Kaggle competition hardware) |\n| Model | GPT-OSS 120B, FP8 KV-cache, 65,536 context |\n| Attempts per problem | 8 (parallel, 16 Jupyter kernels) |\n| Avg time per problem | ~300–400 seconds |\n| Total runtime (50 problems) | ~4–5 hours |\n| Training | None |\n\n---\n\n## References\n\n- Solution notebook: https://www.kaggle.com/code/varianceofx/let-me-improve-this-cooking?scriptVersionId=306081269\n- Adapted notebook: https://huggingface.co/varianceofx/aimo3-2nd-place-solver-adapted\n- Andreas' public base notebook: https://www.kaggle.com/code/andreasbis/aimo-3-gpt-oss-120b-with-tools\n- Parthenos's public base notebook: https://www.kaggle.com/code/nihilisticneuralnet/44-50-let-me-over-cook\n- vLLM: Kwon et al. (2023), PagedAttention, SOSP 2023\n- SymPy: exact symbolic computation: https://sympy.org\n- mpmath: arbitrary-precision arithmetic (64 d.p.): https://mpmath.org\n\n---\n\n*Thanks to Parthenos and Andreas' for sharing the base notebook openly, to the AIMO competition organisers, and to the Kaggle community for a great competition.*\n", "votes": 15 }, { "id": 3463023, "postDate": "2026-05-25T16:39:09.417Z", "content": "Why notebook shows private score as 35 in place of 44
", "rawMarkdown": "Why notebook shows private score as 35 in place of 44", "votes": 1, "replies": [ { "id": 3463127, "postDate": "2026-05-26T00:46:19.463Z", "content": "Apologies, that was version 12. I've changed the link to version 10
", "rawMarkdown": "Apologies, that was version 12. I've changed the link to version 10", "replies": [ { "id": 3463275, "postDate": "2026-05-26T13:39:54.023Z", "content": "Thanks but how u got two private score when we can submit one version only i am not doubting just curiosity
", "rawMarkdown": "Thanks but how u got two private score when we can submit one version only i am not doubting just curiosity " }, { "id": 3463293, "postDate": "2026-05-26T14:24:01.360Z", "content": "The versions refer to different iterations of the same notebook during experimentation. Version 10 was the one selected for final submission rerun with private LB score (44), while later versions (score 35) are public scores that were just my other attempts.
", "rawMarkdown": "The versions refer to different iterations of the same notebook during experimentation. Version 10 was the one selected for final submission rerun with private LB score (44), while later versions (score 35) are public scores that were just my other attempts." } ] } ] }, { "id": 3462749, "postDate": "2026-05-24T19:03:12.110Z", "content": "Nice! Looks like you made some serious improvements. Well done.
", "rawMarkdown": "Nice! Looks like you made some serious improvements. Well done." }, { "id": 3463694, "postDate": "2026-05-27T17:33:56.187Z", "content": "Excellent write-up. The vote dominant ensemble idea and the modular answer reminder in the system prompt are especially insightful.
", "rawMarkdown": "Excellent write-up. The vote dominant ensemble idea and the modular answer reminder in the system prompt are especially insightful." } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T04:26:45Z", "competition_id": 118448, "competition_slug": "ai-mathematical-olympiad-progress-prize-3", "team_id": 15328337, "solution_url": "/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/2nd-place-solution-ai-mathematical-olympiad-prog" }