Kaggle Notebook Url: Kaggle Notebook\nGoogle Colab URL: Google Colab\nHugging Face Model URL: unsloth/gpt-oss-120b\nKaggle Model URL: danielhanchen/gpt-oss-120b\nAcknowledgments: I would like to thank the organizers from AIMO Prize, XTX Markets, and Kaggle for this year’s AIMO-3 challenge.\nSpecial thanks to Simon Frieder for being so accessible and providing valuable insights throughout the event. Finally, I am grateful to the broader research community for inspiring this work through their contributions.
\nThis write-up describes an AIMO-3 submission built around an inference-only reasoning pipeline for olympiad-style mathematics under strict Kaggle notebook constraints. The system uses GPT-OSS-120B served locally through vLLM together with a stateful Python tool for exact arithmetic, symbolic manipulation, numerical verification, and limited brute-force checking. Instead of relying on a single chain of thought, the notebook generates multiple solution attempts in parallel, extracts boxed integer answers, and aggregates them by majority vote. The pipeline also includes practical runtime engineering such as model-cache warmup, compile-cache reuse, deadline-aware stopping, and bounded concurrency so that a very large open model remains usable within the competition environment.\nThe submission achieved 39/50 on the public leaderboard and 40.5/50 on the private leaderboard. The main claim supported by this write-up is that a carefully engineered tool-integrated inference pipeline can be competitive even without fine-tuning, provided that raw model reasoning is coupled to exact computation, runtime control, and disciplined answer aggregation. At the same time, the write-up also reports negative evidence, including a confident failure on IMO 2025 Problem 6, to clarify that the system is competitive but not broadly reliable on frontier olympiad problems.
\nThis write-up is based on the final submission notebook , which was the artifact chosen for the private leaderboard rerun and therefore identifies both the underlying artifact and the current scores state explicitly.\nThe intended reproducible release is the same submission version rather than an upgraded post-competition variant. No unrestricted result is claimed here. If the public release later differs in minor ways needed for Hugging Face or Colab publication, those differences would be documented explicitly together with the final URLs.
\nThe competition required solving olympiad-style integer-answer mathematics problems through a Kaggle notebook submission interface, not in an unconstrained research environment. The practical task was therefore dual: the system had to reason well enough to solve difficult problems, and it had to remain stable enough to finish a full notebook run within Kaggle's execution budget.\nThis shaped the design goal. Rather than optimize only for isolated single-problem successes, I aimed for a pipeline that could remain competitive under the notebook runtime limit, behave stably across reruns, and generalize better than a narrowly hand-tuned leaderboard trick. In practice that meant combining a strong base model with exact computation, multiple reasoning trajectories, bounded execution, and answer aggregation.
\nAt a high level, the submission is a tool-integrated inference pipeline built around GPT-OSS-120B, a persistent Python execution environment, and multi-sample answer aggregation.
\nmath, numpy, and sympy, enabling exact computation and verification inside the reasoning loop.The notebook runs GPT-OSS-120B from /kaggle/input/gpt-oss-120b/transformers/default/1 through vLLM. The active generator configuration uses MAX_LEN = 70_000, K = 8, temperature = 1.0, top_p = 1.0, min_p = 0.02, and majority-vote answer aggregation. The notebook also contains verifier scaffolding, but ENABLE_VERIFIER = False and ENABLE_VERIFIER_FEEDBACK = False in the submitted configuration, so the final documented system is best understood as a single-model submission with optional verifier code present but inactive.\nThe Python tool is implemented as a stateful Jupyter-backed execution environment with timeout handling, health checks, and session reuse. On each problem, the harness launches parallel trajectories with deadline-aware timeouts, early stopping when a majority answer is reached, and defensive fallbacks when streams fail, contexts overflow, or no valid boxed answer is recovered.
Inserted below diagram summarizing the main components of the pipeline.\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F12319208%2F969126fa64603116b7f68654eec6bfbc%2FKaggle%20Notebook%20vLLM-2026-04-22-134356.png?generation=1776867590720448&alt=media\")
\nFigure 1. High-level overview of the submitted inference pipeline. The notebook prepares the runtime, launches a local vLLM server for GPT-OSS-120B, runs K = 8 parallel reasoning trajectories with access to a persistent Python tool, and aggregates extracted boxed answers by majority vote before returning the final submission.
math, numpy, and sympy, enabling exact computation and verification during the reasoning process.My working hypothesis was that the base model already had enough raw mathematical capability to solve a meaningful subset of the benchmark, but that practical failures would come from instability rather than from complete lack of knowledge. In particular, I expected long reasoning chains to fail through arithmetic slips, context drift, premature commitment to weak approaches, and run-to-run inconsistency.\nThat hypothesis led to three design choices. First, exact computation should be available inside the reasoning loop, not only as post-processing. Second, parallel sampling should reduce dependence on a single brittle trajectory. Third, some amount of explicit checking should help when using a decoding regime intended to preserve diverse lines of attack. The project therefore evolved toward a reliability-first pipeline rather than toward a purely model-centric or training-centric solution.
\nWhat other approaches did I consider, and why were they rejected?
\n| Alternative | \nWhat I Tried | \nOutcome | \nWhy I Did Not Use It | \n
|---|---|---|---|
| Fine-tuning or supervised finetuning on model-generated proofs | \nI explored enhancing the GPT-OSS-20B and GPT-OSS-120B models using Supervised Fine-Tuning (SFT). This involved training on roughly 6,000 synthetic samples, consisting of proofs and solutions generated by GPT-OSS-120B models, and I considered more extensions such as self-distillation or semi-online adaptation . | \nThis line of work appeared potentially promising, but I did not have enough time within the competition window to validate a training pipeline to the level of rigor required for the final submission. In my notes, this remained exploratory rather than a settled competitive advantage. | \nI did not base the final submission on this path because the strongest evidence available during the competition favored an inference-time system I could test end-to-end inside the Kaggle environment. | \n
| More aggressive intervention strategies such as frequent interruption, hinted prompting, and heavier reflection schedules | \nI experimented informally with reflection, hints, prompt diversity, and different styles of intervention while monitoring pipeline behavior on difficult reference problems such as problems 9 and 10. | \nSome interventions appeared helpful in isolated cases, especially when they improved grounding or reduced arithmetic mistakes, but other forms of interruption seemed to harm coherence by breaking the model's ongoing reasoning process. The evidence here was preliminary rather than conclusive. | \nI used a restrained version of reflection in the final pipeline, but I did not make heavier intervention or hint-dependent strategies a central claim because they were not yet supported by sufficiently rigorous ablation evidence. | \n
The pivotal insight was that the competition should be treated as a reliability problem, not only as a raw-capability problem. I started from the question of how to improve mathematical problem solving, but over time it became clear that the decisive issue was not simply whether the model could ever reach a correct solution. The more important question was whether a full pipeline could reach correct solutions consistently enough, within Kaggle's constraints, across different reruns and across different problem types. That shift in perspective pushed the project away from a purely model-centric and toward a systems-and-inference mindset.\nOnce I viewed the problem this way, the design became more coherent. Prompting, tool use, reflection, runtime engineering, and answer aggregation were no longer separate tricks; they were all components of the same attempt to convert fragile reasoning into a more stable process. In that sense, the key insight was not that any single intervention solved the problem by itself, but that combining a strong base model with exact computation, multiple trajectories, and controlled execution was more promising than pushing harder on any one axis alone. This also explains why I prioritized the pipeline over fine-tuning in the final submission: the pipeline could be tested directly under the actual competition constraints and therefore supported a stronger reproducibility story.
\nThe strongest internal benchmark encoded directly in the notebook is the Kaggle reference-set gateway used for offline validation. In addition, I used repeated-run diagnostics on selected hard reference problems, especially problems 9 and 10, to assess stability across reruns. I also stress-tested the full pipeline on IMO 2025 Problem 6, where it failed confidently.\nThe most important empirical takeaways were the following.
\n| Question | \nEvidence | \nTakeaway | \n
|---|---|---|
| Does tool use help? | \nDevelopment-time qualitative comparison and final inclusion in the best submission | \nTool access appeared useful for exact computation and verification, although I do not claim a complete isolated causal estimate. | \n
| Does reflection always help? | \nMixed diagnostics on hard problems | \nLight checking sometimes helped; heavier interruption could harm coherence. | \n
| Is the method broadly reliable on very hard olympiad problems? | \nIMO 2025 Problem 6 stress test | \nNo. The pipeline can still fail confidently on sufficiently difficult problems. | \n
| These experiments guided the final design, but not all were logged as exhaustive controlled ablations. Accordingly, the strongest formal evidence remains the official public leaderboard result, while the internal evidence mainly supports claims about stability, plausibility, and failure modes. | \n\n | \n |
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F12319208%2F83f9b4e512eb5def8ad94e121238f4e8%2Fimo2025_p6_vote_patterns.png?generation=1776879236132782&alt=media\")
\nFigure 2. shows the result of running the full pipeline on IMO 2025 Problem 6, which is widely regarded as one of the hardest problems in recent memory. I ran the notebook on this problem several times with different random seeds and observed stable convergence to the same wrong answer, even though the reasoning traces were coherent and used tools correctly.\nI used IMO 2025 Problem 6 as an informal proxy for what the hardest end of AIMO-3 might look like in terms of structural complexity, long-horizon reasoning, and sensitivity to early problem interpretation.\nThese runs are informative because they show not only disagreement, but also stable convergence to the same wrong answer. This suggests a failure mode deeper than sampling noise or a local arithmetic error: the model may fail to form the right structural understanding of the problem early enough to choose an effective solution strategy. Once that initial framing is wrong, downstream reasoning can remain coherent, use tools correctly, and still converge to an incorrect answer.\nI include this result as negative evidence rather than anecdote. It suggests that for very hard problems, the bottleneck may be not only verification or computation, but correct problem understanding at the outset. Tool use and majority vote improve reliability when viable solution paths are present, but they do not guarantee recovery from a flawed initial representation of the task.
First, I explored training-oriented ideas, including supervised finetuning on model-generated proofs and the broader possibility of self-distillation or stronger fine-tuning pipelines. My view during the competition was that these directions might eventually improve smaller models or improve grounding, but I did not reach a level of validation that justified making them part of the final submission. Because of the competition timeline, inference-time engineering offered a much shorter path to something I could actually test end-to-end under Kaggle constraints.\nSecond, I considered more adaptive strategies such as semi test-time reinforcement learning or online adaptation from reasoning feedback. I did not implement these in the competition submission, mainly because they would have required more time, compute, and methodological control than the notebook setting permitted. I still regard them as plausible future directions, especially for turning intermediate reasoning traces into learning signals rather than using them only for immediate correction.\nThird, some observations from development were useful precisely because they were mixed. For example, explicit self-reflection sometimes helped the model recover from local mistakes, but frequent interruption also appeared capable of harming coherence by breaking the flow of reasoning. Likewise, adding more mathematical heuristics or olympiad-specific cues to prompts sometimes seemed to reduce time-to-answer on selected problems, but I did not gather sufficiently rigorous evidence to make that a principal claim. I also tested the full pipeline on IMO 2025 Problem 6 as a stress test and observed a confident failure rather than a hesitant near-miss. That negative result is important: it suggests that the pipeline can still be systematically wrong on very hard problems even when it produces high-confidence reasoning. I include these points because they shaped the final design, but they should be interpreted as development insights rather than fully established empirical findings.
\nThe official public leaderboard score for the documented submission was 39/50, also, the public leaderboard score for the documented submission was 40.5/50. This establishes that the submission is a genuinely competitive restricted notebook system rather than a toy demonstration.\nAt the same time, the public leaderboard alone does not prove that each design choice made a measurable causal contribution, nor does it eliminate the possibility of public-set overfitting. The fair reading is that the pipeline has strong public evidence of effectiveness, moderate internal evidence of robustness, and incomplete evidence about which individual interventions mattered most. The confident failure on IMO 2025 Problem 6 is important negative evidence and should constrain any broader claims of generality.
\nThis submission has five important limitations.
\nThis submission shows that a carefully engineered inference-only pipeline can remain competitive in AIMO-3 without fine-tuning when raw model reasoning is combined with exact tool use, bounded execution, and answer aggregation. The main lesson is that performance depended as much on pipeline reliability as on raw model capability.\nThe strongest formal evidence is the public leaderboard score of 39/50 and private leaderboard score of 40.5/50. The supporting internal evidence suggests that the chosen pipeline was more stable than a brittle single-trajectory baseline, but it does not justify sweeping claims about universal correctness. The right conclusion is therefore modest but meaningful: tool-integrated, runtime-aware inference can be competitive under Kaggle constraints, but serious failure modes remain.
\nThe active prompt list in the submitted notebook is:
\nSOLVER_PROMPTVERIFIER_PROMPT\nAcross these prompts, the shared behavioral constraints are consistent: reason carefully, use Python for exact computation or verification when appropriate, do not guess, and return a final boxed integer answer.| Parameter | \nValue | \n
|---|---|
SEED | \n42 | \n
MAX_LEN | \n70_000 | \n
USE_BUDGET | \nFalse | \n
K | \n8 | \n
TEMPERATURE | \n1.0 | \n
TOP_P | \n1.0 | \n
MIN_P | \n0.02 | \n
TOP_logprobs | \n5 | \n
GENERATOR_PORT | \n8000 | \n
GENERATOR_GPU_MEMORY_UTILIZATION | \n0.96 | \n
GENERATOR_MAX_NUM_SEQS | \n256 | \n
ENABLE_VERIFIER | \nFalse | \n
ENABLE_VERIFIER_FEEDBACK | \nFalse | \n
Adding to @sambealing's comments below, I have a few additional suggestions and questions:
\nAblations: As Sam mentioned, it would be helpful to report any evidence or ablations that motivated the design choices described in Section 5.1. Adding plots could also make it easier to quickly compare the effects of different implementations.
SFT Dataset: You mention using a custom dataset of 60k examples for SFT. Generating this many high-quality datapoints is a substantial effort, so it would be valuable to add a dedicated section describing the data sources, the full generation pipeline, and the motivation behind creating your own dataset. It would also be useful to explain why you chose not to use an existing large math dataset, such as one of the datasets from the Math Corpus Prize, which includes many gpt-oss-120b trace datasets, or a public dataset such as OpenMathReasoning.
Reproducibility: To make your work easier to reuse, is the full training and evaluation code available somewhere? I could not find a direct link in the writeup. Ideally, the code would be well documented and easily accessible, for example, through a GitHub repository with a clear README.
Looking forward to your response :)
", "votes": 1, "replies": [ { "id": 3457850, "author_name": "Anthony Nwafor", "author_url": "", "post_date": "2026-05-14T13:22:31.380000", "content": "Hi @philipvonderlind, thank you for questions and thoughtful suggestions,
\nOn ablations, I agree that the current write-up would benefit from more quantitative evidence supporting the design choices in §5.1. Those choices were mainly informed by reference-set diagnostics and repeated runs on selected hard problems during development, rather than by a fully controlled ablation study. I can make that clearer in the write-up.
\nOn the SFT dataset point, the dataset was assembled from a mix of selected CrystalMath[Math Corpus Prize] problems and other public-source problems, which I then used to generate gpt-oss-120B traces. The main reason for creating a custom dataset was that, for the problems I was focusing on at the time, I was not able to find existinggpt-oss-120B high-thinking-effort traces that matched the cases I wanted to examine. I was also aware of resources such as OpenMathReasoning, but in this exploratory setting I wanted to work with traces generated by gpt-oss-120B itself rather than rely only on reasoning data produced by other models. This was motivated partly by weaknesses I had observed on hard cases, especially AIMO-3 reference Problem 10, and partly by the practical intuition that model-matched traces might be more useful for this particular SFT direction. That said, this remained exploratory work and was not part of the final submitted inference pipeline, so I did not mean to present it as a validated core contribution.
On reproducibility, the current write-up links the final Kaggle notebook and colab version of the submission, and i also attached the exploratory SFT and semi-Test Time RL notebooks in the comment section so reviewers can see what was tried beyond the final workflow. I am also planning to ship a much cleaner and more complete version through a Github repository with a clear README so the training and evaluation workflow is easier to follow and reuse. The attached notebooks were meant as supporting reference and material in the meantime, not as the final packaged reproducibility release.
Hi, thank you for your submission to the Writeup Prize. @philipvonderlind and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.
\n2025 = 45^2), so some strategies (e.g. experimenting with small cases) don't necessarily yield useful results. Did you stress-test the pipeline on any other hard olympiad problems with different challenges, or could the \"stable convergence to a wrong answer\" failure mode be specific to problems where small-case intuition is actively misleading (presumably there was some sort of pattern as to why the model focussed on 3037)?Look forward to reading your responses on these.
", "votes": 1, "replies": [ { "id": 3457822, "author_name": "Anthony Nwafor", "author_url": "", "post_date": "2026-05-14T12:06:19.060000", "content": "Hi @sambealing thank you for the questions
\nOn the first point, i do not have a controlled quantitative ablation suit beyond the official AIMO evalation. The statements in §6 were intended as development-time observations from reference-set diagnostics and repeated runs as selected hard problems, not as strong claims backed by exhuastive tables. I am happy to revise the write-up to make that more clearer.
\nOn the second point, I agree that IMO 2025 P6 should not be treated as fully representative of all hard olympiad problems. I included it as a stress test showing that the pipeline can converge stably to a wrong answer even when the reasoning trace is coherent and tool-supported. The intended claim was limited: tool use, self-reflection (bounded self-refinement), and majority voting do not guarantee recovery once the model commits to a flawed structural interpretation. It was not meant as evidence that this exact failure model is universal across all hard problems.
", "votes": 0, "replies": [] } ] }, { "id": 3450600, "author_name": "Anthony Nwafor", "author_url": "", "post_date": "2026-04-30T04:48:30.157000", "content": "Attached for reference are the files related to approaches and experiments that were evaluated but not incorporated into the final workflow. They were included here to document what was tried, provide additional context, and support any future review or follow-up if needed.
", "votes": 0, "replies": [] } ], "comment_count": 5, "messages": [ { "id": 3457726, "postDate": "2026-05-14T08:34:42.890Z", "content": "Adding to @sambealing's comments below, I have a few additional suggestions and questions:
\nAblations: As Sam mentioned, it would be helpful to report any evidence or ablations that motivated the design choices described in Section 5.1. Adding plots could also make it easier to quickly compare the effects of different implementations.
SFT Dataset: You mention using a custom dataset of 60k examples for SFT. Generating this many high-quality datapoints is a substantial effort, so it would be valuable to add a dedicated section describing the data sources, the full generation pipeline, and the motivation behind creating your own dataset. It would also be useful to explain why you chose not to use an existing large math dataset, such as one of the datasets from the Math Corpus Prize, which includes many gpt-oss-120b trace datasets, or a public dataset such as OpenMathReasoning.
Reproducibility: To make your work easier to reuse, is the full training and evaluation code available somewhere? I could not find a direct link in the writeup. Ideally, the code would be well documented and easily accessible, for example, through a GitHub repository with a clear README.
Looking forward to your response :)
", "rawMarkdown": "Adding to @sambealing's comments below, I have a few additional suggestions and questions:\n\n1. **Ablations**: As Sam mentioned, it would be helpful to report any evidence or ablations that motivated the design choices described in Section `5.1`. Adding plots could also make it easier to quickly compare the effects of different implementations.\n\n2. **SFT Dataset**: You mention using a custom dataset of 60k examples for SFT. Generating this many high-quality datapoints is a substantial effort, so it would be valuable to add a dedicated section describing the data sources, the full generation pipeline, and the motivation behind creating your own dataset. It would also be useful to explain why you chose not to use an existing large math dataset, such as one of the datasets from the Math Corpus Prize, which includes many `gpt-oss-120b` trace datasets, or a public dataset such as [OpenMathReasoning](https://huggingface.co/datasets/nvidia/OpenMathReasoning).\n\n3. **Reproducibility**: To make your work easier to reuse, is the full training and evaluation code available somewhere? I could not find a direct link in the writeup. Ideally, the code would be well documented and easily accessible, for example, through a GitHub repository with a clear `README`.\n\nLooking forward to your response :)", "votes": 1, "replies": [ { "id": 3457850, "postDate": "2026-05-14T13:22:31.380Z", "content": "Hi @philipvonderlind, thank you for questions and thoughtful suggestions,
\nOn ablations, I agree that the current write-up would benefit from more quantitative evidence supporting the design choices in §5.1. Those choices were mainly informed by reference-set diagnostics and repeated runs on selected hard problems during development, rather than by a fully controlled ablation study. I can make that clearer in the write-up.
\nOn the SFT dataset point, the dataset was assembled from a mix of selected CrystalMath[Math Corpus Prize] problems and other public-source problems, which I then used to generate gpt-oss-120B traces. The main reason for creating a custom dataset was that, for the problems I was focusing on at the time, I was not able to find existinggpt-oss-120B high-thinking-effort traces that matched the cases I wanted to examine. I was also aware of resources such as OpenMathReasoning, but in this exploratory setting I wanted to work with traces generated by gpt-oss-120B itself rather than rely only on reasoning data produced by other models. This was motivated partly by weaknesses I had observed on hard cases, especially AIMO-3 reference Problem 10, and partly by the practical intuition that model-matched traces might be more useful for this particular SFT direction. That said, this remained exploratory work and was not part of the final submitted inference pipeline, so I did not mean to present it as a validated core contribution.
On reproducibility, the current write-up links the final Kaggle notebook and colab version of the submission, and i also attached the exploratory SFT and semi-Test Time RL notebooks in the comment section so reviewers can see what was tried beyond the final workflow. I am also planning to ship a much cleaner and more complete version through a Github repository with a clear README so the training and evaluation workflow is easier to follow and reuse. The attached notebooks were meant as supporting reference and material in the meantime, not as the final packaged reproducibility release.
Hi, thank you for your submission to the Writeup Prize. @philipvonderlind and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.
\n2025 = 45^2), so some strategies (e.g. experimenting with small cases) don't necessarily yield useful results. Did you stress-test the pipeline on any other hard olympiad problems with different challenges, or could the \"stable convergence to a wrong answer\" failure mode be specific to problems where small-case intuition is actively misleading (presumably there was some sort of pattern as to why the model focussed on 3037)?Look forward to reading your responses on these.
", "rawMarkdown": "Hi, thank you for your submission to the Writeup Prize. @philipvonderlind and I will post some questions below that help us in assessing your submission more closely, as described in the [updated discussion post](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/689703).\n\n1. **Reference-set ablations:** §6 reports qualitative findings on the Kaggle reference set, and §8 acknowledges the lack of exhaustive ablation tables. Do you have any quantitative evidence to support the takeaways you mention? This could be on benchmark datasets other than AIMO.\n2. **IMO 2025 P6 representativeness:** This problem has an unusual structure trap in that the required construction relies on the side length being a perfect square (`2025 = 45^2`), so some strategies (e.g. experimenting with small cases) don't necessarily yield useful results. Did you stress-test the pipeline on any other hard olympiad problems with different challenges, or could the \"stable convergence to a wrong answer\" failure mode be specific to problems where small-case intuition is actively misleading (presumably there was some sort of pattern as to why the model focussed on 3037)?\n\nLook forward to reading your responses on these.", "votes": 1, "replies": [ { "id": 3457822, "postDate": "2026-05-14T12:06:19.060Z", "content": "Hi @sambealing thank you for the questions
\nOn the first point, i do not have a controlled quantitative ablation suit beyond the official AIMO evalation. The statements in §6 were intended as development-time observations from reference-set diagnostics and repeated runs as selected hard problems, not as strong claims backed by exhuastive tables. I am happy to revise the write-up to make that more clearer.
\nOn the second point, I agree that IMO 2025 P6 should not be treated as fully representative of all hard olympiad problems. I included it as a stress test showing that the pipeline can converge stably to a wrong answer even when the reasoning trace is coherent and tool-supported. The intended claim was limited: tool use, self-reflection (bounded self-refinement), and majority voting do not guarantee recovery once the model commits to a flawed structural interpretation. It was not meant as evidence that this exact failure model is universal across all hard problems.
", "rawMarkdown": "Hi @sambealing thank you for the questions\n\nOn the first point, i do not have a controlled quantitative ablation suit beyond the official AIMO evalation. The statements in §6 were intended as development-time observations from reference-set diagnostics and repeated runs as selected hard problems, not as strong claims backed by exhuastive tables. I am happy to revise the write-up to make that more clearer.\n\nOn the second point, I agree that IMO 2025 P6 should not be treated as fully representative of all hard olympiad problems. I included it as a stress test showing that the pipeline can converge stably to a wrong answer even when the reasoning trace is coherent and tool-supported. The intended claim was limited: tool use, self-reflection (bounded self-refinement), and majority voting do not guarantee recovery once the model commits to a flawed structural interpretation. It was not meant as evidence that this exact failure model is universal across all hard problems." } ] }, { "id": 3446979, "postDate": "2026-04-22T14:41:54.777Z", "content": "Kaggle Notebook Url: Kaggle Notebook\nGoogle Colab URL: Google Colab\nHugging Face Model URL: unsloth/gpt-oss-120b\nKaggle Model URL: danielhanchen/gpt-oss-120b\nAcknowledgments: I would like to thank the organizers from AIMO Prize, XTX Markets, and Kaggle for this year’s AIMO-3 challenge.\nSpecial thanks to Simon Frieder for being so accessible and providing valuable insights throughout the event. Finally, I am grateful to the broader research community for inspiring this work through their contributions.
\nThis write-up describes an AIMO-3 submission built around an inference-only reasoning pipeline for olympiad-style mathematics under strict Kaggle notebook constraints. The system uses GPT-OSS-120B served locally through vLLM together with a stateful Python tool for exact arithmetic, symbolic manipulation, numerical verification, and limited brute-force checking. Instead of relying on a single chain of thought, the notebook generates multiple solution attempts in parallel, extracts boxed integer answers, and aggregates them by majority vote. The pipeline also includes practical runtime engineering such as model-cache warmup, compile-cache reuse, deadline-aware stopping, and bounded concurrency so that a very large open model remains usable within the competition environment.\nThe submission achieved 39/50 on the public leaderboard and 40.5/50 on the private leaderboard. The main claim supported by this write-up is that a carefully engineered tool-integrated inference pipeline can be competitive even without fine-tuning, provided that raw model reasoning is coupled to exact computation, runtime control, and disciplined answer aggregation. At the same time, the write-up also reports negative evidence, including a confident failure on IMO 2025 Problem 6, to clarify that the system is competitive but not broadly reliable on frontier olympiad problems.
\nThis write-up is based on the final submission notebook , which was the artifact chosen for the private leaderboard rerun and therefore identifies both the underlying artifact and the current scores state explicitly.\nThe intended reproducible release is the same submission version rather than an upgraded post-competition variant. No unrestricted result is claimed here. If the public release later differs in minor ways needed for Hugging Face or Colab publication, those differences would be documented explicitly together with the final URLs.
\nThe competition required solving olympiad-style integer-answer mathematics problems through a Kaggle notebook submission interface, not in an unconstrained research environment. The practical task was therefore dual: the system had to reason well enough to solve difficult problems, and it had to remain stable enough to finish a full notebook run within Kaggle's execution budget.\nThis shaped the design goal. Rather than optimize only for isolated single-problem successes, I aimed for a pipeline that could remain competitive under the notebook runtime limit, behave stably across reruns, and generalize better than a narrowly hand-tuned leaderboard trick. In practice that meant combining a strong base model with exact computation, multiple reasoning trajectories, bounded execution, and answer aggregation.
\nAt a high level, the submission is a tool-integrated inference pipeline built around GPT-OSS-120B, a persistent Python execution environment, and multi-sample answer aggregation.
\nmath, numpy, and sympy, enabling exact computation and verification inside the reasoning loop.The notebook runs GPT-OSS-120B from /kaggle/input/gpt-oss-120b/transformers/default/1 through vLLM. The active generator configuration uses MAX_LEN = 70_000, K = 8, temperature = 1.0, top_p = 1.0, min_p = 0.02, and majority-vote answer aggregation. The notebook also contains verifier scaffolding, but ENABLE_VERIFIER = False and ENABLE_VERIFIER_FEEDBACK = False in the submitted configuration, so the final documented system is best understood as a single-model submission with optional verifier code present but inactive.\nThe Python tool is implemented as a stateful Jupyter-backed execution environment with timeout handling, health checks, and session reuse. On each problem, the harness launches parallel trajectories with deadline-aware timeouts, early stopping when a majority answer is reached, and defensive fallbacks when streams fail, contexts overflow, or no valid boxed answer is recovered.
Inserted below diagram summarizing the main components of the pipeline.\n\nFigure 1. High-level overview of the submitted inference pipeline. The notebook prepares the runtime, launches a local vLLM server for GPT-OSS-120B, runs K = 8 parallel reasoning trajectories with access to a persistent Python tool, and aggregates extracted boxed answers by majority vote before returning the final submission.
math, numpy, and sympy, enabling exact computation and verification during the reasoning process.My working hypothesis was that the base model already had enough raw mathematical capability to solve a meaningful subset of the benchmark, but that practical failures would come from instability rather than from complete lack of knowledge. In particular, I expected long reasoning chains to fail through arithmetic slips, context drift, premature commitment to weak approaches, and run-to-run inconsistency.\nThat hypothesis led to three design choices. First, exact computation should be available inside the reasoning loop, not only as post-processing. Second, parallel sampling should reduce dependence on a single brittle trajectory. Third, some amount of explicit checking should help when using a decoding regime intended to preserve diverse lines of attack. The project therefore evolved toward a reliability-first pipeline rather than toward a purely model-centric or training-centric solution.
\nWhat other approaches did I consider, and why were they rejected?
\n| Alternative | \nWhat I Tried | \nOutcome | \nWhy I Did Not Use It | \n
|---|---|---|---|
| Fine-tuning or supervised finetuning on model-generated proofs | \nI explored enhancing the GPT-OSS-20B and GPT-OSS-120B models using Supervised Fine-Tuning (SFT). This involved training on roughly 6,000 synthetic samples, consisting of proofs and solutions generated by GPT-OSS-120B models, and I considered more extensions such as self-distillation or semi-online adaptation . | \nThis line of work appeared potentially promising, but I did not have enough time within the competition window to validate a training pipeline to the level of rigor required for the final submission. In my notes, this remained exploratory rather than a settled competitive advantage. | \nI did not base the final submission on this path because the strongest evidence available during the competition favored an inference-time system I could test end-to-end inside the Kaggle environment. | \n
| More aggressive intervention strategies such as frequent interruption, hinted prompting, and heavier reflection schedules | \nI experimented informally with reflection, hints, prompt diversity, and different styles of intervention while monitoring pipeline behavior on difficult reference problems such as problems 9 and 10. | \nSome interventions appeared helpful in isolated cases, especially when they improved grounding or reduced arithmetic mistakes, but other forms of interruption seemed to harm coherence by breaking the model's ongoing reasoning process. The evidence here was preliminary rather than conclusive. | \nI used a restrained version of reflection in the final pipeline, but I did not make heavier intervention or hint-dependent strategies a central claim because they were not yet supported by sufficiently rigorous ablation evidence. | \n
The pivotal insight was that the competition should be treated as a reliability problem, not only as a raw-capability problem. I started from the question of how to improve mathematical problem solving, but over time it became clear that the decisive issue was not simply whether the model could ever reach a correct solution. The more important question was whether a full pipeline could reach correct solutions consistently enough, within Kaggle's constraints, across different reruns and across different problem types. That shift in perspective pushed the project away from a purely model-centric and toward a systems-and-inference mindset.\nOnce I viewed the problem this way, the design became more coherent. Prompting, tool use, reflection, runtime engineering, and answer aggregation were no longer separate tricks; they were all components of the same attempt to convert fragile reasoning into a more stable process. In that sense, the key insight was not that any single intervention solved the problem by itself, but that combining a strong base model with exact computation, multiple trajectories, and controlled execution was more promising than pushing harder on any one axis alone. This also explains why I prioritized the pipeline over fine-tuning in the final submission: the pipeline could be tested directly under the actual competition constraints and therefore supported a stronger reproducibility story.
\nThe strongest internal benchmark encoded directly in the notebook is the Kaggle reference-set gateway used for offline validation. In addition, I used repeated-run diagnostics on selected hard reference problems, especially problems 9 and 10, to assess stability across reruns. I also stress-tested the full pipeline on IMO 2025 Problem 6, where it failed confidently.\nThe most important empirical takeaways were the following.
\n| Question | \nEvidence | \nTakeaway | \n
|---|---|---|
| Does tool use help? | \nDevelopment-time qualitative comparison and final inclusion in the best submission | \nTool access appeared useful for exact computation and verification, although I do not claim a complete isolated causal estimate. | \n
| Does reflection always help? | \nMixed diagnostics on hard problems | \nLight checking sometimes helped; heavier interruption could harm coherence. | \n
| Is the method broadly reliable on very hard olympiad problems? | \nIMO 2025 Problem 6 stress test | \nNo. The pipeline can still fail confidently on sufficiently difficult problems. | \n
| These experiments guided the final design, but not all were logged as exhaustive controlled ablations. Accordingly, the strongest formal evidence remains the official public leaderboard result, while the internal evidence mainly supports claims about stability, plausibility, and failure modes. | \n\n | \n |
\nFigure 2. shows the result of running the full pipeline on IMO 2025 Problem 6, which is widely regarded as one of the hardest problems in recent memory. I ran the notebook on this problem several times with different random seeds and observed stable convergence to the same wrong answer, even though the reasoning traces were coherent and used tools correctly.\nI used IMO 2025 Problem 6 as an informal proxy for what the hardest end of AIMO-3 might look like in terms of structural complexity, long-horizon reasoning, and sensitivity to early problem interpretation.\nThese runs are informative because they show not only disagreement, but also stable convergence to the same wrong answer. This suggests a failure mode deeper than sampling noise or a local arithmetic error: the model may fail to form the right structural understanding of the problem early enough to choose an effective solution strategy. Once that initial framing is wrong, downstream reasoning can remain coherent, use tools correctly, and still converge to an incorrect answer.\nI include this result as negative evidence rather than anecdote. It suggests that for very hard problems, the bottleneck may be not only verification or computation, but correct problem understanding at the outset. Tool use and majority vote improve reliability when viable solution paths are present, but they do not guarantee recovery from a flawed initial representation of the task.
First, I explored training-oriented ideas, including supervised finetuning on model-generated proofs and the broader possibility of self-distillation or stronger fine-tuning pipelines. My view during the competition was that these directions might eventually improve smaller models or improve grounding, but I did not reach a level of validation that justified making them part of the final submission. Because of the competition timeline, inference-time engineering offered a much shorter path to something I could actually test end-to-end under Kaggle constraints.\nSecond, I considered more adaptive strategies such as semi test-time reinforcement learning or online adaptation from reasoning feedback. I did not implement these in the competition submission, mainly because they would have required more time, compute, and methodological control than the notebook setting permitted. I still regard them as plausible future directions, especially for turning intermediate reasoning traces into learning signals rather than using them only for immediate correction.\nThird, some observations from development were useful precisely because they were mixed. For example, explicit self-reflection sometimes helped the model recover from local mistakes, but frequent interruption also appeared capable of harming coherence by breaking the flow of reasoning. Likewise, adding more mathematical heuristics or olympiad-specific cues to prompts sometimes seemed to reduce time-to-answer on selected problems, but I did not gather sufficiently rigorous evidence to make that a principal claim. I also tested the full pipeline on IMO 2025 Problem 6 as a stress test and observed a confident failure rather than a hesitant near-miss. That negative result is important: it suggests that the pipeline can still be systematically wrong on very hard problems even when it produces high-confidence reasoning. I include these points because they shaped the final design, but they should be interpreted as development insights rather than fully established empirical findings.
\nThe official public leaderboard score for the documented submission was 39/50, also, the public leaderboard score for the documented submission was 40.5/50. This establishes that the submission is a genuinely competitive restricted notebook system rather than a toy demonstration.\nAt the same time, the public leaderboard alone does not prove that each design choice made a measurable causal contribution, nor does it eliminate the possibility of public-set overfitting. The fair reading is that the pipeline has strong public evidence of effectiveness, moderate internal evidence of robustness, and incomplete evidence about which individual interventions mattered most. The confident failure on IMO 2025 Problem 6 is important negative evidence and should constrain any broader claims of generality.
\nThis submission has five important limitations.
\nThis submission shows that a carefully engineered inference-only pipeline can remain competitive in AIMO-3 without fine-tuning when raw model reasoning is combined with exact tool use, bounded execution, and answer aggregation. The main lesson is that performance depended as much on pipeline reliability as on raw model capability.\nThe strongest formal evidence is the public leaderboard score of 39/50 and private leaderboard score of 40.5/50. The supporting internal evidence suggests that the chosen pipeline was more stable than a brittle single-trajectory baseline, but it does not justify sweeping claims about universal correctness. The right conclusion is therefore modest but meaningful: tool-integrated, runtime-aware inference can be competitive under Kaggle constraints, but serious failure modes remain.
\nThe active prompt list in the submitted notebook is:
\nSOLVER_PROMPTVERIFIER_PROMPT\nAcross these prompts, the shared behavioral constraints are consistent: reason carefully, use Python for exact computation or verification when appropriate, do not guess, and return a final boxed integer answer.| Parameter | \nValue | \n
|---|---|
SEED | \n42 | \n
MAX_LEN | \n70_000 | \n
USE_BUDGET | \nFalse | \n
K | \n8 | \n
TEMPERATURE | \n1.0 | \n
TOP_P | \n1.0 | \n
MIN_P | \n0.02 | \n
TOP_logprobs | \n5 | \n
GENERATOR_PORT | \n8000 | \n
GENERATOR_GPU_MEMORY_UTILIZATION | \n0.96 | \n
GENERATOR_MAX_NUM_SEQS | \n256 | \n
ENABLE_VERIFIER | \nFalse | \n
ENABLE_VERIFIER_FEEDBACK | \nFalse | \n
Attached for reference are the files related to approaches and experiments that were evaluated but not incorporated into the final workflow. They were included here to document what was tried, provide additional context, and support any future review or follow-up if needed.
", "rawMarkdown": "Attached for reference are the files related to approaches and experiments that were evaluated but not incorporated into the final workflow. They were included here to document what was tried, provide additional context, and support any future review or follow-up if needed." } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T04:24:06Z", "competition_id": 118448, "competition_slug": "ai-mathematical-olympiad-progress-prize-3", "team_id": 14801006, "solution_url": "/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/competitive-olympiad-reasoning-under-kaggle-constr" }