![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F30948873%2F09bc96d70bcabd34e60ad0352dbefbf4%2Fs4.png?generation=1780602446926721&alt=media\")
Link: TAMU-TACO Huggingface
\nOur final submission did not train a new model. We used the pretrained gpt-oss-120b model and focused on making its inference behavior stable under the AIMO3 evaluation rules.
The main idea was to treat AIMO3 not only as a math reasoning benchmark, but as a sequential resource allocation problem. The system sees one hidden problem at a time, has a 5-hour GPU notebook budget, runs on one H100 80GB GPU, and cannot know whether future problems will be easy or hard. This means that every second spent on the current problem can help or hurt the rest of the run.
\nWe found that private-score robustness was mainly controlled by the distribution of hard problems in the problem order, and by four coupled inference parameters:
\nhigh_problem_timeout = 900\nbase_problem_timeout = 300\nearly_stop = 4\nattempts = 8\nTo choose these values, we built AutoResearchAgent, a human-in-the-loop experiment engine that constructed an AIMO3-like local test set, stress-tested different hard-problem placements, ran repeated experiments, logged score and system metrics, and helped us select robust parameters. Human insight defined the search space and final decisions; the agent made the research loop measurable and repeatable.
A straightforward way to think about this competition is: \"How can we make the model solve each individual math problem better?\"
\nThat was not the framing that gave us the final gain. We instead asked:
\n\n\nHow should a strong pretrained reasoning model spend limited time and vRAM across an unknown 50-problem sequence?
\n
This distinction mattered. If the system spends too much time on one extremely hard problem, it may lose the chance to answer several later problems that were actually solvable. If it spends too little time, it may cut off hard problems that the model was close to solving. If it runs too few parallel attempts, it leaves H100 memory underused. If it runs too many, it risks OOM or slower inference.
\nSo the winning strategy was not a single prompt trick. It was a global inference policy that answered four questions for every problem:
\nThe final notebook is the execution layer. The core contribution is the resource-allocation strategy and the experimental process used to lock it.
\nThe AIMO3 rules create several constraints that are easy to underestimate:
\n[0, 99999].![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F30948873%2Fdbba5cf024c8ec671a73a2eaf2cd1db3%2Fs3.png?generation=1780602486661406&alt=media\")
The most important consequence is that the system cannot plan with knowledge of the full test set. It has to decide how aggressively to spend compute before it knows what comes next.
\nThis also changes how GPU memory should be viewed. If the current problem uses too few attempts, the unused vRAM cannot be transferred to a future problem, because that future problem has not been served yet. At the same time, using too much parallelism can cause memory pressure, lower throughput, or make a single problem consume too much of the global time budget.
\nIn our view, AIMO3 was therefore a systems problem wrapped around a math reasoning benchmark.
\nThe biggest source of private-test instability was not only whether a problem was hard. It was where hard problems appeared in the sequence.
\nA few hard problems can dominate runtime. If they appear early, an uncontrolled system can spend a large fraction of the 5-hour budget before seeing easier problems. If they appear in the middle, they can interrupt a stable pace. If they appear late, the system only handles them well if it has preserved enough time.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F30948873%2F20f7b0b7113be420d383313dff652816%2Fs5.png?generation=1780602506555970&alt=media\")
This is why we did not tune for one random validation order. A parameter set can look good on a lucky shuffle and fail under an unfavorable private order. Our tuning objective was to find parameters that survived multiple hard-problem placements, not parameters that achieved the highest score in one run.
\nThis was one of the central lessons of our solution: robustness came from modeling the distribution of hard problems across the run.
\nAfter early experiments and system analysis, we narrowed the critical search space to four parameters.
\n| Parameter | \nFinal value | \nRole | \n
|---|---|---|
base_problem_timeout | \n300 seconds | \nProtects time for ordinary solvable problems. | \n
high_problem_timeout | \n900 seconds | \nGives near-limit hard problems a chance without letting them drain the run. | \n
attempts | \n8 | \nUses H100 vRAM through parallel reasoning paths. | \n
early_stop | \n4 | \nStops once answer consensus is reliable enough. | \n
These parameters are coupled. They should not be understood as four independent knobs.
\nbase_problem_timeout and high_problem_timeout define the time policy. The base timeout protects the run from spending all its time on early hard problems. The high timeout gives difficult but solvable problems enough room when the global budget allows it.
attempts and early_stop define the sampling policy. More attempts improve self-consistency and better use the GPU, but too many attempts increase memory and latency risk. A lower early-stop threshold saves time but may trust weak agreement. A higher threshold is more reliable but may wait too long.
The final values were selected to maximize stable score, not single-run peak score.
\nSThe public and private test problems were hidden, and leaderboard feedback was too sparse for reliable tuning. We needed a local environment where we could repeatedly test inference policies.
\nWe built a 100-problem local validation set S.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F30948873%2F48e733b6da5fdd91a6af27b138f083e1%2Fs7.png?generation=1780602515926402&alt=media\")
The construction used the 10 official AIMO3 reference problems as difficulty anchors:
\nHere, pass@1 means solving with one attempt, and pass@3 means allowing up to three independent attempts. We used this simple calibration to separate problems that were immediately within reach from problems that required more sampling or were beyond the model's reliable range.
\nWe then screened candidate problems from public math sources including AIMO2, OpenMathReasoning, and Putnam-AXIOM. GPT-5 high judged difficulty and answer-format suitability, and human review was used to remove or replace unsuitable problems.
\nThe final mix was:
\n70 basic\n20 harder\n10 hardest\nFor tuning, we merged harder + hardest into a single Hard group:
70 Basic + 30 Hard\nThis local test set was not used to train the model. It was used only to tune the inference policy.
\nFor each candidate hyperparameter vector H, we tested four placements of the 30 Hard problems:
hard_front50\nhard_middle50\nhard_back50\nhard_random\nEach placement was repeated 3 times, so one candidate H was evaluated through:
4 placements x 3 repeats = 12 full evaluations per H\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F30948873%2F9bb06f67ac5ece961abbf5bee43e2671%2Fs8.png?generation=1780602524303799&alt=media\")
This protocol made the hidden private-test risk measurable. A configuration that was good only when hard problems were late or randomly spread out would not be selected. We wanted a parameter region that stayed strong when hard problems appeared in the front, middle, back, or random positions.
\nAutoResearchAgent was our experiment engine. It was not the whole idea, and it did not replace human judgment.
The human-designed part came first:
\nAutoResearchAgent then made this strategy executable:
\nS.H within our predefined ranges.![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F30948873%2Fc43b7de56056913ed6ea5696e33dda69%2Fs9.png?generation=1780602532386371&alt=media\")
The key point is that this was a human-in-the-loop research loop. The agent gave us scale, structure, and auditability. The core research judgment came from modeling the competition constraints correctly. The experiment phase used the same gpt-oss-120b / vLLM / Jupyter-tool inference stack as the final notebook, so the logged behavior reflected real runtime trade-offs rather than a lightweight proxy.
The initial values came from the competition clock and pilot experiments.
\nThe full GPU notebook budget is 5 hours:
\n5 hours = 18000 seconds\nWe reserved 600 seconds for setup and vLLM startup:
\n18000 - 600 = 17400 seconds\nWith 50 problems, this gives:
\n17400 / 50 = 348 seconds per problem\nSo the initial timeout values were:
\nhigh_problem_timeout = 348\nbase_problem_timeout = 348\nEarly pilot tests used:
\nearly_stop = 3\nattempts = 6\nIn 20 Basic problems from S, 16 produced at least 3 matching answers under this setup. GPU vRAM usage was roughly 67.8-71.6GB out of 80GB, showing that 6-way inference was already using the H100 effectively.
We then bounded the search:
\n348 <= high_problem_timeout <= 1000\n200 <= base_problem_timeout <= 348\n3 <= early_stop <= 5\n3 <= attempts <= 8\nThe lower bound of 200 seconds for base_problem_timeout came from the observation that the fastest correct AIMO3 example solution under the pilot setup took 212 seconds. The upper bound of 8 attempts came from memory. In our H100 80GB tests, 8-way inference with a 16k-32k total token budget per path used about 68.6-74.2GB of vRAM. Going beyond 8 attempts would increase OOM risk or require batching the same problem into multiple inference waves.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F30948873%2Ff818872b098105e2c024af10002806b6%2Fs11.png?generation=1780602540154555&alt=media\")
The search had two phases.
\nWe fixed:
\nattempts = 6\nearly_stop = 3\nThen we tuned:
\nhigh_problem_timeout\nbase_problem_timeout\nThis phase had:
\n46 candidate rounds\n46 x 4 x 3 = 552 placement/repeat experiments\nThe stable high-score region was:
\nhigh_problem_timeout = 894-906\nbase_problem_timeout = 288-306\nAfter manual metric review, we locked the conservative values:
\nhigh_problem_timeout = 900\nbase_problem_timeout = 300\nWe fixed:
\nhigh_problem_timeout = 900\nbase_problem_timeout = 300\nThen we tuned:
\nattempts\nearly_stop\nThis phase had:
\n18 candidate rounds\nup to 18 x 4 x 3 = 216 placement/repeat experiments\nThe best score-variance balance came from:
\nattempts = 8\nearly_stop = 4\nOverall, AutoResearchAgent covered 64 candidate rounds and up to 768 full placement/repeat experiments. Each full placement/repeat experiment ran the 100-problem local set S under one hard-problem ordering. The hyperparameter tuning environment used local research machines with 64 NVIDIA H100 80GB GPUs. This was separate from the final Kaggle inference environment, where the submitted notebook used one H100 80GB GPU.
The final Kaggle notebook implemented the locked policy with the pretrained gpt-oss-120b model.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F30948873%2Fdaacd00bd5f0cee196aac97694ea7f17%2Fs13.png?generation=1780602550546976&alt=media\")
For each problem:
\nbase_problem_timeout=300 and high_problem_timeout=900.[0, 99999].0 as a safe fallback.Important runtime choices included:
\nvLLM local API server\nkv_cache_dtype = fp8_e4m3\ngpu_memory_utilization = 0.96\ncontext_tokens = 65536\nbatch_size = 256\n--async-scheduling\n--enable-prefix-caching\n16 persistent Jupyter kernels for Python tools\nThe Jupyter kernels were pre-started and warmed with common math libraries such as math, numpy, sympy, itertools, collections, and mpmath. This avoided paying a cold-start cost when the model decided to use Python during a reasoning trace.
We used two layers of answer aggregation.
\nIf 4 attempts produced the same valid integer answer, we treated that as reliable consensus and stopped the rest of the attempts for that problem.
\nThis was important because easy and medium problems often reached consensus before all 8 attempts finished. Early stopping saved time for later hard problems.
\nIf no answer reached 4 votes, we used entropy-weighted voting among valid parsed answers.
\nFor each generated token, vLLM returned top logprobs. We converted the top-5 logprobs into probabilities:
\np_i = exp(logprob_i)\nThen we computed token entropy:
\nH_token = -sum_i p_i * log2(p_i)\nThe mean entropy of an attempt was the average token entropy over generated tokens with available logprobs. Lower entropy means the model distribution was more concentrated, so that attempt received more weight:
\nvote_weight = 1 / mean_entropy\nThis gave us a fallback when the model did not reach strict answer consensus.
\nEarly in the competition, we explored the usual questions:
\n\\boxed{}?The final decision was to not fine-tune gpt-oss-120b. The pretrained model already had strong mathematical reasoning ability. The larger gain came from configuring inference so that the model's ability was used reliably under the exact competition constraints.
We also avoided optimizing only for public leaderboard feedback or a single local shuffle. The hidden private order was too important. Once we understood that hard-problem placement could change the whole run, the research effort moved toward local stress tests and robust parameter selection.
\nThe final Kaggle notebook did not train on external data and did not call external services during inference.
\nOutside the final notebook, we used public math datasets only to build the local validation set S for inference-parameter tuning. Candidate sources included:
GPT-5 high was used outside the Kaggle submission for difficulty filtering, test-set construction support, and research-summary analysis. It was not part of the final inference notebook.
\nFinal Kaggle submission environment:
\nPlatform: Kaggle Notebook\nGPU: 1 x NVIDIA H100 80GB\nInternet: disabled\nModel: pretrained gpt-oss-120b\nTraining: none\nAutoResearchAgent tuning environment:
\nPlatform: local research machines\nGPU: 64 x NVIDIA H100 80GB\nOS: Ubuntu 24.04\nCPU: AMD EPYC 9005 Series\nRAM: 4TB 6000MT/s ECC DDR5 RDIMM\nThe distinction is important: the final winning notebook was a single-H100 Kaggle submission. The larger research compute was used for experiment automation and parameter selection.
\nWe believe there were five main differentiators.
\nFirst, we modeled the rules, not only the math problems. The sequential serving format, unknown order, 5-hour runtime, and single H100 memory limit directly determined the optimal inference policy.
\nSecond, we identified hard-problem placement as the key robustness risk. This pushed us to test front, middle, back, and random hard-problem distributions instead of relying on one shuffle.
\nThird, we found the four parameters that actually controlled the run: base_problem_timeout, high_problem_timeout, attempts, and early_stop.
Fourth, we built AutoResearchAgent to make the research loop measurable. It logged score, variance, timeout, OOM, entropy, valid-answer behavior, and resource usage for every candidate setting.
\nFifth, we matched the final notebook to the H100 system. Parallel attempts used GPU memory aggressively, persistent Jupyter kernels used CPU/RAM for tools, early stopping protected time, and dynamic timeout allocation protected the full 50-problem run.
\nWith a strong pretrained reasoning model, model training is not the only path to a winning solution. In AIMO3, correct problem modeling was decisive.
\ngpt-oss-120b had the mathematical ability. The challenge was to expose that ability under a strict, sequential, time-limited evaluation. Our solution was to convert the competition rules into a measurable inference-time resource-allocation problem, stress-test the hidden hard-problem order risk, and lock a robust policy before final submission.
The broader lesson is that for reasoning benchmarks with strict runtime constraints, inference policy can be as important as model choice.
", "comments": [ { "id": 3467141, "author_name": "Yi-Chia Chen", "author_url": "", "post_date": "2026-06-05T14:56:58.390000", "content": "Thanks for the detailed writeup! Using 64 H100s for AutoResearch is quite impressive.
", "votes": 1, "replies": [] }, { "id": 3466972, "author_name": "Hiểu Vy", "author_url": "", "post_date": "2026-06-05T03:58:09.317000", "content": "Could you share the submission notebook?
", "votes": 0, "replies": [ { "id": 3467091, "author_name": "Simon Frieder", "author_url": "", "post_date": "2026-06-05T12:10:11.187000", "content": "I got in touch with them, it's online now.
", "votes": 1, "replies": [] } ] } ], "comment_count": 3, "messages": [ { "id": 3467141, "postDate": "2026-06-05T14:56:58.390Z", "content": "Thanks for the detailed writeup! Using 64 H100s for AutoResearch is quite impressive.
", "rawMarkdown": "Thanks for the detailed writeup! Using 64 H100s for AutoResearch is quite impressive.", "votes": 1 }, { "id": 3466853, "postDate": "2026-06-04T19:55:20.263Z", "content": "Link: TAMU-TACO Huggingface
\nOur final submission did not train a new model. We used the pretrained gpt-oss-120b model and focused on making its inference behavior stable under the AIMO3 evaluation rules.
The main idea was to treat AIMO3 not only as a math reasoning benchmark, but as a sequential resource allocation problem. The system sees one hidden problem at a time, has a 5-hour GPU notebook budget, runs on one H100 80GB GPU, and cannot know whether future problems will be easy or hard. This means that every second spent on the current problem can help or hurt the rest of the run.
\nWe found that private-score robustness was mainly controlled by the distribution of hard problems in the problem order, and by four coupled inference parameters:
\nhigh_problem_timeout = 900\nbase_problem_timeout = 300\nearly_stop = 4\nattempts = 8\nTo choose these values, we built AutoResearchAgent, a human-in-the-loop experiment engine that constructed an AIMO3-like local test set, stress-tested different hard-problem placements, ran repeated experiments, logged score and system metrics, and helped us select robust parameters. Human insight defined the search space and final decisions; the agent made the research loop measurable and repeatable.
A straightforward way to think about this competition is: \"How can we make the model solve each individual math problem better?\"
\nThat was not the framing that gave us the final gain. We instead asked:
\n\n\nHow should a strong pretrained reasoning model spend limited time and vRAM across an unknown 50-problem sequence?
\n
This distinction mattered. If the system spends too much time on one extremely hard problem, it may lose the chance to answer several later problems that were actually solvable. If it spends too little time, it may cut off hard problems that the model was close to solving. If it runs too few parallel attempts, it leaves H100 memory underused. If it runs too many, it risks OOM or slower inference.
\nSo the winning strategy was not a single prompt trick. It was a global inference policy that answered four questions for every problem:
\nThe final notebook is the execution layer. The core contribution is the resource-allocation strategy and the experimental process used to lock it.
\nThe AIMO3 rules create several constraints that are easy to underestimate:
\n[0, 99999].
The most important consequence is that the system cannot plan with knowledge of the full test set. It has to decide how aggressively to spend compute before it knows what comes next.
\nThis also changes how GPU memory should be viewed. If the current problem uses too few attempts, the unused vRAM cannot be transferred to a future problem, because that future problem has not been served yet. At the same time, using too much parallelism can cause memory pressure, lower throughput, or make a single problem consume too much of the global time budget.
\nIn our view, AIMO3 was therefore a systems problem wrapped around a math reasoning benchmark.
\nThe biggest source of private-test instability was not only whether a problem was hard. It was where hard problems appeared in the sequence.
\nA few hard problems can dominate runtime. If they appear early, an uncontrolled system can spend a large fraction of the 5-hour budget before seeing easier problems. If they appear in the middle, they can interrupt a stable pace. If they appear late, the system only handles them well if it has preserved enough time.
\n
This is why we did not tune for one random validation order. A parameter set can look good on a lucky shuffle and fail under an unfavorable private order. Our tuning objective was to find parameters that survived multiple hard-problem placements, not parameters that achieved the highest score in one run.
\nThis was one of the central lessons of our solution: robustness came from modeling the distribution of hard problems across the run.
\nAfter early experiments and system analysis, we narrowed the critical search space to four parameters.
\n| Parameter | \nFinal value | \nRole | \n
|---|---|---|
base_problem_timeout | \n300 seconds | \nProtects time for ordinary solvable problems. | \n
high_problem_timeout | \n900 seconds | \nGives near-limit hard problems a chance without letting them drain the run. | \n
attempts | \n8 | \nUses H100 vRAM through parallel reasoning paths. | \n
early_stop | \n4 | \nStops once answer consensus is reliable enough. | \n
These parameters are coupled. They should not be understood as four independent knobs.
\nbase_problem_timeout and high_problem_timeout define the time policy. The base timeout protects the run from spending all its time on early hard problems. The high timeout gives difficult but solvable problems enough room when the global budget allows it.
attempts and early_stop define the sampling policy. More attempts improve self-consistency and better use the GPU, but too many attempts increase memory and latency risk. A lower early-stop threshold saves time but may trust weak agreement. A higher threshold is more reliable but may wait too long.
The final values were selected to maximize stable score, not single-run peak score.
\nSThe public and private test problems were hidden, and leaderboard feedback was too sparse for reliable tuning. We needed a local environment where we could repeatedly test inference policies.
\nWe built a 100-problem local validation set S.
The construction used the 10 official AIMO3 reference problems as difficulty anchors:
\nHere, pass@1 means solving with one attempt, and pass@3 means allowing up to three independent attempts. We used this simple calibration to separate problems that were immediately within reach from problems that required more sampling or were beyond the model's reliable range.
\nWe then screened candidate problems from public math sources including AIMO2, OpenMathReasoning, and Putnam-AXIOM. GPT-5 high judged difficulty and answer-format suitability, and human review was used to remove or replace unsuitable problems.
\nThe final mix was:
\n70 basic\n20 harder\n10 hardest\nFor tuning, we merged harder + hardest into a single Hard group:
70 Basic + 30 Hard\nThis local test set was not used to train the model. It was used only to tune the inference policy.
\nFor each candidate hyperparameter vector H, we tested four placements of the 30 Hard problems:
hard_front50\nhard_middle50\nhard_back50\nhard_random\nEach placement was repeated 3 times, so one candidate H was evaluated through:
4 placements x 3 repeats = 12 full evaluations per H\n
This protocol made the hidden private-test risk measurable. A configuration that was good only when hard problems were late or randomly spread out would not be selected. We wanted a parameter region that stayed strong when hard problems appeared in the front, middle, back, or random positions.
\nAutoResearchAgent was our experiment engine. It was not the whole idea, and it did not replace human judgment.
The human-designed part came first:
\nAutoResearchAgent then made this strategy executable:
\nS.H within our predefined ranges.
The key point is that this was a human-in-the-loop research loop. The agent gave us scale, structure, and auditability. The core research judgment came from modeling the competition constraints correctly. The experiment phase used the same gpt-oss-120b / vLLM / Jupyter-tool inference stack as the final notebook, so the logged behavior reflected real runtime trade-offs rather than a lightweight proxy.
The initial values came from the competition clock and pilot experiments.
\nThe full GPU notebook budget is 5 hours:
\n5 hours = 18000 seconds\nWe reserved 600 seconds for setup and vLLM startup:
\n18000 - 600 = 17400 seconds\nWith 50 problems, this gives:
\n17400 / 50 = 348 seconds per problem\nSo the initial timeout values were:
\nhigh_problem_timeout = 348\nbase_problem_timeout = 348\nEarly pilot tests used:
\nearly_stop = 3\nattempts = 6\nIn 20 Basic problems from S, 16 produced at least 3 matching answers under this setup. GPU vRAM usage was roughly 67.8-71.6GB out of 80GB, showing that 6-way inference was already using the H100 effectively.
We then bounded the search:
\n348 <= high_problem_timeout <= 1000\n200 <= base_problem_timeout <= 348\n3 <= early_stop <= 5\n3 <= attempts <= 8\nThe lower bound of 200 seconds for base_problem_timeout came from the observation that the fastest correct AIMO3 example solution under the pilot setup took 212 seconds. The upper bound of 8 attempts came from memory. In our H100 80GB tests, 8-way inference with a 16k-32k total token budget per path used about 68.6-74.2GB of vRAM. Going beyond 8 attempts would increase OOM risk or require batching the same problem into multiple inference waves.
The search had two phases.
\nWe fixed:
\nattempts = 6\nearly_stop = 3\nThen we tuned:
\nhigh_problem_timeout\nbase_problem_timeout\nThis phase had:
\n46 candidate rounds\n46 x 4 x 3 = 552 placement/repeat experiments\nThe stable high-score region was:
\nhigh_problem_timeout = 894-906\nbase_problem_timeout = 288-306\nAfter manual metric review, we locked the conservative values:
\nhigh_problem_timeout = 900\nbase_problem_timeout = 300\nWe fixed:
\nhigh_problem_timeout = 900\nbase_problem_timeout = 300\nThen we tuned:
\nattempts\nearly_stop\nThis phase had:
\n18 candidate rounds\nup to 18 x 4 x 3 = 216 placement/repeat experiments\nThe best score-variance balance came from:
\nattempts = 8\nearly_stop = 4\nOverall, AutoResearchAgent covered 64 candidate rounds and up to 768 full placement/repeat experiments. Each full placement/repeat experiment ran the 100-problem local set S under one hard-problem ordering. The hyperparameter tuning environment used local research machines with 64 NVIDIA H100 80GB GPUs. This was separate from the final Kaggle inference environment, where the submitted notebook used one H100 80GB GPU.
The final Kaggle notebook implemented the locked policy with the pretrained gpt-oss-120b model.
For each problem:
\nbase_problem_timeout=300 and high_problem_timeout=900.[0, 99999].0 as a safe fallback.Important runtime choices included:
\nvLLM local API server\nkv_cache_dtype = fp8_e4m3\ngpu_memory_utilization = 0.96\ncontext_tokens = 65536\nbatch_size = 256\n--async-scheduling\n--enable-prefix-caching\n16 persistent Jupyter kernels for Python tools\nThe Jupyter kernels were pre-started and warmed with common math libraries such as math, numpy, sympy, itertools, collections, and mpmath. This avoided paying a cold-start cost when the model decided to use Python during a reasoning trace.
We used two layers of answer aggregation.
\nIf 4 attempts produced the same valid integer answer, we treated that as reliable consensus and stopped the rest of the attempts for that problem.
\nThis was important because easy and medium problems often reached consensus before all 8 attempts finished. Early stopping saved time for later hard problems.
\nIf no answer reached 4 votes, we used entropy-weighted voting among valid parsed answers.
\nFor each generated token, vLLM returned top logprobs. We converted the top-5 logprobs into probabilities:
\np_i = exp(logprob_i)\nThen we computed token entropy:
\nH_token = -sum_i p_i * log2(p_i)\nThe mean entropy of an attempt was the average token entropy over generated tokens with available logprobs. Lower entropy means the model distribution was more concentrated, so that attempt received more weight:
\nvote_weight = 1 / mean_entropy\nThis gave us a fallback when the model did not reach strict answer consensus.
\nEarly in the competition, we explored the usual questions:
\n\\boxed{}?The final decision was to not fine-tune gpt-oss-120b. The pretrained model already had strong mathematical reasoning ability. The larger gain came from configuring inference so that the model's ability was used reliably under the exact competition constraints.
We also avoided optimizing only for public leaderboard feedback or a single local shuffle. The hidden private order was too important. Once we understood that hard-problem placement could change the whole run, the research effort moved toward local stress tests and robust parameter selection.
\nThe final Kaggle notebook did not train on external data and did not call external services during inference.
\nOutside the final notebook, we used public math datasets only to build the local validation set S for inference-parameter tuning. Candidate sources included:
GPT-5 high was used outside the Kaggle submission for difficulty filtering, test-set construction support, and research-summary analysis. It was not part of the final inference notebook.
\nFinal Kaggle submission environment:
\nPlatform: Kaggle Notebook\nGPU: 1 x NVIDIA H100 80GB\nInternet: disabled\nModel: pretrained gpt-oss-120b\nTraining: none\nAutoResearchAgent tuning environment:
\nPlatform: local research machines\nGPU: 64 x NVIDIA H100 80GB\nOS: Ubuntu 24.04\nCPU: AMD EPYC 9005 Series\nRAM: 4TB 6000MT/s ECC DDR5 RDIMM\nThe distinction is important: the final winning notebook was a single-H100 Kaggle submission. The larger research compute was used for experiment automation and parameter selection.
\nWe believe there were five main differentiators.
\nFirst, we modeled the rules, not only the math problems. The sequential serving format, unknown order, 5-hour runtime, and single H100 memory limit directly determined the optimal inference policy.
\nSecond, we identified hard-problem placement as the key robustness risk. This pushed us to test front, middle, back, and random hard-problem distributions instead of relying on one shuffle.
\nThird, we found the four parameters that actually controlled the run: base_problem_timeout, high_problem_timeout, attempts, and early_stop.
Fourth, we built AutoResearchAgent to make the research loop measurable. It logged score, variance, timeout, OOM, entropy, valid-answer behavior, and resource usage for every candidate setting.
\nFifth, we matched the final notebook to the H100 system. Parallel attempts used GPU memory aggressively, persistent Jupyter kernels used CPU/RAM for tools, early stopping protected time, and dynamic timeout allocation protected the full 50-problem run.
\nWith a strong pretrained reasoning model, model training is not the only path to a winning solution. In AIMO3, correct problem modeling was decisive.
\ngpt-oss-120b had the mathematical ability. The challenge was to expose that ability under a strict, sequential, time-limited evaluation. Our solution was to convert the competition rules into a measurable inference-time resource-allocation problem, stress-test the hidden hard-problem order risk, and lock a robust policy before final submission.
The broader lesson is that for reasoning benchmarks with strict runtime constraints, inference policy can be as important as model choice.
", "rawMarkdown": "# TAMU-TACO 5th Place Solution for AIMO3\n\n## Huggingface Space\n\nLink: [TAMU-TACO Huggingface](https://huggingface.co/spaces/peiranli0930/AIMO3-TAMU-TACO)\n\n## TL;DR\n\nOur final submission did not train a new model. We used the pretrained `gpt-oss-120b` model and focused on making its inference behavior stable under the AIMO3 evaluation rules.\n\nThe main idea was to treat AIMO3 not only as a math reasoning benchmark, but as a sequential resource allocation problem. The system sees one hidden problem at a time, has a 5-hour GPU notebook budget, runs on one H100 80GB GPU, and cannot know whether future problems will be easy or hard. This means that every second spent on the current problem can help or hurt the rest of the run.\n\nWe found that private-score robustness was mainly controlled by the distribution of hard problems in the problem order, and by four coupled inference parameters:\n\n```text\nhigh_problem_timeout = 900\nbase_problem_timeout = 300\nearly_stop = 4\nattempts = 8\n```\n\nTo choose these values, we built `AutoResearchAgent`, a human-in-the-loop experiment engine that constructed an AIMO3-like local test set, stress-tested different hard-problem placements, ran repeated experiments, logged score and system metrics, and helped us select robust parameters. Human insight defined the search space and final decisions; the agent made the research loop measurable and repeatable.\n\n\n\n## 1. What Was Different About Our Approach\n\nA straightforward way to think about this competition is: \"How can we make the model solve each individual math problem better?\"\n\nThat was not the framing that gave us the final gain. We instead asked:\n\n> How should a strong pretrained reasoning model spend limited time and vRAM across an unknown 50-problem sequence?\n\nThis distinction mattered. If the system spends too much time on one extremely hard problem, it may lose the chance to answer several later problems that were actually solvable. If it spends too little time, it may cut off hard problems that the model was close to solving. If it runs too few parallel attempts, it leaves H100 memory underused. If it runs too many, it risks OOM or slower inference.\n\nSo the winning strategy was not a single prompt trick. It was a global inference policy that answered four questions for every problem:\n\n- How much time must be protected for ordinary solvable problems?\n- How much extra time can a near-solvable hard problem receive?\n- How many independent reasoning paths should run in parallel?\n- When is answer agreement strong enough to stop early?\n\nThe final notebook is the execution layer. The core contribution is the resource-allocation strategy and the experimental process used to lock it.\n\n## 2. AIMO3 as a Sequential Resource Allocation Problem\n\nThe AIMO3 rules create several constraints that are easy to underestimate:\n\n- Problems are hidden and served one by one.\n- The next problem is not visible until the current problem is finished.\n- The GPU notebook runtime is limited to 5 hours.\n- Setup and vLLM startup consume part of that budget.\n- The final submission runs on a single H100 80GB GPU.\n- The answer must be an integer in `[0, 99999]`.\n- Internet is disabled during the Kaggle rerun.\n\n\n\nThe most important consequence is that the system cannot plan with knowledge of the full test set. It has to decide how aggressively to spend compute before it knows what comes next.\n\nThis also changes how GPU memory should be viewed. If the current problem uses too few attempts, the unused vRAM cannot be transferred to a future problem, because that future problem has not been served yet. At the same time, using too much parallelism can cause memory pressure, lower throughput, or make a single problem consume too much of the global time budget.\n\nIn our view, AIMO3 was therefore a systems problem wrapped around a math reasoning benchmark.\n\n## 3. Why Hard-Problem Order Was the Hidden Risk\n\nThe biggest source of private-test instability was not only whether a problem was hard. It was where hard problems appeared in the sequence.\n\nA few hard problems can dominate runtime. If they appear early, an uncontrolled system can spend a large fraction of the 5-hour budget before seeing easier problems. If they appear in the middle, they can interrupt a stable pace. If they appear late, the system only handles them well if it has preserved enough time.\n\n\n\nThis is why we did not tune for one random validation order. A parameter set can look good on a lucky shuffle and fail under an unfavorable private order. Our tuning objective was to find parameters that survived multiple hard-problem placements, not parameters that achieved the highest score in one run.\n\nThis was one of the central lessons of our solution: robustness came from modeling the distribution of hard problems across the run.\n\n## 4. The Four Parameters That Controlled the Run\n\nAfter early experiments and system analysis, we narrowed the critical search space to four parameters.\n\n| Parameter | Final value | Role |\n| --- | ---: | --- |\n| `base_problem_timeout` | `300` seconds | Protects time for ordinary solvable problems. |\n| `high_problem_timeout` | `900` seconds | Gives near-limit hard problems a chance without letting them drain the run. |\n| `attempts` | `8` | Uses H100 vRAM through parallel reasoning paths. |\n| `early_stop` | `4` | Stops once answer consensus is reliable enough. |\n\nThese parameters are coupled. They should not be understood as four independent knobs.\n\n`base_problem_timeout` and `high_problem_timeout` define the time policy. The base timeout protects the run from spending all its time on early hard problems. The high timeout gives difficult but solvable problems enough room when the global budget allows it.\n\n`attempts` and `early_stop` define the sampling policy. More attempts improve self-consistency and better use the GPU, but too many attempts increase memory and latency risk. A lower early-stop threshold saves time but may trust weak agreement. A higher threshold is more reliable but may wait too long.\n\nThe final values were selected to maximize stable score, not single-run peak score.\n\n## 5. Building a Local Test Set `S`\n\nThe public and private test problems were hidden, and leaderboard feedback was too sparse for reliable tuning. We needed a local environment where we could repeatedly test inference policies.\n\nWe built a 100-problem local validation set `S`.\n\n\n\nThe construction used the 10 official AIMO3 reference problems as difficulty anchors:\n\n- Problems solved by GPT-5 high at pass@1 became Basic anchors.\n- Problems solved only at pass@3 became Harder anchors.\n- The problem not solved at pass@3 became the Hardest anchor.\n\nHere, pass@1 means solving with one attempt, and pass@3 means allowing up to three independent attempts. We used this simple calibration to separate problems that were immediately within reach from problems that required more sampling or were beyond the model's reliable range.\n\nWe then screened candidate problems from public math sources including AIMO2, OpenMathReasoning, and Putnam-AXIOM. GPT-5 high judged difficulty and answer-format suitability, and human review was used to remove or replace unsuitable problems.\n\nThe final mix was:\n\n```text\n70 basic\n20 harder\n10 hardest\n```\n\nFor tuning, we merged `harder + hardest` into a single Hard group:\n\n```text\n70 Basic + 30 Hard\n```\n\nThis local test set was not used to train the model. It was used only to tune the inference policy.\n\n## 6. Stress-Testing Hard-Problem Placement\n\nFor each candidate hyperparameter vector `H`, we tested four placements of the 30 Hard problems:\n\n```text\nhard_front50\nhard_middle50\nhard_back50\nhard_random\n```\n\nEach placement was repeated 3 times, so one candidate `H` was evaluated through:\n\n```text\n4 placements x 3 repeats = 12 full evaluations per H\n```\n\n\n\nThis protocol made the hidden private-test risk measurable. A configuration that was good only when hard problems were late or randomly spread out would not be selected. We wanted a parameter region that stayed strong when hard problems appeared in the front, middle, back, or random positions.\n\n## 7. AutoResearchAgent\n\n`AutoResearchAgent` was our experiment engine. It was not the whole idea, and it did not replace human judgment.\n\nThe human-designed part came first:\n\n- Identify that the rules create a global resource-allocation problem.\n- Identify hard-problem placement as the main robustness risk.\n- Identify the four critical parameters.\n- Define the local test set construction.\n- Define the hard-placement stress tests.\n- Define the two-phase search.\n\nAutoResearchAgent then made this strategy executable:\n\n- Build the local validation set `S`.\n- Run candidate hyperparameter vectors across hard placements.\n- Log per-problem correctness, elapsed time, valid answers, entropy, early stopping, timeout, OOM, GPU utilization, vRAM usage, CPU utilization, and RAM usage.\n- Summarize each experiment suite.\n- Use GPT-5 high to analyze summaries from a systems perspective.\n- Propose the next candidate `H` within our predefined ranges.\n- Leave final region selection to human expert review.\n\n\n\nThe key point is that this was a human-in-the-loop research loop. The agent gave us scale, structure, and auditability. The core research judgment came from modeling the competition constraints correctly. The experiment phase used the same `gpt-oss-120b` / vLLM / Jupyter-tool inference stack as the final notebook, so the logged behavior reflected real runtime trade-offs rather than a lightweight proxy.\n\n## 8. Search Design and Experiment Count\n\nThe initial values came from the competition clock and pilot experiments.\n\nThe full GPU notebook budget is 5 hours:\n\n```text\n5 hours = 18000 seconds\n```\n\nWe reserved 600 seconds for setup and vLLM startup:\n\n```text\n18000 - 600 = 17400 seconds\n```\n\nWith 50 problems, this gives:\n\n```text\n17400 / 50 = 348 seconds per problem\n```\n\nSo the initial timeout values were:\n\n```text\nhigh_problem_timeout = 348\nbase_problem_timeout = 348\n```\n\nEarly pilot tests used:\n\n```text\nearly_stop = 3\nattempts = 6\n```\n\nIn 20 Basic problems from `S`, 16 produced at least 3 matching answers under this setup. GPU vRAM usage was roughly 67.8-71.6GB out of 80GB, showing that 6-way inference was already using the H100 effectively.\n\nWe then bounded the search:\n\n```text\n348 <= high_problem_timeout <= 1000\n200 <= base_problem_timeout <= 348\n3 <= early_stop <= 5\n3 <= attempts <= 8\n```\n\nThe lower bound of 200 seconds for `base_problem_timeout` came from the observation that the fastest correct AIMO3 example solution under the pilot setup took 212 seconds. The upper bound of 8 attempts came from memory. In our H100 80GB tests, 8-way inference with a 16k-32k total token budget per path used about 68.6-74.2GB of vRAM. Going beyond 8 attempts would increase OOM risk or require batching the same problem into multiple inference waves.\n\n\n\nThe search had two phases.\n\n### Phase 1: Time Allocation\n\nWe fixed:\n\n```text\nattempts = 6\nearly_stop = 3\n```\n\nThen we tuned:\n\n```text\nhigh_problem_timeout\nbase_problem_timeout\n```\n\nThis phase had:\n\n```text\n46 candidate rounds\n46 x 4 x 3 = 552 placement/repeat experiments\n```\n\nThe stable high-score region was:\n\n```text\nhigh_problem_timeout = 894-906\nbase_problem_timeout = 288-306\n```\n\nAfter manual metric review, we locked the conservative values:\n\n```text\nhigh_problem_timeout = 900\nbase_problem_timeout = 300\n```\n\n### Phase 2: Parallel Sampling and Consensus\n\nWe fixed:\n\n```text\nhigh_problem_timeout = 900\nbase_problem_timeout = 300\n```\n\nThen we tuned:\n\n```text\nattempts\nearly_stop\n```\n\nThis phase had:\n\n```text\n18 candidate rounds\nup to 18 x 4 x 3 = 216 placement/repeat experiments\n```\n\nThe best score-variance balance came from:\n\n```text\nattempts = 8\nearly_stop = 4\n```\n\nOverall, AutoResearchAgent covered 64 candidate rounds and up to 768 full placement/repeat experiments. Each full placement/repeat experiment ran the 100-problem local set `S` under one hard-problem ordering. The hyperparameter tuning environment used local research machines with 64 NVIDIA H100 80GB GPUs. This was separate from the final Kaggle inference environment, where the submitted notebook used one H100 80GB GPU.\n\n## 9. Final Notebook Pipeline\n\nThe final Kaggle notebook implemented the locked policy with the pretrained `gpt-oss-120b` model.\n\n\n\nFor each problem:\n\n1. The notebook computes a dynamic time budget from the global remaining time and the number of problems left.\n2. The budget is clipped between `base_problem_timeout=300` and `high_problem_timeout=900`.\n3. Up to 8 independent reasoning attempts are launched through a local vLLM OpenAI-compatible API server.\n4. Each attempt can call a pre-started Jupyter Python kernel for symbolic or numerical computation.\n5. During streaming generation, an attempt can stop once it emits a valid boxed integer.\n6. If the same valid integer appears 4 times, unfinished attempts are stopped early.\n7. If no answer reaches the early-stop threshold, valid answers are combined with entropy-weighted voting.\n8. The final output is forced to be an integer in `[0, 99999]`.\n9. If no valid answer is found, the notebook returns `0` as a safe fallback.\n\nImportant runtime choices included:\n\n```text\nvLLM local API server\nkv_cache_dtype = fp8_e4m3\ngpu_memory_utilization = 0.96\ncontext_tokens = 65536\nbatch_size = 256\n--async-scheduling\n--enable-prefix-caching\n16 persistent Jupyter kernels for Python tools\n```\n\nThe Jupyter kernels were pre-started and warmed with common math libraries such as `math`, `numpy`, `sympy`, `itertools`, `collections`, and `mpmath`. This avoided paying a cold-start cost when the model decided to use Python during a reasoning trace.\n\n## 10. Answer Aggregation\n\nWe used two layers of answer aggregation.\n\n### Common of N\n\nIf 4 attempts produced the same valid integer answer, we treated that as reliable consensus and stopped the rest of the attempts for that problem.\n\nThis was important because easy and medium problems often reached consensus before all 8 attempts finished. Early stopping saved time for later hard problems.\n\n### Entropy-Weighted Voting\n\nIf no answer reached 4 votes, we used entropy-weighted voting among valid parsed answers.\n\nFor each generated token, vLLM returned top logprobs. We converted the top-5 logprobs into probabilities:\n\n```text\np_i = exp(logprob_i)\n```\n\nThen we computed token entropy:\n\n```text\nH_token = -sum_i p_i * log2(p_i)\n```\n\nThe mean entropy of an attempt was the average token entropy over generated tokens with available logprobs. Lower entropy means the model distribution was more concentrated, so that attempt received more weight:\n\n```text\nvote_weight = 1 / mean_entropy\n```\n\nThis gave us a fallback when the model did not reach strict answer consensus.\n\n## 11. Experiments and Directions We Did Not Use\n\nEarly in the competition, we explored the usual questions:\n\n- Which open-source reasoning model should be the backbone?\n- Is extra fine-tuning or RL training necessary?\n- How can we make the output format reliably contain `\\boxed{}`?\n- How should tool-integrated reasoning be implemented?\n- How many parallel samples can the H100 support safely?\n- How should we stop attempts without losing reliability?\n\nThe final decision was to not fine-tune `gpt-oss-120b`. The pretrained model already had strong mathematical reasoning ability. The larger gain came from configuring inference so that the model's ability was used reliably under the exact competition constraints.\n\nWe also avoided optimizing only for public leaderboard feedback or a single local shuffle. The hidden private order was too important. Once we understood that hard-problem placement could change the whole run, the research effort moved toward local stress tests and robust parameter selection.\n\n## 12. External Data and External Models\n\nThe final Kaggle notebook did not train on external data and did not call external services during inference.\n\nOutside the final notebook, we used public math datasets only to build the local validation set `S` for inference-parameter tuning. Candidate sources included:\n\n- AIMO2\n- OpenMathReasoning\n- Putnam-AXIOM\n\nGPT-5 high was used outside the Kaggle submission for difficulty filtering, test-set construction support, and research-summary analysis. It was not part of the final inference notebook.\n\n## 13. Hardware\n\nFinal Kaggle submission environment:\n\n```text\nPlatform: Kaggle Notebook\nGPU: 1 x NVIDIA H100 80GB\nInternet: disabled\nModel: pretrained gpt-oss-120b\nTraining: none\n```\n\nAutoResearchAgent tuning environment:\n\n```text\nPlatform: local research machines\nGPU: 64 x NVIDIA H100 80GB\nOS: Ubuntu 24.04\nCPU: AMD EPYC 9005 Series\nRAM: 4TB 6000MT/s ECC DDR5 RDIMM\n```\n\nThe distinction is important: the final winning notebook was a single-H100 Kaggle submission. The larger research compute was used for experiment automation and parameter selection.\n\n## 14. What Set Us Apart\n\nWe believe there were five main differentiators.\n\nFirst, we modeled the rules, not only the math problems. The sequential serving format, unknown order, 5-hour runtime, and single H100 memory limit directly determined the optimal inference policy.\n\nSecond, we identified hard-problem placement as the key robustness risk. This pushed us to test front, middle, back, and random hard-problem distributions instead of relying on one shuffle.\n\nThird, we found the four parameters that actually controlled the run: `base_problem_timeout`, `high_problem_timeout`, `attempts`, and `early_stop`.\n\nFourth, we built AutoResearchAgent to make the research loop measurable. It logged score, variance, timeout, OOM, entropy, valid-answer behavior, and resource usage for every candidate setting.\n\nFifth, we matched the final notebook to the H100 system. Parallel attempts used GPU memory aggressively, persistent Jupyter kernels used CPU/RAM for tools, early stopping protected time, and dynamic timeout allocation protected the full 50-problem run.\n\n## 15. Main Takeaway\n\nWith a strong pretrained reasoning model, model training is not the only path to a winning solution. In AIMO3, correct problem modeling was decisive.\n\n`gpt-oss-120b` had the mathematical ability. The challenge was to expose that ability under a strict, sequential, time-limited evaluation. Our solution was to convert the competition rules into a measurable inference-time resource-allocation problem, stress-test the hidden hard-problem order risk, and lock a robust policy before final submission.\n\nThe broader lesson is that for reasoning benchmarks with strict runtime constraints, inference policy can be as important as model choice.\n", "votes": 1 }, { "id": 3466972, "postDate": "2026-06-05T03:58:09.317Z", "content": "Could you share the submission notebook?
", "rawMarkdown": "Could you share the submission notebook?", "replies": [ { "id": 3467091, "postDate": "2026-06-05T12:10:11.187Z", "content": "I got in touch with them, it's online now.
", "rawMarkdown": "I got in touch with them, it's online now.", "votes": 1 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T04:27:17Z", "competition_id": 118448, "competition_slug": "ai-mathematical-olympiad-progress-prize-3", "team_id": 14864949, "solution_url": "/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/aimo3-tamu-taco-solution" }