![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1563363%2Fef53d83ee10bc3abac80303a2c81184d%2FTime-Budget.png?generation=1778943579039470&alt=media\")
Independent Research • April 2025
\n\nThis report documents an independent submission to the AI Mathematical Olympiad (AIMO)\nProgress Prize 3 competition. Operating under a strict 5-hour GPU budget on a single\nNVIDIA H100 80GB GPU, a comprehensive set of inference-time optimizations for\nGPT-OSS-120B were investigated.\n
\n\nContributions include:\n
\n\nThe AIMO Progress Prize 3 competition requires solving 50 mathematical problems under\na strict 5-hour GPU constraint. Problems are competition-grade, requiring multi-step\nsymbolic and numerical reasoning.\n
\n\nEach participant receives a single GPU allocation for the entire run, with approximately\n20 minutes consumed by environment setup and model loading, leaving roughly\n336 seconds per question.\n
\n\nThis work is entirely inference-based — no model weights were fine-tuned.\nThe approach combines multiple factors:\n
\n\nGPT-OSS-120B appeared to perform best when given sufficient time and token budget to\nreason through problems fully.\n
\n\nNote: The best-performing configuration (score 24) could not be submitted to the competition because a critical library dependency was accidentally deleted just before the deadline.\n
\n| Component | \nSpecification | \n
|---|---|
| GPU | \nNVIDIA H100 80GB (SXM or PCIe; single GPU, tp_size=1) | \n
| GPU Memory | \n~60–65 GB total during inference (96% utilization) | \n
| CUDA Architecture | \nHopper (sm_90) — required for fp8_e4m3 KV cache | \n
| KV Cache dtype | \nfp8_e4m3 | \n
| Model | \nGPT-OSS-120B (MoE, 120B params, 5.1B active) | \n
| Serving Framework | \nModified NemoSkills + vLLM 0.11.2 | \n
| Python | \nPython 3.12.12 | \n
| Parameter | \nValue | \n
|---|---|
| Total GPU time | \n5 hours | \n
| Setup & model loading | \n~20 minutes | \n
| Effective inference window | \n~280 minutes | \n
| Base per-question budget | \n~320 seconds + buffer borrowing | \n
| Global buffer | \n350 seconds (dynamically updated) | \n
\nThree levels of benchmarking were used, applied in the order below as each revealed\na gap that motivated the next.\n
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1563363%2F59e78444f7f0314db3ae36fc745f4ead%2FBenchmarking.png?generation=1778943755757867&alt=media\")
\nThe first benchmark consisted of 20 problems drawn from the AIMO 2 and AIMO 3\nreference problem sets. The model scored 95% accuracy (19/20).\n
\n| Metric | \nValue | \n
|---|---|
| Total problems | \n20 | \n
| Correct answers | \n19/20 (95%) | \n
| Incorrect answer | \nProblem #10 (model: 29,443; actual: 8,687) | \n
| Avg time per problem | \n230 seconds | \n
| Avg tokens generated | \n13,543 tokens | \n
\nAfter observing the gap between 90–95% reference accuracy and the 42% leaderboard score,\na harder benchmark was constructed from 49 problems sourced from IMO, AMC/AIME,\nand similar archives.\n
\n| Property | \nValue | \n
|---|---|
| Total problems | \n49 | \n
| Problem sources | \n\n |
| Model accuracy | \n65% | \n
Internal BenchMark Dataset - https://huggingface.co/HarpreetK/AIMO3/blob/main/Internal_Benchmark_Final.xlsx\nSources - AOPS,https://matharena.ai/apex/, ARML, HMMT
\n\nA targeted 10-problem subset was extracted consisting of questions where the previous\nattempt had either returned 29443 (timeout/extraction failure) or produced an incorrect answer.\n
\n| Benchmark | \nScore | \nNotes | \n
|---|---|---|
| AIMO Reference (20) | \n95% (19/20) | \nToo easy; gap not explained | \n
| Internal Benchmark (49) | \n65% | \nCloser to AIMO 3 difficulty | \n
| AIMO 3 Leaderboard (50) | \n21/50 (42%) | \nActual submission | \n
| Best config (50) | \n24/50 (48%) | \nNot submitted — library deletion | \n
Flow Diagram is attached. Please open it independently in a new window to zoom on it
\n\nA modified version of the NemoSkills repository (winner of AIMO 2) was used as the baseline,\nproviding a scaffold for request scheduling and Python code execution sandboxing.\nMCP and LiteLLM abstraction layers were stripped to reduce overhead.\n
\n\n\nNote: These were conclusions based on research. Ideally, both approaches would have been compared directly.\n\n
\n\nThe BufferBorrower class implements a time-pressure-based allocation strategy.\nBorrowable time differed across notebooks:\n
\n\nThe final best-performing approach used a single-phase prompt applied uniformly to all 50 problems.\n
\n\n\nThe key jump was from score 18 (Step-Back) to score 24 (Non-Step-Back):\na 6-point improvement.\n\n
\n\nUp to 8 self-consistent samples are generated per problem.\nMajority voting uses frequency count only.\n
\n\nMetric Used: Exact Match Accuracy.\n
\n\npython -m nemo_skills.inference.server.serve_vllm \\\n --model=.../gpt-oss-120b/.../1 --port=5000 --num_gpus=1 \\\n --max_model_len=38000 --max_num_batched_tokens=65000 \\\n --max_num_seqs=11 --max-cudagraph-capture-size=2048 \\\n --gpu_memory_utilization=0.95 --kv_cache_dtype=fp8_e4m3 \\\n --enable-prefix-caching --async-scheduling\n| Parameter | \nValue | \nRationale | \n
|---|---|---|
| max_tokens | \n38,000 | \nUpper bound per problem | \n
| tokens_to_generate | \n35,940 | \nmax_tokens minus overhead | \n
| max_batch_size | \n8 | \nNumber of self-consistent samples | \n
| global_buffer | \n350 seconds | \nReserve for hard problems | \n
| temperature | \n1.0 | \nPer NVIDIA NeMo recommendation | \n
| top_p | \n1.0 | \nPer NVIDIA NeMo recommendation | \n
All 15 configurations tested.
\n\nGPT-OSS alternates between full attention and sliding window attention (SWA).\n
\n\nSliding window sizes 128, 200, and 8,000 were tested.\n
\n\nTemperature fixed at 1.0 (NVIDIA recommendation).\n
\n| Parameter | \nScore 18 (SB) | \nScore 20 | \nScore 21 (SUB) | \nScore 24 (Best) | \n
|---|---|---|---|---|
| Prompting | \nStep-Back full | \nNon-SB + dual | \nNon-Step-Back | \nNon-Step-Back | \n
| Temperature | \nFixed 1.0 | \nVaried | \nFixed 1.0 | \nFixed 1.0 | \n
| max_model_len | \n65,536 | \n36,000 | \n38,000 | \n80,000 | \n
| Leaderboard score | \n18/50 | \n20/50 | \n21/50 | \n24/50 | \n
\nReduced token budget caused score drop from 24 to 21.\n
\n\nDual-prompt strategy and varied-temperature sampling hurt performance.\n
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1563363%2F98ad341eccad2211873d4d0c07f0e423%2FSOTA%20Comparison.png?generation=1778953075190961&alt=media\")
Below models were added to SOTA Comparison\nQwen (without tool calling),\nGemini Pro 3.1,\nand Claude Opus 4.7.\nOne particularly interesting observation was that Gemini Pro 3.1 and Claude Opus 4.7 appear to be highly complementary systems.
\nTogether, the two models successfully solved every question in my curated evaluation dataset.
\nMore specifically:
\nClaude demonstrated stronger mathematical reasoning overall,\nespecially on difficult mathematical problems,\nwhile both models solved easier problems reliably,\nand on harder problems, each model solved cases that the other failed to solve.
\n\nHuggingFace repo:\n\nhttps://huggingface.co/HarpreetK/AIMO3\n\n
\n| Artifact | \nLocation / Value | \nStatus | \n
|---|---|---|
| Base model | \nGPT-OSS-120B | \nProvided | \n
| Submitted notebook | \ngpt-oss-score21.ipynb | \nProvided | \n
| Best notebook (24) | \ngpt-oss-nostepback-score24.ipynb | \nProvided | \n
| Backend | \nMarlin (via vLLM 0.11.2) | \nProvided | \n
refer the pdf for the same
", "comments": [ { "id": 3459320, "author_name": "Preet", "author_url": "", "post_date": "2026-05-17T16:48:53.103000", "content": "\n", "votes": 0, "replies": [] }, { "id": 3458323, "author_name": "Sam Bealing", "author_url": "", "post_date": "2026-05-15T16:15:36.507000", "content": "The final pdf is updated. Please do the refer the writeup here. I did fix few sections here on the writeup but formatting here is much tougher than in a word document. This writeup does not contain the full list of edits only the pdf does. So please check the v2 pdf attached to this writeup section. The edits are mentioned on page 1 and against the sections also. Just to save time - all the edits in the pdf except \"sections 5.2 Added Note, 8(last point),12 (Prompt Appendix)\" are already part of the replies that I gave to you both below. So those edits do not provide any new information\n
\n
\n Huggingface repo is also updated
Following up on the questions from @philipvonderlind, here are some additional questions to help us assess your submission:
\nLook forward to reading your responses on these.
", "votes": 0, "replies": [ { "id": 3458701, "author_name": "Preet", "author_url": "", "post_date": "2026-05-16T13:13:15.077000", "content": "Context Setting
\n\nPhase 1 - For the Step-Back approach, three time buckets were defined based on the estimated difficulty level of the problem. Considering the total competition time available, the average time allocated per question could not realistically exceed approximately 320–340 seconds.\n
\n\nFrom this baseline allocation, the additional time required for the Step-Back phase was subtracted. The Step-Back phase refers to an initial pass where the model is asked only to:\n
\n\nBased on the difficulty level predicted by the model, a corresponding baseline time budget was assigned. Each problem could receive time from only one of predefined categories, as shown in the corresponding if-else logic.\n
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1563363%2F2426fefee5f859d596c369fa59066f4f%2Fif-else.png?generation=1778937574722126&alt=media\")
\nOnce the stepback approach is completed, the time-counter indicates how much time was consumed. And this time is subtracted from the baseline time (one of the 3 categories defined above) and a fraction of the global buffer was added to get the net effective inference time (baseline - stepback_time + fraction_of_buffer). (How to compute fraction which gets multiplied with global buffer is indicated in my answer to Philip's question -please refer that)\n
\nAlso a dynamic prompt was chosen based on phase 1 (domain and techniques required to solve the problem)\n
\nPhase 2 - The effective inference time and dynamic prompt was passed onto this phase of the problem solving where actual answer was produced\n(StepBack Time and tokens were capped to a maximum so that it does not eat into problem solving time)\n
\nOnce the question is attempted the actual execution time is measured using time.perf_counter(). The measured time is then compared against the allocated budget:\n
\n\nThe overall philosophy remained similar, but the implementation was intentionally simplified.\n
\n\nAIMO3 was expected to contain increasingly difficult problems. In practice, distinguishing reliably between “hard,” “harder,” and “hardest” problems before fully attempting them proved difficult. Because of this, dynamically assigning substantially different budgets based on predicted difficulty was not considered reliable. \n
\n
\nInstead of categorizing problems by difficulty, a flat baseline of 320 seconds per question was used. \n
\nThe reasoning behind this decision was that harder problems did not necessarily benefit from significantly larger time allocations. Empirically, if the model was capable of solving a problem correctly, it typically did so within roughly the same time range.\n
\nAnyways the unused time would become part of the global buffer.\n
\nAs with the Step-Back approach, the actual solving time for each question was also tracked using time.perf_counter() and compared against the allocated budget (baseline + fraction_of_buffer):\n
The difference between this and Non-StepBack Approach was how the fraction of global buffer was calculated and how the baseline budget was determined - For Non-StepBack - flat 320 seconds were allocated as opposed to the 3 categories defined in stepback approach\n
\nActual Answer of the question
\nThe additional time assigned to a question was capped at the minimum of:\n
\n
Since time per question and global buffer is checked on each question. Therefore, that prevents it from running out on later questions. This could also result in case where global buffer rans out. But if that happens then the addition time from the global buffer cannot be borrowed since it is 0 and the baseline time which is roughly approximate time per question would serve as a hard limit \nDetailed Explanation is given to Philip in the comments below to compute time pressure and global buffer borrowing in Answer 3\nSide Note: Different buffer caps were tested across the few submissions I made
\n\nI would like to clarify one point regarding the token budget. The token budget was not exactly doubled; rather, it was increased by approximately 22%.\n
\n\nIn my view, the improvement in performance was likely driven more by the additional time available for solving each question than by the increase in token budget itself.\n
\n\nFor reference, notebooks that scored in the 40s using the GPT-OSS model were operating with a context length of 65,536 tokens, and I used the same context length in my notebook that scored 18.\n
\n\nLess time available per question was one of the reasons why the Step-Back approach was not used in later iterations. I had initial reservations about it but wanted to check if stepback phase resulted in quick convergence to answer compensating for the time consumed during phase 1 and also better accuracy.\n
\n\nThese were actual problems taken from official IMO-style mathematics competitions. I handpicked the problems from both Hugging Face dataset repositories(this also had difficulty level assigned) and the official competition websites. None of the internal benchmark data was synthetically generated; therefore, the possibility of ground-truth labeling errors is close to 0\nSources Used\nMathArena (https://matharena.ai/apex/)\nAOPS\nARML2 & ARML3\nHMMT\nIMO\n
\n\nThe filtering criteria were based on two primary factors:\n
\n\nThe final set of 10 problems was a subset of this curated dataset and primarily consisted of questions for which the model either failed to generate the correct answer or ran out of allocated solving time.\n
\n\nThe objective was not to continuously expand the benchmark dataset, but rather to evaluate the model’s behavior and limitations on a carefully curated set of difficult problems.\n
", "votes": 0, "replies": [] } ] }, { "id": 3458097, "author_name": "Preet", "author_url": "", "post_date": "2026-05-15T05:04:32.700000", "content": "Hello @sambealing,
\nI see that Philip have tagged you for questions but I do not see any questions from you I'd be happy to answer any questions that you might have.
\nSincerely,\nHarpreet
", "votes": 0, "replies": [ { "id": 3458324, "author_name": "Sam Bealing", "author_url": "", "post_date": "2026-05-15T16:16:01.643000", "content": "Just posted questions. I'm not as quick as Philip getting all my questions written 😉
", "votes": 0, "replies": [] } ] }, { "id": 3457841, "author_name": "Philip Vonderlind", "author_url": "", "post_date": "2026-05-14T13:04:47.323000", "content": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post a few questions below to help us assess your submission more closely, as described in the updated discussion post.
\nFormatting: The formatting of this post may have become distorted when copied from your project write-up. If possible, please fix it going forward, since we cannot refer to the old copy.
Attention Backend: I noticed that you ran several ablations on the serving setup. However, it seems that two concepts may have been mixed together: GPU kernels, such as Marlin or Triton, and inference engines, such as vLLM or TensorRT-LLM. Could you clarify what exactly was tested? For example, was it vLLM + Triton kernel, vLLM + Marlin kernel, or another configuration? Additionally, why did you not test FlashAttention kernels, which are often faster in practice than Triton and are directly supported in vLLM?
Architecture Details: A core contribution of your work appears to be the buffer-borrowing and step-back components introduced in Sections 4.4 and 4.5 of the PDF, respectively. However, these components are currently described only at a high level. Could you add a more detailed explanation of how they work, what issues they are intended to address, and what motivated their design?
Looking forward to your response :)
", "votes": 0, "replies": [ { "id": 3457960, "author_name": "Preet", "author_url": "", "post_date": "2026-05-14T17:32:24.490000", "content": "Hi Philip,
\nThank you for your questions. Could you please tell me if I have to provide answers directly here or in the pdf.
\nSincerely!
", "votes": 0, "replies": [ { "id": 3458322, "author_name": "Sam Bealing", "author_url": "", "post_date": "2026-05-15T16:14:35.790000", "content": "Please could you answer the questions here for visibility. If you are also able to update your write up, then that is also helpful so everything is in one place. Please read Frieder's comments about edits before doing this.
", "votes": 0, "replies": [] } ] }, { "id": 3458705, "author_name": "Preet", "author_url": "", "post_date": "2026-05-16T13:17:56.983000", "content": "Formatting - I fixed it. Thank you for pointing it out.\n Side Note: The overall system design could not be clearly included in the PDF because the diagram became too small. Therefore, the flow diagram has been attached separately so it can be zoomed in and viewed properly.\nThis was already part of the original writeup — just highlighting it here in case it was overlooked.
\n\nYes, several backend-level ablations and experiments were conducted.\n
\n\nOne major component of the backend was the inference engine itself. Few inference frameworks were evaluated:\n
\n\nInitial experiments with TensorRT-LLM. This could be easily implemented using docker but in Kaggle environment docker could not be used. I tried to implement workaround but I faced challenges while implementing the workaround. I finally did not pursue this route when I faced datatype mismatch issues with gpt-oss and tensorrt-llm framework
\n\nThe issue may or may not be fixable, since TensorRT-LLM generally works well with many large models. I spent some time attempting to make it work, but due to competition time constraints, I decided not to pursue this direction further.\nIt may still be possible to get GPT-OSS running successfully with additional engineering effort since gpt-oss work with tensorrt installed using docker\n
\n\nAnother important backend component involved the CUDA kernels used internally within the transformer architecture.\n
\n\nIf you inspect the runtime logs, you will notice that. This is a snippet from vllm logs\n
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1563363%2F6263e95693f01da4278972702d1e1a19%2Fmarlin%20backend.png?generation=1778940390118017&alt=media\")
Using FLASH_ATTNUsing Marlin Backend\nTo clarify, my submitted code does use FLASH Attention. However flash and marlin correspond to different optimized CUDA kernels and execution backends used for accelerating transformer operations such as attention layers, matrix multiplications, and quantized inference workloads.\n
\n\nPlease refer the architecture below to understand where each of them is used. The parts of the transformer where Marlin and Flash are used is given below\n
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1563363%2Fecf0d43f2f65690e43c6af0365fb1801%2Fgpt_oss_120b_layer_with_kernels.svg?generation=1778938778997700&alt=media\")
Marlin performed better than Triton in this setup.\nThis is what was indicated in the table that I added to section 5.2 and Figure 6.
\n\nFor the attention component of the transformer, one of the research papers indicated that GPT-OSS-120B could operate using four different attention backends.\n
\n\nAt the time I conducted these experiments, I had not yet seen the research you shared regarding FLASH Attention outperforming Triton. Nevertheless, it was still valuable to experimentally validate which backend performed best for this specific inference pipeline and use case.\n
\n\nThe following attention backends were evaluated:\n
\n\nKey Observation:
\nFLASH Attention outperformed Triton in my experiments. The remaining two attention types were ultimately not usable due to lack of support for sink-token configuration.\n
\nI missed including this information in my original write-up, so thank you for asking the question. It gave me an opportunity to clarify these backend-level experiments and observations.\n
\n\nUnder normal transformer behavior, each token primarily attends to semantically relevant tokens within the sequence.\n
\n\nHowever, in practice, certain tokens tend to attract disproportionately high attention from many other tokens. These are commonly referred to as sink tokens.\n
\n\nExamples of sink tokens can include:\n
\n\nSink tokens can sometimes improve:\n
\n\nThe sink-token configuration was already built into the GPT-OSS model itself. I intentionally chose not to disable it while testing alternative attention backends because I was concerned that changing this behavior could negatively impact model performance and long-context stability.\n
\n\nThe following logic was used to dynamically estimate time pressure and determine how much additional buffer time could be allocated to a question during inference.\n
\n\n The compute_time_pressure() function estimated the current level of time pressure based on:\n
\n The core idea was to compare:\n
\ntotal_available_time / total_questions)remaining_time / remaining_questions)\n A pressure score was then computed as:\n
\n\n To avoid extreme instability, the final pressure value was clamped between:\n
\n0.3 (minimum pressure)1.5 (maximum pressure)\n The compute_borrow_fraction() function determined what fraction of the global time buffer\n could be borrowed for the current question. The borrowing decision depended on several factors:\n
\n Time Ratio (c_ratio)
\n Measures how much of the allocated Step-Back time budget was consumed.\n
\n Token Ratio (s_ratio)
\n Measures how much of the allocated Step-Back token budget was consumed.\n
\n Mathematical Logic:\n
\n\n Two normalized ratios were first computed.\n These two signals were then combined using a Euclidean-style formulation:\n
\ncombined_signal = sqrt((c_ratio² + s_ratio²) / 2)\n\n
\n This produced a unified measure representing overall computational intensity for the current problem.\n
\n\n The system then adjusted this value using the previously computed time-pressure signal:\n
\n\n Finally, the computed borrow fraction was constrained within predefined limits:\n
\n0.0b_maxdef compute_time_pressure(\n self,\n remaining_time: float,\n questions_completed: int,\n ) -> float:\n remaining_q = max(1, self.total_questions - questions_completed)\n if remaining_time <= 0:\n return 1.5\n ideal_pace = self.total_available_time / self.total_questions\n available_pace = remaining_time / remaining_q\n pressure = ideal_pace / available_pace\n return max(0.3, min(1.5, pressure))\n def compute_borrow_fraction(\n self,\n remaining_time: float,\n questions_completed: int,\n stepback_time: float,\n stepback_time_budget: float,\n stepback_tokens: int,\n stepback_tokens_budget: int,\n global_buffer: float = 0.0,\n ) -> float:\n \"\"\"\n Compute what fraction of the buffer to borrow.\n Returns\n -------\n float\n Borrow fraction in [0, b_max].\n \"\"\"
\n c_ratio = min(1.0, stepback_time / stepback_time_budget) if stepback_time > 0 else 0.5\n s_ratio = stepback_tokens / stepback_tokens_budget if stepback_tokens > 0 else 0.0\n combined_signal = math.sqrt((c_ratio ** 2 + s_ratio ** 2) / 2)\n sigmoid_out = self._sigmoid(combined_signal)\n pressure = self.compute_time_pressure(remaining_time, questions_completed)\n pressure_mult = 1.0 / pressure\n borrow_fraction = self.b_max * combined_signal * pressure_mult\n borrow_fraction = max(0.0, min(self.b_max, borrow_fraction))\n:-
\n– Much of work in the field of Artificial Intelligence and Machine Learning is inspired from how humans function. Neural network corresponds to the neurons in the brain. “Attention is all you need” is how our brains process lengthy information. Therefore, my rationale was that if us humans get guidance then we perform a task better so the llm model can behave similarly. \n
\nFurther, the principles identified during stepback phase 1 formed the basis of dynamic prompting. Prompting based on domains and techniques used would in my opinion result in better outcomes and I wanted to try this further. However, phase 1 took away from the total time to solve the problem. Therefore, this approach had to be discarded.
\nIn addition, I would like to highlight three further contributions and observations that were either underemphasized in the original write-up or significantly strengthened through post-submission experiments.\n
\n\nvLLM generates CUDA graphs for two distinct execution phases:\n
\n\nThese two phases exhibit very different shape distributions.\n
\n\nThe mixed prefill+decode phase experiences progressively increasing input lengths due to:\n
\n\nIn contrast, the pure decode phase typically generates one token at a time and therefore operates under a very different execution profile.\n
\n\nTable in Section 5.2 already includes an initial ablation study on graph capture sizing. After submission, I conducted additional experiments to support a hypothesis I had originally held on intuition alone — one I did not assert in version 1 of the write-up due to the lack of data i.e .
\nIf graph_size was equal to the max-num-seqs parameter passed to vllm inference server then throughput increased atleast by 65%. This maybe due to decode graph matching the number of sequences
\n\nAnother implementation detail omitted from the original write-up: the pipeline uses streaming responses rather than single-shot generation. Non Streaming Responses resulted in frequent read timeout and connection timeout issues\n
\nThis was part of the work that went into building the pipeline, but I forgot to mention it\n
\nDuring experimentation, non-streaming inference frequently resulted in:\n
\n\nSwitching to streaming-based generation resolved these issues and significantly improved runtime stability.\n
\n\nThe revised evaluation now includes additional comparisons against:\n
\n\nOne particularly interesting observation was that Gemini Pro 3.1 and Claude Opus 4.7 appear to be highly complementary systems.\n
\n\nTogether, the two models successfully solved every question in my curated evaluation dataset at pass@1.\n
\nHowever, Claude demonstrated stronger mathematical reasoning than Gemini 3.1 Pro, especially on difficult mathematical problems. While both models solved easier problems reliably and on harder problems, each model solved cases that the other failed to solve.\n
\n
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1563363%2Fb1266b4ed3af90b5d6cbab51f7c2df76%2Fperformance_comparison_updated_2.png?generation=1779036696326203&alt=media\")
\n", "rawMarkdown": ">The final pdf is updated. Please do the refer the writeup here. I did fix few sections here on the writeup but formatting here is much tougher than in a word document. This writeup does not contain the full list of edits only the pdf does. So please check the v2 pdf attached to this writeup section. The edits are mentioned on page 1 and against the sections also. Just to save time - all the edits in the pdf except \"sections 5.2 Added Note, 8(last point),12 (Prompt Appendix)\" are already part of the replies that I gave to you both below. So those edits do not provide any new information\n
\n
\n Huggingface repo is also updated
Following up on the questions from @philipvonderlind, here are some additional questions to help us assess your submission:
\nLook forward to reading your responses on these.
", "rawMarkdown": "Following up on the questions from @philipvonderlind, here are some additional questions to help us assess your submission:\n\n1. **Global time buffer mechanics:** The writeup mentions a global buffer of 350 seconds that's dynamically updated and allows time borrowing from future problems. Could you describe in more detail how the borrowing logic decides when to lend time vs hold it back, and what safeguards prevent the run from running out of time on later problems (particularly if the harder problems happen to fall near the end of the run)?\n2. **Step-back removal vs token budget:** The 18→24 improvement bundles Step-Back removal and a token budget doubling. You explicitly note that you can't isolate their individual contributions, but from development you presumably have some intuition. Informally, which of the two felt like the larger contributor?\n3. **Internal benchmark construction and answer accuracy:** Beyond the AIMO reference set, you reference an internal benchmark and a 10-problem failing subset used during development. Could you describe how you collected and curated these (sources, filtering criteria), and what steps you took to verify the ground-truth answers? Across the maths corpus prize submissions we've reviewed, even widely-used datasets often contain ground-truth errors that can quietly distort ablation conclusions.\n\nLook forward to reading your responses on these.\n", "replies": [ { "id": 3458701, "postDate": "2026-05-16T13:13:15.077Z", "content": "Context Setting
\n\nPhase 1 - For the Step-Back approach, three time buckets were defined based on the estimated difficulty level of the problem. Considering the total competition time available, the average time allocated per question could not realistically exceed approximately 320–340 seconds.\n
\n\nFrom this baseline allocation, the additional time required for the Step-Back phase was subtracted. The Step-Back phase refers to an initial pass where the model is asked only to:\n
\n\nBased on the difficulty level predicted by the model, a corresponding baseline time budget was assigned. Each problem could receive time from only one of predefined categories, as shown in the corresponding if-else logic.\n
\n
\nOnce the stepback approach is completed, the time-counter indicates how much time was consumed. And this time is subtracted from the baseline time (one of the 3 categories defined above) and a fraction of the global buffer was added to get the net effective inference time (baseline - stepback_time + fraction_of_buffer). (How to compute fraction which gets multiplied with global buffer is indicated in my answer to Philip's question -please refer that)\n
\nAlso a dynamic prompt was chosen based on phase 1 (domain and techniques required to solve the problem)\n
\nPhase 2 - The effective inference time and dynamic prompt was passed onto this phase of the problem solving where actual answer was produced\n(StepBack Time and tokens were capped to a maximum so that it does not eat into problem solving time)\n
\nOnce the question is attempted the actual execution time is measured using time.perf_counter(). The measured time is then compared against the allocated budget:\n
\n\nThe overall philosophy remained similar, but the implementation was intentionally simplified.\n
\n\nAIMO3 was expected to contain increasingly difficult problems. In practice, distinguishing reliably between “hard,” “harder,” and “hardest” problems before fully attempting them proved difficult. Because of this, dynamically assigning substantially different budgets based on predicted difficulty was not considered reliable. \n
\n
\nInstead of categorizing problems by difficulty, a flat baseline of 320 seconds per question was used. \n
\nThe reasoning behind this decision was that harder problems did not necessarily benefit from significantly larger time allocations. Empirically, if the model was capable of solving a problem correctly, it typically did so within roughly the same time range.\n
\nAnyways the unused time would become part of the global buffer.\n
\nAs with the Step-Back approach, the actual solving time for each question was also tracked using time.perf_counter() and compared against the allocated budget (baseline + fraction_of_buffer):\n
The difference between this and Non-StepBack Approach was how the fraction of global buffer was calculated and how the baseline budget was determined - For Non-StepBack - flat 320 seconds were allocated as opposed to the 3 categories defined in stepback approach\n
\nActual Answer of the question
\nThe additional time assigned to a question was capped at the minimum of:\n
\n
Since time per question and global buffer is checked on each question. Therefore, that prevents it from running out on later questions. This could also result in case where global buffer rans out. But if that happens then the addition time from the global buffer cannot be borrowed since it is 0 and the baseline time which is roughly approximate time per question would serve as a hard limit \nDetailed Explanation is given to Philip in the comments below to compute time pressure and global buffer borrowing in Answer 3\nSide Note: Different buffer caps were tested across the few submissions I made
\n\nI would like to clarify one point regarding the token budget. The token budget was not exactly doubled; rather, it was increased by approximately 22%.\n
\n\nIn my view, the improvement in performance was likely driven more by the additional time available for solving each question than by the increase in token budget itself.\n
\n\nFor reference, notebooks that scored in the 40s using the GPT-OSS model were operating with a context length of 65,536 tokens, and I used the same context length in my notebook that scored 18.\n
\n\nLess time available per question was one of the reasons why the Step-Back approach was not used in later iterations. I had initial reservations about it but wanted to check if stepback phase resulted in quick convergence to answer compensating for the time consumed during phase 1 and also better accuracy.\n
\n\nThese were actual problems taken from official IMO-style mathematics competitions. I handpicked the problems from both Hugging Face dataset repositories(this also had difficulty level assigned) and the official competition websites. None of the internal benchmark data was synthetically generated; therefore, the possibility of ground-truth labeling errors is close to 0\nSources Used\nMathArena (https://matharena.ai/apex/)\nAOPS\nARML2 & ARML3\nHMMT\nIMO\n
\n\nThe filtering criteria were based on two primary factors:\n
\n\nThe final set of 10 problems was a subset of this curated dataset and primarily consisted of questions for which the model either failed to generate the correct answer or ran out of allocated solving time.\n
\n\nThe objective was not to continuously expand the benchmark dataset, but rather to evaluate the model’s behavior and limitations on a carefully curated set of difficult problems.\n
", "rawMarkdown": "\n\n\nPhase 1 - For the Step-Back approach, three time buckets were defined based on the estimated difficulty level of the problem. Considering the total competition time available, the average time allocated per question could not realistically exceed approximately 320–340 seconds.\n
\n\n\nFrom this baseline allocation, the additional time required for the Step-Back phase was subtracted. The Step-Back phase refers to an initial pass where the model is asked only to:\n
\n\n\nBased on the difficulty level predicted by the model, a corresponding baseline time budget was assigned. Each problem could receive time from only one of predefined categories, as shown in the corresponding if-else logic.\n
\n\n\n\n\nOnce the stepback approach is completed, the time-counter indicates how much time was consumed. And this time is subtracted from the baseline time (one of the 3 categories defined above) and a fraction of the global buffer was added to get the net effective inference time (baseline - stepback_time + fraction_of_buffer). (How to compute fraction which gets multiplied with global buffer is indicated in my answer to Philip's question -please refer that)\n
\nPhase 2 - The effective inference time and dynamic prompt was passed onto this phase of the problem solving where actual answer was produced\n(StepBack Time and tokens were capped to a maximum so that it does not eat into problem solving time)\n
\n\nOnce the question is attempted the actual execution time is measured using time.perf_counter(). The measured time is then compared against the allocated budget:\n\n\n\n\n\nThe overall philosophy remained similar, but the implementation was intentionally simplified.\n
\n\n\n\n AIMO3 was expected to contain increasingly difficult problems. In practice, distinguishing reliably between “hard,” “harder,” and “hardest” problems before fully attempting them proved difficult. Because of this, dynamically assigning substantially different budgets based on predicted difficulty was not considered reliable. \n
\n
\nInstead of categorizing problems by difficulty, a flat baseline of 320 seconds per question was used. \n\n
\nThe reasoning behind this decision was that harder problems did not necessarily benefit from significantly larger time allocations. Empirically, if the model was capable of solving a problem correctly, it typically did so within roughly the same time range.\n\n
\nAnyways the unused time would become part of the global buffer.\n
\nAs with the Step-Back approach, the actual solving time for each question was also tracked using time.perf_counter() and compared against the allocated budget (baseline + fraction_of_buffer):\n
\nThe additional time assigned to a question was capped at the minimum of:\n
\n
\nI would like to clarify one point regarding the token budget. The token budget was not exactly doubled; rather, it was increased by approximately 22%.\n
\n\n\nIn my view, the improvement in performance was likely driven more by the additional time available for solving each question than by the increase in token budget itself.\n
\n\n\nFor reference, notebooks that scored in the 40s using the GPT-OSS model were operating with a context length of 65,536 tokens, and I used the same context length in my notebook that scored 18.\n
\n\n\nLess time available per question was one of the reasons why the Step-Back approach was not used in later iterations. I had initial reservations about it but wanted to check if stepback phase resulted in quick convergence to answer compensating for the time consumed during phase 1 and also better accuracy.\n
\n\n\nThese were actual problems taken from official IMO-style mathematics competitions. I handpicked the problems from both Hugging Face dataset repositories(this also had difficulty level assigned) and the official competition websites. None of the internal benchmark data was synthetically generated; therefore, the possibility of ground-truth labeling errors is close to 0\nSources Used\nMathArena (https://matharena.ai/apex/)\nAOPS\nARML2 & ARML3\nHMMT\nIMO\n\n
\n\n\nThe filtering criteria were based on two primary factors:\n
\n\n\nThe final set of 10 problems was a subset of this curated dataset and primarily consisted of questions for which the model either failed to generate the correct answer or ran out of allocated solving time.\n
\n\n\nThe objective was not to continuously expand the benchmark dataset, but rather to evaluate the model’s behavior and limitations on a carefully curated set of difficult problems.\n
" } ] }, { "id": 3458097, "postDate": "2026-05-15T05:04:32.700Z", "content": "Hello @sambealing,
\nI see that Philip have tagged you for questions but I do not see any questions from you I'd be happy to answer any questions that you might have.
\nSincerely,\nHarpreet
", "rawMarkdown": "Hello @sambealing,\n\n I see that Philip have tagged you for questions but I do not see any questions from you I'd be happy to answer any questions that you might have.\n\nSincerely,\nHarpreet", "replies": [ { "id": 3458324, "postDate": "2026-05-15T16:16:01.643Z", "content": "Just posted questions. I'm not as quick as Philip getting all my questions written 😉
", "rawMarkdown": "Just posted questions. I'm not as quick as Philip getting all my questions written 😉" } ] }, { "id": 3457841, "postDate": "2026-05-14T13:04:47.323Z", "content": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post a few questions below to help us assess your submission more closely, as described in the updated discussion post.
\nFormatting: The formatting of this post may have become distorted when copied from your project write-up. If possible, please fix it going forward, since we cannot refer to the old copy.
Attention Backend: I noticed that you ran several ablations on the serving setup. However, it seems that two concepts may have been mixed together: GPU kernels, such as Marlin or Triton, and inference engines, such as vLLM or TensorRT-LLM. Could you clarify what exactly was tested? For example, was it vLLM + Triton kernel, vLLM + Marlin kernel, or another configuration? Additionally, why did you not test FlashAttention kernels, which are often faster in practice than Triton and are directly supported in vLLM?
Architecture Details: A core contribution of your work appears to be the buffer-borrowing and step-back components introduced in Sections 4.4 and 4.5 of the PDF, respectively. However, these components are currently described only at a high level. Could you add a more detailed explanation of how they work, what issues they are intended to address, and what motivated their design?
Looking forward to your response :)
", "rawMarkdown": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post a few questions below to help us assess your submission more closely, as described in the [updated discussion post](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/689703).\n\n1. **Formatting**: The formatting of this post may have become distorted when copied from your project write-up. If possible, please fix it going forward, since we cannot refer to the old copy.\n\n2. **Attention Backend**: I noticed that you ran several ablations on the serving setup. However, it seems that two concepts may have been mixed together: GPU kernels, such as Marlin or Triton, and inference engines, such as vLLM or TensorRT-LLM. Could you clarify what exactly was tested? For example, was it `vLLM + Triton kernel`, `vLLM + Marlin kernel`, or another configuration? Additionally, why did you not test [FlashAttention kernels](https://github.com/dao-ailab/flash-attention), which are often faster in practice than Triton and are directly supported in vLLM?\n\n3. **Architecture Details**: A core contribution of your work appears to be the buffer-borrowing and step-back components introduced in Sections `4.4` and `4.5` of the PDF, respectively. However, these components are currently described only at a high level. Could you add a more detailed explanation of how they work, what issues they are intended to address, and what motivated their design?\n\nLooking forward to your response :)", "replies": [ { "id": 3457960, "postDate": "2026-05-14T17:32:24.490Z", "content": "Hi Philip,
\nThank you for your questions. Could you please tell me if I have to provide answers directly here or in the pdf.
\nSincerely!
", "rawMarkdown": "Hi Philip,\n\nThank you for your questions. Could you please tell me if I have to provide answers directly here or in the pdf.\n\nSincerely!", "replies": [ { "id": 3458322, "postDate": "2026-05-15T16:14:35.790Z", "content": "Please could you answer the questions here for visibility. If you are also able to update your write up, then that is also helpful so everything is in one place. Please read Frieder's comments about edits before doing this.
", "rawMarkdown": "Please could you answer the questions here for visibility. If you are also able to update your write up, then that is also helpful so everything is in one place. Please read [Frieder's comments](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/699061) about edits before doing this." } ] }, { "id": 3458705, "postDate": "2026-05-16T13:17:56.983Z", "content": "Formatting - I fixed it. Thank you for pointing it out.\n Side Note: The overall system design could not be clearly included in the PDF because the diagram became too small. Therefore, the flow diagram has been attached separately so it can be zoomed in and viewed properly.\nThis was already part of the original writeup — just highlighting it here in case it was overlooked.
\n\nYes, several backend-level ablations and experiments were conducted.\n
\n\nOne major component of the backend was the inference engine itself. Few inference frameworks were evaluated:\n
\n\nInitial experiments with TensorRT-LLM. This could be easily implemented using docker but in Kaggle environment docker could not be used. I tried to implement workaround but I faced challenges while implementing the workaround. I finally did not pursue this route when I faced datatype mismatch issues with gpt-oss and tensorrt-llm framework
\n\nThe issue may or may not be fixable, since TensorRT-LLM generally works well with many large models. I spent some time attempting to make it work, but due to competition time constraints, I decided not to pursue this direction further.\nIt may still be possible to get GPT-OSS running successfully with additional engineering effort since gpt-oss work with tensorrt installed using docker\n
\n\nAnother important backend component involved the CUDA kernels used internally within the transformer architecture.\n
\n\nIf you inspect the runtime logs, you will notice that. This is a snippet from vllm logs\n
\n
Using FLASH_ATTNUsing Marlin Backend\nTo clarify, my submitted code does use FLASH Attention. However flash and marlin correspond to different optimized CUDA kernels and execution backends used for accelerating transformer operations such as attention layers, matrix multiplications, and quantized inference workloads.\n
\n\nPlease refer the architecture below to understand where each of them is used. The parts of the transformer where Marlin and Flash are used is given below\n
\n
Marlin performed better than Triton in this setup.\nThis is what was indicated in the table that I added to section 5.2 and Figure 6.
\n\nFor the attention component of the transformer, one of the research papers indicated that GPT-OSS-120B could operate using four different attention backends.\n
\n\nAt the time I conducted these experiments, I had not yet seen the research you shared regarding FLASH Attention outperforming Triton. Nevertheless, it was still valuable to experimentally validate which backend performed best for this specific inference pipeline and use case.\n
\n\nThe following attention backends were evaluated:\n
\n\nKey Observation:
\nFLASH Attention outperformed Triton in my experiments. The remaining two attention types were ultimately not usable due to lack of support for sink-token configuration.\n
\nI missed including this information in my original write-up, so thank you for asking the question. It gave me an opportunity to clarify these backend-level experiments and observations.\n
\n\nUnder normal transformer behavior, each token primarily attends to semantically relevant tokens within the sequence.\n
\n\nHowever, in practice, certain tokens tend to attract disproportionately high attention from many other tokens. These are commonly referred to as sink tokens.\n
\n\nExamples of sink tokens can include:\n
\n\nSink tokens can sometimes improve:\n
\n\nThe sink-token configuration was already built into the GPT-OSS model itself. I intentionally chose not to disable it while testing alternative attention backends because I was concerned that changing this behavior could negatively impact model performance and long-context stability.\n
\n\nThe following logic was used to dynamically estimate time pressure and determine how much additional buffer time could be allocated to a question during inference.\n
\n\n The compute_time_pressure() function estimated the current level of time pressure based on:\n
\n The core idea was to compare:\n
\ntotal_available_time / total_questions)remaining_time / remaining_questions)\n A pressure score was then computed as:\n
\n\n To avoid extreme instability, the final pressure value was clamped between:\n
\n0.3 (minimum pressure)1.5 (maximum pressure)\n The compute_borrow_fraction() function determined what fraction of the global time buffer\n could be borrowed for the current question. The borrowing decision depended on several factors:\n
\n Time Ratio (c_ratio)
\n Measures how much of the allocated Step-Back time budget was consumed.\n
\n Token Ratio (s_ratio)
\n Measures how much of the allocated Step-Back token budget was consumed.\n
\n Mathematical Logic:\n
\n\n Two normalized ratios were first computed.\n These two signals were then combined using a Euclidean-style formulation:\n
\ncombined_signal = sqrt((c_ratio² + s_ratio²) / 2)\n\n
\n This produced a unified measure representing overall computational intensity for the current problem.\n
\n\n The system then adjusted this value using the previously computed time-pressure signal:\n
\n\n Finally, the computed borrow fraction was constrained within predefined limits:\n
\n0.0b_maxdef compute_time_pressure(\n self,\n remaining_time: float,\n questions_completed: int,\n ) -> float:\n remaining_q = max(1, self.total_questions - questions_completed)\n if remaining_time <= 0:\n return 1.5\n ideal_pace = self.total_available_time / self.total_questions\n available_pace = remaining_time / remaining_q\n pressure = ideal_pace / available_pace\n return max(0.3, min(1.5, pressure))\n def compute_borrow_fraction(\n self,\n remaining_time: float,\n questions_completed: int,\n stepback_time: float,\n stepback_time_budget: float,\n stepback_tokens: int,\n stepback_tokens_budget: int,\n global_buffer: float = 0.0,\n ) -> float:\n \"\"\"\n Compute what fraction of the buffer to borrow.\n Returns\n -------\n float\n Borrow fraction in [0, b_max].\n \"\"\"
\n c_ratio = min(1.0, stepback_time / stepback_time_budget) if stepback_time > 0 else 0.5\n s_ratio = stepback_tokens / stepback_tokens_budget if stepback_tokens > 0 else 0.0\n combined_signal = math.sqrt((c_ratio ** 2 + s_ratio ** 2) / 2)\n sigmoid_out = self._sigmoid(combined_signal)\n pressure = self.compute_time_pressure(remaining_time, questions_completed)\n pressure_mult = 1.0 / pressure\n borrow_fraction = self.b_max * combined_signal * pressure_mult\n borrow_fraction = max(0.0, min(self.b_max, borrow_fraction))\n:-
\n– Much of work in the field of Artificial Intelligence and Machine Learning is inspired from how humans function. Neural network corresponds to the neurons in the brain. “Attention is all you need” is how our brains process lengthy information. Therefore, my rationale was that if us humans get guidance then we perform a task better so the llm model can behave similarly. \n
\nFurther, the principles identified during stepback phase 1 formed the basis of dynamic prompting. Prompting based on domains and techniques used would in my opinion result in better outcomes and I wanted to try this further. However, phase 1 took away from the total time to solve the problem. Therefore, this approach had to be discarded.
\nIn addition, I would like to highlight three further contributions and observations that were either underemphasized in the original write-up or significantly strengthened through post-submission experiments.\n
\n\nvLLM generates CUDA graphs for two distinct execution phases:\n
\n\nThese two phases exhibit very different shape distributions.\n
\n\nThe mixed prefill+decode phase experiences progressively increasing input lengths due to:\n
\n\nIn contrast, the pure decode phase typically generates one token at a time and therefore operates under a very different execution profile.\n
\n\nTable in Section 5.2 already includes an initial ablation study on graph capture sizing. After submission, I conducted additional experiments to support a hypothesis I had originally held on intuition alone — one I did not assert in version 1 of the write-up due to the lack of data i.e .
\nIf graph_size was equal to the max-num-seqs parameter passed to vllm inference server then throughput increased atleast by 65%. This maybe due to decode graph matching the number of sequences
\n\nAnother implementation detail omitted from the original write-up: the pipeline uses streaming responses rather than single-shot generation. Non Streaming Responses resulted in frequent read timeout and connection timeout issues\n
\nThis was part of the work that went into building the pipeline, but I forgot to mention it\n
\nDuring experimentation, non-streaming inference frequently resulted in:\n
\n\nSwitching to streaming-based generation resolved these issues and significantly improved runtime stability.\n
\n\nThe revised evaluation now includes additional comparisons against:\n
\n\nOne particularly interesting observation was that Gemini Pro 3.1 and Claude Opus 4.7 appear to be highly complementary systems.\n
\n\nTogether, the two models successfully solved every question in my curated evaluation dataset at pass@1.\n
\nHowever, Claude demonstrated stronger mathematical reasoning than Gemini 3.1 Pro, especially on difficult mathematical problems. While both models solved easier problems reliably and on harder problems, each model solved cases that the other failed to solve.\n
\n
\n
\nYes, several backend-level ablations and experiments were conducted.\n
\n\n\nOne major component of the backend was the inference engine itself. Few inference frameworks were evaluated:\n
\n\n\n Initial experiments with TensorRT-LLM. This could be easily implemented using docker but in Kaggle environment docker could not be used. I tried to implement workaround but I faced challenges while implementing the workaround. I finally did not pursue this route when I faced datatype mismatch issues with gpt-oss and tensorrt-llm framework
\n\n\nThe issue may or may not be fixable, since TensorRT-LLM generally works well with many large models. I spent some time attempting to make it work, but due to competition time constraints, I decided not to pursue this direction further.\nIt may still be possible to get GPT-OSS running successfully with additional engineering effort since gpt-oss work with tensorrt installed using docker\n
\n\nAnother important backend component involved the CUDA kernels used internally within the transformer architecture.\n
\n\n\nIf you inspect the runtime logs, you will notice that. This is a snippet from vllm logs\n
\n\n\nUsing FLASH_ATTNUsing Marlin Backend\nTo clarify, my submitted code does use FLASH Attention. However flash and marlin correspond to different optimized CUDA kernels and execution backends used for accelerating transformer operations such as attention layers, matrix multiplications, and quantized inference workloads.\n
\n\n\nPlease refer the architecture below to understand where each of them is used. The parts of the transformer where Marlin and Flash are used is given below\n
\n\n\n\n\nFor the attention component of the transformer, one of the research papers indicated that GPT-OSS-120B could operate using four different attention backends.\n
\n\n\nAt the time I conducted these experiments, I had not yet seen the research you shared regarding FLASH Attention outperforming Triton. Nevertheless, it was still valuable to experimentally validate which backend performed best for this specific inference pipeline and use case.\n
\n\n\nThe following attention backends were evaluated:\n
\n\n\nKey Observation:
\nFLASH Attention outperformed Triton in my experiments. The remaining two attention types were ultimately not usable due to lack of support for sink-token configuration.\n
\nI missed including this information in my original write-up, so thank you for asking the question. It gave me an opportunity to clarify these backend-level experiments and observations.\n
\n\n\nUnder normal transformer behavior, each token primarily attends to semantically relevant tokens within the sequence.\n
\n\n\nHowever, in practice, certain tokens tend to attract disproportionately high attention from many other tokens. These are commonly referred to as sink tokens.\n
\n\n\nExamples of sink tokens can include:\n
\n\n\nSink tokens can sometimes improve:\n
\n\n\nThe sink-token configuration was already built into the GPT-OSS model itself. I intentionally chose not to disable it while testing alternative attention backends because I was concerned that changing this behavior could negatively impact model performance and long-context stability.\n
\n\n\nThe following logic was used to dynamically estimate time pressure and determine how much additional buffer time could be allocated to a question during inference.\n
\n\n\n\n The compute_time_pressure() function estimated the current level of time pressure based on:\n
\n The core idea was to compare:\n
\n\ntotal_available_time / total_questions)remaining_time / remaining_questions)\n A pressure score was then computed as:\n
\n\n\n To avoid extreme instability, the final pressure value was clamped between:\n
\n\n0.3 (minimum pressure)1.5 (maximum pressure)\n The compute_borrow_fraction() function determined what fraction of the global time buffer\n could be borrowed for the current question. The borrowing decision depended on several factors:\n
\n Time Ratio (c_ratio)
\n Measures how much of the allocated Step-Back time budget was consumed.\n
\n Token Ratio (s_ratio)
\n Measures how much of the allocated Step-Back token budget was consumed.\n
\n Mathematical Logic:\n
\n\n\n Two normalized ratios were first computed.\n These two signals were then combined using a Euclidean-style formulation:\n
\n\n\ncombined_signal = sqrt((c_ratio² + s_ratio²) / 2)\n\n\n
\n This produced a unified measure representing overall computational intensity for the current problem.\n
\n\n\n The system then adjusted this value using the previously computed time-pressure signal:\n
\n\n\n Finally, the computed borrow fraction was constrained within predefined limits:\n
\n\n0.0b_max\nIn addition, I would like to highlight three further contributions and observations that were either underemphasized in the original write-up or significantly strengthened through post-submission experiments.\n
\n\n\nvLLM generates CUDA graphs for two distinct execution phases:\n
\n\n\nThese two phases exhibit very different shape distributions.\n
\n\n\nThe mixed prefill+decode phase experiences progressively increasing input lengths due to:\n
\n\n\nIn contrast, the pure decode phase typically generates one token at a time and therefore operates under a very different execution profile.\n
\n\n\n\n\nTable in Section 5.2 already includes an initial ablation study on graph capture sizing. After submission, I conducted additional experiments to support a hypothesis I had originally held on intuition alone — one I did not assert in version 1 of the write-up due to the lack of data i.e .
\n\n\nAnother implementation detail omitted from the original write-up: the pipeline uses streaming responses rather than single-shot generation. Non Streaming Responses resulted in frequent read timeout and connection timeout issues\n
\nThis was part of the work that went into building the pipeline, but I forgot to mention it\n
\nDuring experimentation, non-streaming inference frequently resulted in:\n
\n\n\nSwitching to streaming-based generation resolved these issues and significantly improved runtime stability.\n
\n\n\nThe revised evaluation now includes additional comparisons against:\n
\n\n\nOne particularly interesting observation was that Gemini Pro 3.1 and Claude Opus 4.7 appear to be highly complementary systems.\n
\n\n\nTogether, the two models successfully solved every question in my curated evaluation dataset at pass@1.\n
\n\n\nHowever, Claude demonstrated stronger mathematical reasoning than Gemini 3.1 Pro, especially on difficult mathematical problems. While both models solved easier problems reliably and on harder problems, each model solved cases that the other failed to solve.\n
\nIndependent Research • April 2025
\n\nThis report documents an independent submission to the AI Mathematical Olympiad (AIMO)\nProgress Prize 3 competition. Operating under a strict 5-hour GPU budget on a single\nNVIDIA H100 80GB GPU, a comprehensive set of inference-time optimizations for\nGPT-OSS-120B were investigated.\n
\n\nContributions include:\n
\n\nThe AIMO Progress Prize 3 competition requires solving 50 mathematical problems under\na strict 5-hour GPU constraint. Problems are competition-grade, requiring multi-step\nsymbolic and numerical reasoning.\n
\n\nEach participant receives a single GPU allocation for the entire run, with approximately\n20 minutes consumed by environment setup and model loading, leaving roughly\n336 seconds per question.\n
\n\nThis work is entirely inference-based — no model weights were fine-tuned.\nThe approach combines multiple factors:\n
\n\nGPT-OSS-120B appeared to perform best when given sufficient time and token budget to\nreason through problems fully.\n
\n\nNote: The best-performing configuration (score 24) could not be submitted to the competition because a critical library dependency was accidentally deleted just before the deadline.\n
\n| Component | \nSpecification | \n
|---|---|
| GPU | \nNVIDIA H100 80GB (SXM or PCIe; single GPU, tp_size=1) | \n
| GPU Memory | \n~60–65 GB total during inference (96% utilization) | \n
| CUDA Architecture | \nHopper (sm_90) — required for fp8_e4m3 KV cache | \n
| KV Cache dtype | \nfp8_e4m3 | \n
| Model | \nGPT-OSS-120B (MoE, 120B params, 5.1B active) | \n
| Serving Framework | \nModified NemoSkills + vLLM 0.11.2 | \n
| Python | \nPython 3.12.12 | \n
| Parameter | \nValue | \n
|---|---|
| Total GPU time | \n5 hours | \n
| Setup & model loading | \n~20 minutes | \n
| Effective inference window | \n~280 minutes | \n
| Base per-question budget | \n~320 seconds + buffer borrowing | \n
| Global buffer | \n350 seconds (dynamically updated) | \n
\nThree levels of benchmarking were used, applied in the order below as each revealed\na gap that motivated the next.\n
\n
\nThe first benchmark consisted of 20 problems drawn from the AIMO 2 and AIMO 3\nreference problem sets. The model scored 95% accuracy (19/20).\n
\n| Metric | \nValue | \n
|---|---|
| Total problems | \n20 | \n
| Correct answers | \n19/20 (95%) | \n
| Incorrect answer | \nProblem #10 (model: 29,443; actual: 8,687) | \n
| Avg time per problem | \n230 seconds | \n
| Avg tokens generated | \n13,543 tokens | \n
\nAfter observing the gap between 90–95% reference accuracy and the 42% leaderboard score,\na harder benchmark was constructed from 49 problems sourced from IMO, AMC/AIME,\nand similar archives.\n
\n| Property | \nValue | \n
|---|---|
| Total problems | \n49 | \n
| Problem sources | \n\n |
| Model accuracy | \n65% | \n
Internal BenchMark Dataset - https://huggingface.co/HarpreetK/AIMO3/blob/main/Internal_Benchmark_Final.xlsx\nSources - AOPS,https://matharena.ai/apex/, ARML, HMMT
\n\nA targeted 10-problem subset was extracted consisting of questions where the previous\nattempt had either returned 29443 (timeout/extraction failure) or produced an incorrect answer.\n
\n| Benchmark | \nScore | \nNotes | \n
|---|---|---|
| AIMO Reference (20) | \n95% (19/20) | \nToo easy; gap not explained | \n
| Internal Benchmark (49) | \n65% | \nCloser to AIMO 3 difficulty | \n
| AIMO 3 Leaderboard (50) | \n21/50 (42%) | \nActual submission | \n
| Best config (50) | \n24/50 (48%) | \nNot submitted — library deletion | \n
Flow Diagram is attached. Please open it independently in a new window to zoom on it
\n\nA modified version of the NemoSkills repository (winner of AIMO 2) was used as the baseline,\nproviding a scaffold for request scheduling and Python code execution sandboxing.\nMCP and LiteLLM abstraction layers were stripped to reduce overhead.\n
\n\n\nNote: These were conclusions based on research. Ideally, both approaches would have been compared directly.\n\n
\n\nThe BufferBorrower class implements a time-pressure-based allocation strategy.\nBorrowable time differed across notebooks:\n
\n\nThe final best-performing approach used a single-phase prompt applied uniformly to all 50 problems.\n
\n\n\nThe key jump was from score 18 (Step-Back) to score 24 (Non-Step-Back):\na 6-point improvement.\n\n
\n\nUp to 8 self-consistent samples are generated per problem.\nMajority voting uses frequency count only.\n
\n\nMetric Used: Exact Match Accuracy.\n
\n\npython -m nemo_skills.inference.server.serve_vllm \\\n --model=.../gpt-oss-120b/.../1 --port=5000 --num_gpus=1 \\\n --max_model_len=38000 --max_num_batched_tokens=65000 \\\n --max_num_seqs=11 --max-cudagraph-capture-size=2048 \\\n --gpu_memory_utilization=0.95 --kv_cache_dtype=fp8_e4m3 \\\n --enable-prefix-caching --async-scheduling\n| Parameter | \nValue | \nRationale | \n
|---|---|---|
| max_tokens | \n38,000 | \nUpper bound per problem | \n
| tokens_to_generate | \n35,940 | \nmax_tokens minus overhead | \n
| max_batch_size | \n8 | \nNumber of self-consistent samples | \n
| global_buffer | \n350 seconds | \nReserve for hard problems | \n
| temperature | \n1.0 | \nPer NVIDIA NeMo recommendation | \n
| top_p | \n1.0 | \nPer NVIDIA NeMo recommendation | \n
All 15 configurations tested.
\n\nGPT-OSS alternates between full attention and sliding window attention (SWA).\n
\n\nSliding window sizes 128, 200, and 8,000 were tested.\n
\n\nTemperature fixed at 1.0 (NVIDIA recommendation).\n
\n| Parameter | \nScore 18 (SB) | \nScore 20 | \nScore 21 (SUB) | \nScore 24 (Best) | \n
|---|---|---|---|---|
| Prompting | \nStep-Back full | \nNon-SB + dual | \nNon-Step-Back | \nNon-Step-Back | \n
| Temperature | \nFixed 1.0 | \nVaried | \nFixed 1.0 | \nFixed 1.0 | \n
| max_model_len | \n65,536 | \n36,000 | \n38,000 | \n80,000 | \n
| Leaderboard score | \n18/50 | \n20/50 | \n21/50 | \n24/50 | \n
\nReduced token budget caused score drop from 24 to 21.\n
\n\nDual-prompt strategy and varied-temperature sampling hurt performance.\n
\n
Below models were added to SOTA Comparison\nQwen (without tool calling),\nGemini Pro 3.1,\nand Claude Opus 4.7.\nOne particularly interesting observation was that Gemini Pro 3.1 and Claude Opus 4.7 appear to be highly complementary systems.
\nTogether, the two models successfully solved every question in my curated evaluation dataset.
\nMore specifically:
\nClaude demonstrated stronger mathematical reasoning overall,\nespecially on difficult mathematical problems,\nwhile both models solved easier problems reliably,\nand on harder problems, each model solved cases that the other failed to solve.
\n\nHuggingFace repo:\n\nhttps://huggingface.co/HarpreetK/AIMO3\n\n
\n| Artifact | \nLocation / Value | \nStatus | \n
|---|---|---|
| Base model | \nGPT-OSS-120B | \nProvided | \n
| Submitted notebook | \ngpt-oss-score21.ipynb | \nProvided | \n
| Best notebook (24) | \ngpt-oss-nostepback-score24.ipynb | \nProvided | \n
| Backend | \nMarlin (via vLLM 0.11.2) | \nProvided | \n
refer the pdf for the same
", "rawMarkdown": "Independent Research • April 2025
\n\n\nThis report documents an independent submission to the AI Mathematical Olympiad (AIMO)\nProgress Prize 3 competition. Operating under a strict 5-hour GPU budget on a single\nNVIDIA H100 80GB GPU, a comprehensive set of inference-time optimizations for\nGPT-OSS-120B were investigated.\n
\n\n\nContributions include:\n
\n\n\nThe AIMO Progress Prize 3 competition requires solving 50 mathematical problems under\na strict 5-hour GPU constraint. Problems are competition-grade, requiring multi-step\nsymbolic and numerical reasoning.\n
\n\n\nEach participant receives a single GPU allocation for the entire run, with approximately\n20 minutes consumed by environment setup and model loading, leaving roughly\n336 seconds per question.\n
\n\n\nThis work is entirely inference-based — no model weights were fine-tuned.\nThe approach combines multiple factors:\n
\n\n\nGPT-OSS-120B appeared to perform best when given sufficient time and token budget to\nreason through problems fully.\n
\n\n\nNote: The best-performing configuration (score 24) could not be submitted to the competition because a critical library dependency was accidentally deleted just before the deadline.\n
\n\n| Component | \nSpecification | \n
|---|---|
| GPU | \nNVIDIA H100 80GB (SXM or PCIe; single GPU, tp_size=1) | \n
| GPU Memory | \n~60–65 GB total during inference (96% utilization) | \n
| CUDA Architecture | \nHopper (sm_90) — required for fp8_e4m3 KV cache | \n
| KV Cache dtype | \nfp8_e4m3 | \n
| Model | \nGPT-OSS-120B (MoE, 120B params, 5.1B active) | \n
| Serving Framework | \nModified NemoSkills + vLLM 0.11.2 | \n
| Python | \nPython 3.12.12 | \n
| Parameter | \nValue | \n
|---|---|
| Total GPU time | \n5 hours | \n
| Setup & model loading | \n~20 minutes | \n
| Effective inference window | \n~280 minutes | \n
| Base per-question budget | \n~320 seconds + buffer borrowing | \n
| Global buffer | \n350 seconds (dynamically updated) | \n
\nThree levels of benchmarking were used, applied in the order below as each revealed\na gap that motivated the next.\n
\n\n\n\nThe first benchmark consisted of 20 problems drawn from the AIMO 2 and AIMO 3\nreference problem sets. The model scored 95% accuracy (19/20).\n
\n\n| Metric | \nValue | \n
|---|---|
| Total problems | \n20 | \n
| Correct answers | \n19/20 (95%) | \n
| Incorrect answer | \nProblem #10 (model: 29,443; actual: 8,687) | \n
| Avg time per problem | \n230 seconds | \n
| Avg tokens generated | \n13,543 tokens | \n
\nAfter observing the gap between 90–95% reference accuracy and the 42% leaderboard score,\na harder benchmark was constructed from 49 problems sourced from IMO, AMC/AIME,\nand similar archives.\n
\n\n| Property | \nValue | \n
|---|---|
| Total problems | \n49 | \n
| Problem sources | \n\n |
| Model accuracy | \n65% | \n
\nA targeted 10-problem subset was extracted consisting of questions where the previous\nattempt had either returned 29443 (timeout/extraction failure) or produced an incorrect answer.\n
\n\n| Benchmark | \nScore | \nNotes | \n
|---|---|---|
| AIMO Reference (20) | \n95% (19/20) | \nToo easy; gap not explained | \n
| Internal Benchmark (49) | \n65% | \nCloser to AIMO 3 difficulty | \n
| AIMO 3 Leaderboard (50) | \n21/50 (42%) | \nActual submission | \n
| Best config (50) | \n24/50 (48%) | \nNot submitted — library deletion | \n
\nA modified version of the NemoSkills repository (winner of AIMO 2) was used as the baseline,\nproviding a scaffold for request scheduling and Python code execution sandboxing.\nMCP and LiteLLM abstraction layers were stripped to reduce overhead.\n
\n\n\n\nNote: These were conclusions based on research. Ideally, both approaches would have been compared directly.\n\n
\n\n\nThe BufferBorrower class implements a time-pressure-based allocation strategy.\nBorrowable time differed across notebooks:\n
\n\n\nThe final best-performing approach used a single-phase prompt applied uniformly to all 50 problems.\n
\n\n\n\nThe key jump was from score 18 (Step-Back) to score 24 (Non-Step-Back):\na 6-point improvement.\n\n
\n\n\nUp to 8 self-consistent samples are generated per problem.\nMajority voting uses frequency count only.\n
\n\n\nMetric Used: Exact Match Accuracy.\n
\n\n\npython -m nemo_skills.inference.server.serve_vllm \\\n --model=.../gpt-oss-120b/.../1 --port=5000 --num_gpus=1 \\\n --max_model_len=38000 --max_num_batched_tokens=65000 \\\n --max_num_seqs=11 --max-cudagraph-capture-size=2048 \\\n --gpu_memory_utilization=0.95 --kv_cache_dtype=fp8_e4m3 \\\n --enable-prefix-caching --async-scheduling\n| Parameter | \nValue | \nRationale | \n
|---|---|---|
| max_tokens | \n38,000 | \nUpper bound per problem | \n
| tokens_to_generate | \n35,940 | \nmax_tokens minus overhead | \n
| max_batch_size | \n8 | \nNumber of self-consistent samples | \n
| global_buffer | \n350 seconds | \nReserve for hard problems | \n
| temperature | \n1.0 | \nPer NVIDIA NeMo recommendation | \n
| top_p | \n1.0 | \nPer NVIDIA NeMo recommendation | \n
All 15 configurations tested.
\n\n\nGPT-OSS alternates between full attention and sliding window attention (SWA).\n
\n\n\nSliding window sizes 128, 200, and 8,000 were tested.\n
\n\n\nTemperature fixed at 1.0 (NVIDIA recommendation).\n
\n\n| Parameter | \nScore 18 (SB) | \nScore 20 | \nScore 21 (SUB) | \nScore 24 (Best) | \n
|---|---|---|---|---|
| Prompting | \nStep-Back full | \nNon-SB + dual | \nNon-Step-Back | \nNon-Step-Back | \n
| Temperature | \nFixed 1.0 | \nVaried | \nFixed 1.0 | \nFixed 1.0 | \n
| max_model_len | \n65,536 | \n36,000 | \n38,000 | \n80,000 | \n
| Leaderboard score | \n18/50 | \n20/50 | \n21/50 | \n24/50 | \n
\nReduced token budget caused score drop from 24 to 21.\n
\n\n\nDual-prompt strategy and varied-temperature sampling hurt performance.\n
\n\n\nHuggingFace repo:\n\nhttps://huggingface.co/HarpreetK/AIMO3\n\n
\n\n| Artifact | \nLocation / Value | \nStatus | \n
|---|---|---|
| Base model | \nGPT-OSS-120B | \nProvided | \n
| Submitted notebook | \ngpt-oss-score21.ipynb | \nProvided | \n
| Best notebook (24) | \ngpt-oss-nostepback-score24.ipynb | \nProvided | \n
| Backend | \nMarlin (via vLLM 0.11.2) | \nProvided | \n