![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F23110311%2F319f0b6de502290fff5b1b1f46194bf3%2FTIR_ACCURACY.png?generation=1776346717057232&alt=media\")
This writeup presents an advanced inference-time optimization strategy developed by the Potrhino team for the final evaluation of the AI Mathematical Olympiad (AIMO) Progress Prize 3, ranking 373rd out of 4,138. Recognizing that solving Olympiad-level mathematics under strict temporal and computational constraints nowadays is primarily an efficiency-bottlenecked task, we pivot from traditional model fine-tuning toward adaptive test-time orchestration. Leveraging the open-weight gpt-oss-120b model, our approach integrates:
\nThrough rigorous ablation studies, we demonstrate that the synergy of these techniques significantly elevates the model’s performance ceiling, achieving the following scores:
\nOur results establish that strategic resource management and deterministic tool augmentation can improve mathematical reasoning performance. To ensure full reproducibility, our code, environment configurations, and experimental logs are made publicly available.
\nIn recent years, Large Language Models (LLMs) have demonstrated remarkable proficiency across a broad spectrum of natural language tasks, evolving from fluent text generators into versatile problem solvers. Despite these advancements, complex mathematical reasoning remains a challenge, particularly at the Olympiad level. Proficiency in this domain is critical, as it serves as a cornerstone for solving intricate problems in fields ranging from advanced engineering to quantitative finance. While recent breakthroughs have seen Artificial Intelligence (AI) achieve human-level performance on International Mathematical Olympiad (IMO) problems, a notable performance gap persists between state-of-the-art closed-source models and their open-weight counterparts.
\nThe AI Mathematical Olympiad Progress Prize 3 (AIMO3) [1] aims to bridge this divide by challenging participants to develop open-source algorithms and models capable of solving high-level mathematical problems under strict computational and time constraints. While AIMO Progress Prize 2 was won in April 2025 by Nvidia's NemoSkills team [2], AIMO3 utilizes entirely original problems to eliminate the risk of data contamination. Every ground-truth label is an integer within the range [0, 99999]. Submissions, consisting of Kaggle notebooks, are evaluated on a private test set of 50 math problems, where scoring is based on the consistency of two independent predictions: for each problem, a score of 1.0 is awarded if both answers are correct, 0.5 if only one is correct, and 0 otherwise. The competition enforces a rigorous offline environment, limiting execution to 5 hours on GPU-accelerated notebooks (H100, P100, or T4×2) or 12 hours on CPU-based notebooks.
\nThis writeup details the strategy presented by the Potrhino team, which achieved a score of 43/50 in the public evaluation. The proposed approach leverages the gpt-oss-120b [7] model on H100 hardware, integrating:
\nThese methodologies are designed to optimize test-time inference under strict computational and time constraints. The following sections introduce the fundamental concepts, then describe the proposed strategy, and finally provide a comprehensive analysis of the results.
\nWhile LLMs have evolved into versatile problem solvers, purely text-based reasoning often struggles with tasks requiring precise numerical computation, long-horizon search, or rigorous verification. To address these inherent limitations, Tool-Integrated Reasoning (TIR) has emerged as an increasingly adopted paradigm. By equipping LLMs with external tools, such as Python code interpreters, TIR effectively expands the model's capabilities and transcends the \"capability ceiling\" of purely linguistic inference.
\nRecent research demonstrates that TIR-augmented models decisively outperform pure-text counterparts across challenging mathematical benchmarks. This performance gain is particularly evident in the pass@k metric, as the interpreter provides a deterministic execution environment that mitigates common calculation errors. Furthermore, this advantage extends beyond mere arithmetic. It also facilitates complex problem-solving strategies, such as brute-force search and algorithmic simulation, which would be intractably verbose or prone to failure in a standard autoregressive setup.
\nFigure 1 shows pass@k curves for the TIR (RL-trained) and pure-text models (Qwen3 8B) across benchmarks: (a) AIME24, (b) AIME25 [3]. This illustrates how TIR provides a significant gain in accuracy on AIME problems, which makes it an ideal candidate for AIMO3.
\n
While scaling the parameters of LLMs generally improves performance, scale alone is often insufficient for mastering complex arithmetic and symbolic reasoning. Chain-of-Thought (CoT) prompting addresses this limitation by eliciting a series of intermediate reasoning steps before arriving at a final conclusion. This approach offers several critical advantages for Olympiad-level mathematics:
\nCoT transforms a model from a direct predictor into a systematic reasoner, enabling it to bridge the gap between problem understanding and the final output. Figure 2 shows the solve rate of CoT and other prompting variations using LaMDA 137B and PaLM 540B on the GSM8K dataset [4]. This illustrates how CoT outperforms other prompting variations, highlighting its effectiveness in improving reasoning for math problems. However, CoT alone does not guarantee correctness, as errors can propagate across intermediate steps.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F23110311%2F13633ce68b5c9d7569e1b919b209264a%2FCoT_ACCURACY.png?generation=1776551872549459&alt=media\")
A fundamental characteristic of complex problem-solving is the existence of multiple diverse reasoning paths that converge on the same correct solution. Self-Consistency leverages this intuition by replacing traditional greedy decoding with a sampling-based approach. The method involves generating a diverse set of reasoning paths and subsequently selecting the most consistent final answer by marginalizing out the sampled paths, a process commonly implemented as majority voting. This technique significantly enhances the robustness of CoT prompting. While an LLM may occasionally produce erroneous intermediate steps, incorrect reasoning paths are statistically less likely to converge on the same final value compared to valid logical sequences. Consequently, the correct answer tends to exhibit higher agreement across the sample population. Beyond raw accuracy gains, Self-Consistency provides two additional advantages:
\nA primary constraint of Self-Consistency is the increased test-time compute required. However, empirical evidence suggests that performance gains often saturate relatively quickly. Therefore, a strategically limited number of paths (e.g., k=5 or k=10) can capture the majority of the benefits without exceeding time limits. This trade-off is especially critical in constrained environments such as AIMO3, where inference budget is strictly limited. \nFigure 3 shows how Self-Consistency improves accuracy across various reasoning tasks, using LaMDA-137B [5].
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F23110311%2F5a7fdb7a5084e8f86210ad0329bbb23c%2FSelf_consistency_ACCURACY.png?generation=1776348545912708&alt=media\")
While parallel reasoning techniques like Self-Consistency enhance accuracy, they often encounter two primary bottlenecks: the linear scaling of inference overhead and diminishing returns caused by the equal weighting of all reasoning traces. Standard majority voting fails to account for quality variations, potentially allowing low-confidence or \"hallucinated\" traces to compromise the final output.\nTo address these limitations, the Deep Think with Confidence (DeepConf) methodology integrates confidence-aware filtering based on local token-level statistics. Unlike global measures that average confidence across an entire sequence, thereby masking localized reasoning failures, DeepConf monitors fluctuations at critical intermediate steps. This approach offers two distinct operational modes:
\nEmpirical results demonstrate that DeepConf not only improves accuracy by mitigating noise from low-quality traces but also optimizes resource allocation. In offline mode, DeepConf@512 achieves 99.9% accuracy on AIME 2025 using gpt-oss-120b (without tools), effectively saturating the benchmark compared to 97.0% for cons@512 (consensus/majority voting) and 91.8% for pass@1. In online mode, DeepConf reduces token generation by up to 84.7% compared to standard parallel sampling while maintaining or exceeding baseline accuracy. This efficiency is critical for maximizing the number of effective reasoning paths under strict computational and time constraints. This idea directly inspired our adaptive pruning and early termination strategy. Figure 4 shows the aforementioned results for both online and offline modes on AIME 2025 [6].
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F23110311%2F66b614b3c5d06a433690b0af1d1aeb5d%2FDeep_conf_ACCURACY.png?generation=1776348951716265&alt=media\")
gpt-oss-120b is an open-weight, reasoning-focused autoregressive model released under the Apache 2.0 license. Optimized for agentic workflows, it excels in instruction following, structured output generation, and tool integration (e.g., Python execution). The model architecture is a Mixture-of-Experts (MoE) Transformer, featuring 36 layers with a total of 116.8B parameters, of which 5.1B are active per token. The MoE weights are quantized to MXFP4 format (4.25 bits per parameter), comprising over 90% of the total parameter count. This enables operation within the constraints of a single 80GB GPU (such as the H100). The model was trained using the Harmony chat format, which utilizes specialized tokens for message boundaries and a role-based hierarchy to resolve instruction conflicts (System > Developer > User > Assistant > Tool). A distinctive feature of gpt-oss-120b is its support for three distinct reasoning levels (Low, Medium, and High), where higher levels correlate with increased CoT length and logical depth (as shown in Figure 5 [7]).
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F23110311%2Fdfdbd0b9257ab6a1010135c1e9bef41c%2Fgpt-oss-cot.png?generation=1776349514505657&alt=media\")
The model is natively trained to utilize a suite of agentic tools:
\nBy interleaving CoT with TIR, gpt-oss-120b achieves high accuracy on mathematical benchmarks (as illustrated in Figure 6) while maintaining rapid inference speed and efficient memory utilization. We selected gpt-oss-120b due to its strong reasoning capabilities, native support for tool use, and compatibility with single-GPU deployment under AIMO constraints.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F23110311%2F73d193171bc7b245bc1f3888488b99f3%2Fgpt-oss-benchmark_table.png?generation=1776349624566200&alt=media\")
The primary challenge of the AIMO3 competition lies in developing an AI system capable of solving IMO-level mathematical problems under stringent time and computational constraints. While modern LLMs have shown increasing proficiency in this domain, success hinges on identifying the optimal trade-off between reasoning accuracy and execution latency. \nOur approach addresses this challenge by leveraging the gpt-oss-120b model as a high-capacity foundation and concentrating development efforts on test-time optimization. The overall approach is structured around a set of strategic pillars, with hyperparameters calibrated to balance robustness and efficiency, while remaining amenable to further fine-grained optimization. These pillars include:
\nWe select the open-weight gpt-oss-120b model as the primary backbone. Given its strong baseline performance on mathematical benchmarks, further supervised fine-tuning (SFT) is deemed unnecessary. Specifically, the model achieves strong performance on AIME 2024 and AIME 2025 [7] when utilizing the \"High\" CoT reasoning level combined with TIR. Beyond raw accuracy, the model aligns well with the competition's hardware and temporal constraints. Utilizing the MXFP4 quantized format, gpt-oss-120b fits comfortably within the VRAM of a single H100 GPU, maintaining high inference throughput despite its 120B parameters. Furthermore, the average response length for AIME 2025 problems is reported to remain below 16k tokens [7], which supports an effective balance between depth of reasoning with execution speed. Consequently, we hypothesize that further SFT would yield diminishing returns relative to the substantial computational costs and the risk of increasing latency. This design choice also improves reproducibility, as it avoids reliance on additional proprietary training data or compute-intensive fine-tuning stages.
\nThe model's reasoning process is augmented by a Python tool, which can be invoked at any step of its reasoning chain. The model generates code blocks containing print() statements to retrieve intermediate results, which are then executed in a stateful Jupyter notebook environment.
To facilitate complex symbolic and numerical tasks, the execution environment is pre-configured with essential scientific libraries (e.g., math, numpy, sympy). To handle the rigorous demands of Olympiad-level problems, such as deeply nested recursion, operations involving very large integers or high-precision floating-point arithmetic, we increase the system’s default limits as follow:
recursion_limit = 2000;precision = 100000;int_max_digits = 1000000.To prevent long execution outputs from saturating the model’s context window, we implement a centralized truncation strategy. Outputs exceeding a predefined threshold (L_max = 1000 characters) are clipped: only the initial and final L_max/2 characters are preserved, separated by a ... TRUNCATED ... marker. This ensures the model receives both the beginning and final portions of the output while discarding redundant intermediate iterations. This encourages more concise tool usage and provides the model with the segments of the output that typically contain the most relevant information.
To ensure that inefficient or infinite loops do not deplete the problem's time budget, we implement an adaptive timeout strategy for code execution. This strategy balances the need for rapid feedback on simpler problems with the requirement for long-running computations on more complex ones:
\nThe execution timeout for each call is determined by the following formula:
\n$$Texec = \\min( \\min(Tinit \\cdot 1.3^n, 2 \\cdot Tinit), Tremaining )$$
\nWhere:
\nTinit is the starting execution timeout (set to 8s);n is the number of consecutive timeouts for the current problem;Tremaining is the residual time allocated to the specific problem.The selection of Tinit=8s and the growth factor of 1.3 is driven by an empirical balance between code complexity and session latency. A baseline of 8s is sufficient for the majority of standard mathematical scripts (e.g., symbolic manipulations or short loops). The exponential scaling factor of 1.3 is specifically calibrated to allow the execution limit to double after exactly three consecutive timeouts. Capping the growth at 2 * Tinit prevents unoptimized or runaway scripts from depleting the global time budget, which is a critical risk in time-constrained environments. Furthermore, we hypothesize that if a model fails to produce an efficient solution within three attempts (and nearly doubling the initial timeout), the bottleneck is likely structural rather than purely computational. Beyond this threshold, additional time yields diminishing returns, as the model is encouraged to shift toward an alternative reasoning strategy or a more efficient mathematical simulation rather than marginally optimizing failing code.\nThe interaction between the LLM and the execution environment is managed via a dedicated worker process; the detailed operational logic of this Jupyter worker is provided in Algorithm 1 (see Appendix).
The total execution time is partitioned across all 50 problems to ensure robustness against varying problem difficulties. After accounting for the initial setup overhead, including environment initialization and model loading, we implement a dynamic time-budgeting algorithm. The budget for a given problem, Tp, is determined by:
$$Tp = \\max( \\min(Tpool, Tavg \\cdot Cmax), Tavg \\cdot Cmin )$$
\nWhere:
\nTpool is the total remaining time after subtracting the reserved buffer for all future problems (Tavg * Cmin for each remaining problem);Tavg represents the baseline allocation, calculated as the initial total available time divided by the total number of problems;Cmax is a scaling constant for upper bound (2.1);Cmin is a scaling constant for lower bound (0.86).This mechanism ensures that a single outlier (an extremely difficult problem) cannot deplete the resources required for the rest of the test set. Furthermore, it allows simpler problems to conclude early and return their unused budget to the global pool, thereby granting additional computational time to high-complexity cases without risking a global submission timeout.\nThe coefficients Cmin and Cmax were empirically selected to balance risk and performance. A Cmin of 0.86 ensures that even in the worst-case scenario, every problem is guaranteed approximately 85% of the baseline average time. Conversely, a Cmax of 2.1 allows the solver to spend more than double the average time on difficult problems without destabilizing the overall schedule. This upper bound was also found to be sufficient for the model to reach the maximum effective depth of its context window. This dynamic approach ensures that the \"time-to-solution\" is maximized for challenging problems while maintaining a zero-failure rate regarding the global 5-hour deadline. The detailed operational logic is provided in Algorithm 2 (see Appendix).
The prompts used for solving mathematical problems are intentionally minimal and role-based. Since gpt-oss-120b is natively optimized for multi-level CoT reasoning, overly verbose or complex prompts can introduce unwanted bias or lead the model toward suboptimal reasoning paths. Instead, we leveraged the model's role-based hierarchy to enforce constraints while preserving its inherent problem-solving flexibility. The strategy is organized across following prompts:
\nprint() for output, preventing empty execution results.SYSTEM: \"You are a rigorous world-class math competitor. You must return the final answer in \\boxed{}. The final answer must be a single integer (digits only) between 0 and 99999 inclusive.\"
\nDEVELOPER: \"Reason carefully to solve the problem and use the Python tool when necessary. You have access to math, numpy, sympy, mpmath, decimal, itertools, and pulp. Never guess.\"
\nTOOL DESC: \"Use this tool to execute Python code. The environment is a stateful Jupyter notebook. You must use print() to output results. Internet access is off.\"
\nUSER: \"{problem}\"
\nTo convert the model's natural language output into the required competition format, the system employs a multi-stage extraction pipeline. It first attempts to extract the answer within standard LaTeX delimiters. If no match is found, it falls back to a secondary search for internal tags typically generated by the model's final reasoning tokens. The extraction utilizes the following regular expressions:
\noxed\\s*\\{[^\\d-]*(-?[0-9,]+\\.?[0-9]*)[^\\d]*\\}assistantfinal\\s*[^\\n\\d-]*(-?[0-9,]+\\.?[0-9]*)[^\\d]*In cases where multiple matches occur within a single reasoning path, the system processes them in reverse chronological order (from the most recent to the first). This heuristic assumes that the model's final conclusion, often following a self-correction or a summary step, is more likely to be accurate than initial estimates. For each extracted candidate, a normalization function is applied to ensure compliance with the competition's constraints:
\n100,000 operation is applied. This ensures the output falls within the required [0, 99999] range, effectively handling cases where the model calculates a correct large value but omits the competition-specific modulus in its final response.A candidate is considered a valid final answer only if it results in an integer within the specified range. If a reasoning path concludes without a tool call and fails to yield a valid answer, a self-correction loop is triggered to recover a valid formatted answer. A follow-up prompt is appended to the conversation to guide the model:
\nUSER (Recovery): \"The previous response did not produce a valid final answer. Continue solving the problem. Remember that the final answer must be a single integer (digits only) between 0 and 99999 inclusive, and must be placed in \\boxed{}.\"
\nTo prevent infinite loops and manage the time budget, this recovery attempt is limited to a single iteration; if a second failure occurs, the trace is terminated.
\nWhereas our configuration supports a maximum concurrency of 7.52x, we initialize 7 parallel reasoning paths per problem to maintain a stable and efficient Self-Consistency pipeline. To enhance accuracy and minimize computational redundancy, we implement an advanced voting mechanism that integrates confidence/entropy-based tie-breaking, consensus-driven early termination, and a fallback to tool outputs.
\nStarting from the second completed reasoning path, the system applies a majority vote each time a new answer is generated. In the event of a tie, where multiple candidates share the same maximum frequency, we apply a scoring metric to select the most reliable candidate (i.e., the one with the highest score). The score for a candidate answer a, denoted as S, is calculated as:
$$S_{tie, a} = \\sum_{i \\in \\text{Paths}(a)} S_{a, i}$$
\nWhere S represents the individual score of reasoning path i that yielded answer a:
$$S_{a, i} = p_i \\cdot (1 - H_{g, i}) \\cdot C_{g, i}$$
\nWhere:
\np is a coefficient set to 0.75 if path i required Format Recovery (indicating a failure in the initial formatting), and 1.0 otherwise. As reported in the Main results section, paths requiring recovery are statistically less reliable.H is the mean entropy of the entire reasoning path i, considering the top 20 tokens for each generation step (inspired by [6]):$$H_{g, i} = \\frac{\\sum_{t=1}^{L} \\sum_{j=1}^{20} - \\exp(lp_{t,j}) \\log_2(\\exp(lp_{t,j}))}{L \\cdot \\log_2(20)}$$
\nC is the mean confidence of the entire reasoning path i (inspired by [6]):$$C_{g, i} = \\exp\\left(\\frac{1}{L} \\sum_{t=1}^{L} lp_t\\right)$$
\nlp is the log-probability of the j-th top candidate token at generation step t;lp is the log-probability of the token selected at generation step t;L is the length of reasoning path, in tokens.To optimize the inference budget, the system evaluates the consensus of the leading answer after each vote. If the agreement among reasoning paths is sufficiently robust, the process terminates early. We define four hierarchical levels of consensus based on answer frequency, the number of active tasks, and the remaining time budget.
\nmax_vote > max(2, active_tasks/2) or max_vote > still_running_tasks + second_high_frequency max_vote >= max(2, active_tasks // 2) and max_vote - 1 > second_high_frequencymax_vote >= max(2, active_tasks // 2) and max_vote > second_high_frequency and elapsed_time > available_time * 0.5max_vote >= max(2, active_tasks // 2 - 1) and max_vote - 1 > second_high_frequency and elapsed_time > available_time * 0.75Where:
\nmax_vote is the frequency of the current best answer (max frequency among answers);active_tasks includes both running tasks and completed tasks that have already produced a valid answer (i.e., excluding only those that terminated without producing a valid answer);still_running_tasks is the number of still running tasks;second_high_frequency is the frequency of the second best answer;available_time is the available time for the current problem;elapsed_time is the elapsed time from the start of the inference for the current problem.In all cases, at least 2 valid votes are required before triggering any consensus condition, to prevent early termination on potentially noisy single-path results.
\nIf all reasoning paths conclude without a valid formatted answer (often due to model timeouts during the final CoT steps), the system performs a vote on the last valid numerical output returned by the Python interpreter within each reasoning path. This fallback mechanism significantly improves robustness; empirical analysis shows that the model frequently computes the correct value via TIR (as shown in the Main results section) but may fail to encapsulate it in the required \\boxed{} format if the execution limit is reached.
\nTo optimize computational efficiency for problems requiring extended reasoning chains, we implement an early pruning strategy designed to terminate less promising reasoning paths, inspired by the DeepConf framework [6]. When the elapsed time for a problem reaches a predefined fraction of its allocated budget, and the number of active paths still exceeds a given threshold, the system evaluates all paths and prunes the lowest-scoring ones. This mechanism reallocates GPU resources to the remaining high-scoring paths, allowing them to reach greater logical depth within the residual time. In our implementation, pruning is triggered when the elapsed time exceeds 60% of the allocated budget and more than 75% of the initial reasoning paths remain active.\nEach reasoning path i is evaluated using a composite score S that integrates global and local token-level statistics. The pruning score is defined as:
$$S_i = p_i \\cdot (1 - H_{g, i}) \\cdot (1 - H_{tail, i}) \\cdot C_{g, i} \\cdot C_{tail, i}$$
\nWhere p, H and C are the same as defined in Section 3.5.1, while H and C are analogous to H and C, but are computed using only the last 2048 tokens of the reasoning path (inspired by [6]).
This scoring function effectively penalizes paths exhibiting high uncertainty (entropy) or low probability (confidence). While global metrics (H, C) provide an overview of the path's stability, the inclusion of tail metrics (H, C) is critical for detecting real-time \"hallucinations\" or logical instabilities (e.g., repetitive loops) that often emerge late in the generation process. By monitoring these fluctuations, the system can terminate decaying reasoning chains and focus the remaining temporal budget on the most robust candidates, maximizing the effectiveness of the Self-Consistency pool under strict time constraints. These combined metrics improve the accuracy of identifying and pruning reasoning paths that are likely to produce incorrect answers (as shown in the Main Results section).
The proposed methodology was implemented and evaluated within the Kaggle competition environment (Python notebook, snapshot of 2025-11-07) , leveraging an NVIDIA H100 GPU accelerator. The software stack relies on the following dependencies:
\nThe following sections describe the experimental setup in detail, including the local validation strategy, default hyperparameter configurations, and an in-depth analysis of performance across multiple experimental runs, with a particular focus on ablation studies, overall performance and comparison with state of the art.
\nThe gpt-oss-120b model is loaded into GPU memory at the beginning of execution and persists throughout the entire inference process. To accelerate the initialization phase, model artifacts are first copied into RAM using multiprocessing, reducing load time from approximately 900s to ~250s. During this stage, all environment variables required for vLLM optimization and logging are configured.\nTo ensure strict compliance with competition time constraints and to prevent resource contention, the system’s core logic is decoupled into isolated processes using the multiprocessing library. These processes communicate via standardized queues, enabling robust and fault-tolerant execution.\nThe overall architecture is organized as follows:
\nDetailed pseudocode for system orchestration and inference workflows is provided in Algorithm 3 and Algorithm 4 in the Appendix.
\nFor local testing, validation, and hyperparameter optimization, we employ a high-quality mathematical reasoning dataset comprising 75 problems. This benchmark was specifically curated for the AIMO3 competition, with a focus on evaluating reasoning reliability through executable Python-based TIR (see imo-benchmark.csv in [15]). In alignment with the competition requirements, each problem is associated with a ground-truth integer answer within the range [0, 99999].\nThe primary motivation for selecting this dataset is its high level of difficulty. This complexity is critical for accurately measuring performance gains during the optimization process. In contrast to simpler benchmarks, where many problems can be solved through shallow reasoning, this dataset provides a strong discriminative signal, enabling a clear assessment of the effectiveness of advanced reasoning strategies and test-time optimization techniques.
Several parameters were inspired by high-performing public notebooks [12][13][14], which achieved strong results on both reference problems and the public leaderboard. To evaluate the individual contribution of each optimization strategy, we perform a series of controlled experimental runs. Each run isolates a specific component by disabling it while keeping all other parameters constant, following a standard ablation study methodology.\nThe primary hyperparameters used across all runs are summarized in Table 1 in the Appendix. While the validation set consists of 75 problems (compared to 50 in the public/private leaderboard), the configuration remains identical to our final competition submission, with the total time budget scaled proportionally (7.5 hours for validation vs. 5 hours for submission).
\nAdditional vLLM optimizations were applied to reduce latency and memory overhead, including chunked prefill, prefix caching, asynchronous scheduling, and reduced CUDA graph capture size and sequence limits. Standard values were retained for top_p and top_k. All runs in Section 4.4 use this configuration unless otherwise specified. Additional implementation details are available in the submission notebook ([16] refers to the submission notebook, [17] refers to required dependencies, [21] refers to Google Colab version of submission notebook).
\nTo rigorously evaluate the impact of each architectural component, we conducted a series of experimental runs, described in Table 2. Each run is based on the \"final_submission\" configuration, with the deactivation of a specific feature to isolate its effect on performance, except for the \"baseline\" run, where all new optimizations are disabled, and the \"time_constraints_disabled\", where only time constraints are removed.
\n| Run Name | \nDescription | \n
|---|---|
| time_constraints_disabled | \nRemoves time constraints to the final submission to assess the model's unconstrained capability ceiling | \n
| final_submission | \nConfiguration used for the final submission, including all optimizations | \n
| early_break_disabled | \nDisables dynamic pruning of low-scoring reasoning paths | \n
| early_termination_disabled | \nDisables the consensus-driven early termination mechanism | \n
| medium_reasoning | \nReduces the reasoning level of gpt-oss-120b from HIGH to MEDIUM | \n
| dynamic_time_management_disabled | \nUses static time allocation (average time per problem instead of dynamic time management) | \n
| python_tool_disabled | \nUses pure Chain-of-Thought (CoT) reasoning without Python tool integration | \n
| baseline | \nBaseline before new optimizations | \n
The baseline run represents the most advanced configuration achievable using standard practices within the competition's constraints, characterized by:
\nk=7 parallel sampling approach with majority voting;It is important to note that while gpt-oss-120b natively supports Tool-Integrated Reasoning (TIR), the official implementation (requiring gpt-oss>=0.0.5 and Python 3.12) proved incompatible with the Kaggle H100 environment (Python 3.11.13). Consequently, the baseline represents the model's unassisted reasoning limit, while our work introduces a custom-built, compatible TIR orchestration layer.
\nEach run generates a detailed telemetry log capturing key metrics for every reasoning path (runs are available in [18], while generated logs are available in [19]). These logs are aggregated and analyzed to compute per-run performance statistics as well as global reasoning trends, which are presented in the following charts and tables (statistics and charts logic are available in [20]).
\nThe following sections present a detailed analysis of the performance metrics obtained from the experimental runs described above. Results are evaluated across multiple dimensions, including ablation studies designed to isolate the contribution of each component and overall accuracy compared against the state-of-the-art baseline.\nIn particular, we analyze how individual test-time optimization strategies affect final accuracy. These results provide a comprehensive view of the system’s behavior, demonstrating how each component contributes to solving complex mathematical problems under strict time and resource constraints.
\nThis section mainly evaluates the individual contribution of each optimization strategy to the overall system performance. By systematically deactivating specific components, we isolate their impact on accuracy, providing empirical justification for the architectural choices described in Section 3.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F23110311%2Fa534f8cf678921b130b1bd54de970db6%2Fof%20incomplete%20wrong%20and%20correct%20reasoning%20paths%20per%20run.png?generation=1776729740190033&alt=media\")
\nThe chart illustrates the distribution of incomplete, incorrect, and correct reasoning paths across the different experimental configurations. This global metric aggregates all individual traces generated for every problem, regardless of the final consensus answer.\nThe data demonstrates that TIR and dynamic time management significantly improve the yield of correct reasoning paths. Conversely, the HIGH reasoning level is primarily responsible for reducing the incidence of logical fallacies and incorrect outputs. The early break logic provides a marginal gain in the number of correct paths by reallocating resources to more promising chains.\nAs expected, removing time constraints altogether results in the highest proportion of correct paths, establishing the model's theoretical \"capability ceiling.\" Interestingly, disabling consensus-based early termination appears to slightly increase the count of correct paths; however, this is likely an artifact of redundant computations on simpler problems that would have otherwise terminated early, rather than a genuine improvement in solving more complex tasks.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F23110311%2F9503d0ccc3e12c6632d7d4bfb4711391%2Fpass7%20cons7%20and%20conf_conf7%20accuracy%20per%20run.png?generation=1776729780368012&alt=media\")
\nThe chart compares three distinct accuracy metrics across each experimental run:
The results demonstrate that the conf_cons@7 metric consistently outperforms the standard cons@7, although a gap remains compared to the pass@7 theoretical maximum. Consistent with previous observations, Python tool integration provides the most significant accuracy gain. However, each supplementary technique contributes to a cumulative improvement in the final score, most notably the dynamic time management and HIGH reasoning level, followed by the incremental gains from early break and early termination logic. Finally, removing time constraints establishes a higher accuracy ceiling, confirming that the model's performance is currently limited more by the inference budget than by its intrinsic reasoning capabilities.
\nRegarding the gain provided by the Python tool integration, while the used validation set employed is tool-oriented, the benefits of TIR will partially transfer to more \"abstract\" domains, albeit with a shift in how the tool is utilized. For instance:
\nAs for the performance gains driven by dynamic time management, there is room for further optimization. As reported in Section 3.3., it is based on the scaling constants Cmin and Cmax. Initially, the parameter Cmax=2.1 was empirically selected to allow the model to reach its maximum effective context window depth on difficult problems. The following table summarizes additional configurations evaluated during the final phases of the competition:
| Cmin | \nCmax | \nPublic LB Scores | \nMean Score | \n
|---|---|---|---|
| 0.88 | \n2.2 | \n37 | \n37.0 | \n
| 0.86 | \n2.1 | \n39, 37, 38, 43 | \n39.25 (Final Submission) | \n
| 0.85 | \n2.0 | \n36 | \n36.0 | \n
| 0.80 | \n2.0 | \n39 | \n39.0 | \n
| 0.80 | \n2.1 | \n42, 41 | \n41.5 | \n
While the selected final submission (Cmin=0.86, Cmax=2.1) achieved the highest peak score of 43/50, subsequent analysis suggests that a lower floor (Cmin=0.80) paired with Cmax=2.1 yields a more stable mean performance (41.5).\nThe configuration (0.80, 2.1) was identified very close to the competition deadline, consequently, we prioritized the (0.86, 2.1) configuration for the final submission as it had already demonstrated high-ceiling potential in earlier runs.\nThese results indicate that performance is sensitive to the Cmin floor. A lower Cmin grants the system more flexibility to reallocate time from \"easy\" problems to \"hard\" ones, whereas an excessively high Cmax may lead to diminishing returns if the model exceeds its useful reasoning depth.
Mathematical reasoning tasks are sensitive to problem ordering and stochastic sampling, and variance estimates are crucial to distinguish architectural gains from statistical noise. The experimental runs were conducted near the competition deadline to provide a reproducible and objective summary of each technique’s behavior on a high-difficulty public dataset. We acknowledge that these specific local runs reflect a single execution. Due to the significant computational intensity—each 75-problem run requiring approximately 7.5 hours on an NVIDIA H100 GPU—and strict hardware availability constraints, conducting multiple full-scale repetitions was not feasible within the remaining timeframe. \nHowever, the significance of the observed performance gaps is supported by the following factors:
\nk=7) reduces the impact of stochastic sampling.| Notebook Configuration | \nPublic LB Scores | \nMean Score | \n
|---|---|---|
| Final Submission | \n39, 37, 38, 43 | \n39.25 | \n
| Early Break Disabled | \n36, 38 | \n37 | \n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F23110311%2Fd6b8d4ef1100854418395f55909038a5%2FCorrect%20selection%20of%20wrong%20answers%20()%20by%20chosing%20the%20worst%20value%20of%20different%20metrics.png?generation=1776729854362148&alt=media\")
\nThis chart evaluates the capability of different token-level metrics to correctly identify incorrect reasoning paths within \"ambiguous\" problems (defined as those where the model generated both correct and incorrect answers). By selecting the path with the worst value for a given metric, we measure how frequently that metric successfully points to a wrong answer.\nThe results indicate that tail entropy is the most reliable indicator of a failing reasoning chain, achieving the highest precision in identifying incorrect paths. This is followed by global mean entropy, tail confidence, and mean confidence. These findings empirically justify the composite \"early_break_score\" used in our pruning logic (Section 3.6). By using this metric, the system can effectively detect late-stage hallucinations or logical instabilities, allowing for the strategic termination of suboptimal paths and the reallocation of computational budget to more promising candidates.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F23110311%2Fb9ceb877d71d35ec7b4cefa680537dc2%2FTie-breaking%20success%20rate%20()%20per%20summed%20metric.png?generation=1778966896557181&alt=media\")
\nThis chart illustrates the effectiveness of various metrics when used to resolve ties in \"ambiguous\" problems (those containing both correct and incorrect paths where no clear majority exists). The success rate is measured by selecting the answer with the highest aggregate score, where the score is the sum of the metric values across all paths yielding that specific answer.\nWhile the selected final submission utilized a tie_break_score based on global mean confidence and global mean entropy (Section 3.5.1), subsequent analyses reveal that alternative metrics may offer superior discriminative power. In particular, incorporating local confidence bottlenecks and reasoning-path length can produce substantially stronger tie-breaking performance.
| Metric | \nFormula | \n
|---|---|
| Revised tie_break_score_A | \n$$(1 - \\text{length_perc}) \\cdot \\text{min_conf}$$ | \n
| Revised tie_break_score_B | \n$$\\text{tie_break_score} \\cdot (1 - \\text{length_perc}) \\cdot \\text{min_conf}$$ | \n
| Revised tie_break_score_C | \n$$\\text{tie_break_score} \\cdot (1 - \\text{length_perc}) \\cdot \\text{min_conf} \\cdot \\text{tail_conf}$$ | \n
Here, length_perc denotes the ratio between the generated response length and the model context limit (response_length / max_model_len).\nThese results suggest that localized confidence failures are more informative than global averages. Metrics such as mean_conf and mean_entropy can remain relatively stable even when a reasoning trajectory contains a critical logical error. In contrast, min_conf and tail_conf act as bottleneck indicators: a single low-confidence transition often coincides with a hallucinated deduction or unstable reasoning step and can act as a \"pivot point\" leading to reasoning divergence.\nThe inverse correlation between reasoning-path length and correctness is also notable. For Olympiad-style mathematical problems, successful solutions tend to emerge from relatively concise reasoning chains, whereas excessively long trajectories are more likely to exhibit reasoning drift, repetition, or self-inconsistent exploration.\nThese results suggest that combining global stability metrics with local confidence statistics and trajectory-length regularization may provide a more robust tie-breaking strategy for adaptive reasoning systems.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F23110311%2F81ad1a4852a908d9f57d0444e443ab52%2FCorrect%20and%20wrong%20answers%20()%20which%20used%20format-recovery%20mechanism.png?generation=1776730054933917&alt=media\")
\nThe chart illustrates the distribution of correct and incorrect answers for reasoning paths that required the Format Recovery mechanism during inference. The data reveals a significant correlation between formatting failures and logical errors: reasoning paths necessitating recovery are nearly twice as likely to produce an incorrect answer compared to those that adhere to the requested format natively. This empirical evidence justifies our decision to incorporate a penalty coefficient for recovered paths within both the tie-breaking and early-break scoring functions (Sections 3.5.1 and 3.6). By discounting these structurally weaker traces, the system increases the probability of selecting the most robust and consistent reasoning chains as the final output.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F23110311%2F86ad830e51b650eff4617779595a3d12%2FGuessed%20correct%20answers%20()%20by%20last%20tool%20valid%20execution.png?generation=1776730097700401&alt=media\")
\nThis chart evaluates the correspondence between the final valid integer output from the Python interpreter and the ground-truth answer across all reasoning path of experimental runs. The data demonstrates that utilizing the last valid tool output as a fallback is a highly effective strategy for problems that reach the execution timeout before a formal conclusion can be generated.\nIn many high-complexity cases, the model successfully computes the correct value via a tool call in the final stages of reasoning, but lacks the remaining temporal budget to synthesize the final Chain-of-Thought (CoT) and the required LaTeX formatting. \nOn the 75-problem validation set, excluding the python_tool_disabled and baseline ablations (which do not use the Python tool), the tool-output fallback mechanism was triggered 45 times in total. A fallback event occurs when all 7 reasoning paths fail to produce a correctly formatted final answer, but at least one reasoning path generated a valid numerical result during a Python tool execution. Among these 45 fallback activations, the mechanism recovered the correct answer 3 times, corresponding to a conditional accuracy of approximately 6.67% once the fallback fires.\nAlthough the conditional success rate is relatively low, the fallback was intentionally designed as a last-resort recovery mechanism rather than a primary selection strategy. In these cases, the standard extraction and aggregation pipeline had already failed completely, meaning that without the fallback the system would deterministically output an incorrect answer. Consequently, even a small number of successful recoveries translates directly into measurable score improvements. Importantly, the fallback introduces essentially no additional inference-time cost, since it only reuses numerical outputs already produced during earlier Python tool executions rather than generating new reasoning trajectories. This empirically justifies our implementation of a consensus-based vote on tool outputs (Section 3.5.3) when no reasoning path reaches a natural termination within the allotted time.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F23110311%2Fbd19d9e9ea9f5db56bf1b739f841a479%2FGlobal%20length%20of%20reasoning%20paths%20per%20run.png?generation=1776730147413787&alt=media\")
\nThe chart illustrates the total length (in tokens) of all reasoning paths generated across each experimental run. The following observations can be made:
The integration of the strategies discussed in the previous sections culminated in a robust mathematical solver. This framework effectively raised the model's capability ceiling by optimizing the inference process. To quantify this gain, we compared our final submission against a standard state-of-the-art (SOTA) baseline. This baseline represents the most advanced configuration achievable using standard practices within the competition's constraints, as described in Section 4.4.
\nIn local validation (on selected hard problems), our final submission achieved a score of 41/75, which increased to 47/75 when time constraints were removed. In comparison, the initial baseline score was 17/75.
\nOn the AIMO3 Public Leaderboard, our selected final submission achieved a maximum score of 43/50 and a mean score of 39.25/50 (based on four submissions with range 37-43).
\nOn the AIMO3 Private Leaderboard, our selected final submission achieved a score of 42.0/50, ranking 373rd out of 4,138.
\nThese results could be further improved by increasing the number of parallel paths per problem and expanding the context window on more powerful hardware.
\nIn this work, we presented a novel strategy designed for the AIMO3 competition. By integrating a custom-built, fault-tolerant Tool-Integrated Reasoning (TIR) layer with adaptive test-time optimizations, including dynamic time management, lightweight role-based CoT prompting with robust format recovery, Self-Consistency with consensus-based early termination and confidence-/entropy-based tie-breaking, adaptive pruning of less promising reasoning paths, we successfully extended the mathematical reasoning capabilities of the gpt-oss-120b model.\nOur experimental results demonstrate that the bottleneck in automated mathematical problem-solving often lies in the efficiency of the inference process, rather than just the model's intrinsic knowledge. The transition from a standard baseline to our optimized strategy yielded a significant performance gain, confirming that a deterministic execution environment paired with strategic test-time optimizations can achieve state-of-the-art results.\nMoving forward, we identify several promising avenues for improvement:
\n[1] Simon Frieder, Sam Bealing, Philip Vonderlind, Sida Li, Arsenii Nikolaiev, Geoff C. Smith, Kevin Buzzard, Timothy Gowers, Peter J. Liu, Po-Shen Loh, Lester Mackey, Leonardo de Moura, Dan Roberts, D. Sculley, Terence Tao, David Balduzzi, Simon Coyle, Alex Gerko, Ryan Holbrook, Addison Howard, and XTX Markets. AI Mathematical Olympiad - Progress Prize 3. https://kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3, 2025. Kaggle.
\n[2] Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Henkel, Benedikt Schifferer, Wei Du, Igor Gitman. AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset, 2025. URL https://arxiv.org/pdf/2504.16891.
\n[3] Heng Lin and Zhongwen Xu. Understanding Tool-Integrated Reasoning, 2025. URL https://arxiv.org/pdf/2508.19201.
\n[4] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, Denny Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, 2023. URL https://arxiv.org/pdf/2201.11903.
\n[5] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2023. URL https://arxiv.org/pdf/2203.11171.
\n[6] Yichao Fu, Xuewei Wang, Yuandong Tian, Jiawei Zhao. Deep think with confidence, 2025. URL https://arxiv.org/pdf/2508.15260.
\n[7] OpenAI. gpt-oss-120b & gpt-oss-20b Model Card, 2025. URL https://arxiv.org/pdf/2508.10925, https://huggingface.co/openai/gpt-oss-120b, https://www.kaggle.com/models/danielhanchen/gpt-oss-120b/Transformers/default/1.
\n[8] openai-harmony 0.0.8. URL https://pypi.org/project/openai-harmony/0.0.8/.
\n[9] vllm 0.11.2. URL https://pypi.org/project/vllm/0.11.2/, https://docs.vllm.ai/projects/recipes/en/latest/OpenAI/GPT-OSS.html#troubleshooting, https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken, https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken.
\n[10] numpy 2.2.0. URL https://pypi.org/project/numpy/2.2.0/.
\n[11] PuLP 3.3.0. https://pypi.org/project/PuLP/3.3.0/
\n[12] ShelterW. [Way to 30+/50] GPT-OSS - w/ python tool. URL https://www.kaggle.com/code/shelterw/way-to-30-50-gpt-oss-w-python-tool?scriptVersionId=285192299
\n[13] Andreas Bisiadis. AIMO 3 | GPT-OSS-120B (with tools). URL https://www.kaggle.com/code/andreasbis/aimo-3-gpt-oss-120b-with-tools?scriptVersionId=290379281
\n[14] parthenos. [44/50] LET ME (over)COOK!!!. URL https://www.kaggle.com/code/nihilisticneuralnet/44-50-skill-optional-luck-zaroori?scriptVersionId=291499043
\n[15] wenliangtlh. AIMO3-High-Difficulty-Tool-Calling-Dataset. URL https://www.kaggle.com/datasets/wenliangtlh/aimo3-high-difficulty-tool-calling-dataset
\n[16] Potrhino. AIMO3-v3, version 11 (final submission notebook). URL https://www.kaggle.com/code/potrhino/aimo3-v3?scriptVersionId=302225896
\n[17] Potrhino. AIMO3-dependencies-v1, version 10 (dependencies of final submission notebook). URL https://www.kaggle.com/code/potrhino/aimo3-dependencies-v1?scriptVersionId=297334166
\n[18] Potrhino. AIMO3-v3-v11-runs (validation runs notebook for main results). URL https://www.kaggle.com/code/potrhino/aimo3-v3-v11-runs?scriptVersionId=313133192
\n[19] Potrhino. AIMO3-local-logs (validation results dataset). URL https://www.kaggle.com/datasets/potrhino/aimo3-local-logs
\n[20] Potrhino. AIMO3-v3-v11-results (validation stats & charts notebook). URL https://www.kaggle.com/code/potrhino/aimo3-v3-v11-results?scriptVersionId=313200617
\n[21] Potrhino. potrhino-aimo3-v3.ipynb (Google Colab version of final submission notebook). https://colab.research.google.com/drive/1vXAcdEiVcTM7oeQTFcsZsfQ6-ms33TK5?usp=sharing
\n| Param | \nValue | \nNote | \n
|---|---|---|
| device | \ncuda | \nEnables NVIDIA H100 GPU acceleration required to run gpt-oss-120b | \n
| dtype | \nauto | \nUses default precision (bfloat16) | \n
| kv-cache-dtype | \nfp8_e4m3 | \nReduces memory footprint vs. bf16, enabling higher parallelism with negligible precision loss | \n
| gpu_memory_utilization | \n0.97 | \nMaximizes GPU memory usage during inference | \n
| max_model_len | \n64*1024 | \nBalances reasoning depth and VRAM constraints | \n
| enable_tool | \nTrue | \nEnables Python tool usage during reasoning | \n
| reasoning | \nHIGH | \nActivates the highest CoT reasoning level | \n
| temperature | \n1.0 | \nBalances stochasticity for diverse Self-Consistency sampling | \n
| min_p | \n0.02 | \nFilters low-probability tokens while preserving diversity | \n
| reasoning_paths | \n7 | \nNumber of parallel reasoning paths for majority voting | \n
| max_time | \nint(560601.5 - 3.560) | \nTotal available time budget | \n
| problems_number | \n50+25 | \nTotal number of problems in validation | \n
| enable_dynamic_time_management | \nTrue | \nEnables adaptive per-problem time allocation | \n
| min_time_coef | \n0.86 | \nLower bound for time allocation scaling | \n
| max_time_coef | \n2.1 | \nUpper bound for time allocation scaling | \n
| top_logprobs | \n20 | \nUsed to compute entropy based on top-20 token probabilities [6] | \n
| group_logprobs_len | \n2048 | \nWindow size for tail entropy/confidence estimation [6] | \n
| enable_early_break | \nTrue | \nEnables pruning of low-scoring reasoning paths | \n
| execution_timeout | \n8 | \nInitial execution timeout (seconds) for Python tool calls | \n
| max_tool_step | \n200 | \nMaximum number of tool calls per reasoning path | \n
| max_tool_output_len | \n1000 | \nMaximum tool output length before truncation | \n
| max_tool_error | \n5 | \nMaximum number of tool errors before environment reset | \n
| precision | \n100000 | \nNumerical precision for Python tool environment | \n
| int_max_digits | \n1000000 | \nMaximum integer size supported for Python tool environment | \n
| recursion_limit | \n2000 | \nMaximum recursion depth for Python tool environment | \n
| enable_early_termination | \nTrue | \nEnables consensus-based early termination | \n
Initialize Jupyter kernel session\n Configure Environment:\n - Pre-import libraries (math, numpy, sympy, etc.)\n - Set high float precision and integer max_digits\n - Increase recursion depth limit\n\n While True:\n Receive {Python_code, available_time} from Main Process\n\n If Python_code is None:\n Shutdown Kernel and Exit\n\n If Python_code is \"CLEAN\":\n Restart Jupyter Kernel and re-initialize environment\n Return empty confirmation\n\n Else:\n # Determine execution limit using Adaptive Timeout logic\n T_limit = Calculate_Timeout(available_time, consecutive_timeouts)\n\n # Execute code in the stateful session\n Output = Execute_in_Jupyter(Python_code, timeout=T_limit)\n\n # Post-process Output (Output Length Control)\n If length(Output) > THRESHOLD:\n Output = Truncate_Center(Output, max_length=THRESHOLD)\n\n Return Output\n #Init constants\n START_TIME = time.now() \n ...initialization logic...\n INIT_OVERHEAD = time.now() - START_TIME\n TOTAL_CLOCK = int(5*60*60 - 3.5*60) #5 hours with a safety margin of 3.5 minutes (in seconds)\n DEADLINE = START_TIME + TOTAL_CLOCK\n TOTAL_PROBLEMS = 50\n C_min = 0.86 \n C_max = 2.1 \n T_avg = (TOTAL_CLOCK - INIT_OVERHEAD) / TOTAL_PROBLEMS \n\n #Main loop\n processed_problems = 0\n for each incoming problem P:\n # Reserve time for all future problems\n remaining_problems = TOTAL_PROBLEMS - processed_problems - 1\n reserved_buffer = remaining_problems * (T_avg * C_min)\n # Calculate current budget\n T_pool = DEADLINE - time.now() - reserved_buffer\n # Apply elastic constraints\n available_time = max( min(T_pool, T_avg * C_max), T_avg * C_min )\n # Final problem consumes all remaining time\n if remaining_problems == 0:\n available_time = DEADLINE - time.now()\n #Solve the problem within available_time\n solve(P, time_limit=available_time)\n processed_problems += 1\n Init global configurations\n Spawn Process \"PM\": Load LLM into memory\n Spawn Process Pool \"PTP\": Stateful Python Tool environments (one per parallel path)\n\n For each incoming problem \"P\":\n If \"P\" is None: \n Shutdown \"PTP\" pool and \"PM\" process\n Break\n\n Compute available time \"T\" # Via Dynamic Time Management logic\n Init \"AG\": List for valid reasoning path answers\n Init \"ATG\": List for fallback tool-only answers\n Init Stop Event \"S\": For early termination\n Init Global Metrics \"GM\": For cross-thread pruning/scoring\n\n Start Thread Pool \"TP\":\n Execute parallel inference on \"P\" using \"PM\" (LLM) and \"PTP\" (Tools)\n\n Init \"best_answer\" to 0\n Init \"max_vote\" to 0\n For each completed thread \"j\" in \"TP\":\n Get response metrics \"A_j\"\n If \"A_j\" contains a valid extracted answer:\n Add \"A_j\" to \"AG\"\n # Update Best Answer and relative vote using Self-Consistency + Tie-Break\n \"best_answer\", \"max_vote\" = SmartVote(\"AG\")\n\n If Consensus-Based Early Termination criteria met:\n Signal Stop Event \"S\" to halt remaining threads\n Break\n Else:\n # Fallback: check if the last Python tool output is a valid number\n If \"AT_j\" contains valid tool result:\n Add \"AT_j\" to \"ATG\"\n\n # Final Fallback: If no reasoning path finished, vote on Tool outputs\n If \"max_vote\" == 0 and \"ATG\" is not empty:\n \"best_answer\", \"max_vote\" = SmartVote(\"ATG\")\n\n Shutdown Thread Pool \"TP\" (cleanup)\n Return \"best_answer\"\n Init conversation \"C\" with role-based prompts and tool definitions\n Init \"available_steps\" = 200\n Init \"FormatRecoveryEnabled\" = True\n Init \"response_text\", \"logprobs\", \"top_logprobs\" as empty\n\n While \"available_steps\" > 0:\n If context_window_exhausted: # Safety margin of 4096 to allow final answer/tool output\n Break \n\n Open Response Stream \"RS\" from LLM\n For each chunk in \"RS\":\n If Stop_Event \"S\" is set or global_time_limit reached:\n Break\n\n Update \"response_text\", \"logprobs\", and \"top_logprobs\"\n\n # Adaptive Pruning Check (every 2048 tokens)\n If Pruning_Enabled and tokens_since_last_check > 2048:\n Compute reasoning metrics (Entropy/Confidence)\n Update Global Metrics \"GM[i]\"\n If pruning_criteria_met: # Based on elapsed time, active tasks and relative score\n Break\n\n If Stop_Event \"S\" is set: Break\n\n Extract Numerical Answer \"A_j\"\n If \"A_j\" is valid: \n Break # Success: Answer found\n\n if global_time_limit reached or pruning_criteria_met:\n break\n\n Update conversation \"C\" with current reasoning step (reasoning, tool call)\n\n If current response contains a Tool Call: \n Extract generated Python code\n Execute code in Jupyter session \"PTP[i]\" -> Get result \"RES\"\n Update conversation \"C\" with Tool Output\n If \"RES\" is a valid integer answer:\n Save as Fallback Tool Answer \"AT_j\"\n Else: # No tool call and no valid answer found\n If \"FormatRecoveryEnabled\":\n Update \"C\" with a new User Prompt for Format recovery\n \"FormatRecoveryEnabled\" = False # Only one recovery attempt allowed\n Else:\n Break # Failed to recover format\n\n \"available_steps\" -= 1\n\n Clean Jupyter Session \"PTP[i]\"\n Compute final response metrics (Mean/Tail Entropy, Confidence, etc.)\n Return Metrics and Candidate Answers (\"A_j\", \"AT_j\")\nEdited on May 17th 2026 for the rebuttal phase:
\nCmin and Cmax variants) and provided a dedicated ablation study table comparing their Public LB mean scores.Adding to the comments from @sambealing below, I have a few additional questions:
\nCoT: Why did you choose to use CoT-style prompting with a model that has already been extensively trained for reasoning? The official OpenAI reasoning best practices recommends avoiding CoT-style prompts for models with built-in reasoning capabilities, since they already reason internally. Did you also test your approach with a different prompt format that does not explicitly encourage CoT-style reasoning?
Time Management: Time management is a crucial part of this competition, with many possible areas for improvement. Since you present an adaptive time-management system as a key component, it would be helpful to see ablations comparing different strategies and configurations.
Variance on Local Evaluations: As we have seen in AIMO3, variance has a large impact on notebook scores due to problem difficulty and random ordering. Did you run your experiments multiple times and record local variance? I could only find reported averages for the public leaderboard scores, but not for the individual configurations tested on your 75-problem evaluation dataset. Adding variance estimates would allow readers to better assess the statistical significance of the differences between configurations.
Hi @philipvonderlind ,
\nthank you for your time and for these insightful questions.
\nPlease find my detailed responses to each point below.
\nWhile official OpenAI best practices suggest that CoT-style prompts are unnecessary for models with built-in reasoning, my empirical results with gpt-oss-120b tell a more nuanced story. In local runs, I observed that CoT prompts increased the number of correct responses for the hardest problems, although they generally generated longer responses for simpler tasks compared to formats that do not explicitly encourage CoT. To evaluate this, I submitted two versions of a previous notebook that differed only in the \"User\" prompt, but used a fixed configuration for \"System\", \"Developer\" and \"Tool\", as following:
\nmath, numpy, sympy and mpmath to solve the problem.\"The CoT version achieved a score of 39 on the Public Leaderboard, while the Non-CoT version achieved a score of 37.\nFurthermore, in https://arxiv.org/pdf/2508.15260 the authors achieve 99.9% accuracy on AIME 2025 using gpt-oss-120b by specifically appending \"Please reason step by step, and put your final answer within \\boxed{}.\" to the user prompt (Section \"F EXPERIMENTAL SETTINGS\").
\nI also tested several other prompting strategies:
\nUltimately, I found that using the word \"carefully\" provided the best tradeoff. It encourages precision, caution, and consistency without forcing the model into the extreme verbosity of a full \"step-by-step\" instruction. This approach can help to reduces errors on difficult problems while maintaining efficiency on simpler ones.
\nThe dynamic time management system was designed to mitigate the inefficiency of static allocation, where simple problems can waste budget and complex ones are cut short. Our ablation study confirms that the final_submission (dynamic) significantly outperforms the dynamic_time_management_disabled (static) run, which relies on a fixed average time per problem.\nThe dynamic time management is based on the scaling constants Cmin and Cmax (Section 3.3.). Initially, the parameter Cmax=2.1 was empirically selected to allow the model to reach its maximum effective context window depth on difficult problems. The following table summarizes additional configurations evaluated during the final phases of the competition:
| Cmin | \nCmax | \nPublic LB Scores | \nMean Score | \n
|---|---|---|---|
| 0.88 | \n2.2 | \n37 | \n37.0 | \n
| 0.86 | \n2.1 | \n39, 37, 38, 43 | \n39.25 (Final Submission) | \n
| 0.85 | \n2.0 | \n36 | \n36.0 | \n
| 0.80 | \n2.0 | \n39 | \n39.0 | \n
| 0.80 | \n2.1 | \n42, 41 | \n41.5 | \n
While the selected final submission (Cmin=0.86, Cmax=2.1) achieved the highest peak score of 43/50, subsequent analysis suggests that a lower floor (Cmin=0.80) paired with Cmax=2.1 yields a more stable mean performance (41.5).\nThe configuration (0.80, 2.1) was identified very close to the competition deadline, consequently, we prioritized the (0.86, 2.1) configuration for the final submission as it had already demonstrated high-ceiling potential in earlier runs.\nThese results indicate that performance is sensitive to the Cmin floor. A lower Cmin grants the system more flexibility to reallocate time from \"easy\" problems to \"hard\" ones, whereas an excessively high Cmax may lead to diminishing returns if the model exceeds its useful reasoning depth.\nWe agree that further systematic ablations on these hyperparameters could define a more precise \"Pareto front\" for time-constrained mathematical reasoning.
We appreciate the emphasis on variance. Mathematical reasoning tasks are indeed sensitive to problem ordering and stochastic sampling, and variance estimates are crucial to distinguish architectural gains from statistical noise.\nThe experimental runs reported in Section 4.5 were conducted near the competition deadline to provide a reproducible and objective summary of each technique’s behavior on a high-difficulty public dataset. We acknowledge that these specific local runs reflect a single execution. Due to the significant computational intensity—each 75-problem run requiring approximately 7.5 hours on an NVIDIA H100 GPU—and strict hardware availability constraints, conducting multiple full-scale repetitions was not feasible within the remaining timeframe.\nHowever, we believe the significance of the observed performance gaps is supported by the following factors:
\nk=7) reduces the impact of stochastic sampling.| Notebook Configuration | \nPublic LB Scores | \nMean Score | \n
|---|---|---|
| Final Submission | \n39, 37, 38, 43 | \n39.25 | \n
| Early Break Disabled | \n36, 38 | \n37 | \n
Hi, thank you for your submission to the Writeup Prize. @philipvonderlind and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.
\nmean_conf · (1-mean_entropy) aggregation. Did you have a chance to resubmit with the revised scoring function before the deadline, and if not, do you have any data of how it might affect performance on other benchmarks?Look forward to reading your responses on these.
", "votes": 0, "replies": [ { "id": 3458925, "author_name": "Potrhino", "author_url": "", "post_date": "2026-05-17T00:11:20.943000", "content": "Hi @sambealing ,
\nthank you for your time and for these insightful questions.
\nPlease find my detailed responses to each point below.
\nUnfortunately, I completed the tie-breaking analysis only after the competition deadline, so I was unable to submit a revised scoring function to the Public Leaderboard. However, post-hoc analysis on the experimental runs described in Section 4.5.1.4 indicates that the original aggregation (described in Section 3.5.1.) was not the most discriminative signal for selecting among candidate reasoning paths.\nIn particular, incorporating local confidence bottlenecks and reasoning-path length produced substantially stronger tie-breaking performance:
\n| Metric | \nTie-breaking accuracy | \n
|---|---|
| Original tie_break_score | \n0.492 | \n
Revised tie_break_score_A (1−length_perc)⋅min_conf | \n0.542 | \n
Revised tie_break_score_B tie_break_score⋅(1−length_perc)⋅min_conf | \n0.610 | \n
Revised tie_break_score_C tie_break_score⋅(1−length_perc)⋅min_conf⋅tail_conf | \n0.661 | \n
Here, length_perc denotes the ratio between the generated response length and the model context limit (response_length / max_model_len).\nThese results suggest that localized confidence failures are more informative than global averages. Metrics such as mean_conf and mean_entropy can remain relatively stable even when a reasoning trajectory contains a critical logical error. In contrast, min_conf and tail_conf act as bottleneck indicators: a single low-confidence transition often coincides with a hallucinated deduction or unstable reasoning step.\nThe inverse correlation between reasoning-path length and correctness was also notable. For Olympiad-style mathematical problems, successful solutions tended to emerge from relatively concise reasoning chains, whereas excessively long trajectories were more likely to exhibit reasoning drift, repetition, or self-inconsistent exploration.\nThese results, summarized in the accompanying chart, suggest that combining global stability metrics with local confidence statistics and trajectory-length regularization may provide a more robust tie-breaking strategy for adaptive reasoning systems.\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F23110311%2F566687e5063e21dec95c4570a1294636%2FTie-breaking%20success%20rate%20()%20per%20summed%20metric.png?generation=1778976268035266&alt=media\")
On the 75-problem validation set, excluding the python_tool_disabled and baseline ablations (which do not use the Python tool), the tool-output fallback mechanism was triggered 45 times in total.\nA fallback event occurs when all 7 reasoning paths fail to produce a correctly formatted final answer, but at least one reasoning path generated a valid numerical result during a Python tool execution.\nAmong these 45 fallback activations, the mechanism recovered the correct answer 3 times, corresponding to a conditional accuracy of approximately 6.67% once the fallback fires.\nAlthough the conditional success rate is relatively low, the fallback was intentionally designed as a last-resort recovery mechanism rather than a primary selection strategy. In these cases, the standard extraction and aggregation pipeline had already failed completely, meaning that without the fallback the system would deterministically output an incorrect answer.\nConsequently, even a small number of successful recoveries translates directly into measurable score improvements. Importantly, the fallback introduces essentially no additional inference-time cost, since it only reuses numerical outputs already produced during earlier Python tool executions rather than generating new reasoning trajectories.
\nWhile the validation set was indeed tool-oriented, I believe the benefits of TIR would partially transfer to more \"abstract\" domains, albeit with a shift in how the tool is utilized. For instance:
\nsympy, the model can offload complex algebraic manipulations or coordinate geometry calculations to a deterministic solver. This minimizes \"calculation drift\" in long CoT sequences, where models often make small sign errors or arithmetic slips.Therefore, while the magnitude of the gain might vary depending on the problem type, I expect the system to maintain a competitive edge even on benchmarks where tool-use is less intuitive.
", "votes": 0, "replies": [] } ] } ], "comment_count": 5, "messages": [ { "id": 3443347, "postDate": "2026-04-16T12:27:55.997Z", "content": "This writeup presents an advanced inference-time optimization strategy developed by the Potrhino team for the final evaluation of the AI Mathematical Olympiad (AIMO) Progress Prize 3, ranking 373rd out of 4,138. Recognizing that solving Olympiad-level mathematics under strict temporal and computational constraints nowadays is primarily an efficiency-bottlenecked task, we pivot from traditional model fine-tuning toward adaptive test-time orchestration. Leveraging the open-weight gpt-oss-120b model, our approach integrates:
\nThrough rigorous ablation studies, we demonstrate that the synergy of these techniques significantly elevates the model’s performance ceiling, achieving the following scores:
\nOur results establish that strategic resource management and deterministic tool augmentation can improve mathematical reasoning performance. To ensure full reproducibility, our code, environment configurations, and experimental logs are made publicly available.
\nIn recent years, Large Language Models (LLMs) have demonstrated remarkable proficiency across a broad spectrum of natural language tasks, evolving from fluent text generators into versatile problem solvers. Despite these advancements, complex mathematical reasoning remains a challenge, particularly at the Olympiad level. Proficiency in this domain is critical, as it serves as a cornerstone for solving intricate problems in fields ranging from advanced engineering to quantitative finance. While recent breakthroughs have seen Artificial Intelligence (AI) achieve human-level performance on International Mathematical Olympiad (IMO) problems, a notable performance gap persists between state-of-the-art closed-source models and their open-weight counterparts.
\nThe AI Mathematical Olympiad Progress Prize 3 (AIMO3) [1] aims to bridge this divide by challenging participants to develop open-source algorithms and models capable of solving high-level mathematical problems under strict computational and time constraints. While AIMO Progress Prize 2 was won in April 2025 by Nvidia's NemoSkills team [2], AIMO3 utilizes entirely original problems to eliminate the risk of data contamination. Every ground-truth label is an integer within the range [0, 99999]. Submissions, consisting of Kaggle notebooks, are evaluated on a private test set of 50 math problems, where scoring is based on the consistency of two independent predictions: for each problem, a score of 1.0 is awarded if both answers are correct, 0.5 if only one is correct, and 0 otherwise. The competition enforces a rigorous offline environment, limiting execution to 5 hours on GPU-accelerated notebooks (H100, P100, or T4×2) or 12 hours on CPU-based notebooks.
\nThis writeup details the strategy presented by the Potrhino team, which achieved a score of 43/50 in the public evaluation. The proposed approach leverages the gpt-oss-120b [7] model on H100 hardware, integrating:
\nThese methodologies are designed to optimize test-time inference under strict computational and time constraints. The following sections introduce the fundamental concepts, then describe the proposed strategy, and finally provide a comprehensive analysis of the results.
\nWhile LLMs have evolved into versatile problem solvers, purely text-based reasoning often struggles with tasks requiring precise numerical computation, long-horizon search, or rigorous verification. To address these inherent limitations, Tool-Integrated Reasoning (TIR) has emerged as an increasingly adopted paradigm. By equipping LLMs with external tools, such as Python code interpreters, TIR effectively expands the model's capabilities and transcends the \"capability ceiling\" of purely linguistic inference.
\nRecent research demonstrates that TIR-augmented models decisively outperform pure-text counterparts across challenging mathematical benchmarks. This performance gain is particularly evident in the pass@k metric, as the interpreter provides a deterministic execution environment that mitigates common calculation errors. Furthermore, this advantage extends beyond mere arithmetic. It also facilitates complex problem-solving strategies, such as brute-force search and algorithmic simulation, which would be intractably verbose or prone to failure in a standard autoregressive setup.
\nFigure 1 shows pass@k curves for the TIR (RL-trained) and pure-text models (Qwen3 8B) across benchmarks: (a) AIME24, (b) AIME25 [3]. This illustrates how TIR provides a significant gain in accuracy on AIME problems, which makes it an ideal candidate for AIMO3.
\n
While scaling the parameters of LLMs generally improves performance, scale alone is often insufficient for mastering complex arithmetic and symbolic reasoning. Chain-of-Thought (CoT) prompting addresses this limitation by eliciting a series of intermediate reasoning steps before arriving at a final conclusion. This approach offers several critical advantages for Olympiad-level mathematics:
\nCoT transforms a model from a direct predictor into a systematic reasoner, enabling it to bridge the gap between problem understanding and the final output. Figure 2 shows the solve rate of CoT and other prompting variations using LaMDA 137B and PaLM 540B on the GSM8K dataset [4]. This illustrates how CoT outperforms other prompting variations, highlighting its effectiveness in improving reasoning for math problems. However, CoT alone does not guarantee correctness, as errors can propagate across intermediate steps.
\n
A fundamental characteristic of complex problem-solving is the existence of multiple diverse reasoning paths that converge on the same correct solution. Self-Consistency leverages this intuition by replacing traditional greedy decoding with a sampling-based approach. The method involves generating a diverse set of reasoning paths and subsequently selecting the most consistent final answer by marginalizing out the sampled paths, a process commonly implemented as majority voting. This technique significantly enhances the robustness of CoT prompting. While an LLM may occasionally produce erroneous intermediate steps, incorrect reasoning paths are statistically less likely to converge on the same final value compared to valid logical sequences. Consequently, the correct answer tends to exhibit higher agreement across the sample population. Beyond raw accuracy gains, Self-Consistency provides two additional advantages:
\nA primary constraint of Self-Consistency is the increased test-time compute required. However, empirical evidence suggests that performance gains often saturate relatively quickly. Therefore, a strategically limited number of paths (e.g., k=5 or k=10) can capture the majority of the benefits without exceeding time limits. This trade-off is especially critical in constrained environments such as AIMO3, where inference budget is strictly limited. \nFigure 3 shows how Self-Consistency improves accuracy across various reasoning tasks, using LaMDA-137B [5].
\n
While parallel reasoning techniques like Self-Consistency enhance accuracy, they often encounter two primary bottlenecks: the linear scaling of inference overhead and diminishing returns caused by the equal weighting of all reasoning traces. Standard majority voting fails to account for quality variations, potentially allowing low-confidence or \"hallucinated\" traces to compromise the final output.\nTo address these limitations, the Deep Think with Confidence (DeepConf) methodology integrates confidence-aware filtering based on local token-level statistics. Unlike global measures that average confidence across an entire sequence, thereby masking localized reasoning failures, DeepConf monitors fluctuations at critical intermediate steps. This approach offers two distinct operational modes:
\nEmpirical results demonstrate that DeepConf not only improves accuracy by mitigating noise from low-quality traces but also optimizes resource allocation. In offline mode, DeepConf@512 achieves 99.9% accuracy on AIME 2025 using gpt-oss-120b (without tools), effectively saturating the benchmark compared to 97.0% for cons@512 (consensus/majority voting) and 91.8% for pass@1. In online mode, DeepConf reduces token generation by up to 84.7% compared to standard parallel sampling while maintaining or exceeding baseline accuracy. This efficiency is critical for maximizing the number of effective reasoning paths under strict computational and time constraints. This idea directly inspired our adaptive pruning and early termination strategy. Figure 4 shows the aforementioned results for both online and offline modes on AIME 2025 [6].
\n
gpt-oss-120b is an open-weight, reasoning-focused autoregressive model released under the Apache 2.0 license. Optimized for agentic workflows, it excels in instruction following, structured output generation, and tool integration (e.g., Python execution). The model architecture is a Mixture-of-Experts (MoE) Transformer, featuring 36 layers with a total of 116.8B parameters, of which 5.1B are active per token. The MoE weights are quantized to MXFP4 format (4.25 bits per parameter), comprising over 90% of the total parameter count. This enables operation within the constraints of a single 80GB GPU (such as the H100). The model was trained using the Harmony chat format, which utilizes specialized tokens for message boundaries and a role-based hierarchy to resolve instruction conflicts (System > Developer > User > Assistant > Tool). A distinctive feature of gpt-oss-120b is its support for three distinct reasoning levels (Low, Medium, and High), where higher levels correlate with increased CoT length and logical depth (as shown in Figure 5 [7]).
\n
The model is natively trained to utilize a suite of agentic tools:
\nBy interleaving CoT with TIR, gpt-oss-120b achieves high accuracy on mathematical benchmarks (as illustrated in Figure 6) while maintaining rapid inference speed and efficient memory utilization. We selected gpt-oss-120b due to its strong reasoning capabilities, native support for tool use, and compatibility with single-GPU deployment under AIMO constraints.
\n
The primary challenge of the AIMO3 competition lies in developing an AI system capable of solving IMO-level mathematical problems under stringent time and computational constraints. While modern LLMs have shown increasing proficiency in this domain, success hinges on identifying the optimal trade-off between reasoning accuracy and execution latency. \nOur approach addresses this challenge by leveraging the gpt-oss-120b model as a high-capacity foundation and concentrating development efforts on test-time optimization. The overall approach is structured around a set of strategic pillars, with hyperparameters calibrated to balance robustness and efficiency, while remaining amenable to further fine-grained optimization. These pillars include:
\nWe select the open-weight gpt-oss-120b model as the primary backbone. Given its strong baseline performance on mathematical benchmarks, further supervised fine-tuning (SFT) is deemed unnecessary. Specifically, the model achieves strong performance on AIME 2024 and AIME 2025 [7] when utilizing the \"High\" CoT reasoning level combined with TIR. Beyond raw accuracy, the model aligns well with the competition's hardware and temporal constraints. Utilizing the MXFP4 quantized format, gpt-oss-120b fits comfortably within the VRAM of a single H100 GPU, maintaining high inference throughput despite its 120B parameters. Furthermore, the average response length for AIME 2025 problems is reported to remain below 16k tokens [7], which supports an effective balance between depth of reasoning with execution speed. Consequently, we hypothesize that further SFT would yield diminishing returns relative to the substantial computational costs and the risk of increasing latency. This design choice also improves reproducibility, as it avoids reliance on additional proprietary training data or compute-intensive fine-tuning stages.
\nThe model's reasoning process is augmented by a Python tool, which can be invoked at any step of its reasoning chain. The model generates code blocks containing print() statements to retrieve intermediate results, which are then executed in a stateful Jupyter notebook environment.
To facilitate complex symbolic and numerical tasks, the execution environment is pre-configured with essential scientific libraries (e.g., math, numpy, sympy). To handle the rigorous demands of Olympiad-level problems, such as deeply nested recursion, operations involving very large integers or high-precision floating-point arithmetic, we increase the system’s default limits as follow:
recursion_limit = 2000;precision = 100000;int_max_digits = 1000000.To prevent long execution outputs from saturating the model’s context window, we implement a centralized truncation strategy. Outputs exceeding a predefined threshold (L_max = 1000 characters) are clipped: only the initial and final L_max/2 characters are preserved, separated by a ... TRUNCATED ... marker. This ensures the model receives both the beginning and final portions of the output while discarding redundant intermediate iterations. This encourages more concise tool usage and provides the model with the segments of the output that typically contain the most relevant information.
To ensure that inefficient or infinite loops do not deplete the problem's time budget, we implement an adaptive timeout strategy for code execution. This strategy balances the need for rapid feedback on simpler problems with the requirement for long-running computations on more complex ones:
\nThe execution timeout for each call is determined by the following formula:
\n$$Texec = \\min( \\min(Tinit \\cdot 1.3^n, 2 \\cdot Tinit), Tremaining )$$
\nWhere:
\nTinit is the starting execution timeout (set to 8s);n is the number of consecutive timeouts for the current problem;Tremaining is the residual time allocated to the specific problem.The selection of Tinit=8s and the growth factor of 1.3 is driven by an empirical balance between code complexity and session latency. A baseline of 8s is sufficient for the majority of standard mathematical scripts (e.g., symbolic manipulations or short loops). The exponential scaling factor of 1.3 is specifically calibrated to allow the execution limit to double after exactly three consecutive timeouts. Capping the growth at 2 * Tinit prevents unoptimized or runaway scripts from depleting the global time budget, which is a critical risk in time-constrained environments. Furthermore, we hypothesize that if a model fails to produce an efficient solution within three attempts (and nearly doubling the initial timeout), the bottleneck is likely structural rather than purely computational. Beyond this threshold, additional time yields diminishing returns, as the model is encouraged to shift toward an alternative reasoning strategy or a more efficient mathematical simulation rather than marginally optimizing failing code.\nThe interaction between the LLM and the execution environment is managed via a dedicated worker process; the detailed operational logic of this Jupyter worker is provided in Algorithm 1 (see Appendix).
The total execution time is partitioned across all 50 problems to ensure robustness against varying problem difficulties. After accounting for the initial setup overhead, including environment initialization and model loading, we implement a dynamic time-budgeting algorithm. The budget for a given problem, Tp, is determined by:
$$Tp = \\max( \\min(Tpool, Tavg \\cdot Cmax), Tavg \\cdot Cmin )$$
\nWhere:
\nTpool is the total remaining time after subtracting the reserved buffer for all future problems (Tavg * Cmin for each remaining problem);Tavg represents the baseline allocation, calculated as the initial total available time divided by the total number of problems;Cmax is a scaling constant for upper bound (2.1);Cmin is a scaling constant for lower bound (0.86).This mechanism ensures that a single outlier (an extremely difficult problem) cannot deplete the resources required for the rest of the test set. Furthermore, it allows simpler problems to conclude early and return their unused budget to the global pool, thereby granting additional computational time to high-complexity cases without risking a global submission timeout.\nThe coefficients Cmin and Cmax were empirically selected to balance risk and performance. A Cmin of 0.86 ensures that even in the worst-case scenario, every problem is guaranteed approximately 85% of the baseline average time. Conversely, a Cmax of 2.1 allows the solver to spend more than double the average time on difficult problems without destabilizing the overall schedule. This upper bound was also found to be sufficient for the model to reach the maximum effective depth of its context window. This dynamic approach ensures that the \"time-to-solution\" is maximized for challenging problems while maintaining a zero-failure rate regarding the global 5-hour deadline. The detailed operational logic is provided in Algorithm 2 (see Appendix).
The prompts used for solving mathematical problems are intentionally minimal and role-based. Since gpt-oss-120b is natively optimized for multi-level CoT reasoning, overly verbose or complex prompts can introduce unwanted bias or lead the model toward suboptimal reasoning paths. Instead, we leveraged the model's role-based hierarchy to enforce constraints while preserving its inherent problem-solving flexibility. The strategy is organized across following prompts:
\nprint() for output, preventing empty execution results.SYSTEM: \"You are a rigorous world-class math competitor. You must return the final answer in \\boxed{}. The final answer must be a single integer (digits only) between 0 and 99999 inclusive.\"
\nDEVELOPER: \"Reason carefully to solve the problem and use the Python tool when necessary. You have access to math, numpy, sympy, mpmath, decimal, itertools, and pulp. Never guess.\"
\nTOOL DESC: \"Use this tool to execute Python code. The environment is a stateful Jupyter notebook. You must use print() to output results. Internet access is off.\"
\nUSER: \"{problem}\"
\nTo convert the model's natural language output into the required competition format, the system employs a multi-stage extraction pipeline. It first attempts to extract the answer within standard LaTeX delimiters. If no match is found, it falls back to a secondary search for internal tags typically generated by the model's final reasoning tokens. The extraction utilizes the following regular expressions:
\noxed\\s*\\{[^\\d-]*(-?[0-9,]+\\.?[0-9]*)[^\\d]*\\}assistantfinal\\s*[^\\n\\d-]*(-?[0-9,]+\\.?[0-9]*)[^\\d]*In cases where multiple matches occur within a single reasoning path, the system processes them in reverse chronological order (from the most recent to the first). This heuristic assumes that the model's final conclusion, often following a self-correction or a summary step, is more likely to be accurate than initial estimates. For each extracted candidate, a normalization function is applied to ensure compliance with the competition's constraints:
\n100,000 operation is applied. This ensures the output falls within the required [0, 99999] range, effectively handling cases where the model calculates a correct large value but omits the competition-specific modulus in its final response.A candidate is considered a valid final answer only if it results in an integer within the specified range. If a reasoning path concludes without a tool call and fails to yield a valid answer, a self-correction loop is triggered to recover a valid formatted answer. A follow-up prompt is appended to the conversation to guide the model:
\nUSER (Recovery): \"The previous response did not produce a valid final answer. Continue solving the problem. Remember that the final answer must be a single integer (digits only) between 0 and 99999 inclusive, and must be placed in \\boxed{}.\"
\nTo prevent infinite loops and manage the time budget, this recovery attempt is limited to a single iteration; if a second failure occurs, the trace is terminated.
\nWhereas our configuration supports a maximum concurrency of 7.52x, we initialize 7 parallel reasoning paths per problem to maintain a stable and efficient Self-Consistency pipeline. To enhance accuracy and minimize computational redundancy, we implement an advanced voting mechanism that integrates confidence/entropy-based tie-breaking, consensus-driven early termination, and a fallback to tool outputs.
\nStarting from the second completed reasoning path, the system applies a majority vote each time a new answer is generated. In the event of a tie, where multiple candidates share the same maximum frequency, we apply a scoring metric to select the most reliable candidate (i.e., the one with the highest score). The score for a candidate answer a, denoted as S, is calculated as:
$$S_{tie, a} = \\sum_{i \\in \\text{Paths}(a)} S_{a, i}$$
\nWhere S represents the individual score of reasoning path i that yielded answer a:
$$S_{a, i} = p_i \\cdot (1 - H_{g, i}) \\cdot C_{g, i}$$
\nWhere:
\np is a coefficient set to 0.75 if path i required Format Recovery (indicating a failure in the initial formatting), and 1.0 otherwise. As reported in the Main results section, paths requiring recovery are statistically less reliable.H is the mean entropy of the entire reasoning path i, considering the top 20 tokens for each generation step (inspired by [6]):$$H_{g, i} = \\frac{\\sum_{t=1}^{L} \\sum_{j=1}^{20} - \\exp(lp_{t,j}) \\log_2(\\exp(lp_{t,j}))}{L \\cdot \\log_2(20)}$$
\nC is the mean confidence of the entire reasoning path i (inspired by [6]):$$C_{g, i} = \\exp\\left(\\frac{1}{L} \\sum_{t=1}^{L} lp_t\\right)$$
\nlp is the log-probability of the j-th top candidate token at generation step t;lp is the log-probability of the token selected at generation step t;L is the length of reasoning path, in tokens.To optimize the inference budget, the system evaluates the consensus of the leading answer after each vote. If the agreement among reasoning paths is sufficiently robust, the process terminates early. We define four hierarchical levels of consensus based on answer frequency, the number of active tasks, and the remaining time budget.
\nmax_vote > max(2, active_tasks/2) or max_vote > still_running_tasks + second_high_frequency max_vote >= max(2, active_tasks // 2) and max_vote - 1 > second_high_frequencymax_vote >= max(2, active_tasks // 2) and max_vote > second_high_frequency and elapsed_time > available_time * 0.5max_vote >= max(2, active_tasks // 2 - 1) and max_vote - 1 > second_high_frequency and elapsed_time > available_time * 0.75Where:
\nmax_vote is the frequency of the current best answer (max frequency among answers);active_tasks includes both running tasks and completed tasks that have already produced a valid answer (i.e., excluding only those that terminated without producing a valid answer);still_running_tasks is the number of still running tasks;second_high_frequency is the frequency of the second best answer;available_time is the available time for the current problem;elapsed_time is the elapsed time from the start of the inference for the current problem.In all cases, at least 2 valid votes are required before triggering any consensus condition, to prevent early termination on potentially noisy single-path results.
\nIf all reasoning paths conclude without a valid formatted answer (often due to model timeouts during the final CoT steps), the system performs a vote on the last valid numerical output returned by the Python interpreter within each reasoning path. This fallback mechanism significantly improves robustness; empirical analysis shows that the model frequently computes the correct value via TIR (as shown in the Main results section) but may fail to encapsulate it in the required \\boxed{} format if the execution limit is reached.
\nTo optimize computational efficiency for problems requiring extended reasoning chains, we implement an early pruning strategy designed to terminate less promising reasoning paths, inspired by the DeepConf framework [6]. When the elapsed time for a problem reaches a predefined fraction of its allocated budget, and the number of active paths still exceeds a given threshold, the system evaluates all paths and prunes the lowest-scoring ones. This mechanism reallocates GPU resources to the remaining high-scoring paths, allowing them to reach greater logical depth within the residual time. In our implementation, pruning is triggered when the elapsed time exceeds 60% of the allocated budget and more than 75% of the initial reasoning paths remain active.\nEach reasoning path i is evaluated using a composite score S that integrates global and local token-level statistics. The pruning score is defined as:
$$S_i = p_i \\cdot (1 - H_{g, i}) \\cdot (1 - H_{tail, i}) \\cdot C_{g, i} \\cdot C_{tail, i}$$
\nWhere p, H and C are the same as defined in Section 3.5.1, while H and C are analogous to H and C, but are computed using only the last 2048 tokens of the reasoning path (inspired by [6]).
This scoring function effectively penalizes paths exhibiting high uncertainty (entropy) or low probability (confidence). While global metrics (H, C) provide an overview of the path's stability, the inclusion of tail metrics (H, C) is critical for detecting real-time \"hallucinations\" or logical instabilities (e.g., repetitive loops) that often emerge late in the generation process. By monitoring these fluctuations, the system can terminate decaying reasoning chains and focus the remaining temporal budget on the most robust candidates, maximizing the effectiveness of the Self-Consistency pool under strict time constraints. These combined metrics improve the accuracy of identifying and pruning reasoning paths that are likely to produce incorrect answers (as shown in the Main Results section).
The proposed methodology was implemented and evaluated within the Kaggle competition environment (Python notebook, snapshot of 2025-11-07) , leveraging an NVIDIA H100 GPU accelerator. The software stack relies on the following dependencies:
\nThe following sections describe the experimental setup in detail, including the local validation strategy, default hyperparameter configurations, and an in-depth analysis of performance across multiple experimental runs, with a particular focus on ablation studies, overall performance and comparison with state of the art.
\nThe gpt-oss-120b model is loaded into GPU memory at the beginning of execution and persists throughout the entire inference process. To accelerate the initialization phase, model artifacts are first copied into RAM using multiprocessing, reducing load time from approximately 900s to ~250s. During this stage, all environment variables required for vLLM optimization and logging are configured.\nTo ensure strict compliance with competition time constraints and to prevent resource contention, the system’s core logic is decoupled into isolated processes using the multiprocessing library. These processes communicate via standardized queues, enabling robust and fault-tolerant execution.\nThe overall architecture is organized as follows:
\nDetailed pseudocode for system orchestration and inference workflows is provided in Algorithm 3 and Algorithm 4 in the Appendix.
\nFor local testing, validation, and hyperparameter optimization, we employ a high-quality mathematical reasoning dataset comprising 75 problems. This benchmark was specifically curated for the AIMO3 competition, with a focus on evaluating reasoning reliability through executable Python-based TIR (see imo-benchmark.csv in [15]). In alignment with the competition requirements, each problem is associated with a ground-truth integer answer within the range [0, 99999].\nThe primary motivation for selecting this dataset is its high level of difficulty. This complexity is critical for accurately measuring performance gains during the optimization process. In contrast to simpler benchmarks, where many problems can be solved through shallow reasoning, this dataset provides a strong discriminative signal, enabling a clear assessment of the effectiveness of advanced reasoning strategies and test-time optimization techniques.
Several parameters were inspired by high-performing public notebooks [12][13][14], which achieved strong results on both reference problems and the public leaderboard. To evaluate the individual contribution of each optimization strategy, we perform a series of controlled experimental runs. Each run isolates a specific component by disabling it while keeping all other parameters constant, following a standard ablation study methodology.\nThe primary hyperparameters used across all runs are summarized in Table 1 in the Appendix. While the validation set consists of 75 problems (compared to 50 in the public/private leaderboard), the configuration remains identical to our final competition submission, with the total time budget scaled proportionally (7.5 hours for validation vs. 5 hours for submission).
\nAdditional vLLM optimizations were applied to reduce latency and memory overhead, including chunked prefill, prefix caching, asynchronous scheduling, and reduced CUDA graph capture size and sequence limits. Standard values were retained for top_p and top_k. All runs in Section 4.4 use this configuration unless otherwise specified. Additional implementation details are available in the submission notebook ([16] refers to the submission notebook, [17] refers to required dependencies, [21] refers to Google Colab version of submission notebook).
\nTo rigorously evaluate the impact of each architectural component, we conducted a series of experimental runs, described in Table 2. Each run is based on the \"final_submission\" configuration, with the deactivation of a specific feature to isolate its effect on performance, except for the \"baseline\" run, where all new optimizations are disabled, and the \"time_constraints_disabled\", where only time constraints are removed.
\n| Run Name | \nDescription | \n
|---|---|
| time_constraints_disabled | \nRemoves time constraints to the final submission to assess the model's unconstrained capability ceiling | \n
| final_submission | \nConfiguration used for the final submission, including all optimizations | \n
| early_break_disabled | \nDisables dynamic pruning of low-scoring reasoning paths | \n
| early_termination_disabled | \nDisables the consensus-driven early termination mechanism | \n
| medium_reasoning | \nReduces the reasoning level of gpt-oss-120b from HIGH to MEDIUM | \n
| dynamic_time_management_disabled | \nUses static time allocation (average time per problem instead of dynamic time management) | \n
| python_tool_disabled | \nUses pure Chain-of-Thought (CoT) reasoning without Python tool integration | \n
| baseline | \nBaseline before new optimizations | \n
The baseline run represents the most advanced configuration achievable using standard practices within the competition's constraints, characterized by:
\nk=7 parallel sampling approach with majority voting;It is important to note that while gpt-oss-120b natively supports Tool-Integrated Reasoning (TIR), the official implementation (requiring gpt-oss>=0.0.5 and Python 3.12) proved incompatible with the Kaggle H100 environment (Python 3.11.13). Consequently, the baseline represents the model's unassisted reasoning limit, while our work introduces a custom-built, compatible TIR orchestration layer.
\nEach run generates a detailed telemetry log capturing key metrics for every reasoning path (runs are available in [18], while generated logs are available in [19]). These logs are aggregated and analyzed to compute per-run performance statistics as well as global reasoning trends, which are presented in the following charts and tables (statistics and charts logic are available in [20]).
\nThe following sections present a detailed analysis of the performance metrics obtained from the experimental runs described above. Results are evaluated across multiple dimensions, including ablation studies designed to isolate the contribution of each component and overall accuracy compared against the state-of-the-art baseline.\nIn particular, we analyze how individual test-time optimization strategies affect final accuracy. These results provide a comprehensive view of the system’s behavior, demonstrating how each component contributes to solving complex mathematical problems under strict time and resource constraints.
\nThis section mainly evaluates the individual contribution of each optimization strategy to the overall system performance. By systematically deactivating specific components, we isolate their impact on accuracy, providing empirical justification for the architectural choices described in Section 3.
\n\nThe chart illustrates the distribution of incomplete, incorrect, and correct reasoning paths across the different experimental configurations. This global metric aggregates all individual traces generated for every problem, regardless of the final consensus answer.\nThe data demonstrates that TIR and dynamic time management significantly improve the yield of correct reasoning paths. Conversely, the HIGH reasoning level is primarily responsible for reducing the incidence of logical fallacies and incorrect outputs. The early break logic provides a marginal gain in the number of correct paths by reallocating resources to more promising chains.\nAs expected, removing time constraints altogether results in the highest proportion of correct paths, establishing the model's theoretical \"capability ceiling.\" Interestingly, disabling consensus-based early termination appears to slightly increase the count of correct paths; however, this is likely an artifact of redundant computations on simpler problems that would have otherwise terminated early, rather than a genuine improvement in solving more complex tasks.
\nThe chart compares three distinct accuracy metrics across each experimental run:
The results demonstrate that the conf_cons@7 metric consistently outperforms the standard cons@7, although a gap remains compared to the pass@7 theoretical maximum. Consistent with previous observations, Python tool integration provides the most significant accuracy gain. However, each supplementary technique contributes to a cumulative improvement in the final score, most notably the dynamic time management and HIGH reasoning level, followed by the incremental gains from early break and early termination logic. Finally, removing time constraints establishes a higher accuracy ceiling, confirming that the model's performance is currently limited more by the inference budget than by its intrinsic reasoning capabilities.
\nRegarding the gain provided by the Python tool integration, while the used validation set employed is tool-oriented, the benefits of TIR will partially transfer to more \"abstract\" domains, albeit with a shift in how the tool is utilized. For instance:
\nAs for the performance gains driven by dynamic time management, there is room for further optimization. As reported in Section 3.3., it is based on the scaling constants Cmin and Cmax. Initially, the parameter Cmax=2.1 was empirically selected to allow the model to reach its maximum effective context window depth on difficult problems. The following table summarizes additional configurations evaluated during the final phases of the competition:
| Cmin | \nCmax | \nPublic LB Scores | \nMean Score | \n
|---|---|---|---|
| 0.88 | \n2.2 | \n37 | \n37.0 | \n
| 0.86 | \n2.1 | \n39, 37, 38, 43 | \n39.25 (Final Submission) | \n
| 0.85 | \n2.0 | \n36 | \n36.0 | \n
| 0.80 | \n2.0 | \n39 | \n39.0 | \n
| 0.80 | \n2.1 | \n42, 41 | \n41.5 | \n
While the selected final submission (Cmin=0.86, Cmax=2.1) achieved the highest peak score of 43/50, subsequent analysis suggests that a lower floor (Cmin=0.80) paired with Cmax=2.1 yields a more stable mean performance (41.5).\nThe configuration (0.80, 2.1) was identified very close to the competition deadline, consequently, we prioritized the (0.86, 2.1) configuration for the final submission as it had already demonstrated high-ceiling potential in earlier runs.\nThese results indicate that performance is sensitive to the Cmin floor. A lower Cmin grants the system more flexibility to reallocate time from \"easy\" problems to \"hard\" ones, whereas an excessively high Cmax may lead to diminishing returns if the model exceeds its useful reasoning depth.
Mathematical reasoning tasks are sensitive to problem ordering and stochastic sampling, and variance estimates are crucial to distinguish architectural gains from statistical noise. The experimental runs were conducted near the competition deadline to provide a reproducible and objective summary of each technique’s behavior on a high-difficulty public dataset. We acknowledge that these specific local runs reflect a single execution. Due to the significant computational intensity—each 75-problem run requiring approximately 7.5 hours on an NVIDIA H100 GPU—and strict hardware availability constraints, conducting multiple full-scale repetitions was not feasible within the remaining timeframe. \nHowever, the significance of the observed performance gaps is supported by the following factors:
\nk=7) reduces the impact of stochastic sampling.| Notebook Configuration | \nPublic LB Scores | \nMean Score | \n
|---|---|---|
| Final Submission | \n39, 37, 38, 43 | \n39.25 | \n
| Early Break Disabled | \n36, 38 | \n37 | \n
\nThis chart evaluates the capability of different token-level metrics to correctly identify incorrect reasoning paths within \"ambiguous\" problems (defined as those where the model generated both correct and incorrect answers). By selecting the path with the worst value for a given metric, we measure how frequently that metric successfully points to a wrong answer.\nThe results indicate that tail entropy is the most reliable indicator of a failing reasoning chain, achieving the highest precision in identifying incorrect paths. This is followed by global mean entropy, tail confidence, and mean confidence. These findings empirically justify the composite \"early_break_score\" used in our pruning logic (Section 3.6). By using this metric, the system can effectively detect late-stage hallucinations or logical instabilities, allowing for the strategic termination of suboptimal paths and the reallocation of computational budget to more promising candidates.
\nThis chart illustrates the effectiveness of various metrics when used to resolve ties in \"ambiguous\" problems (those containing both correct and incorrect paths where no clear majority exists). The success rate is measured by selecting the answer with the highest aggregate score, where the score is the sum of the metric values across all paths yielding that specific answer.\nWhile the selected final submission utilized a tie_break_score based on global mean confidence and global mean entropy (Section 3.5.1), subsequent analyses reveal that alternative metrics may offer superior discriminative power. In particular, incorporating local confidence bottlenecks and reasoning-path length can produce substantially stronger tie-breaking performance.
| Metric | \nFormula | \n
|---|---|
| Revised tie_break_score_A | \n$$(1 - \\text{length_perc}) \\cdot \\text{min_conf}$$ | \n
| Revised tie_break_score_B | \n$$\\text{tie_break_score} \\cdot (1 - \\text{length_perc}) \\cdot \\text{min_conf}$$ | \n
| Revised tie_break_score_C | \n$$\\text{tie_break_score} \\cdot (1 - \\text{length_perc}) \\cdot \\text{min_conf} \\cdot \\text{tail_conf}$$ | \n
Here, length_perc denotes the ratio between the generated response length and the model context limit (response_length / max_model_len).\nThese results suggest that localized confidence failures are more informative than global averages. Metrics such as mean_conf and mean_entropy can remain relatively stable even when a reasoning trajectory contains a critical logical error. In contrast, min_conf and tail_conf act as bottleneck indicators: a single low-confidence transition often coincides with a hallucinated deduction or unstable reasoning step and can act as a \"pivot point\" leading to reasoning divergence.\nThe inverse correlation between reasoning-path length and correctness is also notable. For Olympiad-style mathematical problems, successful solutions tend to emerge from relatively concise reasoning chains, whereas excessively long trajectories are more likely to exhibit reasoning drift, repetition, or self-inconsistent exploration.\nThese results suggest that combining global stability metrics with local confidence statistics and trajectory-length regularization may provide a more robust tie-breaking strategy for adaptive reasoning systems.
\nThe chart illustrates the distribution of correct and incorrect answers for reasoning paths that required the Format Recovery mechanism during inference. The data reveals a significant correlation between formatting failures and logical errors: reasoning paths necessitating recovery are nearly twice as likely to produce an incorrect answer compared to those that adhere to the requested format natively. This empirical evidence justifies our decision to incorporate a penalty coefficient for recovered paths within both the tie-breaking and early-break scoring functions (Sections 3.5.1 and 3.6). By discounting these structurally weaker traces, the system increases the probability of selecting the most robust and consistent reasoning chains as the final output.
\nThis chart evaluates the correspondence between the final valid integer output from the Python interpreter and the ground-truth answer across all reasoning path of experimental runs. The data demonstrates that utilizing the last valid tool output as a fallback is a highly effective strategy for problems that reach the execution timeout before a formal conclusion can be generated.\nIn many high-complexity cases, the model successfully computes the correct value via a tool call in the final stages of reasoning, but lacks the remaining temporal budget to synthesize the final Chain-of-Thought (CoT) and the required LaTeX formatting. \nOn the 75-problem validation set, excluding the python_tool_disabled and baseline ablations (which do not use the Python tool), the tool-output fallback mechanism was triggered 45 times in total. A fallback event occurs when all 7 reasoning paths fail to produce a correctly formatted final answer, but at least one reasoning path generated a valid numerical result during a Python tool execution. Among these 45 fallback activations, the mechanism recovered the correct answer 3 times, corresponding to a conditional accuracy of approximately 6.67% once the fallback fires.\nAlthough the conditional success rate is relatively low, the fallback was intentionally designed as a last-resort recovery mechanism rather than a primary selection strategy. In these cases, the standard extraction and aggregation pipeline had already failed completely, meaning that without the fallback the system would deterministically output an incorrect answer. Consequently, even a small number of successful recoveries translates directly into measurable score improvements. Importantly, the fallback introduces essentially no additional inference-time cost, since it only reuses numerical outputs already produced during earlier Python tool executions rather than generating new reasoning trajectories. This empirically justifies our implementation of a consensus-based vote on tool outputs (Section 3.5.3) when no reasoning path reaches a natural termination within the allotted time.
\nThe chart illustrates the total length (in tokens) of all reasoning paths generated across each experimental run. The following observations can be made:
The integration of the strategies discussed in the previous sections culminated in a robust mathematical solver. This framework effectively raised the model's capability ceiling by optimizing the inference process. To quantify this gain, we compared our final submission against a standard state-of-the-art (SOTA) baseline. This baseline represents the most advanced configuration achievable using standard practices within the competition's constraints, as described in Section 4.4.
\nIn local validation (on selected hard problems), our final submission achieved a score of 41/75, which increased to 47/75 when time constraints were removed. In comparison, the initial baseline score was 17/75.
\nOn the AIMO3 Public Leaderboard, our selected final submission achieved a maximum score of 43/50 and a mean score of 39.25/50 (based on four submissions with range 37-43).
\nOn the AIMO3 Private Leaderboard, our selected final submission achieved a score of 42.0/50, ranking 373rd out of 4,138.
\nThese results could be further improved by increasing the number of parallel paths per problem and expanding the context window on more powerful hardware.
\nIn this work, we presented a novel strategy designed for the AIMO3 competition. By integrating a custom-built, fault-tolerant Tool-Integrated Reasoning (TIR) layer with adaptive test-time optimizations, including dynamic time management, lightweight role-based CoT prompting with robust format recovery, Self-Consistency with consensus-based early termination and confidence-/entropy-based tie-breaking, adaptive pruning of less promising reasoning paths, we successfully extended the mathematical reasoning capabilities of the gpt-oss-120b model.\nOur experimental results demonstrate that the bottleneck in automated mathematical problem-solving often lies in the efficiency of the inference process, rather than just the model's intrinsic knowledge. The transition from a standard baseline to our optimized strategy yielded a significant performance gain, confirming that a deterministic execution environment paired with strategic test-time optimizations can achieve state-of-the-art results.\nMoving forward, we identify several promising avenues for improvement:
\n[1] Simon Frieder, Sam Bealing, Philip Vonderlind, Sida Li, Arsenii Nikolaiev, Geoff C. Smith, Kevin Buzzard, Timothy Gowers, Peter J. Liu, Po-Shen Loh, Lester Mackey, Leonardo de Moura, Dan Roberts, D. Sculley, Terence Tao, David Balduzzi, Simon Coyle, Alex Gerko, Ryan Holbrook, Addison Howard, and XTX Markets. AI Mathematical Olympiad - Progress Prize 3. https://kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3, 2025. Kaggle.
\n[2] Ivan Moshkov, Darragh Hanley, Ivan Sorokin, Shubham Toshniwal, Christof Henkel, Benedikt Schifferer, Wei Du, Igor Gitman. AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset, 2025. URL https://arxiv.org/pdf/2504.16891.
\n[3] Heng Lin and Zhongwen Xu. Understanding Tool-Integrated Reasoning, 2025. URL https://arxiv.org/pdf/2508.19201.
\n[4] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, Denny Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, 2023. URL https://arxiv.org/pdf/2201.11903.
\n[5] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2023. URL https://arxiv.org/pdf/2203.11171.
\n[6] Yichao Fu, Xuewei Wang, Yuandong Tian, Jiawei Zhao. Deep think with confidence, 2025. URL https://arxiv.org/pdf/2508.15260.
\n[7] OpenAI. gpt-oss-120b & gpt-oss-20b Model Card, 2025. URL https://arxiv.org/pdf/2508.10925, https://huggingface.co/openai/gpt-oss-120b, https://www.kaggle.com/models/danielhanchen/gpt-oss-120b/Transformers/default/1.
\n[8] openai-harmony 0.0.8. URL https://pypi.org/project/openai-harmony/0.0.8/.
\n[9] vllm 0.11.2. URL https://pypi.org/project/vllm/0.11.2/, https://docs.vllm.ai/projects/recipes/en/latest/OpenAI/GPT-OSS.html#troubleshooting, https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken, https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken.
\n[10] numpy 2.2.0. URL https://pypi.org/project/numpy/2.2.0/.
\n[11] PuLP 3.3.0. https://pypi.org/project/PuLP/3.3.0/
\n[12] ShelterW. [Way to 30+/50] GPT-OSS - w/ python tool. URL https://www.kaggle.com/code/shelterw/way-to-30-50-gpt-oss-w-python-tool?scriptVersionId=285192299
\n[13] Andreas Bisiadis. AIMO 3 | GPT-OSS-120B (with tools). URL https://www.kaggle.com/code/andreasbis/aimo-3-gpt-oss-120b-with-tools?scriptVersionId=290379281
\n[14] parthenos. [44/50] LET ME (over)COOK!!!. URL https://www.kaggle.com/code/nihilisticneuralnet/44-50-skill-optional-luck-zaroori?scriptVersionId=291499043
\n[15] wenliangtlh. AIMO3-High-Difficulty-Tool-Calling-Dataset. URL https://www.kaggle.com/datasets/wenliangtlh/aimo3-high-difficulty-tool-calling-dataset
\n[16] Potrhino. AIMO3-v3, version 11 (final submission notebook). URL https://www.kaggle.com/code/potrhino/aimo3-v3?scriptVersionId=302225896
\n[17] Potrhino. AIMO3-dependencies-v1, version 10 (dependencies of final submission notebook). URL https://www.kaggle.com/code/potrhino/aimo3-dependencies-v1?scriptVersionId=297334166
\n[18] Potrhino. AIMO3-v3-v11-runs (validation runs notebook for main results). URL https://www.kaggle.com/code/potrhino/aimo3-v3-v11-runs?scriptVersionId=313133192
\n[19] Potrhino. AIMO3-local-logs (validation results dataset). URL https://www.kaggle.com/datasets/potrhino/aimo3-local-logs
\n[20] Potrhino. AIMO3-v3-v11-results (validation stats & charts notebook). URL https://www.kaggle.com/code/potrhino/aimo3-v3-v11-results?scriptVersionId=313200617
\n[21] Potrhino. potrhino-aimo3-v3.ipynb (Google Colab version of final submission notebook). https://colab.research.google.com/drive/1vXAcdEiVcTM7oeQTFcsZsfQ6-ms33TK5?usp=sharing
\n| Param | \nValue | \nNote | \n
|---|---|---|
| device | \ncuda | \nEnables NVIDIA H100 GPU acceleration required to run gpt-oss-120b | \n
| dtype | \nauto | \nUses default precision (bfloat16) | \n
| kv-cache-dtype | \nfp8_e4m3 | \nReduces memory footprint vs. bf16, enabling higher parallelism with negligible precision loss | \n
| gpu_memory_utilization | \n0.97 | \nMaximizes GPU memory usage during inference | \n
| max_model_len | \n64*1024 | \nBalances reasoning depth and VRAM constraints | \n
| enable_tool | \nTrue | \nEnables Python tool usage during reasoning | \n
| reasoning | \nHIGH | \nActivates the highest CoT reasoning level | \n
| temperature | \n1.0 | \nBalances stochasticity for diverse Self-Consistency sampling | \n
| min_p | \n0.02 | \nFilters low-probability tokens while preserving diversity | \n
| reasoning_paths | \n7 | \nNumber of parallel reasoning paths for majority voting | \n
| max_time | \nint(560601.5 - 3.560) | \nTotal available time budget | \n
| problems_number | \n50+25 | \nTotal number of problems in validation | \n
| enable_dynamic_time_management | \nTrue | \nEnables adaptive per-problem time allocation | \n
| min_time_coef | \n0.86 | \nLower bound for time allocation scaling | \n
| max_time_coef | \n2.1 | \nUpper bound for time allocation scaling | \n
| top_logprobs | \n20 | \nUsed to compute entropy based on top-20 token probabilities [6] | \n
| group_logprobs_len | \n2048 | \nWindow size for tail entropy/confidence estimation [6] | \n
| enable_early_break | \nTrue | \nEnables pruning of low-scoring reasoning paths | \n
| execution_timeout | \n8 | \nInitial execution timeout (seconds) for Python tool calls | \n
| max_tool_step | \n200 | \nMaximum number of tool calls per reasoning path | \n
| max_tool_output_len | \n1000 | \nMaximum tool output length before truncation | \n
| max_tool_error | \n5 | \nMaximum number of tool errors before environment reset | \n
| precision | \n100000 | \nNumerical precision for Python tool environment | \n
| int_max_digits | \n1000000 | \nMaximum integer size supported for Python tool environment | \n
| recursion_limit | \n2000 | \nMaximum recursion depth for Python tool environment | \n
| enable_early_termination | \nTrue | \nEnables consensus-based early termination | \n
Initialize Jupyter kernel session\n Configure Environment:\n - Pre-import libraries (math, numpy, sympy, etc.)\n - Set high float precision and integer max_digits\n - Increase recursion depth limit\n\n While True:\n Receive {Python_code, available_time} from Main Process\n\n If Python_code is None:\n Shutdown Kernel and Exit\n\n If Python_code is \"CLEAN\":\n Restart Jupyter Kernel and re-initialize environment\n Return empty confirmation\n\n Else:\n # Determine execution limit using Adaptive Timeout logic\n T_limit = Calculate_Timeout(available_time, consecutive_timeouts)\n\n # Execute code in the stateful session\n Output = Execute_in_Jupyter(Python_code, timeout=T_limit)\n\n # Post-process Output (Output Length Control)\n If length(Output) > THRESHOLD:\n Output = Truncate_Center(Output, max_length=THRESHOLD)\n\n Return Output\n #Init constants\n START_TIME = time.now() \n ...initialization logic...\n INIT_OVERHEAD = time.now() - START_TIME\n TOTAL_CLOCK = int(5*60*60 - 3.5*60) #5 hours with a safety margin of 3.5 minutes (in seconds)\n DEADLINE = START_TIME + TOTAL_CLOCK\n TOTAL_PROBLEMS = 50\n C_min = 0.86 \n C_max = 2.1 \n T_avg = (TOTAL_CLOCK - INIT_OVERHEAD) / TOTAL_PROBLEMS \n\n #Main loop\n processed_problems = 0\n for each incoming problem P:\n # Reserve time for all future problems\n remaining_problems = TOTAL_PROBLEMS - processed_problems - 1\n reserved_buffer = remaining_problems * (T_avg * C_min)\n # Calculate current budget\n T_pool = DEADLINE - time.now() - reserved_buffer\n # Apply elastic constraints\n available_time = max( min(T_pool, T_avg * C_max), T_avg * C_min )\n # Final problem consumes all remaining time\n if remaining_problems == 0:\n available_time = DEADLINE - time.now()\n #Solve the problem within available_time\n solve(P, time_limit=available_time)\n processed_problems += 1\n Init global configurations\n Spawn Process \"PM\": Load LLM into memory\n Spawn Process Pool \"PTP\": Stateful Python Tool environments (one per parallel path)\n\n For each incoming problem \"P\":\n If \"P\" is None: \n Shutdown \"PTP\" pool and \"PM\" process\n Break\n\n Compute available time \"T\" # Via Dynamic Time Management logic\n Init \"AG\": List for valid reasoning path answers\n Init \"ATG\": List for fallback tool-only answers\n Init Stop Event \"S\": For early termination\n Init Global Metrics \"GM\": For cross-thread pruning/scoring\n\n Start Thread Pool \"TP\":\n Execute parallel inference on \"P\" using \"PM\" (LLM) and \"PTP\" (Tools)\n\n Init \"best_answer\" to 0\n Init \"max_vote\" to 0\n For each completed thread \"j\" in \"TP\":\n Get response metrics \"A_j\"\n If \"A_j\" contains a valid extracted answer:\n Add \"A_j\" to \"AG\"\n # Update Best Answer and relative vote using Self-Consistency + Tie-Break\n \"best_answer\", \"max_vote\" = SmartVote(\"AG\")\n\n If Consensus-Based Early Termination criteria met:\n Signal Stop Event \"S\" to halt remaining threads\n Break\n Else:\n # Fallback: check if the last Python tool output is a valid number\n If \"AT_j\" contains valid tool result:\n Add \"AT_j\" to \"ATG\"\n\n # Final Fallback: If no reasoning path finished, vote on Tool outputs\n If \"max_vote\" == 0 and \"ATG\" is not empty:\n \"best_answer\", \"max_vote\" = SmartVote(\"ATG\")\n\n Shutdown Thread Pool \"TP\" (cleanup)\n Return \"best_answer\"\n Init conversation \"C\" with role-based prompts and tool definitions\n Init \"available_steps\" = 200\n Init \"FormatRecoveryEnabled\" = True\n Init \"response_text\", \"logprobs\", \"top_logprobs\" as empty\n\n While \"available_steps\" > 0:\n If context_window_exhausted: # Safety margin of 4096 to allow final answer/tool output\n Break \n\n Open Response Stream \"RS\" from LLM\n For each chunk in \"RS\":\n If Stop_Event \"S\" is set or global_time_limit reached:\n Break\n\n Update \"response_text\", \"logprobs\", and \"top_logprobs\"\n\n # Adaptive Pruning Check (every 2048 tokens)\n If Pruning_Enabled and tokens_since_last_check > 2048:\n Compute reasoning metrics (Entropy/Confidence)\n Update Global Metrics \"GM[i]\"\n If pruning_criteria_met: # Based on elapsed time, active tasks and relative score\n Break\n\n If Stop_Event \"S\" is set: Break\n\n Extract Numerical Answer \"A_j\"\n If \"A_j\" is valid: \n Break # Success: Answer found\n\n if global_time_limit reached or pruning_criteria_met:\n break\n\n Update conversation \"C\" with current reasoning step (reasoning, tool call)\n\n If current response contains a Tool Call: \n Extract generated Python code\n Execute code in Jupyter session \"PTP[i]\" -> Get result \"RES\"\n Update conversation \"C\" with Tool Output\n If \"RES\" is a valid integer answer:\n Save as Fallback Tool Answer \"AT_j\"\n Else: # No tool call and no valid answer found\n If \"FormatRecoveryEnabled\":\n Update \"C\" with a new User Prompt for Format recovery\n \"FormatRecoveryEnabled\" = False # Only one recovery attempt allowed\n Else:\n Break # Failed to recover format\n\n \"available_steps\" -= 1\n\n Clean Jupyter Session \"PTP[i]\"\n Compute final response metrics (Mean/Tail Entropy, Confidence, etc.)\n Return Metrics and Candidate Answers (\"A_j\", \"AT_j\")\nEdited on May 17th 2026 for the rebuttal phase:
\nCmin and Cmax variants) and provided a dedicated ablation study table comparing their Public LB mean scores.Adding to the comments from @sambealing below, I have a few additional questions:
\nCoT: Why did you choose to use CoT-style prompting with a model that has already been extensively trained for reasoning? The official OpenAI reasoning best practices recommends avoiding CoT-style prompts for models with built-in reasoning capabilities, since they already reason internally. Did you also test your approach with a different prompt format that does not explicitly encourage CoT-style reasoning?
Time Management: Time management is a crucial part of this competition, with many possible areas for improvement. Since you present an adaptive time-management system as a key component, it would be helpful to see ablations comparing different strategies and configurations.
Variance on Local Evaluations: As we have seen in AIMO3, variance has a large impact on notebook scores due to problem difficulty and random ordering. Did you run your experiments multiple times and record local variance? I could only find reported averages for the public leaderboard scores, but not for the individual configurations tested on your 75-problem evaluation dataset. Adding variance estimates would allow readers to better assess the statistical significance of the differences between configurations.
Hi @philipvonderlind ,
\nthank you for your time and for these insightful questions.
\nPlease find my detailed responses to each point below.
\nWhile official OpenAI best practices suggest that CoT-style prompts are unnecessary for models with built-in reasoning, my empirical results with gpt-oss-120b tell a more nuanced story. In local runs, I observed that CoT prompts increased the number of correct responses for the hardest problems, although they generally generated longer responses for simpler tasks compared to formats that do not explicitly encourage CoT. To evaluate this, I submitted two versions of a previous notebook that differed only in the \"User\" prompt, but used a fixed configuration for \"System\", \"Developer\" and \"Tool\", as following:
\nmath, numpy, sympy and mpmath to solve the problem.\"The CoT version achieved a score of 39 on the Public Leaderboard, while the Non-CoT version achieved a score of 37.\nFurthermore, in https://arxiv.org/pdf/2508.15260 the authors achieve 99.9% accuracy on AIME 2025 using gpt-oss-120b by specifically appending \"Please reason step by step, and put your final answer within \\boxed{}.\" to the user prompt (Section \"F EXPERIMENTAL SETTINGS\").
\nI also tested several other prompting strategies:
\nUltimately, I found that using the word \"carefully\" provided the best tradeoff. It encourages precision, caution, and consistency without forcing the model into the extreme verbosity of a full \"step-by-step\" instruction. This approach can help to reduces errors on difficult problems while maintaining efficiency on simpler ones.
\nThe dynamic time management system was designed to mitigate the inefficiency of static allocation, where simple problems can waste budget and complex ones are cut short. Our ablation study confirms that the final_submission (dynamic) significantly outperforms the dynamic_time_management_disabled (static) run, which relies on a fixed average time per problem.\nThe dynamic time management is based on the scaling constants Cmin and Cmax (Section 3.3.). Initially, the parameter Cmax=2.1 was empirically selected to allow the model to reach its maximum effective context window depth on difficult problems. The following table summarizes additional configurations evaluated during the final phases of the competition:
| Cmin | \nCmax | \nPublic LB Scores | \nMean Score | \n
|---|---|---|---|
| 0.88 | \n2.2 | \n37 | \n37.0 | \n
| 0.86 | \n2.1 | \n39, 37, 38, 43 | \n39.25 (Final Submission) | \n
| 0.85 | \n2.0 | \n36 | \n36.0 | \n
| 0.80 | \n2.0 | \n39 | \n39.0 | \n
| 0.80 | \n2.1 | \n42, 41 | \n41.5 | \n
While the selected final submission (Cmin=0.86, Cmax=2.1) achieved the highest peak score of 43/50, subsequent analysis suggests that a lower floor (Cmin=0.80) paired with Cmax=2.1 yields a more stable mean performance (41.5).\nThe configuration (0.80, 2.1) was identified very close to the competition deadline, consequently, we prioritized the (0.86, 2.1) configuration for the final submission as it had already demonstrated high-ceiling potential in earlier runs.\nThese results indicate that performance is sensitive to the Cmin floor. A lower Cmin grants the system more flexibility to reallocate time from \"easy\" problems to \"hard\" ones, whereas an excessively high Cmax may lead to diminishing returns if the model exceeds its useful reasoning depth.\nWe agree that further systematic ablations on these hyperparameters could define a more precise \"Pareto front\" for time-constrained mathematical reasoning.
We appreciate the emphasis on variance. Mathematical reasoning tasks are indeed sensitive to problem ordering and stochastic sampling, and variance estimates are crucial to distinguish architectural gains from statistical noise.\nThe experimental runs reported in Section 4.5 were conducted near the competition deadline to provide a reproducible and objective summary of each technique’s behavior on a high-difficulty public dataset. We acknowledge that these specific local runs reflect a single execution. Due to the significant computational intensity—each 75-problem run requiring approximately 7.5 hours on an NVIDIA H100 GPU—and strict hardware availability constraints, conducting multiple full-scale repetitions was not feasible within the remaining timeframe.\nHowever, we believe the significance of the observed performance gaps is supported by the following factors:
\nk=7) reduces the impact of stochastic sampling.| Notebook Configuration | \nPublic LB Scores | \nMean Score | \n
|---|---|---|
| Final Submission | \n39, 37, 38, 43 | \n39.25 | \n
| Early Break Disabled | \n36, 38 | \n37 | \n
Hi, thank you for your submission to the Writeup Prize. @philipvonderlind and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.
\nmean_conf · (1-mean_entropy) aggregation. Did you have a chance to resubmit with the revised scoring function before the deadline, and if not, do you have any data of how it might affect performance on other benchmarks?Look forward to reading your responses on these.
", "rawMarkdown": "Hi, thank you for your submission to the Writeup Prize. @philipvonderlind and I will post some questions below that help us in assessing your submission more closely, as described in the [updated discussion post](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/689703).\n\n1. **Better tie-breaking metrics (§4.5.1.4):** You found that reasoning-path length and minimum token confidence are stronger discriminators than your final `mean_conf · (1-mean_entropy)` aggregation. Did you have a chance to resubmit with the revised scoring function before the deadline, and if not, do you have any data of how it might affect performance on other benchmarks?\n2. **Tool-fallback fire rate (§3.5.3):** §4.5.1.6 evaluates the tool-output fallback's accuracy, but I couldn't see the actual fire rate. On your 75-problem validation set, on what fraction did all 7 reasoning paths fail to produce a formatted final answer (i.e. when the fallback actually fires), and how often does the fallback then pick the correct answer?\n3. **TIR-friendly validation set:** Your 75-problem set is sourced from a problem set specifically focussed on tool-calling. Do you think the TIR gains you report would transfer to problem types where tool use is less natural (e.g. pure symbolic algebra, geometry proofs), or could the gains concentrate on problems that are already tool-friendly?\n\nLook forward to reading your responses on these.\n", "replies": [ { "id": 3458925, "postDate": "2026-05-17T00:11:20.943Z", "content": "Hi @sambealing ,
\nthank you for your time and for these insightful questions.
\nPlease find my detailed responses to each point below.
\nUnfortunately, I completed the tie-breaking analysis only after the competition deadline, so I was unable to submit a revised scoring function to the Public Leaderboard. However, post-hoc analysis on the experimental runs described in Section 4.5.1.4 indicates that the original aggregation (described in Section 3.5.1.) was not the most discriminative signal for selecting among candidate reasoning paths.\nIn particular, incorporating local confidence bottlenecks and reasoning-path length produced substantially stronger tie-breaking performance:
\n| Metric | \nTie-breaking accuracy | \n
|---|---|
| Original tie_break_score | \n0.492 | \n
Revised tie_break_score_A (1−length_perc)⋅min_conf | \n0.542 | \n
Revised tie_break_score_B tie_break_score⋅(1−length_perc)⋅min_conf | \n0.610 | \n
Revised tie_break_score_C tie_break_score⋅(1−length_perc)⋅min_conf⋅tail_conf | \n0.661 | \n
Here, length_perc denotes the ratio between the generated response length and the model context limit (response_length / max_model_len).\nThese results suggest that localized confidence failures are more informative than global averages. Metrics such as mean_conf and mean_entropy can remain relatively stable even when a reasoning trajectory contains a critical logical error. In contrast, min_conf and tail_conf act as bottleneck indicators: a single low-confidence transition often coincides with a hallucinated deduction or unstable reasoning step.\nThe inverse correlation between reasoning-path length and correctness was also notable. For Olympiad-style mathematical problems, successful solutions tended to emerge from relatively concise reasoning chains, whereas excessively long trajectories were more likely to exhibit reasoning drift, repetition, or self-inconsistent exploration.\nThese results, summarized in the accompanying chart, suggest that combining global stability metrics with local confidence statistics and trajectory-length regularization may provide a more robust tie-breaking strategy for adaptive reasoning systems.\n
On the 75-problem validation set, excluding the python_tool_disabled and baseline ablations (which do not use the Python tool), the tool-output fallback mechanism was triggered 45 times in total.\nA fallback event occurs when all 7 reasoning paths fail to produce a correctly formatted final answer, but at least one reasoning path generated a valid numerical result during a Python tool execution.\nAmong these 45 fallback activations, the mechanism recovered the correct answer 3 times, corresponding to a conditional accuracy of approximately 6.67% once the fallback fires.\nAlthough the conditional success rate is relatively low, the fallback was intentionally designed as a last-resort recovery mechanism rather than a primary selection strategy. In these cases, the standard extraction and aggregation pipeline had already failed completely, meaning that without the fallback the system would deterministically output an incorrect answer.\nConsequently, even a small number of successful recoveries translates directly into measurable score improvements. Importantly, the fallback introduces essentially no additional inference-time cost, since it only reuses numerical outputs already produced during earlier Python tool executions rather than generating new reasoning trajectories.
\nWhile the validation set was indeed tool-oriented, I believe the benefits of TIR would partially transfer to more \"abstract\" domains, albeit with a shift in how the tool is utilized. For instance:
\nsympy, the model can offload complex algebraic manipulations or coordinate geometry calculations to a deterministic solver. This minimizes \"calculation drift\" in long CoT sequences, where models often make small sign errors or arithmetic slips.Therefore, while the magnitude of the gain might vary depending on the problem type, I expect the system to maintain a competitive edge even on benchmarks where tool-use is less intuitive.
", "rawMarkdown": "Hi @sambealing ,\n\nthank you for your time and for these insightful questions. \n\nPlease find my detailed responses to each point below.\n\n\n### 1. **Better tie-breaking metrics**: \n\nUnfortunately, I completed the tie-breaking analysis only after the competition deadline, so I was unable to submit a revised scoring function to the Public Leaderboard. However, post-hoc analysis on the experimental runs described in Section 4.5.1.4 indicates that the original aggregation (described in Section 3.5.1.) was not the most discriminative signal for selecting among candidate reasoning paths.\nIn particular, incorporating local confidence bottlenecks and reasoning-path length produced substantially stronger tie-breaking performance:\n| Metric | Tie-breaking accuracy |\n| --- | --- |\n| Original tie_break_score | 0.492 |\n| Revised tie_break_score_A `(1−length_perc)⋅min_conf` | 0.542 |\n| Revised tie_break_score_B `tie_break_score⋅(1−length_perc)⋅min_conf` | 0.610 |\n| Revised tie_break_score_C `tie_break_score⋅(1−length_perc)⋅min_conf⋅tail_conf` | 0.661 |\n\nHere, `length_perc` denotes the ratio between the generated response length and the model context limit (response_length / max_model_len).\nThese results suggest that localized confidence failures are more informative than global averages. Metrics such as mean_conf and mean_entropy can remain relatively stable even when a reasoning trajectory contains a critical logical error. In contrast, min_conf and tail_conf act as bottleneck indicators: a single low-confidence transition often coincides with a hallucinated deduction or unstable reasoning step.\nThe inverse correlation between reasoning-path length and correctness was also notable. For Olympiad-style mathematical problems, successful solutions tended to emerge from relatively concise reasoning chains, whereas excessively long trajectories were more likely to exhibit reasoning drift, repetition, or self-inconsistent exploration.\nThese results, summarized in the accompanying chart, suggest that combining global stability metrics with local confidence statistics and trajectory-length regularization may provide a more robust tie-breaking strategy for adaptive reasoning systems.\n%20per%20summed%20metric.png?generation=1778976268035266&alt=media)\n\n\n### 2. **Tool-fallback fire rate**: \n\nOn the 75-problem validation set, excluding the python_tool_disabled and baseline ablations (which do not use the Python tool), the tool-output fallback mechanism was triggered **45** times in total.\nA fallback event occurs when all 7 reasoning paths fail to produce a correctly formatted final answer, but at least one reasoning path generated a valid numerical result during a Python tool execution.\nAmong these 45 fallback activations, the mechanism recovered the correct answer **3** times, corresponding to a conditional accuracy of approximately **6.67%** once the fallback fires.\nAlthough the conditional success rate is relatively low, the fallback was intentionally designed as a last-resort recovery mechanism rather than a primary selection strategy. In these cases, the standard extraction and aggregation pipeline had already failed completely, meaning that without the fallback the system would deterministically output an incorrect answer.\nConsequently, even a small number of successful recoveries translates directly into measurable score improvements. Importantly, the fallback introduces essentially no additional inference-time cost, since it only reuses numerical outputs already produced during earlier Python tool executions rather than generating new reasoning trajectories.\n\n\n### 3. **TIR-friendly validation set**: \n\nWhile the validation set was indeed tool-oriented, I believe the benefits of TIR would partially transfer to more \"abstract\" domains, albeit with a shift in how the tool is utilized. For instance:\n- **Symbolic Algebra & Geometry**: By leveraging libraries like `sympy`, the model can offload complex algebraic manipulations or coordinate geometry calculations to a deterministic solver. This minimizes \"calculation drift\" in long CoT sequences, where models often make small sign errors or arithmetic slips.\n- **Verification, not just Solution**: Even in proof-based or symbolic problems, TIR acts as a powerful verifier. The model can write scripts to test specific cases or validate intermediate numerical properties, providing a feedback loop that pure CoT lacks.\n- **Logical Structuring**: Writing code forces the model to decompose abstract problems into discrete, executable steps. This \"algorithmic thinking\" often help in maintaining a coherent logical flow even in pure proof tasks, reducing the likelihood of circular reasoning or vague hand-waving.\n\nTherefore, while the magnitude of the gain might vary depending on the problem type, I expect the system to maintain a competitive edge even on benchmarks where tool-use is less intuitive." } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T04:19:32Z", "competition_id": 118448, "competition_slug": "ai-mathematical-olympiad-progress-prize-3", "team_id": 14802138, "solution_url": "/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/potrhino-adaptive-test-time-optimization" }