Entropy-Weighted Self-Consistency for Olympiad-Level Mathematical Reasoning:\nA Competition Report on AIMO Progress Prize 3
\nKunal Aarse\ninfinitekunal4@gmail.com
\nWe present our solution to the AI Mathematical Olympiad Progress Prize 3 (AIMO3) competition on Kaggle, which tasks participants with solving 50 original olympiad-level mathematics problems spanning algebra, combinatorics, geometry, and number theory. Our approach centers on entropy-weighted self-consistency voting over multiple parallel inference attempts using the GPT-OSS-120B model served via vLLM under MXFP4 quantization on a single NVIDIA H100 GPU. Each candidate answer is weighted by the inverse of the mean Shannon entropy computed from top-5 token log-probabilities, rewarding confident, low-entropy completions. We further integrate a sandboxed Tool-Integrated Reasoning (TIR) environment providing persistent, stateful Jupyter kernels for symbolic computation via SymPy, numerical computation via NumPy, and high-precision arithmetic via mpmath. Our best submission achieved a public leaderboard score of 44/50 using 8 parallel attempts with flat time budget allocation and temperature T=1.0. We systematically ablated model choice (GPT-OSS-120B, GPT-OSS-20B, Qwen3-30B-A3B-fp8), attempt count (8, 16, 32), temperature (0.8, 0.9, 1.0), time budget strategy (flat vs. geometric decay), and fine-tuning approaches (SFT and GRPO). Notably, fine-tuning on mathematical corpora did not improve performance on this competition, and geometric budget allocation—despite theoretical advantages—produced inconsistent results (29–38) compared to the simpler flat-budget baseline.
\nMathematical reasoning remains one of the most challenging frontiers for artificial intelligence. Unlike natural language understanding, which benefits from statistical patterns and surface-level correlations, mathematical problem-solving at the olympiad level demands rigorous multi-step deduction, creative insight, and error-free computation. The AI Mathematical Olympiad (AIMO) Progress Prize series—organized by XTX Markets and hosted on Kaggle—has emerged as the primary benchmark for open-source mathematical AI, offering progressively harder problem sets and substantial prize pools to incentivize frontier research.\nAIMO3 represents a significant escalation in difficulty over its predecessors. The competition features 110 original problems (50 for the public leaderboard) spanning all major olympiad categories, designed to be resistant to memorization and requiring genuine mathematical reasoning. Critically, the answer format was changed from 3-digit (AIMO1, AIMO2) to a 5-digit range (0–99999), eliminating implicit modulo reductions and demanding more precise computation. Furthermore, the competition provided participants with access to NVIDIA H100 GPUs, enabling inference with models far larger than previously feasible in competition settings.\nOur submission explores the following central research questions:\n • Can entropy-weighted voting over parallel attempts outperform simple majority voting for olympiad-level mathematics?\n • What is the optimal balance between number of attempts and per-attempt compute budget under a fixed 5-hour wall-clock constraint?\n • Do SFT and GRPO fine-tuning on mathematical corpora improve base model performance on out-of-distribution olympiad problems?\n • How does dynamic (geometric decay) budget allocation compare to uniform time distribution?\nWe report that entropy-weighting provides a consistent advantage over raw voting, that 8 attempts with maximum per-attempt budget outperforms 16 or 32 attempts with reduced budgets, and that fine-tuning—despite theoretical promise—failed to generalize to the AIMO3 problem distribution. The remainder of this paper is structured as follows: Section 2 reviews related work including previous AIMO winners; Section 3 describes our methodology; Section 4 presents experimental setup; Section 5 reports results and analysis; Section 6 concludes with lessons learned and future directions. Full source code is provided in Appendices A, B, and C.
\nThe inaugural AIMO Progress Prize (July 2024) was won by Project Numina, a collaboration between Hugging Face and the NuminaMath team. Their winning approach fine-tuned a DeepSeek-Math-7B model on a curated corpus of 860K competition-grade problems sourced from AMC, AIME, MATH, and synthetic augmentation pipelines. The key technical contributions were: (1) a two-stage supervised fine-tuning pipeline combining chain-of-thought (CoT) and tool-integrated reasoning (TIR) data; (2) rejection sampling to filter training samples where the model-generated solution reached the correct answer; and (3) self-consistency voting across 32 parallel samples at inference time. Numina demonstrated that a relatively small (7B parameter) model, when fine-tuned on high-quality mathematical data at scale, could dramatically outperform much larger untuned models.\nThe CMU Math team (second place) employed a ToRA-style (Tool-integrated Reasoning Agent) approach, interleaving natural language reasoning steps with Python code execution. Their architecture emphasized tight coupling between symbolic reasoning in text and numerical verification through code, drawing on the insight that the model's natural language reasoning and computational results should mutually constrain each other. CMU Math's approach highlighted the importance of the execution environment quality: reliable, fast code execution directly impacted final score by enabling the model to verify intermediate steps.
\nAIMO2 (April 2025) saw a step-change in both problem difficulty (returning to 3-digit answers but with harder problems) and solution sophistication. The winning submission from Nvidia's NemoSkills team built upon their open-source NeMo-Skills framework, applying Group Relative Policy Optimization (GRPO)—a variant of reinforcement learning from verifiable rewards (RLVR)—to fine-tune Qwen2.5-Math and DeepSeek-R1 variants. GRPO optimizes the model's policy by sampling groups of responses, scoring each against a verifiable outcome (correct/incorrect numerical answer), and training the model to prefer correct trajectories while applying a KL penalty to prevent distribution collapse. The NemoSkills team showed that GRPO substantially improved pass@k rates, enabling reliable answers with fewer samples than their SFT-only baselines. Their technical report (Abadal et al., 2025; arXiv:2504.16891) provides a comprehensive ablation of training data quality, GRPO hyperparameters, and inference-time compute scaling.\nThe Imagination Research team (second place in AIMO2) focused on diverse self-consistency: rather than sampling from a single model, they employed a multi-model ensemble combining outputs from several mathematical reasoning models of different architectures and sizes. Their key insight was that model diversity in the voting pool—where different models make independent errors—improves aggregate accuracy beyond what homogeneous multi-sampling can achieve. They also employed answer clustering with a tolerance window (±2 for numerical proximity), which reduced the sensitivity of majority voting to small arithmetic errors in otherwise correct solution paths.
\nWang et al. (2023) introduced self-consistency sampling as a simple but highly effective decoding strategy: sample multiple reasoning paths at high temperature and select the most frequent answer. Subsequent work has explored weighted variants. Li et al. (2024) demonstrated that using model confidence scores (derived from token log-probabilities) as voting weights outperforms uniform majority voting, particularly for harder problems where the model occasionally produces highly confident correct answers alongside many uncertain wrong ones. Our entropy-weighting approach extends this line of work by computing Shannon entropy H = -Σ p log₂ p over the top-K token distribution at each decoding step, averaging across the full trajectory, and using 1/H as the answer weight.
\nTool-integrated reasoning (Gou et al., 2023; Chen et al., 2022) augments language model inference with access to Python interpreters, enabling exact symbolic computation, brute-force enumeration, and numerical verification. In the AIMO setting, TIR is essential for problems requiring modular arithmetic, polynomial factorization, and combinatorial enumeration. Our implementation uses isolated Jupyter kernels (one per parallel attempt) pre-loaded with SymPy, NumPy, mpmath (64-decimal-digit precision), and standard library modules. This architecture prevents cross-contamination between attempts while enabling the full expressiveness of the Python scientific stack within each attempt.
\nWe evaluated three models available within the competition's offline constraint:\n • GPT-OSS-120B (gpt-oss-120b): The primary model, a 120-billion parameter open-weight reasoning model served in MXFP4 (4-bit microscaled floating point) quantization via vLLM with fp8_e4m3 KV cache. MXFP4 reduces memory footprint to approximately 60–65 GB, enabling the full 120B model to reside on a single H100-80GB GPU with 96% memory utilization.\n • GPT-OSS-20B (gpt-oss-20b): A smaller 20B variant serving as a baseline and fallback configuration. Despite its smaller size, this model benefits from the same reasoning architecture and training regime.\n • Qwen3-30B-A3B-thinking-fp8: A Mixture-of-Experts (MoE) reasoning model from the Qwen3 family with 30B total parameters but only 3B active per forward pass (A3B architecture). Served in fp8 quantization, this model offered low latency per token while maintaining competitive mathematical reasoning quality.\nThe GPT-OSS-120B model under MXFP4 quantization consistently outperformed the other two models on the public leaderboard reference problems, validating the strong empirical relationship between model scale and olympiad-level reasoning quality. However, its larger size means fewer parallel attempts are possible within a fixed time budget, creating a fundamental tension between model quality and sampling diversity.
\nAll inference is served through a local vLLM OpenAI-compatible API endpoint launched at initialization. Key infrastructure components include:\n • vLLM async scheduling with prefix caching enabled, reducing redundant computation for parallel attempts sharing the same system and user prompts.\n • A model weight preloading step that reads all checkpoint files into the OS page cache using 16 parallel worker threads before launching the vLLM process, reducing server startup latency from 4–6 minutes to under 90 seconds.\n • Streaming token generation with real-time \\boxed{} pattern scanning, enabling early termination as soon as a valid answer is detected without waiting for the complete response.\n • Per-attempt quadratic seed spacing: attempt_seed = (seed + i)², ensuring decorrelated random number streams across parallel attempts without complex seeding logic.
\nEach inference request uses a structured system prompt implementing a five-phase problem-solving methodology: (1) UNDERSTAND — problem restatement and constraint identification; (2) EXPLORE — multi-strategy brainstorming; (3) PLAN — approach selection; (4) EXECUTE — systematic derivation; (5) VERIFY — answer validation. The system prompt additionally specifies ReasoningEffort.HIGH, activating the model's extended thinking mode.\nA tool prompt instructs the model on appropriate use of the Python execution environment: computations that are error-prone by hand, numerical verification of analytical results, conjecture testing, and small-case brute-force. The preference prompt appended to each user query enumerates the available scientific libraries (SymPy for symbolic computation, NumPy for numerical computation, mpmath for arbitrary-precision arithmetic) with concrete guidance on when each is most appropriate.
\nOur central methodological contribution is entropy-weighted voting. For each decoding step, vLLM returns the top-K (K=5) token log-probabilities. We compute the Shannon entropy of this distribution:\nH(t) = -Σₖ pₖ log₂ pₖ where pₖ = exp(logprob_k) / Σⱼ exp(logprob_j)\nThe mean entropy H̄ = (1/T) Σₜ H(t) is computed over all T tokens in the response. Lower mean entropy indicates a more confident, self-consistent generation. The voting weight for answer a from attempt i is w_i = 1 / max(H̄_i, ε), where ε = 10⁻⁹ prevents division by zero. The final answer is the candidate maximizing total weighted score: â = argmax_a Σᵢ wᵢ · 1[aᵢ = a].\nThis formulation rewards attempts where the model expresses high token-level confidence throughout its reasoning chain, which empirically correlates with correctness on mathematical problems. In contrast to raw majority voting—which treats all completed attempts equally—entropy weighting can select a minority answer if that answer comes from significantly more confident completions.
\nThe flat budget strategy (Document 1 / Document 4) allocates per-problem time as:\nbudget = min(time_left − (N−1) × base_timeout, ceiling_timeout)\nwhere time_left is the remaining notebook runtime, N is the number of problems remaining, base_timeout = 300s is the minimum guarantee per problem, and ceiling_timeout = 900s is the hard maximum. This strategy is conservative and predictable: early problems receive generous budgets, and unused time flows naturally to subsequent problems through the time_left term.
\nThe geometric decay strategy (Document 3) models the intuition that earlier problems in the randomly-ordered test set should receive more compute, since problem difficulty is unknown and early over-investment is recoverable, while late over-investment is catastrophic. The budget for the i-th problem is the i-th term of a geometric series summing exactly to the total notebook limit:\nbudget_i = T_remaining × (1−r) / (1−r^N)\nwhere r = 0.97 is the geometric ratio and N is the number of remaining problems. Simultaneously, the number of attempts decays geometrically: attempts_i = max(attempts_min, round(attempts_max × r^i)), with attempts_max = 16 and attempts_min = 4. A pace correction mechanism compares actual elapsed time against an ideal elapsed time trajectory, triggering emergency caps when the solver falls more than 300s or 600s behind schedule.
\nWe attempted two fine-tuning approaches using the Unsloth library with the TRL training framework:\n • Supervised Fine-Tuning (SFT): Fine-tuned on a dataset of IMO shortlist problems, Putnam competition solutions, and AIME problems with full solution chains. Training used LoRA adapters (rank 64, α=128) on the attention and MLP projection layers.\n • Group Relative Policy Optimization (GRPO): Trained using verifiable reward signals on the AIMO3 reference problems. Following the NemoSkills approach, groups of 4 responses were sampled per problem, scored as 1 (correct) or 0 (incorrect), and the policy gradient was computed relative to group baselines with KL regularization.\nBoth approaches used Unsloth's memory-efficient training pipeline to fit within the H100 GPU's 80GB VRAM budget while training the 120B model with MXFP4 precision. However, merging LoRA SFT adapters into the MXFP4 base model encountered compatibility issues between the Unsloth quantization format and standard adapter merging routines. When these issues were resolved and the merged model was evaluated, leaderboard scores did not improve over the base model, suggesting that the fine-tuning data distribution did not align well with the AIMO3 problem design philosophy (which explicitly aims to be resistant to training contamination through entirely novel problem construction).
\nAll experiments were conducted on a Kaggle notebook session equipped with a single NVIDIA H100 SXM5 80GB GPU, 2 CPU cores, and approximately 29 GB of system RAM. The competition enforces a 5-hour (18,000 second) wall-clock limit; we used a conservative 17,000–17,400 second budget to account for Kaggle overhead variance.
\nTable 1 summarizes the hyperparameter configurations evaluated across all submissions.
\nTable 1: Hyperparameter Configurations Evaluated\nModel Attempts Temp. Budget Fine-tune LB Score\nGPT-OSS-120B 8 1.0 Flat None 44 (Best)\nGPT-OSS-120B 8 0.9 Flat None 38–42\nGPT-OSS-120B 8 0.8 Flat None 36–40\nGPT-OSS-120B 16 1.0 Flat None 40–43\nGPT-OSS-120B 32 1.0 Flat None 37–41\nGPT-OSS-120B 16 1.0 Geometric None 29–38\nGPT-OSS-120B 8 1.0 Flat SFT No improvement\nGPT-OSS-120B 8 1.0 Flat GRPO No improvement\nGPT-OSS-20B 8 1.0 Flat None 28–34\nQwen3-30B-A3B-fp8 8 1.0 Flat None 30–36
\nPublic leaderboard evaluation: the Kaggle API serves 50 test problems sequentially in random order. Each submitted notebook is run once to generate the public score. Private leaderboard evaluation (after competition deadline) runs the notebook twice over 110 private problems; scoring uses a penalized accuracy where both runs correct = 1.0, one correct = 0.5, both wrong = 0.0. Our public score of 44 corresponds to answering 44 of 50 public problems correctly in a single run.
\nTemperature controls diversity in the self-consistency sampling pool. Lower temperatures produce more deterministic outputs, while higher temperatures introduce more exploration. Our ablation across T ∈ {0.8, 0.9, 1.0} with 8 attempts yielded:\n • T = 0.8: Scores 36–40. Insufficient diversity; multiple attempts converged to identical wrong answers, reducing the effective sample size.\n • T = 0.9: Scores 38–42. Better diversity, but still some convergence failures on harder geometry and combinatorics problems.\n • T = 1.0: Best score 44. Maximum diversity enabled the voting pool to include correct solution paths that lower temperatures never explored. The entropy-weighting mechanism was particularly effective here: high-temperature completions have high mean entropy overall, but correct answers often came from completions with significantly lower entropy than the distribution of wrong answers, allowing entropy-weighting to correctly identify them.\nThis result is counterintuitive relative to standard NLP tasks, where T=1.0 typically introduces too much noise. For olympiad mathematics, the problem space is so constrained that exploration is more valuable than precision in the decoding process—precision is provided by the verification steps within each reasoning chain.\n5.2 Attempt Count Ablation\nIncreasing attempts from 8 to 16 did not consistently improve scores, and 32 attempts performed worse than 8. The likely explanation is a compute-diversity tradeoff: with a fixed 5-hour budget and the large 120B model, increasing attempts necessarily reduces the per-attempt token budget. Attempts truncated by the deadline produced no answer (counted as None, excluded from voting), effectively reducing the useful voting pool. With 32 attempts and a 300-second problem budget, roughly 60% of attempts were truncating before reaching a \\boxed{} answer, yielding fewer usable votes than the 8-attempt baseline where nearly all attempts completed.\nThis finding suggests that for large models in time-constrained settings, depth (per-attempt reasoning quality) dominates breadth (number of attempts) beyond a modest ensemble size.\n5.3 Geometric Budget Allocation Analysis\nThe geometric decay strategy was motivated by two intuitions: (1) front-loading compute to earlier problems allows the model to \"warm up\" on easier instances; (2) problems closer to the end of the session, where the model has less time, are likely harder on average due to random ordering and natural difficulty correlation.\nIn practice, this strategy produced scores ranging from 29 to 38, substantially worse than the flat baseline (44). Post-hoc analysis suggests two failure modes:\n 1. Early over-investment: The geometric strategy allocated up to 660 seconds to the first problem. If that problem happened to be one the model could solve quickly (within 200 seconds), the remaining ~460 seconds was burned on redundant attempts hitting the early_stop threshold. This time was not recoverable for later harder problems.\n 2. Pace correction instability: The pace correction mechanism—which reduces budget and attempts when the solver falls behind the ideal geometric trajectory—triggered aggressively on problems that required many Python tool calls, reducing the quality of subsequent problems' solutions in a cascading failure mode.\nThe flat budget strategy's superior performance demonstrates that in the absence of per-problem difficulty signals, uniform allocation is more robust than adaptive strategies that assume any particular difficulty distribution.\n5.4 Fine-Tuning Analysis\nBoth SFT and GRPO fine-tuning failed to improve over the base model on AIMO3. We identify several reasons for this negative result:\n • Distribution mismatch: AIMO3 problems are explicitly designed to resist training contamination by being entirely novel. Fine-tuning on historical competition problems (IMO shortlists, Putnam, AIME) likely reinforced solution patterns that are not applicable to AIMO3's construction.\n • Quantization incompatibility: The MXFP4 quantization format used by GPT-OSS-120B is not natively supported by standard LoRA merging pipelines. Post-merge re-quantization introduced accuracy degradation that offset any reasoning improvements from the adapter.\n • Adapter-format fragility: Sequential single-adapter training (SFT then GRPO on the same adapter) is theoretically preferable to merging separately trained adapters. However, the small number of GRPO training examples available within the competition's data constraints was insufficient to produce statistically stable policy gradients.\n5.5 Model Comparison\nGPT-OSS-120B significantly outperformed both GPT-OSS-20B (28–34) and Qwen3-30B-A3B-fp8 (30–36). The 6× parameter advantage of the 120B model translated to approximately 10–14 additional correct answers, suggesting that model scale remains the dominant factor for olympiad-level reasoning in the absence of problem-specific fine-tuning. The Qwen3 MoE architecture, despite its active-parameter efficiency, did not close this gap—likely because the mathematical reasoning capability scales with total trained parameters rather than active inference parameters.
\nWe presented a comprehensive exploration of inference-time strategies for olympiad-level mathematical reasoning in the AIMO3 competition context. Our best submission achieving 44/50 on the public leaderboard establishes several empirical findings:\n 3. Entropy-weighted self-consistency voting provides a consistent and principled improvement over raw majority voting by leveraging token-level confidence signals.\n 4. Higher sampling temperature (T=1.0) is optimal for mathematical self-consistency, contradicting conventional wisdom from natural language generation tasks.\n 5. Depth (per-attempt token budget) dominates breadth (number of attempts) for large reasoning models under tight compute constraints. Eight well-resourced attempts outperform 16 or 32 resource-starved ones.\n 6. Flat uniform budget allocation outperforms geometric decay strategies in the absence of per-problem difficulty information. Adaptive strategies require reliable difficulty signals to work well.\n 7. Fine-tuning on historical competition mathematics does not generalize to novel olympiad problems explicitly designed to resist training contamination. Inference-time compute scaling remains more reliable than training-time optimization for out-of-distribution competition mathematics.\nThe gap between our 44/50 open-source result and the 50/50 results achievable by closed-source systems with sufficient compute (as reported by competition organizers) highlights the continued challenge of open-source mathematical AI. Future work should explore: multi-model ensembles for diversity-through-heterogeneity; online difficulty estimation from early attempts to dynamically allocate compute; better-calibrated confidence measures beyond simple token entropy; and training data construction that more closely mirrors the novel problem construction methodology of AIMO3.
\nReferences\n[1] Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2023). Self-consistency improves chain of thought reasoning in language models. ICLR 2023.\n[2] Gou, Z., Shao, Z., Gong, Y., Yang, Y., Huang, M., Duan, N., Chen, W., & Zhang, T. (2023). ToRA: A tool-integrated reasoning agent for mathematical problem solving. arXiv:2309.17452.\n[3] Chen, W., Ma, X., Wang, X., & Cohen, W. W. (2022). Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv:2211.12588.\n[4] Numina. (2024). NuminaMath: A solution to AIMO Progress Prize 1. Kaggle Competition Writeup.\n[5] Abadal, S., et al. (2025). NemoSkills: Scalable open-source mathematical reasoning via GRPO fine-tuning. arXiv:2504.16891.\n[6] Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023). Efficient memory management for large language model serving with PagedAttention. SOSP 2023.\n[7] Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., Re, C., Stoica, I., & Zhang, C. (2023). FlexGen: High-throughput generative inference of large language models with a single GPU. ICML 2023.\n[8] Li, Y., Lin, Z., Zhang, S., Fu, Q., Chen, B., Lou, J. G., & Chen, W. (2024). Making language models better reasoners with step-aware verifier. ACL 2023.\n[9] Frieder, S., et al. (2025). AI Mathematical Olympiad - Progress Prize 3. Kaggle Competition. https://kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3.\n[10] Han, J., et al. (2024). InternLM-Math: Open math large language models toward verifiable reasoning. arXiv:2402.06332.
\nAppendix A: Best Submission Code (44/50)\nThis appendix contains the complete source code for our best-scoring submission, achieving 44/50 on the public leaderboard. This uses GPT-OSS-120B with 8 parallel attempts, flat time budget allocation, temperature T=1.0, and entropy-weighted voting.
\nclass CFG:\n served_model_name = \"gpt-oss\"\n model_path = \"/kaggle/input/gpt-oss-120b/transformers/default/1\"\n kv_cache_dtype = \"fp8_e4m3\"\n dtype = \"auto\"\n high_problem_timeout = 900\n base_problem_timeout = 300\n notebook_limit = 17400\n server_timeout = 180\n session_timeout = 960\n jupyter_timeout = 6\n sandbox_timeout = 3\n stream_interval = 200\n context_tokens = 65536\n buffer_tokens = 512\n search_tokens = 32\n top_logprobs = 5\n batch_size = 256\n early_stop = 4\n attempts = 8 # Fixed 8 attempts — best configuration\n workers = 16\n turns = 128\n seed = 42\n gpu_memory_utilization = 0.96\n temperature = 1.0 # T=1.0 outperformed 0.8 and 0.9\n min_p = 0.02\ndef solve_problem(self, problem: str) -> int:\n # Flat budget: generous for each problem, guarantee floor\n elapsed_global = time.time() - self.notebook_start_time\n time_left = self.cfg.notebook_limit - elapsed_global\n problems_left_others = max(0, self.problems_remaining - 1)\n reserved_time = problems_left_others * self.cfg.base_problem_timeout\n\n budget = time_left - reserved_time\n budget = min(budget, self.cfg.high_problem_timeout) # cap at 900s\n budget = max(budget, self.cfg.base_problem_timeout) # floor at 300s\n\n deadline = time.time() + budget\ndef _compute_mean_entropy(self, logprobs_buffer: list) -> float:\n if not logprobs_buffer:\n return float(\"inf\")\n total_entropy = 0.0\n token_count = 0\n for top_logprobs_dict in logprobs_buffer:\n if not isinstance(top_logprobs_dict, dict) or not top_logprobs_dict:\n continue\n token_entropy = 0.0\n for token_str, log_prob in top_logprobs_dict.items():\n prob = math.exp(log_prob)\n if prob > 0:\n token_entropy -= prob * math.log2(prob)\n total_entropy += token_entropy\n token_count += 1\n return total_entropy / token_count if token_count > 0 else float(\"inf\")\n\ndef _select_answer(self, detailed_results: list) -> int:\n answer_weights = defaultdict(float)\n answer_votes = defaultdict(int)\n for result in detailed_results:\n answer, entropy = result[\"Answer\"], result[\"Entropy\"]\n if answer is not None:\n weight = 1.0 / max(entropy, 1e-9)\n answer_weights[answer] += weight\n answer_votes[answer] += 1\n scored = sorted(answer_weights.items(), key=lambda x: x[1], reverse=True)\n return scored[0][0] if scored else 0\nAppendix B: Geometric Time Allocation Code (Score: 29–38)\nThis appendix documents the geometric decay budget allocation strategy, which underperformed the flat baseline. It is included for completeness and to enable future researchers to understand why adaptive strategies require reliable difficulty signals.
\nclass CFG:\n # Geometric decay parameters\n geo_ratio = 0.97 # each problem gets 3% less budget than prior\n attempts_max = 16 # problem 1 gets maximum attempts\n attempts_min = 4 # floor — never fewer than 4\n tier_1_timeout = 900\n base_problem_timeout = 180\n notebook_limit = 17000 # conservative buffer\n# Projected budgets at competition start (T=17000, N=50, r=0.97):\n# Problem 1: ~660s, 16 attempts\n# Problem 10: ~491s, 12 attempts\n# Problem 25: ~303s, 8 attempts\n# Problem 40: ~187s, 5 attempts\n# Problem 50: ~180s, 4 attempts\n\nr = self.cfg.geo_ratio\nN = max(self.problems_remaining, 1)\nif abs(1.0 - r ** N) < 1e-9:\n geo_budget = time_left / N\nelse:\n geo_budget = time_left * (1.0 - r) / (1.0 - r ** N)\n\nbudget = min(geo_budget, self.cfg.tier_1_timeout)\nbudget = max(budget, self.cfg.base_problem_timeout)\n\n# Geometric attempts decay\ndynamic_attempts = max(\n self.cfg.attempts_min,\n round(self.cfg.attempts_max * (r ** problems_solved))\n)\n# Ideal elapsed = sum of first i terms of the geometric series\nif problems_solved > 0:\n per_problem_0 = self.cfg.notebook_limit * (1.0 - r) / (1.0 - r ** 50)\n ideal_elapsed = per_problem_0 * (1.0 - r ** problems_solved) / (1.0 - r)\nelse:\n ideal_elapsed = problems_solved * 298.0\n\npace_delta = elapsed_global - ideal_elapsed # positive = behind schedule\n\nif pace_delta > 600:\n budget = min(budget, self.cfg.base_problem_timeout * 2)\n dynamic_attempts = self.cfg.attempts_min\n pace_label = \" -> EMERGENCY CAP\"\nelif pace_delta > 300:\n budget = min(budget, 600)\n pace_label = \" -> PACE CAP (600s)\"\nAppendix C: Additional Experiment Details
\ncmd = [\n sys.executable, \"-m\", \"vllm.entrypoints.openai.api_server\",\n \"--seed\", str(cfg.seed),\n \"--model\", cfg.model_path,\n \"--served-model-name\", cfg.served_model_name,\n \"--tensor-parallel-size\", \"1\",\n \"--max-num-seqs\", str(cfg.batch_size), # 256\n \"--gpu-memory-utilization\", \"0.96\",\n \"--dtype\", \"auto\",\n \"--kv-cache-dtype\", \"fp8_e4m3\",\n \"--max-model-len\", \"65536\",\n \"--stream-interval\", \"200\",\n \"--async-scheduling\",\n \"--disable-log-stats\",\n \"--enable-prefix-caching\",\n]\nclass AIMO3Sandbox:\n \"\"\"Isolated Jupyter kernel for tool-integrated reasoning.\"\"\"\n\n def __init__(self, timeout: float):\n # Assign unique ports to prevent inter-kernel conflicts\n ports = self._get_next_ports(5)\n self._km = KernelManager()\n self._km.shell_port = ports[0]\n self._km.iopub_port = ports[1]\n self._km.stdin_port = ports[2]\n self._km.hb_port = ports[3]\n self._km.control_port = ports[4]\n self._km.start_kernel(env=env)\n\n # Pre-import scientific stack in every kernel\n self.execute(\n \"import math, numpy, sympy, itertools, collections, mpmath\\n\"\n \"mpmath.mp.dps = 64 # 64 decimal digits precision\"\n )\ndef _scan_for_answer(self, text: str) -> int | None:\n # Primary: \\boxed{N} pattern\n pattern = r\"\\boxed\\s{\\s([0-9,]+)\\s*}\"\n matches = re.findall(pattern, text)\n if matches:\n clean_value = matches[-1].replace(\",\", \"\")\n value = int(clean_value)\n if 0 <= value <= 99999:\n return value
\n# Fallback: \"final answer is N\" pattern\npattern = r\"final\\s+answer\\s+is\\s*([0-9,]+)\"\nmatches = re.findall(pattern, text, re.IGNORECASE)\nif matches:\n clean_value = matches[-1].replace(\",\", \"\")\n value = int(clean_value)\n if 0 <= value <= 99999:\n return value\n\nreturn None\nTable C1: Score Progression Over Competition Duration\nExperiment Key Change Score Range Best Score\nBaseline (20B) GPT-OSS-20B, 8 att. 28–34 34\nQwen3 MoE Qwen3-30B-A3B-fp8 30–36 36\nT=0.8 120B, T=0.8 36–40 40\nT=0.9 120B, T=0.9 38–42 42\nT=1.0, 8 att. Flat budget, 8 att. 42–44 44 ★\nT=1.0, 16 att. Flat budget, 16 att. 40–43 43\nT=1.0, 32 att. Flat budget, 32 att. 37–41 41\nGeometric Budget r=0.97 decay, 16→4 att. 29–38 38\nSFT Fine-tune LoRA SFT on 120B No gain —\nGRPO Fine-tune GRPO on 120B No gain —
\nThanks a lot @nihilisticneuralnet @andreasbis your work is very significant in this competition, I think these people can be given additional prizes or a job invitation.
\nOur team really spent a lot of time finding the best results and improving our final pipeline. We tried:
\n# Examples of validation, this is Yandex School of Data Analysis tasks\ntest_questions = [\n # Number Theory (1-6)\n (\"Find the remainder when 7^2024 is divided by 100.\", 1, \"Cycle of 7^n mod 100 is 4\"),\n (\"How many positive integers < 1000 divisible by 7 or 11 but NOT both?\", 208, \"Inclusion-exclusion\"),\n (\"Largest Armstrong number < 1000?\", 407, \"1, 153, 370, 371, 407\"),\n (\"Find gcd(2024, 1980).\", 44, \"Euclidean algorithm\"),\n (\"Sum of digits of 3^100 modulo 9?\", 0, \"Digital root: 3^100 ≡ 0 mod 9\"),\n (\"How many primes between 50 and 100?\", 10, \"53,59,61,67,71,73,79,83,89,97\"),\n\n # Sequences & Recurrence (7-10)\n (\"Sequence: a₁=1, a₂=1, aₙ=aₙ₋₁+2·aₙ₋₂. Find a₁₅.\", 10923, \"Linear recurrence\"),\n (\"Fibonacci: F₁=1, F₂=1. Find F₂₀ mod 1000.\", 57021 % 1000, \"Modular Fibonacci\"),\n (\"Geometric series: sum_{k=0}^{10} 2^k = ?\", 2047, \"2^11 - 1\"),\n (\"Arithmetic series: 1+4+7+...+100 = ?\", 1717, \"n=34 terms, sum=n*(a₁+aₙ)/2\"),\n\n # Combinatorics (11-16)\n (\"Ways to arrange 5 books if 2 must NOT be adjacent?\", 72, \"5! - 2·4!\"),\n (\"Non-negative integer solutions to x+y+z=20?\", 231, \"C(22,2)\"),\n (\"Coin flipped 10 times: P(exactly 6 heads)=a/b reduced. Find a+b.\", 617, \"C(10,6)/2^10=105/512\"),\n (\"How many 4-digit numbers with distinct digits?\", 4536, \"9·9·8·7\"),\n (\"Ways to choose 3 from 10 people?\", 120, \"C(10,3)\"),\n (\"Permutations of 'MATH'?\", 24, \"4!\"),\n\n # Geometry / Algebra (17-22)\n (\"Triangle: A=60°, AB=5, AC=8. Find BC².\", 49, \"Law of cosines\"),\n (\"Sum of digits of 2ⁿ = 31, n<30. Find n.\", 20, \"2^20=1048576\"),\n (\"Roots of x²-5x+6=0. Sum of squares of roots?\", 13, \"(r₁+r₂)²-2r₁r₂\"),\n (\"Circle radius 5, chord length 8. Distance from center to chord?\", 3, \"Pythagoras: √(25-16)\"),\n (\"Solve |2x-3|=7. Sum of solutions?\", 3, \"x=5 or x=-2\"),\n (\"Quadratic x²+bx+c has roots 3 and -5. Find b+c.\", -23, \"(x-3)(x+5)=x²+2x-15\"),\n\n # Logic & Attention (23-26)\n (\"n²+n+41 prime for n=0..39. How many primes?\", 40, \"Euler's polynomial\"),\n (\"'Lucky': divisible by 3 but not 9, in [1,198]. Count?\", 44, \"2 per block of 9 × 22\"),\n (\"Alice: ×3, +12, ÷3, -original. Result?\", 4, \"Algebraic simplification\"),\n (\"Last digit of 2^(2^2024)?\", 6, \"Cycle [2,4,8,6], exponent mod 4 = 0\"),\n\n # Advanced (27-30) \n (\"Vector space P₃(R). f(g)=GCD(x²-1,g)+g'. Is f linear? 1=YES, 0=NO.\", 0, \"Counterexample exists\"),\n (\"Cheaters: maximize E[solved]-P(caught). Optimal k?\", 3, \"Piecewise quadratic max\"),\n (\"Asymptotic: ∫₀¹ sin(tx)ln(x)dx ~ C·tⁿ·(ln t)ᵐ. Return |m|+|n|+round(100|C|).\", 158, \"n=-1,m=0,C=-π/2\"),\n (\"Nested squares n=7, final at (4,3). Count sequences?\", 400, \"C(6,3)×C(6,2)\"),\n ]\nweighted entropy actually worked!
", "votes": 1, "replies": [] }, { "id": 3446838, "author_name": "Jerry Chi", "author_url": "", "post_date": "2026-04-22T07:54:44.110000", "content": "My few-shot example prompts did not give me a good outcome. Good learning from your ones.
", "votes": 2, "replies": [] } ], "comment_count": 2, "messages": [ { "id": 3446431, "postDate": "2026-04-21T15:51:16.897Z", "content": "Entropy-Weighted Self-Consistency for Olympiad-Level Mathematical Reasoning:\nA Competition Report on AIMO Progress Prize 3
\nKunal Aarse\ninfinitekunal4@gmail.com
\nWe present our solution to the AI Mathematical Olympiad Progress Prize 3 (AIMO3) competition on Kaggle, which tasks participants with solving 50 original olympiad-level mathematics problems spanning algebra, combinatorics, geometry, and number theory. Our approach centers on entropy-weighted self-consistency voting over multiple parallel inference attempts using the GPT-OSS-120B model served via vLLM under MXFP4 quantization on a single NVIDIA H100 GPU. Each candidate answer is weighted by the inverse of the mean Shannon entropy computed from top-5 token log-probabilities, rewarding confident, low-entropy completions. We further integrate a sandboxed Tool-Integrated Reasoning (TIR) environment providing persistent, stateful Jupyter kernels for symbolic computation via SymPy, numerical computation via NumPy, and high-precision arithmetic via mpmath. Our best submission achieved a public leaderboard score of 44/50 using 8 parallel attempts with flat time budget allocation and temperature T=1.0. We systematically ablated model choice (GPT-OSS-120B, GPT-OSS-20B, Qwen3-30B-A3B-fp8), attempt count (8, 16, 32), temperature (0.8, 0.9, 1.0), time budget strategy (flat vs. geometric decay), and fine-tuning approaches (SFT and GRPO). Notably, fine-tuning on mathematical corpora did not improve performance on this competition, and geometric budget allocation—despite theoretical advantages—produced inconsistent results (29–38) compared to the simpler flat-budget baseline.
\nMathematical reasoning remains one of the most challenging frontiers for artificial intelligence. Unlike natural language understanding, which benefits from statistical patterns and surface-level correlations, mathematical problem-solving at the olympiad level demands rigorous multi-step deduction, creative insight, and error-free computation. The AI Mathematical Olympiad (AIMO) Progress Prize series—organized by XTX Markets and hosted on Kaggle—has emerged as the primary benchmark for open-source mathematical AI, offering progressively harder problem sets and substantial prize pools to incentivize frontier research.\nAIMO3 represents a significant escalation in difficulty over its predecessors. The competition features 110 original problems (50 for the public leaderboard) spanning all major olympiad categories, designed to be resistant to memorization and requiring genuine mathematical reasoning. Critically, the answer format was changed from 3-digit (AIMO1, AIMO2) to a 5-digit range (0–99999), eliminating implicit modulo reductions and demanding more precise computation. Furthermore, the competition provided participants with access to NVIDIA H100 GPUs, enabling inference with models far larger than previously feasible in competition settings.\nOur submission explores the following central research questions:\n • Can entropy-weighted voting over parallel attempts outperform simple majority voting for olympiad-level mathematics?\n • What is the optimal balance between number of attempts and per-attempt compute budget under a fixed 5-hour wall-clock constraint?\n • Do SFT and GRPO fine-tuning on mathematical corpora improve base model performance on out-of-distribution olympiad problems?\n • How does dynamic (geometric decay) budget allocation compare to uniform time distribution?\nWe report that entropy-weighting provides a consistent advantage over raw voting, that 8 attempts with maximum per-attempt budget outperforms 16 or 32 attempts with reduced budgets, and that fine-tuning—despite theoretical promise—failed to generalize to the AIMO3 problem distribution. The remainder of this paper is structured as follows: Section 2 reviews related work including previous AIMO winners; Section 3 describes our methodology; Section 4 presents experimental setup; Section 5 reports results and analysis; Section 6 concludes with lessons learned and future directions. Full source code is provided in Appendices A, B, and C.
\nThe inaugural AIMO Progress Prize (July 2024) was won by Project Numina, a collaboration between Hugging Face and the NuminaMath team. Their winning approach fine-tuned a DeepSeek-Math-7B model on a curated corpus of 860K competition-grade problems sourced from AMC, AIME, MATH, and synthetic augmentation pipelines. The key technical contributions were: (1) a two-stage supervised fine-tuning pipeline combining chain-of-thought (CoT) and tool-integrated reasoning (TIR) data; (2) rejection sampling to filter training samples where the model-generated solution reached the correct answer; and (3) self-consistency voting across 32 parallel samples at inference time. Numina demonstrated that a relatively small (7B parameter) model, when fine-tuned on high-quality mathematical data at scale, could dramatically outperform much larger untuned models.\nThe CMU Math team (second place) employed a ToRA-style (Tool-integrated Reasoning Agent) approach, interleaving natural language reasoning steps with Python code execution. Their architecture emphasized tight coupling between symbolic reasoning in text and numerical verification through code, drawing on the insight that the model's natural language reasoning and computational results should mutually constrain each other. CMU Math's approach highlighted the importance of the execution environment quality: reliable, fast code execution directly impacted final score by enabling the model to verify intermediate steps.
\nAIMO2 (April 2025) saw a step-change in both problem difficulty (returning to 3-digit answers but with harder problems) and solution sophistication. The winning submission from Nvidia's NemoSkills team built upon their open-source NeMo-Skills framework, applying Group Relative Policy Optimization (GRPO)—a variant of reinforcement learning from verifiable rewards (RLVR)—to fine-tune Qwen2.5-Math and DeepSeek-R1 variants. GRPO optimizes the model's policy by sampling groups of responses, scoring each against a verifiable outcome (correct/incorrect numerical answer), and training the model to prefer correct trajectories while applying a KL penalty to prevent distribution collapse. The NemoSkills team showed that GRPO substantially improved pass@k rates, enabling reliable answers with fewer samples than their SFT-only baselines. Their technical report (Abadal et al., 2025; arXiv:2504.16891) provides a comprehensive ablation of training data quality, GRPO hyperparameters, and inference-time compute scaling.\nThe Imagination Research team (second place in AIMO2) focused on diverse self-consistency: rather than sampling from a single model, they employed a multi-model ensemble combining outputs from several mathematical reasoning models of different architectures and sizes. Their key insight was that model diversity in the voting pool—where different models make independent errors—improves aggregate accuracy beyond what homogeneous multi-sampling can achieve. They also employed answer clustering with a tolerance window (±2 for numerical proximity), which reduced the sensitivity of majority voting to small arithmetic errors in otherwise correct solution paths.
\nWang et al. (2023) introduced self-consistency sampling as a simple but highly effective decoding strategy: sample multiple reasoning paths at high temperature and select the most frequent answer. Subsequent work has explored weighted variants. Li et al. (2024) demonstrated that using model confidence scores (derived from token log-probabilities) as voting weights outperforms uniform majority voting, particularly for harder problems where the model occasionally produces highly confident correct answers alongside many uncertain wrong ones. Our entropy-weighting approach extends this line of work by computing Shannon entropy H = -Σ p log₂ p over the top-K token distribution at each decoding step, averaging across the full trajectory, and using 1/H as the answer weight.
\nTool-integrated reasoning (Gou et al., 2023; Chen et al., 2022) augments language model inference with access to Python interpreters, enabling exact symbolic computation, brute-force enumeration, and numerical verification. In the AIMO setting, TIR is essential for problems requiring modular arithmetic, polynomial factorization, and combinatorial enumeration. Our implementation uses isolated Jupyter kernels (one per parallel attempt) pre-loaded with SymPy, NumPy, mpmath (64-decimal-digit precision), and standard library modules. This architecture prevents cross-contamination between attempts while enabling the full expressiveness of the Python scientific stack within each attempt.
\nWe evaluated three models available within the competition's offline constraint:\n • GPT-OSS-120B (gpt-oss-120b): The primary model, a 120-billion parameter open-weight reasoning model served in MXFP4 (4-bit microscaled floating point) quantization via vLLM with fp8_e4m3 KV cache. MXFP4 reduces memory footprint to approximately 60–65 GB, enabling the full 120B model to reside on a single H100-80GB GPU with 96% memory utilization.\n • GPT-OSS-20B (gpt-oss-20b): A smaller 20B variant serving as a baseline and fallback configuration. Despite its smaller size, this model benefits from the same reasoning architecture and training regime.\n • Qwen3-30B-A3B-thinking-fp8: A Mixture-of-Experts (MoE) reasoning model from the Qwen3 family with 30B total parameters but only 3B active per forward pass (A3B architecture). Served in fp8 quantization, this model offered low latency per token while maintaining competitive mathematical reasoning quality.\nThe GPT-OSS-120B model under MXFP4 quantization consistently outperformed the other two models on the public leaderboard reference problems, validating the strong empirical relationship between model scale and olympiad-level reasoning quality. However, its larger size means fewer parallel attempts are possible within a fixed time budget, creating a fundamental tension between model quality and sampling diversity.
\nAll inference is served through a local vLLM OpenAI-compatible API endpoint launched at initialization. Key infrastructure components include:\n • vLLM async scheduling with prefix caching enabled, reducing redundant computation for parallel attempts sharing the same system and user prompts.\n • A model weight preloading step that reads all checkpoint files into the OS page cache using 16 parallel worker threads before launching the vLLM process, reducing server startup latency from 4–6 minutes to under 90 seconds.\n • Streaming token generation with real-time \\boxed{} pattern scanning, enabling early termination as soon as a valid answer is detected without waiting for the complete response.\n • Per-attempt quadratic seed spacing: attempt_seed = (seed + i)², ensuring decorrelated random number streams across parallel attempts without complex seeding logic.
\nEach inference request uses a structured system prompt implementing a five-phase problem-solving methodology: (1) UNDERSTAND — problem restatement and constraint identification; (2) EXPLORE — multi-strategy brainstorming; (3) PLAN — approach selection; (4) EXECUTE — systematic derivation; (5) VERIFY — answer validation. The system prompt additionally specifies ReasoningEffort.HIGH, activating the model's extended thinking mode.\nA tool prompt instructs the model on appropriate use of the Python execution environment: computations that are error-prone by hand, numerical verification of analytical results, conjecture testing, and small-case brute-force. The preference prompt appended to each user query enumerates the available scientific libraries (SymPy for symbolic computation, NumPy for numerical computation, mpmath for arbitrary-precision arithmetic) with concrete guidance on when each is most appropriate.
\nOur central methodological contribution is entropy-weighted voting. For each decoding step, vLLM returns the top-K (K=5) token log-probabilities. We compute the Shannon entropy of this distribution:\nH(t) = -Σₖ pₖ log₂ pₖ where pₖ = exp(logprob_k) / Σⱼ exp(logprob_j)\nThe mean entropy H̄ = (1/T) Σₜ H(t) is computed over all T tokens in the response. Lower mean entropy indicates a more confident, self-consistent generation. The voting weight for answer a from attempt i is w_i = 1 / max(H̄_i, ε), where ε = 10⁻⁹ prevents division by zero. The final answer is the candidate maximizing total weighted score: â = argmax_a Σᵢ wᵢ · 1[aᵢ = a].\nThis formulation rewards attempts where the model expresses high token-level confidence throughout its reasoning chain, which empirically correlates with correctness on mathematical problems. In contrast to raw majority voting—which treats all completed attempts equally—entropy weighting can select a minority answer if that answer comes from significantly more confident completions.
\nThe flat budget strategy (Document 1 / Document 4) allocates per-problem time as:\nbudget = min(time_left − (N−1) × base_timeout, ceiling_timeout)\nwhere time_left is the remaining notebook runtime, N is the number of problems remaining, base_timeout = 300s is the minimum guarantee per problem, and ceiling_timeout = 900s is the hard maximum. This strategy is conservative and predictable: early problems receive generous budgets, and unused time flows naturally to subsequent problems through the time_left term.
\nThe geometric decay strategy (Document 3) models the intuition that earlier problems in the randomly-ordered test set should receive more compute, since problem difficulty is unknown and early over-investment is recoverable, while late over-investment is catastrophic. The budget for the i-th problem is the i-th term of a geometric series summing exactly to the total notebook limit:\nbudget_i = T_remaining × (1−r) / (1−r^N)\nwhere r = 0.97 is the geometric ratio and N is the number of remaining problems. Simultaneously, the number of attempts decays geometrically: attempts_i = max(attempts_min, round(attempts_max × r^i)), with attempts_max = 16 and attempts_min = 4. A pace correction mechanism compares actual elapsed time against an ideal elapsed time trajectory, triggering emergency caps when the solver falls more than 300s or 600s behind schedule.
\nWe attempted two fine-tuning approaches using the Unsloth library with the TRL training framework:\n • Supervised Fine-Tuning (SFT): Fine-tuned on a dataset of IMO shortlist problems, Putnam competition solutions, and AIME problems with full solution chains. Training used LoRA adapters (rank 64, α=128) on the attention and MLP projection layers.\n • Group Relative Policy Optimization (GRPO): Trained using verifiable reward signals on the AIMO3 reference problems. Following the NemoSkills approach, groups of 4 responses were sampled per problem, scored as 1 (correct) or 0 (incorrect), and the policy gradient was computed relative to group baselines with KL regularization.\nBoth approaches used Unsloth's memory-efficient training pipeline to fit within the H100 GPU's 80GB VRAM budget while training the 120B model with MXFP4 precision. However, merging LoRA SFT adapters into the MXFP4 base model encountered compatibility issues between the Unsloth quantization format and standard adapter merging routines. When these issues were resolved and the merged model was evaluated, leaderboard scores did not improve over the base model, suggesting that the fine-tuning data distribution did not align well with the AIMO3 problem design philosophy (which explicitly aims to be resistant to training contamination through entirely novel problem construction).
\nAll experiments were conducted on a Kaggle notebook session equipped with a single NVIDIA H100 SXM5 80GB GPU, 2 CPU cores, and approximately 29 GB of system RAM. The competition enforces a 5-hour (18,000 second) wall-clock limit; we used a conservative 17,000–17,400 second budget to account for Kaggle overhead variance.
\nTable 1 summarizes the hyperparameter configurations evaluated across all submissions.
\nTable 1: Hyperparameter Configurations Evaluated\nModel Attempts Temp. Budget Fine-tune LB Score\nGPT-OSS-120B 8 1.0 Flat None 44 (Best)\nGPT-OSS-120B 8 0.9 Flat None 38–42\nGPT-OSS-120B 8 0.8 Flat None 36–40\nGPT-OSS-120B 16 1.0 Flat None 40–43\nGPT-OSS-120B 32 1.0 Flat None 37–41\nGPT-OSS-120B 16 1.0 Geometric None 29–38\nGPT-OSS-120B 8 1.0 Flat SFT No improvement\nGPT-OSS-120B 8 1.0 Flat GRPO No improvement\nGPT-OSS-20B 8 1.0 Flat None 28–34\nQwen3-30B-A3B-fp8 8 1.0 Flat None 30–36
\nPublic leaderboard evaluation: the Kaggle API serves 50 test problems sequentially in random order. Each submitted notebook is run once to generate the public score. Private leaderboard evaluation (after competition deadline) runs the notebook twice over 110 private problems; scoring uses a penalized accuracy where both runs correct = 1.0, one correct = 0.5, both wrong = 0.0. Our public score of 44 corresponds to answering 44 of 50 public problems correctly in a single run.
\nTemperature controls diversity in the self-consistency sampling pool. Lower temperatures produce more deterministic outputs, while higher temperatures introduce more exploration. Our ablation across T ∈ {0.8, 0.9, 1.0} with 8 attempts yielded:\n • T = 0.8: Scores 36–40. Insufficient diversity; multiple attempts converged to identical wrong answers, reducing the effective sample size.\n • T = 0.9: Scores 38–42. Better diversity, but still some convergence failures on harder geometry and combinatorics problems.\n • T = 1.0: Best score 44. Maximum diversity enabled the voting pool to include correct solution paths that lower temperatures never explored. The entropy-weighting mechanism was particularly effective here: high-temperature completions have high mean entropy overall, but correct answers often came from completions with significantly lower entropy than the distribution of wrong answers, allowing entropy-weighting to correctly identify them.\nThis result is counterintuitive relative to standard NLP tasks, where T=1.0 typically introduces too much noise. For olympiad mathematics, the problem space is so constrained that exploration is more valuable than precision in the decoding process—precision is provided by the verification steps within each reasoning chain.\n5.2 Attempt Count Ablation\nIncreasing attempts from 8 to 16 did not consistently improve scores, and 32 attempts performed worse than 8. The likely explanation is a compute-diversity tradeoff: with a fixed 5-hour budget and the large 120B model, increasing attempts necessarily reduces the per-attempt token budget. Attempts truncated by the deadline produced no answer (counted as None, excluded from voting), effectively reducing the useful voting pool. With 32 attempts and a 300-second problem budget, roughly 60% of attempts were truncating before reaching a \\boxed{} answer, yielding fewer usable votes than the 8-attempt baseline where nearly all attempts completed.\nThis finding suggests that for large models in time-constrained settings, depth (per-attempt reasoning quality) dominates breadth (number of attempts) beyond a modest ensemble size.\n5.3 Geometric Budget Allocation Analysis\nThe geometric decay strategy was motivated by two intuitions: (1) front-loading compute to earlier problems allows the model to \"warm up\" on easier instances; (2) problems closer to the end of the session, where the model has less time, are likely harder on average due to random ordering and natural difficulty correlation.\nIn practice, this strategy produced scores ranging from 29 to 38, substantially worse than the flat baseline (44). Post-hoc analysis suggests two failure modes:\n 1. Early over-investment: The geometric strategy allocated up to 660 seconds to the first problem. If that problem happened to be one the model could solve quickly (within 200 seconds), the remaining ~460 seconds was burned on redundant attempts hitting the early_stop threshold. This time was not recoverable for later harder problems.\n 2. Pace correction instability: The pace correction mechanism—which reduces budget and attempts when the solver falls behind the ideal geometric trajectory—triggered aggressively on problems that required many Python tool calls, reducing the quality of subsequent problems' solutions in a cascading failure mode.\nThe flat budget strategy's superior performance demonstrates that in the absence of per-problem difficulty signals, uniform allocation is more robust than adaptive strategies that assume any particular difficulty distribution.\n5.4 Fine-Tuning Analysis\nBoth SFT and GRPO fine-tuning failed to improve over the base model on AIMO3. We identify several reasons for this negative result:\n • Distribution mismatch: AIMO3 problems are explicitly designed to resist training contamination by being entirely novel. Fine-tuning on historical competition problems (IMO shortlists, Putnam, AIME) likely reinforced solution patterns that are not applicable to AIMO3's construction.\n • Quantization incompatibility: The MXFP4 quantization format used by GPT-OSS-120B is not natively supported by standard LoRA merging pipelines. Post-merge re-quantization introduced accuracy degradation that offset any reasoning improvements from the adapter.\n • Adapter-format fragility: Sequential single-adapter training (SFT then GRPO on the same adapter) is theoretically preferable to merging separately trained adapters. However, the small number of GRPO training examples available within the competition's data constraints was insufficient to produce statistically stable policy gradients.\n5.5 Model Comparison\nGPT-OSS-120B significantly outperformed both GPT-OSS-20B (28–34) and Qwen3-30B-A3B-fp8 (30–36). The 6× parameter advantage of the 120B model translated to approximately 10–14 additional correct answers, suggesting that model scale remains the dominant factor for olympiad-level reasoning in the absence of problem-specific fine-tuning. The Qwen3 MoE architecture, despite its active-parameter efficiency, did not close this gap—likely because the mathematical reasoning capability scales with total trained parameters rather than active inference parameters.
\nWe presented a comprehensive exploration of inference-time strategies for olympiad-level mathematical reasoning in the AIMO3 competition context. Our best submission achieving 44/50 on the public leaderboard establishes several empirical findings:\n 3. Entropy-weighted self-consistency voting provides a consistent and principled improvement over raw majority voting by leveraging token-level confidence signals.\n 4. Higher sampling temperature (T=1.0) is optimal for mathematical self-consistency, contradicting conventional wisdom from natural language generation tasks.\n 5. Depth (per-attempt token budget) dominates breadth (number of attempts) for large reasoning models under tight compute constraints. Eight well-resourced attempts outperform 16 or 32 resource-starved ones.\n 6. Flat uniform budget allocation outperforms geometric decay strategies in the absence of per-problem difficulty information. Adaptive strategies require reliable difficulty signals to work well.\n 7. Fine-tuning on historical competition mathematics does not generalize to novel olympiad problems explicitly designed to resist training contamination. Inference-time compute scaling remains more reliable than training-time optimization for out-of-distribution competition mathematics.\nThe gap between our 44/50 open-source result and the 50/50 results achievable by closed-source systems with sufficient compute (as reported by competition organizers) highlights the continued challenge of open-source mathematical AI. Future work should explore: multi-model ensembles for diversity-through-heterogeneity; online difficulty estimation from early attempts to dynamically allocate compute; better-calibrated confidence measures beyond simple token entropy; and training data construction that more closely mirrors the novel problem construction methodology of AIMO3.
\nReferences\n[1] Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2023). Self-consistency improves chain of thought reasoning in language models. ICLR 2023.\n[2] Gou, Z., Shao, Z., Gong, Y., Yang, Y., Huang, M., Duan, N., Chen, W., & Zhang, T. (2023). ToRA: A tool-integrated reasoning agent for mathematical problem solving. arXiv:2309.17452.\n[3] Chen, W., Ma, X., Wang, X., & Cohen, W. W. (2022). Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv:2211.12588.\n[4] Numina. (2024). NuminaMath: A solution to AIMO Progress Prize 1. Kaggle Competition Writeup.\n[5] Abadal, S., et al. (2025). NemoSkills: Scalable open-source mathematical reasoning via GRPO fine-tuning. arXiv:2504.16891.\n[6] Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023). Efficient memory management for large language model serving with PagedAttention. SOSP 2023.\n[7] Sheng, Y., Zheng, L., Yuan, B., Li, Z., Ryabinin, M., Chen, B., Liang, P., Re, C., Stoica, I., & Zhang, C. (2023). FlexGen: High-throughput generative inference of large language models with a single GPU. ICML 2023.\n[8] Li, Y., Lin, Z., Zhang, S., Fu, Q., Chen, B., Lou, J. G., & Chen, W. (2024). Making language models better reasoners with step-aware verifier. ACL 2023.\n[9] Frieder, S., et al. (2025). AI Mathematical Olympiad - Progress Prize 3. Kaggle Competition. https://kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3.\n[10] Han, J., et al. (2024). InternLM-Math: Open math large language models toward verifiable reasoning. arXiv:2402.06332.
\nAppendix A: Best Submission Code (44/50)\nThis appendix contains the complete source code for our best-scoring submission, achieving 44/50 on the public leaderboard. This uses GPT-OSS-120B with 8 parallel attempts, flat time budget allocation, temperature T=1.0, and entropy-weighted voting.
\nclass CFG:\n served_model_name = \"gpt-oss\"\n model_path = \"/kaggle/input/gpt-oss-120b/transformers/default/1\"\n kv_cache_dtype = \"fp8_e4m3\"\n dtype = \"auto\"\n high_problem_timeout = 900\n base_problem_timeout = 300\n notebook_limit = 17400\n server_timeout = 180\n session_timeout = 960\n jupyter_timeout = 6\n sandbox_timeout = 3\n stream_interval = 200\n context_tokens = 65536\n buffer_tokens = 512\n search_tokens = 32\n top_logprobs = 5\n batch_size = 256\n early_stop = 4\n attempts = 8 # Fixed 8 attempts — best configuration\n workers = 16\n turns = 128\n seed = 42\n gpu_memory_utilization = 0.96\n temperature = 1.0 # T=1.0 outperformed 0.8 and 0.9\n min_p = 0.02\ndef solve_problem(self, problem: str) -> int:\n # Flat budget: generous for each problem, guarantee floor\n elapsed_global = time.time() - self.notebook_start_time\n time_left = self.cfg.notebook_limit - elapsed_global\n problems_left_others = max(0, self.problems_remaining - 1)\n reserved_time = problems_left_others * self.cfg.base_problem_timeout\n\n budget = time_left - reserved_time\n budget = min(budget, self.cfg.high_problem_timeout) # cap at 900s\n budget = max(budget, self.cfg.base_problem_timeout) # floor at 300s\n\n deadline = time.time() + budget\ndef _compute_mean_entropy(self, logprobs_buffer: list) -> float:\n if not logprobs_buffer:\n return float(\"inf\")\n total_entropy = 0.0\n token_count = 0\n for top_logprobs_dict in logprobs_buffer:\n if not isinstance(top_logprobs_dict, dict) or not top_logprobs_dict:\n continue\n token_entropy = 0.0\n for token_str, log_prob in top_logprobs_dict.items():\n prob = math.exp(log_prob)\n if prob > 0:\n token_entropy -= prob * math.log2(prob)\n total_entropy += token_entropy\n token_count += 1\n return total_entropy / token_count if token_count > 0 else float(\"inf\")\n\ndef _select_answer(self, detailed_results: list) -> int:\n answer_weights = defaultdict(float)\n answer_votes = defaultdict(int)\n for result in detailed_results:\n answer, entropy = result[\"Answer\"], result[\"Entropy\"]\n if answer is not None:\n weight = 1.0 / max(entropy, 1e-9)\n answer_weights[answer] += weight\n answer_votes[answer] += 1\n scored = sorted(answer_weights.items(), key=lambda x: x[1], reverse=True)\n return scored[0][0] if scored else 0\nAppendix B: Geometric Time Allocation Code (Score: 29–38)\nThis appendix documents the geometric decay budget allocation strategy, which underperformed the flat baseline. It is included for completeness and to enable future researchers to understand why adaptive strategies require reliable difficulty signals.
\nclass CFG:\n # Geometric decay parameters\n geo_ratio = 0.97 # each problem gets 3% less budget than prior\n attempts_max = 16 # problem 1 gets maximum attempts\n attempts_min = 4 # floor — never fewer than 4\n tier_1_timeout = 900\n base_problem_timeout = 180\n notebook_limit = 17000 # conservative buffer\n# Projected budgets at competition start (T=17000, N=50, r=0.97):\n# Problem 1: ~660s, 16 attempts\n# Problem 10: ~491s, 12 attempts\n# Problem 25: ~303s, 8 attempts\n# Problem 40: ~187s, 5 attempts\n# Problem 50: ~180s, 4 attempts\n\nr = self.cfg.geo_ratio\nN = max(self.problems_remaining, 1)\nif abs(1.0 - r ** N) < 1e-9:\n geo_budget = time_left / N\nelse:\n geo_budget = time_left * (1.0 - r) / (1.0 - r ** N)\n\nbudget = min(geo_budget, self.cfg.tier_1_timeout)\nbudget = max(budget, self.cfg.base_problem_timeout)\n\n# Geometric attempts decay\ndynamic_attempts = max(\n self.cfg.attempts_min,\n round(self.cfg.attempts_max * (r ** problems_solved))\n)\n# Ideal elapsed = sum of first i terms of the geometric series\nif problems_solved > 0:\n per_problem_0 = self.cfg.notebook_limit * (1.0 - r) / (1.0 - r ** 50)\n ideal_elapsed = per_problem_0 * (1.0 - r ** problems_solved) / (1.0 - r)\nelse:\n ideal_elapsed = problems_solved * 298.0\n\npace_delta = elapsed_global - ideal_elapsed # positive = behind schedule\n\nif pace_delta > 600:\n budget = min(budget, self.cfg.base_problem_timeout * 2)\n dynamic_attempts = self.cfg.attempts_min\n pace_label = \" -> EMERGENCY CAP\"\nelif pace_delta > 300:\n budget = min(budget, 600)\n pace_label = \" -> PACE CAP (600s)\"\nAppendix C: Additional Experiment Details
\ncmd = [\n sys.executable, \"-m\", \"vllm.entrypoints.openai.api_server\",\n \"--seed\", str(cfg.seed),\n \"--model\", cfg.model_path,\n \"--served-model-name\", cfg.served_model_name,\n \"--tensor-parallel-size\", \"1\",\n \"--max-num-seqs\", str(cfg.batch_size), # 256\n \"--gpu-memory-utilization\", \"0.96\",\n \"--dtype\", \"auto\",\n \"--kv-cache-dtype\", \"fp8_e4m3\",\n \"--max-model-len\", \"65536\",\n \"--stream-interval\", \"200\",\n \"--async-scheduling\",\n \"--disable-log-stats\",\n \"--enable-prefix-caching\",\n]\nclass AIMO3Sandbox:\n \"\"\"Isolated Jupyter kernel for tool-integrated reasoning.\"\"\"\n\n def __init__(self, timeout: float):\n # Assign unique ports to prevent inter-kernel conflicts\n ports = self._get_next_ports(5)\n self._km = KernelManager()\n self._km.shell_port = ports[0]\n self._km.iopub_port = ports[1]\n self._km.stdin_port = ports[2]\n self._km.hb_port = ports[3]\n self._km.control_port = ports[4]\n self._km.start_kernel(env=env)\n\n # Pre-import scientific stack in every kernel\n self.execute(\n \"import math, numpy, sympy, itertools, collections, mpmath\\n\"\n \"mpmath.mp.dps = 64 # 64 decimal digits precision\"\n )\ndef _scan_for_answer(self, text: str) -> int | None:\n # Primary: \\boxed{N} pattern\n pattern = r\"\\boxed\\s{\\s([0-9,]+)\\s*}\"\n matches = re.findall(pattern, text)\n if matches:\n clean_value = matches[-1].replace(\",\", \"\")\n value = int(clean_value)\n if 0 <= value <= 99999:\n return value
\n# Fallback: \"final answer is N\" pattern\npattern = r\"final\\s+answer\\s+is\\s*([0-9,]+)\"\nmatches = re.findall(pattern, text, re.IGNORECASE)\nif matches:\n clean_value = matches[-1].replace(\",\", \"\")\n value = int(clean_value)\n if 0 <= value <= 99999:\n return value\n\nreturn None\nTable C1: Score Progression Over Competition Duration\nExperiment Key Change Score Range Best Score\nBaseline (20B) GPT-OSS-20B, 8 att. 28–34 34\nQwen3 MoE Qwen3-30B-A3B-fp8 30–36 36\nT=0.8 120B, T=0.8 36–40 40\nT=0.9 120B, T=0.9 38–42 42\nT=1.0, 8 att. Flat budget, 8 att. 42–44 44 ★\nT=1.0, 16 att. Flat budget, 16 att. 40–43 43\nT=1.0, 32 att. Flat budget, 32 att. 37–41 41\nGeometric Budget r=0.97 decay, 16→4 att. 29–38 38\nSFT Fine-tune LoRA SFT on 120B No gain —\nGRPO Fine-tune GRPO on 120B No gain —
\nThanks a lot @nihilisticneuralnet @andreasbis your work is very significant in this competition, I think these people can be given additional prizes or a job invitation.
\nOur team really spent a lot of time finding the best results and improving our final pipeline. We tried:
\n# Examples of validation, this is Yandex School of Data Analysis tasks\ntest_questions = [\n # Number Theory (1-6)\n (\"Find the remainder when 7^2024 is divided by 100.\", 1, \"Cycle of 7^n mod 100 is 4\"),\n (\"How many positive integers < 1000 divisible by 7 or 11 but NOT both?\", 208, \"Inclusion-exclusion\"),\n (\"Largest Armstrong number < 1000?\", 407, \"1, 153, 370, 371, 407\"),\n (\"Find gcd(2024, 1980).\", 44, \"Euclidean algorithm\"),\n (\"Sum of digits of 3^100 modulo 9?\", 0, \"Digital root: 3^100 ≡ 0 mod 9\"),\n (\"How many primes between 50 and 100?\", 10, \"53,59,61,67,71,73,79,83,89,97\"),\n\n # Sequences & Recurrence (7-10)\n (\"Sequence: a₁=1, a₂=1, aₙ=aₙ₋₁+2·aₙ₋₂. Find a₁₅.\", 10923, \"Linear recurrence\"),\n (\"Fibonacci: F₁=1, F₂=1. Find F₂₀ mod 1000.\", 57021 % 1000, \"Modular Fibonacci\"),\n (\"Geometric series: sum_{k=0}^{10} 2^k = ?\", 2047, \"2^11 - 1\"),\n (\"Arithmetic series: 1+4+7+...+100 = ?\", 1717, \"n=34 terms, sum=n*(a₁+aₙ)/2\"),\n\n # Combinatorics (11-16)\n (\"Ways to arrange 5 books if 2 must NOT be adjacent?\", 72, \"5! - 2·4!\"),\n (\"Non-negative integer solutions to x+y+z=20?\", 231, \"C(22,2)\"),\n (\"Coin flipped 10 times: P(exactly 6 heads)=a/b reduced. Find a+b.\", 617, \"C(10,6)/2^10=105/512\"),\n (\"How many 4-digit numbers with distinct digits?\", 4536, \"9·9·8·7\"),\n (\"Ways to choose 3 from 10 people?\", 120, \"C(10,3)\"),\n (\"Permutations of 'MATH'?\", 24, \"4!\"),\n\n # Geometry / Algebra (17-22)\n (\"Triangle: A=60°, AB=5, AC=8. Find BC².\", 49, \"Law of cosines\"),\n (\"Sum of digits of 2ⁿ = 31, n<30. Find n.\", 20, \"2^20=1048576\"),\n (\"Roots of x²-5x+6=0. Sum of squares of roots?\", 13, \"(r₁+r₂)²-2r₁r₂\"),\n (\"Circle radius 5, chord length 8. Distance from center to chord?\", 3, \"Pythagoras: √(25-16)\"),\n (\"Solve |2x-3|=7. Sum of solutions?\", 3, \"x=5 or x=-2\"),\n (\"Quadratic x²+bx+c has roots 3 and -5. Find b+c.\", -23, \"(x-3)(x+5)=x²+2x-15\"),\n\n # Logic & Attention (23-26)\n (\"n²+n+41 prime for n=0..39. How many primes?\", 40, \"Euler's polynomial\"),\n (\"'Lucky': divisible by 3 but not 9, in [1,198]. Count?\", 44, \"2 per block of 9 × 22\"),\n (\"Alice: ×3, +12, ÷3, -original. Result?\", 4, \"Algebraic simplification\"),\n (\"Last digit of 2^(2^2024)?\", 6, \"Cycle [2,4,8,6], exponent mod 4 = 0\"),\n\n # Advanced (27-30) \n (\"Vector space P₃(R). f(g)=GCD(x²-1,g)+g'. Is f linear? 1=YES, 0=NO.\", 0, \"Counterexample exists\"),\n (\"Cheaters: maximize E[solved]-P(caught). Optimal k?\", 3, \"Piecewise quadratic max\"),\n (\"Asymptotic: ∫₀¹ sin(tx)ln(x)dx ~ C·tⁿ·(ln t)ᵐ. Return |m|+|n|+round(100|C|).\", 158, \"n=-1,m=0,C=-π/2\"),\n (\"Nested squares n=7, final at (4,3). Count sequences?\", 400, \"C(6,3)×C(6,2)\"),\n ]\nweighted entropy actually worked!
", "rawMarkdown": "weighted entropy actually worked!", "votes": 1 }, { "id": 3446838, "postDate": "2026-04-22T07:54:44.110Z", "content": "My few-shot example prompts did not give me a good outcome. Good learning from your ones.
", "rawMarkdown": "My few-shot example prompts did not give me a good outcome. Good learning from your ones.", "votes": 2 } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T04:22:28Z", "competition_id": 118448, "competition_slug": "ai-mathematical-olympiad-progress-prize-3", "team_id": 14798985, "solution_url": "/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/entropy-weighted-self-consistency-for-olympiad-lev" }