This writeup was originally published as a Project writeup on April 22, 2026. It has been migrated here to properly link it to this competition, as Kaggle currently does not provide an option to re-link an existing writeup to a competition.
\nhttps://www.kaggle.com/writeups/khoinguyennguyen/adaptive-tree-of-thoughts-with-eagle3-accelerated
\n| Submission | \nPublic LB | \nPrivate LB | \n
|---|---|---|
| Base GPT-120B, simple prompt (intermediate) | \n39--41 (2× hit 41) | \n-- | \n
| SFT GPT-120B, structured prompt + technique hints | \n36 | \n38.5 | \n
This writeup is based on the SFT submission to the private leaderboard. Private scores will be added during the rebuttal phase.\n(Updated: The Private LB score is 38.5, while being higher than Public LB score 36, the structure prompt seems to be penalizing as it limits the solving attempts to more correlated and concentrated set of approaches and reduces diversity)
\n| Resource | \nLink | \n
|---|---|
| Complete write-up (50pp, 6 figures, 14 appendices) | \nZenodo DOI: 10.5281/zenodo.19695613 | \n
| Final submission notebook (SFT) | \nKaggle | \n
| Base stable notebook | \nKaggle | \n
| Eagle3 speculative decoding notebook | \nKaggle | \n
| Blackboard multi-agent solver | \nKaggle | \n
| Cross-model hard problem benchmarks | \nKaggle | \n
| SFT model weights | \nHuggingFace | \n
The complete write-up PDF covers contextual analysis, hypotheses, solution landscape, architectural rationale, all 67 parameters, scoring formulas, and prompt text. This post is a summary. The PDF is the document of record.
\nComplete write-up: https://doi.org/10.5281/zenodo.19695613
\nFinal notebook - Google Colab Version COLAB (Ready To Run)
\nThis submission uses GPT-120B with Eagle3 speculative decoding inside an adaptive Tree-of-Thoughts architecture. The system runs 8 to 16 parallel attempts per problem. Reinforcement-wave attempts inherit full Jupyter kernel state from first-wave snapshots. A quadratic scaling function manages resources across 50 problems. Answer selection uses effort-weighted scoring. The base model scores 39-41 on the public leaderboard.
\nFor the final submission, we applied Neural Boosting SFT to fine-tune the last transformer layer's MoE expert weights on a single consumer GPU. This is submitted as an experiment on the private distribution.
\nWe began by examining publicly available data from the AIMO-2 evaluations (AIMO Prize, 2025).
\nAIMO-2 winners (NemoSkills, NVIDIA) deployed a 14B dense model on 4×L4 GPUs, 96GB total VRAM, and scored 34/50. The commercial model o3-preview scored 47/50 on the same problems. The organizers subsequently re-ran the winning models on 8×H100 GPUs (640GB total) with all time constraints removed. NemoSkills scored 35. imagination-research scored 35.
\nThe infrastructure upgrade from 96GB to 640GB produced a +1 point improvement. This indicates the bottleneck is not hardware or time budget.
\nThree data points from the GPT model family are relevant. o3-preview scored 47/50 on AIMO-2. GPT-120B achieves pass@20 of 47.56 on AIMO-3 public. GPT-20B reached 30+ in several public notebooks including ours. These models share a training lineage and consistently outperform dense models of comparable scale on competition math.
\nTwo qualifications are necessary. First, GPT-OSS inherits training methodology from OpenAI's pipeline. Its performance does not establish that open-source models generally are approaching commercial capability. A controlled comparison would require testing Qwen-3.5-122B and Nemotron-120B-Super under equivalent TIR conditions, which was not done. Second, competition math tests a specific reasoning regime. The actual commercial vs open-source gap on harder cognitive tasks may be significantly larger than these scores suggest.
\nThis analysis informed our primary design decision: to invest in extraction architecture rather than model modification. The model already produces correct answers at sufficient rate. The engineering problem is capturing them.
\nFull analysis in the PDF (§1.2, §2.1).
\nWe compared GPT-120B reasoning traces against Qwen-series models on competition problems with TIR enabled. A consistent behavioral pattern emerged.
\nGPT-120B reads a problem and immediately computes small cases through code. It observes patterns in the output, generalizes, and iterates. Code functions as the primary reasoning medium.
\nNon-GPT models read a problem and deliberate in text first. They form a structured plan, write code to verify it, then revise. Code functions as a verification step after text-based reasoning.
\nWe mapped this to two established frameworks: Boyd's OODA loop (Observe-Orient-Decide-Act) for the GPT pattern, and Deming's PDCA cycle (Plan-Do-Check-Act) for the non-GPT pattern. This is an observational characterization used for engineering decision-making. It is not a mechanistic claim.
\nThe engineering implication is specific: structured prompts that impose planning should add overhead on an OODA-mode model without changing its underlying behavior. We tested this. The simpler prompt scored 39-41. The structured prompt scored 36. The result is consistent with the observation, though other explanations are possible.
\nThree origin hypotheses are discussed in the PDF (§2.2). None are verified.
\nThe organizer pass@20 evaluation shows GPT-120B reaches 47.56/50 with 20 diverse attempts. The capability exists. The problem is extraction under time, memory, and reliability constraints. The architecture addresses this through four mechanisms.
\nWave 1 launches 8 independent attempts. Each snapshots its conversation history and dill-serialized Jupyter kernel state into a shared database. After 75 seconds, Wave 2 launches up to 8 reinforcement attempts. Each restores a first-wave snapshot - variables, functions, computed results - and continues from the branch point in a different direction.
\nA reinforcement attempt branching at turn 5 inherits 60+ seconds of prior computation. The standard community approach of independent flat attempts discards this progress. Kernel state restoration uses a 3-tier fallback for robustness.
\nThe system provides escalating time-awareness nudges (gentle, warning, urgent) but does not force truncation. Context compression presents a dual choice: submit the answer or summarize for continued reasoning. The model has more information about its reasoning state than any fixed policy can encode.
\nThe function min(1, ratio²) adjusts resources based on remaining time per remaining problem, where ratio is the average budget ratio for remaining problems. At ratio 0.7, it allocates 49% vs 70% for linear. This reduces the number of solving attempts when time is tight. The rationale: if harder problems come first, the budget ratio drops below 1, and the solver allocates fewer attempts to increase throughput per attempt. Since the hard problems have already passed, easier problems arrive later and require fewer attempts regardless. This is an adaptive optimization dynamic that self-corrects regardless of problem ordering.
Three-factor scoring replaces majority voting: reasoning effort, code execution effectiveness, and baseline vote contribution. Independent convergence from unrelated branches adds a tiebreaker. On hard problems where the correct answer appears in a minority of attempts, majority voting selects against it. Effort-weighted scoring mitigates this.
\nEight categories of defensive mechanisms: VRAM management, process isolation, timeout hierarchy (6 nested levels), thread safety, graceful degradation, budget safety, sandbox management, and environment propagation. In blind deployment, resilience determines whether a 45-problem solution scores 45 or crashes and scores 30.
\nFull parameter table (~67 parameters), resilience inventory, and scoring formulas in the PDF (§4, Appendices A-D, H).
\nThe writeup evaluation criteria include model contribution. This section addresses that criterion.
\nThe fine-tuning constraint on GPT-120B is well-defined. The MoE expert weights are the natural target for behavioral modification. Four obstacles prevent standard approaches: GPT-OSS fuses experts into packed tensors (LoRA cannot work on them), full-model SFT requires multi-GPU compute, heavy CoT SFT risks overwriting the OODA-like reasoning mode, and intermediate layer modifications degrade Eagle3 throughput.
\nMost teams focused on prompt engineering. We developed an alternative approach.
\nSplit the model into HEAD (all layers except the last, frozen MXFP4, runs at inference speed through vLLM) and TAIL (last layer in bf16, trainable). The speculators library (vllm-project) extracts hidden states from HEAD. TAIL trains on those states with cross-entropy loss. No architecture change. Full weight access to attention, MoE experts, and router in the trainable layer.
\nOne RTX Pro 6000 on Google Colab. Under 2 hours. ~2,500 mixed samples (pattern recognition capability training and anti-forgetting samples).
\nAll Eagle3-dependent layers remain untouched. The goal is adding pattern recognition capacity orthogonal to existing knowledge, not shifting the output distribution. Training was stopped when harder reference problems showed longer reasoning traces while easier problems showed no change. Minimal drift was the target.
\nThe method supports a bank of domain-specific last-layer variants stored individually and merged at deployment. Hot-swapping via in-engine weight reload would eliminate the merge step but requires vLLM support not yet available.
\nMethod details and pipeline diagram in the PDF (§4.8, Appendix L, Figure 4).
\n| Configuration | \nPublic LB | \n
|---|---|
| Simple prompt, base GPT-120B | \n39-41 | \n
| Structured prompt, base GPT-120B | \n36 | \n
| Structured prompt + technique hints, SFT GPT-120B | \n36 | \n
Three factors contributed to the score reduction.
\nFirst, domain technique hints on the base model triggered a cascade: the model applied advanced techniques to problems that did not require them, producing longer traces. Adaptive scaling then reduced resources for subsequent problems.
\nSecond, the SFT submission retained technique hints as a deliberate experiment. The hypothesis: SFT-trained pattern recognition may enable selective hint use that the base model cannot manage. Whether this holds on the private distribution is an open question.
\nThird, time spent on the OODA/PDCA analysis and Neural Boosting SFT development reduced the time available for parameter tuning and submission iteration.
\nWe submitted the third one (36 Public LB) as an experiment of several hypotheses on private leaderboard rather than using the already stable 41 with simple prompt.
\nDetailed analysis in the PDF (§3.3, §5).
\nThree patterns from this competition may generalize.
\nThe AIMO-2 team data shows a 10-15 point gap between pass@64 and maj@64 (Moshkov et al., 2025, Figure 2). On hard problems, majority voting selects against the correct answer. Effort-weighted scoring is one mitigation. Inter-attempt communication (solver-verifier debate, as prototyped in our Blackboard notebook) may be another, though synchronization risks prevented deployment in this competition.
\nBenchmark scores predicted AIMO performance poorly. GPT-120B outscored models with higher standalone benchmarks. Behavioral criteria - how a model reasons when computation is available - may be a more useful selection signal than benchmark rankings for TIR-enabled applications.
\nCompetition problems test monologue-style solving. If commercial models have reached multi-model routing and internal debate capabilities, the actual commercial vs open-source gap on tasks requiring emergent complexity may be substantially larger than competition formats can measure.
\n50 pages. 6 figures. 28 references. 14 appendices.
\nDownload: https://doi.org/10.5281/zenodo.19695613
\nThanks to the AIMO Prize organizers, XTX Markets, and Kaggle. Special thanks to Simon Frieder for the evaluation framework design.
\nCommunity contributors: nihilisticneuralnet, andreasbis, ryanholbrook, friederrr, wenliang1990 (Eagle3 draft model), and others who shared work through discussion posts and notebooks.
", "comments": [], "comment_count": 0, "messages": [ { "id": 3449144, "postDate": "2026-04-27T15:45:21.243Z", "content": "This writeup was originally published as a Project writeup on April 22, 2026. It has been migrated here to properly link it to this competition, as Kaggle currently does not provide an option to re-link an existing writeup to a competition.
\nhttps://www.kaggle.com/writeups/khoinguyennguyen/adaptive-tree-of-thoughts-with-eagle3-accelerated
\n| Submission | \nPublic LB | \nPrivate LB | \n
|---|---|---|
| Base GPT-120B, simple prompt (intermediate) | \n39--41 (2× hit 41) | \n-- | \n
| SFT GPT-120B, structured prompt + technique hints | \n36 | \n38.5 | \n
This writeup is based on the SFT submission to the private leaderboard. Private scores will be added during the rebuttal phase.\n(Updated: The Private LB score is 38.5, while being higher than Public LB score 36, the structure prompt seems to be penalizing as it limits the solving attempts to more correlated and concentrated set of approaches and reduces diversity)
\n| Resource | \nLink | \n
|---|---|
| Complete write-up (50pp, 6 figures, 14 appendices) | \nZenodo DOI: 10.5281/zenodo.19695613 | \n
| Final submission notebook (SFT) | \nKaggle | \n
| Base stable notebook | \nKaggle | \n
| Eagle3 speculative decoding notebook | \nKaggle | \n
| Blackboard multi-agent solver | \nKaggle | \n
| Cross-model hard problem benchmarks | \nKaggle | \n
| SFT model weights | \nHuggingFace | \n
The complete write-up PDF covers contextual analysis, hypotheses, solution landscape, architectural rationale, all 67 parameters, scoring formulas, and prompt text. This post is a summary. The PDF is the document of record.
\nComplete write-up: https://doi.org/10.5281/zenodo.19695613
\nFinal notebook - Google Colab Version COLAB (Ready To Run)
\nThis submission uses GPT-120B with Eagle3 speculative decoding inside an adaptive Tree-of-Thoughts architecture. The system runs 8 to 16 parallel attempts per problem. Reinforcement-wave attempts inherit full Jupyter kernel state from first-wave snapshots. A quadratic scaling function manages resources across 50 problems. Answer selection uses effort-weighted scoring. The base model scores 39-41 on the public leaderboard.
\nFor the final submission, we applied Neural Boosting SFT to fine-tune the last transformer layer's MoE expert weights on a single consumer GPU. This is submitted as an experiment on the private distribution.
\nWe began by examining publicly available data from the AIMO-2 evaluations (AIMO Prize, 2025).
\nAIMO-2 winners (NemoSkills, NVIDIA) deployed a 14B dense model on 4×L4 GPUs, 96GB total VRAM, and scored 34/50. The commercial model o3-preview scored 47/50 on the same problems. The organizers subsequently re-ran the winning models on 8×H100 GPUs (640GB total) with all time constraints removed. NemoSkills scored 35. imagination-research scored 35.
\nThe infrastructure upgrade from 96GB to 640GB produced a +1 point improvement. This indicates the bottleneck is not hardware or time budget.
\nThree data points from the GPT model family are relevant. o3-preview scored 47/50 on AIMO-2. GPT-120B achieves pass@20 of 47.56 on AIMO-3 public. GPT-20B reached 30+ in several public notebooks including ours. These models share a training lineage and consistently outperform dense models of comparable scale on competition math.
\nTwo qualifications are necessary. First, GPT-OSS inherits training methodology from OpenAI's pipeline. Its performance does not establish that open-source models generally are approaching commercial capability. A controlled comparison would require testing Qwen-3.5-122B and Nemotron-120B-Super under equivalent TIR conditions, which was not done. Second, competition math tests a specific reasoning regime. The actual commercial vs open-source gap on harder cognitive tasks may be significantly larger than these scores suggest.
\nThis analysis informed our primary design decision: to invest in extraction architecture rather than model modification. The model already produces correct answers at sufficient rate. The engineering problem is capturing them.
\nFull analysis in the PDF (§1.2, §2.1).
\nWe compared GPT-120B reasoning traces against Qwen-series models on competition problems with TIR enabled. A consistent behavioral pattern emerged.
\nGPT-120B reads a problem and immediately computes small cases through code. It observes patterns in the output, generalizes, and iterates. Code functions as the primary reasoning medium.
\nNon-GPT models read a problem and deliberate in text first. They form a structured plan, write code to verify it, then revise. Code functions as a verification step after text-based reasoning.
\nWe mapped this to two established frameworks: Boyd's OODA loop (Observe-Orient-Decide-Act) for the GPT pattern, and Deming's PDCA cycle (Plan-Do-Check-Act) for the non-GPT pattern. This is an observational characterization used for engineering decision-making. It is not a mechanistic claim.
\nThe engineering implication is specific: structured prompts that impose planning should add overhead on an OODA-mode model without changing its underlying behavior. We tested this. The simpler prompt scored 39-41. The structured prompt scored 36. The result is consistent with the observation, though other explanations are possible.
\nThree origin hypotheses are discussed in the PDF (§2.2). None are verified.
\nThe organizer pass@20 evaluation shows GPT-120B reaches 47.56/50 with 20 diverse attempts. The capability exists. The problem is extraction under time, memory, and reliability constraints. The architecture addresses this through four mechanisms.
\nWave 1 launches 8 independent attempts. Each snapshots its conversation history and dill-serialized Jupyter kernel state into a shared database. After 75 seconds, Wave 2 launches up to 8 reinforcement attempts. Each restores a first-wave snapshot - variables, functions, computed results - and continues from the branch point in a different direction.
\nA reinforcement attempt branching at turn 5 inherits 60+ seconds of prior computation. The standard community approach of independent flat attempts discards this progress. Kernel state restoration uses a 3-tier fallback for robustness.
\nThe system provides escalating time-awareness nudges (gentle, warning, urgent) but does not force truncation. Context compression presents a dual choice: submit the answer or summarize for continued reasoning. The model has more information about its reasoning state than any fixed policy can encode.
\nThe function min(1, ratio²) adjusts resources based on remaining time per remaining problem, where ratio is the average budget ratio for remaining problems. At ratio 0.7, it allocates 49% vs 70% for linear. This reduces the number of solving attempts when time is tight. The rationale: if harder problems come first, the budget ratio drops below 1, and the solver allocates fewer attempts to increase throughput per attempt. Since the hard problems have already passed, easier problems arrive later and require fewer attempts regardless. This is an adaptive optimization dynamic that self-corrects regardless of problem ordering.
Three-factor scoring replaces majority voting: reasoning effort, code execution effectiveness, and baseline vote contribution. Independent convergence from unrelated branches adds a tiebreaker. On hard problems where the correct answer appears in a minority of attempts, majority voting selects against it. Effort-weighted scoring mitigates this.
\nEight categories of defensive mechanisms: VRAM management, process isolation, timeout hierarchy (6 nested levels), thread safety, graceful degradation, budget safety, sandbox management, and environment propagation. In blind deployment, resilience determines whether a 45-problem solution scores 45 or crashes and scores 30.
\nFull parameter table (~67 parameters), resilience inventory, and scoring formulas in the PDF (§4, Appendices A-D, H).
\nThe writeup evaluation criteria include model contribution. This section addresses that criterion.
\nThe fine-tuning constraint on GPT-120B is well-defined. The MoE expert weights are the natural target for behavioral modification. Four obstacles prevent standard approaches: GPT-OSS fuses experts into packed tensors (LoRA cannot work on them), full-model SFT requires multi-GPU compute, heavy CoT SFT risks overwriting the OODA-like reasoning mode, and intermediate layer modifications degrade Eagle3 throughput.
\nMost teams focused on prompt engineering. We developed an alternative approach.
\nSplit the model into HEAD (all layers except the last, frozen MXFP4, runs at inference speed through vLLM) and TAIL (last layer in bf16, trainable). The speculators library (vllm-project) extracts hidden states from HEAD. TAIL trains on those states with cross-entropy loss. No architecture change. Full weight access to attention, MoE experts, and router in the trainable layer.
\nOne RTX Pro 6000 on Google Colab. Under 2 hours. ~2,500 mixed samples (pattern recognition capability training and anti-forgetting samples).
\nAll Eagle3-dependent layers remain untouched. The goal is adding pattern recognition capacity orthogonal to existing knowledge, not shifting the output distribution. Training was stopped when harder reference problems showed longer reasoning traces while easier problems showed no change. Minimal drift was the target.
\nThe method supports a bank of domain-specific last-layer variants stored individually and merged at deployment. Hot-swapping via in-engine weight reload would eliminate the merge step but requires vLLM support not yet available.
\nMethod details and pipeline diagram in the PDF (§4.8, Appendix L, Figure 4).
\n| Configuration | \nPublic LB | \n
|---|---|
| Simple prompt, base GPT-120B | \n39-41 | \n
| Structured prompt, base GPT-120B | \n36 | \n
| Structured prompt + technique hints, SFT GPT-120B | \n36 | \n
Three factors contributed to the score reduction.
\nFirst, domain technique hints on the base model triggered a cascade: the model applied advanced techniques to problems that did not require them, producing longer traces. Adaptive scaling then reduced resources for subsequent problems.
\nSecond, the SFT submission retained technique hints as a deliberate experiment. The hypothesis: SFT-trained pattern recognition may enable selective hint use that the base model cannot manage. Whether this holds on the private distribution is an open question.
\nThird, time spent on the OODA/PDCA analysis and Neural Boosting SFT development reduced the time available for parameter tuning and submission iteration.
\nWe submitted the third one (36 Public LB) as an experiment of several hypotheses on private leaderboard rather than using the already stable 41 with simple prompt.
\nDetailed analysis in the PDF (§3.3, §5).
\nThree patterns from this competition may generalize.
\nThe AIMO-2 team data shows a 10-15 point gap between pass@64 and maj@64 (Moshkov et al., 2025, Figure 2). On hard problems, majority voting selects against the correct answer. Effort-weighted scoring is one mitigation. Inter-attempt communication (solver-verifier debate, as prototyped in our Blackboard notebook) may be another, though synchronization risks prevented deployment in this competition.
\nBenchmark scores predicted AIMO performance poorly. GPT-120B outscored models with higher standalone benchmarks. Behavioral criteria - how a model reasons when computation is available - may be a more useful selection signal than benchmark rankings for TIR-enabled applications.
\nCompetition problems test monologue-style solving. If commercial models have reached multi-model routing and internal debate capabilities, the actual commercial vs open-source gap on tasks requiring emergent complexity may be substantially larger than competition formats can measure.
\n50 pages. 6 figures. 28 references. 14 appendices.
\nDownload: https://doi.org/10.5281/zenodo.19695613
\nThanks to the AIMO Prize organizers, XTX Markets, and Kaggle. Special thanks to Simon Frieder for the evaluation framework design.
\nCommunity contributors: nihilisticneuralnet, andreasbis, ryanholbrook, friederrr, wenliang1990 (Eagle3 draft model), and others who shared work through discussion posts and notebooks.
", "rawMarkdown": "*This writeup was originally published as a Project writeup on April 22, 2026. It has been migrated here to properly link it to this competition, as Kaggle currently does not provide an option to re-link an existing writeup to a competition.*\n\nhttps://www.kaggle.com/writeups/khoinguyennguyen/adaptive-tree-of-thoughts-with-eagle3-accelerated\n\n# Adaptive Tree-of-Thoughts with Eagle3-Accelerated GPT-120B\n\n## Scores\n\n| Submission | Public LB | Private LB |\n|------------|-----------|------------|\n| Base GPT-120B, simple prompt (intermediate) | 39--41 (2× hit 41) | -- |\n| SFT GPT-120B, structured prompt + technique hints | 36 | 38.5|\n\nThis writeup is based on the SFT submission to the private leaderboard. Private scores will be added during the rebuttal phase.\n(Updated: The Private LB score is 38.5, while being higher than Public LB score 36, the structure prompt seems to be penalizing as it limits the solving attempts to more correlated and concentrated set of approaches and reduces diversity)\n## Reproducibility\n\n| Resource | Link |\n|----------|------|\n| **Complete write-up** (50pp, 6 figures, 14 appendices) | [Zenodo DOI: 10.5281/zenodo.19695613](https://doi.org/10.5281/zenodo.19695613) |\n| Final submission notebook (SFT) | [Kaggle](https://www.kaggle.com/code/khoinguyennguyen/adaptive-tree-of-thoughts-with-gpt-120b-eagle3) |\n| Base stable notebook | [Kaggle](https://www.kaggle.com/code/khoinguyennguyen/eagle3-tot-super-original-stable-top-p) |\n| Eagle3 speculative decoding notebook | [Kaggle](https://www.kaggle.com/code/khoinguyennguyen/eagle3-specdecoding-retry) |\n| Blackboard multi-agent solver | [Kaggle](https://www.kaggle.com/code/khoinguyennguyen/aimo3-blackboard-cross-model-multi-agent-solver) |\n| Cross-model hard problem benchmarks | [Kaggle](https://www.kaggle.com/code/khoinguyennguyen/aimo3-cross-model-hard-problem-benchmarks) |\n| SFT model weights | [HuggingFace](https://huggingface.co/khoinguyenbk/gpt-120b-last-layer-sft-pure-strategy-001-merged-mxfp4) |\n\nThe complete write-up PDF covers contextual analysis, hypotheses, solution landscape, architectural rationale, all 67 parameters, scoring formulas, and prompt text. This post is a summary. The PDF is the document of record.\n\n**Complete write-up: [https://doi.org/10.5281/zenodo.19695613](https://doi.org/10.5281/zenodo.19695613)**\n\n**Final notebook - Google Colab Version** [COLAB](https://colab.research.google.com/drive/1iAV0iPKLKBKqljG6TmoTa3FQBL9-YmBv#scrollTo=GbfFLqW2SgCF) (Ready To Run)\n\n---\n\n## Summary\n\nThis submission uses GPT-120B with Eagle3 speculative decoding inside an adaptive Tree-of-Thoughts architecture. The system runs 8 to 16 parallel attempts per problem. Reinforcement-wave attempts inherit full Jupyter kernel state from first-wave snapshots. A quadratic scaling function manages resources across 50 problems. Answer selection uses effort-weighted scoring. The base model scores 39-41 on the public leaderboard.\n\nFor the final submission, we applied Neural Boosting SFT to fine-tune the last transformer layer's MoE expert weights on a single consumer GPU. This is submitted as an experiment on the private distribution.\n\n---\n\n## Analysis of the AIMO-2 gap\n\nWe began by examining publicly available data from the AIMO-2 evaluations (AIMO Prize, 2025).\n\nAIMO-2 winners (NemoSkills, NVIDIA) deployed a 14B dense model on 4×L4 GPUs, 96GB total VRAM, and scored 34/50. The commercial model o3-preview scored 47/50 on the same problems. The organizers subsequently re-ran the winning models on 8×H100 GPUs (640GB total) with all time constraints removed. NemoSkills scored 35. imagination-research scored 35.\n\nThe infrastructure upgrade from 96GB to 640GB produced a +1 point improvement. This indicates the bottleneck is not hardware or time budget.\n\nThree data points from the GPT model family are relevant. o3-preview scored 47/50 on AIMO-2. GPT-120B achieves pass@20 of 47.56 on AIMO-3 public. GPT-20B reached 30+ in several public notebooks including ours. These models share a training lineage and consistently outperform dense models of comparable scale on competition math.\n\nTwo qualifications are necessary. First, GPT-OSS inherits training methodology from OpenAI's pipeline. Its performance does not establish that open-source models generally are approaching commercial capability. A controlled comparison would require testing Qwen-3.5-122B and Nemotron-120B-Super under equivalent TIR conditions, which was not done. Second, competition math tests a specific reasoning regime. The actual commercial vs open-source gap on harder cognitive tasks may be significantly larger than these scores suggest.\n\nThis analysis informed our primary design decision: to invest in extraction architecture rather than model modification. The model already produces correct answers at sufficient rate. The engineering problem is capturing them.\n\nFull analysis in the PDF (§1.2, §2.1).\n\n---\n\n## Observed reasoning behavior\n\nWe compared GPT-120B reasoning traces against Qwen-series models on competition problems with TIR enabled. A consistent behavioral pattern emerged.\n\nGPT-120B reads a problem and immediately computes small cases through code. It observes patterns in the output, generalizes, and iterates. Code functions as the primary reasoning medium.\n\nNon-GPT models read a problem and deliberate in text first. They form a structured plan, write code to verify it, then revise. Code functions as a verification step after text-based reasoning.\n\nWe mapped this to two established frameworks: Boyd's OODA loop (Observe-Orient-Decide-Act) for the GPT pattern, and Deming's PDCA cycle (Plan-Do-Check-Act) for the non-GPT pattern. This is an observational characterization used for engineering decision-making. It is not a mechanistic claim.\n\nThe engineering implication is specific: structured prompts that impose planning should add overhead on an OODA-mode model without changing its underlying behavior. We tested this. The simpler prompt scored 39-41. The structured prompt scored 36. The result is consistent with the observation, though other explanations are possible.\n\nThree origin hypotheses are discussed in the PDF (§2.2). None are verified.\n\n---\n\n## Architecture\n\nThe organizer pass@20 evaluation shows GPT-120B reaches 47.56/50 with 20 diverse attempts. The capability exists. The problem is extraction under time, memory, and reliability constraints. The architecture addresses this through four mechanisms.\n\n### Kernel state inheritance\n\nWave 1 launches 8 independent attempts. Each snapshots its conversation history and dill-serialized Jupyter kernel state into a shared database. After 75 seconds, Wave 2 launches up to 8 reinforcement attempts. Each restores a first-wave snapshot - variables, functions, computed results - and continues from the branch point in a different direction.\n\nA reinforcement attempt branching at turn 5 inherits 60+ seconds of prior computation. The standard community approach of independent flat attempts discards this progress. Kernel state restoration uses a 3-tier fallback for robustness.\n\n### Time management without hard cutoffs\n\nThe system provides escalating time-awareness nudges (gentle, warning, urgent) but does not force truncation. Context compression presents a dual choice: submit the answer or summarize for continued reasoning. The model has more information about its reasoning state than any fixed policy can encode.\n\n### Quadratic resource scaling\n\nThe function `min(1, ratio²)` adjusts resources based on remaining time per remaining problem, where ratio is the average budget ratio for remaining problems. At ratio 0.7, it allocates 49% vs 70% for linear. This reduces the number of solving attempts when time is tight. The rationale: if harder problems come first, the budget ratio drops below 1, and the solver allocates fewer attempts to increase throughput per attempt. Since the hard problems have already passed, easier problems arrive later and require fewer attempts regardless. This is an adaptive optimization dynamic that self-corrects regardless of problem ordering.\n\n### Effort-weighted answer selection\n\nThree-factor scoring replaces majority voting: reasoning effort, code execution effectiveness, and baseline vote contribution. Independent convergence from unrelated branches adds a tiebreaker. On hard problems where the correct answer appears in a minority of attempts, majority voting selects against it. Effort-weighted scoring mitigates this.\n\n### Resilience\n\nEight categories of defensive mechanisms: VRAM management, process isolation, timeout hierarchy (6 nested levels), thread safety, graceful degradation, budget safety, sandbox management, and environment propagation. In blind deployment, resilience determines whether a 45-problem solution scores 45 or crashes and scores 30.\n\nFull parameter table (~67 parameters), resilience inventory, and scoring formulas in the PDF (§4, Appendices A-D, H).\n\n---\n\n## Model contribution: fine-tuning GPT-OSS experts on a single GPU\n\nThe writeup evaluation criteria include model contribution. This section addresses that criterion.\n\nThe fine-tuning constraint on GPT-120B is well-defined. The MoE expert weights are the natural target for behavioral modification. Four obstacles prevent standard approaches: GPT-OSS fuses experts into packed tensors (LoRA cannot work on them), full-model SFT requires multi-GPU compute, heavy CoT SFT risks overwriting the OODA-like reasoning mode, and intermediate layer modifications degrade Eagle3 throughput.\n\nMost teams focused on prompt engineering. We developed an alternative approach.\n\n### Method\n\nSplit the model into HEAD (all layers except the last, frozen MXFP4, runs at inference speed through vLLM) and TAIL (last layer in bf16, trainable). The speculators library (vllm-project) extracts hidden states from HEAD. TAIL trains on those states with cross-entropy loss. No architecture change. Full weight access to attention, MoE experts, and router in the trainable layer.\n\n### Cost (Chosen version)\n\nOne RTX Pro 6000 on Google Colab. Under 2 hours. ~2,500 mixed samples (pattern recognition capability training and anti-forgetting samples).\n\n### Design rationale\n\nAll Eagle3-dependent layers remain untouched. The goal is adding pattern recognition capacity orthogonal to existing knowledge, not shifting the output distribution. Training was stopped when harder reference problems showed longer reasoning traces while easier problems showed no change. Minimal drift was the target.\n\n### Broader applicability\n\nThe method supports a bank of domain-specific last-layer variants stored individually and merged at deployment. Hot-swapping via in-engine weight reload would eliminate the merge step but requires vLLM support not yet available.\n\nMethod details and pipeline diagram in the PDF (§4.8, Appendix L, Figure 4).\n\n---\n\n## Score analysis: 41 to 36\n\n| Configuration | Public LB |\n|---------------|-----------|\n| Simple prompt, base GPT-120B | 39-41 |\n| Structured prompt, base GPT-120B | 36 |\n| Structured prompt + technique hints, SFT GPT-120B | 36 |\n\nThree factors contributed to the score reduction.\n\nFirst, domain technique hints on the base model triggered a cascade: the model applied advanced techniques to problems that did not require them, producing longer traces. Adaptive scaling then reduced resources for subsequent problems.\n\nSecond, the SFT submission retained technique hints as a deliberate experiment. The hypothesis: SFT-trained pattern recognition may enable selective hint use that the base model cannot manage. Whether this holds on the private distribution is an open question.\n\nThird, time spent on the OODA/PDCA analysis and Neural Boosting SFT development reduced the time available for parameter tuning and submission iteration.\n\nWe submitted the third one (36 Public LB) as an experiment of several hypotheses on private leaderboard rather than using the already stable 41 with simple prompt.\n\nDetailed analysis in the PDF (§3.3, §5).\n\n---\n\n## Observations for future work\n\nThree patterns from this competition may generalize.\n\nThe AIMO-2 team data shows a 10-15 point gap between pass@64 and maj@64 (Moshkov et al., 2025, Figure 2). On hard problems, majority voting selects against the correct answer. Effort-weighted scoring is one mitigation. Inter-attempt communication (solver-verifier debate, as prototyped in our Blackboard notebook) may be another, though synchronization risks prevented deployment in this competition.\n\nBenchmark scores predicted AIMO performance poorly. GPT-120B outscored models with higher standalone benchmarks. Behavioral criteria - how a model reasons when computation is available - may be a more useful selection signal than benchmark rankings for TIR-enabled applications.\n\nCompetition problems test monologue-style solving. If commercial models have reached multi-model routing and internal debate capabilities, the actual commercial vs open-source gap on tasks requiring emergent complexity may be substantially larger than competition formats can measure.\n\n---\n\n## Complete write-up\n\n50 pages. 6 figures. 28 references. 14 appendices.\n\n**Download: [https://doi.org/10.5281/zenodo.19695613](https://doi.org/10.5281/zenodo.19695613)**\n\n---\n\n## Acknowledgments\n\nThanks to the AIMO Prize organizers, XTX Markets, and Kaggle. Special thanks to Simon Frieder for the evaluation framework design.\n\nCommunity contributors: nihilisticneuralnet, andreasbis, ryanholbrook, friederrr, wenliang1990 (Eagle3 draft model), and others who shared work through discussion posts and notebooks.", "votes": 3 } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T04:25:58Z", "competition_id": 118448, "competition_slug": "ai-mathematical-olympiad-progress-prize-3", "team_id": 14837208, "solution_url": "/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/adaptive-treeofthoughts-with-eagle3-accelerated-mg" }