![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2982177%2Fb65b8e1b242f7fe4389db7bc51539bdb%2Ffigure01_variance.png?generation=1779031464367902&alt=media\")
\nFigure 1: Competition score variance: Base Model (v12, six runs) vs SFT Model (v08a, four runs) on the public leaderboard.Submission basis: Public LB score: 40.0/50 (average over 4 runs). Private LB score: 42.0/50 (average over 2 runs)
\nResources:
\nOur solution for AIMO Progress Prize 3 competition fine-tunes attention layers on 1,788 self-generated Tool-Integrated Reasoning (TIR) traces with difficulty-balanced sampling. The core idea is promoting behavioral regularization through supervised fine-tuning (SFT) on GPT-OSS-120B, rather than teaching this Mixture of Experts (MoE) model new capabilities. GPT-OSS-120B is already capable of solving most of the questions in the competition, albeit not reliably. In our approach, fine-tuning promotes correct solution traces that were already generated by GPT-OSS-120B. As a result, the model doesn't learn new solution methods, but converges more reliably on correct ones it already knows. This manifests as significantly reduced run-to-run variance (SD 1.2 for SFT model vs 2.0 for the base model, both on the public data set) and slightly higher mean (40.0 for SFT vs 39.8 for the base model). Our model scored 42.0/50 on the private data set (averaged over two runs), indicating that our SFT efforts did not cause overfitting. Beyond variance reduction, we measure concrete behavioral changes at inference: the SFT model uses ~18% less compute and ~17% fewer Python tool calls while reaching the same answers on the same problems (Section 9.2), consistent with regularization tightening reasoning trajectories rather than expanding capability. We support our findings with systematic ablation studies across data composition, learning rate scaling, training context length, DPO, verification strategies, and prompt engineering; most of which produced regressions. We also document 10 iterations of post-vote verification approaches and their impact on performance. We believe SFT can be an efficient tool for guiding large MoE models when solving complex questions and our approach demonstrates an interesting use case of SFT on GPT-OSS-120B.
\nGPT-OSS-120B [1] is a 120B-parameter Mixture-of-Experts (MoE) model with only 5.1B parameters active per token (128 experts, 4 active). In Microscaling FP4 (MXFP4) quantization it occupies ~65GB, fitting a single H100 with room for 108K tokens of key-value (KV) cache. It was trained with native tool-use and so the model knows how to call Python, read execution results, and iterate. These capabilities make the model uniquely suited for tool-integrated reasoning (TIR), which is essential for competition-level math where brute-force computation and symbolic verification complement mathematical reasoning. Not surprisingly, GPT-OSS-120B has been the most popular model among AIMO 3 participants. It is arguably the most capable model for solving olympiad-level questions that can fit in the 80GB VRAM of H100, while leaving room for KV cache across multiple parallel attempts.
\nOne of the apparent challenges of fine-tuning a model such as GPT-OSS-120B is that the model is already highly capable in mathematical reasoning. The original training likely included a much larger and more diverse dataset than what we can practically use for fine-tuning, leaving limited room for improvement under small fine-tuning budgets. In fact, many AIMO 3 participants reported diminished performance, or no change, as a result of fine-tuning.
\nWith these observations in mind, we took a different approach to SFT. Rather than attempting to teach the model new solution methods, we used SFT to regularize its behavior, narrowing the distribution of reasoning strategies toward ones that actually work. In our workflow, we first designed an inference pipeline using the base model. Then we ran the base model on 1,200 olympiad-level problems using the exact same inference pipeline (8 parallel attempts per problem at T=1.0), keeping full traces generated by the model. We used the fraction of correct attempts as a proxy for problem difficulty, e.g., 8/8 correct would be an easy problem, and 1/8 would be a hard problem. We kept only the correct traces as training data. In this way, the training set consists entirely of what the model organically produces when it succeeds. The goal is not to expand the model's capabilities, but to increase the odds that on hard problems (where perhaps only 2 or 3 out of 8 base-model attempts find the correct answer) the fine-tuned model will find it more reliably, pushing that ratio to 4 or 5 out of 8.
\nThe base model has high variance (SD 2.0, range 37–43 across 6 competition runs), and fine-tuning's primary effect is reducing that variance. This matters for the competition scoring formula for the private data set, which averages two independent runs: a model with mean 40 and SD 1 outperforms a model with mean 40 and SD 2, because a lower variance ensures the model is much less likely to have a bad run that drags down the average and hence the final score on the private data set. On the private data set the SFT model averaged 42.0/50 over two runs, slightly above the 40.0 public-LB mean, indicating the SFT model does not overfit to the public data set. Beyond run-to-run variance, the SFT model also shows concrete behavioral changes at inference time (~18% less total compute and ~17% fewer Python tool calls on the same problems while reaching the same answers, Section 9.2), supporting the regularization interpretation over a capability-improvement interpretation.
\nIn this paper we describe the components and design considerations for the eventual model we submitted for the private leaderboard scoring. We also designed several experiments to determine optimality of this solution, which will be discussed in more detail in the \"Ablation Studies\" section. Details on how to replicate results both on Kaggle and elsewhere (e.g. Colab, RunPod) are available on HuggingFace [link above].
\nThe competition offered daily attempts to solve the \"public data set\", consisting of 50 problems. The final evaluation was on a \"private data set\" with a different set of 50 problems. The competition hosts characterized the difficulty distribution of both sets using state-of-the-art open-source models [9]. Based on their analysis, approximately 27 problems can be solved in a single pass (pass@1), about 38 can be solved within 3 passes, and roughly 42 within 5 passes. This leaves approximately 8 problems that require significantly more attempts or are beyond the models' reliable reach.
\nFor GPT-OSS-120B specifically, we estimate that approximately 35 problems are reliably solvable (correct on most runs), while about 7-9 problems have varying degrees of success: sometimes correct, sometimes not, depending on sampling luck. This estimate is based on our competition runs, where we observed a maximum score of 43 and a minimum of 37 across 6 submissions with the base model (no fine-tuning). Our inference pipeline solves each problem with 8 parallel attempts at T=1.0 and selects the answer by entropy-weighted majority vote (detailed in Section 3).
\n| Model | \nCompetition runs | \nMean | \nSD | \n
|---|---|---|---|
| Base model (GPT-OSS-120B) | \n39, 39, 40, 43, 37, 41 | \n39.8 | \n2.0 | \n
This variance originates at the problem level. For easy problems, all 8 attempts agree; these contribute a stable +1 to the score every run. For hard problems where only 2 or 3 out of 8 attempts find the correct answer, the majority vote can go either way depending on sampling luck. A problem at 3/8 baseline accuracy flips between correct and incorrect across runs. When several such problems flip simultaneously, the total score can swing by 4–6 points. We attribute this variance to GPT-OSS-120B's intrinsic sampling at T=1.0, not to artifacts of the evaluation procedure, such as time budget per problem or randomized ordering of problems. We discuss these two procedural factors in more detail in Section 3.
\nThe variance makes it challenging to evaluate changes from single competition runs. A score of 36 (2 SD below mean) strongly suggests regression, but distinguishing 38 from 40 is within noise. We therefore could not rely solely on the public leaderboard for model development and needed a more controlled evaluation methodology.
\nThis problem-level variance is precisely the mechanism our SFT targets: by pushing hard problems from 2-3/8 to 4-5/8 correct attempts, we move them from \"unreliably correct\" to \"reliably correct,\" reducing the number of problems that contribute to run-to-run variance. Figure 1 previews the achieved reduction on the public leaderboard: the fine-tuned model (v08a) reduces the SD by approximately 40% (2.0 → 1.2) while leaving the mean essentially unchanged (39.8 → 40.0). The training and inference setup behind v08a is detailed in Sections 4 and 5; the full per-experiment results matrix is in Section 7.
\n\nFigure 1: Competition score variance: Base Model (v12, six runs) vs SFT Model (v08a, four runs) on the public leaderboard.
Given the high variance on the public leaderboard and the one-submission-per-day limitation, we constructed a separate 50-problem evaluation set for controlled model comparisons. We sourced problems from the PolyMath dataset [2], a curated collection of 11,000 high-difficulty problems submitted to the AIMO 3 Math Corpus Prize by Dekoninck, Hiss, Petrov, Rögnvaldsson, and Sun. PolyMath was particularly useful because it provides oss_20b_medium_pass_rate: the pass rate of GPT-OSS-20B (a 20B-parameter model from the same model family) evaluated under the PolyMath team's \"medium\" inference configuration. The values take discrete levels (0, 0.5, 1.0), giving us a coarse difficulty calibration directly relevant to the GPT-OSS model family. Note that this PolyMath-provided difficulty label is distinct from our own difficulty classification (Section 4.3), which uses the 120B competition model's success rate across 8 attempts.
Our validation set was designed to stress-test the model at its difficulty frontier:
\noss_20b_med_pass ≈ 0.5) and 26 hard problems (oss_20b_med_pass = 0)This validation set (referred to as \"PolyMath-Val-50\" throughout) is distinct from the competition's 50-problem test set. It served as our primary metric for comparing LoRA variants. Across 4 runs of the base model using slightly different code versions, the validation score was consistently 32/50, indicating that this set produces stable, reproducible measurements when the model itself is unchanged. This stability, compared to the ±3 point swings on the public leaderboard, made PolyMath-Val-50 essential for experiment design: even 1-point differences can be meaningful here, whereas on the competition set they are noise.
\nIn addition to Kaggle, we ran evaluations on RunPod H100s for experiments that exceeded Kaggle's compute budget (e.g., multi-seed replications, Unsloth training + evaluation). We observed a consistent 1-2 point offset: models that score 32/50 on Kaggle score 30-31/50 on RunPod. This is likely due to slower inference throughput on RunPod (~1.5x), which means each attempt generates fewer tokens within the same time budget. All RunPod comparisons in this work are internally consistent: the offset cancels when comparing models within the same environment.
\n| Model | \nKaggle Val-50 | \nRunPod Val-50 | \n
|---|---|---|
| Base model (v12) | \n32/50 (×4 runs) | \n31, 30 | \n
| SFT model (v08a) | \n32/50 | \n30, 30 | \n
Our inference code is adapted from a public notebook by Andreas Bisiadis [6] (public LB 42/50), which provided the core TIR + entropy-weighted voting architecture, and from a variant by nihilisticneuralnet titled \"Let Me (over)COOK!!\" [11] (public LB 44/50), which was built on Andreas's version and contributed the more verbose system prompt that we adopted in our final configuration. We iterated through 27 versions of this pipeline during the competition (v01–v27), extensively testing modifications at each step (detailed ablation results in Section 6.1). None improved on our base configuration, giving us confidence that it was the right baseline for both data generation and final deployment.
\nThe key inference parameters and architectural decisions:
\n| Parameter | \nValue | \n
|---|---|
| Temperature | \n1.0 | \n
| min_p | \n0.02 | \n
| Attempts per problem | \n8 | \n
| Max context | \n65,536 tokens | \n
| Max generation per turn | \n2,048 tokens | \n
| Top logprobs | \n5 | \n
| Jupyter timeout | \n6 seconds | \n
| Max TIR turns | \n128 | \n
| Early stop | \n4/8 consensus | \n
| GPU memory utilization | \n0.96 | \n
1. Completions API with openai_harmony: GPT-OSS was trained with a specific message encoding protocol. The openai_harmony library encodes messages as raw token IDs with channel routing (<|channel|>python, <|channel|>final) and tool namespaces. Using the native Completions API is essential: our first integration used the Chat Completions API and scored only 13/50 vs 41/50 with the native protocol.
2. TIR with persistent Jupyter kernels: each attempt runs a multi-turn loop (up to 128 turns). The model generates tokens, we parse for recipient=='python', execute code in a sandboxed Jupyter kernel, feed the result back, and let the model continue. We maintain a pool of 16 persistent kernels pre-initialized with numpy/sympy/mpmath.
# Core TIR loop (simplified)\nfor turn in range(128):\n response = client.completions.create(\n model=model_name, prompt=token_ids,\n max_tokens=2048, temperature=1.0, logprobs=5,\n stop=stop_token_ids)\n messages = parse_messages_from_completion_tokens(response)\n for msg in messages:\n if msg.recipient == \"python\":\n result = kernel_pool.execute(msg.content, timeout=6)\n token_ids += encode_tool_result(result)\n elif msg.recipient == \"final\":\n answer = parse_boxed_answer(msg.content)\n if answer is not None:\n return answer, compute_entropy(response), token_ids\n3. Entropy-weighted majority voting: for each problem, 8 parallel attempts at T=1.0, answer selected by weighted majority vote where each attempt's weight is 1 / (1 + entropy). Entropy is computed from top-5 logprobs at the final answer token. We tested raw majority voting (no entropy weighting); it scored 36/50, roughly a 3-point drop from the entropy-weighted version on the same ablation.
4. Time management: each problem gets a budget of min(300 + available_bank, 900) seconds, with early stop at 4/8 consensus. In other words, each question is guaranteed to have 300 seconds minimum, and can be offered up to 900 seconds if extra time is available. The time bank accumulates unused time from fast problems. In practice, the bank rarely constrains: no problem used more than 744s of its 900s budget on PolyMath-Val-50, and the bank grew to ~5,000s unused. Likewise, we ran the inference model without time budgeting other than setting a maximum time of 900 in the public data set and the computation was completed within 5 hours. We are including it as insurance against a potentially harder private test set, which may exceed the time limit.
5. Procedural variance controls (problem ordering and time-budget binding): Returning to the claim in Section 2.1 that competition-run variance is intrinsic to GPT-OSS-120B sampling rather than an artifact of evaluation procedure, we verified two procedural factors directly from the run logs. Problem ordering: Problems are processed in effectively randomized order across runs. On three PolyMath-Val-50 runs (v08a, v11a, v11d) the processed sequences differed across runs and were mixed across difficulty type and oss_20b_med_pass value. Time-budget binding: on the same three runs, the maximum time used by any single problem was 695s, 715s, and 747s respectively, against the 900s minimum budget; the bank-grown budget for slow problems reached 1100-1800s, and this extra headroom was never used. The same inference code and time-management policy applied to public-leaderboard submissions. We therefore conclude that neither problem ordering nor time-clipping contributes to the 37-43 spread reported in Section 2.1, and the spread is mainly due to the intrinsic variance in GPT-OSS-120B chain-of-thought.
This section describes the core idea behind our fine-tuning approach. The mechanical details of LoRA configuration, learning rate calibration, and the training platform comparison are in Section 5.
\nA natural starting point for SFT is to use existing mathematical datasets. Our first attempt used NuminaMath-TIR [3], which contains thousands of math problems with solutions. The result was catastrophic: SFT on NuminaMath completely suppressed code generation, with 0 out of 400 attempts using Python after training. The text-heavy NuminaMath solution format overwrote the model's TIR behavior, replacing iterative code-based reasoning with single-pass text answers. This taught us a critical lesson: the training data must match the inference format exactly. The lesson is about format compatibility, not source strength: NuminaMath-TIR traces are generated by GPT-4, a strong model on math benchmarks, yet using them for SFT destroyed our model's TIR behavior. What matters is that the training data structurally matches the format the model will produce at inference time.
\nSelf-generated traces solve this problem by construction. We run the base model using the same inference pipeline it will use at competition time (the same openai_harmony encoding, the same Jupyter kernels, the same tool-call routing) and capture the full conversation traces. The correct traces (reasoning + code + execution output + \\boxed{} answer) become our training data. The model is trained on its own successful behavior, not on an external format it was never designed to produce.
We selected 1,200 problems from the PolyMath dataset [2] for trace generation:
\noss_20b_medium_pass_rate (the 20B model's pass rate, see Section 2.2): 1,125 problems with pass rate 0.5 (solvable by the 20B model roughly half the time) and 75 with pass rate 0 (unsolvable by the 20B model but potentially solvable by our 120B model)![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2982177%2Fdaca97b30fc38f97f673795e1b3f0f8e%2Ffigure02_data_pipeline.png?generation=1779031473549272&alt=media\")
\nFigure 2: Data generation pipeline
Data generation ran on Kaggle H100 (200 problems) and RunPod H100 (1,000 problems). We validated equivalence by comparing results on 4 shared problems: RunPod matched or beat Kaggle (4/4 vs 3/4 correct).
\nDifficulty-bucket sampling was the most impactful design decision. Since easy problems produce 8/8 correct traces and hard problems produce only 1–2/8, using all correct traces without resampling creates a training set naturally dominated by easy-problem reasoning patterns, not because we chose to emphasize easy problems, but simply because they generate more correct traces per problem.
\nIn our first LoRA attempt (SFT v03), 59% of training examples came from 8/8-easy problems (Figure 3, left). This resulted in a score of 25/50 on PolyMath-Val-50, a 7-point regression from the base model's 32/50. The model learned to be more decisive on problems it already solved reliably, while becoming less effective on harder problems.
\nOur balanced sampling strategy downsamples easy problems and upsamples hard ones:
\n| Difficulty bucket | \nProblems | \nSampling strategy | \nTraining share | \n
|---|---|---|---|
| 8/8 (easy) | \n~350 | \n1 trace each | \n~27% | \n
| 7/8 | \n~150 | \n2 traces each | \n~14% | \n
| 5-6/8 (medium) | \n~280 | \nall traces | \n~33% | \n
| 1-4/8 (hard) | \n~260 | \nall traces × 2 | \n~26% | \n
Within the 8/8 and 7/8 buckets where we downsample, we keep the shortest correct traces per problem rather than sampling randomly. Shorter correct traces tend to represent more compact reasoning paths with less rambling, consistent with the broader regularization toward efficient correct strategies. The 5-6/8 and 1-4/8 buckets keep all correct traces (with 1-4/8 duplicated 2×), so no within-bucket selection is applied there.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2982177%2F41effb2210fb81726bfa3e69dc87361e%2Ffigure03_data_balance.png?generation=1779031483621901&alt=media\")
\nFigure 3: Training data composition: unbalanced vs balanced sampling
The balanced version (v04, 430 examples) immediately jumped to 30/50. With the full dataset and proper balancing (v08a, 1788 examples), we reached 32/50 on PolyMath-Val-50 and 40.0 mean on competition (4 runs, SD 1.2).
\nThe controlled data composition experiment (Section 5.3) confirms that both easy and hard traces are necessary: removing hard traces is worst (30/50), removing easy traces costs 1 point (31/50), and overweighting hard traces beyond 2× provides no further benefit (31/50). The asymmetry in the bucket weights is intentional and follows from the thesis: regularization should pull the model toward conditions where it usually succeeds, so no single bucket is allowed to dominate (no bucket trains more than ~33%) and the hard bucket is not over-emphasized. Over-weighting the 1-4/8 bucket past 2× would push the model to imitate its rare-success patterns at the expense of its typical-success patterns.
\nThe thesis that SFT acts as behavioral regularization rather than capability transfer is supported by several lines of evidence:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2982177%2F7f09c6dc7c0d92698bd54c9af176fdd1%2Ffigure04_temperature.png?generation=1779031490764506&alt=media\")
\nFigure 4: Temperature reduction vs SFT as two routes to lower variance. Both T=0.8 and SFT reach SD 1.2, but T=0.8 loses 2.5 points on the mean (37.3 vs 39.8) while SFT raises it to 40.0.
The 9 \"wobbly\" problems (those that differentiate model variants) sit at the boundary where the base model has a ~50% chance of finding the correct approach through 8 attempts. SFT pushes these past the threshold. The wobbly score predicts the total: each additional wobbly problem solved adds one point to the score (e.g., 27 + 2 wobbly = 29/50, 27 + 5 = 32/50).
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2982177%2Fe98468424a629ad9a8eedb33246496cc%2Ffigure05_problem_stability_kaggle.png?generation=1779031499899151&alt=media\")
\nFigure 5: Problem stability across SFT variants on Kaggle. Problems are sorted along the y-axis by difficulty bucket (easy → hard) and within bucket by question type (Algebra, Combinatorics, Geometry, Number Theory) to make per-problem patterns legible. The actual processing order at inference time was effectively randomized across runs (Section 3, item 5).
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2982177%2Ff915c442ddabcc899dbbffa917e7361d%2Ffigure06_problem_stability_runpod.png?generation=1779031508670206&alt=media\")
\nFigure 6: Problem stability, RunPod multi-seed comparison. Same y-axis convention as Figure 5: sorted by difficulty and question type for visualization; the actual processing order at inference was randomized.
Attention-only LoRA: full-linear LoRA on a MoE model produces a 4.9 GB adapter incompatible with vLLM's [10] MoE LoRA support. Attention-only (q/k/v/o_proj) produces a 91 MB adapter that works. This targets 144 tensors: the shared attention layers that route information across all experts.
\n| Parameter | \nValue | \nWhy | \n
|---|---|---|
| Target modules | \nq/k/v/o_proj | \nOnly attention layers work with vLLM MoE; 144 tensors, 91 MB | \n
| Rank | \n32 | \nRank 32 solved Norwegian integers (PID 3994) where rank 16 couldn't; 31/50 vs 30/50 | \n
| Alpha | \n32 (scaling = 1.0) | \nStandard choice for rank = alpha | \n
| Epochs | \n1 | \n2 epochs overtrained (v05a), losing problems that 1 epoch retained | \n
| Batch size | \n8 | \n— | \n
| LR | \nScaled per data size | \nSee Section 5.2 | \n
When we doubled training data from 430 to 794 examples at the same LR (2e-4), performance dropped (v06: 30/50). Halving the LR restored it (v06b: 32/50). The total gradient movement (steps × LR) must stay roughly constant.
| Version | \nExamples | \nSteps | \nLR | \nsteps × LR | \nPolyMath-Val-50 | \n
|---|---|---|---|---|---|
| v05c | \n430 | \n51 | \n2e-4 | \n0.010 | \n31/50 | \n
| v06b | \n794 | \n100 | \n1e-4 | \n0.010 | \n32/50 | \n
| v08a | \n1788 | \n224 | \n4e-5 | \n0.009 | \n32/50 | \n
| v06 (too high) | \n794 | \n100 | \n2e-4 | \n0.020 | \n30/50 | \n
Our rule: keep steps × LR ≈ 0.01. This held across all successful runs.
With the full 1,071-problem dataset, we ran a controlled experiment varying data composition. All runs: rank 32, 1 epoch, batch 8, attention-only, LR scaled to keep steps × LR ≈ 0.01.
\n| Run | \nData change | \nExamples | \nLR | \nPolyMath-Val-50 | \n
|---|---|---|---|---|
| v08a | \nAll buckets, balanced | \n1788 | \n4e-5 | \n32/50 | \n
| v08b | \nRemove 8/8 (easy) bucket | \n1156 | \n6e-5 | \n31/50 | \n
| v08c | \nRemove 1-4/8 (hard) bucket | \n1402 | \n6e-5 | \n30/50 | \n
| v08d | \n4× hard upweight | \n2174 | \n3e-5 | \n31/50 | \n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2982177%2Fff321c320df3b253088286c128502b0e%2Ffigure07_data_composition.png?generation=1779031516987786&alt=media\")
\nFigure 7: Data composition ablation
Findings:
\nA distinctive aspect of our approach is the use of Tinker [4] (Thinking Machines' fine-tuning API for GPT-OSS) as our primary training platform. Tinker was a sponsor of the AIMO 3 competition, yet few participants reported success with fine-tuning through it. Our pipeline (train on Tinker, download adapter, merge on CPU, deploy via vLLM) proved reliable and cost-effective (~$13 per training run, ~$191 total).
\nTinker imposes a 32K context cap on training examples. Since inference runs at 65K context, this means some problems whose correct traces exceed 32K tokens are excluded from training. Critically, this filtering is not uniform across difficulty levels (Figure 8): at 32K, 57% of (5-6)/8 problems (70/123) and 39% of (1-4)/8 problems (48/124) are filtered out, compared to under 1% of (8)/8 problems. Harder problems produce longer traces (more computation steps, more error-correction loops) and are disproportionately lost to the context cap. Rather than treating this as a limitation, we explored whether longer context during training actually helps, using Unsloth [5] (QLoRA on RunPod H100) at 48K and 65K context.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2982177%2F72ac9ccaa87aa8b0209d4bb0cc22bc9e%2Ffigure08_context_filter.png?generation=1779031525900014&alt=media\")
\nFigure 8: Context window cap impact by difficulty bucket. The 32K cap (Tinker) disproportionately filters out the medium and hard buckets, whose correct traces tend to be longer.
| Version | \nContext | \nPlatform | \nData variant | \nExamples | \nRunPod Val-50 | \nKaggle Val-50 | \nCompetition | \n
|---|---|---|---|---|---|---|---|
| v12 (base) | \n— | \nno LoRA | \n— | \n— | \n31, 30 | \n32 (×4 runs) | \n39.8 mean (6 runs, SD 2.0) | \n
| SFT v08a | \n32K | \nTinker | \nbalanced | \n1,788 | \n30, 30 | \n32 | \n40.0 mean (4 runs, SD 1.2) | \n
| SFT v08a | \n32K | \nUnsloth | \nbalanced | \n1,788 | \n32 | \n— | \n— | \n
| SFT v11a | \n48K | \nUnsloth | \nbalanced | \n2,106 | \n— | \n33 | \n39.0 mean (2 runs, SD 2.83) | \n
| SFT v11c | \n65K | \nUnsloth | \n4× hard upsample | \n2,572 | \n— | \n33, 29 | \n— | \n
| SFT v11d | \n65K | \nUnsloth | \nv08a as-is | \n2,223 | \n31 | \n— | \n— | \n
| SFT v11e | \n65K | \nUnsloth | \nsmart upsample | \n2,400 | \n31 | \n— | \n— | \n
| SFT v11f | \n65K | \nUnsloth | \nno (1-4)/8 bucket | \n2,139 | \n— | \n31 | \n— | \n
Figure 9 visualizes the competition-data column of the table for the two main SFT candidates against the base model. The v08a / v11a comparison was load-bearing for our final-submission decision: v11a's single high run (41) is matched by a low run (37, below any v08a run), giving it a higher SD (2.83) over only 2 runs. v08a's tighter cluster across 4 runs made it the safer choice under the private-LB two-run averaging.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2982177%2Fbabba8c4fb358a5789e77f44ed730c5c%2Ffigure09_sft_variants.png?generation=1779031535893708&alt=media\")
\nFigure 9: SFT variants on competition data. v08a (Tinker 32K) clusters tightly at mean 40.0 (SD 1.2 over 4 runs), while v11a (Unsloth 48K) spans 37-41 (SD 2.83 over 2 runs). v11a's higher upside is offset by a worse downside, making v08a the safer selection under two-run private-LB averaging.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2982177%2F1225461091546d1ca5791dadd545d218%2Ffigure10_context_window.png?generation=1779031543955875&alt=media\")
\nFigure 10: Training context length vs performance
The results tell a nuanced story about context length and training platform.
\nLonger context introduces noise rather than helping. Four independent 65K variants (v11d, v11e, v11f, plus v11c repeat) all scored 31/50 or below on Kaggle. The single 48K run (v11a) scored 33/50, but when we repeated a 65K model (v11c), it swung from 33/50 to 29/50 on the same problems, revealing large run-to-run variance in the Unsloth-trained models.
\nThe platform/context confound is partially resolved. Our initial comparison confounded two variables: context length and training platform. Tinker uses full-precision LoRA on Thinking Machines' infrastructure; Unsloth uses 4-bit NF4 QLoRA with an SDPA attention patch on RunPod H100s. To isolate these factors, we trained an Unsloth model at 32K context with the same v08a data and learning rate.
\nThe result was striking: Unsloth at 32K scored 32/50 on RunPod, matching the Kaggle-level performance of the Tinker model, and 2 points above Tinker's own RunPod score (30/50). At the problem level, Unsloth 32K solved two problems (PID 480, PID 2002) that Tinker consistently missed on RunPod across both seeds. This suggests that the 32K context length, not the training platform, is the key factor in producing effective adapters. QLoRA at matched context length performs at least as well as full-precision LoRA, and possibly better.
\nThis was a late-stage discovery (post-competition). With only a single RunPod run for Unsloth 32K, we cannot make definitive claims; our variance analysis shows that single-run differences of 2 points can be noise. However, the fact that it matches Kaggle-level scores despite the consistent 1-2 point RunPod offset observed for all other models is suggestive. If this result holds across multiple seeds, it would imply that our competition submission (Tinker 32K) was a conservative but not optimal choice, and that Unsloth QLoRA at 32K context may offer both lower cost (~$15 vs ~$13 per run) and comparable or better performance.
\nWhat we can state with confidence: 32K training context produces the best adapters regardless of platform, and our competition choice of Tinker was justified by its proven stability (SD 1.2 across 4 competition runs).
\nAttention-only LoRA enables an efficient merge strategy. We extract the 144 base attention tensors, add LoRA deltas (W' = W + B @ A * scaling) in float32, cast back to BF16, and save as a single 1.78 GB safetensors shard. The model's safetensors index is modified to point attention keys at the new shard, and symlinks reconstruct the full model directory at inference time.
# Sparse merge: 144 attention tensors, 14 seconds on CPU\nATTN_KEYS = [\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"]\nfor name in base_tensor_index:\n if any(k in name for k in ATTN_KEYS):\n base_w = load_base_tensor(name).float()\n lora_a = adapter[f\"base_model.model.{name}.lora_A.weight\"]\n lora_b = adapter[f\"base_model.model.{name}.lora_B.weight\"]\n merged = base_w + (lora_b.float() @ lora_a.float()) * (alpha / rank)\n new_shard[name] = merged.to(torch.bfloat16)\nMerge takes 14 seconds on Kaggle CPU. No --enable-lora flag needed: vLLM loads the merged model as a normal model with full KV cache (108K tokens vs 48K with LoRA adapter). Our first LoRA competition submission used --enable-lora and timed out at 5 hours due to 37% inference overhead.
Before turning to fine-tuning, we exhaustively explored inference-side improvements. Every experiment changed exactly one variable from the v12 baseline (32/50 PolyMath-Val-50, 39.8 competition mean across 6 runs).
\n| Version | \nChange | \nIndividual scores | \nMean | \nVerdict | \n
|---|---|---|---|---|
| v12 | \nbaseline (8 attempts, T=1.0) | \n39, 39, 40, 43, 37, 41 | \n39.8 | \n— | \n
| v16 | \n8→6 attempts | \n36 | \n36.0 | \nFewer attempts = less diversity | \n
| v17 | \nminimalist prompt | \n38 | \n38.0 | \nVerbose prompt is necessary priming | \n
| v22 | \ntighter prompt (\"don't brute force\") | \n38 | \n38.0 | \nSuppresses TIR code usage | \n
| v25 | \nT=0.8 | \n38, 36, 38 | \n37.3 | \nDiversity > consistency | \n
| v18 | \n+4 attempts on split votes | \n37 | \n37.0 | \nVote dilution: wrong samples strengthen wrong majority | \n
| v23 | \njupyter_timeout 6→15s | \n32 (val only) | \n— | \nWrong code doesn't become right with more time | \n
| v26 | \nconditional 12 attempts | \n34 | \n34.0 | \nTime cost outweighs marginal gain | \n
| v27 | \nsequential 4+4 batches | \n32 (val only) | \n— | \nGPU contention not the bottleneck | \n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2982177%2F1c4fc86390e081b35986bf31d188a8b7%2Ffigure11_inference_ablation.png?generation=1779031553930334&alt=media\")
\nFigure 11: Inference-side ablations: competition scores
Why every inference modification failed: the model's failure modes are deterministic. When it gets a problem wrong, all 8 attempts converge on the same wrong approach regardless of prompt, temperature, or number of samples. This was confirmed by v19's diagnostic analysis: across 245 attempts on 50 problems, no metadata signal (entropy, wall time, error count, code/answer correlation) distinguishes correct from incorrect attempts within a problem. All apparent correlations were cross-problem difficulty confounds.
\nThis exhaustive exploration gave us confidence that the v12 inference configuration was the right baseline on which to build our SFT approach. The bottleneck is the model's reasoning distribution, not the inference pipeline.
\nWe spent 10 iterations trying to improve answers after the initial vote:
\n| Version | \nApproach | \nResult | \nWhy it failed | \n
|---|---|---|---|
| v13 | \nCode-based verification | \n0/6 | \nModel rubber-stamps its own wrong answers | \n
| v13_dual | \nMixed temperature (T=0.3 + T=1.0) | \n0/6 | \nT=0.3 converges on same wrong answers | \n
| v14 | \nCounter-proposals from verification code | \n0/6 | \nSingle counter can't override majority | \n
| v15 | \nTrace analysis for discriminating test cases | \n— | \nRevealed the problem but not a solution | \n
| v17 | \nGenSelect round-robin (AIMO-2 style) | \n0/18 matchups | \nModel can't adjudicate between competing solutions | \n
| v18 | \nExtended voting (+4 attempts) | \n37/50 | \nMore wrong samples = stronger wrong majority | \n
| v19 | \nSignal-based reweighting | \n32/50 (val) | \nNo signal works (see 6.1) | \n
| v20 | \n4 diverse parallel prompts | \n23/50 | \nAll prompts converge on same wrong answer | \n
| v21 | \n\"Your answer is WRONG, re-solve\" + minority flip | \n34/50 (net -1) | \nThe one flip was incorrect | \n
The key insight: the model CAN sometimes find the correct answer when told \"your answer is wrong\" (v21, Prompt B found the correct answer to the Norwegian integers problem in 1/3 tries). But it cannot reliably verify which answers are correct. Self-verification is fundamentally broken: the model confirms its own wrong answers with high confidence. This is consistent with recent findings on LLM self-verification [7] and NVIDIA's experience that GenSelect requires purpose-trained selection models [3].
\nFuture work: an efficient external verifier (one that can reliably reject wrong answers without re-solving) could unlock the sequential self-correction approach. The Norwegian integers experiment (where Prompt B's \"assume wrong, re-solve\" found the correct answer 8687 from scratch) demonstrates the potential.
\nConclusion: no post-hoc mechanism we tried could fix deterministic reasoning failures. Only training changes the reasoning distribution. This is what motivated our transition to fine-tuning.
\nAfter establishing that SFT helps, we hypothesized that DPO (Direct Preference Optimization) could further sharpen the model's preference for correct reasoning paths. DPO was a viable approach in AIMO-2 [8], where the 2nd-place team used SFT→DPO successfully on a 14B model.
\nSetup: starting from v08a's SFT checkpoint, we trained DPO with 517 preference pairs (wrong trace vs shortest correct trace for the same problem), beta=0.1, LR=1e-5, 1 epoch. Training metrics converged well (margin=4.04, accuracy=1.0).
\nResult: PolyMath-Val-50 31/50 (-1), competition 35/50 (-5). Severe regression.
\nControl: DPO without SFT base (v09b) couldn't learn at all; margin stayed near zero, loss didn't converge. This confirms SFT is a necessary precursor, but DPO on top destroys what SFT built.
\nWhy DPO failed on GPT-OSS-120B: we believe the model is already so capable that DPO's contrast signal is too coarse. Wrong traces fail for many different reasons (wrong formula, computational error, timeout), and a single \"preferred/dispreferred\" label doesn't capture this diversity. DPO may also undo SFT's regularization effect by pulling the model too far from its natural distribution. This contrasts with AIMO-2's 14B model, which had more room for capability improvement.
\nScope caveat on preference optimization: our DPO experiment used contrasting pairs sourced entirely from the same base model's own traces (correct vs incorrect attempts from the 8-per-problem datagen). We did not explore preference optimization with strong external models (for example, using a GPT-5-class, Kimi, or GLM model as the source of preferred traces or as a judge over candidate pairs), which would be a substantively different setup. Our negative DPO result here speaks to within-distribution preference optimization on top of an already-capable model; it does not rule out PO methods that introduce a stronger external signal. If we revisit this work, that experiment is at the top of our list (see Section 9.6).
\nError trimming (v10b): we tried removing Python error blocks (tracebacks, exceptions) from training traces to give the model \"cleaner\" examples. Score dropped from 32/50 to 29/50. The model learns from seeing errors and recovery: the error-then-fix pattern is part of what it needs to reproduce at inference time.
\nSelf-correction traces (v09c): we constructed 220 synthetic correction examples: wrong trace (truncated before \\boxed{}) → pivot phrase (\"Wait, let me verify this answer by checking with a different approach.\") → full correct trace. PolyMath-Val-50: 31/50. The model found correct minority answers on 6 extra problems but couldn't flip the majority vote. The correction behavior learned in training didn't reliably transfer to the voting-based inference setting.
| Version | \nChange | \nExamples | \nLR | \nVal-50 | \nCompetition | \n
|---|---|---|---|---|---|
| v12 | \nbase (no LoRA) | \n— | \n— | \n32/50 | \n39.8 (6 runs) | \n
| v03 | \nunbalanced data | \n1006 | \n— | \n25/50 | \n— | \n
| v04 | \nbalanced, rank 16 | \n430 | \n2e-4 | \n30/50 | \n— (timed out) | \n
| v05c | \nbalanced, rank 32 | \n430 | \n2e-4 | \n31/50 | \n37.0 (3 runs) | \n
| v06 | \n2× data, same LR | \n794 | \n2e-4 | \n30/50 | \n40 | \n
| v06b | \n2× data, half LR | \n794 | \n1e-4 | \n32/50 | \n— | \n
| v07a | \n3× data | \n1333 | \n1e-4 | \n32/50 | \n37 | \n
| v08a | \nfull data, balanced | \n1788 | \n4e-5 | \n32/50 | \n40.0 (4 runs) | \n
| v08b | \nremove easy bucket | \n1156 | \n6e-5 | \n31/50 | \n— | \n
| v08c | \nremove hard bucket | \n1402 | \n6e-5 | \n30/50 | \n— | \n
| v08d | \n4× hard upsample | \n2174 | \n3e-5 | \n31/50 | \n— | \n
| v09a | \nSFT→DPO | \n368 pairs | \n1e-5 | \n31/50 | \n35 | \n
| v09c | \nSFT + correction traces | \n2444 | \n3.5e-5 | \n31/50 | \n— | \n
| v10a | \n+ pass-2 hard traces | \n1911 | \n4e-5 | \n31/50 | \n— | \n
| v10b | \n+ error code trimming | \n1941 | \n4e-5 | \n29/50 | \n— | \n
| v11a | \nUnsloth 48K | \n2106 | \n4e-5 | \n33/50 | \n39.0 (2 runs) | \n
| v11c-f | \nUnsloth 65K (×4) | \nvarious | \nvarious | \n31/50 each | \n— | \n
| Component | \nDetails | \n
|---|---|
| Inference | \nKaggle H100 (80GB), vLLM 0.16.0 (gpt-oss fork), 5-hour wall clock | \n
| Training (Tinker) | \nThinking Machines fine-tuning API, 32K context cap, ~$13/run, ~$191 total | \n
| Training (Unsloth) | \nRunPod H100, QLoRA NF4, SDPA patch required, ~$100 across 6 runs | \n
| Data generation | \nKaggle H100 (200 problems) + RunPod H100 (1,000 problems, ~$72) | \n
| Evaluation | \nPolyMath-Val-50 (primary) + 10-problem reference test (smoke check) | \n
| Merge pipeline | \nSparse attention merge: 144 tensors, 1.78 GB shard, 14s on CPU | \n
We hypothesize that large MoE models like GPT-OSS-120B have a broad distribution over reasoning strategies. For a given problem, the model \"knows\" multiple approaches: some correct, some wrong. At T=1.0, sampling from this broad distribution means each of 8 attempts may take a different path, and the wrong paths sometimes outvote the correct ones.
\nSFT on correct traces narrows this distribution toward strategies that actually work, without eliminating the diversity needed for voting to succeed. It's analogous to lowering temperature on the strategy selection (which approach to try) without lowering temperature on the execution (how to carry out the chosen approach). This explains why SFT outperforms actual temperature reduction: T=0.8 uniformly suppresses all diversity (including useful diversity on problems where the model has multiple valid approaches), while SFT selectively suppresses only the wrong strategies.
\nThe aggregate mean and SD numbers in Section 7 do not directly answer how the model behaves differently after SFT. To get a concrete behavioral signal, we compared v12 (base, no LoRA) and v08a (SFT, our final submission model) on the same 50 PolyMath-Val-50 problems, extracting per-problem timing and Python tool-call counts from the inference logs of both runs.
\nHeadline: same score, less exploration.
\n| Metric | \nv12 (base) | \nv08a (SFT) | \nΔ | \n
|---|---|---|---|
| PolyMath-Val-50 score | \n32/50 | \n32/50 | \n0 | \n
| Total compute (sum of per-problem times) | \n3h 44m | \n3h 04m | \n−17.8% | \n
| Per-problem mean time | \n269s | \n221s | \n−48s | \n
| Per-problem time stdev | \n204s | \n166s | \n−38s | \n
| Problems completing in <300s (strong early-stop indicator) | \n32 | \n38 | \n+6 | \n
The score is unchanged, but v08a finishes the same 50 problems in ~40 minutes less wall-clock time and reaches the 4/8 consensus threshold before the 300s minimum budget on 6 more problems. The per-problem time variance also narrows (204s → 166s), the same SD-narrowing effect seen in competition scores, here at a different time scale.
\nPer-problem matching by correctness transition. With the same 50 problems on both runs, every problem falls into one of four categories:
\n| Transition | \nn | \nMean Δt | \nInterpretation | \n
|---|---|---|---|
| correct → correct | \n30 | \n−35s | \nSFT confirms the same correct answer with less exploration | \n
| wrong → wrong | \n16 | \n−49s | \nSFT model also gives up sooner on problems it cannot solve | \n
| wrong → correct (gain) | \n2 | \n+10s | \nTwo problems newly solved; slightly more time spent verifying | \n
| correct → wrong (loss) | \n2 | \n−290s | \nTwo losses: fast convergence to a wrong answer | \n
The cleanest behavioral signal is the correct→correct row: on the 30 problems both versions solve, the SFT model reaches 4/8 consensus a mean 35 seconds sooner. Same final answer, fewer attempts before consensus fires. The wrong→wrong row shows the same direction: ~49 seconds sooner on problems neither version can solve. Regularization tightens the trajectory in both directions: less exploration when the model is going to be right, less exploration when it isn't.
\nThe 2/2 gain/loss split is too small for a directional claim, but the two losses are informative on their own. PIDs 2002 (Geometry) and 522 (Algebra) were correct in v12 and wrong in v08a, and completed much faster: the v08a trajectory converged to a wrong answer where the v12 trajectory eventually found the correct one with more exploration. This is the \"confident divergence\" failure mode: when regularization tightens trajectory in a direction that turns out to be wrong on a specific problem, the model commits sooner than the base version would have. It is a real cost of the approach, balanced by faster convergence on the dominant 30 + 16 stable problems.
\nPython tool usage corroborates the timing story. The same per-problem matching gives us call and error counts for the stateful Jupyter sandbox:
\nThe compute reduction comes from fewer tool calls, not from better tool calls: the model is doing less exploratory computation, not improving the quality of the computation it does. Per-transition breakdown:
\n| Transition | \nn | \nv12 calls/pb | \nv08a calls/pb | \nv12 err% | \nv08a err% | \n
|---|---|---|---|---|---|
| correct → correct | \n30 | \n107.0 | \n94.6 | \n8.4% | \n9.4% | \n
| wrong → wrong | \n16 | \n143.2 | \n115.1 | \n13.4% | \n14.6% | \n
| wrong → correct (gain) | \n2 | \n59.0 | \n59.0 | \n28.8% | \n30.5% | \n
| correct → wrong (loss) | \n2 | \n161.5 | \n76.0 | \n20.1% | \n9.9% | \n
On the 30 correct→correct problems, the SFT model averages ~12 fewer Python calls per problem and arrives at the same answer: concrete evidence of less exploration on problems it can solve. The correct→wrong loss row is striking: v12 made 161 calls per problem at a 20% error rate (clearly hunting through the sandbox) and eventually arrived at the right answer; v08a more than halved the call count (76 calls) with a much cleaner 9.9% error rate, and converged to the wrong answer. The cost of regularization on these problems is \"exhaustive but eventually correct exploration\" replaced by \"confident shortcut to the wrong answer.\"
\nPer problem-type. All four types speed up. Algebra −43s, Combinatorics −67s, Geometry −40s, Number Theory −38s (mean Δt). Number Theory has both of the score gains. Combinatorics has 14/14 problems stable in correctness despite the largest mean time reduction, a particularly clean case where regularization helped convergence without affecting correctness.
\nConnection to brevity-biased data selection. Section 4.3 describes the within-bucket selection rule for downsampled buckets: in the 8/8 and 7/8 buckets, we keep the shortest correct trace per problem rather than sampling randomly. This biases the training data toward more compact correct paths, which is a plausible mechanism for the faster inference-time convergence observed here. We did not isolate this experimentally with a counterfactual training run using random within-bucket selection, so this is a hypothesis rather than an established causal claim. But the inference-time behavior (less time, fewer Python calls, same answers on the dominant 30 + 16 stable problems) is consistent with the training-time bias.
\nCaveats. The analysis above comes from one v12 PolyMath-Val-50 run and one v08a PolyMath-Val-50 run (n=1 per side). The 30/16 stable counts and the time deltas within them carry the actual signal at this sample size; the 2/2 gain/loss split is too small for directional claims. The compute reduction (−17.8%) and the Python-call reduction (−16.7%) are large enough to be unlikely from sampling noise alone, but a multi-seed replication would establish tighter confidence on the magnitudes. The PolyMath-Val-50 problems are validation-set problems and not the competition private set, so the exact distribution shift is not guaranteed to carry to out-of-distribution problems.
\nThe thesis that SFT acts as behavioral regularization carries a corollary: the training data should stay as close as possible to the model's own natural correct trajectories. Every attempt we made to \"improve\" the training data by modification (substituting traces from a stronger source, removing intermediate errors, synthesizing self-correction patterns, or applying preference signals on top) regressed. We summarize these here as one mechanism, because they tell a single story.
\nExternal source, same format (NuminaMath). SFT on NuminaMath-TIR (Section 4.1), a dataset of high-quality math solutions generated by GPT-4, completely suppressed code generation: 0 of 400 attempts used Python after training. The dataset itself is strong; the failure is about format mismatch. NuminaMath's text-heavy single-pass format overwrote the model's tool-integrated reasoning behavior. Source strength does not compensate when the training data does not match the format the model will produce at inference.
\nRemoving intermediate errors (v10b). We tried filtering Python error blocks (tracebacks, exceptions) out of training traces on the intuition that \"cleaner\" examples would teach the model better behavior. PolyMath-Val-50 dropped from 32/50 to 29/50 (Section 6.4). Errors are not noise to be removed; the error-then-fix pattern is part of how the model recovers at inference, and the training trace has to include it.
\nSynthesizing self-correction patterns (v09c). We constructed 220 examples of \"wrong trace, pivot phrase, correct trace\" to teach the model to flip from an incorrect answer to a correct one mid-trajectory (Section 6.4). The learned behavior did not transfer to the voting-based inference setting: the model could find correct minority answers on a few problems but could not change the majority vote. Synthesizing a behavior the base model does not exhibit naturally did not survive contact with our actual inference pipeline.
\nPreference signal on top of SFT (DPO, v09a). Training DPO on top of v08a with contrasting pairs sourced from the base model's own traces regressed competition score from 41 to 35 (Section 6.3). Even within-distribution modification beyond plain SFT, pushing the model preferentially toward shorter correct traces and away from wrong ones, degraded behavior. The model is already capable; the contrast signal appears to undo SFT's regularization by pulling further from the natural distribution than is helpful.
\nThe unifying interpretation. SFT on self-generated correct traces works because it nudges the model toward more reliable patterns that it already produces. Each of the modifications above introduces some asymmetry between training and the model's natural inference behavior: a foreign format, an absent error-recovery pattern, a synthesized transition the base model would not generate, or a contrasting push beyond what plain reinforcement provides. We did not try every conceivable trace modification, but the consistency of the negative results across four very different attempts makes us confident that trace integrity, in the sense of keeping the training set as a faithful subset of the model's own correct trajectories, is a load-bearing design choice rather than an incidental one. Reviewers asked whether stronger filtering or stronger preference signals might help; based on these results, we expect them to require care. The filter would need to separate hallucinated reasoning from legitimate error-and-recovery patterns, and the preference signal would likely need an external strong-model source rather than within-distribution pairs (see Section 9.6 for both as follow-ups).
\nSingle-model approach: we used only GPT-OSS-120B. Ensembling with a second model could improve diversity, but the H100 VRAM constraint makes this challenging.
\nTinker 32K context cap: training traces are truncated to 32K tokens, while inference runs at 65K. Some hard problems produce 50K+ token traces that are excluded from training. This may explain why the 13 always-wrong problems don't improve: their correct traces may be too long to train on.
\nPlatform/context confound partially resolved: our Unsloth 32K experiment (Section 5.4) suggests context length, not training platform, is the key variable. However, this finding rests on a single RunPod run; multiple seeds are needed for confirmation.
\nLimited competition runs: v08a has 4 competition data points. While SD 1.2 is encouraging, 4 points cannot definitively establish low variance.
\nThe project spanned ~7 weeks (March 1 – April 15, 2026):
\nWe thank the AIMO Progress Prize organizers and XTX Markets for designing a competition that rewards reproducibility and open science. We acknowledge Thinking Machines for offering Tinker credits, which were crucial for our SFT experiments. We utilized Claude Code (Opus 4.6) by Anthropic throughout this project for assistance with code development, data analysis, and drafting the report.
\n[1] OpenAI. GPT-OSS-120B. https://huggingface.co/openai/gpt-oss-120b
\n[2] Dekoninck, J., Hiss, H., Petrov, I., Rögnvaldsson, K., & Sun, C. PolyMath: A curated dataset of 11,000 high-difficulty problems. AIMO 3 Math Corpus Prize submission. https://huggingface.co/datasets/AIMO-Corpus/PolyMath
\n[3] Moshkov, I. et al. AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset. arXiv:2504.16891, 2025.
\n[4] Thinking Machines. Tinker: Fine-tuning API for GPT-OSS. https://www.thinkingmachines.ai/
\n[5] Unsloth. QLoRA training framework. https://github.com/unslothai/unsloth
\n[6] Bisiadis, A. AIMO 3 GPT-OSS-120B with Tools. https://www.kaggle.com/code/andreasbis/aimo-3-gpt-oss-120b-with-tools
\n[7] Bialas, P. et al. Can Language Models Falsify? Towards Robust Mathematical Reasoning. arXiv:2412.15184, 2024.
\n[8] imagination-research. AIMO-2 2nd Place Solution: SFT→DPO pipeline. https://github.com/imagination-research/aimo2/
\n[9] Frieder, S. AIMO 3 Problem Difficulty Analysis. https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/679559
\n[10] vLLM Project. vLLM: Easy, fast, and cheap LLM serving. https://github.com/vllm-project/vllm (gpt-oss fork v0.16.0 required)
\n[11] nihilisticneuralnet. [44/50] Let Me (over)COOK!! https://www.kaggle.com/code/nihilisticneuralnet/44-50-let-me-over-cook
", "comments": [ { "id": 3459321, "author_name": "Onur Fidaner", "author_url": "", "post_date": "2026-05-17T16:50:56.140000", "content": "Edited on 05/17/2026 for the rebuttal phase:
\nI've published an updated version of the writeup incorporating the feedback from @philipvonderlind and @sambealings:\nThe \"large MoE\" framing for GPT-OSS-120B has been toned down throughout; a new Section 9.3 (\"Trace integrity\") consolidates evidence that self-generated, unmodified traces were load-bearing for our approach, addressing the trace-quality and PO-target questions; Section 3 verifies from run logs that PolyMath-Val-50 problem ordering was effectively randomized and the time budget never bound across the three runs we inspected; Section 4.3 now describes the brevity-selection within difficulty buckets; and a new Section 9.2 presents concrete behavioral evidence of regularization, showing that the SFT model uses ~18% less compute and ~17% fewer Python tool calls while reaching the same answers on the same 50 problems. I've also added the private leaderboard result (42.0/50 averaged over two runs) to the abstract and Section 2.1. Lastly, I made modifications to the figures and the text overall to help with the flow and improve readability.
\nMany thanks to Philip and Sam for the careful review.
", "votes": 0, "replies": [] }, { "id": 3458587, "author_name": "Sam Bealing", "author_url": "", "post_date": "2026-05-16T08:12:30.247000", "content": "Following up on the questions from @philipvonderlind (and thanks for the very thorough response), here are some additional questions to help us assess your submission:
\nLook forward to reading your responses on these.
", "votes": 0, "replies": [ { "id": 3458726, "author_name": "Onur Fidaner", "author_url": "", "post_date": "2026-05-16T14:17:56.827000", "content": "Thanks for the follow-up. Both questions are very helpful, and here are my answers:
\nThe sampling is balanced across difficulty buckets. Difficulty is measured by the base model's vote share — the fraction of 8 base-model attempts (T=1.0) that produce the correct answer, ranging from 1/8 (hard) to 8/8 (easy). The bucket table from Section 4.3 of the writeup:
\n| Difficulty bucket | \nProblems | \nSampling strategy | \nTraining share | \n
|---|---|---|---|
| 8/8 (easy) | \n~350 | \n1 trace each | \n~27% | \n
| 7/8 | \n~150 | \n2 traces each | \n~14% | \n
| 5-6/8 (medium) | \n~280 | \nall traces | \n~33% | \n
| 1-4/8 (hard) | \n~260 | \nall traces × 2 | \n~26% | \n
We downsample the easy 8/8 bucket and upsample the hard 1-4/8 bucket. Downsampling is done based on brevity; i.e., in the 8/8 bucket we keep the 1 shortest correct trace per problem and in the 7/8 bucket we keep the 2 shortest traces. In downsampling, the idea was that shorter correct traces tend to represent more compact CoT paths with less rambling. Otherwise, the motivation for the balancing of hard/medium/easy traces comes from an early experiment: v03 was unbalanced (59% of training data came from 8/8 easy problems, simply because they generate more correct traces per problem) and scored 25/50 on PolyMath-Val-50, a 7-point regression below the base model's 32/50. v04 (same data sources, balanced) immediately rose to 30/50. v08a (1,788 traces, fully balanced) reached 32/50.
\nThe Section 5.3 ablation confirms both extremes hurt: removing the 8/8 (easy) bucket cost 1 point (v08b), removing the 1-4/8 (hard) buckets cost 2 points (v08c), and 4× upsampling the hard bucket provided no further benefit (v08d, 31/50). Hence the eventual choice was an optimal mix of easy/medium/hard questions.
\nIn summary, we sample to expose the model to its full range of typical reasoning, not just hard problems. Over-weighting hard problems would bias the regularization toward conditions where the model might overcomplicate its CoT and fail as a result, pushing the reasoning distribution toward less-reliable strategies. The balanced sampling keeps the model's nominal range across a question difficulty scale, which we found to be important to not regress in otherwise solvable problems while attempting to solve harder problems. For easier problems there is a bias towards brevity. I will be adding some additional discussion and clarification in the relevant sections in the writeup.
\nThis is a very good question, and I should have included more discussion on that. As you point out, the writeup leans heavily on aggregate mean/SD numbers without a clean measurement of behavioral change. To address this, I ran a post-hoc analysis matching the v12 (base, no SFT) and v08a (SFT, the final submission model) runs at the per-problem level on the same 50 PolyMath-Val-50 validation set problems. The results are very interesting:
\n| Metric | \nv12 (base) | \nv08a (SFT) | \nΔ | \n
|---|---|---|---|
| PolyMath-Val-50 score | \n32/50 | \n32/50 | \n0 (no capability change) | \n
| Total compute (sum of per-problem times) | \n3h 44m | \n3h 04m | \n−17.8% | \n
| Per-problem mean time | \n269s | \n221s | \n−48s | \n
| Per-problem time stdev | \n204s | \n166s | \n−38s | \n
| Problems completing in <300s (strong early-stop indicator) | \n32 | \n38 | \n+6 | \n
Same problems solved, ~40 minutes less total compute, and 6 more problems hit the 4/8-consensus threshold before exhausting the 300s budget floor. So even though the score didn't necessarily improve, the behavior certainly changed.
\nWhere the time saving comes from:
\n| Transition | \nn problems | \nMean Δt | \nWhat it shows | \n
|---|---|---|---|
| correct → correct | \n30 | \n−35s | \nSFT model confirms the same correct answer with less exploration | \n
| wrong → wrong | \n16 | \n−49s | \nSFT model also gives up sooner on problems it cannot solve | \n
| wrong → correct (gain) | \n2 | \n+10s | \nTwo problems newly solved; slightly more time spent verifying | \n
| correct → wrong (loss) | \n2 | \n−290s | \nTwo losses — fast convergence to a wrong answer | \n
The cleanest behavioral signal is the correct→correct row: on the 30 problems both models solve, the SFT model reaches 4/8 consensus a mean 35 seconds sooner. Same final answer, fewer attempts needed before consensus fires. The wrong→wrong row shows a parallel pattern — the SFT model recognizes hopeless problems faster, ~49s sooner on average. So the regularization tightens the trajectory in both directions: less exploration when the model is going to be right, and less exploration when it isn't. This may be attributable to the downsampling approach in easier questions.
\nConcrete correct→correct speedups (PID, type, v12 time → v08a time):
\nThe 2/2 gain/loss split shows individual problems can flip in either direction. The 2 losses (PIDs 2002 in Geometry and 522 in Algebra) are the \"confident divergence\" failure mode — SFT converges quickly to a wrong answer. These are not a directional signal at this sample size, but I believe it shows that SFT shifts the reasoning behavior.
\nBehavior Change regarding Python tool usage The same per-problem matching also gives us call and error counts for the stateful Jupyter sandbox:
\nThe model issues fewer calls overall, with roughly the same error rate per call. Therefore one can say the regularization is correlated with less exploratory tool use. I think this is an interesting future research direction in this line of thought. It will be interesting to try SFT that not only reduces tool use, but also increases its usefulness. Here is the per-transition breakdown:
\n| Transition | \nn | \nv12 calls/pb | \nv08a calls/pb | \nv12 err% | \nv08a err% | \n
|---|---|---|---|---|---|
| correct → correct | \n30 | \n107.0 | \n94.6 | \n8.4% | \n9.4% | \n
| wrong → wrong | \n16 | \n143.2 | \n115.1 | \n13.4% | \n14.6% | \n
| wrong → correct (gain) | \n2 | \n59.0 | \n59.0 | \n28.8% | \n30.5% | \n
| correct → wrong (loss) | \n2 | \n161.5 | \n76.0 | \n20.1% | \n9.9% | \n
On the 30 correct→correct problems, the SFT model averages ~12 fewer Python calls per problem and arrives at the same answer — clean evidence of less exploration on problems it can solve. On the 2 correct→wrong losses, v12 was doing 161 calls at a 20% error rate (clearly fishing) and eventually finding the right answer; v08a more than halves the call count (76 calls, much cleaner 9.9% error rate) and produces the wrong answer. So on the rare losses, SFT replaced \"exhaustive but eventually correct exploration\" with \"confident shortcut to the wrong answer.\" This is one metric to address to improve overall performance with further SFT.
\nPer problem-type (timing): all four types speed up. Algebra −43s, Combinatorics −67s, Geometry −40s, Number Theory −38s (mean Δt per type). Number Theory has both of the score gains; Combinatorics has zero score changes (14/14 stable) despite the largest mean time reduction. Hence, regularization helps convergence without affecting correctness across problem types.
\nOnce again thank you. I'll be adding these findings to the updated writeup. Once I finish the writeup, I'll post a short summary of changes on Simon's discussion thread.
", "votes": 0, "replies": [] } ] }, { "id": 3456843, "author_name": "Philip Vonderlind", "author_url": "", "post_date": "2026-05-12T16:57:29.053000", "content": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.
\nThe submission is already very nice overall. I especially like the plots and tables you included.
\nMoE Size: I would be cautious about describing gpt-oss-120b as a large MoE, since it only has around 5B active parameters. I would reserve that description for models such as DeepSeek-R1. This is only a minor side remark, however.
Trace Quality: Since preference optimization depends heavily on the quality of the preference dataset, did you take any special precautions to avoid including low-quality or fully hallucinated traces? Relying only on final-answer verification could still allow many flawed reasoning traces into the training data. The model may have difficulty learning useful behavior if most of the intermediate signals are noisy or incorrect, even when the final answer is right.
PO Target: In preference optimization, it is common to use a stronger model as a reference or source of preferred responses. Did you explore using very strong models such as GPT-5.x, Kimi, or GLM in any way? One possible complication I could imagine is compatibility with the Harmony chat format and tool calls, although it should in principle, be possible to convert traces from other models into the required format.
Results on PolyMath: As this competition has shown, variance is an important factor to account for. Did you always evaluate PolyMath-v50 in the same problem order? If so, this could potentially skew results, for example, if the system overfits to that order by consistently seeing harder problems first and easier ones later, especially under a limited time budget.
Looking forward to your response :)
", "votes": 0, "replies": [ { "id": 3457990, "author_name": "Onur Fidaner", "author_url": "", "post_date": "2026-05-14T18:52:44.577000", "content": "Thank you for the thoughtful comments and the kind words about the plots and tables. Below are my responses to each of your comments. I am working on an updated version of the write up to address your comments and sharpen a few other things. I'll post a summary of those edits on the discussion thread once the writeup is updated.
\nYou're right, GPT-OSS-120B is closer to a mid-sized dense model in inference cost than to a true large MoE like DeepSeek-R1. I'll clarify the language in the writeup.
\nI'll address 2 and 3 together since they both ask about the role of stronger external models can play in our pipeline.
\nPreference Optimization: We tried DPO early in our experimentation (Code version v09a, discussed in Section 6.3). Training itself converged well (margin 4.04, accuracy 1.0), but the competition score (public data set) regressed from 41/50 to 35/50. Base model competition score had mean 39.8 with SD ~2.0 (range 37 to 43 across 6 runs), so the regression to 35 is well outside the noise floor. We used the correct and wrong traces from the multiple runs on GPT-OSS-120B. I think there could be a way to get that to work (as demonstrated by LB#2 solution in AIMO2). Given the tight time budget, we focused the remaining budget on SFT, which had already shown promise in terms of stable gains. Hence, the final submission for the private data set is based on SFT, and DPO is documented as a failed branch within that scope. I think it is worth experimenting whether DPO could succeed with a better-curated preference dataset / different set of traces.
\nObviously, your broader point is also relevant for SFT itself: Could traces from a stronger model improve our training data? My experience suggests potentially yes, but with caveats for GPT-OSS-120B. Here's what we tried:
\nNuminaMath-TIR (the AIMO-1 winner dataset) traces were generated by GPT-4o using the same final-answer filtering you describe. I tried these traces with SFT as my first TIR attempt (v02). The result was a complete collapse of code generation: 0 out of 400 attempts used Python after training. In a way, this is not unexpected, since GPT-4o does not have TIR embedded in its weights. Numina generated their dataset using the ToRA pattern (prompt GPT-4o to write Python, execute the code in an external loop, splice results back into the prompt). On the other hand, GPT-OSS-120B has native tool use as a built-in capability, with harmony-format tool calls directly from inside its chain of thought. The trace format is very different (around 1.5K tokens of plain Python execution vs around 15K tokens of multi-step harmony tool calls). So, while not conclusive in terms of whether stronger models like GPT-5 could help, this experiment points to the fact that output format matching is crucial. When other LLMs with potentially different CoT format are used, probably an intermediate step is needed to make it compatible with OSS-120B's style for SFT to be successful.
The cleaner test of format-sensitivity we actually have is with our own self-generated traces, which are surprisingly fragile to modification. In v10b (Section 4), we kept the source model fixed but trimmed code blocks that produced errors-before-recovery, expecting a small gain from the cleanup. PolyMath-Val-50 performance instead dropped from 32 to 29, a 3-point regression that is well outside the typical run-to-run variance we observed on Tinker-trained models. Same source, same problems, only the editing changed. My conclusion was that the model needs to see its own error-and-recovery patterns for successful SFT — what may look like noise or bantering in intermediate reasoning actually carries useful signal. As you suggest, defining metrics around reasoning efficiency in traces and optimizing them is probably a viable strategy, but in the form I tried it, it regressed. I also tried a version where I started with a wrong-answer trace and stopped midway for course-correction toward the correct trace. That attempt was half-baked and also regressed. Both led me to conclude that GPT-OSS-120B SFT is very sensitive to the output style of the traces, so I stuck with organic, unmodified traces generated by the model itself.
One interesting observation was that if there is a good \"verifier\" code block to determine the correctness of an answer, then you can use prompt engineering to steer the reasoning. E.g., the solver gives an answer, then using the verifier block we determine it's wrong. Then we hit back saying \"The answer XYZ is known to be wrong, and go back and retry using a different approach\" This approach swayed the reasoning towards the correct answer, but of course you need to have that accurate verifier.
For all these reasons, I chose self-generated traces to support the core idea. The paper's framing is behavioral regularization, not capability transfer. We are biasing the model toward strategies it already finds (sometimes), not teaching new ones. Traces from Claude or GPT-5 would be a different and more ambitious bet, closer to capability transfer, and would first need to clear the format-compatibility bar. Overall, I agree that there are many interesting things to try along this line of discussion.
\nThese ideas were not feasible within the time frame of the competition but I definitely want to do a follow-up. Thank you for pointing these out. For this writeup, I will be updating to make the SFT framing more explicit and clarify some of the points.
\nI double checked my evaluation output files to look at this question from two dimensions:
\n1. Order of Questions: PolyMath-Val-50 runs effectively processed problems in randomized order. For example, the first eight problems in the v08a evaluation run (the one that produced our headline 32/50 number) were processed in this sequence (problem IDs): 3049, 455, 886, 1581, 2228, 4088, 3307, 522. These are mixed across norm_type (Algebra, Combinatorics, Number Theory, Geometry) and across the dataset's oss_20b_med_pass difficulty buckets (both 0.0 and 0.5). The same effective randomization shows up in our v11a (Unsloth 48K) and v11d (Unsloth 65K) hardtest runs, and on our separate 10-problem reference test set as well. So problems were not consistently encountered hardest-first or easiest-first across runs. Although in Fig 5a, I ordered them based on problem type and easiness (Always correct/wobbly/always wrong) so that we can compare different models for each question type/difficulty type. But in the runs they were presented in random order. I will clarify that in the text.
2. Time Budget: More importantly for your concern, the time budget was never binding in practice. Across the v08a, v11a, and v11d runs:
\nSo even if our evaluations had been ordered hardest-first, no problem would have been time-clipped for landing late in the run. Order-induced score effects on PolyMath-Val-50 would therefore be small, and attributable to other factors. Nevertheless, they were randomized in the tests. I also have multi-run comparisons on the same model and same order (e.g., v11c, which swung 33 to 29 across two runs), which suggests that sampling variance is already large enough to dominate any plausible order effect.
\nOf course, the performance variance we observed on the public LB is very real, and is the central problem addressed by my method. For the public data set, I tried a variant of my BKM code offering up to 900s to each question, risking a time-out at the end, and it still completed on time. I have no insights about the private data set, but at least for this algorithm public data set is less likely to be time constrained. The observed variance most likely comes from the variance of the model, hence the self-regularization method to make it smaller.
\nThanks again for these comments. They've been useful for sharpening both the paper and my own thinking about the next steps. I really want to prepare an Arxiv submission based on that, so I'd welcome any other comments you guys may have. Once I finish my edits, I'll be posting a short summary of the resulting paper edits on Simon's discussion thread shortly.
", "votes": 0, "replies": [] } ] } ], "comment_count": 5, "messages": [ { "id": 3447298, "postDate": "2026-04-22T23:15:44.077Z", "content": "Submission basis: Public LB score: 40.0/50 (average over 4 runs). Private LB score: 42.0/50 (average over 2 runs)
\nResources:
\nOur solution for AIMO Progress Prize 3 competition fine-tunes attention layers on 1,788 self-generated Tool-Integrated Reasoning (TIR) traces with difficulty-balanced sampling. The core idea is promoting behavioral regularization through supervised fine-tuning (SFT) on GPT-OSS-120B, rather than teaching this Mixture of Experts (MoE) model new capabilities. GPT-OSS-120B is already capable of solving most of the questions in the competition, albeit not reliably. In our approach, fine-tuning promotes correct solution traces that were already generated by GPT-OSS-120B. As a result, the model doesn't learn new solution methods, but converges more reliably on correct ones it already knows. This manifests as significantly reduced run-to-run variance (SD 1.2 for SFT model vs 2.0 for the base model, both on the public data set) and slightly higher mean (40.0 for SFT vs 39.8 for the base model). Our model scored 42.0/50 on the private data set (averaged over two runs), indicating that our SFT efforts did not cause overfitting. Beyond variance reduction, we measure concrete behavioral changes at inference: the SFT model uses ~18% less compute and ~17% fewer Python tool calls while reaching the same answers on the same problems (Section 9.2), consistent with regularization tightening reasoning trajectories rather than expanding capability. We support our findings with systematic ablation studies across data composition, learning rate scaling, training context length, DPO, verification strategies, and prompt engineering; most of which produced regressions. We also document 10 iterations of post-vote verification approaches and their impact on performance. We believe SFT can be an efficient tool for guiding large MoE models when solving complex questions and our approach demonstrates an interesting use case of SFT on GPT-OSS-120B.
\nGPT-OSS-120B [1] is a 120B-parameter Mixture-of-Experts (MoE) model with only 5.1B parameters active per token (128 experts, 4 active). In Microscaling FP4 (MXFP4) quantization it occupies ~65GB, fitting a single H100 with room for 108K tokens of key-value (KV) cache. It was trained with native tool-use and so the model knows how to call Python, read execution results, and iterate. These capabilities make the model uniquely suited for tool-integrated reasoning (TIR), which is essential for competition-level math where brute-force computation and symbolic verification complement mathematical reasoning. Not surprisingly, GPT-OSS-120B has been the most popular model among AIMO 3 participants. It is arguably the most capable model for solving olympiad-level questions that can fit in the 80GB VRAM of H100, while leaving room for KV cache across multiple parallel attempts.
\nOne of the apparent challenges of fine-tuning a model such as GPT-OSS-120B is that the model is already highly capable in mathematical reasoning. The original training likely included a much larger and more diverse dataset than what we can practically use for fine-tuning, leaving limited room for improvement under small fine-tuning budgets. In fact, many AIMO 3 participants reported diminished performance, or no change, as a result of fine-tuning.
\nWith these observations in mind, we took a different approach to SFT. Rather than attempting to teach the model new solution methods, we used SFT to regularize its behavior, narrowing the distribution of reasoning strategies toward ones that actually work. In our workflow, we first designed an inference pipeline using the base model. Then we ran the base model on 1,200 olympiad-level problems using the exact same inference pipeline (8 parallel attempts per problem at T=1.0), keeping full traces generated by the model. We used the fraction of correct attempts as a proxy for problem difficulty, e.g., 8/8 correct would be an easy problem, and 1/8 would be a hard problem. We kept only the correct traces as training data. In this way, the training set consists entirely of what the model organically produces when it succeeds. The goal is not to expand the model's capabilities, but to increase the odds that on hard problems (where perhaps only 2 or 3 out of 8 base-model attempts find the correct answer) the fine-tuned model will find it more reliably, pushing that ratio to 4 or 5 out of 8.
\nThe base model has high variance (SD 2.0, range 37–43 across 6 competition runs), and fine-tuning's primary effect is reducing that variance. This matters for the competition scoring formula for the private data set, which averages two independent runs: a model with mean 40 and SD 1 outperforms a model with mean 40 and SD 2, because a lower variance ensures the model is much less likely to have a bad run that drags down the average and hence the final score on the private data set. On the private data set the SFT model averaged 42.0/50 over two runs, slightly above the 40.0 public-LB mean, indicating the SFT model does not overfit to the public data set. Beyond run-to-run variance, the SFT model also shows concrete behavioral changes at inference time (~18% less total compute and ~17% fewer Python tool calls on the same problems while reaching the same answers, Section 9.2), supporting the regularization interpretation over a capability-improvement interpretation.
\nIn this paper we describe the components and design considerations for the eventual model we submitted for the private leaderboard scoring. We also designed several experiments to determine optimality of this solution, which will be discussed in more detail in the \"Ablation Studies\" section. Details on how to replicate results both on Kaggle and elsewhere (e.g. Colab, RunPod) are available on HuggingFace [link above].
\nThe competition offered daily attempts to solve the \"public data set\", consisting of 50 problems. The final evaluation was on a \"private data set\" with a different set of 50 problems. The competition hosts characterized the difficulty distribution of both sets using state-of-the-art open-source models [9]. Based on their analysis, approximately 27 problems can be solved in a single pass (pass@1), about 38 can be solved within 3 passes, and roughly 42 within 5 passes. This leaves approximately 8 problems that require significantly more attempts or are beyond the models' reliable reach.
\nFor GPT-OSS-120B specifically, we estimate that approximately 35 problems are reliably solvable (correct on most runs), while about 7-9 problems have varying degrees of success: sometimes correct, sometimes not, depending on sampling luck. This estimate is based on our competition runs, where we observed a maximum score of 43 and a minimum of 37 across 6 submissions with the base model (no fine-tuning). Our inference pipeline solves each problem with 8 parallel attempts at T=1.0 and selects the answer by entropy-weighted majority vote (detailed in Section 3).
\n| Model | \nCompetition runs | \nMean | \nSD | \n
|---|---|---|---|
| Base model (GPT-OSS-120B) | \n39, 39, 40, 43, 37, 41 | \n39.8 | \n2.0 | \n
This variance originates at the problem level. For easy problems, all 8 attempts agree; these contribute a stable +1 to the score every run. For hard problems where only 2 or 3 out of 8 attempts find the correct answer, the majority vote can go either way depending on sampling luck. A problem at 3/8 baseline accuracy flips between correct and incorrect across runs. When several such problems flip simultaneously, the total score can swing by 4–6 points. We attribute this variance to GPT-OSS-120B's intrinsic sampling at T=1.0, not to artifacts of the evaluation procedure, such as time budget per problem or randomized ordering of problems. We discuss these two procedural factors in more detail in Section 3.
\nThe variance makes it challenging to evaluate changes from single competition runs. A score of 36 (2 SD below mean) strongly suggests regression, but distinguishing 38 from 40 is within noise. We therefore could not rely solely on the public leaderboard for model development and needed a more controlled evaluation methodology.
\nThis problem-level variance is precisely the mechanism our SFT targets: by pushing hard problems from 2-3/8 to 4-5/8 correct attempts, we move them from \"unreliably correct\" to \"reliably correct,\" reducing the number of problems that contribute to run-to-run variance. Figure 1 previews the achieved reduction on the public leaderboard: the fine-tuned model (v08a) reduces the SD by approximately 40% (2.0 → 1.2) while leaving the mean essentially unchanged (39.8 → 40.0). The training and inference setup behind v08a is detailed in Sections 4 and 5; the full per-experiment results matrix is in Section 7.
\n\nFigure 1: Competition score variance: Base Model (v12, six runs) vs SFT Model (v08a, four runs) on the public leaderboard.
Given the high variance on the public leaderboard and the one-submission-per-day limitation, we constructed a separate 50-problem evaluation set for controlled model comparisons. We sourced problems from the PolyMath dataset [2], a curated collection of 11,000 high-difficulty problems submitted to the AIMO 3 Math Corpus Prize by Dekoninck, Hiss, Petrov, Rögnvaldsson, and Sun. PolyMath was particularly useful because it provides oss_20b_medium_pass_rate: the pass rate of GPT-OSS-20B (a 20B-parameter model from the same model family) evaluated under the PolyMath team's \"medium\" inference configuration. The values take discrete levels (0, 0.5, 1.0), giving us a coarse difficulty calibration directly relevant to the GPT-OSS model family. Note that this PolyMath-provided difficulty label is distinct from our own difficulty classification (Section 4.3), which uses the 120B competition model's success rate across 8 attempts.
Our validation set was designed to stress-test the model at its difficulty frontier:
\noss_20b_med_pass ≈ 0.5) and 26 hard problems (oss_20b_med_pass = 0)This validation set (referred to as \"PolyMath-Val-50\" throughout) is distinct from the competition's 50-problem test set. It served as our primary metric for comparing LoRA variants. Across 4 runs of the base model using slightly different code versions, the validation score was consistently 32/50, indicating that this set produces stable, reproducible measurements when the model itself is unchanged. This stability, compared to the ±3 point swings on the public leaderboard, made PolyMath-Val-50 essential for experiment design: even 1-point differences can be meaningful here, whereas on the competition set they are noise.
\nIn addition to Kaggle, we ran evaluations on RunPod H100s for experiments that exceeded Kaggle's compute budget (e.g., multi-seed replications, Unsloth training + evaluation). We observed a consistent 1-2 point offset: models that score 32/50 on Kaggle score 30-31/50 on RunPod. This is likely due to slower inference throughput on RunPod (~1.5x), which means each attempt generates fewer tokens within the same time budget. All RunPod comparisons in this work are internally consistent: the offset cancels when comparing models within the same environment.
\n| Model | \nKaggle Val-50 | \nRunPod Val-50 | \n
|---|---|---|
| Base model (v12) | \n32/50 (×4 runs) | \n31, 30 | \n
| SFT model (v08a) | \n32/50 | \n30, 30 | \n
Our inference code is adapted from a public notebook by Andreas Bisiadis [6] (public LB 42/50), which provided the core TIR + entropy-weighted voting architecture, and from a variant by nihilisticneuralnet titled \"Let Me (over)COOK!!\" [11] (public LB 44/50), which was built on Andreas's version and contributed the more verbose system prompt that we adopted in our final configuration. We iterated through 27 versions of this pipeline during the competition (v01–v27), extensively testing modifications at each step (detailed ablation results in Section 6.1). None improved on our base configuration, giving us confidence that it was the right baseline for both data generation and final deployment.
\nThe key inference parameters and architectural decisions:
\n| Parameter | \nValue | \n
|---|---|
| Temperature | \n1.0 | \n
| min_p | \n0.02 | \n
| Attempts per problem | \n8 | \n
| Max context | \n65,536 tokens | \n
| Max generation per turn | \n2,048 tokens | \n
| Top logprobs | \n5 | \n
| Jupyter timeout | \n6 seconds | \n
| Max TIR turns | \n128 | \n
| Early stop | \n4/8 consensus | \n
| GPU memory utilization | \n0.96 | \n
1. Completions API with openai_harmony: GPT-OSS was trained with a specific message encoding protocol. The openai_harmony library encodes messages as raw token IDs with channel routing (<|channel|>python, <|channel|>final) and tool namespaces. Using the native Completions API is essential: our first integration used the Chat Completions API and scored only 13/50 vs 41/50 with the native protocol.
2. TIR with persistent Jupyter kernels: each attempt runs a multi-turn loop (up to 128 turns). The model generates tokens, we parse for recipient=='python', execute code in a sandboxed Jupyter kernel, feed the result back, and let the model continue. We maintain a pool of 16 persistent kernels pre-initialized with numpy/sympy/mpmath.
# Core TIR loop (simplified)\nfor turn in range(128):\n response = client.completions.create(\n model=model_name, prompt=token_ids,\n max_tokens=2048, temperature=1.0, logprobs=5,\n stop=stop_token_ids)\n messages = parse_messages_from_completion_tokens(response)\n for msg in messages:\n if msg.recipient == \"python\":\n result = kernel_pool.execute(msg.content, timeout=6)\n token_ids += encode_tool_result(result)\n elif msg.recipient == \"final\":\n answer = parse_boxed_answer(msg.content)\n if answer is not None:\n return answer, compute_entropy(response), token_ids\n3. Entropy-weighted majority voting: for each problem, 8 parallel attempts at T=1.0, answer selected by weighted majority vote where each attempt's weight is 1 / (1 + entropy). Entropy is computed from top-5 logprobs at the final answer token. We tested raw majority voting (no entropy weighting); it scored 36/50, roughly a 3-point drop from the entropy-weighted version on the same ablation.
4. Time management: each problem gets a budget of min(300 + available_bank, 900) seconds, with early stop at 4/8 consensus. In other words, each question is guaranteed to have 300 seconds minimum, and can be offered up to 900 seconds if extra time is available. The time bank accumulates unused time from fast problems. In practice, the bank rarely constrains: no problem used more than 744s of its 900s budget on PolyMath-Val-50, and the bank grew to ~5,000s unused. Likewise, we ran the inference model without time budgeting other than setting a maximum time of 900 in the public data set and the computation was completed within 5 hours. We are including it as insurance against a potentially harder private test set, which may exceed the time limit.
5. Procedural variance controls (problem ordering and time-budget binding): Returning to the claim in Section 2.1 that competition-run variance is intrinsic to GPT-OSS-120B sampling rather than an artifact of evaluation procedure, we verified two procedural factors directly from the run logs. Problem ordering: Problems are processed in effectively randomized order across runs. On three PolyMath-Val-50 runs (v08a, v11a, v11d) the processed sequences differed across runs and were mixed across difficulty type and oss_20b_med_pass value. Time-budget binding: on the same three runs, the maximum time used by any single problem was 695s, 715s, and 747s respectively, against the 900s minimum budget; the bank-grown budget for slow problems reached 1100-1800s, and this extra headroom was never used. The same inference code and time-management policy applied to public-leaderboard submissions. We therefore conclude that neither problem ordering nor time-clipping contributes to the 37-43 spread reported in Section 2.1, and the spread is mainly due to the intrinsic variance in GPT-OSS-120B chain-of-thought.
This section describes the core idea behind our fine-tuning approach. The mechanical details of LoRA configuration, learning rate calibration, and the training platform comparison are in Section 5.
\nA natural starting point for SFT is to use existing mathematical datasets. Our first attempt used NuminaMath-TIR [3], which contains thousands of math problems with solutions. The result was catastrophic: SFT on NuminaMath completely suppressed code generation, with 0 out of 400 attempts using Python after training. The text-heavy NuminaMath solution format overwrote the model's TIR behavior, replacing iterative code-based reasoning with single-pass text answers. This taught us a critical lesson: the training data must match the inference format exactly. The lesson is about format compatibility, not source strength: NuminaMath-TIR traces are generated by GPT-4, a strong model on math benchmarks, yet using them for SFT destroyed our model's TIR behavior. What matters is that the training data structurally matches the format the model will produce at inference time.
\nSelf-generated traces solve this problem by construction. We run the base model using the same inference pipeline it will use at competition time (the same openai_harmony encoding, the same Jupyter kernels, the same tool-call routing) and capture the full conversation traces. The correct traces (reasoning + code + execution output + \\boxed{} answer) become our training data. The model is trained on its own successful behavior, not on an external format it was never designed to produce.
We selected 1,200 problems from the PolyMath dataset [2] for trace generation:
\noss_20b_medium_pass_rate (the 20B model's pass rate, see Section 2.2): 1,125 problems with pass rate 0.5 (solvable by the 20B model roughly half the time) and 75 with pass rate 0 (unsolvable by the 20B model but potentially solvable by our 120B model)\nFigure 2: Data generation pipeline
Data generation ran on Kaggle H100 (200 problems) and RunPod H100 (1,000 problems). We validated equivalence by comparing results on 4 shared problems: RunPod matched or beat Kaggle (4/4 vs 3/4 correct).
\nDifficulty-bucket sampling was the most impactful design decision. Since easy problems produce 8/8 correct traces and hard problems produce only 1–2/8, using all correct traces without resampling creates a training set naturally dominated by easy-problem reasoning patterns, not because we chose to emphasize easy problems, but simply because they generate more correct traces per problem.
\nIn our first LoRA attempt (SFT v03), 59% of training examples came from 8/8-easy problems (Figure 3, left). This resulted in a score of 25/50 on PolyMath-Val-50, a 7-point regression from the base model's 32/50. The model learned to be more decisive on problems it already solved reliably, while becoming less effective on harder problems.
\nOur balanced sampling strategy downsamples easy problems and upsamples hard ones:
\n| Difficulty bucket | \nProblems | \nSampling strategy | \nTraining share | \n
|---|---|---|---|
| 8/8 (easy) | \n~350 | \n1 trace each | \n~27% | \n
| 7/8 | \n~150 | \n2 traces each | \n~14% | \n
| 5-6/8 (medium) | \n~280 | \nall traces | \n~33% | \n
| 1-4/8 (hard) | \n~260 | \nall traces × 2 | \n~26% | \n
Within the 8/8 and 7/8 buckets where we downsample, we keep the shortest correct traces per problem rather than sampling randomly. Shorter correct traces tend to represent more compact reasoning paths with less rambling, consistent with the broader regularization toward efficient correct strategies. The 5-6/8 and 1-4/8 buckets keep all correct traces (with 1-4/8 duplicated 2×), so no within-bucket selection is applied there.
\n\nFigure 3: Training data composition: unbalanced vs balanced sampling
The balanced version (v04, 430 examples) immediately jumped to 30/50. With the full dataset and proper balancing (v08a, 1788 examples), we reached 32/50 on PolyMath-Val-50 and 40.0 mean on competition (4 runs, SD 1.2).
\nThe controlled data composition experiment (Section 5.3) confirms that both easy and hard traces are necessary: removing hard traces is worst (30/50), removing easy traces costs 1 point (31/50), and overweighting hard traces beyond 2× provides no further benefit (31/50). The asymmetry in the bucket weights is intentional and follows from the thesis: regularization should pull the model toward conditions where it usually succeeds, so no single bucket is allowed to dominate (no bucket trains more than ~33%) and the hard bucket is not over-emphasized. Over-weighting the 1-4/8 bucket past 2× would push the model to imitate its rare-success patterns at the expense of its typical-success patterns.
\nThe thesis that SFT acts as behavioral regularization rather than capability transfer is supported by several lines of evidence:
\n\nFigure 4: Temperature reduction vs SFT as two routes to lower variance. Both T=0.8 and SFT reach SD 1.2, but T=0.8 loses 2.5 points on the mean (37.3 vs 39.8) while SFT raises it to 40.0.
The 9 \"wobbly\" problems (those that differentiate model variants) sit at the boundary where the base model has a ~50% chance of finding the correct approach through 8 attempts. SFT pushes these past the threshold. The wobbly score predicts the total: each additional wobbly problem solved adds one point to the score (e.g., 27 + 2 wobbly = 29/50, 27 + 5 = 32/50).
\n\nFigure 5: Problem stability across SFT variants on Kaggle. Problems are sorted along the y-axis by difficulty bucket (easy → hard) and within bucket by question type (Algebra, Combinatorics, Geometry, Number Theory) to make per-problem patterns legible. The actual processing order at inference time was effectively randomized across runs (Section 3, item 5).
\nFigure 6: Problem stability, RunPod multi-seed comparison. Same y-axis convention as Figure 5: sorted by difficulty and question type for visualization; the actual processing order at inference was randomized.
Attention-only LoRA: full-linear LoRA on a MoE model produces a 4.9 GB adapter incompatible with vLLM's [10] MoE LoRA support. Attention-only (q/k/v/o_proj) produces a 91 MB adapter that works. This targets 144 tensors: the shared attention layers that route information across all experts.
\n| Parameter | \nValue | \nWhy | \n
|---|---|---|
| Target modules | \nq/k/v/o_proj | \nOnly attention layers work with vLLM MoE; 144 tensors, 91 MB | \n
| Rank | \n32 | \nRank 32 solved Norwegian integers (PID 3994) where rank 16 couldn't; 31/50 vs 30/50 | \n
| Alpha | \n32 (scaling = 1.0) | \nStandard choice for rank = alpha | \n
| Epochs | \n1 | \n2 epochs overtrained (v05a), losing problems that 1 epoch retained | \n
| Batch size | \n8 | \n— | \n
| LR | \nScaled per data size | \nSee Section 5.2 | \n
When we doubled training data from 430 to 794 examples at the same LR (2e-4), performance dropped (v06: 30/50). Halving the LR restored it (v06b: 32/50). The total gradient movement (steps × LR) must stay roughly constant.
| Version | \nExamples | \nSteps | \nLR | \nsteps × LR | \nPolyMath-Val-50 | \n
|---|---|---|---|---|---|
| v05c | \n430 | \n51 | \n2e-4 | \n0.010 | \n31/50 | \n
| v06b | \n794 | \n100 | \n1e-4 | \n0.010 | \n32/50 | \n
| v08a | \n1788 | \n224 | \n4e-5 | \n0.009 | \n32/50 | \n
| v06 (too high) | \n794 | \n100 | \n2e-4 | \n0.020 | \n30/50 | \n
Our rule: keep steps × LR ≈ 0.01. This held across all successful runs.
With the full 1,071-problem dataset, we ran a controlled experiment varying data composition. All runs: rank 32, 1 epoch, batch 8, attention-only, LR scaled to keep steps × LR ≈ 0.01.
\n| Run | \nData change | \nExamples | \nLR | \nPolyMath-Val-50 | \n
|---|---|---|---|---|
| v08a | \nAll buckets, balanced | \n1788 | \n4e-5 | \n32/50 | \n
| v08b | \nRemove 8/8 (easy) bucket | \n1156 | \n6e-5 | \n31/50 | \n
| v08c | \nRemove 1-4/8 (hard) bucket | \n1402 | \n6e-5 | \n30/50 | \n
| v08d | \n4× hard upweight | \n2174 | \n3e-5 | \n31/50 | \n
\nFigure 7: Data composition ablation
Findings:
\nA distinctive aspect of our approach is the use of Tinker [4] (Thinking Machines' fine-tuning API for GPT-OSS) as our primary training platform. Tinker was a sponsor of the AIMO 3 competition, yet few participants reported success with fine-tuning through it. Our pipeline (train on Tinker, download adapter, merge on CPU, deploy via vLLM) proved reliable and cost-effective (~$13 per training run, ~$191 total).
\nTinker imposes a 32K context cap on training examples. Since inference runs at 65K context, this means some problems whose correct traces exceed 32K tokens are excluded from training. Critically, this filtering is not uniform across difficulty levels (Figure 8): at 32K, 57% of (5-6)/8 problems (70/123) and 39% of (1-4)/8 problems (48/124) are filtered out, compared to under 1% of (8)/8 problems. Harder problems produce longer traces (more computation steps, more error-correction loops) and are disproportionately lost to the context cap. Rather than treating this as a limitation, we explored whether longer context during training actually helps, using Unsloth [5] (QLoRA on RunPod H100) at 48K and 65K context.
\n\nFigure 8: Context window cap impact by difficulty bucket. The 32K cap (Tinker) disproportionately filters out the medium and hard buckets, whose correct traces tend to be longer.
| Version | \nContext | \nPlatform | \nData variant | \nExamples | \nRunPod Val-50 | \nKaggle Val-50 | \nCompetition | \n
|---|---|---|---|---|---|---|---|
| v12 (base) | \n— | \nno LoRA | \n— | \n— | \n31, 30 | \n32 (×4 runs) | \n39.8 mean (6 runs, SD 2.0) | \n
| SFT v08a | \n32K | \nTinker | \nbalanced | \n1,788 | \n30, 30 | \n32 | \n40.0 mean (4 runs, SD 1.2) | \n
| SFT v08a | \n32K | \nUnsloth | \nbalanced | \n1,788 | \n32 | \n— | \n— | \n
| SFT v11a | \n48K | \nUnsloth | \nbalanced | \n2,106 | \n— | \n33 | \n39.0 mean (2 runs, SD 2.83) | \n
| SFT v11c | \n65K | \nUnsloth | \n4× hard upsample | \n2,572 | \n— | \n33, 29 | \n— | \n
| SFT v11d | \n65K | \nUnsloth | \nv08a as-is | \n2,223 | \n31 | \n— | \n— | \n
| SFT v11e | \n65K | \nUnsloth | \nsmart upsample | \n2,400 | \n31 | \n— | \n— | \n
| SFT v11f | \n65K | \nUnsloth | \nno (1-4)/8 bucket | \n2,139 | \n— | \n31 | \n— | \n
Figure 9 visualizes the competition-data column of the table for the two main SFT candidates against the base model. The v08a / v11a comparison was load-bearing for our final-submission decision: v11a's single high run (41) is matched by a low run (37, below any v08a run), giving it a higher SD (2.83) over only 2 runs. v08a's tighter cluster across 4 runs made it the safer choice under the private-LB two-run averaging.
\n\nFigure 9: SFT variants on competition data. v08a (Tinker 32K) clusters tightly at mean 40.0 (SD 1.2 over 4 runs), while v11a (Unsloth 48K) spans 37-41 (SD 2.83 over 2 runs). v11a's higher upside is offset by a worse downside, making v08a the safer selection under two-run private-LB averaging.
\nFigure 10: Training context length vs performance
The results tell a nuanced story about context length and training platform.
\nLonger context introduces noise rather than helping. Four independent 65K variants (v11d, v11e, v11f, plus v11c repeat) all scored 31/50 or below on Kaggle. The single 48K run (v11a) scored 33/50, but when we repeated a 65K model (v11c), it swung from 33/50 to 29/50 on the same problems, revealing large run-to-run variance in the Unsloth-trained models.
\nThe platform/context confound is partially resolved. Our initial comparison confounded two variables: context length and training platform. Tinker uses full-precision LoRA on Thinking Machines' infrastructure; Unsloth uses 4-bit NF4 QLoRA with an SDPA attention patch on RunPod H100s. To isolate these factors, we trained an Unsloth model at 32K context with the same v08a data and learning rate.
\nThe result was striking: Unsloth at 32K scored 32/50 on RunPod, matching the Kaggle-level performance of the Tinker model, and 2 points above Tinker's own RunPod score (30/50). At the problem level, Unsloth 32K solved two problems (PID 480, PID 2002) that Tinker consistently missed on RunPod across both seeds. This suggests that the 32K context length, not the training platform, is the key factor in producing effective adapters. QLoRA at matched context length performs at least as well as full-precision LoRA, and possibly better.
\nThis was a late-stage discovery (post-competition). With only a single RunPod run for Unsloth 32K, we cannot make definitive claims; our variance analysis shows that single-run differences of 2 points can be noise. However, the fact that it matches Kaggle-level scores despite the consistent 1-2 point RunPod offset observed for all other models is suggestive. If this result holds across multiple seeds, it would imply that our competition submission (Tinker 32K) was a conservative but not optimal choice, and that Unsloth QLoRA at 32K context may offer both lower cost (~$15 vs ~$13 per run) and comparable or better performance.
\nWhat we can state with confidence: 32K training context produces the best adapters regardless of platform, and our competition choice of Tinker was justified by its proven stability (SD 1.2 across 4 competition runs).
\nAttention-only LoRA enables an efficient merge strategy. We extract the 144 base attention tensors, add LoRA deltas (W' = W + B @ A * scaling) in float32, cast back to BF16, and save as a single 1.78 GB safetensors shard. The model's safetensors index is modified to point attention keys at the new shard, and symlinks reconstruct the full model directory at inference time.
# Sparse merge: 144 attention tensors, 14 seconds on CPU\nATTN_KEYS = [\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"]\nfor name in base_tensor_index:\n if any(k in name for k in ATTN_KEYS):\n base_w = load_base_tensor(name).float()\n lora_a = adapter[f\"base_model.model.{name}.lora_A.weight\"]\n lora_b = adapter[f\"base_model.model.{name}.lora_B.weight\"]\n merged = base_w + (lora_b.float() @ lora_a.float()) * (alpha / rank)\n new_shard[name] = merged.to(torch.bfloat16)\nMerge takes 14 seconds on Kaggle CPU. No --enable-lora flag needed: vLLM loads the merged model as a normal model with full KV cache (108K tokens vs 48K with LoRA adapter). Our first LoRA competition submission used --enable-lora and timed out at 5 hours due to 37% inference overhead.
Before turning to fine-tuning, we exhaustively explored inference-side improvements. Every experiment changed exactly one variable from the v12 baseline (32/50 PolyMath-Val-50, 39.8 competition mean across 6 runs).
\n| Version | \nChange | \nIndividual scores | \nMean | \nVerdict | \n
|---|---|---|---|---|
| v12 | \nbaseline (8 attempts, T=1.0) | \n39, 39, 40, 43, 37, 41 | \n39.8 | \n— | \n
| v16 | \n8→6 attempts | \n36 | \n36.0 | \nFewer attempts = less diversity | \n
| v17 | \nminimalist prompt | \n38 | \n38.0 | \nVerbose prompt is necessary priming | \n
| v22 | \ntighter prompt (\"don't brute force\") | \n38 | \n38.0 | \nSuppresses TIR code usage | \n
| v25 | \nT=0.8 | \n38, 36, 38 | \n37.3 | \nDiversity > consistency | \n
| v18 | \n+4 attempts on split votes | \n37 | \n37.0 | \nVote dilution: wrong samples strengthen wrong majority | \n
| v23 | \njupyter_timeout 6→15s | \n32 (val only) | \n— | \nWrong code doesn't become right with more time | \n
| v26 | \nconditional 12 attempts | \n34 | \n34.0 | \nTime cost outweighs marginal gain | \n
| v27 | \nsequential 4+4 batches | \n32 (val only) | \n— | \nGPU contention not the bottleneck | \n
\nFigure 11: Inference-side ablations: competition scores
Why every inference modification failed: the model's failure modes are deterministic. When it gets a problem wrong, all 8 attempts converge on the same wrong approach regardless of prompt, temperature, or number of samples. This was confirmed by v19's diagnostic analysis: across 245 attempts on 50 problems, no metadata signal (entropy, wall time, error count, code/answer correlation) distinguishes correct from incorrect attempts within a problem. All apparent correlations were cross-problem difficulty confounds.
\nThis exhaustive exploration gave us confidence that the v12 inference configuration was the right baseline on which to build our SFT approach. The bottleneck is the model's reasoning distribution, not the inference pipeline.
\nWe spent 10 iterations trying to improve answers after the initial vote:
\n| Version | \nApproach | \nResult | \nWhy it failed | \n
|---|---|---|---|
| v13 | \nCode-based verification | \n0/6 | \nModel rubber-stamps its own wrong answers | \n
| v13_dual | \nMixed temperature (T=0.3 + T=1.0) | \n0/6 | \nT=0.3 converges on same wrong answers | \n
| v14 | \nCounter-proposals from verification code | \n0/6 | \nSingle counter can't override majority | \n
| v15 | \nTrace analysis for discriminating test cases | \n— | \nRevealed the problem but not a solution | \n
| v17 | \nGenSelect round-robin (AIMO-2 style) | \n0/18 matchups | \nModel can't adjudicate between competing solutions | \n
| v18 | \nExtended voting (+4 attempts) | \n37/50 | \nMore wrong samples = stronger wrong majority | \n
| v19 | \nSignal-based reweighting | \n32/50 (val) | \nNo signal works (see 6.1) | \n
| v20 | \n4 diverse parallel prompts | \n23/50 | \nAll prompts converge on same wrong answer | \n
| v21 | \n\"Your answer is WRONG, re-solve\" + minority flip | \n34/50 (net -1) | \nThe one flip was incorrect | \n
The key insight: the model CAN sometimes find the correct answer when told \"your answer is wrong\" (v21, Prompt B found the correct answer to the Norwegian integers problem in 1/3 tries). But it cannot reliably verify which answers are correct. Self-verification is fundamentally broken: the model confirms its own wrong answers with high confidence. This is consistent with recent findings on LLM self-verification [7] and NVIDIA's experience that GenSelect requires purpose-trained selection models [3].
\nFuture work: an efficient external verifier (one that can reliably reject wrong answers without re-solving) could unlock the sequential self-correction approach. The Norwegian integers experiment (where Prompt B's \"assume wrong, re-solve\" found the correct answer 8687 from scratch) demonstrates the potential.
\nConclusion: no post-hoc mechanism we tried could fix deterministic reasoning failures. Only training changes the reasoning distribution. This is what motivated our transition to fine-tuning.
\nAfter establishing that SFT helps, we hypothesized that DPO (Direct Preference Optimization) could further sharpen the model's preference for correct reasoning paths. DPO was a viable approach in AIMO-2 [8], where the 2nd-place team used SFT→DPO successfully on a 14B model.
\nSetup: starting from v08a's SFT checkpoint, we trained DPO with 517 preference pairs (wrong trace vs shortest correct trace for the same problem), beta=0.1, LR=1e-5, 1 epoch. Training metrics converged well (margin=4.04, accuracy=1.0).
\nResult: PolyMath-Val-50 31/50 (-1), competition 35/50 (-5). Severe regression.
\nControl: DPO without SFT base (v09b) couldn't learn at all; margin stayed near zero, loss didn't converge. This confirms SFT is a necessary precursor, but DPO on top destroys what SFT built.
\nWhy DPO failed on GPT-OSS-120B: we believe the model is already so capable that DPO's contrast signal is too coarse. Wrong traces fail for many different reasons (wrong formula, computational error, timeout), and a single \"preferred/dispreferred\" label doesn't capture this diversity. DPO may also undo SFT's regularization effect by pulling the model too far from its natural distribution. This contrasts with AIMO-2's 14B model, which had more room for capability improvement.
\nScope caveat on preference optimization: our DPO experiment used contrasting pairs sourced entirely from the same base model's own traces (correct vs incorrect attempts from the 8-per-problem datagen). We did not explore preference optimization with strong external models (for example, using a GPT-5-class, Kimi, or GLM model as the source of preferred traces or as a judge over candidate pairs), which would be a substantively different setup. Our negative DPO result here speaks to within-distribution preference optimization on top of an already-capable model; it does not rule out PO methods that introduce a stronger external signal. If we revisit this work, that experiment is at the top of our list (see Section 9.6).
\nError trimming (v10b): we tried removing Python error blocks (tracebacks, exceptions) from training traces to give the model \"cleaner\" examples. Score dropped from 32/50 to 29/50. The model learns from seeing errors and recovery: the error-then-fix pattern is part of what it needs to reproduce at inference time.
\nSelf-correction traces (v09c): we constructed 220 synthetic correction examples: wrong trace (truncated before \\boxed{}) → pivot phrase (\"Wait, let me verify this answer by checking with a different approach.\") → full correct trace. PolyMath-Val-50: 31/50. The model found correct minority answers on 6 extra problems but couldn't flip the majority vote. The correction behavior learned in training didn't reliably transfer to the voting-based inference setting.
| Version | \nChange | \nExamples | \nLR | \nVal-50 | \nCompetition | \n
|---|---|---|---|---|---|
| v12 | \nbase (no LoRA) | \n— | \n— | \n32/50 | \n39.8 (6 runs) | \n
| v03 | \nunbalanced data | \n1006 | \n— | \n25/50 | \n— | \n
| v04 | \nbalanced, rank 16 | \n430 | \n2e-4 | \n30/50 | \n— (timed out) | \n
| v05c | \nbalanced, rank 32 | \n430 | \n2e-4 | \n31/50 | \n37.0 (3 runs) | \n
| v06 | \n2× data, same LR | \n794 | \n2e-4 | \n30/50 | \n40 | \n
| v06b | \n2× data, half LR | \n794 | \n1e-4 | \n32/50 | \n— | \n
| v07a | \n3× data | \n1333 | \n1e-4 | \n32/50 | \n37 | \n
| v08a | \nfull data, balanced | \n1788 | \n4e-5 | \n32/50 | \n40.0 (4 runs) | \n
| v08b | \nremove easy bucket | \n1156 | \n6e-5 | \n31/50 | \n— | \n
| v08c | \nremove hard bucket | \n1402 | \n6e-5 | \n30/50 | \n— | \n
| v08d | \n4× hard upsample | \n2174 | \n3e-5 | \n31/50 | \n— | \n
| v09a | \nSFT→DPO | \n368 pairs | \n1e-5 | \n31/50 | \n35 | \n
| v09c | \nSFT + correction traces | \n2444 | \n3.5e-5 | \n31/50 | \n— | \n
| v10a | \n+ pass-2 hard traces | \n1911 | \n4e-5 | \n31/50 | \n— | \n
| v10b | \n+ error code trimming | \n1941 | \n4e-5 | \n29/50 | \n— | \n
| v11a | \nUnsloth 48K | \n2106 | \n4e-5 | \n33/50 | \n39.0 (2 runs) | \n
| v11c-f | \nUnsloth 65K (×4) | \nvarious | \nvarious | \n31/50 each | \n— | \n
| Component | \nDetails | \n
|---|---|
| Inference | \nKaggle H100 (80GB), vLLM 0.16.0 (gpt-oss fork), 5-hour wall clock | \n
| Training (Tinker) | \nThinking Machines fine-tuning API, 32K context cap, ~$13/run, ~$191 total | \n
| Training (Unsloth) | \nRunPod H100, QLoRA NF4, SDPA patch required, ~$100 across 6 runs | \n
| Data generation | \nKaggle H100 (200 problems) + RunPod H100 (1,000 problems, ~$72) | \n
| Evaluation | \nPolyMath-Val-50 (primary) + 10-problem reference test (smoke check) | \n
| Merge pipeline | \nSparse attention merge: 144 tensors, 1.78 GB shard, 14s on CPU | \n
We hypothesize that large MoE models like GPT-OSS-120B have a broad distribution over reasoning strategies. For a given problem, the model \"knows\" multiple approaches: some correct, some wrong. At T=1.0, sampling from this broad distribution means each of 8 attempts may take a different path, and the wrong paths sometimes outvote the correct ones.
\nSFT on correct traces narrows this distribution toward strategies that actually work, without eliminating the diversity needed for voting to succeed. It's analogous to lowering temperature on the strategy selection (which approach to try) without lowering temperature on the execution (how to carry out the chosen approach). This explains why SFT outperforms actual temperature reduction: T=0.8 uniformly suppresses all diversity (including useful diversity on problems where the model has multiple valid approaches), while SFT selectively suppresses only the wrong strategies.
\nThe aggregate mean and SD numbers in Section 7 do not directly answer how the model behaves differently after SFT. To get a concrete behavioral signal, we compared v12 (base, no LoRA) and v08a (SFT, our final submission model) on the same 50 PolyMath-Val-50 problems, extracting per-problem timing and Python tool-call counts from the inference logs of both runs.
\nHeadline: same score, less exploration.
\n| Metric | \nv12 (base) | \nv08a (SFT) | \nΔ | \n
|---|---|---|---|
| PolyMath-Val-50 score | \n32/50 | \n32/50 | \n0 | \n
| Total compute (sum of per-problem times) | \n3h 44m | \n3h 04m | \n−17.8% | \n
| Per-problem mean time | \n269s | \n221s | \n−48s | \n
| Per-problem time stdev | \n204s | \n166s | \n−38s | \n
| Problems completing in <300s (strong early-stop indicator) | \n32 | \n38 | \n+6 | \n
The score is unchanged, but v08a finishes the same 50 problems in ~40 minutes less wall-clock time and reaches the 4/8 consensus threshold before the 300s minimum budget on 6 more problems. The per-problem time variance also narrows (204s → 166s), the same SD-narrowing effect seen in competition scores, here at a different time scale.
\nPer-problem matching by correctness transition. With the same 50 problems on both runs, every problem falls into one of four categories:
\n| Transition | \nn | \nMean Δt | \nInterpretation | \n
|---|---|---|---|
| correct → correct | \n30 | \n−35s | \nSFT confirms the same correct answer with less exploration | \n
| wrong → wrong | \n16 | \n−49s | \nSFT model also gives up sooner on problems it cannot solve | \n
| wrong → correct (gain) | \n2 | \n+10s | \nTwo problems newly solved; slightly more time spent verifying | \n
| correct → wrong (loss) | \n2 | \n−290s | \nTwo losses: fast convergence to a wrong answer | \n
The cleanest behavioral signal is the correct→correct row: on the 30 problems both versions solve, the SFT model reaches 4/8 consensus a mean 35 seconds sooner. Same final answer, fewer attempts before consensus fires. The wrong→wrong row shows the same direction: ~49 seconds sooner on problems neither version can solve. Regularization tightens the trajectory in both directions: less exploration when the model is going to be right, less exploration when it isn't.
\nThe 2/2 gain/loss split is too small for a directional claim, but the two losses are informative on their own. PIDs 2002 (Geometry) and 522 (Algebra) were correct in v12 and wrong in v08a, and completed much faster: the v08a trajectory converged to a wrong answer where the v12 trajectory eventually found the correct one with more exploration. This is the \"confident divergence\" failure mode: when regularization tightens trajectory in a direction that turns out to be wrong on a specific problem, the model commits sooner than the base version would have. It is a real cost of the approach, balanced by faster convergence on the dominant 30 + 16 stable problems.
\nPython tool usage corroborates the timing story. The same per-problem matching gives us call and error counts for the stateful Jupyter sandbox:
\nThe compute reduction comes from fewer tool calls, not from better tool calls: the model is doing less exploratory computation, not improving the quality of the computation it does. Per-transition breakdown:
\n| Transition | \nn | \nv12 calls/pb | \nv08a calls/pb | \nv12 err% | \nv08a err% | \n
|---|---|---|---|---|---|
| correct → correct | \n30 | \n107.0 | \n94.6 | \n8.4% | \n9.4% | \n
| wrong → wrong | \n16 | \n143.2 | \n115.1 | \n13.4% | \n14.6% | \n
| wrong → correct (gain) | \n2 | \n59.0 | \n59.0 | \n28.8% | \n30.5% | \n
| correct → wrong (loss) | \n2 | \n161.5 | \n76.0 | \n20.1% | \n9.9% | \n
On the 30 correct→correct problems, the SFT model averages ~12 fewer Python calls per problem and arrives at the same answer: concrete evidence of less exploration on problems it can solve. The correct→wrong loss row is striking: v12 made 161 calls per problem at a 20% error rate (clearly hunting through the sandbox) and eventually arrived at the right answer; v08a more than halved the call count (76 calls) with a much cleaner 9.9% error rate, and converged to the wrong answer. The cost of regularization on these problems is \"exhaustive but eventually correct exploration\" replaced by \"confident shortcut to the wrong answer.\"
\nPer problem-type. All four types speed up. Algebra −43s, Combinatorics −67s, Geometry −40s, Number Theory −38s (mean Δt). Number Theory has both of the score gains. Combinatorics has 14/14 problems stable in correctness despite the largest mean time reduction, a particularly clean case where regularization helped convergence without affecting correctness.
\nConnection to brevity-biased data selection. Section 4.3 describes the within-bucket selection rule for downsampled buckets: in the 8/8 and 7/8 buckets, we keep the shortest correct trace per problem rather than sampling randomly. This biases the training data toward more compact correct paths, which is a plausible mechanism for the faster inference-time convergence observed here. We did not isolate this experimentally with a counterfactual training run using random within-bucket selection, so this is a hypothesis rather than an established causal claim. But the inference-time behavior (less time, fewer Python calls, same answers on the dominant 30 + 16 stable problems) is consistent with the training-time bias.
\nCaveats. The analysis above comes from one v12 PolyMath-Val-50 run and one v08a PolyMath-Val-50 run (n=1 per side). The 30/16 stable counts and the time deltas within them carry the actual signal at this sample size; the 2/2 gain/loss split is too small for directional claims. The compute reduction (−17.8%) and the Python-call reduction (−16.7%) are large enough to be unlikely from sampling noise alone, but a multi-seed replication would establish tighter confidence on the magnitudes. The PolyMath-Val-50 problems are validation-set problems and not the competition private set, so the exact distribution shift is not guaranteed to carry to out-of-distribution problems.
\nThe thesis that SFT acts as behavioral regularization carries a corollary: the training data should stay as close as possible to the model's own natural correct trajectories. Every attempt we made to \"improve\" the training data by modification (substituting traces from a stronger source, removing intermediate errors, synthesizing self-correction patterns, or applying preference signals on top) regressed. We summarize these here as one mechanism, because they tell a single story.
\nExternal source, same format (NuminaMath). SFT on NuminaMath-TIR (Section 4.1), a dataset of high-quality math solutions generated by GPT-4, completely suppressed code generation: 0 of 400 attempts used Python after training. The dataset itself is strong; the failure is about format mismatch. NuminaMath's text-heavy single-pass format overwrote the model's tool-integrated reasoning behavior. Source strength does not compensate when the training data does not match the format the model will produce at inference.
\nRemoving intermediate errors (v10b). We tried filtering Python error blocks (tracebacks, exceptions) out of training traces on the intuition that \"cleaner\" examples would teach the model better behavior. PolyMath-Val-50 dropped from 32/50 to 29/50 (Section 6.4). Errors are not noise to be removed; the error-then-fix pattern is part of how the model recovers at inference, and the training trace has to include it.
\nSynthesizing self-correction patterns (v09c). We constructed 220 examples of \"wrong trace, pivot phrase, correct trace\" to teach the model to flip from an incorrect answer to a correct one mid-trajectory (Section 6.4). The learned behavior did not transfer to the voting-based inference setting: the model could find correct minority answers on a few problems but could not change the majority vote. Synthesizing a behavior the base model does not exhibit naturally did not survive contact with our actual inference pipeline.
\nPreference signal on top of SFT (DPO, v09a). Training DPO on top of v08a with contrasting pairs sourced from the base model's own traces regressed competition score from 41 to 35 (Section 6.3). Even within-distribution modification beyond plain SFT, pushing the model preferentially toward shorter correct traces and away from wrong ones, degraded behavior. The model is already capable; the contrast signal appears to undo SFT's regularization by pulling further from the natural distribution than is helpful.
\nThe unifying interpretation. SFT on self-generated correct traces works because it nudges the model toward more reliable patterns that it already produces. Each of the modifications above introduces some asymmetry between training and the model's natural inference behavior: a foreign format, an absent error-recovery pattern, a synthesized transition the base model would not generate, or a contrasting push beyond what plain reinforcement provides. We did not try every conceivable trace modification, but the consistency of the negative results across four very different attempts makes us confident that trace integrity, in the sense of keeping the training set as a faithful subset of the model's own correct trajectories, is a load-bearing design choice rather than an incidental one. Reviewers asked whether stronger filtering or stronger preference signals might help; based on these results, we expect them to require care. The filter would need to separate hallucinated reasoning from legitimate error-and-recovery patterns, and the preference signal would likely need an external strong-model source rather than within-distribution pairs (see Section 9.6 for both as follow-ups).
\nSingle-model approach: we used only GPT-OSS-120B. Ensembling with a second model could improve diversity, but the H100 VRAM constraint makes this challenging.
\nTinker 32K context cap: training traces are truncated to 32K tokens, while inference runs at 65K. Some hard problems produce 50K+ token traces that are excluded from training. This may explain why the 13 always-wrong problems don't improve: their correct traces may be too long to train on.
\nPlatform/context confound partially resolved: our Unsloth 32K experiment (Section 5.4) suggests context length, not training platform, is the key variable. However, this finding rests on a single RunPod run; multiple seeds are needed for confirmation.
\nLimited competition runs: v08a has 4 competition data points. While SD 1.2 is encouraging, 4 points cannot definitively establish low variance.
\nThe project spanned ~7 weeks (March 1 – April 15, 2026):
\nWe thank the AIMO Progress Prize organizers and XTX Markets for designing a competition that rewards reproducibility and open science. We acknowledge Thinking Machines for offering Tinker credits, which were crucial for our SFT experiments. We utilized Claude Code (Opus 4.6) by Anthropic throughout this project for assistance with code development, data analysis, and drafting the report.
\n[1] OpenAI. GPT-OSS-120B. https://huggingface.co/openai/gpt-oss-120b
\n[2] Dekoninck, J., Hiss, H., Petrov, I., Rögnvaldsson, K., & Sun, C. PolyMath: A curated dataset of 11,000 high-difficulty problems. AIMO 3 Math Corpus Prize submission. https://huggingface.co/datasets/AIMO-Corpus/PolyMath
\n[3] Moshkov, I. et al. AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning dataset. arXiv:2504.16891, 2025.
\n[4] Thinking Machines. Tinker: Fine-tuning API for GPT-OSS. https://www.thinkingmachines.ai/
\n[5] Unsloth. QLoRA training framework. https://github.com/unslothai/unsloth
\n[6] Bisiadis, A. AIMO 3 GPT-OSS-120B with Tools. https://www.kaggle.com/code/andreasbis/aimo-3-gpt-oss-120b-with-tools
\n[7] Bialas, P. et al. Can Language Models Falsify? Towards Robust Mathematical Reasoning. arXiv:2412.15184, 2024.
\n[8] imagination-research. AIMO-2 2nd Place Solution: SFT→DPO pipeline. https://github.com/imagination-research/aimo2/
\n[9] Frieder, S. AIMO 3 Problem Difficulty Analysis. https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/679559
\n[10] vLLM Project. vLLM: Easy, fast, and cheap LLM serving. https://github.com/vllm-project/vllm (gpt-oss fork v0.16.0 required)
\n[11] nihilisticneuralnet. [44/50] Let Me (over)COOK!! https://www.kaggle.com/code/nihilisticneuralnet/44-50-let-me-over-cook
", "rawMarkdown": "**Submission basis**: Public LB score: 40.0/50 (average over 4 runs). Private LB score: 42.0/50 (average over 2 runs)\n\n**Resources**:\n- **Kaggle notebook**: https://www.kaggle.com/code/ofidaner/aimo-v036\n- **HuggingFace** (adapter, training data, scripts, dependencies): https://huggingface.co/ofidaner/aimo3-gpt-oss-120b-sft-v08a\n- **Google Colab** (A100 80GB inference notebook): https://colab.research.google.com/drive/1O3FXBR3EozCWlQrP8B--IFE2RBXSaq0P\n- **Kaggle model**: https://www.kaggle.com/models/ofidaner/gpt-oss-120b-merged-v08a\n- **Kaggle datasets**: [ofidaner/aimo-vllm-wheels](https://www.kaggle.com/datasets/ofidaner/aimo-vllm-wheels), [ofidaner/aimo-tiktoken](https://www.kaggle.com/datasets/ofidaner/aimo-tiktoken)\n- **Competition overview**: https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/overview\n- **Competition data**: https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/data\n\n---\n\n## Abstract\n\nOur solution for AIMO Progress Prize 3 competition fine-tunes attention layers on 1,788 self-generated Tool-Integrated Reasoning (TIR) traces with difficulty-balanced sampling. The core idea is promoting **behavioral regularization** through supervised fine-tuning (SFT) on GPT-OSS-120B, rather than teaching this Mixture of Experts (MoE) model new capabilities. GPT-OSS-120B is already capable of solving most of the questions in the competition, albeit not reliably. In our approach, fine-tuning promotes correct solution traces that were already generated by GPT-OSS-120B. As a result, the model doesn't learn new solution methods, but converges more reliably on correct ones it already knows. This manifests as significantly reduced run-to-run variance (SD 1.2 for SFT model vs 2.0 for the base model, both on the public data set) and slightly higher mean (40.0 for SFT vs 39.8 for the base model). Our model scored 42.0/50 on the private data set (averaged over two runs), indicating that our SFT efforts did not cause overfitting. Beyond variance reduction, we measure concrete behavioral changes at inference: the SFT model uses ~18% less compute and ~17% fewer Python tool calls while reaching the same answers on the same problems (Section 9.2), consistent with regularization tightening reasoning trajectories rather than expanding capability. We support our findings with systematic ablation studies across data composition, learning rate scaling, training context length, DPO, verification strategies, and prompt engineering; most of which produced regressions. We also document 10 iterations of post-vote verification approaches and their impact on performance. We believe SFT can be an efficient tool for guiding large MoE models when solving complex questions and our approach demonstrates an interesting use case of SFT on GPT-OSS-120B.\n\n## 1. Introduction\n\nGPT-OSS-120B [1] is a 120B-parameter Mixture-of-Experts (MoE) model with only 5.1B parameters active per token (128 experts, 4 active). In Microscaling FP4 (MXFP4) quantization it occupies ~65GB, fitting a single H100 with room for 108K tokens of key-value (KV) cache. It was trained with **native tool-use** and so the model knows how to call Python, read execution results, and iterate. These capabilities make the model uniquely suited for tool-integrated reasoning (TIR), which is essential for competition-level math where brute-force computation and symbolic verification complement mathematical reasoning. Not surprisingly, GPT-OSS-120B has been the most popular model among AIMO 3 participants. It is arguably the most capable model for solving olympiad-level questions that can fit in the 80GB VRAM of H100, while leaving room for KV cache across multiple parallel attempts.\n\nOne of the apparent challenges of fine-tuning a model such as GPT-OSS-120B is that the model is already highly capable in mathematical reasoning. The original training likely included a much larger and more diverse dataset than what we can practically use for fine-tuning, leaving limited room for improvement under small fine-tuning budgets. In fact, many AIMO 3 participants reported diminished performance, or no change, as a result of fine-tuning.\n\nWith these observations in mind, we took a different approach to SFT. Rather than attempting to teach the model new solution methods, we used SFT to **regularize its behavior**, narrowing the distribution of reasoning strategies toward ones that actually work. In our workflow, we first designed an inference pipeline using the base model. Then we ran the base model on 1,200 olympiad-level problems using the exact same inference pipeline (8 parallel attempts per problem at T=1.0), keeping full traces generated by the model. We used the fraction of correct attempts as a proxy for problem difficulty, e.g., 8/8 correct would be an easy problem, and 1/8 would be a hard problem. We kept only the correct traces as training data. In this way, the training set consists entirely of what the model organically produces when it succeeds. The goal is not to expand the model's capabilities, but to increase the odds that on hard problems (where perhaps only 2 or 3 out of 8 base-model attempts find the correct answer) the fine-tuned model will find it more reliably, pushing that ratio to 4 or 5 out of 8.\n\nThe base model has high **variance** (SD 2.0, range 37–43 across 6 competition runs), and fine-tuning's primary effect is reducing that variance. This matters for the competition scoring formula for the private data set, which averages two independent runs: a model with mean 40 and SD 1 outperforms a model with mean 40 and SD 2, because a lower variance ensures the model is much less likely to have a bad run that drags down the average and hence the final score on the private data set. On the private data set the SFT model averaged 42.0/50 over two runs, slightly above the 40.0 public-LB mean, indicating the SFT model does not overfit to the public data set. Beyond run-to-run variance, the SFT model also shows concrete behavioral changes at inference time (~18% less total compute and ~17% fewer Python tool calls on the same problems while reaching the same answers, Section 9.2), supporting the regularization interpretation over a capability-improvement interpretation.\n\nIn this paper we describe the components and design considerations for the eventual model we submitted for the private leaderboard scoring. We also designed several experiments to determine optimality of this solution, which will be discussed in more detail in the \"Ablation Studies\" section. Details on how to replicate results both on Kaggle and elsewhere (e.g. Colab, RunPod) are available on HuggingFace [link above].\n\n## 2. Evaluation Methodology\n\n### 2.1 Competition data sets\n\nThe competition offered daily attempts to solve the \"public data set\", consisting of 50 problems. The final evaluation was on a \"private data set\" with a different set of 50 problems. The competition hosts characterized the difficulty distribution of both sets using state-of-the-art open-source models [9]. Based on their analysis, approximately 27 problems can be solved in a single pass (pass@1), about 38 can be solved within 3 passes, and roughly 42 within 5 passes. This leaves approximately 8 problems that require significantly more attempts or are beyond the models' reliable reach.\n\nFor GPT-OSS-120B specifically, we estimate that approximately 35 problems are reliably solvable (correct on most runs), while about 7-9 problems have varying degrees of success: sometimes correct, sometimes not, depending on sampling luck. This estimate is based on our competition runs, where we observed a maximum score of 43 and a minimum of 37 across 6 submissions with the base model (no fine-tuning). Our inference pipeline solves each problem with 8 parallel attempts at T=1.0 and selects the answer by entropy-weighted majority vote (detailed in Section 3).\n\n| Model | Competition runs | Mean | SD |\n|---|---|---|---|\n| Base model (GPT-OSS-120B) | 39, 39, 40, 43, 37, 41 | 39.8 | 2.0 |\n\nEdited on 05/17/2026 for the rebuttal phase:
\nI've published an updated version of the writeup incorporating the feedback from @philipvonderlind and @sambealings:\nThe \"large MoE\" framing for GPT-OSS-120B has been toned down throughout; a new Section 9.3 (\"Trace integrity\") consolidates evidence that self-generated, unmodified traces were load-bearing for our approach, addressing the trace-quality and PO-target questions; Section 3 verifies from run logs that PolyMath-Val-50 problem ordering was effectively randomized and the time budget never bound across the three runs we inspected; Section 4.3 now describes the brevity-selection within difficulty buckets; and a new Section 9.2 presents concrete behavioral evidence of regularization, showing that the SFT model uses ~18% less compute and ~17% fewer Python tool calls while reaching the same answers on the same 50 problems. I've also added the private leaderboard result (42.0/50 averaged over two runs) to the abstract and Section 2.1. Lastly, I made modifications to the figures and the text overall to help with the flow and improve readability.
\nMany thanks to Philip and Sam for the careful review.
", "rawMarkdown": "Edited on 05/17/2026 for the rebuttal phase:\n\nI've published an updated version of the writeup incorporating the feedback from @philipvonderlind and @sambealings:\nThe \"large MoE\" framing for GPT-OSS-120B has been toned down throughout; a new Section 9.3 (\"Trace integrity\") consolidates evidence that self-generated, unmodified traces were load-bearing for our approach, addressing the trace-quality and PO-target questions; Section 3 verifies from run logs that PolyMath-Val-50 problem ordering was effectively randomized and the time budget never bound across the three runs we inspected; Section 4.3 now describes the brevity-selection within difficulty buckets; and a new Section 9.2 presents concrete behavioral evidence of regularization, showing that the SFT model uses ~18% less compute and ~17% fewer Python tool calls while reaching the same answers on the same 50 problems. I've also added the private leaderboard result (42.0/50 averaged over two runs) to the abstract and Section 2.1. Lastly, I made modifications to the figures and the text overall to help with the flow and improve readability.\n\nMany thanks to Philip and Sam for the careful review." }, { "id": 3458587, "postDate": "2026-05-16T08:12:30.247Z", "content": "Following up on the questions from @philipvonderlind (and thanks for the very thorough response), here are some additional questions to help us assess your submission:
\nLook forward to reading your responses on these.
", "rawMarkdown": "Following up on the questions from @philipvonderlind (and thanks for the very thorough response), here are some additional questions to help us assess your submission:\n\n1. **Difficulty sampling for training traces:** Could you describe how you sampled the 1,788 self-generated training traces with respect to problem difficulty? For instance, were they drawn uniformly across difficulty levels, biased toward harder problems (where correct base-model traces are rarer), or biased toward easier ones?\n2. **What \"behavioral regularization\" looks like in practice:** From comparing SFT vs base traces on the same problems, can you point to a concrete behavioural difference you've observed (e.g. \"the SFT model now consistently uses tool X for problem type Y, while base did so only sometimes\"), or is the framing currently supported mainly by the aggregate mean / SD numbers?\n\nLook forward to reading your responses on these.", "replies": [ { "id": 3458726, "postDate": "2026-05-16T14:17:56.827Z", "content": "Thanks for the follow-up. Both questions are very helpful, and here are my answers:
\nThe sampling is balanced across difficulty buckets. Difficulty is measured by the base model's vote share — the fraction of 8 base-model attempts (T=1.0) that produce the correct answer, ranging from 1/8 (hard) to 8/8 (easy). The bucket table from Section 4.3 of the writeup:
\n| Difficulty bucket | \nProblems | \nSampling strategy | \nTraining share | \n
|---|---|---|---|
| 8/8 (easy) | \n~350 | \n1 trace each | \n~27% | \n
| 7/8 | \n~150 | \n2 traces each | \n~14% | \n
| 5-6/8 (medium) | \n~280 | \nall traces | \n~33% | \n
| 1-4/8 (hard) | \n~260 | \nall traces × 2 | \n~26% | \n
We downsample the easy 8/8 bucket and upsample the hard 1-4/8 bucket. Downsampling is done based on brevity; i.e., in the 8/8 bucket we keep the 1 shortest correct trace per problem and in the 7/8 bucket we keep the 2 shortest traces. In downsampling, the idea was that shorter correct traces tend to represent more compact CoT paths with less rambling. Otherwise, the motivation for the balancing of hard/medium/easy traces comes from an early experiment: v03 was unbalanced (59% of training data came from 8/8 easy problems, simply because they generate more correct traces per problem) and scored 25/50 on PolyMath-Val-50, a 7-point regression below the base model's 32/50. v04 (same data sources, balanced) immediately rose to 30/50. v08a (1,788 traces, fully balanced) reached 32/50.
\nThe Section 5.3 ablation confirms both extremes hurt: removing the 8/8 (easy) bucket cost 1 point (v08b), removing the 1-4/8 (hard) buckets cost 2 points (v08c), and 4× upsampling the hard bucket provided no further benefit (v08d, 31/50). Hence the eventual choice was an optimal mix of easy/medium/hard questions.
\nIn summary, we sample to expose the model to its full range of typical reasoning, not just hard problems. Over-weighting hard problems would bias the regularization toward conditions where the model might overcomplicate its CoT and fail as a result, pushing the reasoning distribution toward less-reliable strategies. The balanced sampling keeps the model's nominal range across a question difficulty scale, which we found to be important to not regress in otherwise solvable problems while attempting to solve harder problems. For easier problems there is a bias towards brevity. I will be adding some additional discussion and clarification in the relevant sections in the writeup.
\nThis is a very good question, and I should have included more discussion on that. As you point out, the writeup leans heavily on aggregate mean/SD numbers without a clean measurement of behavioral change. To address this, I ran a post-hoc analysis matching the v12 (base, no SFT) and v08a (SFT, the final submission model) runs at the per-problem level on the same 50 PolyMath-Val-50 validation set problems. The results are very interesting:
\n| Metric | \nv12 (base) | \nv08a (SFT) | \nΔ | \n
|---|---|---|---|
| PolyMath-Val-50 score | \n32/50 | \n32/50 | \n0 (no capability change) | \n
| Total compute (sum of per-problem times) | \n3h 44m | \n3h 04m | \n−17.8% | \n
| Per-problem mean time | \n269s | \n221s | \n−48s | \n
| Per-problem time stdev | \n204s | \n166s | \n−38s | \n
| Problems completing in <300s (strong early-stop indicator) | \n32 | \n38 | \n+6 | \n
Same problems solved, ~40 minutes less total compute, and 6 more problems hit the 4/8-consensus threshold before exhausting the 300s budget floor. So even though the score didn't necessarily improve, the behavior certainly changed.
\nWhere the time saving comes from:
\n| Transition | \nn problems | \nMean Δt | \nWhat it shows | \n
|---|---|---|---|
| correct → correct | \n30 | \n−35s | \nSFT model confirms the same correct answer with less exploration | \n
| wrong → wrong | \n16 | \n−49s | \nSFT model also gives up sooner on problems it cannot solve | \n
| wrong → correct (gain) | \n2 | \n+10s | \nTwo problems newly solved; slightly more time spent verifying | \n
| correct → wrong (loss) | \n2 | \n−290s | \nTwo losses — fast convergence to a wrong answer | \n
The cleanest behavioral signal is the correct→correct row: on the 30 problems both models solve, the SFT model reaches 4/8 consensus a mean 35 seconds sooner. Same final answer, fewer attempts needed before consensus fires. The wrong→wrong row shows a parallel pattern — the SFT model recognizes hopeless problems faster, ~49s sooner on average. So the regularization tightens the trajectory in both directions: less exploration when the model is going to be right, and less exploration when it isn't. This may be attributable to the downsampling approach in easier questions.
\nConcrete correct→correct speedups (PID, type, v12 time → v08a time):
\nThe 2/2 gain/loss split shows individual problems can flip in either direction. The 2 losses (PIDs 2002 in Geometry and 522 in Algebra) are the \"confident divergence\" failure mode — SFT converges quickly to a wrong answer. These are not a directional signal at this sample size, but I believe it shows that SFT shifts the reasoning behavior.
\nBehavior Change regarding Python tool usage The same per-problem matching also gives us call and error counts for the stateful Jupyter sandbox:
\nThe model issues fewer calls overall, with roughly the same error rate per call. Therefore one can say the regularization is correlated with less exploratory tool use. I think this is an interesting future research direction in this line of thought. It will be interesting to try SFT that not only reduces tool use, but also increases its usefulness. Here is the per-transition breakdown:
\n| Transition | \nn | \nv12 calls/pb | \nv08a calls/pb | \nv12 err% | \nv08a err% | \n
|---|---|---|---|---|---|
| correct → correct | \n30 | \n107.0 | \n94.6 | \n8.4% | \n9.4% | \n
| wrong → wrong | \n16 | \n143.2 | \n115.1 | \n13.4% | \n14.6% | \n
| wrong → correct (gain) | \n2 | \n59.0 | \n59.0 | \n28.8% | \n30.5% | \n
| correct → wrong (loss) | \n2 | \n161.5 | \n76.0 | \n20.1% | \n9.9% | \n
On the 30 correct→correct problems, the SFT model averages ~12 fewer Python calls per problem and arrives at the same answer — clean evidence of less exploration on problems it can solve. On the 2 correct→wrong losses, v12 was doing 161 calls at a 20% error rate (clearly fishing) and eventually finding the right answer; v08a more than halves the call count (76 calls, much cleaner 9.9% error rate) and produces the wrong answer. So on the rare losses, SFT replaced \"exhaustive but eventually correct exploration\" with \"confident shortcut to the wrong answer.\" This is one metric to address to improve overall performance with further SFT.
\nPer problem-type (timing): all four types speed up. Algebra −43s, Combinatorics −67s, Geometry −40s, Number Theory −38s (mean Δt per type). Number Theory has both of the score gains; Combinatorics has zero score changes (14/14 stable) despite the largest mean time reduction. Hence, regularization helps convergence without affecting correctness across problem types.
\nOnce again thank you. I'll be adding these findings to the updated writeup. Once I finish the writeup, I'll post a short summary of changes on Simon's discussion thread.
", "rawMarkdown": "Thanks for the follow-up. Both questions are very helpful, and here are my answers:\n\n### Q1. Difficulty sampling for training traces\n\nThe sampling is **balanced** across difficulty buckets. Difficulty is measured by the base model's vote share — the fraction of 8 base-model attempts (T=1.0) that produce the correct answer, ranging from 1/8 (hard) to 8/8 (easy). The bucket table from Section 4.3 of the writeup:\n\n| Difficulty bucket | Problems | Sampling strategy | Training share |\n|---|---|---|---|\n| 8/8 (easy) | ~350 | 1 trace each | ~27% |\n| 7/8 | ~150 | 2 traces each | ~14% |\n| 5-6/8 (medium) | ~280 | all traces | ~33% |\n| 1-4/8 (hard) | ~260 | all traces × 2 | ~26% |\n\nWe **downsample** the easy 8/8 bucket and **upsample** the hard 1-4/8 bucket. Downsampling is done based on brevity; i.e., in the 8/8 bucket we keep the 1 shortest correct trace per problem and in the 7/8 bucket we keep the 2 shortest traces. In downsampling, the idea was that shorter correct traces tend to represent more compact CoT paths with less rambling. Otherwise, the motivation for the balancing of hard/medium/easy traces comes from an early experiment: v03 was unbalanced (59% of training data came from 8/8 easy problems, simply because they generate more correct traces per problem) and scored 25/50 on PolyMath-Val-50, a 7-point regression below the base model's 32/50. v04 (same data sources, balanced) immediately rose to 30/50. v08a (1,788 traces, fully balanced) reached 32/50.\n\nThe Section 5.3 ablation confirms both extremes hurt: removing the 8/8 (easy) bucket cost 1 point (v08b), removing the 1-4/8 (hard) buckets cost 2 points (v08c), and 4× upsampling the hard bucket provided no further benefit (v08d, 31/50). Hence the eventual choice was an optimal mix of easy/medium/hard questions.\n\nIn summary, we sample to expose the model to its full range of typical reasoning, not just hard problems. Over-weighting hard problems would bias the regularization toward conditions where the model might overcomplicate its CoT and fail as a result, pushing the reasoning distribution toward less-reliable strategies. The balanced sampling keeps the model's nominal range across a question difficulty scale, which we found to be important to not regress in otherwise solvable problems while attempting to solve harder problems. For easier problems there is a bias towards brevity. I will be adding some additional discussion and clarification in the relevant sections in the writeup.\n\n### Q2. Concrete behavioral differences from SFT\n\nThis is a very good question, and I should have included more discussion on that. As you point out, the writeup leans heavily on aggregate mean/SD numbers without a clean measurement of behavioral change. To address this, I ran a post-hoc analysis matching the v12 (base, no SFT) and v08a (SFT, the final submission model) runs at the per-problem level on the same 50 PolyMath-Val-50 validation set problems. The results are very interesting:\n\n| Metric | v12 (base) | v08a (SFT) | Δ |\n|---|---|---|---|\n| PolyMath-Val-50 score | 32/50 | 32/50 | 0 (no capability change) |\n| Total compute (sum of per-problem times) | 3h 44m | 3h 04m | **−17.8%** |\n| Per-problem mean time | 269s | 221s | −48s |\n| Per-problem time stdev | 204s | 166s | −38s |\n| Problems completing in <300s (strong early-stop indicator) | 32 | 38 | **+6** |\n\nSame problems solved, ~40 minutes less total compute, and 6 more problems hit the 4/8-consensus threshold before exhausting the 300s budget floor. So even though the score didn't necessarily improve, the behavior certainly changed.\n\n**Where the time saving comes from:**\n\n| Transition | n problems | Mean Δt | What it shows |\n|---|---|---|---|\n| correct → correct | 30 | −35s | SFT model confirms the same correct answer with less exploration |\n| wrong → wrong | 16 | −49s | SFT model also gives up sooner on problems it cannot solve |\n| wrong → correct (gain) | 2 | +10s | Two problems newly solved; slightly more time spent verifying |\n| correct → wrong (loss) | 2 | −290s | Two losses — fast convergence to a wrong answer |\n\nThe cleanest behavioral signal is the correct→correct row: on the 30 problems both models solve, the SFT model reaches 4/8 consensus a mean 35 seconds sooner. Same final answer, fewer attempts needed before consensus fires. The wrong→wrong row shows a parallel pattern — the SFT model recognizes hopeless problems faster, ~49s sooner on average. So the regularization tightens the trajectory in both directions: less exploration when the model is going to be right, and less exploration when it isn't. This may be attributable to the downsampling approach in easier questions.\n\nConcrete correct→correct speedups (PID, type, v12 time → v08a time):\n- 5138 (Combinatorics): 263s → 100s\n- 4369 (Number Theory): 247s → 88s\n- 4238 (Number Theory): 307s → 181s\n\nThe 2/2 gain/loss split shows individual problems can flip in either direction. The 2 losses (PIDs 2002 in Geometry and 522 in Algebra) are the \"confident divergence\" failure mode — SFT converges quickly to a wrong answer. These are not a directional signal at this sample size, but I believe it shows that SFT shifts the reasoning behavior.\n\n**Behavior Change regarding Python tool usage** The same per-problem matching also gives us call and error counts for the stateful Jupyter sandbox:\n\n- Total Python calls: 5942 → 4948 (−994, **−16.7%**), which closely matches the 17.8% compute reduction.\n- Total errors: 675 → 587 (−88).\n- Per-call error rate: 11.4% → 11.9% (essentially unchanged).\n\nThe model issues fewer calls overall, with roughly the same error rate per call. Therefore one can say the regularization is correlated with less exploratory tool use. I think this is an interesting future research direction in this line of thought. It will be interesting to try SFT that not only reduces tool use, but also increases its usefulness. Here is the per-transition breakdown:\n\n| Transition | n | v12 calls/pb | v08a calls/pb | v12 err% | v08a err% |\n|---|---|---|---|---|---|\n| correct → correct | 30 | 107.0 | 94.6 | 8.4% | 9.4% |\n| wrong → wrong | 16 | 143.2 | 115.1 | 13.4% | 14.6% |\n| wrong → correct (gain) | 2 | 59.0 | 59.0 | 28.8% | 30.5% |\n| correct → wrong (loss) | 2 | 161.5 | 76.0 | 20.1% | 9.9% |\n\nOn the 30 correct→correct problems, the SFT model averages ~12 fewer Python calls per problem and arrives at the same answer — clean evidence of less exploration on problems it can solve. On the 2 correct→wrong losses, v12 was doing 161 calls at a 20% error rate (clearly fishing) and eventually finding the right answer; v08a more than halves the call count (76 calls, much cleaner 9.9% error rate) and produces the wrong answer. So on the rare losses, SFT replaced \"exhaustive but eventually correct exploration\" with \"confident shortcut to the wrong answer.\" This is one metric to address to improve overall performance with further SFT.\n\n**Per problem-type (timing):** all four types speed up. Algebra −43s, Combinatorics −67s, Geometry −40s, Number Theory −38s (mean Δt per type). Number Theory has both of the score gains; Combinatorics has zero score changes (14/14 stable) despite the largest mean time reduction. Hence, regularization helps convergence without affecting correctness across problem types.\n\nOnce again thank you. I'll be adding these findings to the updated writeup. Once I finish the writeup, I'll post a short summary of changes on Simon's discussion thread." } ] }, { "id": 3456843, "postDate": "2026-05-12T16:57:29.053Z", "content": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.
\nThe submission is already very nice overall. I especially like the plots and tables you included.
\nMoE Size: I would be cautious about describing gpt-oss-120b as a large MoE, since it only has around 5B active parameters. I would reserve that description for models such as DeepSeek-R1. This is only a minor side remark, however.
Trace Quality: Since preference optimization depends heavily on the quality of the preference dataset, did you take any special precautions to avoid including low-quality or fully hallucinated traces? Relying only on final-answer verification could still allow many flawed reasoning traces into the training data. The model may have difficulty learning useful behavior if most of the intermediate signals are noisy or incorrect, even when the final answer is right.
PO Target: In preference optimization, it is common to use a stronger model as a reference or source of preferred responses. Did you explore using very strong models such as GPT-5.x, Kimi, or GLM in any way? One possible complication I could imagine is compatibility with the Harmony chat format and tool calls, although it should in principle, be possible to convert traces from other models into the required format.
Results on PolyMath: As this competition has shown, variance is an important factor to account for. Did you always evaluate PolyMath-v50 in the same problem order? If so, this could potentially skew results, for example, if the system overfits to that order by consistently seeing harder problems first and easier ones later, especially under a limited time budget.
Looking forward to your response :)
", "rawMarkdown": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the [updated discussion post.](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/689703)\n\nThe submission is already very nice overall. I especially like the plots and tables you included.\n\n1. **MoE Size**: I would be cautious about describing `gpt-oss-120b` as a large MoE, since it only has around 5B active parameters. I would reserve that description for models such as DeepSeek-R1. This is only a minor side remark, however.\n\n2. **Trace Quality**: Since preference optimization depends heavily on the quality of the preference dataset, did you take any special precautions to avoid including low-quality or fully hallucinated traces? Relying only on final-answer verification could still allow many flawed reasoning traces into the training data. The model may have difficulty learning useful behavior if most of the intermediate signals are noisy or incorrect, even when the final answer is right.\n\n3. **PO Target**: In preference optimization, it is common to use a stronger model as a reference or source of preferred responses. Did you explore using very strong models such as GPT-5.x, Kimi, or GLM in any way? One possible complication I could imagine is compatibility with the Harmony chat format and tool calls, although it should in principle, be possible to convert traces from other models into the required format.\n\n4. **Results on PolyMath**: As this competition has shown, variance is an important factor to account for. Did you always evaluate PolyMath-v50 in the same problem order? If so, this could potentially skew results, for example, if the system overfits to that order by consistently seeing harder problems first and easier ones later, especially under a limited time budget.\n\nLooking forward to your response :)\n", "replies": [ { "id": 3457990, "postDate": "2026-05-14T18:52:44.577Z", "content": "Thank you for the thoughtful comments and the kind words about the plots and tables. Below are my responses to each of your comments. I am working on an updated version of the write up to address your comments and sharpen a few other things. I'll post a summary of those edits on the discussion thread once the writeup is updated.
\nYou're right, GPT-OSS-120B is closer to a mid-sized dense model in inference cost than to a true large MoE like DeepSeek-R1. I'll clarify the language in the writeup.
\nI'll address 2 and 3 together since they both ask about the role of stronger external models can play in our pipeline.
\nPreference Optimization: We tried DPO early in our experimentation (Code version v09a, discussed in Section 6.3). Training itself converged well (margin 4.04, accuracy 1.0), but the competition score (public data set) regressed from 41/50 to 35/50. Base model competition score had mean 39.8 with SD ~2.0 (range 37 to 43 across 6 runs), so the regression to 35 is well outside the noise floor. We used the correct and wrong traces from the multiple runs on GPT-OSS-120B. I think there could be a way to get that to work (as demonstrated by LB#2 solution in AIMO2). Given the tight time budget, we focused the remaining budget on SFT, which had already shown promise in terms of stable gains. Hence, the final submission for the private data set is based on SFT, and DPO is documented as a failed branch within that scope. I think it is worth experimenting whether DPO could succeed with a better-curated preference dataset / different set of traces.
\nObviously, your broader point is also relevant for SFT itself: Could traces from a stronger model improve our training data? My experience suggests potentially yes, but with caveats for GPT-OSS-120B. Here's what we tried:
\nNuminaMath-TIR (the AIMO-1 winner dataset) traces were generated by GPT-4o using the same final-answer filtering you describe. I tried these traces with SFT as my first TIR attempt (v02). The result was a complete collapse of code generation: 0 out of 400 attempts used Python after training. In a way, this is not unexpected, since GPT-4o does not have TIR embedded in its weights. Numina generated their dataset using the ToRA pattern (prompt GPT-4o to write Python, execute the code in an external loop, splice results back into the prompt). On the other hand, GPT-OSS-120B has native tool use as a built-in capability, with harmony-format tool calls directly from inside its chain of thought. The trace format is very different (around 1.5K tokens of plain Python execution vs around 15K tokens of multi-step harmony tool calls). So, while not conclusive in terms of whether stronger models like GPT-5 could help, this experiment points to the fact that output format matching is crucial. When other LLMs with potentially different CoT format are used, probably an intermediate step is needed to make it compatible with OSS-120B's style for SFT to be successful.
The cleaner test of format-sensitivity we actually have is with our own self-generated traces, which are surprisingly fragile to modification. In v10b (Section 4), we kept the source model fixed but trimmed code blocks that produced errors-before-recovery, expecting a small gain from the cleanup. PolyMath-Val-50 performance instead dropped from 32 to 29, a 3-point regression that is well outside the typical run-to-run variance we observed on Tinker-trained models. Same source, same problems, only the editing changed. My conclusion was that the model needs to see its own error-and-recovery patterns for successful SFT — what may look like noise or bantering in intermediate reasoning actually carries useful signal. As you suggest, defining metrics around reasoning efficiency in traces and optimizing them is probably a viable strategy, but in the form I tried it, it regressed. I also tried a version where I started with a wrong-answer trace and stopped midway for course-correction toward the correct trace. That attempt was half-baked and also regressed. Both led me to conclude that GPT-OSS-120B SFT is very sensitive to the output style of the traces, so I stuck with organic, unmodified traces generated by the model itself.
One interesting observation was that if there is a good \"verifier\" code block to determine the correctness of an answer, then you can use prompt engineering to steer the reasoning. E.g., the solver gives an answer, then using the verifier block we determine it's wrong. Then we hit back saying \"The answer XYZ is known to be wrong, and go back and retry using a different approach\" This approach swayed the reasoning towards the correct answer, but of course you need to have that accurate verifier.
For all these reasons, I chose self-generated traces to support the core idea. The paper's framing is behavioral regularization, not capability transfer. We are biasing the model toward strategies it already finds (sometimes), not teaching new ones. Traces from Claude or GPT-5 would be a different and more ambitious bet, closer to capability transfer, and would first need to clear the format-compatibility bar. Overall, I agree that there are many interesting things to try along this line of discussion.
\nThese ideas were not feasible within the time frame of the competition but I definitely want to do a follow-up. Thank you for pointing these out. For this writeup, I will be updating to make the SFT framing more explicit and clarify some of the points.
\nI double checked my evaluation output files to look at this question from two dimensions:
\n1. Order of Questions: PolyMath-Val-50 runs effectively processed problems in randomized order. For example, the first eight problems in the v08a evaluation run (the one that produced our headline 32/50 number) were processed in this sequence (problem IDs): 3049, 455, 886, 1581, 2228, 4088, 3307, 522. These are mixed across norm_type (Algebra, Combinatorics, Number Theory, Geometry) and across the dataset's oss_20b_med_pass difficulty buckets (both 0.0 and 0.5). The same effective randomization shows up in our v11a (Unsloth 48K) and v11d (Unsloth 65K) hardtest runs, and on our separate 10-problem reference test set as well. So problems were not consistently encountered hardest-first or easiest-first across runs. Although in Fig 5a, I ordered them based on problem type and easiness (Always correct/wobbly/always wrong) so that we can compare different models for each question type/difficulty type. But in the runs they were presented in random order. I will clarify that in the text.
2. Time Budget: More importantly for your concern, the time budget was never binding in practice. Across the v08a, v11a, and v11d runs:
\nSo even if our evaluations had been ordered hardest-first, no problem would have been time-clipped for landing late in the run. Order-induced score effects on PolyMath-Val-50 would therefore be small, and attributable to other factors. Nevertheless, they were randomized in the tests. I also have multi-run comparisons on the same model and same order (e.g., v11c, which swung 33 to 29 across two runs), which suggests that sampling variance is already large enough to dominate any plausible order effect.
\nOf course, the performance variance we observed on the public LB is very real, and is the central problem addressed by my method. For the public data set, I tried a variant of my BKM code offering up to 900s to each question, risking a time-out at the end, and it still completed on time. I have no insights about the private data set, but at least for this algorithm public data set is less likely to be time constrained. The observed variance most likely comes from the variance of the model, hence the self-regularization method to make it smaller.
\nThanks again for these comments. They've been useful for sharpening both the paper and my own thinking about the next steps. I really want to prepare an Arxiv submission based on that, so I'd welcome any other comments you guys may have. Once I finish my edits, I'll be posting a short summary of the resulting paper edits on Simon's discussion thread shortly.
", "rawMarkdown": "Thank you for the thoughtful comments and the kind words about the plots and tables. Below are my responses to each of your comments. I am working on an updated version of the write up to address your comments and sharpen a few other things. I'll post a summary of those edits on the discussion thread once the writeup is updated.\n\n## 1. MoE Size\n\nYou're right, GPT-OSS-120B is closer to a mid-sized dense model in inference cost than to a true large MoE like DeepSeek-R1. I'll clarify the language in the writeup.\n\n## 2 + 3. Trace Quality and PO Target (combined)\n\nI'll address 2 and 3 together since they both ask about the role of stronger external models can play in our pipeline.\n\n**Preference Optimization:** We tried DPO early in our experimentation (Code version v09a, discussed in Section 6.3). Training itself converged well (margin 4.04, accuracy 1.0), but the competition score (public data set) regressed from 41/50 to 35/50. Base model competition score had mean 39.8 with SD ~2.0 (range 37 to 43 across 6 runs), so the regression to 35 is well outside the noise floor. We used the correct and wrong traces from the multiple runs on GPT-OSS-120B. I think there could be a way to get that to work (as demonstrated by LB#2 solution in AIMO2). Given the tight time budget, we focused the remaining budget on SFT, which had already shown promise in terms of stable gains. Hence, the final submission for the private data set is based on SFT, and DPO is documented as a failed branch within that scope. I think it is worth experimenting whether DPO could succeed with a better-curated preference dataset / different set of traces.\n\nObviously, your broader point is also relevant for SFT itself: Could traces from a stronger model improve our training data? My experience suggests potentially yes, but with caveats for GPT-OSS-120B. Here's what we tried:\n\n1. **NuminaMath-TIR (the AIMO-1 winner dataset)** traces were generated by GPT-4o using the same final-answer filtering you describe. I tried these traces with SFT as my first TIR attempt (v02). The result was a complete collapse of code generation: 0 out of 400 attempts used Python after training. In a way, this is not unexpected, since GPT-4o does not have TIR embedded in its weights. Numina generated their dataset using the ToRA pattern (prompt GPT-4o to write Python, execute the code in an external loop, splice results back into the prompt). On the other hand, GPT-OSS-120B has native tool use as a built-in capability, with harmony-format tool calls directly from inside its chain of thought. The trace format is very different (around 1.5K tokens of plain Python execution vs around 15K tokens of multi-step harmony tool calls). So, while not conclusive in terms of whether stronger models like GPT-5 could help, this experiment points to the fact that output format matching is crucial. When other LLMs with potentially different CoT format are used, probably an intermediate step is needed to make it compatible with OSS-120B's style for SFT to be successful.\n\n2. The cleaner test of format-sensitivity we actually have is with our own **self-generated traces**, which are surprisingly fragile to modification. In v10b (Section 4), we kept the source model fixed but trimmed code blocks that produced errors-before-recovery, expecting a small gain from the cleanup. PolyMath-Val-50 performance instead dropped from 32 to 29, a 3-point regression that is well outside the typical run-to-run variance we observed on Tinker-trained models. Same source, same problems, only the editing changed. My conclusion was that the model needs to see its own error-and-recovery patterns for successful SFT — what may look like noise or bantering in intermediate reasoning actually carries useful signal. As you suggest, defining metrics around reasoning efficiency in traces and optimizing them is probably a viable strategy, but in the form I tried it, it regressed. I also tried a version where I started with a wrong-answer trace and stopped midway for course-correction toward the correct trace. That attempt was half-baked and also regressed. Both led me to conclude that GPT-OSS-120B SFT is very sensitive to the output style of the traces, so I stuck with organic, unmodified traces generated by the model itself.\n\n3. One interesting observation was that if there is a good \"verifier\" code block to determine the correctness of an answer, then you can use prompt engineering to steer the reasoning. E.g., the solver gives an answer, then using the verifier block we determine it's wrong. Then we hit back saying \"The answer XYZ is known to be wrong, and go back and retry using a different approach\" This approach swayed the reasoning towards the correct answer, but of course you need to have that accurate verifier.\n\nFor all these reasons, I chose **self-generated traces** to support the core idea. The paper's framing is *behavioral regularization*, not *capability transfer*. We are biasing the model toward strategies it already finds (sometimes), not teaching new ones. Traces from Claude or GPT-5 would be a different and more ambitious bet, closer to capability transfer, and would first need to clear the format-compatibility bar. Overall, I agree that there are many interesting things to try along this line of discussion.\n\n- Generate traces with a strong model (Claude Opus, GPT-5, or Kimi) restricted to the harmony chat format and Python tool calls. Then, maybe go through a referee LLM stage to make sure they conform to OSS-120B output format and then use these traces for SFT/DPO.\n- External verifier: This could be a separate SFT based tool, which filters for final-answer correctness, plus some form of intermediate-step quality assessment of CoT. Obviously with the one H100 constraint, there is some resource optimization needed.\n- Increase the training data beyond PolyMath. This one is an obvious next step. I'd like to explore a larger mix from Corpus Prize data sets and re-test some of the assumptions and findings.\n\nThese ideas were not feasible within the time frame of the competition but I definitely want to do a follow-up. Thank you for pointing these out. For this writeup, I will be updating to make the SFT framing more explicit and clarify some of the points.\n\n\n## 4. PolyMath-Val-50 Evaluation Order\n\nI double checked my evaluation output files to look at this question from two dimensions:\n\n**1. Order of Questions**: PolyMath-Val-50 runs effectively processed problems in randomized order. For example, the first eight problems in the v08a evaluation run (the one that produced our headline 32/50 number) were processed in this sequence (problem IDs): 3049, 455, 886, 1581, 2228, 4088, 3307, 522. These are mixed across `norm_type` (Algebra, Combinatorics, Number Theory, Geometry) and across the dataset's `oss_20b_med_pass` difficulty buckets (both 0.0 and 0.5). The same effective randomization shows up in our v11a (Unsloth 48K) and v11d (Unsloth 65K) hardtest runs, and on our separate 10-problem reference test set as well. So problems were not consistently encountered hardest-first or easiest-first across runs. Although in Fig 5a, I ordered them based on problem type and easiness (Always correct/wobbly/always wrong) so that we can compare different models for each question type/difficulty type. But in the runs they were presented in random order. I will clarify that in the text.\n\n**2. Time Budget**: More importantly for your concern, the time budget was never binding in practice. Across the v08a, v11a, and v11d runs:\n\n- The minimum per-problem budget was 900s, and the time bank typically accumulated to give later problems somewhere between 1100s and 1800s of headroom.\n- The maximum time any single problem actually used was 695s (v08a), 715s (v11a), and 747s (v11d), all comfortably below the minimum 900s budget.\n- Mean time per problem was roughly 220 to 270s.\n- In every case the decision logic, not the timer, closed each problem out.\n\nSo even if our evaluations had been ordered hardest-first, no problem would have been time-clipped for landing late in the run. Order-induced score effects on PolyMath-Val-50 would therefore be small, and attributable to other factors. Nevertheless, they were randomized in the tests. I also have multi-run comparisons on the same model and same order (e.g., v11c, which swung 33 to 29 across two runs), which suggests that sampling variance is already large enough to dominate any plausible order effect.\n\nOf course, the performance variance we observed on the public LB is very real, and is the central problem addressed by my method. For the public data set, I tried a variant of my BKM code offering up to 900s to each question, risking a time-out at the end, and it still completed on time. I have no insights about the private data set, but at least for this algorithm public data set is less likely to be time constrained. The observed variance most likely comes from the variance of the model, hence the self-regularization method to make it smaller.\n\n\nThanks again for these comments. They've been useful for sharpening both the paper and my own thinking about the next steps. I really want to prepare an Arxiv submission based on that, so I'd welcome any other comments you guys may have. Once I finish my edits, I'll be posting a short summary of the resulting paper edits on Simon's discussion thread shortly." } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T04:24:40Z", "competition_id": 118448, "competition_slug": "ai-mathematical-olympiad-progress-prize-3", "team_id": 15091563, "solution_url": "/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/behavioral-regularization-via-sft-on-gpt-oss-120b" }