![\"Per-layer](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2270821%2Fca39d8cf5ccf892e7080773c16103025%2Fcascade2_quant_layout.png?generation=1776888105108688&alt=media\")
For both @yeoyunsianggeremie and me, this is our third time participating in AIMO. Last year, we went all out and put everything into SFT + RLHF (Kaggle post: \"Soβ¦ We cooked a model π€·\", detailed blog post: \"Improving DeepSeek R1 in Math\"). This year, with Geremie grinding 75+ hours per week at his startup and me in the middle of a release, we had almost no time to compete like we did last year. Luckily, we had companions helping us: Geremie's bogoconic1/Qgentic-AI and my Claude Code Max subscription!
\nDespite joining the competition only in the last couple of weeks, we got delightfully stuck in a few side quests, and ended up creating a few useful artifacts that we think are worth sharing.
\nWe'd like to express our gratitude to AIMO3 organizers and XTX market for hosting such a fun competition and providing H100 GPUs. Special thanks to Simon Frieder (@friederrr) for actively engaging with the community, making it one of the most pleasant competitions I've ever participated in.
\nOur little fun time with AIMO3 resulted in following open-source side quests, useful beyond this competition:
\nWe also uncovered cheating attempts by Cascade 2 on Math problems (searching the web for answers) and reported to model authors here: Cascade-2 cheating attempts on Math is⦠cute.
\n\n\nKaggle notebook: chankhavu/c2-distill-vllm018-watchdog\n Model: chankhavu/c2-legaldistill-30b-a3b-fp8
\n
Our solution uses Nemotron-Cascade-2-30B-A3B, which we soft-cloned via layer-wise linear distillation to make it legal for this competition (Cascade 2 was released after the March 15 cutoff β see the Introduction below), then selectively quantized to FP8 with no quality degradation.
\nThe final harness is intentionally simple: 8 parallel rollouts per problem with Python tool use, entropy-weighted majority voting. We also tried a more sophisticated variant that filled the GPU idle bubbles between early- and late-finishing rollouts with pairwise LLM-as-judge self-verification, but didn't have time to tune it properly to beat plain majority voting. We also put a great deal of work to make speculative decoding work with prefix caching for hybrid-mamba models, but it didn't get into our final submission due to lack of submission budget and time to test things out properly. Summary of our main candidates on local validation and the Kaggle leaderboard:
\n| Submission (model, harness) | \nLocal validation (3 shuffles) | \nLocal validation (mean) | \nPublic LB attempts | \nPrivate LB | \n
|---|---|---|---|---|
| gpt-oss-120b, 44/50 public notebook | \n33, 30, 36 | \n33.0 | \n38, 38, 37 | \nβ | \n
| Cascade 2, 8 rollouts, self-verification (untuned) | \n33, 31 | \n32.0 | \n34 | \nβ | \n
| Cascade 2, 8 rollouts, voting (our final submission) | \n36, 34, 34 | \n34.6 | \n36, 39, 38 | \n41 | \n
Our local validation imitates the Kaggle leaderboard setting β 50 hard IMO-level problems drawn from IMO-AnswerBench (an open benchmark of IMO-style problems with short-integer answers), 5 hours on 1Γ H100 SXM. We shuffle the 50 problems 3 times with fixed seeds and run each solver through each shuffle, which gives us a controlled way to compare candidates without burning LB submissions.
\nThe competition. AIMO3 (the AI Mathematical Olympiad Prize 3, hosted by Kaggle and XTX Markets) evaluates math-problem solvers on 50 held-out IMO-level problems with short-integer answers. Each submission is a notebook that runs on a single NVIDIA H100 SXM (80 GB) with a 5-hour wall-clock budget. Models published after March 15, 2026 are not allowed to be used directly at runtime, which is the rule that shapes most of the model choices below. The public leaderboard is scored on a visible subset; the final standings use a disjoint private set revealed at the end.
\nThe main model candidate everyone converged on this year is gpt-oss-120b β OpenAI's 120 B-parameter open-weights reasoning model with native mxfp4 weights and a Harmony chat format. It's the model behind the strongest public-notebook baseline (the \"44/50\" notebook on the AIMO3 discussion board), and our main point of comparison throughout.
\nIn March 19, @chankhavu's birthday, Nemotron-Cascade-2-30B-A3B (shortened to Cascade 2 in this post) was released β NVIDIA's 30 B hybrid Mamba-MoE reasoning model. Published just after the March 15 cutoff, so not directly usable as-is, but the architecture is identical to the pre-cutoff Nemotron-3-Nano-30B-A3B. This is what made the \"soft-cloning\" approach legal, and it's the model inside our final submission. The hybrid-Mamba architecture is what forced us down a bunch of the side quests below β quantizing it, getting prefix caching to work with speculative decoding, training an Eagle3 speculator, and so on.
\nOther participant's findings. With only a few weeks to compete, we relied heavily on the open-source ecosystem and the community's collective findings. Some highlights:
\nThe competition rules prohibits using models published after March 15 directly during runtime. However, there's no limit on distillation or how distillation should be made. We rely on the following FAQ clarification from the competition organizers:
\n\n\nQuestion: Say I initialize an entirely new model with same architecture as the post - cutoff teacher model, but then train it by layerwise MSE to the teacher on the residual stream; in the limit this could approximate the teacher model weights, I just didn't explicitly say that I used the weights themselves.
\nAnswer: Indeed, while conceptually, these two cases can blur, practically they don't and that is why the AMLT rules makes sense: In practice, with this approach you'd recreate in a way the model that was off limits.
\n
We further clarified with the organizers that this kind of \"soft cloning\" is legal for the competition (here and here).
\nAfter confirming the legality, we performed the following layer-wise MSE distillation from Nemotron-Cascade-2. We initialized a randomly initialized student model with the same architecture and performed the following procedure:
\ni of the teacher Nemotron-Cascade-2 model, where X is the input to that layer and W_i the weight tensor.i. The student model happens to have the same architecture, so we can directly map each F^Teacher}_i to a student F^Student_i.i to minimize the following MSE loss: $$L_i = \\mathbb{E}_{X \\sim D} \\left[ | F^\\text{Teacher}_i(X) - F^\\text{Student}_i(X) |^2_2 \\right]$$ where D is a dataset of randomly sampled inputs. We optimize either by gradient descent or by closed-form MSE optimization techniques. Each student treats the teacher as a black-box function that produces outputs for given inputs.We never optimize towards the teacher weights directly; we only observe the teacher outputs and use them as MSE targets for the student. However, since the sub-models are linear, convergence is mathematically absolute. We published the implementation here. The BF16 model can be found here: chankhavu/bogoconic1-c2-softcpy-v3.
\nImportant clarification: for Kaggle submission, we quantized the cloned Cascade 2 model separately from the cloned BF16 model! We did NOT use our published open-source models.
\n\n\nOpen-source artifacts (totaling 66K downloads on Huggingface as of April 21, 2026):
\n\n
\n- π€ Huggingface model: chankhavu/Nemotron-Cascade-2-30B-A3B-NVFP4
\n- π€ Huggingface model: chankhavu/Nemotron-Cascade-2-30B-A3B-FP8
\n- π€ Huggingface model: chankhavu/Nemotron-Cascade-2-30B-A3B-AWQ-INT4
\nQuantization code: chankhavu/quantize-cascade-2; calibration data: chankhavu/cascade-2-calibration-data.
\n
We followed the Nano-architecture selective quantization recipe from the Nemotron 3 Nano Technical Report (Section 4). This is the same recipe as NVIDIA's official FP8 checkpoint of Nemotron-3-Nano-30B-A3B. Sensitive components are kept in higher precision:
\n
For calibration, we constructed a dataset of 4000 samples from Nemotron-Cascade-2-SFT-Data with the following domain mix, specifically designed for long-context reasoning: math (1000), swe (900), terminal_agent (500), science (500), chat (400), conversational_agent (300), instruction_following (300), safety (100). We trained on sequence length of up to 12,288 tokens (no padding, natural length per sample).
\nWe validated the correctness of our quantized models using NVIDIA-NeMo/Evaluator with config from Nemotron-3-Super-120B's eval config. We see no degradation in reasoning quality of quantized models. More details in Appendix B.
\n\n\nOpen source artifact: hav4ik/vllm branch pc-spec-v2-nonpc-slots
\n
Out of the box, neither vLLM nor SGLang can run Prefix Caching (PC) and Speculative Decoding (SD) together on Hybrid-Mamba models. Mamba's SSM layer keeps a fixed-size state vector updated in place β each token folds into it irreversibly. SD runs K candidate tokens through the model before the rejection sampler decides how many to keep; by then the SSM kernel has already overwritten the canonical state with the full K-token trajectory. Transformers escape this because their KV cache is append-only β rejected entries sit past the sequence end and touch nothing. Mamba has one state vector per layer per request and can't roll it back, so PC + SD needs an explicit mechanism to preserve the accepted state.
\n![\"When](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2270821%2F6f4d1ef1b007dbf7421e0ef6f972dae9%2Fpc_sd_problem.png?generation=1776888162924985&alt=media\")
The non-PC spec-decode path already does the right thing: the SSM and conv Triton kernels maintain K+1 per-request \"ghost\" slots with native init_token_idx rollback β those slots are just gated out whenever PC is active. We re-enable them and add a one-way boundary commit: only when a request's last accepted token lands exactly on a mamba-block boundary (last_accepted_pos % block_size == block_size - 1) do we write the ghost slot back to the PC pool. On every other step the pool is untouched, so the canonical cache state is never contaminated by rejected speculations. Ghost slots are indexed by batch position, so batch reordering moves them automatically β no extra bookkeeping.
One memorable debug moment: the first cut of init_spec_slots allocated each spec slot at the pool's page-padded stride (~50 MB/slot). At 32 seqs Γ K=5 Γ 23 layers that comes out to 52 GB β instant OOM. The fix was to allocate contiguously (~1.3 MB/slot); the kernels read state.stride(0) off the tensor itself, so ghost slots and pool slots can have completely different strides without breaking anything.
We used AIME-25 as a unit test throughout, with a pass threshold of β₯95% per-session accuracy β ironic, because AIME-25 is a notoriously hard competition that few high schoolers solve reliably, but at this point it's well within what a modern reasoning LLM with tool use can do consistently. A reliable gold standard for \"nothing silently corrupted\" is useful regardless.
\nOnce the correctness fix was in, Eagle3 was still a net loss at single-sequence concurrency: a full step took ~18 ms vs ~3.4 ms for no-Eagle, even though the target forward itself is only 4.2 ms. Profiling (H100, K=4, p=1) split the 18 ms budget roughly like this:
\n| Stage | \nTime (ms) | \nNotes | \n
|---|---|---|
| Target forward (K+1 tokens) | \n4.2 | \nactual model compute | \n
| Drafter, K=4 iterations | \n11.2 | \nserial: 4 Γ (~2 ms Python metadata + ~0.8 ms GPU forward) | \n
_build_attention_metadata | \n5.7 | \n7Γ more than no-Eagle β all 52 layers Γ K+1 | \n
_prepare_inputs | \n1.7 | \nK+1 tokens per request | \n
commit_boundary_states (23 mamba layers) | \n3.9 | \n.nonzero() D2H sync every step + .synchronize() wait | \n
| Rejection sampling + misc | \n1.1 | \n\n |
Two things stood out. First, the drafter loop is serially CPU-bound: each iteration needs the previously accepted token before it can build the next metadata object, so Python rebuilds can't overlap the 0.8 ms GPU forwards, and the GPU sits idle ~70% of the drafter phase. Second, commit_boundary_states β Mamba-specific infrastructure β was paying a full host-sync tax on every step, not just the rare steps where a commit actually fires.
![\"Eagle3](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2270821%2Ff9ef240767d03da1ecfd5b621f84b027%2Feagle3_step_timeline.png?generation=1776888210353385&alt=media\")
The drafter issue needs a proper architectural fix (fuse all K iterations into one captured graph with persistent metadata buffers) that we left for upstream. What we did land is a sweep of surgical cuts targeting the other stages:
\ncommit_boundary_states used to call .nonzero() on a CUDA bool mask to find requests that had crossed a mamba-block boundary. .nonzero() forces a device β host sync every step. The new kernel grid is (num_layers, num_requests): each block checks its mask bit on-device and returns early if unset. No host ever sees the mask. β1.5 ms/step.synchronize(). The deferred-commit path was calling sample_complete_event.synchronize() β a blocking CPU wait β to guarantee num_accepted_tokens was visible before launching the commit kernel. Since both run on the same CUDA stream as the sampler, stream ordering already guarantees visibility. Removed. β1.0 ms/step.update_block_table + batched commit indices. Two related cleanups on the commit path that were paying per-layer overhead on pure-decode steps. β4.5 ms/step combined.attn_meta. On pure-decode steps, only state_indices_tensor_d differs between the 23 mamba attention builders, but _prepare_inputs was rebuilding the full metadata struct each time. Cache the first build; for the other 22, copy_ the per-layer pool mapping into the persistent buffer and dataclasses.replace just that field. β0.3 ms/step.End state at single-sequence (K=4, T=1.0): step time 18 β 8.9 ms, throughput 171 β 298 tok/s vs 303 tok/s for PC-only β essentially at parity.
\n![\"Eagle3](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2270821%2Fb3245aec078557c3cb820f13d9fc5eae%2Feagle3_step_timeline_after.png?generation=1776888256293073&alt=media\")
At higher concurrency, where the serial drafter overhead is amortized across the batch, Eagle3 becomes a real win:
\n| Concurrency | \nPC-only | \nEagle3 K=4, T=0.7 | \nvs PC-only | \n
|---|---|---|---|
| p=1 | \n303 tok/s | \n349 tok/s | \n+15% | \n
| p=8 | \n1186 tok/s | \n1552 tok/s | \n+31% | \n
| p=12 | \n1383 tok/s | \n1843 tok/s | \n+33% | \n
On an experimental branch we captured the K-iteration drafter decode loop as a single FULL cudagraph, replayed in 0.16 ms. Draft-phase time dropped from 3.16 ms to 1.39 ms (β56%) with acceptance rates preserved. But overall step time at p=8 was unchanged (13.7 ms) β the CPU savings got absorbed by GPU pipeline stalls in the prep phase, since the drafter's GPU work that previously overlapped the PIECEWISE Python loop now blocked the next iteration. The theoretical payoff is at p=12+ where CPU dispatch becomes the real bottleneck, but we didn't validate that before the deadline.
\nWe had planned to upstream the correctness fix and follow with the perf path, but two things have changed since the competition closed and the PR is currently blocked:
\nThe fork itself remains useful for anyone running Hybrid-Mamba SD inference on Hopper; we just can't claim the upstream PR as a clean deliverable any more.
\n\n\nOpen-source artifacts:
\n\n
\n- π€ Huggingface model: chankhavu/Nemotron-Cascade2-30B-A3B-Eagle3-Long-Context
\n- π€ Huggingface dataset: chankhavu/c2_eagle3_train (2 stage train data)
\n- π€ Huggingface dataset: chankhavu/c2-traces-10197 (collected reasoning traces on 10'197 hard math problems)
\nCode: SpecForge-Nemotron3.
\n
To our knowledge this is the first Eagle3 draft head trained against a hybrid Mamba-Transformer verifier that's verified end-to-end with vLLM. We forked SpecForge as SpecForge-Nemotron3 and added the engineering needed to fit long-context training (L=65536, ttt=6) on 2Γ RTX 6000 Pro Blackwell (96 GB each).
Two stages: a broad SFT bootstrap on ~20K conversations (378M untruncated tokens, from Cascade 2's supervised-finetune distribution) followed by on-policy fine-tuning on ~9.7K Cascade 2 reasoning traces (207M tokens) β ~292M loss-masked tokens total. We prune the 131K target vocab to a 32K draft vocab via top-K token frequency on the loss-masked positions, which keeps 99.77% coverage with ~4Γ smaller embedding / lm_head. Stage 1 uses LR 1e-4 with length-bucketed sampling (eliminates straggler-rank waste at batch_size=1); stage 2 drops to LR 5e-5 and disables bucketing for gradient diversity.
Two classes of fixes were required to get this training loop working at long context.
\nCorrectness β FP32 SSM scan state. Upstream mamba_ssm 2.3.1 silently downcasts the SSM scan boundary state to BF16 inside its Triton kernel. We monkey-patch it to force FP32, bit-equivalent to vLLM 0.19's NemotronH default. Without this, training converges to a worse minimum that costs ~10% absolute on AIME-class math.
Memory footprint β fitting L=65k on 96 GB. NaΓ―ve training peaks north of 120 GiB (OOM on 96 GB). Four independent activation-memory cuts on the draft forward + TTT unroll (chunked fused linear + soft-target CE for the lm_head, freeing the verifier's 131k-vocab logits before TTT unrolling, Liger-style chunked MLP, sliding-window attention on the draft) bring peak to ~88β91 GiB, with the single biggest win being the verifier-logit release. Step time at L=65k ttt=6 is ~4 s/step (high variance across sequence lengths). Separately, a 100Γ speedup in the loss-mask parser that was otherwise re-tokenizing each conversation num_turns times. Full details in Appendix C.
Temperature sweep, mean accepted tokens per step (on AIME-25):
\n| K | \nT=0.0 | \nT=0.7 | \nT=0.8 | \nT=0.9 | \nT=1.0 | \n
|---|---|---|---|---|---|
| K=4 (out of K+1=5) | \n3.33 | \n3.00 | \n3.05 | \n2.88 | \n2.91 | \n
| K=7 (out of K+1=8) | \n3.55 | \n3.26 | \n3.14 | \n3.03 | \n2.90 | \n
Lower temperature β draft head is more confident and its proposals track the target's argmax more closely, so acceptance climbs. You can find more information in Appendix C.
\nThe sandbox we gave Cascade 2 for tool use was a stateful_python_code_exec Jupyter kernel with network egress not blocked β urllib, requests, and pip install all worked. On 10,197 rollouts against IMO-class problems, Cascade 2 issued real HTTP calls in 246 traces (~2.4%).
Top destinations (HTTP URL literals in tool-call code, per-trace dedup), by category:
\nartofproblemsolving.com (64), en.wikipedia.org (58), oeis.org (28), mathworld.wolfram.com (6)google.com (40), duckduckgo.com (39), bing.com (3)math.stackexchange.com (13), api.stackexchange.com (12), stackoverflow.com (3)imo-official.org (11), purplecomet.org (5)raw.githubusercontent.com (8), arxiv.org (3), api.github.com (3)The dominant pattern is AoPS wiki lookups by year/contest/number and Wikipedia for theorem statements. OEIS (28 traces) is usually queried with a computed-prefix sequence. DuckDuckGo Instant Answer API and google.com/search GETs with the prompt pasted verbatim show up often enough to be a recognizable tactic.
A few representative (amusing) traces:
\nurllib.request.urlopen against imo-official.org, written to disk (byte-exact match to the real PDF), PyPDF2.PdfReader over every page, substring-grep for prompt keywords. Extraction worked; the problem wasn't in the document. Answer happened to be 0 β coincidental match.requests.get(...), saved response, tried PyPDF2 / pdftotext / fitz / pdfminer β all four failed with EOF marker not found / No /Root object. First 100 bytes of the \"PDF\" were <!DOCTYPE html>; the server returned an HTML error page with a .pdf extension. Wrong answer.IMO2021SL.pdf and IMO2022SL.pdf into memory via response.read(), never persisted, grep_bytes(b'Mario'). Problem was IMO 2022 SL C2 (\"Mario and Bowser\"). Wrong answer.!wget with a typo (trailing s), DNS fails.Network-using traces end up at 34.6% correct β these are the IMO-level problems the model couldn't crack with direct reasoning in the first place, and no venue in the breakdown meaningfully changes the picture (wikipedia 36.6%, AoPS 27.9%, OEIS 27.0%, Google search 42.9% at the top). That said, the model recognizing a problem as a benchmark item and reaching for the right contest archive is a contamination signal on its own, independent of whether the lookup paid off.
\nWe reported this to the Nemotron team (HF discussion) and released the verified traces: π€ chankhavu/nemotron-cascade2-cheating-attempts (173 rows). For our own runs since, we block egress at the sandbox namespace level.
\nWith the open-source artifacts above in hand β the soft-cloned FP8 Cascade 2, the vLLM PC+SD fix, the Eagle3 speculator β we now turn to the competition itself: how these pieces line up against gpt-oss-120b on the benchmarks, what we actually submitted, and which harnesses we tried but didn't ship.
\n\n\nWe publish the dataset construction notebook here: https://www.kaggle.com/code/chankhavu/local-validation-benchmark.
\nAll throughput analysis in this section are based on the model alone, without speculative decoding.
\n
We construct a candidate pool of 63 problems by combining the 10 official AIMO3 reference problems with 53 hard problems from IMO-AnswerBench β specifically those where gpt-oss-120b scored 6/8 or less on avg@8, restricted to answers in the Kaggle 0β99999 integer range. The pool draws from IMO Shortlists, USAMO, HMMT, Sharygin, RMM, and national TSTs (Vietnam, China, Korea, Japan). We then take a 50-problem subset to match the Kaggle Leaderboard setting used for local validation above.
\nFor the gpt-oss-120b tool-use accuracy stats, we use the answerbench-gpt-oss dataset by @yiyangzheng β they were among the top contenders on the LB, so their inference pipeline is trustworthy.
\n\n\nNote on contamination. IMO-AnswerBench is itself drawn from previously published IMO and IMO-Shortlist problems (and other national olympiads), rewritten so that the answer is an integer in a fixed range. Both gpt-oss-120b and Cascade 2 were trained on data that almost certainly includes some of these problems in their original form, so contamination is a real concern. Cascade 2 in particular was released after IMO-AnswerBench was, which expands the contamination window. We did not test for contamination on our side; we chose to trust the data-curation procedure described in the Nemotron-Cascade 2 paper, which uses n-gram-based deduplication against known evaluation benchmarks. We flag this so the reader can weight the absolute accuracy numbers accordingly; the relative comparisons between solvers are less affected because both candidates draw from the same potentially-contaminated training pool.
\n
\n\nWe published all reasoning traces collected for offline benchmark: chankhavu/benchmark-traces
\n
All three candidates run through the same harness on the 63-problem evaluation set (8 rollouts per problem, tool use enabled, same prompt template): gpt-oss-120b, our FP8-quantized Cascade 2, and the Nemotron-3-Nano-30B-A3B.
\nMetric definitions used throughout this section: Accuracy is avg@1 (probability a single rollout is correct, averaged over all 504 rollouts). Majority vote is maj@8 (the most-common of 8 parallel rollouts matches the expected answer). Pass@8 is the probability that at least one of 8 rollouts is correct (the upper bound an oracle verifier could reach).
\n| Model | \nAccuracy | \nMajority vote | \nPass@8 | \nNo-answer | \nAvg gen tok | \nAvg tools/sess | \n
|---|---|---|---|---|---|---|
| gpt-oss-120b | \n44.0% (222/504) | \n52.4% (33/63) | \n77.8% (49/63) | \n8.9% (45/504) | \n26k | \n20.7 (7% err) | \n
| nano30b | \n41.5% (209/504) | \n52.4% (33/63) | \n63.5% (40/63) | \n1.6% (8/504) | \n47k | \n14.2 (6% err) | \n
| cascade2 | \n47.0% (237/504) | \n55.6% (35/63) | \n74.6% (47/63) | \n2.2% (11/504) | \n34k | \n11.5 (7% err) | \n
Cascade 2 leads on both accuracy and voting with half the tool calls β more sample-efficient when it does reach for Python. gpt-oss-120b's high no-answer rate comes from hard problems where it runs out of context mid-reasoning.
\n| \n | 1 sequence | \n4 sequences | \n8 sequences | \n12 sequences | \n
|---|---|---|---|---|
| gpt-oss-120b, mxfp4, kv8 | \n175.4 tok/s | \n506.4 tok/s | \n810.9 tok/s | \n842.8 tok/s | \n
| Cascade 2, fp8, kv8 | \n301.6 tok/s | \n859.0 tok/s | \n1333.5 tok/s | \n1487.8 tok/s | \n
We use the FP8 Cascade 2 variant because H100 supports FP8 natively. Cascade 2 is faster at every batch size, and the gap widens at higher concurrency: gpt-oss-120b's throughput plateaus by 8 sequences, while Cascade 2 keeps scaling.
\nTwo views of how each model scales with more compute per problem. pass@k = prob. β₯1 of k rollouts is correct; maj@k = probability that the most-common answer across k rollouts is correct. Methodology (x-axis conversion, truncation rule, tokenizer choice) in Appendix D.
\nTime budget. Each dot is k β {1, β¦, 8}; the x-axis converts k into wall-clock seconds per problem on 1Γ H100 SXM at 8 parallel sequences. Each model's dots are spaced differently because their per-rollout cost differs: Cascade 2 is the cheapest per rollout, so its curve packs the most to the left.
\n![\"Test-time](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2270821%2F0f085dcf830df3ae82dd1dcdd2c87eeb%2Ftest_time_scaling.png?generation=1779061888745661&alt=media\")
Within the tested budget, Cascade 2 wins both pass@k and maj@k β it's the more consistent choice for the actual competition setting. It leads on maj@k across the whole range, and it wins pass@k at every plotted budget because it's cheaper per rollout and gets to fit more rollouts in for any given wall-clock. gpt-oss-120b's pass@k curve, however, has a higher upper ceiling β its rollouts are more diverse (helps pass@k, hurts plurality voting), so given more attempts / a larger budget than this chart covers, an oracle verifier picking the right rollout would eventually prefer gpt-oss-120b. That higher pass@k ceiling is what makes gpt-oss-120b interesting with an external verifier (see Self-verification experiments below); within the time budget of this competition, Cascade 2 dominates on both axes.
\nA practical note on that pass@k advantage: realizing it would require an external verifier picking from gpt-oss-120b's rollouts, and Kaggle's resource constraints (5 h on 1Γ H100 SXM) make that infeasible in either obvious form β we cannot fit a separate verifier model alongside gpt-oss-120b's ~120 GB MXFP4 weights on a single H100, and using gpt-oss-120b as its own verifier doubles the per-problem token bill on a budget that already only fits ~8 solution rollouts per problem. Cascade 2's smaller footprint and faster throughput leave headroom for the self-verification experiments below; gpt-oss-120b does not.
\nReasoning-time budget. Left panel plots avg@1, right panel maj@8, both as a function of how long a single rollout is allowed to reason (wall-clock seconds on 1Γ H100 SXM at 8 parallel sequences). Each dot is the result of capping the stored rollouts at a different token length, retaining the last \\boxed{} within the cap as that rollout's answer. This is the \"one longer chain of thought\" axis, complementing the \"more parallel rollouts\" time-budget chart above.
![\"Test-time](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2270821%2Fc32a3b368450b06dd6948c37448d9330%2Ftest_time_scaling_length.png?generation=1779061923631429&alt=media\")
Cascade 2 dominates on both panels. It reaches its ceiling (47.0% avg@1 / 57.1% maj@8) at ~207 s per rollout, while gpt-oss-120b tops out at ~313 s with a lower ceiling (44.0% / 52.9%) β gpt-oss reasons compactly in tokens, but its slower throughput costs it ~50% more wall-clock time. nano30b traces Cascade 2's shape shifted right by ~70 s: same throughput, longer traces β exactly the gap soft-cloning was meant to close.
\nHow the x-axis is built. For each truncation L β {16k, 32k, 48k, 64k, 80k, 96k} we cut each rollout at L tokens, measure the effective reasoning length (the \\boxed{} position, or L if the model didn't stop), and convert tokens β seconds using the model's per-sequence throughput at 8 parallel sequences on 1Γ H100 SXM (101 tok/s/seq for gpt-oss-120b; 167 tok/s/seq for Cascade 2 and nano30b, which share a tokenizer and architecture).
\n\nKaggle notebook: chankhavu/c2-distill-vllm018-watchdog\n Model: chankhavu/c2-legaldistill-30b-a3b-fp8
\n
The solver we actually submitted is compact: no self-verification, no external tools beyond Python, no learned verifier. Five mechanisms wrapped around the FP8 soft-clone:
\nbudget = min(900, max(300, time_left β 300 Γ problems_left_others)). Early problems get the full 900 s; as the wall clock runs down we reserve 300 s per remaining problem and squeeze the current one to the 300 s floor.stateful_python_code_exec calls in a stateful IPython sandbox (30 s per call, 1000-char output cap, math/numpy/sympy/itertools/collections/mpmath preimported with mpmath.mp.dps = 64), append results as tool messages, continue. The rollout returns the integer inside the last \\boxed{β¦} when the model stops calling tools.1 / mean_token_entropy, where the per-token entropy is computed from top-5 logprobs over the generation. The winning answer is the one with the largest sum of rollout weights.Unfortunately, we only finished our Eagle3 work on the last 2 days of the competition, so we never had a clean submission with only Eagle3, without other harnesses that degrades the results (like self-verification or increased number of parallel rollouts).
\n\n\nThe analysis in this section is based on collected traces from simulated online LB runs (50 problems per run, 5h per run, 1x H100 SXM). The data is published here: chankhavu/aimo50-evals.
\n
To sanity-check how the offline numbers translate into the actual Kaggle setup, we replayed our final Cascade 2 voting submission and the gpt-oss-120b 44/50 public-notebook baseline on three independent shuffles of the same 50 problems, each under the real Kaggle budget: 5 hours wall-clock on 1Γ H100 SXM with an adaptive per-problem budget that starts at 900 s and can be squeezed down to 300 s as the wall clock runs low.
\nThese 50 problems are a subset of the 63 problems in our offline benchmark section. We removed problems with pass@8 = 0 for all 3 models: gpt-oss-120b, Nemotron Cascade 2, and Nemotron 3 Nano (so, basically impossible removed problems that were never solved).
![\"Left:](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2270821%2F0cc893180cba5bf84e4cf4ff7916fdbf%2Fkaggle_time_budget.png?generation=1776888430311498&alt=media\")
Two readings. (i) Cascade 2 (mean 34.7/50) is ahead of gpt-oss-120b (mean 33.0/50) on the LB scoreboard from the ~2h mark onward, and reaches gpt-oss-120b's final score about 20 min earlier in the average run. (ii) gpt-oss-120b's per-problem budget gets visibly squeezed toward the end: on aimo50c the last 5 problems ran at 300 s (the minimum), because the notebook was within minutes of the 4.83h ceiling. Cascade 2 held 900 s per problem for most of the run across all three shuffles, with plenty of headroom at the finish.
\nThree per-problem views of the 150 (problem Γ shuffle) pairs:
\n![\"Per-problem](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2270821%2F70f9e23ae071319d893dd81e92f91db5%2Fkaggle_per_problem_stats.png?generation=1779061973301798&alt=media\")
Left β vote convergence predicts correctness. When all 8 rollouts agree (bin 1), Cascade 2 is correct 82.8% of the time vs gpt-oss-120b's 74.6% β the online equivalent of the offline \"consistency β correctness\" story. Cascade 2 never produces a zero-answer case; gpt-oss-120b has 9 of 150 (all wrong). At 4+ distinct answers (both models stumped), Cascade 2 is slightly worse but has more such cases (30 vs 14), so the bin averages understate its overall advantage.
\nMiddle β which model actually matters on 29% of pairs. 53% both right, 18% both wrong, 16% Cascade-2-only right, 13% gpt-oss-only right. Net +5 pair-wise advantage to Cascade 2. Roughly half the LB score is \"easy for everyone\" and one-fifth is \"hard for everyone\"; the remaining third is where model choice actually moves the needle.
\nRight β where the wall-clock time actually goes. Decomposing total wall time across all 3 shuffles into four buckets β generation, successful tool call, Python-error tool call, timed-out tool call β Cascade 2 spends 88% in generation vs gpt-oss-120b's 74%, and 6.3% of Cascade 2's time vs 14.4% of gpt-oss-120b's is wasted on tool calls that produced no useful output (~2.2Γ more for gpt-oss-120b). For both solvers, the wasted time is almost entirely pure 30 s timeouts (12.5% / 5.9% of wall-clock) rather than Python errors (1.9% / 0.4%) β gpt-oss-120b's tool calls just trip the 30 s sandbox cap roughly 2Γ more often per rollout than Cascade 2's. Appendix G dives into the per-error-class breakdown and the most common root causes (forgotten imports, sandbox state rollback after a cell error).
\nCascade 2 voting vs the gpt-oss-120b baseline, log-derived:
\n| Submission (model, harness) | \nLocal validation (3 shuffles) | \nLocal val (mean) | \nTool err (per-call) | \nPublic LB attempts | \nPrivate LB | \n
|---|---|---|---|---|---|
| gpt-oss-120b, 44/50 public notebook | \n33, 30, 36 | \n33.0 | \n15.0% | \n38, 38, 37 | \nβ | \n
| Cascade 2, 8 rollouts, voting (our final submission) | \n36, 34, 34 | \n34.7 | \n10.6% | \n36, 39, 38 | \n41 | \n
Per-call tool error rates count any call that produced no useful output β Python errors and 30 s sandbox timeouts together.
\nThe fact that the results are in similar ballpark as Public LB (30 - 36) is pure coincidence - we did not cherry-pick the problems for our benchmark in any way. It means that the Public LB are really on the same difficulty level as the IMO-AnswerBench, which is a testament of how far we've come since AIMO1!
\nCascade 2 wins local validation by 1.7 points and matches gpt-oss-120b on the LB mean (both 37.7), peaking higher (39 vs 38). Both finalists score meaningfully higher on the LB than on our local mean (gpt-oss +4.7, Cascade 2 voting +3.0), which suggests IMO-AnswerBench (our local set) is slightly harder than the hidden LB set. The self-verification harnesses we tried β folded into the full comparison table at the end of the Self-verification experiments section below β both underperformed plain voting.
\n\n\n
\n- Notebook we submitted to Public LB: https://www.kaggle.com/code/chankhavu/c2-distill-verifier?scriptVersionId=311859342
\n- Caveat up front. We built and evaluated the entire self-verification harness within the last 24 hours before the competition deadline β no time to tune any of the hyperparameters (pair-selection policy, grader prompt, max-verifications cap, temperature, rating scale, ΞΌ-score weighting, fallback thresholds). Treat the numbers below as first-attempt measurements, not a carefully-optimised upper bound on what self-verification can do.
\n
The voting harness leaves an obvious source of slack: on most problems the 8 solution rollouts finish at wildly different times. The first rollout drops its \\boxed{} at 100β200 s, the slowest can take 500+ s, and in between we're sitting on GPU capacity we've already paid for. The self-verification variants from the table above use that capacity to run pairwise grader sessions that rate competing answers against each other.
![\"Left:](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2270821%2Fd71a07030d85e1049dbf373179fd355d%2Fkaggle_self_verification_gantt.png?generation=1776888538729281&alt=media\")
\n
We borrow the core idea β pairwise LLM-as-judge comparisons aggregated into an uncertainty-weighted score β from V1: Toward Verifiable Reasoning in LLMs by Harman et al. Our pair-selection policy differs: V1 picks the next pair adaptively to maximise information gain, while we use a simpler greedy rule (newest answer vs current undefeated bucket). Our winner-selection scoring stays close to theirs.
\nOn top of the voting harness (same model, same sampling, same adaptive budget, same early-stop at four agreeing rollouts), the self-verification variants add three conceptual pieces:
\nEach time a solution rollout completes, the harness attempts to launch up to two verifications in whichever GPU slots are free, subject to a cap on concurrent rollouts. The tracker picks the next comparison in priority order:
\nCompleted verifications also re-trigger a new comparison if there's still wall-clock budget and solution rollouts are still arriving. Verifications are capped at a fixed maximum per problem so they can't swallow the solver's budget on pathological cases.
\nLet undefeated be the set of buckets with zero losses in the win/loss graph. If exactly one bucket is undefeated, we pick it β done. If multiple remain undefeated (or none, in the cycle case), we rank them by an uncertainty-weighted score
\n$$\\mu_i = \\frac{\\sum_j w_{ij} \\cdot v_{ij}}{\\sum_j w_{ij}}\\,,$$
\nwhere $v_{ij} \\in {1, \\tfrac{1}{2}, 0}$ is the outcome (win / tie / loss) of bucket i against bucket j, and $w_{ij} = \\max(|r_i - r_j| / 9,\\, 0.1)$ scales each comparison by its rating margin: a 10β1 verdict carries an order of magnitude more weight than a 5β4 one, with a floor of 0.1 so ties still count. Further ties break by bucket size (number of rollouts that produced that answer), then by comparison count. If fewer than half the distinct answer buckets have been graded at all, we treat the grading as too sparse to trust and fall back to the entropy-weighted majority vote from the baseline voting solver.
\nOn the problem in the chart, voting has nothing useful to say: three rollouts say 1, three say 12, only two say 13 β the correct answer. Voting breaks the tie by entropy-weighted score, picks 12 (score 5.13 vs 1's 4.71 and 13's 3.55), and is wrong. The grader only needs three pairwise comparisons to settle it (13 beats 1 with 10β1, 13 beats 12 with 9β2, and 12 beats 1 with 3β1), leaving 13 as the only undefeated bucket. This is the regime self-verification is supposed to win: when a single well-reasoned rollout can recognise a majority of sloppy rollouts for what they are.
Despite the elegance, the table tells the disappointing story: self-verification underperformed voting on both local validation (32.0 / 33.0 vs 34.7) and public LB (34 vs 36/39/38). We tried two configurations β 8 solution rollouts with verification layered on top, and 6 solution rollouts with more verification budget per problem β and neither beat plain voting.
\nWhen does the grader even do something? Across all 297 problem-shuffles we ran under self-verification (three variants Γ two shuffles Γ ~50 problems), the grader agrees with entropy-weighted voting's top answer 84.5 % of the time (251 / 297) β meaning on 5 out of every 6 problems the verification layer adds literally nothing. On the remaining 15.5 % (46 cases) the grader flips the decision, and its hit rate on those flips is what really matters:
\n| Harness | \nFlips | \nGrader right (voting wrong) | \nGrader wrong (voting right) | \nBoth wrong | \nNet | \n
|---|---|---|---|---|---|
| Cascade 2, 8 rollouts, self-verification | \n11 | \n6 | \n3 | \n2 | \n+3 | \n
| Cascade 2, 6 rollouts, self-verification | \n11 | \n6 | \n2 | \n3 | \n+4 | \n
| Cascade 2 + Eagle3, 6 rollouts | \n24 | \n10 | \n10 | \n4 | \n0 | \n
| All self-verif (aggregate) | \n46 | \n22 | \n15 | \n9 | \n+7 / 297 | \n
So on the two non-Eagle3 runs the grader is net positive but only barely β +7 correct answers across 297 problem-shuffles, roughly 2.4 %. The Eagle3 run is the real tell: its 33 % throughput headroom lets the scheduler fire more than twice as many flip-candidate comparisons (24 vs 11), and the extra firings split right down the middle β ten wins cancelled by ten losses. More grading, no net gain. At the margins where voting and the grader disagree, the grader is essentially a coin flip.
\n![\"Two](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2270821%2Fc2f032a216884a70d93e74af479db525%2Fkaggle_self_verification_time_budget.png?generation=1776888592784811&alt=media\")
Our read: the grader is too lenient to break ties confidently when both answers are backed by plausible-looking reasoning, and β as the chart above makes clear β at 1Γ H100 SXM there simply isn't enough wall-clock budget to run the 15β20 comparisons per problem the tracker would need to converge on harder problems. The per-problem means are close across all three runs (329 s voting vs 342 s self-verification vs 339 s with Eagle3), but the tails tell a very different story: voting's 90th-percentile problem consumes 472 s, self-verification's reaches 589 s, and with Eagle3 it stretches to 796 s β the longer the grader can keep running comparisons, the longer individual problems drag on, and the scheduler simply squeezes the remaining budget harder. Eagle3 buys roughly a 33 % throughput improvement (our motivation for training it in the first place), but even on that budget the extra complexity of the verification layer didn't translate into more correct answers β the Eagle3 run landed at 33.5 / 50 (mean of 36 and 31), which is no better than the plain self-verification variant and still behind voting's 34.7. The ingredients β fast model, honest grader β were right; the budget wasn't.
\n| Submission (model, harness) | \nLocal validation (3 shuffles) | \nLocal val (mean) | \nTool err (per-call) | \nPublic LB attempts | \n
|---|---|---|---|---|
| gpt-oss-120b, 44/50 public notebook | \n33, 30, 36 | \n33.0 | \n15.0% | \n38, 38, 37 | \n
| Cascade 2, 8 rollouts, self-verification | \n33, 31 | \n32.0 | \n5.1% | \n34 | \n
| Cascade 2, 6 rollouts, self-verification | \n35, 31 | \n33.0 | \n5.2% | \nnever submitted | \n
| Cascade 2 + Eagle3, 6 rollouts, self-verification | \n36, 31 | \n33.5 | \n5.2% | \nnever submitted | \n
| Cascade 2, 8 rollouts, voting (our final submission) | \n36, 34, 34 | \n34.7 | \n6.0% | \n36, 39, 38 | \n
The boring harness wins. Our final submission is the simplest thing that works: 8 rollouts of soft-cloned FP8 Cascade 2 with Python tool use, an adaptive 300β900 s per-problem budget, early-stop at 4 agreeing answers, and an entropy-weighted majority vote. On the 3-shuffle local validation (IMO-AnswersBench, 5 h, 1Γ H100 SXM) it scores 34.7 / 50, ahead of the public gpt-oss-120b 44/50 notebook's 33.0 / 50, and it ties gpt-oss-120b on the public LB (mean 37.7, peak 39 / 50 vs 38). Every more-sophisticated Cascade 2 harness we tried β pairwise-comparison self-verification, the same plus Eagle3-accelerated inference β underperformed the baseline. We believe self-verification could beat voting with more tuning, but not in the 24 hours we had before the deadline.
\nWhat the side quests produced. Three open-source artifacts that outlast the competition: a family of quantized Cascade 2 checkpoints (FP8, NVFP4, AWQ-INT4 β 66K downloads and counting), a vLLM fork that correctly runs prefix caching alongside speculative decoding on hybrid-Mamba models (long-standing open problem), and an EAGLE-3 speculator for Cascade 2 reaching 3.55 mean accepted tokens at K=7 along with the SpecForge modifications needed to train it at 64 k context. We also filed a note on Cascade 2's tendency to quietly search the web for answers through its Python tool.
\nWhy this is mostly a first-try harness. We joined the competition late and the Kaggle 1-submission-per-day cap meant essentially every choice we shipped was a first-try call, with no headroom to A/B-test on the hidden LB. That's also why our scheduling, early-stop, and budget policy are deliberately simple: we picked Cascade 2 specifically because it offered the best maj@k under a time budget, and our priors from AIMO 2 were that winning solutions in this class of competition optimized maj@k rather than chasing pass@k at infinity β pass@β tends to be similar across competitive models, but maj@k under a real time budget is where the consistency premium pays off. A more elaborate per-problem scheduling system was on the wishlist (several contestants reported good gains from one), but the engineering dependency we did spend weeks on β PC + SD for hybrid-Mamba models in vLLM β was already over our time budget, and we judged that getting the simple harness right was a safer use of the remaining days than tuning a scheduler we couldn't validate without using up the submission slots.
\nWhat we'd do differently. Finalize PC + SD support in vLLM earlier (this was the engineering dependency that would have unlocked a genuinely competitive self-verification harness), start the self-verification work on day one instead of the last 24 hours, spend some time tuning the grader (prompt, rating scale, pair-selection policy), and develop a smarter per-problem scheduling policy that exploits the random hard/easy ordering. The ingredients were right; we just ran out of room to tune them. Even so, we're glad we got side-tracked β the open-source artifacts are the part of this project we're most proud of.
\nThe recovery loop runs per tensor, one tensor type at a time. For each teacher tensor we (1) generate random probe inputs $X \\sim \\mathcal{N}(0, 1)$, (2) observe the teacher's output $Y = f_\\text{teacher}(X)$ β this is the only step that touches teacher values β and (3) solve for student parameters that minimise the mean-squared error $\\lVert f_\\text{student}(X) - Y \\rVert^2$. Linear layers $y = W x$ are recovered in closed form via the normal equations (it's already an obvious MSE minimiser and the loop would add ~100Γ runtime for no benefit). The other three tensor types β depthwise Conv1d kernels, 1D scalar vectors ($y = w \\odot x$; norms, SSM $A_{\\log}$, $d_t$, $D$), and additive biases ($y = x + b$) β all run an explicit SGD loop with a learning rate that cosine-decays to 0 over 500 steps. Max learning rates are exposed as CLI flags; the choices below come from stability arguments on the per-parameter Hessian.
\nEach recovery loss is a convex quadratic, and SGD is stable exactly when $\\mathrm{lr} < 2/\\lambda_{\\max}(H)$ with $H$ the per-parameter Hessian. For the $y = w \\odot X$ and $y = X + b$ cases the loss is separable across features, so with the normalisation we use (mean over samples, sum over features) every parameter sees the same scalar Hessian $H = 2\\,\\mathbb{E}[X^2] = 2$ for $X \\sim \\mathcal{N}(0, 1)$ β hence $\\mathrm{lr} = 0.5 \\approx 1/H$ lands each element at its optimum in $O(1)$ effective steps, independent of $N$ and of the magnitude of the target.
\nThe unfolded-window conv1d Hessian is not a scaled identity. Its $(k_1, k_2)$ entry is the sample autocorrelation of standard noise at lag $|k_1 - k_2|$: diagonal $\\approx T$, off-diagonals have $\\mathcal{O}(\\sqrt{T})$ noise fluctuations (plus small deterministic shifts near the zero-padded boundary of the window). Divided by $T$ in the loss, the spectrum concentrates around $2$ but spreads to roughly $[1.3, 3.5]$ at typical Mamba sizes ($T = K + 32 = 36$, $K = 4$). $\\mathrm{lr} = 0.5$ is right on the stability edge for $C \\lesssim 10^3$ and clearly past it by $C \\gtrsim 4 \\cdot 10^3$ (it diverges with extreme numeric blow-up). $\\mathrm{lr} = 0.3$ sits inside $2 / 3.5 \\approx 0.57$ with comfortable margin across every Cascade 2 size we ran.
\nThe per-component precisions behind the layout chart in the main section. Each row maps an architectural component to the precision it's kept at in the FP8 and NVFP4 checkpoints, following the NVIDIA Nano sensitivity recipe.
\n| Component | \nPrecision (FP8) | \nPrecision (NVFP4) | \nComment | \n
|---|---|---|---|
| MoE expert GEMMs (routed + shared) | \nFP8 | \nNVFP4 | \nAll 23 MoE layers, 128 routed + 2 shared experts each | \n
| Mamba GEMMs (non-adjacent) | \nFP8 | \nNVFP4 | \n17 of 23 Mamba layers | \n
| Attention layers (all 6) | \nBF16 | \nBF16 | \nMost sensitive β kept BF16 per NVIDIA sensitivity analysis | \n
| Mamba layers adjacent to attention (6) | \nBF16 | \nBF16 | \nLayers {4, 11, 18, 25, 32, 41} β found sensitive in ablations | \n
| Mamba 1D conv | \nBF16 | \nBF16 | \nAll layers | \n
| Router gates | \nFP32 | \nFP32 | \nRouting precision must not degrade | \n
| Embeddings & lm_head | \nBF16 | \nBF16 | \nNot quantized | \n
| KV cache | \nFP8 | \nFP8 | \nAll 6 attention layers | \n
All benchmark runs use NVIDIA-NeMo/Evaluator with the Nemotron-3-Super-120B tool-use config: 8 rollouts per problem, temperature 1.0, tool use enabled (sandbox code execution), enable_thinking=true. Same config across the BF16 reference and our FP8 / NVFP4 checkpoints so numbers are directly comparable.
\n\nCritical serving flag. NemotronH's SSM state must stay in FP32 at inference. vLLM 0.19 reads this from the model's
\nconfig.jsonby default (Cascade 2 ships the override), but anyone serving a variant whose config doesn't include it should pass--mamba_ssm_cache_dtype float32explicitly. Leaving the SSM state at BF16 costs ~10% absolute on AIME-class benchmarks β silent precision regression, no crash. The same applies to SGLang (--mamba-ssm-dtype float32), which defaults to BF16.
| Benchmark | \nNemotron-Cascade-2-30B-A3B (reproduced results) | \nOur FP8 model | \nOur NVFP4 model | \n
|---|---|---|---|
| AIME 2025 (avg@8) | \n98.8 | \n96.7 | \n97.9 | \n
| AIME 2026 (avg@8) | \n94.2 | \n95.0 | \n92.1 | \n
| HMMT Feb 2025 (avg@8) | \n92.9 | \n93.8 | \n90.1 | \n
With the low sample count (8 rollouts per problem), a deviation of Β±2% across runs is expected. Safe to say BF16, FP8, and NVFP4 variants are equivalent in reasoning performance.
\nAt the sweet spot (K=4, T=0.7 on AIME-25):
\n| Position | \n1 | \n2 | \n3 | \n4 | \n5 | \n
|---|---|---|---|---|---|
| Accept rate | \n73.3% | \n49.8% | \n34.5% | \n24.7% | \n17.3% | \n
Rates decay geometrically with a ratio of ~0.7 per position β each additional draft token needs all previous drafts to verify first, and the marginal drafter confidence drops with depth.
\nNaΓ―ve training at L=65k, ttt=6 peaks at 120+ GiB β OOM on a 96 GB GPU. Four independent activation-memory cuts on the draft forward + TTT unroll bring peak down to ~88β91 GiB, approximately 63 GiB verifier weights + 4.2 GiB target_p_padded + 12 GiB attention activations + 4.2 GiB MLP saves + ~5β8 GiB overhead:
[seq, 131072] logits as target (~16 GiB at L=65k), held through three independent references across the draft forward + backward. Precomputing a small draft-vocab target_p_padded (~4 GiB) and dropping the rest before eagle3_model.forward() releases the full-vocab tensor before the expensive phase. Dropping the .float() upcast on the softmax saves another ~8 GiB.[B, T, V] logits tensor is never materialized β the existing LogSoftmaxLoss Triton kernel runs inside a chunked + grad-checkpointed wrapper. Numerically equivalent to the unchunked path on both hidden_states and lm_head_weight grads, verified at realistic shapes.LlamaForCausalLMEagle3, the flex_attention mask generator, and the causal-mask builder. The TTT suffix block is intentionally not windowed.The four savings don't sum linearly β some overlap (e.g., freeing target logits shrinks the memory available for future cuts to operate on). Aggregate per-step wall time is ~4 s (1.2 s on short samples up to 8.6 s on near-max-length conversations at L=65k); grad-checkpointed MLP costs ~8β12% per-step recompute overhead, which the memory savings more than pay for.
\nGeneralParser.parse() was running per-turn re-tokenization in O(num_turns Γ seq_len) β for long reasoning traces with many assistant turns this took several seconds per conversation. We replaced it with a single return_offsets_mapping + bisect pass to map character spans to token spans in O(num_turns Γ log seq_len). Per-conversation preprocessing dropped from several seconds to ~30 ms on long reasoning traces.
Attempt-count view. What the time-budget chart in the body would look like if we drop the per-rollout-cost scaling and plot against raw k. Same data, x-axis = k instead of k Γ seconds-per-rollout.
![\"pass@k](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2270821%2F02418e1ddd660c78b71374af498f4ef3%2Ftest_time_scaling_attempts.png?generation=1779062011546385&alt=media\")
Time-budget chart. For each model m and k β {1, β¦, 8}, we plot the (pass@k, maj@k) point at x = k Γ avg_tokens_per_rollout(m) / tok-per-sec(m) seconds, using the 8-parallel-sequence throughput from the generation-speed table. pass@k and maj@k are computed exactly (no Monte-Carlo) by enumerating all C(8, k) subsets of the 8 available sessions per problem and averaging across the 63 problems. maj@k uses the most-common answer rule: for each subset, the modal answer wins, ties split uniformly across tied answers.
Reasoning-time budget chart. For each truncation length L β {16k, 32k, 48k, 64k, 80k, 96k}:
\n\\boxed{β¦} inside the truncated portion. If found, that token position is the session's effective length and its original predicted answer counts. If not found, the session is treated as \"no answer\" at that L and its effective length is L.Tokenization. Per-model, exact:
\nnum_completion_tokens from each trace record (the model's own Harmony token count for generated output, exclusive of prompt and tool results). Because gpt-oss emits \\boxed{β¦} as its final generation, we approximate its \\boxed{}-end token position as the total generated-token count.generation text with the Nemotron-Cascade-2 tokenizer (nvidia/Nemotron-Cascade-2-30B-A3B) and compute both the total and the \\boxed{}-end positions exactly by running the tokenizer on the full text and on the prefix up to the closing brace of the last \\boxed{β¦}.Each pairwise verification is its own chat session with Python tool use enabled. System message:
\nYou are a helpful assistant and an expert math contest grader.\nUser message (fields {problem}, {sol_A}, {sol_B} substituted from the current problem and two competing solution texts):
Compare two submissions and rate correctness based solely on the final answer.\n\n**Evaluation Guidelines:**\n- Extract each submission's final answer. Use any provided reasoning only\n to help you assess whether the stated final answer is trustworthy. Do\n not award credit for method quality or rigor.\n- Carefully analyze the problem statement and the submissions to assess\n whether each final answer is correct. Grade only the final answer.\n**Problem**\n{problem}\n**Solution A**\n{sol_A}\n**Solution B**\n{sol_B}\n**Output Format:**\nFirst, provide your reasoning (what checks you performed). Then, on\nseparate new lines, give ratings using the EXACT tags below. Add no\nother text after the tags.\n<rating_A>INTEGER_1_TO_10</rating_A>\n<rating_B>INTEGER_1_TO_10</rating_B>\n**Rating Rules:**\n- Rate correctness on a 1-10 scale (10 = certainly correct, 8 = very\n likely correct, 5 = uncertain/borderline, 3 = likely incorrect, 1 =\n certainly incorrect).\n- Higher rating wins. Equal ratings imply a tie.\nPlease provide your analysis now.\nWe parse the rating pair out of the model's final message with an XML-tag regex first, then fall back to \\boxed{N} per side if the model used its trained \\boxed{} format instead of the requested tags.
All the knobs we actually shipped for chankhavu/c2-distill-vllm018-watchdog.
\nModel. chankhavu/c2-legaldistill-30b-a3b-fp8 β the FP8-quantized soft-clone of Nemotron-Cascade-2, re-quantized specifically for the Kaggle submission (not our published open-source FP8 checkpoint; see the \"Quantizing the Cascade 2 model\" section). Runtime: vLLM 0.18.0 on 1Γ H100 SXM under the Kaggle inference server sandbox.
vLLM serve command. Single GPU, FP8 KV cache, large context window, chunked prefill, prefix caching, async scheduling, native function-calling enabled with the qwen3_coder tool-call parser:
vllm serve /path/to/c2-legaldistill-30b-a3b-fp8 \\\n --served-model-name chankhavu/c2-legaldistill-30b-a3b-fp8 \\\n --tensor-parallel-size 1 \\\n --max-num-seqs 64 \\\n --gpu-memory-utilization 0.9 \\\n --dtype auto \\\n --max-model-len 262144 \\\n --max-num-batched-tokens 8192 \\\n --kv-cache-dtype fp8 \\\n --mamba-ssm-cache-dtype float32 \\\n --mamba-cache-mode all \\\n --mamba-block-size 512 \\\n --enable-chunked-prefill \\\n --enable-prefix-caching \\\n --async-scheduling \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --trust-remote-code \\\n --host 0.0.0.0 --port 8000\nWe also run a lightweight watchdog subprocess that polls /proc every 30 s and relaunches vLLM if the APIServer / EngineCore processes disappear (scrapes the VLLM::APIServer / VLLM::EngineCor... comm names β the literal vllm.entrypoints string stops appearing in cmdline after vLLM calls setproctitle during startup). The relaunch itself takes ~100 s, which is why the watchdog restart trigger is \"no live vllm process found\" rather than HTTP-based.
Solver configuration (from CFG):
| Setting | \nValue | \nNotes | \n
|---|---|---|
attempts | \n8 | \nSolution rollouts per problem | \n
workers | \n16 | \nThread pool size (2Γ attempts, so cancellations don't stall new launches) | \n
early_stop | \n4 | \nStop once 4 rollouts agree on an integer answer | \n
turns | \n128 | \nMax conversation turns per rollout | \n
temperature | \n1.0 | \nSampling diversity β all rollouts share the prompt | \n
top_p | \n0.95 | \n\n |
top_logprobs | \n5 | \nTop-5 logprobs for entropy computation | \n
enable_thinking | \nTrue | \nCascade 2's <think>β¦</think> reasoning | \n
context_tokens | \n262144 | \nvLLM --max-model-len | \n
max_completion_tokens | \n131072 | \nPer-turn completion cap | \n
notebook_limit | \n17400 s (4.83 h) | \nKaggle inference ceiling | \n
high_problem_timeout | \n900 s | \nDefault per-problem budget | \n
base_problem_timeout | \n300 s | \nReserved per remaining problem; also the per-problem floor | \n
python_timeout | \n30 s | \nPer Python tool call | \n
max_output_characters | \n1000 | \nSandbox stdout truncation | \n
seed | \n42 | \nUnused (vLLM picks fresh seeds per request β we want the cross-worker diversity) | \n
Adaptive budget formula. For each problem we compute\nbudget = min(high_problem_timeout, max(base_problem_timeout, time_left β base_problem_timeout Γ problems_left_others)), so early problems get the full 900 s; as the wall clock runs down we reserve 300 s per remaining problem and squeeze the current one toward the 300 s floor.
System prompt.
\nYou are a helpful assistant and a mathematical problem solver.\nSolve the given problem step by step using rigorous reasoning.\nUser template ({problem} substituted per problem):
{problem}\n\nThe answer is a non-negative integer between 0 and 99999, inclusive.\nPut your final answer in \\boxed{}.\nTool definition β the model is given exactly one function, a stateful IPython code executor:
\n{\n \"type\": \"function\",\n \"function\": {\n \"name\": \"stateful_python_code_exec\",\n \"description\": \"Call this function to execute Python code in a stateful Jupyter notebook environment. Python will respond with the output of the execution or time out after 120.0 seconds. The environment has math, numpy, sympy, itertools, collections, and mpmath pre-imported.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"code\": {\"type\": \"string\", \"description\": \"Code to execute\"}\n },\n \"required\": [\"code\"]\n }\n }\n}\nPer-call timeout is actually 30 s (from CFG.python_timeout), not 120 s β the description is the one the Cascade 2 model was trained to expect, so we leave it unchanged.
Sandbox preimports. Every new session starts with:
\nimport math\nimport numpy\nimport sympy\nimport itertools\nimport collections\nimport mpmath\nmpmath.mp.dps = 64\nAnswer extraction. A small regex cascade applied to each rollout's final non-tool-call message, in order: (i) \\boxed{DIGITS} β pick the last one; (ii) \\boxed{β¦ = DIGITS} β pick the last integer inside the last \\boxed{}; (iii) final answer is DIGITS. Accepts integers in [0, 99999] only. Before scanning, <think>β¦</think> reasoning blocks are stripped.
Voting. Each rollout's weight is 1 / max(mean_token_entropy, 1e-9), where the per-token entropy is computed from the top-5 logprobs of the generated tokens (not tool-call bytes). The picked answer is the one with the highest sum of weights across the rollouts that produced it.
Hyperparameters. Grader runs at temperature 0, top_p 0.95, same Python sandbox as the solver. Tool-call budget per grader session: 15 calls; past that we force-respond to any pending tool calls with a stub and instruct the model to output its ratings immediately. Maximum verifications per problem: 16. Maximum concurrent rollouts (solution + verification combined) in the thread pool: 8.
\nWhy does gpt-oss-120b's tool-call overhead come out roughly 2Γ Cascade 2's? The whole question is really about time: a typical gpt-oss-120b rollout burns 37 s of wall-clock on errored or timed-out tool calls β 14.0% of its 266 s average rollout β vs 17 s for Cascade 2 (6.3% of its 272 s rollout). ~2.2Γ the absolute, ~2.2Γ the rollout fraction. All numbers below are from the same LB-simulation runs that feed the chart in Β§Per-problem breakdown (aimo50a/b/c, 3 shuffles Γ 50 problems Γ up to 8 rollouts per problem, 1Γ H100 SXM under the 5 h Kaggle budget).
\nPer-rollout averages β every column below is a mean across rollouts (912 for gpt-oss-120b, 850 for Cascade 2; < 50Γ3Γ8 = 1200 because the adaptive scheduler cancels in-flight rollouts once 4 agree on an answer). In the table, \"error\" means any failed tool call β Python exceptions and the 30 s sandbox timeouts together.
\n| Model | \nrollout sec | \ntool calls | \ntool sec | \nerror sec | \n% rollout errored | \n
|---|---|---|---|---|---|
| gpt-oss-120b | \n266 | \n18.4 | \n66.3 | \n37.3 | \n14.0% | \n
| Cascade 2 | \n272 | \n14.3 | \n32.4 | \n17.1 | \n6.3% | \n
Within each model, where the error+timeout time goes (% errs and % err-time are within-model fractions of the error+timeout totals):
All low-frequency classes (< 0.1 occurrences per rollout in both models) are collapsed into \"other\" β this includes RuntimeError, ValueError, KeyError, IndexError, ImportError, and uncategorized Python tracebacks.
\n| Class | \ngpt-oss-120b count/rollout | \nsec/rollout | \n% errs | \n% err-time | \nCascade 2 count/rollout | \nsec/rollout | \n% errs | \n% err-time | \n
|---|---|---|---|---|---|---|---|---|
| [TIMEOUT] (30 s cap) | \n1.01 | \n33.1 | \n36.4% | \n88.6% | \n0.51 | \n16.0 | \n33.1% | \n94.0% | \n
| SyntaxError | \n0.72 | \n0.3 | \n26.1% | \n0.9% | \n0.11 | \n0.0 | \n7.3% | \n0.1% | \n
| NameError | \n0.49 | \n2.3 | \n17.8% | \n6.0% | \n0.15 | \n0.1 | \n9.7% | \n0.7% | \n
| TypeError | \n0.17 | \n0.1 | \n6.1% | \n0.3% | \n0.23 | \n0.0 | \n15.1% | \n0.2% | \n
| AttributeError | \n0.10 | \n0.0 | \n3.6% | \n0.1% | \n0.10 | \n0.1 | \n6.5% | \n0.4% | \n
| other | \n0.28 | \n1.5 | \n10.0% | \n4.1% | \n0.41 | \n0.7 | \n28.3% | \n4.6% | \n
| All errored + timed-out | \n2.77 | \n37.3 | \n100% | \n100% | \n1.53 | \n17.1 | \n100% | \n100% | \n
Timeouts are ~90% of the wasted wall-clock for both solvers (88.6% / 94.0%) β every Python error class is a rounding error by time.
\nNameError carries the strongest cross-model signal (0.49/rollout for gpt-oss-120b vs 0.15 for Cascade 2). gpt-oss-120b's 451 extractable NameErrors split into two causes:
\nsp (16), random (15), deque (9), math (5), and other stdlib aliases are already live, but the pre-import block (math, numpy, sympy, itertools, collections, mpmath) doesn't expose them and the model rarely writes the import line itself.Following up on the questions from @philipvonderlind, here are some additional questions to help us assess your submission:
\nLook forward to reading your responses on these.
", "votes": 1, "replies": [ { "id": 3459537, "author_name": "Chan Kha Vu", "author_url": "", "post_date": "2026-05-17T23:58:58.033000", "content": "Thanks a lot, Sam Bealing (@sambealing) for thoughtful questions! We had similar concerns during AIMO3. In fact, we were skeptical about Cascade 2 at first and were not sure if it was benchmaxxed. Here is why we ultimately chose to trust it.
\n\n\nand also AnswerBench contamination concern: Cascade 2 was released in March 2026, a few months after IMO-AnswerBench was published, whereas gpt-oss-120b was released earlier. Are you worried that Cascade 2's higher score on your local validation set (drawn from AnswerBench) might partly reflect AnswerBench problems having been in Cascade 2's training data, rather than a fair reasoning comparison against gpt-oss-120b?
\n
This is a legitimate concern and we've added a note to Β§Evaluation set acknowledging it explicitly. Short version:
\nThe contamination caveat is now in the writeup body so the reader can weight the absolute numbers themselves.
\n\n\nWhy Cascade 2 over a gpt-oss-120b verifier path: Your offline numbers show gpt-oss-120b has the higher pass@8 (77.8% vs Cascade 2's 74.6%), and you note an oracle verifier would pick gpt-oss-120b. You then invested heavily in the soft-clone + quantization + speculator pipeline for Cascade 2. With hindsight, what made you commit to the Cascade 2 route rather than building (e.g. a learned) verifier for gpt-oss-120b that could close the maj@8 β pass@8 gap?
\n
A few interacting reasons, in roughly the order they mattered:
\nThe resource-constraint version of this argument is now in the writeup as a follow-up paragraph after the oracle-verifier note in Β§Test-time scaling. The training-time / risk-budget version we hadn't articulated; thank you for prompting it.
", "votes": 0, "replies": [] } ] }, { "id": 3457297, "author_name": "Philip Vonderlind", "author_url": "", "post_date": "2026-05-13T13:49:55.270000", "content": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.
\nYour work shows impressive contributions, including implementing a missing feature in vLLM. We especially appreciate the in-depth engineering insights you provided, which even helped resolve issues that other participants encountered during the competition. I only have a few more surface-level questions:
\nvLLM: Was your PR eventually accepted into vLLM? I have read several other writeups from participants who used EAGLE3 and encountered the same issue you describe above. Please keep us posted if there are any updates.
Pass@k Plot: Did you mean to write that gpt-oss-120b wins pass@k at every time budget? Based on the plot, it seems that Cascade outperforms gpt-oss-120b across all tested budgets.
Time Management: Since EAGLE3 primarily improves inference speed and therefore saves time, why did you not spend more effort on developing a more advanced time-management system as well to further increase the gains? Many other contestants found that a good scheduling strategy could help, especially because hard and easy problems appeared in random order.
Tool Calls: Did you investigate the common causes of the significantly higher tool-call error rate in gpt-oss-120b, such as missing packages, outdated APIs, syntax issues, or similar failure modes? It would be interesting to understand why the Nemotron model appears to produce much cleaner tool calls.
Looking forward to your response :)
", "votes": 1, "replies": [ { "id": 3459539, "author_name": "Chan Kha Vu", "author_url": "", "post_date": "2026-05-17T23:59:55.407000", "content": "Hi Philip Vonderlind (@philipvonderlind), thank you for thoughtful questions. Your 2nd question raised a real concern that we were not clear about our wording enough. The \"gpt-oss-120b always wins pass@k\" is false, and it should be attributed to \"wins given attempt budget\", not \"time budget\".
\nYour 4th question called out a real mistake in our numbers and analysis. We carefully re-visited that analysis and re-generated the chart (basically it was a parsing issue on our end due to prompt differences between gpt-oss and cascade 2).
\n\n\nvLLM: Was your PR eventually accepted into vLLM? I have read several other writeups from participants who used EAGLE3 and encountered the same issue you describe above. Please keep us posted if there are any updates.
\n
Not merged yet β we're still working through two issues before it's ready to upstream:
\nvllm-project/vllm#41233 was opened against the same mamba-cache-poisoning bug. It's still open at the time of writing, but it's likely the path upstream takes. Our perf optimizations are orthogonal to that PR (it has no speed/kernels work) and would still apply on top of it; we plan to submit them as a follow-up once the correctness side lands and our Blackwell story is fixed.The fork remains useful for anyone running Hybrid-Mamba SD on Hopper today. We added a new Β§Upstream PR status subsection inside the writeup's vLLM section to track this honestly, and we'll keep you posted as the situation upstream evolves.
\n\n\nPass@k Plot: Did you mean to write that gpt-oss-120b wins pass@k at every time budget? Based on the plot, it seems that Cascade outperforms gpt-oss-120b across all tested budgets.
\n
You're right β the original sentence was wrong. Within the plotted time budget, Cascade 2 wins both pass@k and maj@k: maj@k by a comfortable margin across the whole range, and pass@k by a smaller margin because Cascade 2 is cheaper per rollout and gets to fit more rollouts in any fixed wall-clock budget.
\nWhat we should have written, and now do: gpt-oss-120b has a higher pass@k ceiling that only becomes visible beyond the plotted budget β its rollouts are more diverse (helps pass@k, hurts plurality voting), so given enough attempts a hypothetical oracle verifier would eventually prefer gpt-oss-120b. Inside the actual competition's time budget, Cascade 2 dominates on both axes.
\nThe paragraph next to the chart has been rewritten to reflect this. We've also added a note explaining why we couldn't realize that pass@k ceiling within Kaggle's resource budget anyway: a separate verifier won't fit alongside gpt-oss-120b's ~120 GB MXFP4 weights on a single H100, and using gpt-oss-120b as its own verifier doubles the per-problem token bill on a budget that already only fits ~8 solution rollouts.
\n\n\nTime Management: Since EAGLE3 primarily improves inference speed and therefore saves time, why did you not spend more effort on developing a more advanced time-management system as well to further increase the gains? Many other contestants found that a good scheduling strategy could help, especially because hard and easy problems appeared in random order.
\n
Honest answer: we joined the competition late, and the Kaggle 1-submission-per-day cap meant essentially every scheduling choice we shipped was a first-try call, with no headroom to A/B-test on the hidden LB. A more elaborate per-problem scheduler was on the wishlist, but the engineering dependency we did spend weeks on β PC + SD support for hybrid-Mamba models in vLLM β was already over our time budget, and we judged that getting the simple harness right was a safer use of the remaining days than tuning a scheduler we couldn't validate without burning submission slots.
\nWe also weighted this decision by what optimizing for in the first place. Our priors from AIMO 2 were that winning solutions in this class of competition optimized for maj@k under a time budget, not pass@β β pass@β tends to be similar across competitive models, but maj@k under a real budget is where the consistency premium pays off. That's why we picked Cascade 2, which (per the test-time-scaling chart) gives the best maj@k under a wall-clock budget. Cascade 2's lower per-rollout cost also means an adaptive scheduler has less to squeeze: the simple min(900, max(300, time_left β 300 Γ problems_left_others)) formula already kept us at the 900 s ceiling for most of each shuffle.
In hindsight, a smarter scheduler would have been a high-leverage thing to try, and we've added it to the What we'd do differently list in the Conclusion. The Conclusion now also has a \"Why this is mostly a first-try harness\" paragraph that articulates this trade-off explicitly.
\n\n\nTool Calls: Did you investigate the common causes of the significantly higher tool-call error rate in gpt-oss-120b, such as missing packages, outdated APIs, syntax issues, or similar failure modes? It would be interesting to understand why the Nemotron model appears to produce much cleaner tool calls.
\n
Yes, in depth. We added Appendix G (tool-call error breakdown) which re-pairs every tool response in the LB-sim with the code that triggered it (504 rollouts per model). Headline findings:
\nsp (sympy alias), random, deque, math, etc. The pre-import block (math, numpy, sympy, itertools, collections, mpmath) doesn't expose sympy as sp or random, and gpt-oss-120b writes Jupyter-cell-style snippets that assume these names are already live in the session rather than import-ing them.NameError on what they assumed were live names. Cascade 2 hits the same rollback mechanism, just less often (it writes more self-contained cells with explicit imports at the top of each block).We also re-rendered the per-problem-breakdown chart on the same LB-sim data after discovering that the original chart silently bucketed Cascade 2's bare-text timeout responses as \"successful tool call\" (the two solver recorders formatted timeout strings differently).
", "votes": 0, "replies": [] } ] }, { "id": 3447586, "author_name": "ivan", "author_url": "", "post_date": "2026-04-23T13:12:42.480000", "content": "Fantastic engineering work! I have a question about speculative decoding. You show very good \"Accept rate\", but how it changes during long trajectories? You fine-tuned it with 65k context, but does it work well with long context at test time?
", "votes": 1, "replies": [ { "id": 3447798, "author_name": "Chan Kha Vu", "author_url": "", "post_date": "2026-04-24T01:00:39.007000", "content": "Thanks for your kind words π You've got a good point!
\nAlthough we did fine-tune with maximum of 65K context, in reality there's not many TIR tasks that have more than 40K generated tokens (we did our best to filter out the hardest ones). TIR reasoning is much shorter than COT (which we didn't sampled enough for training). Validation during training was done on a hard subset of IMO-AnswerBench (we filtered all validation benchmarks like AIME and IMO-benches out from training), but they don't have that much rollouts with >65K tokens either.
\nHowever, the 3.55 acceptance rate cited above was on AIME'26, which doesn't need more than 65K tokens. It was my primary \"unit test\" for Eagle3 work, because AIME is simple enough to run in ~10 minutes, but just hard enough to reveal any bugs in Mamba states handling (my criteria was 98% accuracy - if I see a drop of 10%, I know exactly that somewhere a Mamba state got corrupted), and acceptance rate numbers were collected together with accuracy unit testing. We didn't have time to properly test it on enough 65K+ samples. Will come back with some measurements and update the writeup (I'll continue working on the PC+SD anyways).
\nOne thing I want to emphasize: Speculative Decoding is not worth it for small models. I know that people smarter than me already knows this, but the economy of 3B active models is so brutal. 1 token step is just 3.4 ms: 2.8ms Cuda Graph + 0.6ms overhead from CPU/sampling. Naive Eagle3 implementation is 13ms - you need acceptance rate of 4 to break even. We brought that down to 8.9ms - but that still requires 2.6 acceptance rate to break even. Not to mention other overheads that makes it closer to 3.0 acceptance rate to break even. Models like Cascade2 are heartbreakingly ridiculously fast!
", "votes": 2, "replies": [] } ] }, { "id": 3447327, "author_name": "Chan Kha Vu", "author_url": "", "post_date": "2026-04-23T00:08:32.747000", "content": "This is @yeoyunsianggeremie watching me getting stuck at some random side-quests \n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2270821%2F7f5e4044400f6c2e3207611fc92cbad5%2Ffrieren-beyond-journey-s-end-15-frieren-fern-mimic.avif?generation=1776902886558630&alt=media\")
Changes to address rebuttal questions:
\nShort list of edits made to writeup.md (and supporting artifacts) in this rebuttal pass.
kaggle_per_problem_stats.png from scratch. The original right panel used an asymmetric is_error rule: the gpt-oss-120b recorder prefixed its 30 s timeouts with [ERROR], our Cascade 2 recorder left them bare, and the same rule caught one and missed the other. Cascade 2's tool_err went from 0.2% β 6.3% of wall-clock; gpt-oss-120b's stayed at 14.4%. The corrected chart also splits timeouts from Python errors into two visually distinct buckets (purple/red), and the panel-3 subtitle now states the 30 s sandbox timeout cap that applies to both solvers. Full reconciliation in rebuttal_n4.md.test_time_scaling_attempts.png as a visual companion that drops the per-rollout cost scaling. One sentence of framing, no separate findings.Summary of our main candidates lead-in tightened from \"local validation and public LB\" β \"local validation and the Kaggle leaderboard\" (now covers Public + Private columns).For both @yeoyunsianggeremie and me, this is our third time participating in AIMO. Last year, we went all out and put everything into SFT + RLHF (Kaggle post: \"Soβ¦ We cooked a model π€·\", detailed blog post: \"Improving DeepSeek R1 in Math\"). This year, with Geremie grinding 75+ hours per week at his startup and me in the middle of a release, we had almost no time to compete like we did last year. Luckily, we had companions helping us: Geremie's bogoconic1/Qgentic-AI and my Claude Code Max subscription!
\nDespite joining the competition only in the last couple of weeks, we got delightfully stuck in a few side quests, and ended up creating a few useful artifacts that we think are worth sharing.
\nWe'd like to express our gratitude to AIMO3 organizers and XTX market for hosting such a fun competition and providing H100 GPUs. Special thanks to Simon Frieder (@friederrr) for actively engaging with the community, making it one of the most pleasant competitions I've ever participated in.
\nOur little fun time with AIMO3 resulted in following open-source side quests, useful beyond this competition:
\nWe also uncovered cheating attempts by Cascade 2 on Math problems (searching the web for answers) and reported to model authors here: Cascade-2 cheating attempts on Math is⦠cute.
\n\n\nKaggle notebook: chankhavu/c2-distill-vllm018-watchdog\n Model: chankhavu/c2-legaldistill-30b-a3b-fp8
\n
Our solution uses Nemotron-Cascade-2-30B-A3B, which we soft-cloned via layer-wise linear distillation to make it legal for this competition (Cascade 2 was released after the March 15 cutoff β see the Introduction below), then selectively quantized to FP8 with no quality degradation.
\nThe final harness is intentionally simple: 8 parallel rollouts per problem with Python tool use, entropy-weighted majority voting. We also tried a more sophisticated variant that filled the GPU idle bubbles between early- and late-finishing rollouts with pairwise LLM-as-judge self-verification, but didn't have time to tune it properly to beat plain majority voting. We also put a great deal of work to make speculative decoding work with prefix caching for hybrid-mamba models, but it didn't get into our final submission due to lack of submission budget and time to test things out properly. Summary of our main candidates on local validation and the Kaggle leaderboard:
\n| Submission (model, harness) | \nLocal validation (3 shuffles) | \nLocal validation (mean) | \nPublic LB attempts | \nPrivate LB | \n
|---|---|---|---|---|
| gpt-oss-120b, 44/50 public notebook | \n33, 30, 36 | \n33.0 | \n38, 38, 37 | \nβ | \n
| Cascade 2, 8 rollouts, self-verification (untuned) | \n33, 31 | \n32.0 | \n34 | \nβ | \n
| Cascade 2, 8 rollouts, voting (our final submission) | \n36, 34, 34 | \n34.6 | \n36, 39, 38 | \n41 | \n
Our local validation imitates the Kaggle leaderboard setting β 50 hard IMO-level problems drawn from IMO-AnswerBench (an open benchmark of IMO-style problems with short-integer answers), 5 hours on 1Γ H100 SXM. We shuffle the 50 problems 3 times with fixed seeds and run each solver through each shuffle, which gives us a controlled way to compare candidates without burning LB submissions.
\nThe competition. AIMO3 (the AI Mathematical Olympiad Prize 3, hosted by Kaggle and XTX Markets) evaluates math-problem solvers on 50 held-out IMO-level problems with short-integer answers. Each submission is a notebook that runs on a single NVIDIA H100 SXM (80 GB) with a 5-hour wall-clock budget. Models published after March 15, 2026 are not allowed to be used directly at runtime, which is the rule that shapes most of the model choices below. The public leaderboard is scored on a visible subset; the final standings use a disjoint private set revealed at the end.
\nThe main model candidate everyone converged on this year is gpt-oss-120b β OpenAI's 120 B-parameter open-weights reasoning model with native mxfp4 weights and a Harmony chat format. It's the model behind the strongest public-notebook baseline (the \"44/50\" notebook on the AIMO3 discussion board), and our main point of comparison throughout.
\nIn March 19, @chankhavu's birthday, Nemotron-Cascade-2-30B-A3B (shortened to Cascade 2 in this post) was released β NVIDIA's 30 B hybrid Mamba-MoE reasoning model. Published just after the March 15 cutoff, so not directly usable as-is, but the architecture is identical to the pre-cutoff Nemotron-3-Nano-30B-A3B. This is what made the \"soft-cloning\" approach legal, and it's the model inside our final submission. The hybrid-Mamba architecture is what forced us down a bunch of the side quests below β quantizing it, getting prefix caching to work with speculative decoding, training an Eagle3 speculator, and so on.
\nOther participant's findings. With only a few weeks to compete, we relied heavily on the open-source ecosystem and the community's collective findings. Some highlights:
\nThe competition rules prohibits using models published after March 15 directly during runtime. However, there's no limit on distillation or how distillation should be made. We rely on the following FAQ clarification from the competition organizers:
\n\n\nQuestion: Say I initialize an entirely new model with same architecture as the post - cutoff teacher model, but then train it by layerwise MSE to the teacher on the residual stream; in the limit this could approximate the teacher model weights, I just didn't explicitly say that I used the weights themselves.
\nAnswer: Indeed, while conceptually, these two cases can blur, practically they don't and that is why the AMLT rules makes sense: In practice, with this approach you'd recreate in a way the model that was off limits.
\n
We further clarified with the organizers that this kind of \"soft cloning\" is legal for the competition (here and here).
\nAfter confirming the legality, we performed the following layer-wise MSE distillation from Nemotron-Cascade-2. We initialized a randomly initialized student model with the same architecture and performed the following procedure:
\ni of the teacher Nemotron-Cascade-2 model, where X is the input to that layer and W_i the weight tensor.i. The student model happens to have the same architecture, so we can directly map each F^Teacher}_i to a student F^Student_i.i to minimize the following MSE loss: $$L_i = \\mathbb{E}_{X \\sim D} \\left[ | F^\\text{Teacher}_i(X) - F^\\text{Student}_i(X) |^2_2 \\right]$$ where D is a dataset of randomly sampled inputs. We optimize either by gradient descent or by closed-form MSE optimization techniques. Each student treats the teacher as a black-box function that produces outputs for given inputs.We never optimize towards the teacher weights directly; we only observe the teacher outputs and use them as MSE targets for the student. However, since the sub-models are linear, convergence is mathematically absolute. We published the implementation here. The BF16 model can be found here: chankhavu/bogoconic1-c2-softcpy-v3.
\nImportant clarification: for Kaggle submission, we quantized the cloned Cascade 2 model separately from the cloned BF16 model! We did NOT use our published open-source models.
\n\n\nOpen-source artifacts (totaling 66K downloads on Huggingface as of April 21, 2026):
\n\n
\n- π€ Huggingface model: chankhavu/Nemotron-Cascade-2-30B-A3B-NVFP4
\n- π€ Huggingface model: chankhavu/Nemotron-Cascade-2-30B-A3B-FP8
\n- π€ Huggingface model: chankhavu/Nemotron-Cascade-2-30B-A3B-AWQ-INT4
\nQuantization code: chankhavu/quantize-cascade-2; calibration data: chankhavu/cascade-2-calibration-data.
\n
We followed the Nano-architecture selective quantization recipe from the Nemotron 3 Nano Technical Report (Section 4). This is the same recipe as NVIDIA's official FP8 checkpoint of Nemotron-3-Nano-30B-A3B. Sensitive components are kept in higher precision:
\n
For calibration, we constructed a dataset of 4000 samples from Nemotron-Cascade-2-SFT-Data with the following domain mix, specifically designed for long-context reasoning: math (1000), swe (900), terminal_agent (500), science (500), chat (400), conversational_agent (300), instruction_following (300), safety (100). We trained on sequence length of up to 12,288 tokens (no padding, natural length per sample).
\nWe validated the correctness of our quantized models using NVIDIA-NeMo/Evaluator with config from Nemotron-3-Super-120B's eval config. We see no degradation in reasoning quality of quantized models. More details in Appendix B.
\n\n\nOpen source artifact: hav4ik/vllm branch pc-spec-v2-nonpc-slots
\n
Out of the box, neither vLLM nor SGLang can run Prefix Caching (PC) and Speculative Decoding (SD) together on Hybrid-Mamba models. Mamba's SSM layer keeps a fixed-size state vector updated in place β each token folds into it irreversibly. SD runs K candidate tokens through the model before the rejection sampler decides how many to keep; by then the SSM kernel has already overwritten the canonical state with the full K-token trajectory. Transformers escape this because their KV cache is append-only β rejected entries sit past the sequence end and touch nothing. Mamba has one state vector per layer per request and can't roll it back, so PC + SD needs an explicit mechanism to preserve the accepted state.
\n
The non-PC spec-decode path already does the right thing: the SSM and conv Triton kernels maintain K+1 per-request \"ghost\" slots with native init_token_idx rollback β those slots are just gated out whenever PC is active. We re-enable them and add a one-way boundary commit: only when a request's last accepted token lands exactly on a mamba-block boundary (last_accepted_pos % block_size == block_size - 1) do we write the ghost slot back to the PC pool. On every other step the pool is untouched, so the canonical cache state is never contaminated by rejected speculations. Ghost slots are indexed by batch position, so batch reordering moves them automatically β no extra bookkeeping.
One memorable debug moment: the first cut of init_spec_slots allocated each spec slot at the pool's page-padded stride (~50 MB/slot). At 32 seqs Γ K=5 Γ 23 layers that comes out to 52 GB β instant OOM. The fix was to allocate contiguously (~1.3 MB/slot); the kernels read state.stride(0) off the tensor itself, so ghost slots and pool slots can have completely different strides without breaking anything.
We used AIME-25 as a unit test throughout, with a pass threshold of β₯95% per-session accuracy β ironic, because AIME-25 is a notoriously hard competition that few high schoolers solve reliably, but at this point it's well within what a modern reasoning LLM with tool use can do consistently. A reliable gold standard for \"nothing silently corrupted\" is useful regardless.
\nOnce the correctness fix was in, Eagle3 was still a net loss at single-sequence concurrency: a full step took ~18 ms vs ~3.4 ms for no-Eagle, even though the target forward itself is only 4.2 ms. Profiling (H100, K=4, p=1) split the 18 ms budget roughly like this:
\n| Stage | \nTime (ms) | \nNotes | \n
|---|---|---|
| Target forward (K+1 tokens) | \n4.2 | \nactual model compute | \n
| Drafter, K=4 iterations | \n11.2 | \nserial: 4 Γ (~2 ms Python metadata + ~0.8 ms GPU forward) | \n
_build_attention_metadata | \n5.7 | \n7Γ more than no-Eagle β all 52 layers Γ K+1 | \n
_prepare_inputs | \n1.7 | \nK+1 tokens per request | \n
commit_boundary_states (23 mamba layers) | \n3.9 | \n.nonzero() D2H sync every step + .synchronize() wait | \n
| Rejection sampling + misc | \n1.1 | \n\n |
Two things stood out. First, the drafter loop is serially CPU-bound: each iteration needs the previously accepted token before it can build the next metadata object, so Python rebuilds can't overlap the 0.8 ms GPU forwards, and the GPU sits idle ~70% of the drafter phase. Second, commit_boundary_states β Mamba-specific infrastructure β was paying a full host-sync tax on every step, not just the rare steps where a commit actually fires.
The drafter issue needs a proper architectural fix (fuse all K iterations into one captured graph with persistent metadata buffers) that we left for upstream. What we did land is a sweep of surgical cuts targeting the other stages:
\ncommit_boundary_states used to call .nonzero() on a CUDA bool mask to find requests that had crossed a mamba-block boundary. .nonzero() forces a device β host sync every step. The new kernel grid is (num_layers, num_requests): each block checks its mask bit on-device and returns early if unset. No host ever sees the mask. β1.5 ms/step.synchronize(). The deferred-commit path was calling sample_complete_event.synchronize() β a blocking CPU wait β to guarantee num_accepted_tokens was visible before launching the commit kernel. Since both run on the same CUDA stream as the sampler, stream ordering already guarantees visibility. Removed. β1.0 ms/step.update_block_table + batched commit indices. Two related cleanups on the commit path that were paying per-layer overhead on pure-decode steps. β4.5 ms/step combined.attn_meta. On pure-decode steps, only state_indices_tensor_d differs between the 23 mamba attention builders, but _prepare_inputs was rebuilding the full metadata struct each time. Cache the first build; for the other 22, copy_ the per-layer pool mapping into the persistent buffer and dataclasses.replace just that field. β0.3 ms/step.End state at single-sequence (K=4, T=1.0): step time 18 β 8.9 ms, throughput 171 β 298 tok/s vs 303 tok/s for PC-only β essentially at parity.
\n
At higher concurrency, where the serial drafter overhead is amortized across the batch, Eagle3 becomes a real win:
\n| Concurrency | \nPC-only | \nEagle3 K=4, T=0.7 | \nvs PC-only | \n
|---|---|---|---|
| p=1 | \n303 tok/s | \n349 tok/s | \n+15% | \n
| p=8 | \n1186 tok/s | \n1552 tok/s | \n+31% | \n
| p=12 | \n1383 tok/s | \n1843 tok/s | \n+33% | \n
On an experimental branch we captured the K-iteration drafter decode loop as a single FULL cudagraph, replayed in 0.16 ms. Draft-phase time dropped from 3.16 ms to 1.39 ms (β56%) with acceptance rates preserved. But overall step time at p=8 was unchanged (13.7 ms) β the CPU savings got absorbed by GPU pipeline stalls in the prep phase, since the drafter's GPU work that previously overlapped the PIECEWISE Python loop now blocked the next iteration. The theoretical payoff is at p=12+ where CPU dispatch becomes the real bottleneck, but we didn't validate that before the deadline.
\nWe had planned to upstream the correctness fix and follow with the perf path, but two things have changed since the competition closed and the PR is currently blocked:
\nThe fork itself remains useful for anyone running Hybrid-Mamba SD inference on Hopper; we just can't claim the upstream PR as a clean deliverable any more.
\n\n\nOpen-source artifacts:
\n\n
\n- π€ Huggingface model: chankhavu/Nemotron-Cascade2-30B-A3B-Eagle3-Long-Context
\n- π€ Huggingface dataset: chankhavu/c2_eagle3_train (2 stage train data)
\n- π€ Huggingface dataset: chankhavu/c2-traces-10197 (collected reasoning traces on 10'197 hard math problems)
\nCode: SpecForge-Nemotron3.
\n
To our knowledge this is the first Eagle3 draft head trained against a hybrid Mamba-Transformer verifier that's verified end-to-end with vLLM. We forked SpecForge as SpecForge-Nemotron3 and added the engineering needed to fit long-context training (L=65536, ttt=6) on 2Γ RTX 6000 Pro Blackwell (96 GB each).
Two stages: a broad SFT bootstrap on ~20K conversations (378M untruncated tokens, from Cascade 2's supervised-finetune distribution) followed by on-policy fine-tuning on ~9.7K Cascade 2 reasoning traces (207M tokens) β ~292M loss-masked tokens total. We prune the 131K target vocab to a 32K draft vocab via top-K token frequency on the loss-masked positions, which keeps 99.77% coverage with ~4Γ smaller embedding / lm_head. Stage 1 uses LR 1e-4 with length-bucketed sampling (eliminates straggler-rank waste at batch_size=1); stage 2 drops to LR 5e-5 and disables bucketing for gradient diversity.
Two classes of fixes were required to get this training loop working at long context.
\nCorrectness β FP32 SSM scan state. Upstream mamba_ssm 2.3.1 silently downcasts the SSM scan boundary state to BF16 inside its Triton kernel. We monkey-patch it to force FP32, bit-equivalent to vLLM 0.19's NemotronH default. Without this, training converges to a worse minimum that costs ~10% absolute on AIME-class math.
Memory footprint β fitting L=65k on 96 GB. NaΓ―ve training peaks north of 120 GiB (OOM on 96 GB). Four independent activation-memory cuts on the draft forward + TTT unroll (chunked fused linear + soft-target CE for the lm_head, freeing the verifier's 131k-vocab logits before TTT unrolling, Liger-style chunked MLP, sliding-window attention on the draft) bring peak to ~88β91 GiB, with the single biggest win being the verifier-logit release. Step time at L=65k ttt=6 is ~4 s/step (high variance across sequence lengths). Separately, a 100Γ speedup in the loss-mask parser that was otherwise re-tokenizing each conversation num_turns times. Full details in Appendix C.
Temperature sweep, mean accepted tokens per step (on AIME-25):
\n| K | \nT=0.0 | \nT=0.7 | \nT=0.8 | \nT=0.9 | \nT=1.0 | \n
|---|---|---|---|---|---|
| K=4 (out of K+1=5) | \n3.33 | \n3.00 | \n3.05 | \n2.88 | \n2.91 | \n
| K=7 (out of K+1=8) | \n3.55 | \n3.26 | \n3.14 | \n3.03 | \n2.90 | \n
Lower temperature β draft head is more confident and its proposals track the target's argmax more closely, so acceptance climbs. You can find more information in Appendix C.
\nThe sandbox we gave Cascade 2 for tool use was a stateful_python_code_exec Jupyter kernel with network egress not blocked β urllib, requests, and pip install all worked. On 10,197 rollouts against IMO-class problems, Cascade 2 issued real HTTP calls in 246 traces (~2.4%).
Top destinations (HTTP URL literals in tool-call code, per-trace dedup), by category:
\nartofproblemsolving.com (64), en.wikipedia.org (58), oeis.org (28), mathworld.wolfram.com (6)google.com (40), duckduckgo.com (39), bing.com (3)math.stackexchange.com (13), api.stackexchange.com (12), stackoverflow.com (3)imo-official.org (11), purplecomet.org (5)raw.githubusercontent.com (8), arxiv.org (3), api.github.com (3)The dominant pattern is AoPS wiki lookups by year/contest/number and Wikipedia for theorem statements. OEIS (28 traces) is usually queried with a computed-prefix sequence. DuckDuckGo Instant Answer API and google.com/search GETs with the prompt pasted verbatim show up often enough to be a recognizable tactic.
A few representative (amusing) traces:
\nurllib.request.urlopen against imo-official.org, written to disk (byte-exact match to the real PDF), PyPDF2.PdfReader over every page, substring-grep for prompt keywords. Extraction worked; the problem wasn't in the document. Answer happened to be 0 β coincidental match.requests.get(...), saved response, tried PyPDF2 / pdftotext / fitz / pdfminer β all four failed with EOF marker not found / No /Root object. First 100 bytes of the \"PDF\" were <!DOCTYPE html>; the server returned an HTML error page with a .pdf extension. Wrong answer.IMO2021SL.pdf and IMO2022SL.pdf into memory via response.read(), never persisted, grep_bytes(b'Mario'). Problem was IMO 2022 SL C2 (\"Mario and Bowser\"). Wrong answer.!wget with a typo (trailing s), DNS fails.Network-using traces end up at 34.6% correct β these are the IMO-level problems the model couldn't crack with direct reasoning in the first place, and no venue in the breakdown meaningfully changes the picture (wikipedia 36.6%, AoPS 27.9%, OEIS 27.0%, Google search 42.9% at the top). That said, the model recognizing a problem as a benchmark item and reaching for the right contest archive is a contamination signal on its own, independent of whether the lookup paid off.
\nWe reported this to the Nemotron team (HF discussion) and released the verified traces: π€ chankhavu/nemotron-cascade2-cheating-attempts (173 rows). For our own runs since, we block egress at the sandbox namespace level.
\nWith the open-source artifacts above in hand β the soft-cloned FP8 Cascade 2, the vLLM PC+SD fix, the Eagle3 speculator β we now turn to the competition itself: how these pieces line up against gpt-oss-120b on the benchmarks, what we actually submitted, and which harnesses we tried but didn't ship.
\n\n\nWe publish the dataset construction notebook here: https://www.kaggle.com/code/chankhavu/local-validation-benchmark.
\nAll throughput analysis in this section are based on the model alone, without speculative decoding.
\n
We construct a candidate pool of 63 problems by combining the 10 official AIMO3 reference problems with 53 hard problems from IMO-AnswerBench β specifically those where gpt-oss-120b scored 6/8 or less on avg@8, restricted to answers in the Kaggle 0β99999 integer range. The pool draws from IMO Shortlists, USAMO, HMMT, Sharygin, RMM, and national TSTs (Vietnam, China, Korea, Japan). We then take a 50-problem subset to match the Kaggle Leaderboard setting used for local validation above.
\nFor the gpt-oss-120b tool-use accuracy stats, we use the answerbench-gpt-oss dataset by @yiyangzheng β they were among the top contenders on the LB, so their inference pipeline is trustworthy.
\n\n\nNote on contamination. IMO-AnswerBench is itself drawn from previously published IMO and IMO-Shortlist problems (and other national olympiads), rewritten so that the answer is an integer in a fixed range. Both gpt-oss-120b and Cascade 2 were trained on data that almost certainly includes some of these problems in their original form, so contamination is a real concern. Cascade 2 in particular was released after IMO-AnswerBench was, which expands the contamination window. We did not test for contamination on our side; we chose to trust the data-curation procedure described in the Nemotron-Cascade 2 paper, which uses n-gram-based deduplication against known evaluation benchmarks. We flag this so the reader can weight the absolute accuracy numbers accordingly; the relative comparisons between solvers are less affected because both candidates draw from the same potentially-contaminated training pool.
\n
\n\nWe published all reasoning traces collected for offline benchmark: chankhavu/benchmark-traces
\n
All three candidates run through the same harness on the 63-problem evaluation set (8 rollouts per problem, tool use enabled, same prompt template): gpt-oss-120b, our FP8-quantized Cascade 2, and the Nemotron-3-Nano-30B-A3B.
\nMetric definitions used throughout this section: Accuracy is avg@1 (probability a single rollout is correct, averaged over all 504 rollouts). Majority vote is maj@8 (the most-common of 8 parallel rollouts matches the expected answer). Pass@8 is the probability that at least one of 8 rollouts is correct (the upper bound an oracle verifier could reach).
\n| Model | \nAccuracy | \nMajority vote | \nPass@8 | \nNo-answer | \nAvg gen tok | \nAvg tools/sess | \n
|---|---|---|---|---|---|---|
| gpt-oss-120b | \n44.0% (222/504) | \n52.4% (33/63) | \n77.8% (49/63) | \n8.9% (45/504) | \n26k | \n20.7 (7% err) | \n
| nano30b | \n41.5% (209/504) | \n52.4% (33/63) | \n63.5% (40/63) | \n1.6% (8/504) | \n47k | \n14.2 (6% err) | \n
| cascade2 | \n47.0% (237/504) | \n55.6% (35/63) | \n74.6% (47/63) | \n2.2% (11/504) | \n34k | \n11.5 (7% err) | \n
Cascade 2 leads on both accuracy and voting with half the tool calls β more sample-efficient when it does reach for Python. gpt-oss-120b's high no-answer rate comes from hard problems where it runs out of context mid-reasoning.
\n| \n | 1 sequence | \n4 sequences | \n8 sequences | \n12 sequences | \n
|---|---|---|---|---|
| gpt-oss-120b, mxfp4, kv8 | \n175.4 tok/s | \n506.4 tok/s | \n810.9 tok/s | \n842.8 tok/s | \n
| Cascade 2, fp8, kv8 | \n301.6 tok/s | \n859.0 tok/s | \n1333.5 tok/s | \n1487.8 tok/s | \n
We use the FP8 Cascade 2 variant because H100 supports FP8 natively. Cascade 2 is faster at every batch size, and the gap widens at higher concurrency: gpt-oss-120b's throughput plateaus by 8 sequences, while Cascade 2 keeps scaling.
\nTwo views of how each model scales with more compute per problem. pass@k = prob. β₯1 of k rollouts is correct; maj@k = probability that the most-common answer across k rollouts is correct. Methodology (x-axis conversion, truncation rule, tokenizer choice) in Appendix D.
\nTime budget. Each dot is k β {1, β¦, 8}; the x-axis converts k into wall-clock seconds per problem on 1Γ H100 SXM at 8 parallel sequences. Each model's dots are spaced differently because their per-rollout cost differs: Cascade 2 is the cheapest per rollout, so its curve packs the most to the left.
\n
Within the tested budget, Cascade 2 wins both pass@k and maj@k β it's the more consistent choice for the actual competition setting. It leads on maj@k across the whole range, and it wins pass@k at every plotted budget because it's cheaper per rollout and gets to fit more rollouts in for any given wall-clock. gpt-oss-120b's pass@k curve, however, has a higher upper ceiling β its rollouts are more diverse (helps pass@k, hurts plurality voting), so given more attempts / a larger budget than this chart covers, an oracle verifier picking the right rollout would eventually prefer gpt-oss-120b. That higher pass@k ceiling is what makes gpt-oss-120b interesting with an external verifier (see Self-verification experiments below); within the time budget of this competition, Cascade 2 dominates on both axes.
\nA practical note on that pass@k advantage: realizing it would require an external verifier picking from gpt-oss-120b's rollouts, and Kaggle's resource constraints (5 h on 1Γ H100 SXM) make that infeasible in either obvious form β we cannot fit a separate verifier model alongside gpt-oss-120b's ~120 GB MXFP4 weights on a single H100, and using gpt-oss-120b as its own verifier doubles the per-problem token bill on a budget that already only fits ~8 solution rollouts per problem. Cascade 2's smaller footprint and faster throughput leave headroom for the self-verification experiments below; gpt-oss-120b does not.
\nReasoning-time budget. Left panel plots avg@1, right panel maj@8, both as a function of how long a single rollout is allowed to reason (wall-clock seconds on 1Γ H100 SXM at 8 parallel sequences). Each dot is the result of capping the stored rollouts at a different token length, retaining the last \\boxed{} within the cap as that rollout's answer. This is the \"one longer chain of thought\" axis, complementing the \"more parallel rollouts\" time-budget chart above.
Cascade 2 dominates on both panels. It reaches its ceiling (47.0% avg@1 / 57.1% maj@8) at ~207 s per rollout, while gpt-oss-120b tops out at ~313 s with a lower ceiling (44.0% / 52.9%) β gpt-oss reasons compactly in tokens, but its slower throughput costs it ~50% more wall-clock time. nano30b traces Cascade 2's shape shifted right by ~70 s: same throughput, longer traces β exactly the gap soft-cloning was meant to close.
\nHow the x-axis is built. For each truncation L β {16k, 32k, 48k, 64k, 80k, 96k} we cut each rollout at L tokens, measure the effective reasoning length (the \\boxed{} position, or L if the model didn't stop), and convert tokens β seconds using the model's per-sequence throughput at 8 parallel sequences on 1Γ H100 SXM (101 tok/s/seq for gpt-oss-120b; 167 tok/s/seq for Cascade 2 and nano30b, which share a tokenizer and architecture).
\n\nKaggle notebook: chankhavu/c2-distill-vllm018-watchdog\n Model: chankhavu/c2-legaldistill-30b-a3b-fp8
\n
The solver we actually submitted is compact: no self-verification, no external tools beyond Python, no learned verifier. Five mechanisms wrapped around the FP8 soft-clone:
\nbudget = min(900, max(300, time_left β 300 Γ problems_left_others)). Early problems get the full 900 s; as the wall clock runs down we reserve 300 s per remaining problem and squeeze the current one to the 300 s floor.stateful_python_code_exec calls in a stateful IPython sandbox (30 s per call, 1000-char output cap, math/numpy/sympy/itertools/collections/mpmath preimported with mpmath.mp.dps = 64), append results as tool messages, continue. The rollout returns the integer inside the last \\boxed{β¦} when the model stops calling tools.1 / mean_token_entropy, where the per-token entropy is computed from top-5 logprobs over the generation. The winning answer is the one with the largest sum of rollout weights.Unfortunately, we only finished our Eagle3 work on the last 2 days of the competition, so we never had a clean submission with only Eagle3, without other harnesses that degrades the results (like self-verification or increased number of parallel rollouts).
\n\n\nThe analysis in this section is based on collected traces from simulated online LB runs (50 problems per run, 5h per run, 1x H100 SXM). The data is published here: chankhavu/aimo50-evals.
\n
To sanity-check how the offline numbers translate into the actual Kaggle setup, we replayed our final Cascade 2 voting submission and the gpt-oss-120b 44/50 public-notebook baseline on three independent shuffles of the same 50 problems, each under the real Kaggle budget: 5 hours wall-clock on 1Γ H100 SXM with an adaptive per-problem budget that starts at 900 s and can be squeezed down to 300 s as the wall clock runs low.
\nThese 50 problems are a subset of the 63 problems in our offline benchmark section. We removed problems with pass@8 = 0 for all 3 models: gpt-oss-120b, Nemotron Cascade 2, and Nemotron 3 Nano (so, basically impossible removed problems that were never solved).
Two readings. (i) Cascade 2 (mean 34.7/50) is ahead of gpt-oss-120b (mean 33.0/50) on the LB scoreboard from the ~2h mark onward, and reaches gpt-oss-120b's final score about 20 min earlier in the average run. (ii) gpt-oss-120b's per-problem budget gets visibly squeezed toward the end: on aimo50c the last 5 problems ran at 300 s (the minimum), because the notebook was within minutes of the 4.83h ceiling. Cascade 2 held 900 s per problem for most of the run across all three shuffles, with plenty of headroom at the finish.
\nThree per-problem views of the 150 (problem Γ shuffle) pairs:
\n
Left β vote convergence predicts correctness. When all 8 rollouts agree (bin 1), Cascade 2 is correct 82.8% of the time vs gpt-oss-120b's 74.6% β the online equivalent of the offline \"consistency β correctness\" story. Cascade 2 never produces a zero-answer case; gpt-oss-120b has 9 of 150 (all wrong). At 4+ distinct answers (both models stumped), Cascade 2 is slightly worse but has more such cases (30 vs 14), so the bin averages understate its overall advantage.
\nMiddle β which model actually matters on 29% of pairs. 53% both right, 18% both wrong, 16% Cascade-2-only right, 13% gpt-oss-only right. Net +5 pair-wise advantage to Cascade 2. Roughly half the LB score is \"easy for everyone\" and one-fifth is \"hard for everyone\"; the remaining third is where model choice actually moves the needle.
\nRight β where the wall-clock time actually goes. Decomposing total wall time across all 3 shuffles into four buckets β generation, successful tool call, Python-error tool call, timed-out tool call β Cascade 2 spends 88% in generation vs gpt-oss-120b's 74%, and 6.3% of Cascade 2's time vs 14.4% of gpt-oss-120b's is wasted on tool calls that produced no useful output (~2.2Γ more for gpt-oss-120b). For both solvers, the wasted time is almost entirely pure 30 s timeouts (12.5% / 5.9% of wall-clock) rather than Python errors (1.9% / 0.4%) β gpt-oss-120b's tool calls just trip the 30 s sandbox cap roughly 2Γ more often per rollout than Cascade 2's. Appendix G dives into the per-error-class breakdown and the most common root causes (forgotten imports, sandbox state rollback after a cell error).
\nCascade 2 voting vs the gpt-oss-120b baseline, log-derived:
\n| Submission (model, harness) | \nLocal validation (3 shuffles) | \nLocal val (mean) | \nTool err (per-call) | \nPublic LB attempts | \nPrivate LB | \n
|---|---|---|---|---|---|
| gpt-oss-120b, 44/50 public notebook | \n33, 30, 36 | \n33.0 | \n15.0% | \n38, 38, 37 | \nβ | \n
| Cascade 2, 8 rollouts, voting (our final submission) | \n36, 34, 34 | \n34.7 | \n10.6% | \n36, 39, 38 | \n41 | \n
Per-call tool error rates count any call that produced no useful output β Python errors and 30 s sandbox timeouts together.
\nThe fact that the results are in similar ballpark as Public LB (30 - 36) is pure coincidence - we did not cherry-pick the problems for our benchmark in any way. It means that the Public LB are really on the same difficulty level as the IMO-AnswerBench, which is a testament of how far we've come since AIMO1!
\nCascade 2 wins local validation by 1.7 points and matches gpt-oss-120b on the LB mean (both 37.7), peaking higher (39 vs 38). Both finalists score meaningfully higher on the LB than on our local mean (gpt-oss +4.7, Cascade 2 voting +3.0), which suggests IMO-AnswerBench (our local set) is slightly harder than the hidden LB set. The self-verification harnesses we tried β folded into the full comparison table at the end of the Self-verification experiments section below β both underperformed plain voting.
\n\n\n
\n- Notebook we submitted to Public LB: https://www.kaggle.com/code/chankhavu/c2-distill-verifier?scriptVersionId=311859342
\n- Caveat up front. We built and evaluated the entire self-verification harness within the last 24 hours before the competition deadline β no time to tune any of the hyperparameters (pair-selection policy, grader prompt, max-verifications cap, temperature, rating scale, ΞΌ-score weighting, fallback thresholds). Treat the numbers below as first-attempt measurements, not a carefully-optimised upper bound on what self-verification can do.
\n
The voting harness leaves an obvious source of slack: on most problems the 8 solution rollouts finish at wildly different times. The first rollout drops its \\boxed{} at 100β200 s, the slowest can take 500+ s, and in between we're sitting on GPU capacity we've already paid for. The self-verification variants from the table above use that capacity to run pairwise grader sessions that rate competing answers against each other.
\n
We borrow the core idea β pairwise LLM-as-judge comparisons aggregated into an uncertainty-weighted score β from V1: Toward Verifiable Reasoning in LLMs by Harman et al. Our pair-selection policy differs: V1 picks the next pair adaptively to maximise information gain, while we use a simpler greedy rule (newest answer vs current undefeated bucket). Our winner-selection scoring stays close to theirs.
\nOn top of the voting harness (same model, same sampling, same adaptive budget, same early-stop at four agreeing rollouts), the self-verification variants add three conceptual pieces:
\nEach time a solution rollout completes, the harness attempts to launch up to two verifications in whichever GPU slots are free, subject to a cap on concurrent rollouts. The tracker picks the next comparison in priority order:
\nCompleted verifications also re-trigger a new comparison if there's still wall-clock budget and solution rollouts are still arriving. Verifications are capped at a fixed maximum per problem so they can't swallow the solver's budget on pathological cases.
\nLet undefeated be the set of buckets with zero losses in the win/loss graph. If exactly one bucket is undefeated, we pick it β done. If multiple remain undefeated (or none, in the cycle case), we rank them by an uncertainty-weighted score
\n$$\\mu_i = \\frac{\\sum_j w_{ij} \\cdot v_{ij}}{\\sum_j w_{ij}}\\,,$$
\nwhere $v_{ij} \\in {1, \\tfrac{1}{2}, 0}$ is the outcome (win / tie / loss) of bucket i against bucket j, and $w_{ij} = \\max(|r_i - r_j| / 9,\\, 0.1)$ scales each comparison by its rating margin: a 10β1 verdict carries an order of magnitude more weight than a 5β4 one, with a floor of 0.1 so ties still count. Further ties break by bucket size (number of rollouts that produced that answer), then by comparison count. If fewer than half the distinct answer buckets have been graded at all, we treat the grading as too sparse to trust and fall back to the entropy-weighted majority vote from the baseline voting solver.
\nOn the problem in the chart, voting has nothing useful to say: three rollouts say 1, three say 12, only two say 13 β the correct answer. Voting breaks the tie by entropy-weighted score, picks 12 (score 5.13 vs 1's 4.71 and 13's 3.55), and is wrong. The grader only needs three pairwise comparisons to settle it (13 beats 1 with 10β1, 13 beats 12 with 9β2, and 12 beats 1 with 3β1), leaving 13 as the only undefeated bucket. This is the regime self-verification is supposed to win: when a single well-reasoned rollout can recognise a majority of sloppy rollouts for what they are.
Despite the elegance, the table tells the disappointing story: self-verification underperformed voting on both local validation (32.0 / 33.0 vs 34.7) and public LB (34 vs 36/39/38). We tried two configurations β 8 solution rollouts with verification layered on top, and 6 solution rollouts with more verification budget per problem β and neither beat plain voting.
\nWhen does the grader even do something? Across all 297 problem-shuffles we ran under self-verification (three variants Γ two shuffles Γ ~50 problems), the grader agrees with entropy-weighted voting's top answer 84.5 % of the time (251 / 297) β meaning on 5 out of every 6 problems the verification layer adds literally nothing. On the remaining 15.5 % (46 cases) the grader flips the decision, and its hit rate on those flips is what really matters:
\n| Harness | \nFlips | \nGrader right (voting wrong) | \nGrader wrong (voting right) | \nBoth wrong | \nNet | \n
|---|---|---|---|---|---|
| Cascade 2, 8 rollouts, self-verification | \n11 | \n6 | \n3 | \n2 | \n+3 | \n
| Cascade 2, 6 rollouts, self-verification | \n11 | \n6 | \n2 | \n3 | \n+4 | \n
| Cascade 2 + Eagle3, 6 rollouts | \n24 | \n10 | \n10 | \n4 | \n0 | \n
| All self-verif (aggregate) | \n46 | \n22 | \n15 | \n9 | \n+7 / 297 | \n
So on the two non-Eagle3 runs the grader is net positive but only barely β +7 correct answers across 297 problem-shuffles, roughly 2.4 %. The Eagle3 run is the real tell: its 33 % throughput headroom lets the scheduler fire more than twice as many flip-candidate comparisons (24 vs 11), and the extra firings split right down the middle β ten wins cancelled by ten losses. More grading, no net gain. At the margins where voting and the grader disagree, the grader is essentially a coin flip.
\n
Our read: the grader is too lenient to break ties confidently when both answers are backed by plausible-looking reasoning, and β as the chart above makes clear β at 1Γ H100 SXM there simply isn't enough wall-clock budget to run the 15β20 comparisons per problem the tracker would need to converge on harder problems. The per-problem means are close across all three runs (329 s voting vs 342 s self-verification vs 339 s with Eagle3), but the tails tell a very different story: voting's 90th-percentile problem consumes 472 s, self-verification's reaches 589 s, and with Eagle3 it stretches to 796 s β the longer the grader can keep running comparisons, the longer individual problems drag on, and the scheduler simply squeezes the remaining budget harder. Eagle3 buys roughly a 33 % throughput improvement (our motivation for training it in the first place), but even on that budget the extra complexity of the verification layer didn't translate into more correct answers β the Eagle3 run landed at 33.5 / 50 (mean of 36 and 31), which is no better than the plain self-verification variant and still behind voting's 34.7. The ingredients β fast model, honest grader β were right; the budget wasn't.
\n| Submission (model, harness) | \nLocal validation (3 shuffles) | \nLocal val (mean) | \nTool err (per-call) | \nPublic LB attempts | \n
|---|---|---|---|---|
| gpt-oss-120b, 44/50 public notebook | \n33, 30, 36 | \n33.0 | \n15.0% | \n38, 38, 37 | \n
| Cascade 2, 8 rollouts, self-verification | \n33, 31 | \n32.0 | \n5.1% | \n34 | \n
| Cascade 2, 6 rollouts, self-verification | \n35, 31 | \n33.0 | \n5.2% | \nnever submitted | \n
| Cascade 2 + Eagle3, 6 rollouts, self-verification | \n36, 31 | \n33.5 | \n5.2% | \nnever submitted | \n
| Cascade 2, 8 rollouts, voting (our final submission) | \n36, 34, 34 | \n34.7 | \n6.0% | \n36, 39, 38 | \n
The boring harness wins. Our final submission is the simplest thing that works: 8 rollouts of soft-cloned FP8 Cascade 2 with Python tool use, an adaptive 300β900 s per-problem budget, early-stop at 4 agreeing answers, and an entropy-weighted majority vote. On the 3-shuffle local validation (IMO-AnswersBench, 5 h, 1Γ H100 SXM) it scores 34.7 / 50, ahead of the public gpt-oss-120b 44/50 notebook's 33.0 / 50, and it ties gpt-oss-120b on the public LB (mean 37.7, peak 39 / 50 vs 38). Every more-sophisticated Cascade 2 harness we tried β pairwise-comparison self-verification, the same plus Eagle3-accelerated inference β underperformed the baseline. We believe self-verification could beat voting with more tuning, but not in the 24 hours we had before the deadline.
\nWhat the side quests produced. Three open-source artifacts that outlast the competition: a family of quantized Cascade 2 checkpoints (FP8, NVFP4, AWQ-INT4 β 66K downloads and counting), a vLLM fork that correctly runs prefix caching alongside speculative decoding on hybrid-Mamba models (long-standing open problem), and an EAGLE-3 speculator for Cascade 2 reaching 3.55 mean accepted tokens at K=7 along with the SpecForge modifications needed to train it at 64 k context. We also filed a note on Cascade 2's tendency to quietly search the web for answers through its Python tool.
\nWhy this is mostly a first-try harness. We joined the competition late and the Kaggle 1-submission-per-day cap meant essentially every choice we shipped was a first-try call, with no headroom to A/B-test on the hidden LB. That's also why our scheduling, early-stop, and budget policy are deliberately simple: we picked Cascade 2 specifically because it offered the best maj@k under a time budget, and our priors from AIMO 2 were that winning solutions in this class of competition optimized maj@k rather than chasing pass@k at infinity β pass@β tends to be similar across competitive models, but maj@k under a real time budget is where the consistency premium pays off. A more elaborate per-problem scheduling system was on the wishlist (several contestants reported good gains from one), but the engineering dependency we did spend weeks on β PC + SD for hybrid-Mamba models in vLLM β was already over our time budget, and we judged that getting the simple harness right was a safer use of the remaining days than tuning a scheduler we couldn't validate without using up the submission slots.
\nWhat we'd do differently. Finalize PC + SD support in vLLM earlier (this was the engineering dependency that would have unlocked a genuinely competitive self-verification harness), start the self-verification work on day one instead of the last 24 hours, spend some time tuning the grader (prompt, rating scale, pair-selection policy), and develop a smarter per-problem scheduling policy that exploits the random hard/easy ordering. The ingredients were right; we just ran out of room to tune them. Even so, we're glad we got side-tracked β the open-source artifacts are the part of this project we're most proud of.
\nThe recovery loop runs per tensor, one tensor type at a time. For each teacher tensor we (1) generate random probe inputs $X \\sim \\mathcal{N}(0, 1)$, (2) observe the teacher's output $Y = f_\\text{teacher}(X)$ β this is the only step that touches teacher values β and (3) solve for student parameters that minimise the mean-squared error $\\lVert f_\\text{student}(X) - Y \\rVert^2$. Linear layers $y = W x$ are recovered in closed form via the normal equations (it's already an obvious MSE minimiser and the loop would add ~100Γ runtime for no benefit). The other three tensor types β depthwise Conv1d kernels, 1D scalar vectors ($y = w \\odot x$; norms, SSM $A_{\\log}$, $d_t$, $D$), and additive biases ($y = x + b$) β all run an explicit SGD loop with a learning rate that cosine-decays to 0 over 500 steps. Max learning rates are exposed as CLI flags; the choices below come from stability arguments on the per-parameter Hessian.
\nEach recovery loss is a convex quadratic, and SGD is stable exactly when $\\mathrm{lr} < 2/\\lambda_{\\max}(H)$ with $H$ the per-parameter Hessian. For the $y = w \\odot X$ and $y = X + b$ cases the loss is separable across features, so with the normalisation we use (mean over samples, sum over features) every parameter sees the same scalar Hessian $H = 2\\,\\mathbb{E}[X^2] = 2$ for $X \\sim \\mathcal{N}(0, 1)$ β hence $\\mathrm{lr} = 0.5 \\approx 1/H$ lands each element at its optimum in $O(1)$ effective steps, independent of $N$ and of the magnitude of the target.
\nThe unfolded-window conv1d Hessian is not a scaled identity. Its $(k_1, k_2)$ entry is the sample autocorrelation of standard noise at lag $|k_1 - k_2|$: diagonal $\\approx T$, off-diagonals have $\\mathcal{O}(\\sqrt{T})$ noise fluctuations (plus small deterministic shifts near the zero-padded boundary of the window). Divided by $T$ in the loss, the spectrum concentrates around $2$ but spreads to roughly $[1.3, 3.5]$ at typical Mamba sizes ($T = K + 32 = 36$, $K = 4$). $\\mathrm{lr} = 0.5$ is right on the stability edge for $C \\lesssim 10^3$ and clearly past it by $C \\gtrsim 4 \\cdot 10^3$ (it diverges with extreme numeric blow-up). $\\mathrm{lr} = 0.3$ sits inside $2 / 3.5 \\approx 0.57$ with comfortable margin across every Cascade 2 size we ran.
\nThe per-component precisions behind the layout chart in the main section. Each row maps an architectural component to the precision it's kept at in the FP8 and NVFP4 checkpoints, following the NVIDIA Nano sensitivity recipe.
\n| Component | \nPrecision (FP8) | \nPrecision (NVFP4) | \nComment | \n
|---|---|---|---|
| MoE expert GEMMs (routed + shared) | \nFP8 | \nNVFP4 | \nAll 23 MoE layers, 128 routed + 2 shared experts each | \n
| Mamba GEMMs (non-adjacent) | \nFP8 | \nNVFP4 | \n17 of 23 Mamba layers | \n
| Attention layers (all 6) | \nBF16 | \nBF16 | \nMost sensitive β kept BF16 per NVIDIA sensitivity analysis | \n
| Mamba layers adjacent to attention (6) | \nBF16 | \nBF16 | \nLayers {4, 11, 18, 25, 32, 41} β found sensitive in ablations | \n
| Mamba 1D conv | \nBF16 | \nBF16 | \nAll layers | \n
| Router gates | \nFP32 | \nFP32 | \nRouting precision must not degrade | \n
| Embeddings & lm_head | \nBF16 | \nBF16 | \nNot quantized | \n
| KV cache | \nFP8 | \nFP8 | \nAll 6 attention layers | \n
All benchmark runs use NVIDIA-NeMo/Evaluator with the Nemotron-3-Super-120B tool-use config: 8 rollouts per problem, temperature 1.0, tool use enabled (sandbox code execution), enable_thinking=true. Same config across the BF16 reference and our FP8 / NVFP4 checkpoints so numbers are directly comparable.
\n\nCritical serving flag. NemotronH's SSM state must stay in FP32 at inference. vLLM 0.19 reads this from the model's
\nconfig.jsonby default (Cascade 2 ships the override), but anyone serving a variant whose config doesn't include it should pass--mamba_ssm_cache_dtype float32explicitly. Leaving the SSM state at BF16 costs ~10% absolute on AIME-class benchmarks β silent precision regression, no crash. The same applies to SGLang (--mamba-ssm-dtype float32), which defaults to BF16.
| Benchmark | \nNemotron-Cascade-2-30B-A3B (reproduced results) | \nOur FP8 model | \nOur NVFP4 model | \n
|---|---|---|---|
| AIME 2025 (avg@8) | \n98.8 | \n96.7 | \n97.9 | \n
| AIME 2026 (avg@8) | \n94.2 | \n95.0 | \n92.1 | \n
| HMMT Feb 2025 (avg@8) | \n92.9 | \n93.8 | \n90.1 | \n
With the low sample count (8 rollouts per problem), a deviation of Β±2% across runs is expected. Safe to say BF16, FP8, and NVFP4 variants are equivalent in reasoning performance.
\nAt the sweet spot (K=4, T=0.7 on AIME-25):
\n| Position | \n1 | \n2 | \n3 | \n4 | \n5 | \n
|---|---|---|---|---|---|
| Accept rate | \n73.3% | \n49.8% | \n34.5% | \n24.7% | \n17.3% | \n
Rates decay geometrically with a ratio of ~0.7 per position β each additional draft token needs all previous drafts to verify first, and the marginal drafter confidence drops with depth.
\nNaΓ―ve training at L=65k, ttt=6 peaks at 120+ GiB β OOM on a 96 GB GPU. Four independent activation-memory cuts on the draft forward + TTT unroll bring peak down to ~88β91 GiB, approximately 63 GiB verifier weights + 4.2 GiB target_p_padded + 12 GiB attention activations + 4.2 GiB MLP saves + ~5β8 GiB overhead:
[seq, 131072] logits as target (~16 GiB at L=65k), held through three independent references across the draft forward + backward. Precomputing a small draft-vocab target_p_padded (~4 GiB) and dropping the rest before eagle3_model.forward() releases the full-vocab tensor before the expensive phase. Dropping the .float() upcast on the softmax saves another ~8 GiB.[B, T, V] logits tensor is never materialized β the existing LogSoftmaxLoss Triton kernel runs inside a chunked + grad-checkpointed wrapper. Numerically equivalent to the unchunked path on both hidden_states and lm_head_weight grads, verified at realistic shapes.LlamaForCausalLMEagle3, the flex_attention mask generator, and the causal-mask builder. The TTT suffix block is intentionally not windowed.The four savings don't sum linearly β some overlap (e.g., freeing target logits shrinks the memory available for future cuts to operate on). Aggregate per-step wall time is ~4 s (1.2 s on short samples up to 8.6 s on near-max-length conversations at L=65k); grad-checkpointed MLP costs ~8β12% per-step recompute overhead, which the memory savings more than pay for.
\nGeneralParser.parse() was running per-turn re-tokenization in O(num_turns Γ seq_len) β for long reasoning traces with many assistant turns this took several seconds per conversation. We replaced it with a single return_offsets_mapping + bisect pass to map character spans to token spans in O(num_turns Γ log seq_len). Per-conversation preprocessing dropped from several seconds to ~30 ms on long reasoning traces.
Attempt-count view. What the time-budget chart in the body would look like if we drop the per-rollout-cost scaling and plot against raw k. Same data, x-axis = k instead of k Γ seconds-per-rollout.
Time-budget chart. For each model m and k β {1, β¦, 8}, we plot the (pass@k, maj@k) point at x = k Γ avg_tokens_per_rollout(m) / tok-per-sec(m) seconds, using the 8-parallel-sequence throughput from the generation-speed table. pass@k and maj@k are computed exactly (no Monte-Carlo) by enumerating all C(8, k) subsets of the 8 available sessions per problem and averaging across the 63 problems. maj@k uses the most-common answer rule: for each subset, the modal answer wins, ties split uniformly across tied answers.
Reasoning-time budget chart. For each truncation length L β {16k, 32k, 48k, 64k, 80k, 96k}:
\n\\boxed{β¦} inside the truncated portion. If found, that token position is the session's effective length and its original predicted answer counts. If not found, the session is treated as \"no answer\" at that L and its effective length is L.Tokenization. Per-model, exact:
\nnum_completion_tokens from each trace record (the model's own Harmony token count for generated output, exclusive of prompt and tool results). Because gpt-oss emits \\boxed{β¦} as its final generation, we approximate its \\boxed{}-end token position as the total generated-token count.generation text with the Nemotron-Cascade-2 tokenizer (nvidia/Nemotron-Cascade-2-30B-A3B) and compute both the total and the \\boxed{}-end positions exactly by running the tokenizer on the full text and on the prefix up to the closing brace of the last \\boxed{β¦}.Each pairwise verification is its own chat session with Python tool use enabled. System message:
\nYou are a helpful assistant and an expert math contest grader.\nUser message (fields {problem}, {sol_A}, {sol_B} substituted from the current problem and two competing solution texts):
Compare two submissions and rate correctness based solely on the final answer.\n\n**Evaluation Guidelines:**\n- Extract each submission's final answer. Use any provided reasoning only\n to help you assess whether the stated final answer is trustworthy. Do\n not award credit for method quality or rigor.\n- Carefully analyze the problem statement and the submissions to assess\n whether each final answer is correct. Grade only the final answer.\n**Problem**\n{problem}\n**Solution A**\n{sol_A}\n**Solution B**\n{sol_B}\n**Output Format:**\nFirst, provide your reasoning (what checks you performed). Then, on\nseparate new lines, give ratings using the EXACT tags below. Add no\nother text after the tags.\n<rating_A>INTEGER_1_TO_10</rating_A>\n<rating_B>INTEGER_1_TO_10</rating_B>\n**Rating Rules:**\n- Rate correctness on a 1-10 scale (10 = certainly correct, 8 = very\n likely correct, 5 = uncertain/borderline, 3 = likely incorrect, 1 =\n certainly incorrect).\n- Higher rating wins. Equal ratings imply a tie.\nPlease provide your analysis now.\nWe parse the rating pair out of the model's final message with an XML-tag regex first, then fall back to \\boxed{N} per side if the model used its trained \\boxed{} format instead of the requested tags.
All the knobs we actually shipped for chankhavu/c2-distill-vllm018-watchdog.
\nModel. chankhavu/c2-legaldistill-30b-a3b-fp8 β the FP8-quantized soft-clone of Nemotron-Cascade-2, re-quantized specifically for the Kaggle submission (not our published open-source FP8 checkpoint; see the \"Quantizing the Cascade 2 model\" section). Runtime: vLLM 0.18.0 on 1Γ H100 SXM under the Kaggle inference server sandbox.
vLLM serve command. Single GPU, FP8 KV cache, large context window, chunked prefill, prefix caching, async scheduling, native function-calling enabled with the qwen3_coder tool-call parser:
vllm serve /path/to/c2-legaldistill-30b-a3b-fp8 \\\n --served-model-name chankhavu/c2-legaldistill-30b-a3b-fp8 \\\n --tensor-parallel-size 1 \\\n --max-num-seqs 64 \\\n --gpu-memory-utilization 0.9 \\\n --dtype auto \\\n --max-model-len 262144 \\\n --max-num-batched-tokens 8192 \\\n --kv-cache-dtype fp8 \\\n --mamba-ssm-cache-dtype float32 \\\n --mamba-cache-mode all \\\n --mamba-block-size 512 \\\n --enable-chunked-prefill \\\n --enable-prefix-caching \\\n --async-scheduling \\\n --enable-auto-tool-choice \\\n --tool-call-parser qwen3_coder \\\n --trust-remote-code \\\n --host 0.0.0.0 --port 8000\nWe also run a lightweight watchdog subprocess that polls /proc every 30 s and relaunches vLLM if the APIServer / EngineCore processes disappear (scrapes the VLLM::APIServer / VLLM::EngineCor... comm names β the literal vllm.entrypoints string stops appearing in cmdline after vLLM calls setproctitle during startup). The relaunch itself takes ~100 s, which is why the watchdog restart trigger is \"no live vllm process found\" rather than HTTP-based.
Solver configuration (from CFG):
| Setting | \nValue | \nNotes | \n
|---|---|---|
attempts | \n8 | \nSolution rollouts per problem | \n
workers | \n16 | \nThread pool size (2Γ attempts, so cancellations don't stall new launches) | \n
early_stop | \n4 | \nStop once 4 rollouts agree on an integer answer | \n
turns | \n128 | \nMax conversation turns per rollout | \n
temperature | \n1.0 | \nSampling diversity β all rollouts share the prompt | \n
top_p | \n0.95 | \n\n |
top_logprobs | \n5 | \nTop-5 logprobs for entropy computation | \n
enable_thinking | \nTrue | \nCascade 2's <think>β¦</think> reasoning | \n
context_tokens | \n262144 | \nvLLM --max-model-len | \n
max_completion_tokens | \n131072 | \nPer-turn completion cap | \n
notebook_limit | \n17400 s (4.83 h) | \nKaggle inference ceiling | \n
high_problem_timeout | \n900 s | \nDefault per-problem budget | \n
base_problem_timeout | \n300 s | \nReserved per remaining problem; also the per-problem floor | \n
python_timeout | \n30 s | \nPer Python tool call | \n
max_output_characters | \n1000 | \nSandbox stdout truncation | \n
seed | \n42 | \nUnused (vLLM picks fresh seeds per request β we want the cross-worker diversity) | \n
Adaptive budget formula. For each problem we compute\nbudget = min(high_problem_timeout, max(base_problem_timeout, time_left β base_problem_timeout Γ problems_left_others)), so early problems get the full 900 s; as the wall clock runs down we reserve 300 s per remaining problem and squeeze the current one toward the 300 s floor.
System prompt.
\nYou are a helpful assistant and a mathematical problem solver.\nSolve the given problem step by step using rigorous reasoning.\nUser template ({problem} substituted per problem):
{problem}\n\nThe answer is a non-negative integer between 0 and 99999, inclusive.\nPut your final answer in \\boxed{}.\nTool definition β the model is given exactly one function, a stateful IPython code executor:
\n{\n \"type\": \"function\",\n \"function\": {\n \"name\": \"stateful_python_code_exec\",\n \"description\": \"Call this function to execute Python code in a stateful Jupyter notebook environment. Python will respond with the output of the execution or time out after 120.0 seconds. The environment has math, numpy, sympy, itertools, collections, and mpmath pre-imported.\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"code\": {\"type\": \"string\", \"description\": \"Code to execute\"}\n },\n \"required\": [\"code\"]\n }\n }\n}\nPer-call timeout is actually 30 s (from CFG.python_timeout), not 120 s β the description is the one the Cascade 2 model was trained to expect, so we leave it unchanged.
Sandbox preimports. Every new session starts with:
\nimport math\nimport numpy\nimport sympy\nimport itertools\nimport collections\nimport mpmath\nmpmath.mp.dps = 64\nAnswer extraction. A small regex cascade applied to each rollout's final non-tool-call message, in order: (i) \\boxed{DIGITS} β pick the last one; (ii) \\boxed{β¦ = DIGITS} β pick the last integer inside the last \\boxed{}; (iii) final answer is DIGITS. Accepts integers in [0, 99999] only. Before scanning, <think>β¦</think> reasoning blocks are stripped.
Voting. Each rollout's weight is 1 / max(mean_token_entropy, 1e-9), where the per-token entropy is computed from the top-5 logprobs of the generated tokens (not tool-call bytes). The picked answer is the one with the highest sum of weights across the rollouts that produced it.
Hyperparameters. Grader runs at temperature 0, top_p 0.95, same Python sandbox as the solver. Tool-call budget per grader session: 15 calls; past that we force-respond to any pending tool calls with a stub and instruct the model to output its ratings immediately. Maximum verifications per problem: 16. Maximum concurrent rollouts (solution + verification combined) in the thread pool: 8.
\nWhy does gpt-oss-120b's tool-call overhead come out roughly 2Γ Cascade 2's? The whole question is really about time: a typical gpt-oss-120b rollout burns 37 s of wall-clock on errored or timed-out tool calls β 14.0% of its 266 s average rollout β vs 17 s for Cascade 2 (6.3% of its 272 s rollout). ~2.2Γ the absolute, ~2.2Γ the rollout fraction. All numbers below are from the same LB-simulation runs that feed the chart in Β§Per-problem breakdown (aimo50a/b/c, 3 shuffles Γ 50 problems Γ up to 8 rollouts per problem, 1Γ H100 SXM under the 5 h Kaggle budget).
\nPer-rollout averages β every column below is a mean across rollouts (912 for gpt-oss-120b, 850 for Cascade 2; < 50Γ3Γ8 = 1200 because the adaptive scheduler cancels in-flight rollouts once 4 agree on an answer). In the table, \"error\" means any failed tool call β Python exceptions and the 30 s sandbox timeouts together.
\n| Model | \nrollout sec | \ntool calls | \ntool sec | \nerror sec | \n% rollout errored | \n
|---|---|---|---|---|---|
| gpt-oss-120b | \n266 | \n18.4 | \n66.3 | \n37.3 | \n14.0% | \n
| Cascade 2 | \n272 | \n14.3 | \n32.4 | \n17.1 | \n6.3% | \n
Within each model, where the error+timeout time goes (% errs and % err-time are within-model fractions of the error+timeout totals):
All low-frequency classes (< 0.1 occurrences per rollout in both models) are collapsed into \"other\" β this includes RuntimeError, ValueError, KeyError, IndexError, ImportError, and uncategorized Python tracebacks.
\n| Class | \ngpt-oss-120b count/rollout | \nsec/rollout | \n% errs | \n% err-time | \nCascade 2 count/rollout | \nsec/rollout | \n% errs | \n% err-time | \n
|---|---|---|---|---|---|---|---|---|
| [TIMEOUT] (30 s cap) | \n1.01 | \n33.1 | \n36.4% | \n88.6% | \n0.51 | \n16.0 | \n33.1% | \n94.0% | \n
| SyntaxError | \n0.72 | \n0.3 | \n26.1% | \n0.9% | \n0.11 | \n0.0 | \n7.3% | \n0.1% | \n
| NameError | \n0.49 | \n2.3 | \n17.8% | \n6.0% | \n0.15 | \n0.1 | \n9.7% | \n0.7% | \n
| TypeError | \n0.17 | \n0.1 | \n6.1% | \n0.3% | \n0.23 | \n0.0 | \n15.1% | \n0.2% | \n
| AttributeError | \n0.10 | \n0.0 | \n3.6% | \n0.1% | \n0.10 | \n0.1 | \n6.5% | \n0.4% | \n
| other | \n0.28 | \n1.5 | \n10.0% | \n4.1% | \n0.41 | \n0.7 | \n28.3% | \n4.6% | \n
| All errored + timed-out | \n2.77 | \n37.3 | \n100% | \n100% | \n1.53 | \n17.1 | \n100% | \n100% | \n
Timeouts are ~90% of the wasted wall-clock for both solvers (88.6% / 94.0%) β every Python error class is a rounding error by time.
\nNameError carries the strongest cross-model signal (0.49/rollout for gpt-oss-120b vs 0.15 for Cascade 2). gpt-oss-120b's 451 extractable NameErrors split into two causes:
\nsp (16), random (15), deque (9), math (5), and other stdlib aliases are already live, but the pre-import block (math, numpy, sympy, itertools, collections, mpmath) doesn't expose them and the model rarely writes the import line itself.Following up on the questions from @philipvonderlind, here are some additional questions to help us assess your submission:
\nLook forward to reading your responses on these.
", "rawMarkdown": "Following up on the questions from @philipvonderlind, here are some additional questions to help us assess your submission:\n\n1. **AnswerBench contamination concern:** Cascade 2 was released in March 2026, a few months after IMO-AnswerBench was published, whereas gpt-oss-120b was released earlier. Are you worried that Cascade 2's higher score on your local validation set (drawn from AnswerBench) might partly reflect AnswerBench problems having been in Cascade 2's training data, rather than a fair reasoning comparison against gpt-oss-120b?\n2. **Why Cascade 2 over a gpt-oss-120b verifier path:** Your offline numbers show gpt-oss-120b has the higher pass@8 (77.8% vs Cascade 2's 74.6%), and you note an oracle verifier would pick gpt-oss-120b. You then invested heavily in the soft-clone + quantization + speculator pipeline for Cascade 2. With hindsight, what made you commit to the Cascade 2 route rather than building (e.g. a learned) verifier for gpt-oss-120b that could close the maj@8 β pass@8 gap?\n\nLook forward to reading your responses on these.", "votes": 1, "replies": [ { "id": 3459537, "postDate": "2026-05-17T23:58:58.033Z", "content": "Thanks a lot, Sam Bealing (@sambealing) for thoughtful questions! We had similar concerns during AIMO3. In fact, we were skeptical about Cascade 2 at first and were not sure if it was benchmaxxed. Here is why we ultimately chose to trust it.
\n\n\nand also AnswerBench contamination concern: Cascade 2 was released in March 2026, a few months after IMO-AnswerBench was published, whereas gpt-oss-120b was released earlier. Are you worried that Cascade 2's higher score on your local validation set (drawn from AnswerBench) might partly reflect AnswerBench problems having been in Cascade 2's training data, rather than a fair reasoning comparison against gpt-oss-120b?
\n
This is a legitimate concern and we've added a note to Β§Evaluation set acknowledging it explicitly. Short version:
\nThe contamination caveat is now in the writeup body so the reader can weight the absolute numbers themselves.
\n\n\nWhy Cascade 2 over a gpt-oss-120b verifier path: Your offline numbers show gpt-oss-120b has the higher pass@8 (77.8% vs Cascade 2's 74.6%), and you note an oracle verifier would pick gpt-oss-120b. You then invested heavily in the soft-clone + quantization + speculator pipeline for Cascade 2. With hindsight, what made you commit to the Cascade 2 route rather than building (e.g. a learned) verifier for gpt-oss-120b that could close the maj@8 β pass@8 gap?
\n
A few interacting reasons, in roughly the order they mattered:
\nThe resource-constraint version of this argument is now in the writeup as a follow-up paragraph after the oracle-verifier note in Β§Test-time scaling. The training-time / risk-budget version we hadn't articulated; thank you for prompting it.
", "rawMarkdown": "Thanks a lot, Sam Bealing (@sambealing) for thoughtful questions! We had similar concerns during AIMO3. In fact, we were skeptical about Cascade 2 at first and were not sure if it was benchmaxxed. Here is why we ultimately chose to trust it.\n\n## AnswerBench contamination\n\n> and also AnswerBench contamination concern: Cascade 2 was released in March 2026, a few months after IMO-AnswerBench was published, whereas gpt-oss-120b was released earlier. Are you worried that Cascade 2's higher score on your local validation set (drawn from AnswerBench) might partly reflect AnswerBench problems having been in Cascade 2's training data, rather than a fair reasoning comparison against gpt-oss-120b?\n\nThis is a legitimate concern and we've added a note to *Β§Evaluation set* acknowledging it explicitly. Short version:\n\n- **Both models are at some risk.** IMO-AnswerBench is a rewriting of previously-published IMO problems, Shortlists, and national olympiads. Both gpt-oss-120b and Cascade 2 were trained on data that almost certainly includes some of these problems in their original form. So contamination is *the baseline*, not a one-sided concern.\n- **Cascade 2 is more exposed than gpt-oss-120b.** Cascade 2 was released after AnswerBench was published, so the contamination window for Cascade 2 is wider. The reasoning advantage we measure should be discounted accordingly when comparing absolute accuracy.\n- **We did not run our own contamination checks.** We chose to trust the data-curation procedure described in the Nemotron-Cascade-2 paper, which uses n-gram deduplication against known evaluation benchmarks. If that procedure works as documented, the contamination on AnswerBench specifically should be limited β but we can't verify this independently.\n- **What this doesn't undermine:** the *relative* observations that hold up regardless of contamination. The per-rollout cost difference, the maj@k stability difference, the tool-call error breakdown, the time-budget scaling shape β none of these depend on absolute reasoning ability. If Cascade 2's true reasoning capability is lower than its AnswerBench number suggests, our conclusions about \"Cascade 2 wins maj@k under time budget *because* it's cheaper per rollout and more consistent across rollouts\" still hold; we'd just be over-reporting the absolute scoreboard.\n\nThe contamination caveat is now in the writeup body so the reader can weight the absolute numbers themselves.\n\n\n## Why Cascade 2 over a gpt-oss-120b verifier path\n\n> **Why Cascade 2 over a gpt-oss-120b verifier path:** Your offline numbers show gpt-oss-120b has the higher pass@8 (77.8% vs Cascade 2's 74.6%), and you note an oracle verifier would pick gpt-oss-120b. You then invested heavily in the soft-clone + quantization + speculator pipeline for Cascade 2. With hindsight, what made you commit to the Cascade 2 route rather than building (e.g. a learned) verifier for gpt-oss-120b that could close the maj@8 β pass@8 gap?\n\nA few interacting reasons, in roughly the order they mattered:\n\n1. **The verifier route doesn't fit on the H100.** gpt-oss-120b's MXFP4 weights are ~120 GB; that leaves ~60 GB of HBM for KV cache, activations, and the 8 parallel solution rollouts we already need to fit. There is no headroom on a single H100 SXM for a *separate* verifier model running alongside it. Using gpt-oss-120b as its own verifier doubles the per-problem token bill on a budget that, at our throughput, already only fits ~8 solution rollouts. Both forms of \"gpt-oss-120b + verifier\" run out of either memory or wall-clock before they can be meaningfully tuned.\n2. **Cascade 2 leaves room for verification later.** Cascade 2's smaller footprint and ~70% higher throughput meant we could *start* with plain voting and then layer self-verification in the GPU bubbles between fast and slow solution rollouts. The self-verification harness we shipped (untuned, last 24 h) underperformed voting, but the option existed at all because of Cascade 2's overhead. With gpt-oss-120b we wouldn't have had that fallback path even in principle.\n3. **Training a learned verifier is its own project.** \"Build a learned verifier for gpt-oss-120b that closes maj@8 β pass@8\" is a meaningfully larger undertaking than the soft-clone we shipped: it needs verifier-training data (problem + multiple solutions + correctness labels), a training run, an eval harness for the verifier itself, and integration into the inference path.\n4. **AIMO 2 priors.** The historical evidence from AIMO 2 was that winning solutions optimized maj@k under a real time budget, not pass@β. Cascade 2 *measurably* wins maj@k under our time budget; gpt-oss-120b would have needed a working verifier to compete on the same axis. Investing in the model that already wins the relevant metric was the lower-variance bet.\n\nThe resource-constraint version of this argument is now in the writeup as a follow-up paragraph after the oracle-verifier note in Β§Test-time scaling. The training-time / risk-budget version we hadn't articulated; thank you for prompting it.\n" } ] }, { "id": 3457297, "postDate": "2026-05-13T13:49:55.270Z", "content": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.
\nYour work shows impressive contributions, including implementing a missing feature in vLLM. We especially appreciate the in-depth engineering insights you provided, which even helped resolve issues that other participants encountered during the competition. I only have a few more surface-level questions:
\nvLLM: Was your PR eventually accepted into vLLM? I have read several other writeups from participants who used EAGLE3 and encountered the same issue you describe above. Please keep us posted if there are any updates.
Pass@k Plot: Did you mean to write that gpt-oss-120b wins pass@k at every time budget? Based on the plot, it seems that Cascade outperforms gpt-oss-120b across all tested budgets.
Time Management: Since EAGLE3 primarily improves inference speed and therefore saves time, why did you not spend more effort on developing a more advanced time-management system as well to further increase the gains? Many other contestants found that a good scheduling strategy could help, especially because hard and easy problems appeared in random order.
Tool Calls: Did you investigate the common causes of the significantly higher tool-call error rate in gpt-oss-120b, such as missing packages, outdated APIs, syntax issues, or similar failure modes? It would be interesting to understand why the Nemotron model appears to produce much cleaner tool calls.
Looking forward to your response :)
", "rawMarkdown": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the [updated discussion post](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/689703).\n\nYour work shows impressive contributions, including implementing a missing feature in vLLM. We especially appreciate the in-depth engineering insights you provided, which even helped resolve issues that other participants encountered during the competition. I only have a few more surface-level questions:\n\n1. **vLLM**: Was your PR eventually accepted into vLLM? I have read several other writeups from participants who used EAGLE3 and encountered the same issue you describe above. Please keep us posted if there are any updates.\n\n2. **Pass@k Plot**: Did you mean to write that `gpt-oss-120b` wins pass@k at every time budget? Based on the plot, it seems that Cascade outperforms `gpt-oss-120b` across all tested budgets.\n\n3. **Time Management**: Since EAGLE3 primarily improves inference speed and therefore saves time, why did you not spend more effort on developing a more advanced time-management system as well to further increase the gains? Many other contestants found that a good scheduling strategy could help, especially because hard and easy problems appeared in random order.\n\n4. **Tool Calls**: Did you investigate the common causes of the significantly higher tool-call error rate in `gpt-oss-120b`, such as missing packages, outdated APIs, syntax issues, or similar failure modes? It would be interesting to understand why the Nemotron model appears to produce much cleaner tool calls.\n\nLooking forward to your response :)\n", "votes": 1, "replies": [ { "id": 3459539, "postDate": "2026-05-17T23:59:55.407Z", "content": "Hi Philip Vonderlind (@philipvonderlind), thank you for thoughtful questions. Your 2nd question raised a real concern that we were not clear about our wording enough. The \"gpt-oss-120b always wins pass@k\" is false, and it should be attributed to \"wins given attempt budget\", not \"time budget\".
\nYour 4th question called out a real mistake in our numbers and analysis. We carefully re-visited that analysis and re-generated the chart (basically it was a parsing issue on our end due to prompt differences between gpt-oss and cascade 2).
\n\n\nvLLM: Was your PR eventually accepted into vLLM? I have read several other writeups from participants who used EAGLE3 and encountered the same issue you describe above. Please keep us posted if there are any updates.
\n
Not merged yet β we're still working through two issues before it's ready to upstream:
\nvllm-project/vllm#41233 was opened against the same mamba-cache-poisoning bug. It's still open at the time of writing, but it's likely the path upstream takes. Our perf optimizations are orthogonal to that PR (it has no speed/kernels work) and would still apply on top of it; we plan to submit them as a follow-up once the correctness side lands and our Blackwell story is fixed.The fork remains useful for anyone running Hybrid-Mamba SD on Hopper today. We added a new Β§Upstream PR status subsection inside the writeup's vLLM section to track this honestly, and we'll keep you posted as the situation upstream evolves.
\n\n\nPass@k Plot: Did you mean to write that gpt-oss-120b wins pass@k at every time budget? Based on the plot, it seems that Cascade outperforms gpt-oss-120b across all tested budgets.
\n
You're right β the original sentence was wrong. Within the plotted time budget, Cascade 2 wins both pass@k and maj@k: maj@k by a comfortable margin across the whole range, and pass@k by a smaller margin because Cascade 2 is cheaper per rollout and gets to fit more rollouts in any fixed wall-clock budget.
\nWhat we should have written, and now do: gpt-oss-120b has a higher pass@k ceiling that only becomes visible beyond the plotted budget β its rollouts are more diverse (helps pass@k, hurts plurality voting), so given enough attempts a hypothetical oracle verifier would eventually prefer gpt-oss-120b. Inside the actual competition's time budget, Cascade 2 dominates on both axes.
\nThe paragraph next to the chart has been rewritten to reflect this. We've also added a note explaining why we couldn't realize that pass@k ceiling within Kaggle's resource budget anyway: a separate verifier won't fit alongside gpt-oss-120b's ~120 GB MXFP4 weights on a single H100, and using gpt-oss-120b as its own verifier doubles the per-problem token bill on a budget that already only fits ~8 solution rollouts.
\n\n\nTime Management: Since EAGLE3 primarily improves inference speed and therefore saves time, why did you not spend more effort on developing a more advanced time-management system as well to further increase the gains? Many other contestants found that a good scheduling strategy could help, especially because hard and easy problems appeared in random order.
\n
Honest answer: we joined the competition late, and the Kaggle 1-submission-per-day cap meant essentially every scheduling choice we shipped was a first-try call, with no headroom to A/B-test on the hidden LB. A more elaborate per-problem scheduler was on the wishlist, but the engineering dependency we did spend weeks on β PC + SD support for hybrid-Mamba models in vLLM β was already over our time budget, and we judged that getting the simple harness right was a safer use of the remaining days than tuning a scheduler we couldn't validate without burning submission slots.
\nWe also weighted this decision by what optimizing for in the first place. Our priors from AIMO 2 were that winning solutions in this class of competition optimized for maj@k under a time budget, not pass@β β pass@β tends to be similar across competitive models, but maj@k under a real budget is where the consistency premium pays off. That's why we picked Cascade 2, which (per the test-time-scaling chart) gives the best maj@k under a wall-clock budget. Cascade 2's lower per-rollout cost also means an adaptive scheduler has less to squeeze: the simple min(900, max(300, time_left β 300 Γ problems_left_others)) formula already kept us at the 900 s ceiling for most of each shuffle.
In hindsight, a smarter scheduler would have been a high-leverage thing to try, and we've added it to the What we'd do differently list in the Conclusion. The Conclusion now also has a \"Why this is mostly a first-try harness\" paragraph that articulates this trade-off explicitly.
\n\n\nTool Calls: Did you investigate the common causes of the significantly higher tool-call error rate in gpt-oss-120b, such as missing packages, outdated APIs, syntax issues, or similar failure modes? It would be interesting to understand why the Nemotron model appears to produce much cleaner tool calls.
\n
Yes, in depth. We added Appendix G (tool-call error breakdown) which re-pairs every tool response in the LB-sim with the code that triggered it (504 rollouts per model). Headline findings:
\nsp (sympy alias), random, deque, math, etc. The pre-import block (math, numpy, sympy, itertools, collections, mpmath) doesn't expose sympy as sp or random, and gpt-oss-120b writes Jupyter-cell-style snippets that assume these names are already live in the session rather than import-ing them.NameError on what they assumed were live names. Cascade 2 hits the same rollback mechanism, just less often (it writes more self-contained cells with explicit imports at the top of each block).We also re-rendered the per-problem-breakdown chart on the same LB-sim data after discovering that the original chart silently bucketed Cascade 2's bare-text timeout responses as \"successful tool call\" (the two solver recorders formatted timeout strings differently).
", "rawMarkdown": "Hi Philip Vonderlind (@philipvonderlind), thank you for thoughtful questions. Your 2nd question raised a real concern that we were not clear about our wording enough. The \"gpt-oss-120b always wins pass@k\" is false, and it should be attributed to \"wins given attempt budget\", not \"time budget\".\n\nYour 4th question called out a real mistake in our numbers and analysis. We carefully re-visited that analysis and re-generated the chart (basically it was a parsing issue on our end due to prompt differences between gpt-oss and cascade 2).\n\n\n## 1. vLLM PR status\n\n> **vLLM:** Was your PR eventually accepted into vLLM? I have read several other writeups from participants who used EAGLE3 and encountered the same issue you describe above. Please keep us posted if there are any updates.\n\nNot merged yet β we're still working through two issues before it's ready to upstream:\n\n1. **Hopper-only optimizations.** The performance work in our fork (cudagraph capture of the K-iteration drafter loop, the PIECEWISE β FULL upgrade we planned for p=12+, the FlashInfer attention switch) assumed Hopper SM 9.0 behaviour (because AIMO3 competition uses H100). On Blackwell SM 10.0 the same code hits graph-compilation issues we're still resolving. vLLM upstream needs GPU-portable code paths, so the Blackwell-clean rewrite of the kernel layer needs to land before we can merge.\n2. **A community PR now covers the correctness fix.** One week after the competition deadline, [`vllm-project/vllm#41233`](https://github.com/vllm-project/vllm/pull/41233) was opened against the same mamba-cache-poisoning bug. It's still open at the time of writing, but it's likely the path upstream takes. Our perf optimizations are **orthogonal** to that PR (it has no speed/kernels work) and would still apply on top of it; we plan to submit them as a follow-up once the correctness side lands and our Blackwell story is fixed.\n\nThe [fork](https://github.com/hav4ik/vllm/tree/pc-spec-v2-nonpc-slots) remains useful for anyone running Hybrid-Mamba SD on Hopper today. We added a new *Β§Upstream PR status* subsection inside the writeup's vLLM section to track this honestly, and we'll keep you posted as the situation upstream evolves.\n\n## 2. Pass@k plot\n\n> **Pass@k Plot:** Did you mean to write that gpt-oss-120b wins pass@k at every time budget? Based on the plot, it seems that Cascade outperforms gpt-oss-120b across all tested budgets.\n\nYou're right β the original sentence was wrong. Within the plotted time budget, **Cascade 2 wins both pass@k and maj@k**: maj@k by a comfortable margin across the whole range, and pass@k by a smaller margin because Cascade 2 is cheaper per rollout and gets to fit more rollouts in any fixed wall-clock budget.\n\nWhat we *should* have written, and now do: gpt-oss-120b has a higher pass@k *ceiling* that only becomes visible **beyond** the plotted budget β its rollouts are more diverse (helps pass@k, hurts plurality voting), so given enough attempts a hypothetical oracle verifier would eventually prefer gpt-oss-120b. Inside the actual competition's time budget, Cascade 2 dominates on both axes.\n\nThe paragraph next to the chart has been rewritten to reflect this. We've also added a note explaining why we couldn't realize that pass@k ceiling within Kaggle's resource budget anyway: a separate verifier won't fit alongside gpt-oss-120b's ~120 GB MXFP4 weights on a single H100, and using gpt-oss-120b as its own verifier doubles the per-problem token bill on a budget that already only fits ~8 solution rollouts.\n\n## 3. Time management\n\n> **Time Management:** Since EAGLE3 primarily improves inference speed and therefore saves time, why did you not spend more effort on developing a more advanced time-management system as well to further increase the gains? Many other contestants found that a good scheduling strategy could help, especially because hard and easy problems appeared in random order.\n\nHonest answer: we joined the competition late, and the Kaggle 1-submission-per-day cap meant essentially every scheduling choice we shipped was a **first-try call**, with no headroom to A/B-test on the hidden LB. A more elaborate per-problem scheduler was on the wishlist, but the engineering dependency we *did* spend weeks on β PC + SD support for hybrid-Mamba models in vLLM β was already over our time budget, and we judged that getting the simple harness right was a safer use of the remaining days than tuning a scheduler we couldn't validate without burning submission slots.\n\nWe also weighted this decision by what optimizing for in the first place. Our priors from AIMO 2 were that winning solutions in this class of competition optimized for **maj@k under a time budget**, not pass@β β pass@β tends to be similar across competitive models, but maj@k under a real budget is where the consistency premium pays off. That's why we picked Cascade 2, which (per the test-time-scaling chart) gives the best maj@k under a wall-clock budget. Cascade 2's lower per-rollout cost also means an adaptive scheduler has less to squeeze: the simple `min(900, max(300, time_left β 300 Γ problems_left_others))` formula already kept us at the 900 s ceiling for most of each shuffle.\n\nIn hindsight, a smarter scheduler would have been a high-leverage thing to try, and we've added it to the *What we'd do differently* list in the Conclusion. The Conclusion now also has a *\"Why this is mostly a first-try harness\"* paragraph that articulates this trade-off explicitly.\n\n## 4. Tool calls\n\n> **Tool Calls:** Did you investigate the common causes of the significantly higher tool-call error rate in gpt-oss-120b, such as missing packages, outdated APIs, syntax issues, or similar failure modes? It would be interesting to understand why the Nemotron model appears to produce much cleaner tool calls.\n\nYes, in depth. We added **Appendix G** (tool-call error breakdown) which re-pairs every tool response in the LB-sim with the code that triggered it (504 rollouts per model). Headline findings:\n\n1. **Time, not count, is what matters.** A typical gpt-oss-120b rollout burns ~37 s of wall-clock on failed tool calls (14% of its rollout), Cascade 2 ~17 s (6%). Per-call error *rates* are within a few points of each other; the per-rollout *time* difference is ~2.2Γ.\n2. **Timeouts dominate the wasted time.** Pure 30 s sandbox timeouts (no output produced) account for **88.6%** of gpt-oss-120b's wasted tool wall-clock and **94.0%** of Cascade 2's. Every Python error class is a rounding error by time compared to timeouts. This was the most surprising finding for us β the wasted wall-clock is overwhelmingly about *how often the model writes code that doesn't finish in 30 s*, not about syntactic or library failures.\n3. **NameError is the strongest cross-model signal on Python errors.** 0.49/rollout for gpt-oss-120b vs 0.15 for Cascade 2 (3.3Γ). Two root causes split the gpt-oss-120b cases roughly into a third and two-thirds:\n - **Forgotten imports (~1/3)** β `sp` (sympy alias), `random`, `deque`, `math`, etc. The pre-import block (`math, numpy, sympy, itertools, collections, mpmath`) doesn't expose `sympy as sp` or `random`, and gpt-oss-120b writes Jupyter-cell-style snippets that assume these names are already live in the session rather than `import`-ing them.\n - **Sandbox rollback (~2/3)** β standard Jupyter / IPython preserves partial assignments when a cell errors; our sandbox snapshots state *before* each cell and restores that snapshot whenever the cell raises. Models trained on REPL semantics write follow-up code referencing variables from cells that errored, and hit `NameError` on what they assumed were live names. Cascade 2 hits the same rollback mechanism, just less often (it writes more self-contained cells with explicit imports at the top of each block).\n4. **gpt-oss-120b also pays a SyntaxError toll** (0.72 / rollout, 26% of its non-timeout errors), almost all incomplete-bracket / truncation patterns. We suspect this is a public-notebook artefact (smaller code chunks) rather than a model-trait, but didn't run the experiment to confirm.\n\nWe also re-rendered the per-problem-breakdown chart on the same LB-sim data after discovering that the original chart silently bucketed Cascade 2's bare-text timeout responses as \"successful tool call\" (the two solver recorders formatted timeout strings differently)." } ] }, { "id": 3447586, "postDate": "2026-04-23T13:12:42.480Z", "content": "Fantastic engineering work! I have a question about speculative decoding. You show very good \"Accept rate\", but how it changes during long trajectories? You fine-tuned it with 65k context, but does it work well with long context at test time?
", "rawMarkdown": "Fantastic engineering work! I have a question about speculative decoding. You show very good \"Accept rate\", but how it changes during long trajectories? You fine-tuned it with 65k context, but does it work well with long context at test time?", "votes": 1, "replies": [ { "id": 3447798, "postDate": "2026-04-24T01:00:39.007Z", "content": "Thanks for your kind words π You've got a good point!
\nAlthough we did fine-tune with maximum of 65K context, in reality there's not many TIR tasks that have more than 40K generated tokens (we did our best to filter out the hardest ones). TIR reasoning is much shorter than COT (which we didn't sampled enough for training). Validation during training was done on a hard subset of IMO-AnswerBench (we filtered all validation benchmarks like AIME and IMO-benches out from training), but they don't have that much rollouts with >65K tokens either.
\nHowever, the 3.55 acceptance rate cited above was on AIME'26, which doesn't need more than 65K tokens. It was my primary \"unit test\" for Eagle3 work, because AIME is simple enough to run in ~10 minutes, but just hard enough to reveal any bugs in Mamba states handling (my criteria was 98% accuracy - if I see a drop of 10%, I know exactly that somewhere a Mamba state got corrupted), and acceptance rate numbers were collected together with accuracy unit testing. We didn't have time to properly test it on enough 65K+ samples. Will come back with some measurements and update the writeup (I'll continue working on the PC+SD anyways).
\nOne thing I want to emphasize: Speculative Decoding is not worth it for small models. I know that people smarter than me already knows this, but the economy of 3B active models is so brutal. 1 token step is just 3.4 ms: 2.8ms Cuda Graph + 0.6ms overhead from CPU/sampling. Naive Eagle3 implementation is 13ms - you need acceptance rate of 4 to break even. We brought that down to 8.9ms - but that still requires 2.6 acceptance rate to break even. Not to mention other overheads that makes it closer to 3.0 acceptance rate to break even. Models like Cascade2 are heartbreakingly ridiculously fast!
", "rawMarkdown": "Thanks for your kind words π You've got a good point!\n\nAlthough we did fine-tune with maximum of 65K context, in reality there's not many TIR tasks that have more than 40K generated tokens (we did our best to filter out the hardest ones). TIR reasoning is much shorter than COT (which we didn't sampled enough for training). Validation during training was done on a hard subset of IMO-AnswerBench (we filtered all validation benchmarks like AIME and IMO-benches out from training), but they don't have that much rollouts with >65K tokens either.\n\nHowever, the **3.55** acceptance rate cited above was on AIME'26, which doesn't need more than 65K tokens. It was my primary \"unit test\" for Eagle3 work, because AIME is simple enough to run in ~10 minutes, but just hard enough to reveal any bugs in Mamba states handling (my criteria was 98% accuracy - if I see a drop of 10%, I know exactly that somewhere a Mamba state got corrupted), and acceptance rate numbers were collected together with accuracy unit testing. We didn't have time to properly test it on enough 65K+ samples. Will come back with some measurements and update the writeup (I'll continue working on the PC+SD anyways).\n\nOne thing I want to emphasize: Speculative Decoding is not worth it for small models. I know that people smarter than me already knows this, but the economy of 3B active models is so brutal. 1 token step is just **3.4** ms: 2.8ms Cuda Graph + 0.6ms overhead from CPU/sampling. Naive Eagle3 implementation is 13ms - you need acceptance rate of 4 to break even. We brought that down to 8.9ms - but that still requires 2.6 acceptance rate to break even. Not to mention other overheads that makes it closer to 3.0 acceptance rate to break even. Models like Cascade2 are heartbreakingly ridiculously fast!", "votes": 2 } ] }, { "id": 3447327, "postDate": "2026-04-23T00:08:32.747Z", "content": "This is @yeoyunsianggeremie watching me getting stuck at some random side-quests \n
Changes to address rebuttal questions:
\nShort list of edits made to writeup.md (and supporting artifacts) in this rebuttal pass.
kaggle_per_problem_stats.png from scratch. The original right panel used an asymmetric is_error rule: the gpt-oss-120b recorder prefixed its 30 s timeouts with [ERROR], our Cascade 2 recorder left them bare, and the same rule caught one and missed the other. Cascade 2's tool_err went from 0.2% β 6.3% of wall-clock; gpt-oss-120b's stayed at 14.4%. The corrected chart also splits timeouts from Python errors into two visually distinct buckets (purple/red), and the panel-3 subtitle now states the 30 s sandbox timeout cap that applies to both solvers. Full reconciliation in rebuttal_n4.md.test_time_scaling_attempts.png as a visual companion that drops the per-rollout cost scaling. One sentence of framing, no separate findings.Summary of our main candidates lead-in tightened from \"local validation and public LB\" β \"local validation and the Kaggle leaderboard\" (now covers Public + Private columns).