#  Self-Consistency + TIR with Python-Verification-Weighted Voting for AIMO3

1. TL;DR
Two concrete, publicly verifiable contributions to the AIMO3 open-source lineage:
A. SC + TIR scaffold on top of parthenos's streaming baseline.
I published
[Self-Consistency Strategy](https://www.kaggle.com/code/canqiang/self-consistency-strategy)
(notebook slug:
self-consistency-strategy
, V12, 170 upvotes). It merges
[parthenos's streaming-inference reflexion prompting notebook](https://www.kaggle.com/code/nihilisticneuralnet/aimo3-streaming-inference-reflexion-prompting)
(score 27) with
[shelterw's Python-tool line](https://www.kaggle.com/code/shelterw/way-to-30-50-gpt-oss-w-python-tool)
, and adds what those notebooks lacked: (i) the
openai_harmony
chat template with
ReasoningEffort.HIGH
, (ii) a per-thread persistent Jupyter kernel (
BlockingKernelClient
+ port-pool allocator), (iii) K=8 parallel self-consistency with in-stream
\boxed{…}
scanning and a majority-threshold early-stop, (iv) a 16-thread OS-page-cache weight preloader that cuts vLLM startup from ~400 s to ~130 s. This notebook became the foundation ZaynYu forked into
[[40/50] GPT-OSS-120B TIR+DynamicTime+KernelPool](https://www.kaggle.com/code/zaynyu/40-50-gpt-oss-120b-tir-dynamictime-kernelpool)
(615 upvotes, Public LB 40).
B. Python-verification-weighted voting.
In the ZaynYu → daniil1423 → my fork chain, the only non-trivial change I made to the voting logic was to weight each attempt's vote by whether it used the Python tool:
base weight 1.0
+0.5 if the attempt made ≥ 3 tool calls
+0.25 if the attempt made ≥ 1 tool call
−0.3 if > 50% of its tool calls errored
Tie-break key:
(weighted_votes, raw_votes, total_calls)
. This pushed my best same-code submission from the 35–40 band (ZaynYu) to 42/50 peak over n=25 V2 submissions (μ = 38.44, σ = 1.58).
Neither contribution changes the ceiling by much — the ceiling is set by
gpt-oss-120b
's pass@k on this problem set — but both are small, reproducible, and the scaffold in (A) is demonstrably load-bearing downstream.
2. Competition Setup
AIMO3 gives 50 IMO-style problems with integer answers in
[0, 99999]
. Each submission is scored twice; a problem earns 1.0 if both runs agree and are correct, 0.5 if one run is correct, and 0 otherwise. Inference runs offline on a single NVIDIA H100-85GB under a 5-hour wall-clock limit. Models must have been released before 2026-03-15.
I used
gpt-oss-120b
(120B parameters, MXFP4 weights, ~65 GB VRAM) served via
vLLM==0.11.2
. The KV cache budget on 1×H100 with this model is about 15–16 GB, which constrains safe concurrent requests to roughly 8.
3. Lineage and Honest Attribution
This is the relevant portion of the public fork tree. Arrows point from upstream to downstream; numbers in parentheses are approximate Public LB peaks reported in each notebook's title or comments.
Kawchar Husain                                       (upstream prompt template)
        │
        ▼
parthenos · aimo3-streaming-inference-reflexion-prompting (V1, 27)
        │   streaming inference + reflexion prompting; no TIR; no Harmony
        │
shelterw · way-to-30-50-gpt-oss-w-python-tool          (parallel upstream: simple Python tool loop)
        │
        ▼
canqiang (me) · self-consistency-strategy (V12, 32)    ← Contribution A
        │   + Harmony protocol (ReasoningEffort.HIGH)
        │   + per-thread persistent Jupyter kernel pool
        │   + K=8 parallel SC with in-stream \boxed scanning + early-stop
        │   + 16-thread OS-page-cache weight preloader
        │
        ▼
ZaynYu · [40/50] GPT-OSS-120B TIR+DynamicTime+KernelPool (V26, 40)
        │   + dynamic per-problem time budget
        │   + class refactor (AIMO3Solver / AIMO3Sandbox / AIMO3Tool)
        │   + refined system prompts ("world-class IMO competitor", "Never guess")
        │
        ▼
daniil1423 · entropy-trajectory-scoring (40)
        │   + logprob/entropy-based trajectory scoring (I did not use this part)
        │
        ▼
canqiang (me) · fork of ETS ("Weighted Voting with Python Verification" V2, 42)   ← Contribution B
            + python-call-count & error-rate weighted voting
            (kept the rest of the ETS/ZaynYu infrastructure unchanged)
Two things I want to be explicit about:
The 42-score notebook was a
private fork during the competition
(I judged the delta over daniil1423 to be small enough that a separate public notebook during the competition was not obviously worth it). I plan to open-source it after the writeup period. The 170-vote notebook I published earlier is the SC+TIR scaffold (contribution A), not the 42-score fork.
The DynamicTime budget, the AIMO3Solver class refactor, and the entropy-trajectory machinery are
ZaynYu's and daniil1423's work
, not mine. The only voting-logic delta I contributed is the four-rule weighting above.
4. System Overview
4.1 Inference stack
gpt-oss-120b
, native MXFP4 weights, served by
vllm
0.11.2.
--kv-cache-dtype fp8_e4m3
,
--max-model-len 65536
,
--gpu-memory-utilization 0.96
,
--max-num-seqs 256
,
--async-scheduling
.
openai_harmony
for the chat template. System message is constructed with
SystemContent.new().with_reasoning_effort(ReasoningEffort.HIGH).with_tools(python_tool.tool_config)
, which is the native format
gpt-oss-120b
was post-trained on.
Sampling:
temperature=1.0
,
top_p=1.0
,
min_p=0.02
. Temperature is
not
a free knob for aligned models —
[techtech69's writeup](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/entropy-weighted-majority-voting-with-gpt-oss-120b)
(85th, public 43) documents an 8-point drop when they went to 0.6. My ablations agree.
4.2 Per-problem loop
Each incoming problem is handled by
K=8
worker threads sharing a
ThreadPoolExecutor
. Each thread:
Acquires one persistent
AIMO3Sandbox
(Jupyter kernel) from a pool.
Runs up to 128 reasoning turns. Each turn calls vLLM with
stream=True, return_token_ids=True
, streams tokens while:
checking a shared
stop_event
and the per-problem deadline,
scanning incoming text for
\boxed{…}
and bailing out as soon as a valid integer in
[0, 99999]
is found.
If the model emitted a
python
tool call, the sandbox executes the code in a stateful kernel and the output is fed back as a new conversation message.
python_calls
and
python_errors
are tracked per attempt.
When any 5/8 attempts agree on the same answer (
early_stop = 5
), the stop event fires and the remaining threads are cancelled.
4.3 Voting (my contribution B)
After the parallel sweep, votes are aggregated with four small rules on top of raw majority:
# AIMO3Solver._select_answer, used in the 42-score fork of ETS V2
for result in detailed_results:
    answer = result['Answer']
    if answer is None:
        continue
    weight = 1.0
    calls  = result['Python Calls']
    errors = result['Python Errors']
    if calls >= 3:
        weight += 0.5         # strong Python-verified answer
    elif calls >= 1:
        weight += 0.25        # at least one Python check
    if calls > 0 and errors / calls > 0.5:
        weight -= 0.3         # high error rate → unreliable
    stats[answer]['weighted_votes'] += weight
    stats[answer]['votes']  += 1
    stats[answer]['calls']  += calls
    stats[answer]['errors'] += errors

# Tie-break: weighted votes > raw votes > total Python calls
return max(stats.items(),
           key=lambda x: (x[1]['weighted_votes'],
                          x[1]['votes'],
                          x[1]['calls']))[0]
The intuition is trivial: if an answer was arrived at by an attempt that ran Python code several times and those calls mostly returned without an exception, it is more likely to be correct than an answer the model guessed without verification. The coefficients (0.25 / 0.5 / 0.3 / threshold 0.5) are not tuned; they are round numbers that fell out of the first thing I tried. The contribution is the
structure
of the weighting, not the specific values.
5. Reproducibility and the Noise Floor
Running the same notebook twice on Kaggle does
not
give the same score. This is not a subtle effect and it is not my claim — it is reported by both techtech69 and ZaynYu in their own writeups / comments. On vLLM, batched completion order depends on GPU scheduling, floating-point associativity perturbs the softmax, and downstream token sampling diverges.
5.1 Parthenos's upstream (no SC, no TIR): very noisy, centred low
From public comments on
[parthenos's aimo3-streaming-inference-reflexion-prompting](https://www.kaggle.com/code/nihilisticneuralnet/aimo3-streaming-inference-reflexion-prompting)
V1 (author score 27):
Forker
Score
Poemcourt (copy, no changes)
26
Dipankar Mitra (copy, no changes)
21
CloudYU
22
Octavi Grau
16
Artem Goncharov
18
swordsman (after increasing timeout)
22
Many more hit "Notebook threw exception". Spread is roughly 16–26 against an author-reported 27.
5.2 My SC+TIR scaffold (contribution A): higher and tighter
From the comment on my
[self-consistency-strategy](https://www.kaggle.com/code/canqiang/self-consistency-strategy)
:
"i scored 35 with no changes…"
— cm391
Author peak on that notebook was 32 on LB; the commenter got 35 with zero changes, and downstream forks that layered additional infrastructure (ZaynYu, daniil1423) reached 40+.
5.3 ZaynYu's fork of my scaffold: the best community reproduction data
ZaynYu's
[[40/50] GPT-OSS-120B TIR+DynamicTime+KernelPool](https://www.kaggle.com/code/zaynyu/40-50-gpt-oss-120b-tir-dynamictime-kernelpool)
(615 upvotes, 1,362 copies) has 57 comments and is the most-forked downstream child of my SC scaffold. A selection of the scores people reported, quoted literally from that comment thread:
Forker
Variant
Public LB
Tong Hui Kang
V1 (direct copy)
38
Tong Hui Kang
V2
35
Tong Hui Kang
V10
36
Tong Hui Kang
V12
36
Siancy ("Same as parthenos's, no modifications")
V1
41
THUNDER THUNDER
V01
40
THUNDER THUNDER
V02
36
Abhijeet Bhargava
V01 / V03 / V05
38 / 35 / 34
chenjixuan666 (fork of Siancy)
V01
38
parthenos (on ZaynYu's V15)
—
37
Dipankar Mitra
submitted twice
33 / 33
ZaynYu himself on his own notebook:
"The score of the LB by running the same notebook is not guaranteed to be the same."
So, empirically, the scaffold I contributed (A) sits downstream of parthenos's 16–26 band and supports a community reproduction band of roughly 34–41 on top of the same hardware.
5.4 My own V2 notebook (contribution B): 25 identical submissions
I submitted the same V2 code 25 times to measure the noise floor:
n = 25
min = 35
max = 42
mean = 38.44
std  = 1.58
A single run scoring 42 is well inside the tail; a single run scoring 35 is also inside the tail. This is why I frame the Python-weighted voting as "pushed the ceiling up by ~2 points", not "solves more problems". The voting change is not the dominant source of score; the model and the TIR loop are.
6. Ablations
Everything in this table is a single-submission LB score from my own submission list. They are not independent trials — each number is a draw from a noisy distribution with σ ≈ 1–2 on top of a ~35–42 base. I report them with that caveat.
Configuration
Best
Mean
N
Notes
Weighted Voting w/ Python Verification (V2, contribution B)
42
38.44
25
Peak and baseline for my work
ans_verifys (verifier variant)
41
37.67
6
Different verification logic, similar spread
entropy-trajectory-scoring (daniil1423's logprob-entropy scoring)
39
36.78
9
Without my weighted voting, entropy scoring alone
Dual Entropy-Weighted Voting
37
35.71
7
Combining two entropy signals
ZaynYu-style direct fork (36/40 variant)
34
32.67
3
Before my voting change
self-consistency-strategy V12 (contribution A, public)
32
—
2
Before any voting change; scaffold only
parthenos direct fork (streaming + reflexion, no TIR)
26
23
2
Upstream baseline
improved-tir-solution
25
21
3
Different TIR formulation — didn't pay off
The progression from parthenos → SC+TIR scaffold → ZaynYu's DynamicTime → weighted voting is visible across my own submissions, not just across the public fork tree.
7. What Did and Did Not Matter
7.1 Worked
Treating the sampler as an uncertain voter, not an oracle.
[Self-consistency](https://arxiv.org/abs/2203.11171)
is old news, but on AIMO3 it is not optional: single-run accuracy is much noisier than multi-run agreement. This is the single biggest delta above parthenos's baseline.
Persistent Jupyter kernels.
One kernel per thread, created once at notebook startup, reused across all 50 problems. Import overhead for each
sympy
/
numpy
/
mpmath
call disappears.
BlockingKernelClient
+ a per-thread port allocator avoids collisions that bite you as soon as you try to scale past K=2.
Harmony's
ReasoningEffort.HIGH
+
min_p=0.02
.
The model's native training format outperforms any hand-crafted prompt I tried, including "step by step" and "think carefully" variants. This is also what techtech69 reports.
OS-page-cache weight preloading.
16 threads reading every weight file in 1 GB chunks before
vllm
starts saves ~270 s of startup. Every second saved here is a second available to the hardest problems.
In-stream
\boxed{…}
scanning.
Once a valid integer appears in the stream, close the connection. Don't wait for the model to finish.
Weighted voting (my B).
Small but measurable lift on the upper tail of the score distribution. On average maybe +1, on the best submission +2.
7.2 Didn't work, or not for me
Prompt engineering beyond "rigorous competitor, never guess, box your answer".
Every longer system prompt I tried either did nothing or slightly hurt.
Temperature below 1.0.
Distribution collapses, loops, fewer solution paths. Confirmed by techtech69's −8 result at 0.6.
Pushing K past 8.
KV cache is the binding constraint on 1×H100 with
gpt-oss-120b
. 16 and 24 concurrent attempts are preempted by vLLM and score catastrophically worse. techtech69 reports −22 at K=24; I observed the same pattern at smaller scale.
Low
base_problem_timeout
with higher K.
100 s is not enough tokens to finish a hard problem. You lose more to truncated outputs than you gain from additional samples.
Fine-grained entropy-based trajectory scoring.
daniil1423's ETS scoring is clever but in my hands it did not clearly dominate simple weighted majority. Others (Potrhino, techtech69) report stronger results with entropy; this is a "methodology ≠ my hands" disclaimer rather than a claim about the method.
8. Limitations and Things I Got Wrong
I did not run controlled ablations.
Each row in §6 is one submission or a handful of identical resubmissions. To claim "my weighted voting beats raw majority by X points", I would need to run both variants against the same problem set under the same scheduler noise, many times. I have not. The noise floor (σ ≈ 1.6) is wide enough that the 42-peak could plausibly come from a lucky draw on the ZaynYu base alone.
I did not verify weight-coefficient sensitivity.
+0.5 / +0.25 / −0.3
with a 0.5 error-rate threshold is just the first thing I tried. I don't know whether
+0.3 / +0.15 / −0.2
would be better, worse, or identical within the noise.
I piggybacked heavily.
Contribution B is
small
on top of ZaynYu and daniil1423. Without their infrastructure — DynamicTime, AIMO3Solver, the kernel pool with
queue.Queue
, the refined system prompts — the weighted-vote rule has nothing to rest on. The right way to read this writeup is "I added two small bricks to a wall that other people built most of", not "I solved AIMO3 my way".
9. What I'd Do Differently
Record every submission's per-problem answers.
Without the per-problem breakdown, I can't tell which of the 50 problems my weighted vote actually flipped versus which were noise. Next time I'd log every
(problem_id, attempt_i, answer, python_calls, python_errors, token_len)
tuple to a file that's in the submission output, so I can do real ablation offline.
Try Qwen3-Next-80B-A3B-Thinking with a 262 k context.
gpt-oss-120b
caps out at ~43–44 on this hardware because its per-attempt accuracy on hard problems is ~5%. Both techtech69's and the host's pass@n data strongly suggest the gain from switching models is larger than the gain from any more voting tricks.
10. Reproducibility
Notebooks:
Scaffold (contribution A, open source):
[canqiang/self-consistency-strategy](https://www.kaggle.com/code/canqiang/self-consistency-strategy)
(V12, 170 upvotes).
42-score variant (contribution B): a fork of
[daniil1423/entropy-trajectory-scoring](https://www.kaggle.com/code/daniil1423/entropy-trajectory-scoring)
with the
_select_answer
method replaced by the 20-line rule in §4.3. Currently private; I'll open-source it shortly.
Key configuration for the 42-score submission:
class Config:
    model_path               = '/kaggle/input/gpt-oss-120b/transformers/default/1'
    served_model_name        = 'gpt-oss'
    context_tokens           = 65536
    kv_cache_dtype           = 'fp8_e4m3'
    gpu_memory_utilization   = 0.96
    batch_size               = 256

    temperature              = 1.0
    top_p                    = 1.0
    min_p                    = 0.02
    seed                     = 42
    attempts                 = 8
    workers                  = 16
    early_stop               = 5
    turns                    = 128

    high_problem_timeout     = 900
    base_problem_timeout     = 300
    notebook_limit           = 17400
    sandbox_timeout          = 3
    jupyter_timeout          = 10
Hardware:
1×H100 85GB, Kaggle competition runtime, 5h wall clock.
Expected variance:
μ ≈ 38, σ ≈ 1.6 across identical-code submissions. A single run scoring in
[35, 42]
should be considered a successful reproduction.
11. Acknowledgements
Most of what is in this notebook is not mine. I want to name the people whose work I built on, in the order I inherited it:
Kawchar Husain
, for the original prompt structure that parthenos forked from.
[parthenos (nihilisticneuralnet)](https://www.kaggle.com/nihilisticneuralnet)
, for publishing
[aimo3-streaming-inference-reflexion-prompting](https://www.kaggle.com/code/nihilisticneuralnet/aimo3-streaming-inference-reflexion-prompting)
— the streaming-inference reflexion prompt scaffold that my SC+TIR notebook merges with shelterw's line.
[shelterw](https://www.kaggle.com/shelterw)
, for
[[Way to 30+/50] GPT-OSS - w/ python tool](https://www.kaggle.com/code/shelterw/way-to-30-50-gpt-oss-w-python-tool)
, the other upstream I merged into the SC notebook.
[ZaynYu](https://www.kaggle.com/zaynyu)
, for turning my SC+TIR scaffold into a class-structured solver with dynamic time budgeting and a proper sandbox pool — the infrastructure my weighted-voting fork runs on.
[daniil1423](https://www.kaggle.com/daniil1423)
, for the
[entropy-trajectory-scoring notebook](https://www.kaggle.com/code/daniil1423/entropy-trajectory-scoring)
that my 42-score fork directly modifies.
[techtech69](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/entropy-weighted-majority-voting-with-gpt-oss-120b)
and
[Potrhino](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/potrhino-strategy-for-aimo3-adaptive-test-time-op)
, whose AIMO3 writeups on entropy-weighted voting and DeepConf-inspired pruning shaped how I now think about the "voting as uncertainty estimator" framing in §7.
If you are one of these people and I have miscredited any part of your work, please tell me and I will correct this writeup.
12. References
Writeups / notebooks directly referenced above:
techtech69,
[Entropy-Weighted Majority Voting with gpt-oss-120b for AIMO3](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/entropy-weighted-majority-voting-with-gpt-oss-120b)
(85th place, Public LB 43). Kaggle writeup.
Potrhino,
[Adaptive Test-Time Optimization for gpt-oss-120b](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/potrhino-strategy-for-aimo3-adaptive-test-time-op)
(91st place, Public LB 43). Kaggle writeup.
parthenos (nihilisticneuralnet),
[AIMO3: streaming-inference reflexion prompting](https://www.kaggle.com/code/nihilisticneuralnet/aimo3-streaming-inference-reflexion-prompting)
(V1).
ShelterW,
[[Way to 30+/50] GPT-OSS — w/ python tool](https://www.kaggle.com/code/shelterw/way-to-30-50-gpt-oss-w-python-tool)
.
canqiang (me),
[Self-Consistency Strategy](https://www.kaggle.com/code/canqiang/self-consistency-strategy)
(V12).
ZaynYu,
[[40/50] GPT-OSS-120B TIR+DynamicTime+KernelPool](https://www.kaggle.com/code/zaynyu/40-50-gpt-oss-120b-tir-dynamictime-kernelpool)
(V26).
daniil1423,
[entropy-trajectory-scoring](https://www.kaggle.com/code/daniil1423/entropy-trajectory-scoring)
.
External methodology references:
Wang et al.,
[Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://arxiv.org/abs/2203.11171)
, arXiv:2203.11171.
Wei et al.,
[Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://arxiv.org/abs/2201.11903)
, arXiv:2201.11903.
Lin & Xu,
[Understanding Tool-Integrated Reasoning](https://arxiv.org/abs/2508.19201)
, arXiv:2508.19201.
Fu et al.,
[Deep Think with Confidence](https://arxiv.org/abs/2508.15260)
, arXiv:2508.15260.
OpenAI,
[gpt-oss-120b & gpt-oss-20b Model Card](https://arxiv.org/abs/2508.10925)
, arXiv:2508.10925.
Kwon et al.,
[vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention](https://arxiv.org/abs/2309.06180)
, SOSP 2023.