![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F9785820%2F9304c0532d2b7f137a85242127ad8c89%2FChatGPT%20Image%2018%20Nis%202026%2014_44_24.png?generation=1776632382239680&alt=media\")
This write-up presents a stability-focused agent architecture developed for the third AIMO Progress Prize, designed to make final answer selection more reliable under strict compute constraints. Instead of leaning too heavily on raw majority voting, token-level confidence, or other brittle heuristics, the system uses a structured multi-stage workflow in which a small set of base solver attempts generate candidate solutions, summarization nodes turn those traces into more uniform and auditable representations, and tool-enabled judger nodes evaluate the mathematical content of those summaries to support a more consistent final decision. Speculative decoding was incorporated to help the overall workflow fit within the runtime budget. Overall, the approach prioritizes stability, verification, and controlled consensus, particularly on harder problems where direct confidence signals and simple voting tend to become unreliable.
\n
Many participants, myself included, started by analyzing the leading public notebooks[1,2]. While these notebooks provided a strong foundation, they contained some vulnerabilities that severely limited their stability on public leaderboard:
\nThe Dynamic Time Cap: To survive the strict 5-hour runtime limit, the public notebooks implement a time-budgeting mechanism. If the remaining overall budget runs low, it enforces a strict time cap (e.g., 300 seconds) per problem, which often leads to insufficient time to solve moderate-to-hard problems.
\nThe Confidence Trap: The baseline relies heavily on average token logprobs to measure model confidence, select the best answer, and break ties. However, logprob-based selection is highly vulnerable because there is no strong relation between logprobs and mathematical correctness (see the plots).
\nMy strategy was to build a system that could bypass these vulnerabilities and seek a more stable overall solution process, even if that did not necessarily guarantee a better score.
\nThe 300-second time cap introduced by the public notebook's budgeting logic was problematic because we do not know the order or the difficulty distribution of the private test set.
\nTo build some intuition about the effect of this cap, I ran Monte Carlo simulations modeling notebook behavior over 50 questions of varying difficulty. I approximated question difficulty and required compute time using simple heuristics guided by empirical observations. The results suggested that standard public-style approaches may leave many mathematically complex questions without enough compute to reach consensus before hitting the cap or running out of the 5-hour budget, especially in worst-case scenarios.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F9785820%2Fe6cf75fea1e27105ff2f410b32e2f5ec%2F__results___2_0.png?generation=1776629464559420&alt=media\")
To ensure every question received adequate time to explore mathematical pathways without hitting the budget cap, I implemented speculative decoding using the eagle3 draft model alongside gpt-oss-120b. Many thanks to @wenliangtlh for publicly sharing their speculative model [3].
\nThe Gains: This provided a massive ~40% speedup on base attempt runs and a ~30% speedup on Judger nodes.
\nThe Trade-offs: Speculative decoding requires roughly 2x the KV cache storage. Consequently, the maximum vLLM concurrency limit dropped from 6 down to 3 (with a 65k context limit).
\nThis hardware bottleneck forced me to be highly efficient. I strictly limited the base solver to 3 parallel runs and leaned heavily on the Judger nodes. In this architecture, the 3 base attempts act as an initial filter for easier questions, while the 5 parallel Judger nodes are only activated for harder problems requiring consensus.
\nNote on Concurrency: Even though I instantiate 5 Judger nodes, I let vLLM manage the concurrency internally. While a strict concurrency limit of 3 at a 65k context could theoretically cause a bottleneck, in practice, not all Judger runs consume the full 65k token limit simultaneously. The asynchronous scheduling handles the queueing efficiently.
\nWith speculative decoding enabled and the 3-Base/5-Judger node architecture, the notebooks consistently completed within the 5 hour submission limit. I experimented with different configurations (varying parallel runs and early consensus stops), but the 3-Base / 5-Judger setup with no early stop provided a decently stable score and became the basis for my final submission.
\n3. The Logprob Illusion\nThe second vulnerability was using logprobs to select the correct answer. My analysis showed that logprob sums and averages are fundamentally flawed metrics for evaluating mathematical correctness, especially under tie-breaking situations.
\n\n ![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F9785820%2Fdcb33381d1b635c05db5940686e8790f%2FEkran%20Resmi%202026-04-07%2020.26.32.png?generation=1776631712673025&alt=media\")
\n ![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F9785820%2Faa97674281dc2eb99897919d7e0d1785%2Ftie.png?generation=1776631752172369&alt=media\")
\n
As the plots above demonstrate, there is a massive density overlap in the average token logprobs and attempt vote weights between correct and incorrect answers. In tie situations, logprobs have almost no discriminative power.
\nFurthermore, the pass@n plots shared by the hosts [4] suggest that even when the model successfully generates the correct answer, that answer is often not the majority vote on hard questions. This makes logprob aggregation especially weak in the settings that matter most, since summing logprobs is unlikely to recover the correct answer when the dominant traces are already wrong.
\nCrucially, the log probability of a final answer token is entirely conditioned on the reasoning trace that precedes it. So, intuitively, treating one answer as “better” simply because it has a higher logprob than an answer produced through a different sequence of reasoning is mathematically unsound.
\nTo replace logprob heuristics, I implemented a three-phase decision workflow.
\nThe system initiates 3 parallel base attempts (maximizing my concurrency limit). The base solver attempts are essentially the same as what is presented in [2], but with a few notable changes:
\nRemoved _ensure_last_print: The baseline forced a print() wrapper on the final line of code, which caused numerous Python syntax errors. I simply removed it.
\nThe \"42 Trap\": Because the system prompt uses \\boxed{42} as an example, the model often outputs \"42\" prematurely. If a 42 is detected, I let the generation complete rather than stopping early, ensuring the parser captures the actual final \\boxed{} notation.
\nFlexible Regex: I updated the regex to catch answers formatted as equations (e.g., \\boxed{a+b=10}) when they are presented at the end of the traces.
\nEarly Exit: If all 3 base attempts independently arrive at the exact same answer, the system immediately outputs that answer. If they disagree, the pipeline moves to the summarization phase.
\nInitially, I let the base solvers complete their generation and fed the final answer blocks directly to the evaluating nodes. The base outputs varied widely: some were just \\boxed{} answers, while others were long step-by-step essays. From the traces I reviewed, the evaluator showed a noticeable tendency to favor the longest explanation, even when it was wrong.
\nTo address this, I introduced a Summarizer Node. I modified the summarizer prompt to stress objectivity, including the instruction: “Be completely objective, do not whitewash the solution.” This helped make the base traces less stylistically uneven and reduced some of the artificial confidence carried into the next stage, resulting in cleaner, more standardized inputs for the judger nodes. I also tried lowering the temperature with a fixed seed to reduce tone variance further, but that did not produce any clear improvement. In future work, this could be improved with a dedicated summary-standardization step or by using a larger-context model that can process full traces directly.
\nPhase 3: The Judger Node & Smart Voting\nThe last step of the system is an ensemble of 5 Judger nodes. Early versions of the Judger functioned as simple multiple-choice selectors. This failed entirely if all base options were wrong and often caused the model to hang indefinitely.
\nI made two critical adjustments:
\nLoosened the Prompt: I stopped forcing the Judger to pick an existing answer. Instead, I asked it to evaluate the mathematical reasoning of the summaries and come up with a final answer.
\nTool Enablement: I also gave the Judger node access to the Jupyter sandbox.
\nThe \"Smart Voting\" Tie-Break Logic:\nThe system finalizes the answer using a dynamic hierarchy:
\nJudger votes have the highest priority in the final decision logic. Once 3 judger nodes converge on the same answer, the run is terminated early.
Total Votes (Combining Base + Judger votes, if Judgers tie).
Lowest Average Token Length (If votes are still tied(rare), shorter, more concise answers generally indicate less hallucination and are favored over long, rambling traces).
This part was extremely challenging since triggering the summarizer and judger nodes is somewhat arbitrary, depending entirely on whether the base attempts agree. To overcome this, I used execution traces saved from previous experiment runs on 100 randomly selected questions from the validation set presented in [5].
\nTo create a useful test set, I filtered these logs for instances where:
\nThere was no early consensus.
At least one correct answer was presented in the base attempts.
The system had ultimately predicted the wrong answer in the previous run.
Using this curated small subset, I hand-tuned the prompts to elicit behaviors such as varying levels of summarization detail and different degrees of judge involvement.
\n| Metric | \nExp 1 (Baseline) | \nExp 2 (Sum Temp 0.2) | \nExp 3 (Prompt 4) | \nExp 4 (Prompt 5) | \nExp 5 (Prompt 3) | \nExp 6 (Prompt 2) | \n
|---|---|---|---|---|---|---|
| Overall Success | \n6 / 7 (85.7%) | \n5 / 7 (71.4%) | \n7 / 7 (100%) | \n5 / 7 (71.4%) | \n6 / 7 (85.7%) | \n6 / 7 (85.7%) | \n
| Completed Judger Runs | \n21 | \n21 | \n21 | \n21 | \n21 | \n21 | \n
| Correct Judgers | \n18 | \n16 | \n16 | \n16 | \n15 | \n17 | \n
| Judger Accuracy | \n85.71% | \n76.19% | \n76.19% | \n76.19% | \n71.43% | \n80.95% | \n
To validate these offline results, I further tested the Experiment 1 and Experiment 3 prompts dynamically on a hard reference question (Question 10) using multiple replicas.
\n| Metric | \nExp 3 (Prompt 4) | \nExp 1 | \n
|---|---|---|
| Overall System Success | \n3 / 8 (37.5%) | \n5 / 8 (62.5%) | \n
| System Success When Correct Ans. was Present | \n3 / 4 (75.0%) | \n5 / 5 (100.0%) | \n
| Judger Win Rate (When Triggered) | \n1 / 2 (50.0%) | \n3 / 3 (100.0%) | \n
| Zero-Shot Recoveries | \n0 | \n2 | \n
Based on these results, I chose the Experiment 1 summarizer and judger prompts for my final submission. One interesting observation from this test was the judger’s ability to reason independently in some cases: even when all provided base-attempt summaries were incorrect, the judger node was still sometimes able to derive and assign the correct final answer on its own. Although this was not consistent enough to rely on as a primary mechanism, I share a sample run on reference question 10 below:
\n\n ![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F9785820%2Fe7d220a1fc7880d13c47b777d957802a%2FEkran%20Resmi%202026-04-19%2012.54.57.png?generation=1776635091945945&alt=media\")
\n
I tested several variants of the pipeline on a 50-question set that included the 10 host reference questions, a subset of 2025 IMO questions with integer answers (or answers that could be converted into integer form), and 2025 Turkish Mathematical Olympiad questions. I intentionally over-sampled reference question 10, evaluating it 8 times, since it was the hardest problem available and served as a useful stress test for both submission-time runtime estimation and judger behavior under pressure. I share some of the more notable experiments below:
\n| Experiment Name | \nEarly Stop Enabled (Base Attempts) | \nEarly Stop Config (Base Attempts) | \nJudger Early Stop | \nContext Length | \nLocal Score | \nLocal Completion Time | \nPublic LB | \n
|---|---|---|---|---|---|---|---|
| public-notebook 8 runs with early stop 4 | \nyes | \n4 | \nN/A | \n65k | \n45/50 | \n3h 53m | \n- | \n
| 4 base, 3 judger | \nno | \nN/A | \n2 | \n65k | \n46/50 | \n3h 29m | \n41 | \n
| 3 base, 3 judger | \nno | \nN/A | \n2 | \n65k | \n46/50 | \n3h 55m | \n40 | \n
| 8 base with early stop, 4 then 3 judger | \nyes | \n4 | \n2 | \n65k | \n45/50 | \n4h 32m | \n- | \n
| 3 base, 5 judger | \nno | \nN/A | \n3 | \n65k | \n47/50 | \n3h 48m | \n42* | \n
| 3 base, 5 judger (higher context limit) | \nno | \nN/A | \n3 | \n75k | \n49/50 | \n4h 13m | \n42 | \n
| 5 base, 5 judger (lower context limit) | \nno | \nN/A | \n3 | \n45k | \n42/50 | \n3h 36m | \n- | \n
| 5 base, 5 judger** (higher context limit) | \nno | \nN/A | \n3 | \n72k | \n45/50 | \n4h 42m | \n- | \n
| 3 base, 8 judger** | \nno | \nN/A | \n5 | \n65k | \n44/50 | \n4h 11m | \n- | \n
*Final Submission.
\n**These experiments were affected by the hard 900s time limit for particular questions. While my intuition suggests these would score higher, they were not suitable for final submission due to the 5-hour total budget limit.
\nKey Insight: Increasing the number of base attempts did not boost accuracy. My intuition is that by feeding more base attempts into the pipeline, we are simply generating more incorrect solution summaries. When the Judger is presented with too many conflicting, flawed solutions, it suffers from context overload and struggles to isolate the correct logic. Keeping the base attempts low (3) and relying on a higher number of empowered, tool-enabled Judger nodes (5) yields a much more stable system.
\nLowering the number of base attempts and relying more heavily on judger nodes appears promising, but the effective vLLM concurrency limit of 3 makes this difficult to scale within the 5 hour runtime budget.
More broadly, making robust estimates requires repeated experiments or a broader question set. Because compute was limited, I often had to rely on single-run evaluations together with manual trace inspection, which makes some of the conclusions less certain.
It was also genuinely difficult to construct a local evaluation set that reflected the style and difficulty of the actual competition questions. For future competitions, it would be very helpful to provide more reference questions that better match the true competition distribution.
Another limitation is that, while a 3-attempt base setup gave decent results, the hosts’ analysis suggests that model pass@3 performance is only around 39/50. In that sense, the base solvers remain the main weakness of the system. Simply scaling the number of base attempts did not improve performance in my experiments, although with more compute there may be ways to make this work through broader multi-stage or higher-level aggregation strategies.
With more compute, one natural next step would be to run multiple instances of the full pipeline in parallel and combine them through a meta-judger or broader majority-voting scheme. vLLM concurrency on a single H100 is a real bottleneck here, but slightly more VRAM, for example on an H200, could allow for significantly more base agents.
\nWhat did NOT work for me:
\nInitiating many runs and killing the unpromising ones (as decided by the LLM).
Feeding direct raw solutions instead of summaries to the Judgers.
Heuristic-based voting instead of Judger nodes (e.g., factoring in trace lengths, tool errors, or logprobs).
Collaboration between base solver nodes.
This architecture was designed to address some of the weaknesses of time-capped, logprob-based selection by combining speculative decoding with a Summarizer-Judger workflow. Under the constraints of the competition, this made it possible to build a more structured and, in many cases, more stable decision process than simple majority voting or confidence-based tie-breaking alone.
\nAt the same time, the system is still clearly limited by compute, evaluation quality, and the strength of the underlying base solver. Even so, I think the results suggest that structured multi-stage evaluation is a useful direction for improving the reliability of open-weight mathematical reasoning systems.
\nThanks again to the hosts for an excellent competition, and also to the authors of the public notebooks and datasets, whose contributions provided a very strong foundation for experimentation.
\nAndreas Bis. AIMO-3: GPT-OSS-120B with Tools. Kaggle Notebook.
\nhttps://www.kaggle.com/code/andreasbis/aimo-3-gpt-oss-120b-with-tools
NihilisticNeuralNet. 44/50 — Let Me Over Cook. Kaggle Notebook.
\nhttps://www.kaggle.com/code/nihilisticneuralnet/44-50-let-me-over-cook
Wenliang1990. gpt-oss-120b-eagle3-aimo3. Hugging Face.
\nhttps://huggingface.co/wenliang1990/gpt-oss-120b-eagle3-aimo3
AI Mathematical Olympiad – Progress Prize 3 Discussion #679559. Kaggle Discussion.
\nhttps://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/679559
AI Mathematical Olympiad – Progress Prize 3 Discussion #672507. Kaggle Discussion.
\nhttps://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/672507
Following up on the questions from @philipvonderlind, here are some additional questions to help us assess your submission:
\nLook forward to reading your responses on these.
", "votes": 1, "replies": [ { "id": 3458754, "author_name": "rıza temizel", "author_url": "", "post_date": "2026-05-16T15:20:37.873000", "content": "Thank you for the detailed questions and for taking the time to review my write-up closely. I answer each point below.
\n1.Token-length tie-break
\nThe token length tie-breaker is a very rare case in the voting hierarchy, because it is only used after the earlier tie-break rules fail. My intention was not to treat answer length as a strong correctness signal, but to use it as a final deterministic fallback that seemed slightly better than random selection when both the Judger votes and the base-solver contributions were still insufficient to break the tie.
\nI plotted the token lengths of correct and incorrect answers for a sample run below, the difference between the two distributions is small. And, it may not be claimed that token length is a statistically strong separator. However, the plot suggested that shorter traces may be marginally better than a random assigment as a final fallback.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9785820%2F65f0ff44c0329966cf23f6395312c1e6%2Fb95f61e8-6d63-4e4d-9938-0131d782f43c.png?generation=1778943905924501&alt=media\")
I additionally checked the outputs from the “removed restrictions” experiment that I mentioned in my answer to @philipvonderlind's questions, since it contained a useful collection of Judger runs. In that experiment, 5 parallel full pipelines were run on the hard reference question 10 across 9 completed repetitions, giving 45 independent pipeline attempts. Out of those 45 attempts, the Judger stage was triggered 30 times, but the token length tie-breaker activated only 3 times. In 2 of those 3 cases, the correct answer was not even present in the tied pool, so no tie-break rule could have recovered the correct answer. In the remaining case where the correct answer was present among the tied candidates, the token-length tie-breaker selected it correctly.
\nSo my conclusion is that the token-length rule should be viewed as a weak final fallback, not as a major component of the method. It is triggered rarely, and the evidence is not strong enough to claim that length is a reliable correctness signal. However, based on the limited cases I inspected, it appeared slightly preferable to random selection when all other voting mechanisms had already failed.
\n2.Tool-using Judgers
\nFor the first part of the question, I plotted the distribution of the number of Python tool calls for base-solver attempts and Judger nodes for a sample run. The box plot shows that Judger nodes use clearly fewer Python calls than base-solver attempts. However, they still appear to rely on tool access rather than being purely text-only judges, since they almost always make at least some Python calls.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9785820%2F3274f6417f820a851dae7bc6bba9b78d%2FEkran%20Resmi%202026-05-16%2016.14.28.png?generation=1778944397208438&alt=media\")
For the second part, my method for evaluating the Judger was to check cases where at least one correct answer was present among the candidate solutions, and then measure how often the Judger successfully selected that correct answer.
\nIn the tool-enabled Judger setup, there were 6 such cases across 100 sampled questions from reference 5. The tool-enabled Judger selected the correct answer in 5 out of those 6 cases.
\nIn the no-tool Judger setup, there were 5 such cases. Out of these, 2 Judger calls timed out, 1 selected the wrong answer, and only 2 selected the correct answer.
\nSo my overall conclusion was that giving the Judger access to the Jupyter sandbox made it both more efficient and more accurate.
\nThe Colab notebook for the tool-enabled Judger implementation is here:
\n\nThe Colab notebook for the no-tool Judger implementation is here:
\n\nBest regards.
", "votes": 0, "replies": [] } ] }, { "id": 3457715, "author_name": "Philip Vonderlind", "author_url": "", "post_date": "2026-05-14T08:17:24.800000", "content": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.
\nI have a few questions regarding your work:
\nEAGLE3: You propose using speculative decoding with an EAGLE3 model developed by another contestant. While increasing throughput seems beneficial in principle, your results show that it substantially reduces the number of parallel trajectories that vLLM can sample because of the increased KV-cache usage. Did you run any ablations to determine whether the throughput gains outweigh the loss in diversity from using a higher number of n generations?
Early Finishes and Synchronization: As I understand it, your multi-stage pipeline requires all stages to finish before moving on to the next stage. This seems like it could introduce synchronization inefficiencies: if one completion finishes early, compute may be wasted while waiting for the remaining completions to finish. Did you take any measures to mitigate this synchronization issue? If so, it would be helpful to include them in the submission.
Composition: Did you experiment with alternative pipeline architectures? For example, after the summarization and judge stage, did you try adding an iterative self-improvement or correction step before making the final judgment, so the model has an opportunity to revise its answer?
Variance: Since the public leaderboard is highly sensitive to variance, did you run each experiment multiple times, either on your local test set or, where possible, on the public leaderboard? I understand that the daily submission limit may have made repeated leaderboard submissions difficult. However, reporting variance or confidence intervals in the results table would make the results much easier to interpret and compare. This is especially relevant because other contestants have shown that many observed improvements fall within the variance band around the public notebook's mean.
Looking forward to your response :)
", "votes": 1, "replies": [ { "id": 3458747, "author_name": "rıza temizel", "author_url": "", "post_date": "2026-05-16T14:58:12.637000", "content": "Thank you for the detailed questions and for taking the time to review my write-up closely. I answer each point below.
\n1.EAGLE3:
\nMy initial observation was that logprob-based voting was not reliable enough for final answer selection. Based on that, I moved toward a Judger-based design. Judger calls add extra cost, and the public notebook setup already did not reliably allocate enough time for all questions, especially with unlucky question orders. Therefore, adding a Judger stage meant that some base solver attempts had to be reduced.
\nBefore using EAGLE3, in early experiments I removed the hard 300 second per-question cap introduced in the public notebook and tested mainly non-speculative configurations, including:
\n5 base solvers with early stop at 3, followed by a single Judger if early stopping was not reached.
8 parallel base solvers with early stop at 4, plus an additional early-stop rule if the first 3 completed attempts agreed to gain some time budget. If neither early-stop condition was reached, the solutions were sent to a single Judger.
These configurations aimed to preserve more base solver diversity with some time gain, but I had to cap the number of questions that could use the Judger. Even with the Judger cap reduced to 4–5 questions, these notebooks were not guaranteed to finish within the 5-hour runtime limit. Their public LB scores fluctuated between 36 and 41, with the lower scores usually corresponding to API logs showing runtime over 5 hours. I also did not observe a meaningful public LB difference between the smaller (5) and larger (8) base-solver configurations.
\nI then switched to EAGLE3 speculative decoding. This introduced its own trade-off: vLLM concurrency was reduced because of the increased KV-cache usage. However, the throughput gain made the Judger-based pipeline feasible without a hard cap on Judger usage. It also allowed me to use multiple Judger calls on harder questions, which made the leaderboard score more stable.
\nI tested several variants, as shown in the experiment variants table, and the local test results were generally correlated with the public LB: configurations that worked better locally also tended to work better on the leaderboard. One early speculative-decoding variant, not included in the table, used 5 base attempts with early stop at 3 and 3 parallel Judgers, aiming to preserve more baseline solution diversity. That version fluctuated between 38 and 41 on the public LB. In contrast, the final 3-base / 5-Judger configuration scored 42 on two repeated submissions, once with a 65k context limit and once with a 75k context limit.
\nOne interesting observation was that feeding more baseline solutions to the Judger seemed to reduce performance in my experiments. For example, the 5-base / 5-Judger configuration scored 45/50 on my local test set, which was worse than the 3-base / 5-Judger configuration at 47/50. My interpretation is that too many proposed wrong answers can make it harder for the Judger to select the correct one. So in my setup, diversity seemed to be hurting the performance rather than improving it.
\nBased on these experiments, my conclusion is that, for my Judger-heavy pipeline, the throughput gain from EAGLE3 outweighed the loss of base-solver diversity. Overall, according to the experiment variants table, all successfully completed variants outperformed the public notebook on my local test set.
\n2.Early Finishes and Synchronization
\nAlthough the base attempts, summarization calls, and Judger calls are individually parallelizable, the overall workflow is sequential at the stage level: solve, summarize and judge. This is because the Judger expects all solution summaries from the base attempts before it can compare them and make a final decision.
\nOne possible minor optimization would be to start summarizing each base attempt as soon as it finishes, instead of waiting for all base attempts to complete. However, I decided not to do this because each summarization call uses a large context and would add extra parallel concurrency while the remaining base attempts are still running. Also, summarization was not triggered for every question. I was expecting it will be needed for around 10–20 questions, and each summarization call mostly completed in under 20 seconds. I estimated that the total summarization time over all 50 questions was around 5 minutes, which is a very small portion of the full notebook runtime.
\nOverall, there was not much opportunity to reallocate idle compute because the main stages depend on each other.
\nOne possible improvement would be to allocate available compute to the next question, but this was not possible since evaluation API serves questions one at a time.
\n3.Composition
\nYes, I experimented with several alternative agent architectures. But I did not add an extra self-improvement step after the Summarizer-Judger stage in the final submission since finishing within the 5-hour competition limit one of the main priorities. The final 3-base / 5-Judger configuration already completed in a little more than 4.5 hours, so there was not enough safe room to add another correction or self-improvement stage after judging.
\nI tested three notable alternative architectures:
\nFirst, I tried cross-feeding answers between base attempts. In this setup, different base solver outputs were fed back to each other, and each node was asked to compare the other proposed answers against its own answer and then produce a final solution. I did not submit this version for two reasons. First, the second turn became slow because it started from a very large context containing the model’s original solution trace plus the other proposed solutions. On harder questions, this often reached the maximum context length during the comparison step. Second, the model often appeared to insist on its own original answer. My intuition is that the initial solution trace still dominated the context, making it harder for the model to select a different candidate even when another solution was better.
\nOn reference question 10, I tested this architecture with 10 repetitions and it solved only 3/10, mainly because of the context-length issue. I also tried it on samples from reference question 5; even when context length was not the bottleneck, it was slower than my primary approach. Therefore, I did not pursue this architecture further.
\nImplementation can be found on this colab notebook:
\n\nSecond, I tested removing the summarization step and feeding the final presented answers directly to the Judger. This scored 43/50 on my local test set, which was worse than the Summarizer-Judger configurations. Manual trace inspection suggested a clear issue: some base attempts only produced a boxed answer, while others produced long step-by-step explanations. As I mentioned in the write-up, the Judger seemed to have a strong tendency to select the longer and more detailed-looking explanation.
\nNotebook can be found here:
\n\nThird, I explored an observer-node idea, where an additional node would monitor the base attempts and kill unpromising ones early. The goal was to start with a larger number of base attempts but stop weak trajectories before they consumed too much compute and vLLM concurancy become bottleneck. This was a late idea, and I did not push it very far because implementing it reliably was difficult.
\nNotebook can be found here:
\n\nAfter the submission period, I also started a “removed restrictions” experiment as mentioned on the Write-Up Prize Information post. Unfortunately, I could not complete this experiment in time for the write-up deadline.
\nIn the write-up, I wrote: “With more compute, one natural next step would be to run multiple instances of the full pipeline in parallel and combine them through a meta-judger or broader majority-voting scheme.”
\nI implemented a majority-voting version of this idea using 5 parallel instances of the full 3-base / 5-Judger pipeline, and tested it on reference question 10 with 10 repetitions in Colab. Although I set a very large notebook time limit of 60,000 seconds, 1 of the 10 repetitions still did not complete. It is extremely inefficient, mainly due to vLLM concurrence limitations on a single H100, but among the 9 completed repetitions, the majority vote over 5 full pipelines solved \"reference question 10\" correctly 8 times.
\nThe Colab notebook for this experiment is here:
\n\n4.Variance
\nUnfortunately, I did not have enough compute to run each experiment multiple times. However, I submitted the final 3-base / 5-Judger topology twice, changing only the context length: once with 65k and once with 75k. Both scored 42 on the public LB. I selected the 65k version for the final submission because it had more timing safety, and it later scored 42.5 on the private LB. It also had the highest local test score among the configurations that could realistically complete within the runtime limit. This suggests that the configuration was relatively stable and accomplished what was intended in my work.
\nThe screenshot shows with 3-base / 5-Judger variants mentioned above, and both reached the same public score of 42.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F9785820%2Fa9626a5acb5cd5a8fa1b810bafb1f053%2FEkran%20Resmi%202026-05-16%2017.20.46.png?generation=1778943211463388&alt=media\")
Best regards.
", "votes": 0, "replies": [] } ] }, { "id": 3445568, "author_name": "Utkarsh Sharma", "author_url": "", "post_date": "2026-04-20T04:56:50.100000", "content": "@rzatemizel The logprob density overlap plots are easily the most important empirical contribution. The numbers at the top make it concrete: correct answers have mean entropy 0.7053, wrong answers have 0.7503 - a gap of 0.045 on distributions that almost completely overlap. The right panel showing vote-weight overlap in tie-breaking situations is even more damning. This is a direct challenge to every entropy-based voting scheme in the competition, including mine.
\nOne question about the scope of this finding: your analysis uses mean token entropy across the full reasoning trace. Some approaches (including ours) use position-weighted entropy with exponential decay, so tokens near the final \\boxed{} count far more than tokens during early exploration. The intuition is that a solution can be uncertain during exploration but should be confident at the conclusion. Have you checked whether position-weighted entropy near the final answer has better discriminative power than the mean entropy shown in your plots? It's possible that mean entropy is useless (as your data clearly shows) but conclusion-specific confidence still correlates with correctness.
Your 3-base / 5-judger architecture is elegant - the workflow diagram makes the decision logic very clear. The key tradeoff I see is the N=3 ceiling. From the host's pass@n charts you referenced, pass@3 for Model B ≈ 38, meaning on ~12 private-set problems, none of your 3 base attempts might produce the correct answer. On some of those 12, all 3 might agree on the same wrong answer, triggering your early exit and skipping the judger entirely. Did you measure how often 3/3 wrong-agreement occurs? That seems like the one failure mode your architecture cannot recover from, since the judger - impressive as it is - never gets invoked. Am I thinking this right?
The Monte Carlo simulations motivating the speculative decoding tradeoff are really well done. One follow-up: the simulations model a public-notebook-style scheduler. Did you run the same simulation on your own 3-base/5-judger time profile? The judger nodes add latency that the base-only Monte Carlo wouldn't capture, so the failure rates under unlucky ordering might be different for your actual system.
Hi @flamingice2801, thank you for the careful reading.
\nI did not check position-weighted entropy or other similar variants, because I lost confidence in logprobs fairly early. My feeling was that asking the model to select among, or improve, proposed solutions was a more manageable task than relying on token-level confidence, so I ended up focusing on the judger approach instead.
\nOn “some of those 12, all 3 might agree on the same wrong answer, triggering your early exit and skipping the judger entirely”: yes, you are completely right about that. I did observe this kind of situation on the dataset shared in reference 5. For some of those questions, I also checked the solutions with Gemini and GPT-5, and they agreed with gpt-oss on the same answer. That made me suspect that at least some of those questions may have been poorly formulated, or that some reference answers may be incorrect. I am also not sure that a configuration like “8 attempts with early stop at 4” would really avoid the same failure mode. If the model already collapses to the same wrong answer in 3 attempts, then some of the later runs will likely do the same.
\nA fairly common opposite failure mode was getting only one answer plus 2 NaNs, especially on reference question 10. I thought about triggering additional parallel runs in that case, but I did not implement it because it seemed too risky under the 5-hour notebook limit.
\nThe Monte Carlo simulation was mainly meant to quantify the effect of random question ordering in the public-notebook setting. I did not run the same simulation for my exact 3-base / 5-judger pipeline, because that pipeline was already finishing below the 5-hour limit.
\nBest regards.
", "votes": 0, "replies": [ { "id": 3448911, "author_name": "Utkarsh Sharma", "author_url": "", "post_date": "2026-04-27T02:09:12.370000", "content": "That's a fair and honest answer - and the point about 8 attempts with early_stop=4 potentially hitting the same failure mode is well taken. If the model collapses to the same wrong answer in 3/3 attempts, it'll likely do the same in 4/8 or 5/8, and our early_stop would fire on the wrong answer too.
\nYour observation about some reference answers potentially being incorrect is consistent with what we found in our offline evaluation - when we tried to grade our contested resolver notebooks against a hard50 variant, we found problems where the model's \"wrong\" answer was actually more defensible than the listed ground truth.
\nThe practical wisdom of \"if it finishes within 5 hours, don't run a Monte Carlo simulation to prove it\" is honestly the right call for a competition setting.
", "votes": 0, "replies": [] } ] } ] } ], "comment_count": 7, "messages": [ { "id": 3445472, "postDate": "2026-04-19T21:00:40.103Z", "content": "This write-up presents a stability-focused agent architecture developed for the third AIMO Progress Prize, designed to make final answer selection more reliable under strict compute constraints. Instead of leaning too heavily on raw majority voting, token-level confidence, or other brittle heuristics, the system uses a structured multi-stage workflow in which a small set of base solver attempts generate candidate solutions, summarization nodes turn those traces into more uniform and auditable representations, and tool-enabled judger nodes evaluate the mathematical content of those summaries to support a more consistent final decision. Speculative decoding was incorporated to help the overall workflow fit within the runtime budget. Overall, the approach prioritizes stability, verification, and controlled consensus, particularly on harder problems where direct confidence signals and simple voting tend to become unreliable.
\n
Many participants, myself included, started by analyzing the leading public notebooks[1,2]. While these notebooks provided a strong foundation, they contained some vulnerabilities that severely limited their stability on public leaderboard:
\nThe Dynamic Time Cap: To survive the strict 5-hour runtime limit, the public notebooks implement a time-budgeting mechanism. If the remaining overall budget runs low, it enforces a strict time cap (e.g., 300 seconds) per problem, which often leads to insufficient time to solve moderate-to-hard problems.
\nThe Confidence Trap: The baseline relies heavily on average token logprobs to measure model confidence, select the best answer, and break ties. However, logprob-based selection is highly vulnerable because there is no strong relation between logprobs and mathematical correctness (see the plots).
\nMy strategy was to build a system that could bypass these vulnerabilities and seek a more stable overall solution process, even if that did not necessarily guarantee a better score.
\nThe 300-second time cap introduced by the public notebook's budgeting logic was problematic because we do not know the order or the difficulty distribution of the private test set.
\nTo build some intuition about the effect of this cap, I ran Monte Carlo simulations modeling notebook behavior over 50 questions of varying difficulty. I approximated question difficulty and required compute time using simple heuristics guided by empirical observations. The results suggested that standard public-style approaches may leave many mathematically complex questions without enough compute to reach consensus before hitting the cap or running out of the 5-hour budget, especially in worst-case scenarios.
\n
To ensure every question received adequate time to explore mathematical pathways without hitting the budget cap, I implemented speculative decoding using the eagle3 draft model alongside gpt-oss-120b. Many thanks to @wenliangtlh for publicly sharing their speculative model [3].
\nThe Gains: This provided a massive ~40% speedup on base attempt runs and a ~30% speedup on Judger nodes.
\nThe Trade-offs: Speculative decoding requires roughly 2x the KV cache storage. Consequently, the maximum vLLM concurrency limit dropped from 6 down to 3 (with a 65k context limit).
\nThis hardware bottleneck forced me to be highly efficient. I strictly limited the base solver to 3 parallel runs and leaned heavily on the Judger nodes. In this architecture, the 3 base attempts act as an initial filter for easier questions, while the 5 parallel Judger nodes are only activated for harder problems requiring consensus.
\nNote on Concurrency: Even though I instantiate 5 Judger nodes, I let vLLM manage the concurrency internally. While a strict concurrency limit of 3 at a 65k context could theoretically cause a bottleneck, in practice, not all Judger runs consume the full 65k token limit simultaneously. The asynchronous scheduling handles the queueing efficiently.
\nWith speculative decoding enabled and the 3-Base/5-Judger node architecture, the notebooks consistently completed within the 5 hour submission limit. I experimented with different configurations (varying parallel runs and early consensus stops), but the 3-Base / 5-Judger setup with no early stop provided a decently stable score and became the basis for my final submission.
\n3. The Logprob Illusion\nThe second vulnerability was using logprobs to select the correct answer. My analysis showed that logprob sums and averages are fundamentally flawed metrics for evaluating mathematical correctness, especially under tie-breaking situations.
\n\n \n \n
As the plots above demonstrate, there is a massive density overlap in the average token logprobs and attempt vote weights between correct and incorrect answers. In tie situations, logprobs have almost no discriminative power.
\nFurthermore, the pass@n plots shared by the hosts [4] suggest that even when the model successfully generates the correct answer, that answer is often not the majority vote on hard questions. This makes logprob aggregation especially weak in the settings that matter most, since summing logprobs is unlikely to recover the correct answer when the dominant traces are already wrong.
\nCrucially, the log probability of a final answer token is entirely conditioned on the reasoning trace that precedes it. So, intuitively, treating one answer as “better” simply because it has a higher logprob than an answer produced through a different sequence of reasoning is mathematically unsound.
\nTo replace logprob heuristics, I implemented a three-phase decision workflow.
\nThe system initiates 3 parallel base attempts (maximizing my concurrency limit). The base solver attempts are essentially the same as what is presented in [2], but with a few notable changes:
\nRemoved _ensure_last_print: The baseline forced a print() wrapper on the final line of code, which caused numerous Python syntax errors. I simply removed it.
\nThe \"42 Trap\": Because the system prompt uses \\boxed{42} as an example, the model often outputs \"42\" prematurely. If a 42 is detected, I let the generation complete rather than stopping early, ensuring the parser captures the actual final \\boxed{} notation.
\nFlexible Regex: I updated the regex to catch answers formatted as equations (e.g., \\boxed{a+b=10}) when they are presented at the end of the traces.
\nEarly Exit: If all 3 base attempts independently arrive at the exact same answer, the system immediately outputs that answer. If they disagree, the pipeline moves to the summarization phase.
\nInitially, I let the base solvers complete their generation and fed the final answer blocks directly to the evaluating nodes. The base outputs varied widely: some were just \\boxed{} answers, while others were long step-by-step essays. From the traces I reviewed, the evaluator showed a noticeable tendency to favor the longest explanation, even when it was wrong.
\nTo address this, I introduced a Summarizer Node. I modified the summarizer prompt to stress objectivity, including the instruction: “Be completely objective, do not whitewash the solution.” This helped make the base traces less stylistically uneven and reduced some of the artificial confidence carried into the next stage, resulting in cleaner, more standardized inputs for the judger nodes. I also tried lowering the temperature with a fixed seed to reduce tone variance further, but that did not produce any clear improvement. In future work, this could be improved with a dedicated summary-standardization step or by using a larger-context model that can process full traces directly.
\nPhase 3: The Judger Node & Smart Voting\nThe last step of the system is an ensemble of 5 Judger nodes. Early versions of the Judger functioned as simple multiple-choice selectors. This failed entirely if all base options were wrong and often caused the model to hang indefinitely.
\nI made two critical adjustments:
\nLoosened the Prompt: I stopped forcing the Judger to pick an existing answer. Instead, I asked it to evaluate the mathematical reasoning of the summaries and come up with a final answer.
\nTool Enablement: I also gave the Judger node access to the Jupyter sandbox.
\nThe \"Smart Voting\" Tie-Break Logic:\nThe system finalizes the answer using a dynamic hierarchy:
\nJudger votes have the highest priority in the final decision logic. Once 3 judger nodes converge on the same answer, the run is terminated early.
Total Votes (Combining Base + Judger votes, if Judgers tie).
Lowest Average Token Length (If votes are still tied(rare), shorter, more concise answers generally indicate less hallucination and are favored over long, rambling traces).
This part was extremely challenging since triggering the summarizer and judger nodes is somewhat arbitrary, depending entirely on whether the base attempts agree. To overcome this, I used execution traces saved from previous experiment runs on 100 randomly selected questions from the validation set presented in [5].
\nTo create a useful test set, I filtered these logs for instances where:
\nThere was no early consensus.
At least one correct answer was presented in the base attempts.
The system had ultimately predicted the wrong answer in the previous run.
Using this curated small subset, I hand-tuned the prompts to elicit behaviors such as varying levels of summarization detail and different degrees of judge involvement.
\n| Metric | \nExp 1 (Baseline) | \nExp 2 (Sum Temp 0.2) | \nExp 3 (Prompt 4) | \nExp 4 (Prompt 5) | \nExp 5 (Prompt 3) | \nExp 6 (Prompt 2) | \n
|---|---|---|---|---|---|---|
| Overall Success | \n6 / 7 (85.7%) | \n5 / 7 (71.4%) | \n7 / 7 (100%) | \n5 / 7 (71.4%) | \n6 / 7 (85.7%) | \n6 / 7 (85.7%) | \n
| Completed Judger Runs | \n21 | \n21 | \n21 | \n21 | \n21 | \n21 | \n
| Correct Judgers | \n18 | \n16 | \n16 | \n16 | \n15 | \n17 | \n
| Judger Accuracy | \n85.71% | \n76.19% | \n76.19% | \n76.19% | \n71.43% | \n80.95% | \n
To validate these offline results, I further tested the Experiment 1 and Experiment 3 prompts dynamically on a hard reference question (Question 10) using multiple replicas.
\n| Metric | \nExp 3 (Prompt 4) | \nExp 1 | \n
|---|---|---|
| Overall System Success | \n3 / 8 (37.5%) | \n5 / 8 (62.5%) | \n
| System Success When Correct Ans. was Present | \n3 / 4 (75.0%) | \n5 / 5 (100.0%) | \n
| Judger Win Rate (When Triggered) | \n1 / 2 (50.0%) | \n3 / 3 (100.0%) | \n
| Zero-Shot Recoveries | \n0 | \n2 | \n
Based on these results, I chose the Experiment 1 summarizer and judger prompts for my final submission. One interesting observation from this test was the judger’s ability to reason independently in some cases: even when all provided base-attempt summaries were incorrect, the judger node was still sometimes able to derive and assign the correct final answer on its own. Although this was not consistent enough to rely on as a primary mechanism, I share a sample run on reference question 10 below:
\n\n \n
I tested several variants of the pipeline on a 50-question set that included the 10 host reference questions, a subset of 2025 IMO questions with integer answers (or answers that could be converted into integer form), and 2025 Turkish Mathematical Olympiad questions. I intentionally over-sampled reference question 10, evaluating it 8 times, since it was the hardest problem available and served as a useful stress test for both submission-time runtime estimation and judger behavior under pressure. I share some of the more notable experiments below:
\n| Experiment Name | \nEarly Stop Enabled (Base Attempts) | \nEarly Stop Config (Base Attempts) | \nJudger Early Stop | \nContext Length | \nLocal Score | \nLocal Completion Time | \nPublic LB | \n
|---|---|---|---|---|---|---|---|
| public-notebook 8 runs with early stop 4 | \nyes | \n4 | \nN/A | \n65k | \n45/50 | \n3h 53m | \n- | \n
| 4 base, 3 judger | \nno | \nN/A | \n2 | \n65k | \n46/50 | \n3h 29m | \n41 | \n
| 3 base, 3 judger | \nno | \nN/A | \n2 | \n65k | \n46/50 | \n3h 55m | \n40 | \n
| 8 base with early stop, 4 then 3 judger | \nyes | \n4 | \n2 | \n65k | \n45/50 | \n4h 32m | \n- | \n
| 3 base, 5 judger | \nno | \nN/A | \n3 | \n65k | \n47/50 | \n3h 48m | \n42* | \n
| 3 base, 5 judger (higher context limit) | \nno | \nN/A | \n3 | \n75k | \n49/50 | \n4h 13m | \n42 | \n
| 5 base, 5 judger (lower context limit) | \nno | \nN/A | \n3 | \n45k | \n42/50 | \n3h 36m | \n- | \n
| 5 base, 5 judger** (higher context limit) | \nno | \nN/A | \n3 | \n72k | \n45/50 | \n4h 42m | \n- | \n
| 3 base, 8 judger** | \nno | \nN/A | \n5 | \n65k | \n44/50 | \n4h 11m | \n- | \n
*Final Submission.
\n**These experiments were affected by the hard 900s time limit for particular questions. While my intuition suggests these would score higher, they were not suitable for final submission due to the 5-hour total budget limit.
\nKey Insight: Increasing the number of base attempts did not boost accuracy. My intuition is that by feeding more base attempts into the pipeline, we are simply generating more incorrect solution summaries. When the Judger is presented with too many conflicting, flawed solutions, it suffers from context overload and struggles to isolate the correct logic. Keeping the base attempts low (3) and relying on a higher number of empowered, tool-enabled Judger nodes (5) yields a much more stable system.
\nLowering the number of base attempts and relying more heavily on judger nodes appears promising, but the effective vLLM concurrency limit of 3 makes this difficult to scale within the 5 hour runtime budget.
More broadly, making robust estimates requires repeated experiments or a broader question set. Because compute was limited, I often had to rely on single-run evaluations together with manual trace inspection, which makes some of the conclusions less certain.
It was also genuinely difficult to construct a local evaluation set that reflected the style and difficulty of the actual competition questions. For future competitions, it would be very helpful to provide more reference questions that better match the true competition distribution.
Another limitation is that, while a 3-attempt base setup gave decent results, the hosts’ analysis suggests that model pass@3 performance is only around 39/50. In that sense, the base solvers remain the main weakness of the system. Simply scaling the number of base attempts did not improve performance in my experiments, although with more compute there may be ways to make this work through broader multi-stage or higher-level aggregation strategies.
With more compute, one natural next step would be to run multiple instances of the full pipeline in parallel and combine them through a meta-judger or broader majority-voting scheme. vLLM concurrency on a single H100 is a real bottleneck here, but slightly more VRAM, for example on an H200, could allow for significantly more base agents.
\nWhat did NOT work for me:
\nInitiating many runs and killing the unpromising ones (as decided by the LLM).
Feeding direct raw solutions instead of summaries to the Judgers.
Heuristic-based voting instead of Judger nodes (e.g., factoring in trace lengths, tool errors, or logprobs).
Collaboration between base solver nodes.
This architecture was designed to address some of the weaknesses of time-capped, logprob-based selection by combining speculative decoding with a Summarizer-Judger workflow. Under the constraints of the competition, this made it possible to build a more structured and, in many cases, more stable decision process than simple majority voting or confidence-based tie-breaking alone.
\nAt the same time, the system is still clearly limited by compute, evaluation quality, and the strength of the underlying base solver. Even so, I think the results suggest that structured multi-stage evaluation is a useful direction for improving the reliability of open-weight mathematical reasoning systems.
\nThanks again to the hosts for an excellent competition, and also to the authors of the public notebooks and datasets, whose contributions provided a very strong foundation for experimentation.
\nAndreas Bis. AIMO-3: GPT-OSS-120B with Tools. Kaggle Notebook.
\nhttps://www.kaggle.com/code/andreasbis/aimo-3-gpt-oss-120b-with-tools
NihilisticNeuralNet. 44/50 — Let Me Over Cook. Kaggle Notebook.
\nhttps://www.kaggle.com/code/nihilisticneuralnet/44-50-let-me-over-cook
Wenliang1990. gpt-oss-120b-eagle3-aimo3. Hugging Face.
\nhttps://huggingface.co/wenliang1990/gpt-oss-120b-eagle3-aimo3
AI Mathematical Olympiad – Progress Prize 3 Discussion #679559. Kaggle Discussion.
\nhttps://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/679559
AI Mathematical Olympiad – Progress Prize 3 Discussion #672507. Kaggle Discussion.
\nhttps://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/672507
\n \n \n
\n \n
Following up on the questions from @philipvonderlind, here are some additional questions to help us assess your submission:
\nLook forward to reading your responses on these.
", "rawMarkdown": "Following up on the questions from @philipvonderlind, here are some additional questions to help us assess your submission:\n\n1. **Token-length tie-break:** Your Smart Voting tie-break hierarchy uses lowest average token length as the third-level tie-breaker, with the rationale that \"shorter, more concise answers generally indicate less hallucination\". Do you have data supporting this on your dev runs, e.g. when the token-length tie-breaker fires, do shorter-trace answers actually win against longer-trace ones more often than chance? How often does this tiebreak get used in the runs you've carried out?\n2. **Tool-using Judgers:** You gave Judger nodes access to the Jupyter sandbox. From inspecting traces, roughly how often do Judgers actually invoke Python (vs reasoning purely from the summarised candidates), and are tool-using Judger decisions noticeably more reliable than non-tool ones?\n\nLook forward to reading your responses on these.", "votes": 1, "replies": [ { "id": 3458754, "postDate": "2026-05-16T15:20:37.873Z", "content": "Thank you for the detailed questions and for taking the time to review my write-up closely. I answer each point below.
\n1.Token-length tie-break
\nThe token length tie-breaker is a very rare case in the voting hierarchy, because it is only used after the earlier tie-break rules fail. My intention was not to treat answer length as a strong correctness signal, but to use it as a final deterministic fallback that seemed slightly better than random selection when both the Judger votes and the base-solver contributions were still insufficient to break the tie.
\nI plotted the token lengths of correct and incorrect answers for a sample run below, the difference between the two distributions is small. And, it may not be claimed that token length is a statistically strong separator. However, the plot suggested that shorter traces may be marginally better than a random assigment as a final fallback.
\n
I additionally checked the outputs from the “removed restrictions” experiment that I mentioned in my answer to @philipvonderlind's questions, since it contained a useful collection of Judger runs. In that experiment, 5 parallel full pipelines were run on the hard reference question 10 across 9 completed repetitions, giving 45 independent pipeline attempts. Out of those 45 attempts, the Judger stage was triggered 30 times, but the token length tie-breaker activated only 3 times. In 2 of those 3 cases, the correct answer was not even present in the tied pool, so no tie-break rule could have recovered the correct answer. In the remaining case where the correct answer was present among the tied candidates, the token-length tie-breaker selected it correctly.
\nSo my conclusion is that the token-length rule should be viewed as a weak final fallback, not as a major component of the method. It is triggered rarely, and the evidence is not strong enough to claim that length is a reliable correctness signal. However, based on the limited cases I inspected, it appeared slightly preferable to random selection when all other voting mechanisms had already failed.
\n2.Tool-using Judgers
\nFor the first part of the question, I plotted the distribution of the number of Python tool calls for base-solver attempts and Judger nodes for a sample run. The box plot shows that Judger nodes use clearly fewer Python calls than base-solver attempts. However, they still appear to rely on tool access rather than being purely text-only judges, since they almost always make at least some Python calls.
\n
For the second part, my method for evaluating the Judger was to check cases where at least one correct answer was present among the candidate solutions, and then measure how often the Judger successfully selected that correct answer.
\nIn the tool-enabled Judger setup, there were 6 such cases across 100 sampled questions from reference 5. The tool-enabled Judger selected the correct answer in 5 out of those 6 cases.
\nIn the no-tool Judger setup, there were 5 such cases. Out of these, 2 Judger calls timed out, 1 selected the wrong answer, and only 2 selected the correct answer.
\nSo my overall conclusion was that giving the Judger access to the Jupyter sandbox made it both more efficient and more accurate.
\nThe Colab notebook for the tool-enabled Judger implementation is here:
\n\nThe Colab notebook for the no-tool Judger implementation is here:
\n\nBest regards.
", "rawMarkdown": "Thank you for the detailed questions and for taking the time to review my write-up closely. I answer each point below.\n\n**1.Token-length tie-break**\n\nThe token length tie-breaker is a very rare case in the voting hierarchy, because it is only used after the earlier tie-break rules fail. My intention was not to treat answer length as a strong correctness signal, but to use it as a final deterministic fallback that seemed slightly better than random selection when both the Judger votes and the base-solver contributions were still insufficient to break the tie.\n\nI plotted the token lengths of correct and incorrect answers for a sample run below, the difference between the two distributions is small. And, it may not be claimed that token length is a statistically strong separator. However, the plot suggested that shorter traces may be marginally better than a random assigment as a final fallback.\n\n\n\nI additionally checked the outputs from the “removed restrictions” experiment that I mentioned in my answer to @philipvonderlind's questions, since it contained a useful collection of Judger runs. In that experiment, 5 parallel full pipelines were run on the hard reference question 10 across 9 completed repetitions, giving 45 independent pipeline attempts. Out of those 45 attempts, the Judger stage was triggered 30 times, but the token length tie-breaker activated only 3 times. In 2 of those 3 cases, the correct answer was not even present in the tied pool, so no tie-break rule could have recovered the correct answer. In the remaining case where the correct answer was present among the tied candidates, the token-length tie-breaker selected it correctly.\n\nSo my conclusion is that the token-length rule should be viewed as a weak final fallback, not as a major component of the method. It is triggered rarely, and the evidence is not strong enough to claim that length is a reliable correctness signal. However, based on the limited cases I inspected, it appeared slightly preferable to random selection when all other voting mechanisms had already failed.\n\n\n**2.Tool-using Judgers**\n\nFor the first part of the question, I plotted the distribution of the number of Python tool calls for base-solver attempts and Judger nodes for a sample run. The box plot shows that Judger nodes use clearly fewer Python calls than base-solver attempts. However, they still appear to rely on tool access rather than being purely text-only judges, since they almost always make at least some Python calls.\n\n\n\n\nFor the second part, my method for evaluating the Judger was to check cases where at least one correct answer was present among the candidate solutions, and then measure how often the Judger successfully selected that correct answer.\n\nIn the tool-enabled Judger setup, there were 6 such cases across 100 sampled questions from reference 5. The tool-enabled Judger selected the correct answer in 5 out of those 6 cases.\n\nIn the no-tool Judger setup, there were 5 such cases. Out of these, 2 Judger calls timed out, 1 selected the wrong answer, and only 2 selected the correct answer.\n\nSo my overall conclusion was that giving the Judger access to the Jupyter sandbox made it both more efficient and more accurate. \n\nThe Colab notebook for the tool-enabled Judger implementation is here:\n\n[Open the Colab notebook](https://colab.research.google.com/drive/1jQKyJ-M5z1gP2ZthvHpZ1P9v0mK73ZKj?usp=sharing)\n\nThe Colab notebook for the no-tool Judger implementation is here:\n\n[Open the Colab notebook](https://colab.research.google.com/drive/12wFS0xde8gFe51E3YOFGscR77NXmdM0U?usp=sharing)\n\nBest regards. " } ] }, { "id": 3457715, "postDate": "2026-05-14T08:17:24.800Z", "content": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.
\nI have a few questions regarding your work:
\nEAGLE3: You propose using speculative decoding with an EAGLE3 model developed by another contestant. While increasing throughput seems beneficial in principle, your results show that it substantially reduces the number of parallel trajectories that vLLM can sample because of the increased KV-cache usage. Did you run any ablations to determine whether the throughput gains outweigh the loss in diversity from using a higher number of n generations?
Early Finishes and Synchronization: As I understand it, your multi-stage pipeline requires all stages to finish before moving on to the next stage. This seems like it could introduce synchronization inefficiencies: if one completion finishes early, compute may be wasted while waiting for the remaining completions to finish. Did you take any measures to mitigate this synchronization issue? If so, it would be helpful to include them in the submission.
Composition: Did you experiment with alternative pipeline architectures? For example, after the summarization and judge stage, did you try adding an iterative self-improvement or correction step before making the final judgment, so the model has an opportunity to revise its answer?
Variance: Since the public leaderboard is highly sensitive to variance, did you run each experiment multiple times, either on your local test set or, where possible, on the public leaderboard? I understand that the daily submission limit may have made repeated leaderboard submissions difficult. However, reporting variance or confidence intervals in the results table would make the results much easier to interpret and compare. This is especially relevant because other contestants have shown that many observed improvements fall within the variance band around the public notebook's mean.
Looking forward to your response :)
", "rawMarkdown": "Hi, thank you for your submission to the Writeup Prize. @sambealing and I will post some questions below that help us in assessing your submission more closely, as described in the [updated discussion post](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/689703).\n\nI have a few questions regarding your work:\n\n1. **EAGLE3**: You propose using speculative decoding with an EAGLE3 model developed by another contestant. While increasing throughput seems beneficial in principle, your results show that it substantially reduces the number of parallel trajectories that vLLM can sample because of the increased KV-cache usage. Did you run any ablations to determine whether the throughput gains outweigh the loss in diversity from using a higher number of `n` generations?\n\n2. **Early Finishes and Synchronization**: As I understand it, your multi-stage pipeline requires all stages to finish before moving on to the next stage. This seems like it could introduce synchronization inefficiencies: if one completion finishes early, compute may be wasted while waiting for the remaining completions to finish. Did you take any measures to mitigate this synchronization issue? If so, it would be helpful to include them in the submission.\n\n3. **Composition**: Did you experiment with alternative pipeline architectures? For example, after the summarization and judge stage, did you try adding an iterative self-improvement or correction step before making the final judgment, so the model has an opportunity to revise its answer?\n\n4. **Variance**: Since the public leaderboard is highly sensitive to variance, did you run each experiment multiple times, either on your local test set or, where possible, on the public leaderboard? I understand that the daily submission limit may have made repeated leaderboard submissions difficult. However, reporting variance or confidence intervals in the results table would make the results much easier to interpret and compare. This is especially relevant because [other contestants](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/model-capability-dominates) have shown that many observed improvements fall within the variance band around the public notebook's mean.\n\nLooking forward to your response :)", "votes": 1, "replies": [ { "id": 3458747, "postDate": "2026-05-16T14:58:12.637Z", "content": "Thank you for the detailed questions and for taking the time to review my write-up closely. I answer each point below.
\n1.EAGLE3:
\nMy initial observation was that logprob-based voting was not reliable enough for final answer selection. Based on that, I moved toward a Judger-based design. Judger calls add extra cost, and the public notebook setup already did not reliably allocate enough time for all questions, especially with unlucky question orders. Therefore, adding a Judger stage meant that some base solver attempts had to be reduced.
\nBefore using EAGLE3, in early experiments I removed the hard 300 second per-question cap introduced in the public notebook and tested mainly non-speculative configurations, including:
\n5 base solvers with early stop at 3, followed by a single Judger if early stopping was not reached.
8 parallel base solvers with early stop at 4, plus an additional early-stop rule if the first 3 completed attempts agreed to gain some time budget. If neither early-stop condition was reached, the solutions were sent to a single Judger.
These configurations aimed to preserve more base solver diversity with some time gain, but I had to cap the number of questions that could use the Judger. Even with the Judger cap reduced to 4–5 questions, these notebooks were not guaranteed to finish within the 5-hour runtime limit. Their public LB scores fluctuated between 36 and 41, with the lower scores usually corresponding to API logs showing runtime over 5 hours. I also did not observe a meaningful public LB difference between the smaller (5) and larger (8) base-solver configurations.
\nI then switched to EAGLE3 speculative decoding. This introduced its own trade-off: vLLM concurrency was reduced because of the increased KV-cache usage. However, the throughput gain made the Judger-based pipeline feasible without a hard cap on Judger usage. It also allowed me to use multiple Judger calls on harder questions, which made the leaderboard score more stable.
\nI tested several variants, as shown in the experiment variants table, and the local test results were generally correlated with the public LB: configurations that worked better locally also tended to work better on the leaderboard. One early speculative-decoding variant, not included in the table, used 5 base attempts with early stop at 3 and 3 parallel Judgers, aiming to preserve more baseline solution diversity. That version fluctuated between 38 and 41 on the public LB. In contrast, the final 3-base / 5-Judger configuration scored 42 on two repeated submissions, once with a 65k context limit and once with a 75k context limit.
\nOne interesting observation was that feeding more baseline solutions to the Judger seemed to reduce performance in my experiments. For example, the 5-base / 5-Judger configuration scored 45/50 on my local test set, which was worse than the 3-base / 5-Judger configuration at 47/50. My interpretation is that too many proposed wrong answers can make it harder for the Judger to select the correct one. So in my setup, diversity seemed to be hurting the performance rather than improving it.
\nBased on these experiments, my conclusion is that, for my Judger-heavy pipeline, the throughput gain from EAGLE3 outweighed the loss of base-solver diversity. Overall, according to the experiment variants table, all successfully completed variants outperformed the public notebook on my local test set.
\n2.Early Finishes and Synchronization
\nAlthough the base attempts, summarization calls, and Judger calls are individually parallelizable, the overall workflow is sequential at the stage level: solve, summarize and judge. This is because the Judger expects all solution summaries from the base attempts before it can compare them and make a final decision.
\nOne possible minor optimization would be to start summarizing each base attempt as soon as it finishes, instead of waiting for all base attempts to complete. However, I decided not to do this because each summarization call uses a large context and would add extra parallel concurrency while the remaining base attempts are still running. Also, summarization was not triggered for every question. I was expecting it will be needed for around 10–20 questions, and each summarization call mostly completed in under 20 seconds. I estimated that the total summarization time over all 50 questions was around 5 minutes, which is a very small portion of the full notebook runtime.
\nOverall, there was not much opportunity to reallocate idle compute because the main stages depend on each other.
\nOne possible improvement would be to allocate available compute to the next question, but this was not possible since evaluation API serves questions one at a time.
\n3.Composition
\nYes, I experimented with several alternative agent architectures. But I did not add an extra self-improvement step after the Summarizer-Judger stage in the final submission since finishing within the 5-hour competition limit one of the main priorities. The final 3-base / 5-Judger configuration already completed in a little more than 4.5 hours, so there was not enough safe room to add another correction or self-improvement stage after judging.
\nI tested three notable alternative architectures:
\nFirst, I tried cross-feeding answers between base attempts. In this setup, different base solver outputs were fed back to each other, and each node was asked to compare the other proposed answers against its own answer and then produce a final solution. I did not submit this version for two reasons. First, the second turn became slow because it started from a very large context containing the model’s original solution trace plus the other proposed solutions. On harder questions, this often reached the maximum context length during the comparison step. Second, the model often appeared to insist on its own original answer. My intuition is that the initial solution trace still dominated the context, making it harder for the model to select a different candidate even when another solution was better.
\nOn reference question 10, I tested this architecture with 10 repetitions and it solved only 3/10, mainly because of the context-length issue. I also tried it on samples from reference question 5; even when context length was not the bottleneck, it was slower than my primary approach. Therefore, I did not pursue this architecture further.
\nImplementation can be found on this colab notebook:
\n\nSecond, I tested removing the summarization step and feeding the final presented answers directly to the Judger. This scored 43/50 on my local test set, which was worse than the Summarizer-Judger configurations. Manual trace inspection suggested a clear issue: some base attempts only produced a boxed answer, while others produced long step-by-step explanations. As I mentioned in the write-up, the Judger seemed to have a strong tendency to select the longer and more detailed-looking explanation.
\nNotebook can be found here:
\n\nThird, I explored an observer-node idea, where an additional node would monitor the base attempts and kill unpromising ones early. The goal was to start with a larger number of base attempts but stop weak trajectories before they consumed too much compute and vLLM concurancy become bottleneck. This was a late idea, and I did not push it very far because implementing it reliably was difficult.
\nNotebook can be found here:
\n\nAfter the submission period, I also started a “removed restrictions” experiment as mentioned on the Write-Up Prize Information post. Unfortunately, I could not complete this experiment in time for the write-up deadline.
\nIn the write-up, I wrote: “With more compute, one natural next step would be to run multiple instances of the full pipeline in parallel and combine them through a meta-judger or broader majority-voting scheme.”
\nI implemented a majority-voting version of this idea using 5 parallel instances of the full 3-base / 5-Judger pipeline, and tested it on reference question 10 with 10 repetitions in Colab. Although I set a very large notebook time limit of 60,000 seconds, 1 of the 10 repetitions still did not complete. It is extremely inefficient, mainly due to vLLM concurrence limitations on a single H100, but among the 9 completed repetitions, the majority vote over 5 full pipelines solved \"reference question 10\" correctly 8 times.
\nThe Colab notebook for this experiment is here:
\n\n4.Variance
\nUnfortunately, I did not have enough compute to run each experiment multiple times. However, I submitted the final 3-base / 5-Judger topology twice, changing only the context length: once with 65k and once with 75k. Both scored 42 on the public LB. I selected the 65k version for the final submission because it had more timing safety, and it later scored 42.5 on the private LB. It also had the highest local test score among the configurations that could realistically complete within the runtime limit. This suggests that the configuration was relatively stable and accomplished what was intended in my work.
\nThe screenshot shows with 3-base / 5-Judger variants mentioned above, and both reached the same public score of 42.
\n
Best regards.
", "rawMarkdown": "Thank you for the detailed questions and for taking the time to review my write-up closely. I answer each point below.\n\n**1.EAGLE3:**\n\nMy initial observation was that logprob-based voting was not reliable enough for final answer selection. Based on that, I moved toward a Judger-based design. Judger calls add extra cost, and the public notebook setup already did not reliably allocate enough time for all questions, especially with unlucky question orders. Therefore, adding a Judger stage meant that some base solver attempts had to be reduced.\n\nBefore using EAGLE3, in early experiments I removed the hard 300 second per-question cap introduced in the public notebook and tested mainly non-speculative configurations, including:\n\n* 5 base solvers with early stop at 3, followed by a single Judger if early stopping was not reached.\n\n* 8 parallel base solvers with early stop at 4, plus an additional early-stop rule if the first 3 completed attempts agreed to gain some time budget. If neither early-stop condition was reached, the solutions were sent to a single Judger.\n\nThese configurations aimed to preserve more base solver diversity with some time gain, but I had to cap the number of questions that could use the Judger. Even with the Judger cap reduced to 4–5 questions, these notebooks were not guaranteed to finish within the 5-hour runtime limit. Their public LB scores fluctuated between 36 and 41, with the lower scores usually corresponding to API logs showing runtime over 5 hours. I also did not observe a meaningful public LB difference between the smaller (5) and larger (8) base-solver configurations.\n\nI then switched to EAGLE3 speculative decoding. This introduced its own trade-off: vLLM concurrency was reduced because of the increased KV-cache usage. However, the throughput gain made the Judger-based pipeline feasible without a hard cap on Judger usage. It also allowed me to use multiple Judger calls on harder questions, which made the leaderboard score more stable.\n\nI tested several variants, as shown in the experiment variants table, and the local test results were generally correlated with the public LB: configurations that worked better locally also tended to work better on the leaderboard. One early speculative-decoding variant, not included in the table, used 5 base attempts with early stop at 3 and 3 parallel Judgers, aiming to preserve more baseline solution diversity. That version fluctuated between 38 and 41 on the public LB. In contrast, the final 3-base / 5-Judger configuration scored 42 on two repeated submissions, once with a 65k context limit and once with a 75k context limit.\n\nOne interesting observation was that feeding more baseline solutions to the Judger seemed to reduce performance in my experiments. For example, the 5-base / 5-Judger configuration scored 45/50 on my local test set, which was worse than the 3-base / 5-Judger configuration at 47/50. My interpretation is that too many proposed wrong answers can make it harder for the Judger to select the correct one. So in my setup, diversity seemed to be hurting the performance rather than improving it.\n\nBased on these experiments, my conclusion is that, for my Judger-heavy pipeline, the throughput gain from EAGLE3 outweighed the loss of base-solver diversity. Overall, according to the experiment variants table, all successfully completed variants outperformed the public notebook on my local test set.\n\n**2.Early Finishes and Synchronization**\n\nAlthough the base attempts, summarization calls, and Judger calls are individually parallelizable, the overall workflow is sequential at the stage level: solve, summarize and judge. This is because the Judger expects all solution summaries from the base attempts before it can compare them and make a final decision.\n\nOne possible minor optimization would be to start summarizing each base attempt as soon as it finishes, instead of waiting for all base attempts to complete. However, I decided not to do this because each summarization call uses a large context and would add extra parallel concurrency while the remaining base attempts are still running. Also, summarization was not triggered for every question. I was expecting it will be needed for around 10–20 questions, and each summarization call mostly completed in under 20 seconds. I estimated that the total summarization time over all 50 questions was around 5 minutes, which is a very small portion of the full notebook runtime.\n\nOverall, there was not much opportunity to reallocate idle compute because the main stages depend on each other.\n\nOne possible improvement would be to allocate available compute to the next question, but this was not possible since evaluation API serves questions one at a time.\n\n**3.Composition**\n\nYes, I experimented with several alternative agent architectures. But I did not add an extra self-improvement step after the Summarizer-Judger stage in the final submission since finishing within the 5-hour competition limit one of the main priorities. The final 3-base / 5-Judger configuration already completed in a little more than 4.5 hours, so there was not enough safe room to add another correction or self-improvement stage after judging.\n\nI tested three notable alternative architectures:\n\n\nFirst, I tried cross-feeding answers between base attempts. In this setup, different base solver outputs were fed back to each other, and each node was asked to compare the other proposed answers against its own answer and then produce a final solution. I did not submit this version for two reasons. First, the second turn became slow because it started from a very large context containing the model’s original solution trace plus the other proposed solutions. On harder questions, this often reached the maximum context length during the comparison step. Second, the model often appeared to insist on its own original answer. My intuition is that the initial solution trace still dominated the context, making it harder for the model to select a different candidate even when another solution was better. \n\nOn reference question 10, I tested this architecture with 10 repetitions and it solved only 3/10, mainly because of the context-length issue. I also tried it on samples from reference question 5; even when context length was not the bottleneck, it was slower than my primary approach. Therefore, I did not pursue this architecture further.\n\nImplementation can be found on this colab notebook:\n\n[Open the Colab notebook](https://colab.research.google.com/drive/1vrlENPCINafwu6J3-v0riJrCIUT-7S4C?usp=sharing)\n\n\nSecond, I tested removing the summarization step and feeding the final presented answers directly to the Judger. This scored 43/50 on my local test set, which was worse than the Summarizer-Judger configurations. Manual trace inspection suggested a clear issue: some base attempts only produced a boxed answer, while others produced long step-by-step explanations. As I mentioned in the write-up, the Judger seemed to have a strong tendency to select the longer and more detailed-looking explanation. \n\nNotebook can be found here: \n\n[Open the Kaggle notebook](https://www.kaggle.com/code/rzatemizel/nosummarizer-direct-answer-333-judger-3-experiment)\n\n\nThird, I explored an observer-node idea, where an additional node would monitor the base attempts and kill unpromising ones early. The goal was to start with a larger number of base attempts but stop weak trajectories before they consumed too much compute and vLLM concurancy become bottleneck. This was a late idea, and I did not push it very far because implementing it reliably was difficult.\n\nNotebook can be found here: \n\n[Open the Colab notebook](https://colab.research.google.com/drive/1XgG82Xa82MGFdfMMfypUJblBLWGnv12c?usp=sharing)\n\n\nAfter the submission period, I also started a “removed restrictions” experiment as mentioned on the Write-Up Prize Information post. Unfortunately, I could not complete this experiment in time for the write-up deadline.\n\nIn the write-up, I wrote: “With more compute, one natural next step would be to run multiple instances of the full pipeline in parallel and combine them through a meta-judger or broader majority-voting scheme.”\n\nI implemented a majority-voting version of this idea using 5 parallel instances of the full 3-base / 5-Judger pipeline, and tested it on reference question 10 with 10 repetitions in Colab. Although I set a very large notebook time limit of 60,000 seconds, 1 of the 10 repetitions still did not complete. It is extremely inefficient, mainly due to vLLM concurrence limitations on a single H100, but among the 9 completed repetitions, the majority vote over 5 full pipelines solved \"reference question 10\" correctly 8 times.\n\nThe Colab notebook for this experiment is here:\n\n[Open the Colab notebook](https://colab.research.google.com/drive/1S2og3uSK8DGsmyu3FxSUJgfw8jJ8se5B?usp=sharing)\n\n\n**4.Variance**\n\nUnfortunately, I did not have enough compute to run each experiment multiple times. However, I submitted the final 3-base / 5-Judger topology twice, changing only the context length: once with 65k and once with 75k. Both scored 42 on the public LB. I selected the 65k version for the final submission because it had more timing safety, and it later scored 42.5 on the private LB. It also had the highest local test score among the configurations that could realistically complete within the runtime limit. This suggests that the configuration was relatively stable and accomplished what was intended in my work.\n\nThe screenshot shows with 3-base / 5-Judger variants mentioned above, and both reached the same public score of 42.\n\n\n\nBest regards." } ] }, { "id": 3445568, "postDate": "2026-04-20T04:56:50.100Z", "content": "@rzatemizel The logprob density overlap plots are easily the most important empirical contribution. The numbers at the top make it concrete: correct answers have mean entropy 0.7053, wrong answers have 0.7503 - a gap of 0.045 on distributions that almost completely overlap. The right panel showing vote-weight overlap in tie-breaking situations is even more damning. This is a direct challenge to every entropy-based voting scheme in the competition, including mine.
\nOne question about the scope of this finding: your analysis uses mean token entropy across the full reasoning trace. Some approaches (including ours) use position-weighted entropy with exponential decay, so tokens near the final \\boxed{} count far more than tokens during early exploration. The intuition is that a solution can be uncertain during exploration but should be confident at the conclusion. Have you checked whether position-weighted entropy near the final answer has better discriminative power than the mean entropy shown in your plots? It's possible that mean entropy is useless (as your data clearly shows) but conclusion-specific confidence still correlates with correctness.
Your 3-base / 5-judger architecture is elegant - the workflow diagram makes the decision logic very clear. The key tradeoff I see is the N=3 ceiling. From the host's pass@n charts you referenced, pass@3 for Model B ≈ 38, meaning on ~12 private-set problems, none of your 3 base attempts might produce the correct answer. On some of those 12, all 3 might agree on the same wrong answer, triggering your early exit and skipping the judger entirely. Did you measure how often 3/3 wrong-agreement occurs? That seems like the one failure mode your architecture cannot recover from, since the judger - impressive as it is - never gets invoked. Am I thinking this right?
The Monte Carlo simulations motivating the speculative decoding tradeoff are really well done. One follow-up: the simulations model a public-notebook-style scheduler. Did you run the same simulation on your own 3-base/5-judger time profile? The judger nodes add latency that the base-only Monte Carlo wouldn't capture, so the failure rates under unlucky ordering might be different for your actual system.
Hi @flamingice2801, thank you for the careful reading.
\nI did not check position-weighted entropy or other similar variants, because I lost confidence in logprobs fairly early. My feeling was that asking the model to select among, or improve, proposed solutions was a more manageable task than relying on token-level confidence, so I ended up focusing on the judger approach instead.
\nOn “some of those 12, all 3 might agree on the same wrong answer, triggering your early exit and skipping the judger entirely”: yes, you are completely right about that. I did observe this kind of situation on the dataset shared in reference 5. For some of those questions, I also checked the solutions with Gemini and GPT-5, and they agreed with gpt-oss on the same answer. That made me suspect that at least some of those questions may have been poorly formulated, or that some reference answers may be incorrect. I am also not sure that a configuration like “8 attempts with early stop at 4” would really avoid the same failure mode. If the model already collapses to the same wrong answer in 3 attempts, then some of the later runs will likely do the same.
\nA fairly common opposite failure mode was getting only one answer plus 2 NaNs, especially on reference question 10. I thought about triggering additional parallel runs in that case, but I did not implement it because it seemed too risky under the 5-hour notebook limit.
\nThe Monte Carlo simulation was mainly meant to quantify the effect of random question ordering in the public-notebook setting. I did not run the same simulation for my exact 3-base / 5-judger pipeline, because that pipeline was already finishing below the 5-hour limit.
\nBest regards.
", "rawMarkdown": "Hi @flamingice2801, thank you for the careful reading.\n\nI did not check position-weighted entropy or other similar variants, because I lost confidence in logprobs fairly early. My feeling was that asking the model to select among, or improve, proposed solutions was a more manageable task than relying on token-level confidence, so I ended up focusing on the judger approach instead.\n\nOn “some of those 12, all 3 might agree on the same wrong answer, triggering your early exit and skipping the judger entirely”: yes, you are completely right about that. I did observe this kind of situation on the dataset shared in reference 5. For some of those questions, I also checked the solutions with Gemini and GPT-5, and they agreed with gpt-oss on the same answer. That made me suspect that at least some of those questions may have been poorly formulated, or that some reference answers may be incorrect. I am also not sure that a configuration like “8 attempts with early stop at 4” would really avoid the same failure mode. If the model already collapses to the same wrong answer in 3 attempts, then some of the later runs will likely do the same. \n\nA fairly common opposite failure mode was getting only one answer plus 2 NaNs, especially on reference question 10. I thought about triggering additional parallel runs in that case, but I did not implement it because it seemed too risky under the 5-hour notebook limit.\n\nThe Monte Carlo simulation was mainly meant to quantify the effect of random question ordering in the public-notebook setting. I did not run the same simulation for my exact 3-base / 5-judger pipeline, because that pipeline was already finishing below the 5-hour limit.\n\nBest regards. ", "replies": [ { "id": 3448911, "postDate": "2026-04-27T02:09:12.370Z", "content": "That's a fair and honest answer - and the point about 8 attempts with early_stop=4 potentially hitting the same failure mode is well taken. If the model collapses to the same wrong answer in 3/3 attempts, it'll likely do the same in 4/8 or 5/8, and our early_stop would fire on the wrong answer too.
\nYour observation about some reference answers potentially being incorrect is consistent with what we found in our offline evaluation - when we tried to grade our contested resolver notebooks against a hard50 variant, we found problems where the model's \"wrong\" answer was actually more defensible than the listed ground truth.
\nThe practical wisdom of \"if it finishes within 5 hours, don't run a Monte Carlo simulation to prove it\" is honestly the right call for a competition setting.
", "rawMarkdown": "That's a fair and honest answer - and the point about 8 attempts with early_stop=4 potentially hitting the same failure mode is well taken. If the model collapses to the same wrong answer in 3/3 attempts, it'll likely do the same in 4/8 or 5/8, and our early_stop would fire on the wrong answer too.\n\nYour observation about some reference answers potentially being incorrect is consistent with what we found in our offline evaluation - when we tried to grade our contested resolver notebooks against a hard50 variant, we found problems where the model's \"wrong\" answer was actually more defensible than the listed ground truth.\n\nThe practical wisdom of \"if it finishes within 5 hours, don't run a Monte Carlo simulation to prove it\" is honestly the right call for a competition setting. " } ] } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T04:21:34Z", "competition_id": 118448, "competition_slug": "ai-mathematical-olympiad-progress-prize-3", "team_id": 15238969, "solution_url": "/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/fighting-against-variance-under-time-constraint" }