Most notebooks get gpt-oss-120b to work on the problem independently, and implement some voting mechanism to decide which answer to submit.
\nMy hypothesis is that I can beat the market if I enable multiple instances of gpt-oss-120b to collaborate with each other, by reading the results and findings of other solvers after submitting an answer.
\nThe interaction is accessible at aimo.huikang.dev/runs.\nThe visualization was built around Christmas last year. Just before publishing this writeup, I made it much easier to navigate.
\nLinks
\nI assume that people are familiar with the popular 44-score notebook. It has the following features.
\nI now explain the additional features of my submission.
\nWhen the KV cache is full, vLLM cannot generate any more tokens.\nIt picks one of the in-flight generations, pauses it, and frees up its KV cache. In the naive implementation, completions take turns to wait, and every time one exits the queue, the entire KV cache has to be prefilled with the O(n^2) computation. This becomes inefficient with more tool calls, as the prefix cache is frequently evicted.
\nI think it is more likely to get the answer if I have three solvers reaching the 60,000th token compared to eight solvers stuck at the 40,000th token. Therefore, I decided to terminate solvers when a threshold of the KV cache is reached.
\nI terminate the least-progressed solver first, with this priority:
\nIn the widely-forked 44-score notebook, submission is made once there are 4 solvers agreeing on the answer.
\nI think I can implement a logic better than that popular notebook. Given that there are only 8 attempts, if three solvers agree on an answer, it is unlikely that after some time four of the other solvers will agree on a different answer. Therefore, I think we could make a submission as long as 3 solvers agree on the answer.
\nHowever, when I look at the submission logs, I do not think it is true that we can just submit when two solvers agree on the answer.\nIt depends on whether the answer is unanimous. It also depends on how much time we have left.
\nMy submission entry has this answer-submission logic, where it will submit the answer if
\nIn addition, for the last question, I allow the agent to spend all the remaining time on the problem until all solvers submit the same answer.
\nThe main bet I am making is that collaboration between solvers will make it more likely to get the correct answer, as they can compare their answers with each other.
\nThis is an example of a solver being presented with other answers after submitting its own.
\nThe hope is that the solver will fix its answer based on the information provided by the other solvers.
\nassistant\nfinal\n\\boxed{41754}\nuser\nThese are the currently submitted answers:\nSolver 0 (you) is considering 41754\nSolver 2 is submitting 8687\nSolver 3 is submitting 8687\nSolver 5 is submitting 41754\n\nAnother solver has provided this solution:\n\nInsights: f(n)=min LCM(a,b,c) with a+b+c=n ⇒ f(n)/n = min_{d|n, d≥6} r(d), where r(d)=2/3−2/(3d) if d≡1(mod 3), r(d)=2/3−4/(3d) if d≡2(mod 3), r(d)=2/3 for d multiple of 3 (≥9). \n\nM=3^{2025!} ⇒ M≡1 (mod d) for d∤3. \n\nFor each c: find smallest divisor ≥6 of N=M+c in each residue class; compute r. \n\nResults: \ng(0)=2/3 (d=9); g(4M)=5·2/3=10/3; g(1848374)=16/25 (d=25); g(10162574)=30/47 (d=47); g(265710644)=64/97 (d=97); g(44636594)=110/167 (d=167). \n\nSum = 125561848/19033825 ⇒ (p+q) mod 99991 = **8687**. \n\nPitfalls: ignore factor (1+c/M) for c=4M (g(4M)≠2/3) and ensure d≥6, distinct‑divisor condition.\n\nScrutinize your solution, using other solutions as a reference.\nIf you spot any critical mistake in your solution, work towards figuring out the correct answer.\nPrioritize scrutinizing your solution so you can find any mistakes as soon as possible.\nThe summary from the other solvers is produced by gpt-oss-120b itself in a separate LLM call. When a solver proposes or updates an answer, I take a copy of its conversation tokens, append a prompt asking it to summarize its own current approach in under 250 characters (covering key insights, intermediate results, and pitfalls), and prefix the assistant turn with Insights:.\nThe resulting string is what other solvers see as \"Another solver has provided this solution\". The Insights: and Pitfalls: labels in the example above come from the model following this template.
If gpt-oss-120b arrived at the correct answer, presenting the summary of the wrong answer almost never causes gpt-oss-120b to adopt the wrong answer. If given 1 hour to arrive at a unanimous answer for Question 10, it seems that I can get 8687 80% of the time.
\nI had previous iterations where the information was injected before the first answer was submitted.\nHowever, this did not increase the probability of arriving at the correct answer in the same time budget.\nI found that interrupting the chain of thought causes gpt-oss-120b to waste tokens. This is because on every interruption, gpt-oss-120b restarts its chain of thought from scratch, rather than continuing from where it left off. This makes it difficult to inject new information in real time. Therefore, I ended up injecting the information only after the first answer is submitted, to ensure that the performance is not worse than a setup with no collaboration.
\nI allow solvers to retract their submission if they find their submitted answer is wrong.
\nThis makes unanimous agreement more likely, and hopefully the unanimously agreed answer is the correct answer.\nIf unanimous agreement is not achieved before time runs out, the retractions remove the wrong answers from participating in the voting process.
\nThe retraction decision itself is made by gpt-oss-120b in a separate LLM call. I append the solver's conversation tokens with a prompt asking it to reply with either I have confirmed that {answer} is wrong or I have yet to confirm that {answer} is wrong, and string-match the response.
This prompt-based classifier seems to be unreliable.\nTherefore, for an answer to be deleted, I require two consecutive retraction decisions.\nThe first decision marks the answer as pending retraction in its history. Only a second retraction decision (with no intervening confirmation) actually removes the answer from the voting pool.
\nAfter a solver submits an answer, the conversation continues and the solver often re-examines its work in the next iteration.\nThe first time this happens, it is very likely that the solver will spend real effort checking the answer.
\nHowever, if the solver keeps arriving at the same answer, it may simply restate its previous conclusion and stop spending effort on verification.\nThis behavior is undesirable.
\nTherefore, I implemented a backtracking logic.\nIf the solver submits the same answer two iterations in a row, I rewind its conversation by popping tokens off the end of its token stream until two assistant turns have been removed — effectively returning it to the state just before the iteration that first produced that answer. Generation then resumes from that earlier point.
\nThis makes it more likely that the solver will continue to spend effort checking the answer, hopefully figuring out that the answer it has submitted is wrong, and finding the correct answer instead.
\nMost of the changes were done by analyzing the errors and making changes to the code to address them.\nI do not have the GPU budget to run ablation tests.
\n8 submissions were made for the notebook version.\nThe scores, in chronological order, are 42, 40, 40, 36, 39, 40, 40, and 41.\nThe scores have a mean of 39.75.
\nInitially, I did this so that I would have a good chance at the Longest Leader Prize. I did not win the prize. It seems that what was required is a clean implementation of gpt-oss-120b, as seen from the notebooks that first scored between 41 and 44.
\nI thought I could achieve a significantly better score than the 44-score notebook, but I could not prove this, even though my changes made the model perform much better on Question 9 and Question 10. I realized that Questions 9 and 10 from the reference problems probably did not represent the problems in the leaderboard. The two questions involve multiple parts, which makes it easy for gpt-oss-120b to be correct in one part but wrong in another. Collaboration is especially useful, as I can allow gpt-oss-120b to compare notes and figure out who is correct. Perhaps most problems in the public leaderboard do not benefit from comparing notes.
", "comments": [ { "id": 3457793, "author_name": "Philip Vonderlind", "author_url": "", "post_date": "2026-05-14T10:43:05.110000", "content": "Adding to the comments by @sambealing below, I have a few additional questions:
\nSummary Injection: As I understand it, your approach generates summaries from completed traces and injects them into other generations that are still running. Did you test replacing this injection mechanism with an LLM-as-a-judge approach, similar to this submission? Would you say there is a benefit to avoiding distinct generation, summarization, and judgment stages?
Inference Bottleneck: You mention waiting for full traces before proceeding with the summarization step. Did you encounter cases where GPU cycles were wasted because one trace took significantly longer to finish than the others, causing the remaining traces or stages to wait?
Ablations and Time Management: Do you have ablations showing that answer retraction and backtracking improve performance on average in a competition-style evaluation? I imagine time budgeting becomes considerably more complex when each problem requires multiple sequential steps, especially given the synchronization issue mentioned above.
In some sense, my approach is already something like an LLM-as-a-judge (but a very biased one), the LLM will judge its current answer against the summaries of the other answers.
\nAnother feature that I have tried is to get a judge to decide whether a summary is worth injecting Another solver has presented the following finding ... Is this finding genuinely new and useful to you? Answer YES or NO, before I inject the summary. I decided to remove that feature because the performance of solvers running individually is getting higher ceiling scores than mine, and I want to be strictly better than that setup. I no longer need a classifier to decide whether a finding is useful, I will just inject the finding if the boxed answers are different.
My intention is to avoid distinct generation, summarization, and judgment stages. If I have multiple stages, I need to wait for every trace in each stage to complete or I interrupt, and tokens are wasted.
\nA principle of my design here is to keep the main traces uninterrupted, so that I can be more confident that my approach is better than a setup where the solvers solve independently.
\nIn my setup, all traces will continue to work until it passes the voting threshold or it times out. Traces will continue to generate even after the boxed answer is submitted, it will be given the status from the other traces and an instruction to check their answer. If a trace repeats the boxed answer multiple times, it will be backtracked to when it first repeated the boxed answer.
\nIf I assume all traces are doing productive work, I do not think I am wasting any GPU cycles.
\nThere is still a tradeoff being made though. If I terminate traces that submitted a boxed answer, I leave more memory and compute for other traces. Even so, I think it is faster to get a correct majority vote by fixing an early wrong answer than to be overpowered by other correct answers.
\nI observe that answer retraction actually retract wrong answers, and backtracking prevents the model from repeating the same submitted answer. I did not do ablation beyond these observations.
\nOn hindsight, I should have done ablations. Probably I should run 50 solve attempts on q9 and compare whether the win rate at least did not worsen, similar to what I did for the corpus prize submission. However, also on hindsight, improvements in q9 does not necessarily generalize to improvements in the leaderboard.
\nThe time budgeting did not get more complex, the decision to stop depends on the current vote distribution, once the vote passes, all in-flight generation will be stopped.
", "votes": 0, "replies": [] } ] }, { "id": 3457735, "author_name": "Sam Bealing", "author_url": "", "post_date": "2026-05-14T08:46:29.193000", "content": "Hi, thank you for your submission to the Writeup Prize. @philipvonderlind and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.
\nLook forward to reading your responses on these.
", "votes": 1, "replies": [ { "id": 3458858, "author_name": "Tong Hui Kang", "author_url": "", "post_date": "2026-05-16T19:29:26.650000", "content": "I only observe one case in q5 where the solver flipped from a correct answer (21818) to a wrong answer (62140).
\nSpecific cases where the information-sharing step flipped a solver's answer from wrong to right
\nlearned from correct
\n\ncorrected based on difference
\n\ncorrected on its own
\n\nlearned from correct
\n\ncorrected based on difference
\n\ncorrected on its own
\n| \n | q9 (160) | \nq10 (8687) | \n
|---|---|---|
| total traces | \n1411 | \n689 | \n
| did not box (no \\boxed) | \n708 (50.2%) | \n420 (61.0%) | \n
| boxed only once — correct | \n153 (10.8%) | \n15 (2.2%) | \n
| boxed only once — wrong | \n76 (5.4%) | \n91 (13.2%) | \n
| stayed correct (multi-box, all GT) | \n246 (17.4%) | \n32 (4.6%) | \n
| wrong → same wrong (one wrong value ≥2×) | \n136 (9.6%) | \n96 (13.9%) | \n
| wrong → different wrong (≥2 wrong values) | \n13 (0.9%) | \n22 (3.2%) | \n
| wrong → correct | \n79 (5.6%) | \n13 (1.9%) | \n
| learned from correct | \n58 (73.4% of 79) | \n7 (53.8% of 13) | \n
| corrected based on difference | \n13 (16.5% of 79) | \n6 (46.2% of 13) | \n
| corrected on its own | \n8 (10.1% of 79) | \n0 (0.0% of 13) | \n
| correct → wrong | \n0 (0.0%) | \n0 (0.0%) | \n
I think sharing code is a great idea, I should have tried that approach when sharing summaries.
\nAn issue I tried to resolve for my pipeline is that the model is not trying very hard to understand solutions that are different from theirs. I believe if models are provided the code and its output, the models would make a much better effort to understand the differences.
\nSharing code means that you need to ensure that the code works. The code shared needs to be runnable standalone, or otherwise the recipient will waste time debugging the code—and perhaps require a mechanism to test that the code could run on its own. However, even so, for the same tokens and time budget, I think that code is more effective than a pure-text sharing.
", "votes": 0, "replies": [] } ] }, { "id": 3443942, "author_name": "David Babu", "author_url": "", "post_date": "2026-04-17T02:13:19.387000", "content": "Interesting. I felt there were 2 kinds of hard problems. Ones like the Norwegian problem, which is pass@10/20 problem and another one where model quickly converges to wrong answer. I doubt the unsolvable ones are on the latter category where oss120B overfits on the problem and converges to wrong answer(by searching memory) in under a minute.
\nOn the point of interrupting the trace, what I found to be working with oss120B was to interrupt/inject with assistant role. User role interruption or non harmony encoding interruptions often broke the COT
", "votes": 3, "replies": [] }, { "id": 3444907, "author_name": "Durga Kumari", "author_url": "", "post_date": "2026-04-18T17:00:01.610000", "content": "Interesting approach, your observation that collaboration mainly helps on multi-part problems is especially insightful
", "votes": 0, "replies": [] }, { "id": 3444366, "author_name": "Simon Frieder", "author_url": "", "post_date": "2026-04-17T17:56:48.073000", "content": "Have you shared this for the writeup prize, or just for the community?
\nIn the former case, please leave a comment here linking to this writeup here https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/689703
", "votes": 0, "replies": [ { "id": 3444372, "author_name": "Tong Hui Kang", "author_url": "", "post_date": "2026-04-17T18:02:38.160000", "content": "Thanks for reaching out!
\nOn whether I am writing this for the writeup prize, it depends on my final score, before I invest in writing a full write-up that is worth your time to judge.
\nFor now, I am just happy to share my html file.
", "votes": 0, "replies": [ { "id": 3445890, "author_name": "Simon Frieder", "author_url": "", "post_date": "2026-04-20T16:27:22.800000", "content": "Fair enough. But please note that we learned that it will likely be later than the Write-Up Prize deadline that the private LB scores will come in: https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/693267
\nSo it looks like you may have to make a decision whether you want to go all in or not. In any case, I am sure whatever you come up with will be well worth read. You are a veteran AIMO participant by now :)
", "votes": 0, "replies": [] } ] } ] } ], "comment_count": 9, "messages": [ { "id": 3443903, "postDate": "2026-04-17T00:50:28.987Z", "content": "Most notebooks get gpt-oss-120b to work on the problem independently, and implement some voting mechanism to decide which answer to submit.
\nMy hypothesis is that I can beat the market if I enable multiple instances of gpt-oss-120b to collaborate with each other, by reading the results and findings of other solvers after submitting an answer.
\nThe interaction is accessible at aimo.huikang.dev/runs.\nThe visualization was built around Christmas last year. Just before publishing this writeup, I made it much easier to navigate.
\nLinks
\nI assume that people are familiar with the popular 44-score notebook. It has the following features.
\nI now explain the additional features of my submission.
\nWhen the KV cache is full, vLLM cannot generate any more tokens.\nIt picks one of the in-flight generations, pauses it, and frees up its KV cache. In the naive implementation, completions take turns to wait, and every time one exits the queue, the entire KV cache has to be prefilled with the O(n^2) computation. This becomes inefficient with more tool calls, as the prefix cache is frequently evicted.
\nI think it is more likely to get the answer if I have three solvers reaching the 60,000th token compared to eight solvers stuck at the 40,000th token. Therefore, I decided to terminate solvers when a threshold of the KV cache is reached.
\nI terminate the least-progressed solver first, with this priority:
\nIn the widely-forked 44-score notebook, submission is made once there are 4 solvers agreeing on the answer.
\nI think I can implement a logic better than that popular notebook. Given that there are only 8 attempts, if three solvers agree on an answer, it is unlikely that after some time four of the other solvers will agree on a different answer. Therefore, I think we could make a submission as long as 3 solvers agree on the answer.
\nHowever, when I look at the submission logs, I do not think it is true that we can just submit when two solvers agree on the answer.\nIt depends on whether the answer is unanimous. It also depends on how much time we have left.
\nMy submission entry has this answer-submission logic, where it will submit the answer if
\nIn addition, for the last question, I allow the agent to spend all the remaining time on the problem until all solvers submit the same answer.
\nThe main bet I am making is that collaboration between solvers will make it more likely to get the correct answer, as they can compare their answers with each other.
\nThis is an example of a solver being presented with other answers after submitting its own.
\nThe hope is that the solver will fix its answer based on the information provided by the other solvers.
\nassistant\nfinal\n\\boxed{41754}\nuser\nThese are the currently submitted answers:\nSolver 0 (you) is considering 41754\nSolver 2 is submitting 8687\nSolver 3 is submitting 8687\nSolver 5 is submitting 41754\n\nAnother solver has provided this solution:\n\nInsights: f(n)=min LCM(a,b,c) with a+b+c=n ⇒ f(n)/n = min_{d|n, d≥6} r(d), where r(d)=2/3−2/(3d) if d≡1(mod 3), r(d)=2/3−4/(3d) if d≡2(mod 3), r(d)=2/3 for d multiple of 3 (≥9). \n\nM=3^{2025!} ⇒ M≡1 (mod d) for d∤3. \n\nFor each c: find smallest divisor ≥6 of N=M+c in each residue class; compute r. \n\nResults: \ng(0)=2/3 (d=9); g(4M)=5·2/3=10/3; g(1848374)=16/25 (d=25); g(10162574)=30/47 (d=47); g(265710644)=64/97 (d=97); g(44636594)=110/167 (d=167). \n\nSum = 125561848/19033825 ⇒ (p+q) mod 99991 = **8687**. \n\nPitfalls: ignore factor (1+c/M) for c=4M (g(4M)≠2/3) and ensure d≥6, distinct‑divisor condition.\n\nScrutinize your solution, using other solutions as a reference.\nIf you spot any critical mistake in your solution, work towards figuring out the correct answer.\nPrioritize scrutinizing your solution so you can find any mistakes as soon as possible.\nThe summary from the other solvers is produced by gpt-oss-120b itself in a separate LLM call. When a solver proposes or updates an answer, I take a copy of its conversation tokens, append a prompt asking it to summarize its own current approach in under 250 characters (covering key insights, intermediate results, and pitfalls), and prefix the assistant turn with Insights:.\nThe resulting string is what other solvers see as \"Another solver has provided this solution\". The Insights: and Pitfalls: labels in the example above come from the model following this template.
If gpt-oss-120b arrived at the correct answer, presenting the summary of the wrong answer almost never causes gpt-oss-120b to adopt the wrong answer. If given 1 hour to arrive at a unanimous answer for Question 10, it seems that I can get 8687 80% of the time.
\nI had previous iterations where the information was injected before the first answer was submitted.\nHowever, this did not increase the probability of arriving at the correct answer in the same time budget.\nI found that interrupting the chain of thought causes gpt-oss-120b to waste tokens. This is because on every interruption, gpt-oss-120b restarts its chain of thought from scratch, rather than continuing from where it left off. This makes it difficult to inject new information in real time. Therefore, I ended up injecting the information only after the first answer is submitted, to ensure that the performance is not worse than a setup with no collaboration.
\nI allow solvers to retract their submission if they find their submitted answer is wrong.
\nThis makes unanimous agreement more likely, and hopefully the unanimously agreed answer is the correct answer.\nIf unanimous agreement is not achieved before time runs out, the retractions remove the wrong answers from participating in the voting process.
\nThe retraction decision itself is made by gpt-oss-120b in a separate LLM call. I append the solver's conversation tokens with a prompt asking it to reply with either I have confirmed that {answer} is wrong or I have yet to confirm that {answer} is wrong, and string-match the response.
This prompt-based classifier seems to be unreliable.\nTherefore, for an answer to be deleted, I require two consecutive retraction decisions.\nThe first decision marks the answer as pending retraction in its history. Only a second retraction decision (with no intervening confirmation) actually removes the answer from the voting pool.
\nAfter a solver submits an answer, the conversation continues and the solver often re-examines its work in the next iteration.\nThe first time this happens, it is very likely that the solver will spend real effort checking the answer.
\nHowever, if the solver keeps arriving at the same answer, it may simply restate its previous conclusion and stop spending effort on verification.\nThis behavior is undesirable.
\nTherefore, I implemented a backtracking logic.\nIf the solver submits the same answer two iterations in a row, I rewind its conversation by popping tokens off the end of its token stream until two assistant turns have been removed — effectively returning it to the state just before the iteration that first produced that answer. Generation then resumes from that earlier point.
\nThis makes it more likely that the solver will continue to spend effort checking the answer, hopefully figuring out that the answer it has submitted is wrong, and finding the correct answer instead.
\nMost of the changes were done by analyzing the errors and making changes to the code to address them.\nI do not have the GPU budget to run ablation tests.
\n8 submissions were made for the notebook version.\nThe scores, in chronological order, are 42, 40, 40, 36, 39, 40, 40, and 41.\nThe scores have a mean of 39.75.
\nInitially, I did this so that I would have a good chance at the Longest Leader Prize. I did not win the prize. It seems that what was required is a clean implementation of gpt-oss-120b, as seen from the notebooks that first scored between 41 and 44.
\nI thought I could achieve a significantly better score than the 44-score notebook, but I could not prove this, even though my changes made the model perform much better on Question 9 and Question 10. I realized that Questions 9 and 10 from the reference problems probably did not represent the problems in the leaderboard. The two questions involve multiple parts, which makes it easy for gpt-oss-120b to be correct in one part but wrong in another. Collaboration is especially useful, as I can allow gpt-oss-120b to compare notes and figure out who is correct. Perhaps most problems in the public leaderboard do not benefit from comparing notes.
", "rawMarkdown": "Most notebooks get gpt-oss-120b to work on the problem independently, and implement some voting mechanism to decide which answer to submit.\n\nMy hypothesis is that I can beat the market if I enable multiple instances of gpt-oss-120b to collaborate with each other, by reading the results and findings of other solvers after submitting an answer.\n\nThe interaction is accessible at [aimo.huikang.dev/runs](http://aimo.huikang.dev/runs.html?datetime=2026-02-18-05-21-35&problem=86e8e5©=4&solution=0).\nThe visualization was built around Christmas last year. Just before publishing this writeup, I made it much easier to navigate.\n\n---\n\nLinks\n\n- Comment in submission thread: https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/689703#3446458\n- Writeup: https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/attempts-to-get-gpt-oss-120b-to-collaborate\n- Notebook\n - Kaggle https://www.kaggle.com/code/huikang/streaming-inference-private?scriptVersionId=298346744\n - Github https://github.com/tonghuikang/aimo3/blob/master/notebook.py\n - Colab https://colab.research.google.com/drive/1rWFywCyNjDehHiYrsWjzdENA9o4QbpEh (copy might not run)\n- Model\n - Kaggle https://www.kaggle.com/models/huikang/gpt-oss-120b-aimo3/Transformers/160a/9 as described in [Corpus Prize Submission](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/672528)\n - Huggingface: https://huggingface.co/ExampleTestcase/gpt-oss-120b-aimo3\n\n---\n\nI assume that people are familiar with the popular [44-score notebook](https://www.kaggle.com/code/nihilisticneuralnet/44-50-let-me-over-cook). It has the following features.\n\n- Use of gpt-oss-120b\n- vLLM for inference\n- The tool calling harness with the Jupyter client\n- Early stopping once the majority vote is achieved\n\nI now explain the additional features of my submission.\n\n### Solver termination based on KV cache usage\n\nWhen the KV cache is full, vLLM cannot generate any more tokens.\nIt picks one of the in-flight generations, pauses it, and frees up its KV cache. In the naive implementation, completions take turns to wait, and every time one exits the queue, the entire KV cache has to be prefilled with the O(n^2) computation. This becomes inefficient with more tool calls, as the prefix cache is frequently evicted.\n\nI think it is more likely to get the answer if I have three solvers reaching the 60,000th token compared to eight solvers stuck at the 40,000th token. Therefore, I decided to terminate solvers when a threshold of the KV cache is reached.\n\nI terminate the least-progressed solver first, with this priority:\n- a solver that has not made any proposal yet\n- a solver with no answer or proposal\n- a solver with no answer\n- the solver with the fewest tool calls\n\n\n### Adaptive submission logic\n\nIn the widely-forked 44-score notebook, submission is made once there are 4 solvers agreeing on the answer.\n\nI think I can implement a logic better than that popular notebook. Given that there are only 8 attempts, if three solvers agree on an answer, it is unlikely that after some time four of the other solvers will agree on a different answer. Therefore, I think we could make a submission as long as 3 solvers agree on the answer.\n\nHowever, when I look at the submission logs, I do not think it is true that we can just submit when two solvers agree on the answer.\nIt depends on whether the answer is unanimous. It also depends on how much time we have left.\n\nMy submission entry has this answer-submission logic, where it will submit the answer if\n- two solvers have submitted an answer and they agree on the answer\n- there are three solvers remaining (due to the solver termination logic) and two solvers agree on the answer\n- there are three solvers remaining (due to the solver termination logic) and only one solver has a submission\n\nIn addition, for the last question, I allow the agent to spend all the remaining time on the problem until all solvers submit the same answer.\n\n\n### Information sharing between solvers\n\nThe main bet I am making is that collaboration between solvers will make it more likely to get the correct answer, as they can compare their answers with each other.\n\nThis is an example of a solver being [presented](http://localhost:4130/runs.html?datetime=2026-02-18-05-21-35&problem=86e8e5©=4&solution=0) with other answers after submitting its own.\n\nThe hope is that the solver will fix its answer based on the information provided by the other solvers.\n\n```\nassistant\nfinal\n\\boxed{41754}\nuser\nThese are the currently submitted answers:\nSolver 0 (you) is considering 41754\nSolver 2 is submitting 8687\nSolver 3 is submitting 8687\nSolver 5 is submitting 41754\n\nAnother solver has provided this solution:\n\nInsights: f(n)=min LCM(a,b,c) with a+b+c=n ⇒ f(n)/n = min_{d|n, d≥6} r(d), where r(d)=2/3−2/(3d) if d≡1(mod 3), r(d)=2/3−4/(3d) if d≡2(mod 3), r(d)=2/3 for d multiple of 3 (≥9). \n\nM=3^{2025!} ⇒ M≡1 (mod d) for d∤3. \n\nFor each c: find smallest divisor ≥6 of N=M+c in each residue class; compute r. \n\nResults: \ng(0)=2/3 (d=9); g(4M)=5·2/3=10/3; g(1848374)=16/25 (d=25); g(10162574)=30/47 (d=47); g(265710644)=64/97 (d=97); g(44636594)=110/167 (d=167). \n\nSum = 125561848/19033825 ⇒ (p+q) mod 99991 = **8687**. \n\nPitfalls: ignore factor (1+c/M) for c=4M (g(4M)≠2/3) and ensure d≥6, distinct‑divisor condition.\n\nScrutinize your solution, using other solutions as a reference.\nIf you spot any critical mistake in your solution, work towards figuring out the correct answer.\nPrioritize scrutinizing your solution so you can find any mistakes as soon as possible.\n```\n\nThe summary from the other solvers is produced by gpt-oss-120b itself in a separate LLM call. When a solver proposes or updates an answer, I take a copy of its conversation tokens, append a prompt asking it to summarize its own current approach in under 250 characters (covering key insights, intermediate results, and pitfalls), and prefix the assistant turn with `Insights:`.\nThe resulting string is what other solvers see as \"Another solver has provided this solution\". The `Insights:` and `Pitfalls:` labels in the example above come from the model following this template.\n\nIf gpt-oss-120b arrived at the correct answer, presenting the summary of the wrong answer almost never causes gpt-oss-120b to adopt the wrong answer. If given 1 hour to arrive at a unanimous answer for Question 10, it seems that I can get 8687 [80% of the time](https://www.kaggle.com/code/huikang/streaming-inference-private/output?scriptVersionId=293581258&select=stats.csv).\n\nI had previous iterations where the information was injected before the first answer was submitted.\nHowever, this did not increase the probability of arriving at the correct answer in the same time budget.\nI found that interrupting the chain of thought causes gpt-oss-120b to waste tokens. This is because on every interruption, gpt-oss-120b restarts its chain of thought from scratch, rather than continuing from where it left off. This makes it difficult to inject new information in real time. Therefore, I ended up injecting the information only after the first answer is submitted, to ensure that the performance is not worse than a setup with no collaboration.\n\n\n### Answer deletion logic\n\nI allow solvers to retract their submission if they find their submitted answer is wrong.\n\nThis makes unanimous agreement more likely, and hopefully the unanimously agreed answer is the correct answer.\nIf unanimous agreement is not achieved before time runs out, the retractions remove the wrong answers from participating in the voting process.\n\nThe retraction decision itself is made by gpt-oss-120b in a separate LLM call. I append the solver's conversation tokens with a prompt asking it to reply with either `I have confirmed that {answer} is wrong` or `I have yet to confirm that {answer} is wrong`, and string-match the response.\n\nThis prompt-based classifier seems to be unreliable.\nTherefore, for an answer to be deleted, I require two consecutive retraction decisions.\nThe first decision marks the answer as pending retraction in its history. Only a second retraction decision (with no intervening confirmation) actually removes the answer from the voting pool.\n\n\n### Backtracking logic\n\nAfter a solver submits an answer, the conversation continues and the solver often re-examines its work in the next iteration.\nThe first time this happens, it is very likely that the solver will spend real effort checking the answer.\n\nHowever, if the solver keeps arriving at the same answer, it may simply restate its previous conclusion and stop spending effort on verification.\nThis behavior is undesirable.\n\nTherefore, I implemented a backtracking logic.\nIf the solver submits the same answer two iterations in a row, I rewind its conversation by popping tokens off the end of its token stream until two assistant turns have been removed — effectively returning it to the state just before the iteration that first produced that answer. Generation then resumes from that earlier point.\n\nThis makes it more likely that the solver will continue to spend effort checking the answer, hopefully figuring out that the answer it has submitted is wrong, and finding the correct answer instead.\n\n\n---\n\n### Results\n\nMost of the changes were done by analyzing the errors and making changes to the code to address them.\nI do not have the GPU budget to run ablation tests.\n\n8 submissions were made for the notebook version.\nThe scores, in chronological order, are 42, 40, 40, 36, 39, 40, 40, and 41.\nThe scores have a mean of 39.75.\n\n---\n\nInitially, I did this so that I would have a good chance at the Longest Leader Prize. I did not win the prize. It seems that what was required is a clean implementation of gpt-oss-120b, as seen from the notebooks that first scored between 41 and 44.\n\nI thought I could achieve a significantly better score than the 44-score notebook, but I could not prove this, even though my changes made the model perform much better on Question 9 and Question 10. I realized that Questions 9 and 10 from the reference problems probably did not represent the problems in the leaderboard. The two questions involve multiple parts, which makes it easy for gpt-oss-120b to be correct in one part but wrong in another. Collaboration is especially useful, as I can allow gpt-oss-120b to compare notes and figure out who is correct. Perhaps most problems in the public leaderboard do not benefit from comparing notes.\n", "votes": 15 }, { "id": 3457793, "postDate": "2026-05-14T10:43:05.110Z", "content": "Adding to the comments by @sambealing below, I have a few additional questions:
\nSummary Injection: As I understand it, your approach generates summaries from completed traces and injects them into other generations that are still running. Did you test replacing this injection mechanism with an LLM-as-a-judge approach, similar to this submission? Would you say there is a benefit to avoiding distinct generation, summarization, and judgment stages?
Inference Bottleneck: You mention waiting for full traces before proceeding with the summarization step. Did you encounter cases where GPU cycles were wasted because one trace took significantly longer to finish than the others, causing the remaining traces or stages to wait?
Ablations and Time Management: Do you have ablations showing that answer retraction and backtracking improve performance on average in a competition-style evaluation? I imagine time budgeting becomes considerably more complex when each problem requires multiple sequential steps, especially given the synchronization issue mentioned above.
In some sense, my approach is already something like an LLM-as-a-judge (but a very biased one), the LLM will judge its current answer against the summaries of the other answers.
\nAnother feature that I have tried is to get a judge to decide whether a summary is worth injecting Another solver has presented the following finding ... Is this finding genuinely new and useful to you? Answer YES or NO, before I inject the summary. I decided to remove that feature because the performance of solvers running individually is getting higher ceiling scores than mine, and I want to be strictly better than that setup. I no longer need a classifier to decide whether a finding is useful, I will just inject the finding if the boxed answers are different.
My intention is to avoid distinct generation, summarization, and judgment stages. If I have multiple stages, I need to wait for every trace in each stage to complete or I interrupt, and tokens are wasted.
\nA principle of my design here is to keep the main traces uninterrupted, so that I can be more confident that my approach is better than a setup where the solvers solve independently.
\nIn my setup, all traces will continue to work until it passes the voting threshold or it times out. Traces will continue to generate even after the boxed answer is submitted, it will be given the status from the other traces and an instruction to check their answer. If a trace repeats the boxed answer multiple times, it will be backtracked to when it first repeated the boxed answer.
\nIf I assume all traces are doing productive work, I do not think I am wasting any GPU cycles.
\nThere is still a tradeoff being made though. If I terminate traces that submitted a boxed answer, I leave more memory and compute for other traces. Even so, I think it is faster to get a correct majority vote by fixing an early wrong answer than to be overpowered by other correct answers.
\nI observe that answer retraction actually retract wrong answers, and backtracking prevents the model from repeating the same submitted answer. I did not do ablation beyond these observations.
\nOn hindsight, I should have done ablations. Probably I should run 50 solve attempts on q9 and compare whether the win rate at least did not worsen, similar to what I did for the corpus prize submission. However, also on hindsight, improvements in q9 does not necessarily generalize to improvements in the leaderboard.
\nThe time budgeting did not get more complex, the decision to stop depends on the current vote distribution, once the vote passes, all in-flight generation will be stopped.
", "rawMarkdown": "### Summary Injection\n\nIn some sense, my approach is already something like an LLM-as-a-judge (but a very biased one), the LLM will judge its current answer against the summaries of the other answers.\n\nAnother feature that I have tried is to [get a judge](https://aimo.huikang.dev/runs/2025-12-08-23-35-59/communications/review-116wvp.txt) to decide whether a summary is worth injecting `Another solver has presented the following finding ... Is this finding genuinely new and useful to you? Answer YES or NO`, before I inject the summary. I decided to remove that feature because the performance of solvers running individually is getting higher ceiling scores than mine, and I want to be strictly better than that setup. I no longer need a classifier to decide whether a finding is useful, I will just inject the finding if the boxed answers are different.\n\nMy intention is to avoid distinct generation, summarization, and judgment stages. If I have multiple stages, I need to wait for every trace in each stage to complete or I interrupt, and tokens are wasted.\n\nA principle of my design here is to keep the main traces uninterrupted, so that I can be more confident that my approach is better than a setup where the solvers solve independently.\n\n\n### Inference Bottleneck\n\nIn my setup, all traces will continue to work until it passes the voting threshold or it times out. Traces will continue to generate even after the boxed answer is submitted, it will be given the status from the other traces and an instruction to check their answer. If a trace repeats the boxed answer multiple times, it will be backtracked to when it first repeated the boxed answer.\n\nIf I assume all traces are doing productive work, I do not think I am wasting any GPU cycles.\n\nThere is still a tradeoff being made though. If I terminate traces that submitted a boxed answer, I leave more memory and compute for other traces. Even so, I think it is faster to get a correct majority vote by fixing an early wrong answer than to be overpowered by other correct answers.\n\n\n### Ablations and Time Management\n\nI observe that answer retraction actually retract wrong answers, and backtracking prevents the model from repeating the same submitted answer. I did not do ablation beyond these observations.\n\nOn hindsight, I should have done ablations. Probably I should run 50 solve attempts on q9 and compare whether the win rate at least did not worsen, similar to what I did for the corpus prize submission. However, also on hindsight, improvements in q9 does not necessarily generalize to improvements in the leaderboard.\n\nThe time budgeting did not get more complex, the decision to stop depends on the current vote distribution, once the vote passes, all in-flight generation will be stopped." } ] }, { "id": 3457735, "postDate": "2026-05-14T08:46:29.193Z", "content": "Hi, thank you for your submission to the Writeup Prize. @philipvonderlind and I will post some questions below that help us in assessing your submission more closely, as described in the updated discussion post.
\nLook forward to reading your responses on these.
", "rawMarkdown": "Hi, thank you for your submission to the Writeup Prize. @philipvonderlind and I will post some questions below that help us in assessing your submission more closely, as described in the [updated discussion post](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/689703).\n\n1. **Run-viewer case studies:** You note you don't have GPU budget for systematic ablations, but the run viewer at aimo.huikang.dev/runs captures detailed traces. From those traces, do you have specific cases where the information-sharing step flipped a solver's answer from wrong to right (or vice versa), and a rough sense of how often that happens?\n2. **Comparison to Blackboard-style sharing:** Your information-sharing summarises a solver's progress via a separate LLM call. [ippeiogawa's Blackboard](https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/aimo3test) instead pushes raw tool I/O between solvers. How do you think these two approaches compare and what do you think are the advantages and disadvantages of each?\n\nLook forward to reading your responses on these.\n", "votes": 1, "replies": [ { "id": 3458858, "postDate": "2026-05-16T19:29:26.650Z", "content": "I only observe one case in q5 where the solver flipped from a correct answer (21818) to a wrong answer (62140).
\nSpecific cases where the information-sharing step flipped a solver's answer from wrong to right
\nlearned from correct
\n\ncorrected based on difference
\n\ncorrected on its own
\n\nlearned from correct
\n\ncorrected based on difference
\n\ncorrected on its own
\n| \n | q9 (160) | \nq10 (8687) | \n
|---|---|---|
| total traces | \n1411 | \n689 | \n
| did not box (no \\boxed) | \n708 (50.2%) | \n420 (61.0%) | \n
| boxed only once — correct | \n153 (10.8%) | \n15 (2.2%) | \n
| boxed only once — wrong | \n76 (5.4%) | \n91 (13.2%) | \n
| stayed correct (multi-box, all GT) | \n246 (17.4%) | \n32 (4.6%) | \n
| wrong → same wrong (one wrong value ≥2×) | \n136 (9.6%) | \n96 (13.9%) | \n
| wrong → different wrong (≥2 wrong values) | \n13 (0.9%) | \n22 (3.2%) | \n
| wrong → correct | \n79 (5.6%) | \n13 (1.9%) | \n
| learned from correct | \n58 (73.4% of 79) | \n7 (53.8% of 13) | \n
| corrected based on difference | \n13 (16.5% of 79) | \n6 (46.2% of 13) | \n
| corrected on its own | \n8 (10.1% of 79) | \n0 (0.0% of 13) | \n
| correct → wrong | \n0 (0.0%) | \n0 (0.0%) | \n
I think sharing code is a great idea, I should have tried that approach when sharing summaries.
\nAn issue I tried to resolve for my pipeline is that the model is not trying very hard to understand solutions that are different from theirs. I believe if models are provided the code and its output, the models would make a much better effort to understand the differences.
\nSharing code means that you need to ensure that the code works. The code shared needs to be runnable standalone, or otherwise the recipient will waste time debugging the code—and perhaps require a mechanism to test that the code could run on its own. However, even so, for the same tokens and time budget, I think that code is more effective than a pure-text sharing.
", "rawMarkdown": "### Run-viewer case studies\n\n\nI only observe [one case](https://aimo.huikang.dev/runs.html?datetime=2025-12-08-01-13-08&problem=424e18&solution=2) in q5 where the solver flipped from a correct answer (21818) to a wrong answer (62140).\n\n\nSpecific cases where the information-sharing step flipped a solver's answer from wrong to right\n\n\n### q9 (dd7f5e, GT = 160)\n\nlearned from correct\n- [214 -> [160](https://aimo.huikang.dev/runs.html?datetime=2025-12-04-01-43-44&problem=dd7f5e&solution=1)]\n- [114 -> [160](https://aimo.huikang.dev/runs.html?datetime=2025-12-07-21-55-41&problem=dd7f5e&solution=2)]\n\ncorrected based on difference\n- [266 -> [160](https://aimo.huikang.dev/runs.html?datetime=2025-12-04-00-48-09&problem=dd7f5e&solution=2)]\n- [266 -> [160](https://aimo.huikang.dev/runs.html?datetime=2025-12-04-01-38-17&problem=dd7f5e&solution=2)]\n\ncorrected on its own\n- [217 -> [160](https://aimo.huikang.dev/runs.html?datetime=2025-12-04-03-01-27&problem=dd7f5e&solution=1)]\n- [266 -> [160](https://aimo.huikang.dev/runs.html?datetime=2025-12-12-22-20-54&problem=dd7f5e&solution=26)]\n\n### q10 (86e8e5, GT = 8687)\n\nlearned from correct\n- [41754 -> [8687](https://aimo.huikang.dev/runs.html?datetime=2025-12-10-01-27-04&problem=86e8e5&solution=3)]\n- [96985 -> [8687](https://aimo.huikang.dev/runs.html?datetime=2026-02-18-05-21-35&problem=86e8e5©=9&solution=7)]\n\ncorrected based on difference\n- [41754 -> [8687](https://aimo.huikang.dev/runs.html?datetime=2025-12-17-22-22-40&problem=86e8e5©=18&solution=0)]\n- [7535 -> [8687](https://aimo.huikang.dev/runs.html?datetime=2025-12-17-22-22-40&problem=86e8e5©=18&solution=3)]\n\ncorrected on its own\n- (none — every q10 rescue had at least some external signal)\n\n\n| | q9 (160) | q10 (8687) |\n|------------------------------------------------|-------------:|------------:|\n| total traces | 1411 | 689 |\n| did not box (no \\boxed) | 708 (50.2%) | 420 (61.0%) |\n| boxed only once — correct | 153 (10.8%) | 15 (2.2%) |\n| boxed only once — wrong | 76 (5.4%) | 91 (13.2%) |\n| stayed correct (multi-box, all GT) | 246 (17.4%) | 32 (4.6%) |\n| wrong → same wrong (one wrong value ≥2×) | 136 (9.6%) | 96 (13.9%) |\n| wrong → different wrong (≥2 wrong values) | 13 (0.9%) | 22 (3.2%) |\n| wrong → correct | 79 (5.6%) | 13 (1.9%) |\n| learned from correct | 58 (73.4% of 79) | 7 (53.8% of 13) |\n| corrected based on difference | 13 (16.5% of 79) | 6 (46.2% of 13) |\n| corrected on its own | 8 (10.1% of 79) | 0 (0.0% of 13) |\n| correct → wrong | 0 (0.0%) | 0 (0.0%) |\n\n\n### Comparison to Blackboard-style sharing\n\nI think sharing code is a great idea, I should have tried that approach when sharing summaries.\n\nAn issue I tried to resolve for my pipeline is that the model is not trying very hard to understand solutions that are different from theirs. I believe if models are provided the code and its output, the models would make a much better effort to understand the differences.\n\nSharing code means that you need to ensure that the code works. The code shared needs to be runnable standalone, or otherwise the recipient will waste time debugging the code—and perhaps require a mechanism to test that the code could run on its own. However, even so, for the same tokens and time budget, I think that code is more effective than a pure-text sharing." } ] }, { "id": 3443942, "postDate": "2026-04-17T02:13:19.387Z", "content": "Interesting. I felt there were 2 kinds of hard problems. Ones like the Norwegian problem, which is pass@10/20 problem and another one where model quickly converges to wrong answer. I doubt the unsolvable ones are on the latter category where oss120B overfits on the problem and converges to wrong answer(by searching memory) in under a minute.
\nOn the point of interrupting the trace, what I found to be working with oss120B was to interrupt/inject with assistant role. User role interruption or non harmony encoding interruptions often broke the COT
", "rawMarkdown": "Interesting. I felt there were 2 kinds of hard problems. Ones like the Norwegian problem, which is pass@10/20 problem and another one where model quickly converges to wrong answer. I doubt the unsolvable ones are on the latter category where oss120B overfits on the problem and converges to wrong answer(by searching memory) in under a minute.\n\n\nOn the point of interrupting the trace, what I found to be working with oss120B was to interrupt/inject with assistant role. User role interruption or non harmony encoding interruptions often broke the COT", "votes": 3 }, { "id": 3444907, "postDate": "2026-04-18T17:00:01.610Z", "content": "Interesting approach, your observation that collaboration mainly helps on multi-part problems is especially insightful
", "rawMarkdown": "Interesting approach, your observation that collaboration mainly helps on multi-part problems is especially insightful" }, { "id": 3444366, "postDate": "2026-04-17T17:56:48.073Z", "content": "Have you shared this for the writeup prize, or just for the community?
\nIn the former case, please leave a comment here linking to this writeup here https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/689703
", "rawMarkdown": "Have you shared this for the writeup prize, or just for the community?\n\nIn the former case, please leave a comment here linking to this writeup here https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/689703\n", "replies": [ { "id": 3444372, "postDate": "2026-04-17T18:02:38.160Z", "content": "Thanks for reaching out!
\nOn whether I am writing this for the writeup prize, it depends on my final score, before I invest in writing a full write-up that is worth your time to judge.
\nFor now, I am just happy to share my html file.
", "rawMarkdown": "Thanks for reaching out!\n\nOn whether I am writing this for the writeup prize, it depends on my final score, before I invest in writing a full write-up that is worth your time to judge.\n\nFor now, I am just happy to share my html file.", "replies": [ { "id": 3445890, "postDate": "2026-04-20T16:27:22.800Z", "content": "Fair enough. But please note that we learned that it will likely be later than the Write-Up Prize deadline that the private LB scores will come in: https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/693267
\nSo it looks like you may have to make a decision whether you want to go all in or not. In any case, I am sure whatever you come up with will be well worth read. You are a veteran AIMO participant by now :)
", "rawMarkdown": "Fair enough. But please note that we learned that it will likely be later than the Write-Up Prize deadline that the private LB scores will come in: https://www.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-3/discussion/693267\n\nSo it looks like you may have to make a decision whether you want to go all in or not. In any case, I am sure whatever you come up with will be well worth read. You are a veteran AIMO participant by now :)" } ] } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T04:20:14Z", "competition_id": 118448, "competition_slug": "ai-mathematical-olympiad-progress-prize-3", "team_id": 14799743, "solution_url": "/competitions/ai-mathematical-olympiad-progress-prize-3/writeups/attempts-to-get-gpt-oss-120b-to-collaborate" }