![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F213493%2F70b340202a566617522baed38915aa25%2F1_fig_3_training_scores.png?generation=1745675140838902&alt=media\")
We would like to extend our thanks to the hosts from AIMOPrize, XTX Markets and Kaggle for this year's challenge which obviously had a lot of thought and preparation spent on the problems and the Kaggle environment. And a special shout out to Simon Frieder for being so accessible throughout the challenge.
\nWe were thrilled with the results - after developing our own TIR models we struggled to get Public LB to match the strong CV results but at the end private LB was more closely aligned.
\nThe dataset was built by collecting and filtering 540K math problems from AoPS forums. For our submission, 2.2M million Chain-of-Thought (CoT) responses were generated using models like DeepSeek-R1 with incorrect answers filtered out through verification. We also created a high quality Tool-Integrated Reasoning (TIR) dataset of 15K problems and generations.
\nTraining occurred in two phases: first a Qwen2.5-14B model was fine-tuned on CoT data, and second, the smaller TIR dataset was used to further refine, enhancing its problem-solving with structured reasoning.
\nTo retain the strength of TIR, along with the shorter responses of CoT for speed, we found mergekit worked well to merge both models into one, retaining the strength of TIR, while shortening the number of output tokens.
\nFor inference optimization, we implemented a pipeline combining TensorRT-LLM conversion with FP8 quantization, achieving 1.5x speedup over BF16. ReDrafter, a speculative decoding technique developed by Apple, was used for a further 1.8x speedup. Our serving system features dynamic batching with early stopping and a time-buffering strategy which allocates compute resources efficiently. The solution includes tool-integrated reasoning capabilities through a sandboxed execution environment.
We started by collecting an extensive set of mathematical problems from the internet. We developed an LLM-based problem extraction and refinement pipeline to construct a dataset of 540K unique problems. Using this dataset, we then generated 3.2M long reasoning CoT solutions. The process generates multiple solution candidates (up to 32) per problem using models like DeepSeek-R1 and QwQ-32B, with temperature 0.7 and top-p 0.95. Harder problems—identified by Qwen2.5-72B-Math-Instruct's pass-rates—receive more candidates. Incorrect solutions are filtered by verifying answer equivalence with Qwen2.5-32B-Instruct. If no answer is found, the most frequent candidate answer is used. For our submission a 2.2M subset generated by DeepSeek-R1 was used.
\nFor tool integrated reasoning the model prompts code for a calculation, and then the code is executed within a sandbox. We recognised code snippets with the special tokens <tool_call> and <\\tool_call>. The code is then appended to the LLM output, between the text ``` and ```output. An example snippet from an output is below.
, k must divide , k > So, k can be Therefore, b = k - = So, same result. Therefore, sum \\n\\nAlternatively, maybe I can a small program bases b > , compute b + b + , the latter divides the former, collect such bases. sum them. Let\\s a Python code the :\\n\\n<tool_call>\\n# Initialize a list store bases\\nvalid_bases = []\\n\\n# bases upwards\\nfor b range(, ): # Arbitrary upper \\n num1 = * b + \\n num2 = b + \\n num1 % num2 == :\\n valid_bases.append(b)\\n print(f\"Found base: {b}\")\\n\\n# Sum the bases\\nsum_bases = sum(valid_bases)\\nprint(f\"Sum: {sum_bases}\")\\n\\n# sum , take modulo \\nif sum_bases > :\\n result = sum_bases % \\nelse:\\n result = sum_bases\\n\\nprint(f\"Final Result: {result}\")\\n</tool_call>\\n```output\\nFound base: \\nFound base: \\nSum: \\nFinal Result: \\n```\\nThe code confirms that the bases are , summing Therefore, the final answer Since less than , modulo operation needed.\nWe developed a method for integrating code execution into long reasoning generations. Our initial attempts to elicit Tool-Integrated Reasoning (TIR) generations from DeepSeek-R1 through simple prompting proved unsuccessful. We hypothesize that these models struggle to deviate from their standard solution format due to extensive training on reasoning tasks and limited exposure to instruction-following. To overcome this challenge, we built a pipeline that starts with a small-scale reasoning finetuning of an instruction-following model, LIMO. By prompting this model to generate long-reasoning TIR solutions followed by aggressive quality filtering, we established an initial dataset suitable for training. Through multiple iterations of training, generation, and filtering, we constructed a 1.7M TIR solution set that was crucial for improving the accuracy of our final models.
\nFor final stage TIR training, the 1.7M solutions are aggressively filtered down to 15K samples based on number of code executions and correctness of answers.
\nFor our Kaggle submission we used a slightly different training recipe than detailed in the paper. We found this different recipe resulted in fewer tokens than the models publicly released, and we were out of time to experiment more with reducing tokens on the final models even though they performed better.
\nWe first trained a Qwen2.5-14B-Base model with SFT on the 2.2M subset of CoT solutions for 8 epochs. We changed base RoPE to 500k to allow long reasoning. The model is trained with NVIDIA/Nemo-Skills for 8 epochs with 1e-4 LR, AdamW, weight decay 0.01 and 10% linear warmup decaying to a LR of 1e-7. We use batch size of 1024 samples and leverage sequence packing and context parallelization techniques from NVIDIA/NeMo-Aligner which significantly accelerates training on the long-reasoning data. The training lasted 48 hours on 512 H100 (yes, 512!). We could achieve most of the strength of the model with 20% of the compute but we wanted to scale up to see where learning saturates. See figure 3(b) from the linked publication below which shows metrics at different stages of training. Weight averaging is used from different stages for the final weights.
\n
This is followed by a light-weight TIR fine-tuning on the 15k TIR samples. We trained the TIR model for 400 steps with a constant learning rate of 1e-5 and use the last checkpoint without averaging.
\nAs detailed below, we then merge CoT and TIR checkpoints as it both improves accuracy and speeds up generation by reducing solution length and number of code executions.
\nDuring the competition we mostly use problems from the 2024 year for American Invitational Mathematics Examinations (AIME) and Harvard-MIT Mathematics Tournaments (HMMT). Later we added problems from 2025. Our final benchmark Comp-Math-24-25 includes 256 problems.
\n\n| Dataset | \nProblems | \n
|---|---|
| AIME24 | \n30 | \n
| AIME25 | \n30 | \n
| HMMT Nov 2024 | \n62 | \n
| HMMT Feb 2024 | \n68 | \n
| HMMT Feb 2025 | \n66 | \n
In this competition, we explored various methods for merging two LLMs with CoT and TIR behaviors. Our primary goal was to effectively combine the distinct strengths of these two fine-tuning stages to enhance the model's performance. We experimented with several merging techniques from mergekit package. Surprisingly, the most effective approach turned out to be a simple linear combination of the two checkpoints: the CoT checkpoint used before TIR fine-tuning and the best TIR checkpoint attained thereafter. This strategy allowed us to control the extent to which each stage influenced the final model's behavior.
\nIn the following table, we show accuracy and generation statistics of merged models on Comp-Math-24-25 dataset. length indicates the average number of tokens per solution, while code refers to the average number of code executions per solution.
\n\n| Method | \nmaj@16 | \npass@16 | \nlength | \ncode | \n
|---|---|---|---|---|
| CoT | \n62.9 | \n76.2 | \n11203 | \n- | \n
| TIR | \n66.8 | \n80.1 | \n15834 | \n2.73 | \n
| CoT * 0.3 + TIR * 0.7 | \n69.1 | \n81.3 | \n12489 | \n0.85 | \n
We prioritized Int8 weight-only (W8A16) and FP8 quantization, which delivered faster inference than BF16 with minimal accuracy loss. The reduced weight size also freed up memory for larger key-value caches.
\nReDrafter is a speculative decoding technique developed by Apple, and implemented in TensorRT-LLM. We trained a ReDrafter head on a random subset of problems from OpenMathReasoning-1 dataset. Using these problems we generated 100k solutions with the target model. The resulting redrafter generated 3 tokens for every LLM step with 65% acceptance rate delivering a speed up of approximately 1.8X. The TensorRT-LLM repo’s example implementation has an example for a non quantised llama, so we needed some changes to make it work. We tried other drafting methods, but could not get them to work as well. Especially LLM drafting should work and may be better.
| Quantization | \nSpeed tok/sec L4x4 | \n50 question eval time L40x4 | \nAccuracy score % AIME24 | \nAccuracy score % AIME25 | \n
|---|---|---|---|---|
| bf16 | \n210 | \n2h | \n82.7 | \n66.7 | \n
| w8a16 (int8) | \n315 | \n1h 45m | \n82.7 | \n66.7 | \n
| w4a16 (int4) | \n436 | \n1h 35m | \n72.7 | \n60.7 | \n
| f8a16 | \n310 | \n1h 40m | \n83.3 | \n68.7 | \n
| f8a16 + Redrafter | \n554 | \n1h | \n81.3 | \n71.3 | \n
Accuracy score in the table is using maj@12 of the merged model averaged over 5 runs.
\nPretrained models were converted to TensorRT engines using TensorRT-LLM. TensorRT’s inflight batching boosts throughput by dynamically grouping inference requests, releasing each sample as soon as it completes—reducing latency and optimizing GPU utilization. See below some recent benchmarks by vLLM team.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F213493%2F1600cb97cfaf08259271fd42cf57c178%2F2_vLLM%20comparison.jpeg?generation=1745675061119172&alt=media\")
Since samples are processed independently, batches can mix different prompts or seeds seamlessly. TensorRT-LLM includes a number of other optimizations such as custom attention kernels and paged KV caching.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F213493%2F4555c9c735ed6627686e451efe987af2%2F3_inflight_batching.png?generation=1745675044303870&alt=media\")
For each new problem we used different seeds to launch 12 async generations via in-flight batching in TensorRT. The streaming for each sample was monitored for code blocks, stop phrases, max tokens or timeouts. If the LLM generates code, LLM generation stops, and the code block is executed in a sandbox. The sandbox output (or partial error trace) is appended to the LLM and generation continues. The generation continues until another code block. When no other code blocks are encountered, the LLM stops at one of max tokens, timeout time, or stop phrase.
\nWe experimented with various sampling parameters but observed minimal differences in the results. However, our winning submission was based on an \"almost\" greedy search strategy. Despite setting the temperature to 0 and enabling the redrafter_greedy_search parameter, TensorRT-LLM still produced slightly varying outputs within a single batch. We did not investigate this behavior in detail, but speculate that it could be related to rounding or batching issues. Ultimately, we chose this approach because it provided more stable results at small batch sizes and offered a slight improvement in the speed of speculative decoding.
\nWithin the kaggle notebook, generations are monitored for completion. When initial answers are the same we early-stop, e.g. if 4 of the first 5 generations are the same we cancel the remaining generations in TensorRT.
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F213493%2F5c66f8f8ebdbf641c71adb4bbd588391%2F4_nemoskills%20inference%20flow.png?generation=1745675026592359&alt=media\")
During monitoring of async generations, we also early stop after completing 10 of the 12 generations to avoid waiting excessively long for any stragglers. A buffering strategy was implemented, allocating 350 seconds per question as the base time limit. If a question completed early, the unused time was added to a shared buffer. The next question could then draw up to 210 extra seconds from this buffer, making a total of 560 seconds.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F213493%2F448b7da3f0bbf0ead9bf8ef98b25b680%2F5_early_stopping.png?generation=1745674994945752&alt=media\")
Using this inference pipeline and deepseek-distill-14b, we scored in the range of 23 to 29 on public leaderboard using different settings. The score variability was concerning and our own trained models helped to stabilise the scores as well as increasing the accuracy.
\nFor the final chosen submissions, one 14b CoT model and the above MIX TIR model were chosen. The MIX TIR model scored much better on CV, and a little better on Public LB (public LB scores : 32,33,28). The one 28 scoring submission made us nervous and was perhaps due to a larger maj@ used. In the end, the private LB reflected more our CV results than LB results which was quite satisfying.
\nWe had other models as seen which were quite strong on CV (see the report) but the same size model produced more tokens than the previous versions, and we believe in public LB they were timing out before having a chance to properly attempt the questions. Ultimately we were out of time and submissions to properly narrow down the discrepancies seen between CV and public LB, especially given the variance with one submission per day and only 50 questions scored.
\nA number of experiments were made but were not in the final submission choices, mostly due to limited time likely causing early exit before a good answer was made,
\nWhat an incredible solution - truly worth of the name SOTA.
\nThere’s so much to learn from it. Thank you for sharing such an inspiring work.
This year's 1st place is even crazier than last year! Congratulations to the whole team with the incredible achievement, and a family of frontier-level reasoning models. The fact that you trained everything from base Qwen (not even reasoning) models with 1.5B beating 14B DeepSeek is bonkers!
\nI have a question: your team is the only one among the top teams that didn't get nerd-sniped by RL (GRPO in particular) and relied only on SFT. Was it intentional, or you didn't have time to perform further tuning? You did mention about your generation length problem, that in theory should be solvable with RL.
", "votes": 3, "replies": [ { "id": 3187241, "author_name": "Igor Gitman", "author_url": "", "post_date": "2025-04-25T17:45:14.867000", "content": "We mainly bet on SFT with a large dataset generated by a powerful teacher since there is some evidence that it works better for small models. E.g. DeepSeek-R1 report makes this claim, and we also observed it in our experiments. You're right that to address the generation length we could have used RL. The main reason we didn't is simply because we noticed it too late. With the variance in the submissions and our local tests not fully matching what we saw in LB, it took us some time to realize that longer generations are likely the reason our new models weren't scoring well.
\nIf we were to do it again, I'd say the best recipe would be to start with large-scale SFT and then do RL on top to make models use fewer tokens.
", "votes": 8, "replies": [] } ] }, { "id": 3185902, "author_name": "Jason Bourne", "author_url": "", "post_date": "2025-04-24T02:05:16.950000", "content": "Great job!
\nLearning from you guys is the reason why I participate in this competition.
Crazy solution. I never saw a kaggle solution takes so much talent, skills, effort and resourses. 🫠
", "votes": 4, "replies": [] }, { "id": 3194838, "author_name": "planet2020", "author_url": "", "post_date": "2025-05-06T10:59:10.920000", "content": "Thank you for sharing such a detailed solution and a convenient and comprehensive datasets.
\nI tried running your utility but it failed.
cell:
\n!pip install --no-index --no-deps --upgrade --target=/kaggle/working \\
\n --find-links file:///kaggle/input/tensorrt-llm-aimo-wheels-ns-varlen/ nemo_skills
\nError:
\nLooking in links: file:///kaggle/input/tensorrt-llm-aimo-wheels-ns-varlen/
\nWARNING: Location 'file:///kaggle/input/tensorrt-llm-aimo-wheels-ns-varlen/' is ignored: it is neither a file nor a directory.
\nERROR: Could not find a version that satisfies the requirement nemo_skills (from versions: none)
\nERROR: No matching distribution found for nemo_skills
@planet2020 Sorry about missing this for so long and thanks for checking it… that dataset tensorrt-llm-aimo-wheels-ns-varlen was private but is public now to let you run it.
\nFeel free to dm me if you get any more issues; will check in here to see if you get any more issues.
Quick follow up - @cdeotte helped to tested; it works now - just make sure to turn on the L40 GPU. Tensor parallelism is set to 4, and the model is build for the L40.
\nThanks for alerting us to this.
@darraghdog Thanks for update! I tried again today. Unfortunately it still failed.
\nWhen run this cell:
%%bash
\nexport LD_LIBRARY_PATH=\"/kaggle/working/nvidia/nvjitlink/lib:$LD_LIBRARY_PATH\"
\npython -m tensorrt_llm.commands.build -h
Got an error:
\n/usr/bin/python3: Error while finding module specification for 'tensorrt_llm.commands.build' (ModuleNotFoundError: No module named 'tensorrt_llm')
In the previous cell in that notebook:
\n!pip install --no-index --target=/kaggle/working --find-links file:///kaggle/input/tensorrt-llm-aimo-wheels-ns-varlen/ \\
\n tensorrt_llm==0.17.0.dev2024121700 \\
\n tensorrt-cu12_libs==10.7.0 \\
\n tensorrt-cu12_bindings==10.7.0
Then I check tensorrt-llm dataset. It contains tensorrt_llm-0.19.0 instead of tensorrt_llm-0.17.0. I am not sure if this is where the problem comes from.
\nI tried run tensorrt-llm-aimo-wheels ns varlen notebook but still got an error.
\nWhen run this cell:
\n!pip wheel --no-verify --no-cache-dir -w /kaggle/working \\
\n /kaggle/input/tensorrt-llm/tensorrt_llm-0.17.0.dev2024121700-cp310-cp310-linux_x86_64.whl
Got an error:
\nWARNING: Requirement '/kaggle/input/tensorrt-llm/tensorrt_llm-0.17.0.dev2024121700-cp310-cp310-linux_x86_64.whl' looks like a filename, but the file does not exist
\nERROR: tensorrt_llm-0.17.0.dev2024121700-cp310-cp310-linux_x86_64.whl is not a supported wheel on this platform.
Then I tried replace tensorrt_llm-0.17.0 with 0.19.0 version that exists in the tensorrt-llm dataset but still got an error:
\nERROR: tensorrt_llm-0.19.0.dev2025032500-cp310-cp310-linux_x86_64.whl is not a supported wheel on this platform.
I have no idea how to run it without an error.
", "votes": 1, "replies": [] }, { "id": 3202326, "author_name": "Darragh", "author_url": "", "post_date": "2025-05-15T08:06:37.543000", "content": "The lines below is from the linked tensorrt-llm installation script tensorrt-llm-aimo-wheels-ns-varlen,
\n%%bash\nexport LD_LIBRARY_PATH=\"/kaggle/working/nvidia/nvjitlink/lib:$LD_LIBRARY_PATH\"\npython -m tensorrt_llm.commands.build -h
\nYou are right, the linked tensorrt_llm pip wheel dataset has been upgraded to 0.19.0 in the meantime. Please give us a bit of time to get this sorted out for you.
In the meantime, if you want to run the main submission book without reinstalling tensorrt_llm, you should be able to just re-fork it (Copy and edit) in the drop down beside the edit button. This should just use the utility script we attached (the original v05 version is now pinned as default), and as long as L40 is turned on for GPU, this should run. See a test here - its running now, should be done in an hour.
\nThanks again for verifying this.
", "votes": 0, "replies": [] }, { "id": 3202413, "author_name": "planet2020", "author_url": "", "post_date": "2025-05-15T10:29:51.657000", "content": "Thank you for your help. I know that your utility works for running your notebook. But I want to be able to install tensorrt_llm in Kaggle by myself as well as other libraries used for your submission notebook so that any changes of your utility in the future will not affect my notebook.
", "votes": 1, "replies": [] }, { "id": 3202432, "author_name": "Darragh", "author_url": "", "post_date": "2025-05-15T11:04:49.807000", "content": "So the levels we have are,
\n0.17.0.dev2024121700 version. pip-install-tensorrt-llm-ns-varlen2 - we made a new book to test it all works. Uses step 2 as input. f'export LD_LIBRARY_PATH=\"/kaggle/usr/lib/pip_install_tensorrt_llm_ns_varlen2/nvidia/nvjitlink/lib:$LD_LIBRARY_PATH\" &&'Also, if you have a new tensorrt_llm model, make sure it builds on an L4/L40/L40S for best performance.
\nAnd let us know how it goes.
Big thanks to @sorokin for helping through this! I forgot how complicated it is.
", "votes": 2, "replies": [] }, { "id": 3202834, "author_name": "planet2020", "author_url": "", "post_date": "2025-05-16T02:10:55.457000", "content": "@darraghdog It works now. Thank you very much for your guidance.😀
\nI am wondering what reason for you to choose 0.17.0 version instead of 0.19.0? Is it possible to install new version such as 0.19.0 on Kaggle?
We did a lot of experiments on this 0.17.x version and had everything set up with it. I believe 0.19.0 was released towards the end or after the competition, but 0.18.0 was out there. We did do some experiments with it, but it would have meant rebuilding older models and the environment explained above with the new version - with more time we would have done it for sure.
If you are interest to try building the nvidia/OpenMathReasoning, or the Fast-OpenMath-Nemotron-14B models in the inference pipeline with the new pipeline it could be interesting. We work on docs for doing this, just to make a bit simpler, and could share. It would be without ReDrafter, but you could try building one of those models and the pipeline in 0.19.0. Without ReDrafter, it would probably make sense to try the Fast-OpenMath-Nemotron-14B.
Got it. I will definitely try it when I have time. Thank you very much for your patient guidance and I learned a lot from it. Congratulations on winning the first place in AIMO2.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3189641, "author_name": "Ivan Tretyakov", "author_url": "", "post_date": "2025-04-29T14:23:37.553000", "content": "Great job!
\nIt is a pleasure to read the magnum opus of such a large number of the best specialists!
", "votes": 1, "replies": [] }, { "id": 3189079, "author_name": "", "author_url": "", "post_date": "2025-04-28T17:41:41.340000", "content": "This is top notch. Congratulations and thanks a lot for the dataset released. 🙌
\nI am very very happy for noticing one small thing in your solution. You used both prompt+question in the user role. May I know the reason behind this? I even asked the second place team on why they didn't use it, they used both user and system role
\nDid your team find any improvements on doing this? I got some bumps in my LB by changing this. Of course I first saw this in many threads, posts on other communities and the model card on deepseek itself, but not a single public nb had this!
But I really want to know your thoughts on this 🤔
", "votes": 1, "replies": [ { "id": 3189857, "author_name": "Igor Gitman", "author_url": "", "post_date": "2025-04-29T21:31:15.180000", "content": "This is how we trained the model, to have both prompt and question inside the user part. When you do SFT, it's largely your own preference as the model will learn just fine whether you put everything in the system or the user part or split it. After all, it's just a few tokens difference, so as long as you match the setup between training and inference, it doesn't really matter
", "votes": 1, "replies": [] } ] }, { "id": 3187072, "author_name": "Sarah Arshad", "author_url": "", "post_date": "2025-04-25T14:15:57.167000", "content": "Congratulations man!
", "votes": 1, "replies": [] }, { "id": 3186964, "author_name": "mx1024", "author_url": "", "post_date": "2025-04-25T11:00:35.607000", "content": "Great job!
\nYour open data is a huge help to the open-source community.
\nIt’ll really benefit researchers who can’t build such large datasets themselves, just like the Numina team in the last competition.
Congratulations on First Prize in this competition. Can you please advise the running time of your notebook. Appreciate your observation on why your notebook did not solve correctly the remaining problems.
", "votes": 1, "replies": [ { "id": 3186956, "author_name": "Darragh", "author_url": "", "post_date": "2025-04-25T10:43:15.873000", "content": "Thanks, you can check it in the Kaggle submission script. On LB, hard to tell due to runtime being obfuscated.
\nEach question has stats like below printed. For example total_num_generated_tokens is the LLM generated tokens for each of the 12 samples. cumulative time is time since start of script. We watched this quite a bit. Kaggle L4 was slower than our testing L4 for some reason (circa 10 to 20%). We also ran our CV sets in kaggle to check time - a big team luckily meant bigger gpu quota to help out 😃
88c219\n\n'<|im_start|>system<|im_end|><|im_start|>userSolve the following math problem. Make sure to put the answer (and only answer) inside boxed{}.For positive integers $x_1,ldots, x_n$ define (x_1, ldots, x_n)$ to be the sum of their $frac{n(n-1)}{2}$ pairwise greatest common divisors. We say that an integer $n geq 2$ is emph{artificial} if there exist $n$ different positive integers $a_1, ..., a_n$ such that [a_1 + cdots + a_n = G(a_1, ldots, a_n) +1.]Find the sum of all artificial integers $m$ in the range $2 leq m leq 40$.If the final answer is a number larger than 1000, take modulo 1000.<|im_end|><|im_start|>assistant'\n\nFinish times: 254 307 552 561 561 562 562 562 563 563 563 563 Cancelling @ 563\ntotal_num_generated_tokens [9550, 11381, 18992, 18112, 18444, 18243, 17759, 18565, 17498, 17945, 18892, 18746]\ngeneration_time [253, 306, 551, 561, 561, 562, 558, 561, 559, 562, 562, 563]\ncode_rounds_executed [0, 1, 0, 0, 0, 0, 3, 0, 4, 0, 0, 0]\ncode_execution_time [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n\nall extracted answers: [0, 6]\n\nSolved with answer 0; correct answer 810. Time taken 563; new buffer time 700; cumulative time 2504\nscore: 6/8\nvery interesting and clever investigation
", "votes": 1, "replies": [] }, { "id": 3190158, "author_name": "m2rt1n", "author_url": "", "post_date": "2025-04-30T08:00:53.977000", "content": "wow! its so easy
", "votes": 2, "replies": [] }, { "id": 3186392, "author_name": "Steven Boon", "author_url": "", "post_date": "2025-04-24T15:38:04.897000", "content": "Congratulation for your great win and Thank you for your great sharing.
\nInitially thought of building on top of your CoT and TIR model with some merging ideas, but sadly those models was not release ☹ .
\nUnderstood that nVidia might have some legal reasons that prevent you from releasing certain things fully under open source, but can you consider at least open source the TIR and CoT model for the Kaggle related version.
\nIf releasing it on HF is not allowed, maybe you can discuss it with @friederrr to host it on Kaggle Model as part of AIMO full open-source open science reproducibility initiative.
\nThis is not only good for effort to reproduce or innovate on top of it, but also aid in apple-to-apple comparison with other’s open source release under non-merging setting.
\nNumina did it in AIMO1 : https://huggingface.co/collections/AI-MO/numinamath-6697df380293bcfdbc1d978c and attracted good open reproduce effort: https://www.kaggle.com/competitions/ai-mathematical-olympiad-prize/discussion/523210
\nHope the tradition can continue.
\nLast but not least, please do share to us the merging code as well, it is a good reference.
", "votes": 2, "replies": [ { "id": 3186481, "author_name": "Darragh", "author_url": "", "post_date": "2025-04-24T18:09:30.610000", "content": "The MIX model we used in kaggle, as well as the datasets for training are released. Check the model marked kaggle in the Collection of Math Reasoning models linked above.
\nTo rerun inference you need to convert to tensorrt_llm - see qwen conversion. This build would be without redrafter - but you could use the trtllm build and follow along what happens in the submission script.
\nWe plan to make some easy to follow tutorials to add redrafter but you can run without this; if you wish to quantise, maybe try the int8_wo (INT8 weight-only quantization) in the qwen conversion and you should get good speed up.
I refer to the before-merging TIR and CoT model (e.g model CoT & TIR in Table 10 of your paper), the one you released is after-merged final model.
\nYou can refer to Numina AIMO1 Winner release for better understanding:
\nTheir final model is NuminaMath-7B-TIR , but they also released NuminaMath-7B-CoT which is the intermediate trained model .
\nhttps://huggingface.co/collections/AI-MO/numinamath-6697df380293bcfdbc1d978c
There are few reasons from open science research point of view that this is important:
\nSome other open source released models is only trained on CoT or TIR without merging, apple-to-apple comparison can only been perform vs your non-merged CoT or TIR model, not vs the final merged model.
\nThe goal is to evaluate the impact of training dataset alone before the effect of merging.
Researcher can try out different merging ideas to see if better result can be achieve with your non-merged CoT/TIR model.
Some community member might just want to reproduce the model merging part.
We discussed internally and we will not release externally the intermediate model weights at the moment. There is some admin work for it, and it would mean more models on hf which need docs to explain they are from a specific part of the pipeline etc.; and kaggle when we host there. I know it was done last year, but we have a quite a different solution and architecture (trtllm based TIR/specdecoding) than last years winners. I could suggest you take a CoT model like deepseek-14b and mix with our TIR model; the details of the mixing are in the paper.
\nAlternatively you could try a small finetuning of a Math Reasoning CoT model (QwQ or dseek), using ~1k samples for the data we provide and create your own TIR model - and mix that with the original to reproduce the work.
\nI appreciate this is not the answer you are hoping for, but hopefully there is enough released between the new data, family of models that you can try some experiment on different parts.
@binbinai Thanks for raising these points, in particular 3. We are following this up with the NemoSkills team to assess if this is an impediment to reproducibility, and whether this is inherently unresolvable or not.
", "votes": 1, "replies": [] }, { "id": 3201939, "author_name": "Darragh", "author_url": "", "post_date": "2025-05-14T16:17:16.123000", "content": "@friederrr (fyi @binbiai ) as per our mail, we trained a CoT-only model, then finetuned it further on TIR data and then merged the two models. We released all the data we used as well as all details needed for training, so anyone can rerun SFT, get both models and merge them to reproduce the final checkpoint.
\nThanks again for all AIMO's effort here.
Good job man!
\nBut how?
I am here for such write-ups, truly a great work, thanks for sharing and congratulations!
", "votes": 0, "replies": [] }, { "id": 3467809, "author_name": "Ugur Tasdan", "author_url": "", "post_date": "2026-06-07T13:39:32.157000", "content": "Type of writeups and people that I would love to get schooled by. Congratulations.
", "votes": 0, "replies": [] }, { "id": 3193128, "author_name": "Amr Yasser", "author_url": "", "post_date": "2025-05-03T22:23:24.167000", "content": "great job man!
", "votes": 0, "replies": [] }, { "id": 3187914, "author_name": "mohanapavan bezawada", "author_url": "", "post_date": "2025-04-26T17:41:13.130000", "content": "great job man!
", "votes": 0, "replies": [] }, { "id": 3185887, "author_name": "", "author_url": "", "post_date": "2025-04-24T01:29:19.903000", "content": "", "votes": 9, "replies": [] }, { "id": 3186990, "author_name": "bot8877", "author_url": "", "post_date": "2025-04-25T12:00:25.240000", "content": "Great Work !
", "votes": 1, "replies": [] }, { "id": 3185956, "author_name": "tuanjhg", "author_url": "", "post_date": "2025-04-24T03:40:51.060000", "content": "congratulations
", "votes": 1, "replies": [] }, { "id": 3189733, "author_name": "Fullzer4", "author_url": "", "post_date": "2025-04-29T16:54:03.267000", "content": "Congratulations!
", "votes": 0, "replies": [] }, { "id": 3186639, "author_name": "니엘NIEL", "author_url": "", "post_date": "2025-04-25T01:50:27.520000", "content": "Thank you. I learned more
", "votes": 0, "replies": [] } ], "comment_count": 40, "messages": [ { "id": 3185782, "postDate": "2025-04-23T20:04:04.227Z", "content": "We would like to extend our thanks to the hosts from AIMOPrize, XTX Markets and Kaggle for this year's challenge which obviously had a lot of thought and preparation spent on the problems and the Kaggle environment. And a special shout out to Simon Frieder for being so accessible throughout the challenge.
\nWe were thrilled with the results - after developing our own TIR models we struggled to get Public LB to match the strong CV results but at the end private LB was more closely aligned.
\nThe dataset was built by collecting and filtering 540K math problems from AoPS forums. For our submission, 2.2M million Chain-of-Thought (CoT) responses were generated using models like DeepSeek-R1 with incorrect answers filtered out through verification. We also created a high quality Tool-Integrated Reasoning (TIR) dataset of 15K problems and generations.
\nTraining occurred in two phases: first a Qwen2.5-14B model was fine-tuned on CoT data, and second, the smaller TIR dataset was used to further refine, enhancing its problem-solving with structured reasoning.
\nTo retain the strength of TIR, along with the shorter responses of CoT for speed, we found mergekit worked well to merge both models into one, retaining the strength of TIR, while shortening the number of output tokens.
\nFor inference optimization, we implemented a pipeline combining TensorRT-LLM conversion with FP8 quantization, achieving 1.5x speedup over BF16. ReDrafter, a speculative decoding technique developed by Apple, was used for a further 1.8x speedup. Our serving system features dynamic batching with early stopping and a time-buffering strategy which allocates compute resources efficiently. The solution includes tool-integrated reasoning capabilities through a sandboxed execution environment.
We started by collecting an extensive set of mathematical problems from the internet. We developed an LLM-based problem extraction and refinement pipeline to construct a dataset of 540K unique problems. Using this dataset, we then generated 3.2M long reasoning CoT solutions. The process generates multiple solution candidates (up to 32) per problem using models like DeepSeek-R1 and QwQ-32B, with temperature 0.7 and top-p 0.95. Harder problems—identified by Qwen2.5-72B-Math-Instruct's pass-rates—receive more candidates. Incorrect solutions are filtered by verifying answer equivalence with Qwen2.5-32B-Instruct. If no answer is found, the most frequent candidate answer is used. For our submission a 2.2M subset generated by DeepSeek-R1 was used.
\nFor tool integrated reasoning the model prompts code for a calculation, and then the code is executed within a sandbox. We recognised code snippets with the special tokens <tool_call> and <\\tool_call>. The code is then appended to the LLM output, between the text ``` and ```output. An example snippet from an output is below.
, k must divide , k > So, k can be Therefore, b = k - = So, same result. Therefore, sum \\n\\nAlternatively, maybe I can a small program bases b > , compute b + b + , the latter divides the former, collect such bases. sum them. Let\\s a Python code the :\\n\\n<tool_call>\\n# Initialize a list store bases\\nvalid_bases = []\\n\\n# bases upwards\\nfor b range(, ): # Arbitrary upper \\n num1 = * b + \\n num2 = b + \\n num1 % num2 == :\\n valid_bases.append(b)\\n print(f\"Found base: {b}\")\\n\\n# Sum the bases\\nsum_bases = sum(valid_bases)\\nprint(f\"Sum: {sum_bases}\")\\n\\n# sum , take modulo \\nif sum_bases > :\\n result = sum_bases % \\nelse:\\n result = sum_bases\\n\\nprint(f\"Final Result: {result}\")\\n</tool_call>\\n```output\\nFound base: \\nFound base: \\nSum: \\nFinal Result: \\n```\\nThe code confirms that the bases are , summing Therefore, the final answer Since less than , modulo operation needed.\nWe developed a method for integrating code execution into long reasoning generations. Our initial attempts to elicit Tool-Integrated Reasoning (TIR) generations from DeepSeek-R1 through simple prompting proved unsuccessful. We hypothesize that these models struggle to deviate from their standard solution format due to extensive training on reasoning tasks and limited exposure to instruction-following. To overcome this challenge, we built a pipeline that starts with a small-scale reasoning finetuning of an instruction-following model, LIMO. By prompting this model to generate long-reasoning TIR solutions followed by aggressive quality filtering, we established an initial dataset suitable for training. Through multiple iterations of training, generation, and filtering, we constructed a 1.7M TIR solution set that was crucial for improving the accuracy of our final models.
\nFor final stage TIR training, the 1.7M solutions are aggressively filtered down to 15K samples based on number of code executions and correctness of answers.
\nFor our Kaggle submission we used a slightly different training recipe than detailed in the paper. We found this different recipe resulted in fewer tokens than the models publicly released, and we were out of time to experiment more with reducing tokens on the final models even though they performed better.
\nWe first trained a Qwen2.5-14B-Base model with SFT on the 2.2M subset of CoT solutions for 8 epochs. We changed base RoPE to 500k to allow long reasoning. The model is trained with NVIDIA/Nemo-Skills for 8 epochs with 1e-4 LR, AdamW, weight decay 0.01 and 10% linear warmup decaying to a LR of 1e-7. We use batch size of 1024 samples and leverage sequence packing and context parallelization techniques from NVIDIA/NeMo-Aligner which significantly accelerates training on the long-reasoning data. The training lasted 48 hours on 512 H100 (yes, 512!). We could achieve most of the strength of the model with 20% of the compute but we wanted to scale up to see where learning saturates. See figure 3(b) from the linked publication below which shows metrics at different stages of training. Weight averaging is used from different stages for the final weights.
\n
This is followed by a light-weight TIR fine-tuning on the 15k TIR samples. We trained the TIR model for 400 steps with a constant learning rate of 1e-5 and use the last checkpoint without averaging.
\nAs detailed below, we then merge CoT and TIR checkpoints as it both improves accuracy and speeds up generation by reducing solution length and number of code executions.
\nDuring the competition we mostly use problems from the 2024 year for American Invitational Mathematics Examinations (AIME) and Harvard-MIT Mathematics Tournaments (HMMT). Later we added problems from 2025. Our final benchmark Comp-Math-24-25 includes 256 problems.
\n\n| Dataset | \nProblems | \n
|---|---|
| AIME24 | \n30 | \n
| AIME25 | \n30 | \n
| HMMT Nov 2024 | \n62 | \n
| HMMT Feb 2024 | \n68 | \n
| HMMT Feb 2025 | \n66 | \n
In this competition, we explored various methods for merging two LLMs with CoT and TIR behaviors. Our primary goal was to effectively combine the distinct strengths of these two fine-tuning stages to enhance the model's performance. We experimented with several merging techniques from mergekit package. Surprisingly, the most effective approach turned out to be a simple linear combination of the two checkpoints: the CoT checkpoint used before TIR fine-tuning and the best TIR checkpoint attained thereafter. This strategy allowed us to control the extent to which each stage influenced the final model's behavior.
\nIn the following table, we show accuracy and generation statistics of merged models on Comp-Math-24-25 dataset. length indicates the average number of tokens per solution, while code refers to the average number of code executions per solution.
\n\n| Method | \nmaj@16 | \npass@16 | \nlength | \ncode | \n
|---|---|---|---|---|
| CoT | \n62.9 | \n76.2 | \n11203 | \n- | \n
| TIR | \n66.8 | \n80.1 | \n15834 | \n2.73 | \n
| CoT * 0.3 + TIR * 0.7 | \n69.1 | \n81.3 | \n12489 | \n0.85 | \n
We prioritized Int8 weight-only (W8A16) and FP8 quantization, which delivered faster inference than BF16 with minimal accuracy loss. The reduced weight size also freed up memory for larger key-value caches.
\nReDrafter is a speculative decoding technique developed by Apple, and implemented in TensorRT-LLM. We trained a ReDrafter head on a random subset of problems from OpenMathReasoning-1 dataset. Using these problems we generated 100k solutions with the target model. The resulting redrafter generated 3 tokens for every LLM step with 65% acceptance rate delivering a speed up of approximately 1.8X. The TensorRT-LLM repo’s example implementation has an example for a non quantised llama, so we needed some changes to make it work. We tried other drafting methods, but could not get them to work as well. Especially LLM drafting should work and may be better.
| Quantization | \nSpeed tok/sec L4x4 | \n50 question eval time L40x4 | \nAccuracy score % AIME24 | \nAccuracy score % AIME25 | \n
|---|---|---|---|---|
| bf16 | \n210 | \n2h | \n82.7 | \n66.7 | \n
| w8a16 (int8) | \n315 | \n1h 45m | \n82.7 | \n66.7 | \n
| w4a16 (int4) | \n436 | \n1h 35m | \n72.7 | \n60.7 | \n
| f8a16 | \n310 | \n1h 40m | \n83.3 | \n68.7 | \n
| f8a16 + Redrafter | \n554 | \n1h | \n81.3 | \n71.3 | \n
Accuracy score in the table is using maj@12 of the merged model averaged over 5 runs.
\nPretrained models were converted to TensorRT engines using TensorRT-LLM. TensorRT’s inflight batching boosts throughput by dynamically grouping inference requests, releasing each sample as soon as it completes—reducing latency and optimizing GPU utilization. See below some recent benchmarks by vLLM team.
\n
Since samples are processed independently, batches can mix different prompts or seeds seamlessly. TensorRT-LLM includes a number of other optimizations such as custom attention kernels and paged KV caching.
\n
For each new problem we used different seeds to launch 12 async generations via in-flight batching in TensorRT. The streaming for each sample was monitored for code blocks, stop phrases, max tokens or timeouts. If the LLM generates code, LLM generation stops, and the code block is executed in a sandbox. The sandbox output (or partial error trace) is appended to the LLM and generation continues. The generation continues until another code block. When no other code blocks are encountered, the LLM stops at one of max tokens, timeout time, or stop phrase.
\nWe experimented with various sampling parameters but observed minimal differences in the results. However, our winning submission was based on an \"almost\" greedy search strategy. Despite setting the temperature to 0 and enabling the redrafter_greedy_search parameter, TensorRT-LLM still produced slightly varying outputs within a single batch. We did not investigate this behavior in detail, but speculate that it could be related to rounding or batching issues. Ultimately, we chose this approach because it provided more stable results at small batch sizes and offered a slight improvement in the speed of speculative decoding.
\nWithin the kaggle notebook, generations are monitored for completion. When initial answers are the same we early-stop, e.g. if 4 of the first 5 generations are the same we cancel the remaining generations in TensorRT.
During monitoring of async generations, we also early stop after completing 10 of the 12 generations to avoid waiting excessively long for any stragglers. A buffering strategy was implemented, allocating 350 seconds per question as the base time limit. If a question completed early, the unused time was added to a shared buffer. The next question could then draw up to 210 extra seconds from this buffer, making a total of 560 seconds.
\n
Using this inference pipeline and deepseek-distill-14b, we scored in the range of 23 to 29 on public leaderboard using different settings. The score variability was concerning and our own trained models helped to stabilise the scores as well as increasing the accuracy.
\nFor the final chosen submissions, one 14b CoT model and the above MIX TIR model were chosen. The MIX TIR model scored much better on CV, and a little better on Public LB (public LB scores : 32,33,28). The one 28 scoring submission made us nervous and was perhaps due to a larger maj@ used. In the end, the private LB reflected more our CV results than LB results which was quite satisfying.
\nWe had other models as seen which were quite strong on CV (see the report) but the same size model produced more tokens than the previous versions, and we believe in public LB they were timing out before having a chance to properly attempt the questions. Ultimately we were out of time and submissions to properly narrow down the discrepancies seen between CV and public LB, especially given the variance with one submission per day and only 50 questions scored.
\nA number of experiments were made but were not in the final submission choices, mostly due to limited time likely causing early exit before a good answer was made,
\nWhat an incredible solution - truly worth of the name SOTA.
\nThere’s so much to learn from it. Thank you for sharing such an inspiring work.
This year's 1st place is even crazier than last year! Congratulations to the whole team with the incredible achievement, and a family of frontier-level reasoning models. The fact that you trained everything from base Qwen (not even reasoning) models with 1.5B beating 14B DeepSeek is bonkers!
\nI have a question: your team is the only one among the top teams that didn't get nerd-sniped by RL (GRPO in particular) and relied only on SFT. Was it intentional, or you didn't have time to perform further tuning? You did mention about your generation length problem, that in theory should be solvable with RL.
", "rawMarkdown": "This year's 1st place is even crazier than last year! Congratulations to the whole team with the incredible achievement, and a family of frontier-level reasoning models. The fact that you trained everything from base Qwen (not even reasoning) models with 1.5B beating 14B DeepSeek is bonkers!\n\nI have a question: your team is the only one among the top teams that didn't get nerd-sniped by RL (GRPO in particular) and relied only on SFT. Was it intentional, or you didn't have time to perform further tuning? You did mention about your generation length problem, that in theory should be solvable with RL.", "votes": 3, "replies": [ { "id": 3187241, "postDate": "2025-04-25T17:45:14.867Z", "content": "We mainly bet on SFT with a large dataset generated by a powerful teacher since there is some evidence that it works better for small models. E.g. DeepSeek-R1 report makes this claim, and we also observed it in our experiments. You're right that to address the generation length we could have used RL. The main reason we didn't is simply because we noticed it too late. With the variance in the submissions and our local tests not fully matching what we saw in LB, it took us some time to realize that longer generations are likely the reason our new models weren't scoring well.
\nIf we were to do it again, I'd say the best recipe would be to start with large-scale SFT and then do RL on top to make models use fewer tokens.
", "rawMarkdown": "We mainly bet on SFT with a large dataset generated by a powerful teacher since there is some evidence that it works better for small models. E.g. DeepSeek-R1 report makes this claim, and we also observed it in our experiments. You're right that to address the generation length we could have used RL. The main reason we didn't is simply because we noticed it too late. With the variance in the submissions and our local tests not fully matching what we saw in LB, it took us some time to realize that longer generations are likely the reason our new models weren't scoring well.\n\nIf we were to do it again, I'd say the best recipe would be to start with large-scale SFT and then do RL on top to make models use fewer tokens.", "votes": 8 } ] }, { "id": 3185902, "postDate": "2025-04-24T02:05:16.950Z", "content": "Great job!
\nLearning from you guys is the reason why I participate in this competition.
Crazy solution. I never saw a kaggle solution takes so much talent, skills, effort and resourses. 🫠
", "rawMarkdown": "Crazy solution. I never saw a kaggle solution takes so much talent, skills, effort and resourses. 🫠 ", "votes": 4 }, { "id": 3194838, "postDate": "2025-05-06T10:59:10.920Z", "content": "Thank you for sharing such a detailed solution and a convenient and comprehensive datasets.
\nI tried running your utility but it failed.
cell:
\n!pip install --no-index --no-deps --upgrade --target=/kaggle/working \\
\n --find-links file:///kaggle/input/tensorrt-llm-aimo-wheels-ns-varlen/ nemo_skills
\nError:
\nLooking in links: file:///kaggle/input/tensorrt-llm-aimo-wheels-ns-varlen/
\nWARNING: Location 'file:///kaggle/input/tensorrt-llm-aimo-wheels-ns-varlen/' is ignored: it is neither a file nor a directory.
\nERROR: Could not find a version that satisfies the requirement nemo_skills (from versions: none)
\nERROR: No matching distribution found for nemo_skills
@planet2020 Sorry about missing this for so long and thanks for checking it… that dataset tensorrt-llm-aimo-wheels-ns-varlen was private but is public now to let you run it.
\nFeel free to dm me if you get any more issues; will check in here to see if you get any more issues.
Quick follow up - @cdeotte helped to tested; it works now - just make sure to turn on the L40 GPU. Tensor parallelism is set to 4, and the model is build for the L40.
\nThanks for alerting us to this.
@darraghdog Thanks for update! I tried again today. Unfortunately it still failed.
\nWhen run this cell:
%%bash
\nexport LD_LIBRARY_PATH=\"/kaggle/working/nvidia/nvjitlink/lib:$LD_LIBRARY_PATH\"
\npython -m tensorrt_llm.commands.build -h
Got an error:
\n/usr/bin/python3: Error while finding module specification for 'tensorrt_llm.commands.build' (ModuleNotFoundError: No module named 'tensorrt_llm')
In the previous cell in that notebook:
\n!pip install --no-index --target=/kaggle/working --find-links file:///kaggle/input/tensorrt-llm-aimo-wheels-ns-varlen/ \\
\n tensorrt_llm==0.17.0.dev2024121700 \\
\n tensorrt-cu12_libs==10.7.0 \\
\n tensorrt-cu12_bindings==10.7.0
Then I check tensorrt-llm dataset. It contains tensorrt_llm-0.19.0 instead of tensorrt_llm-0.17.0. I am not sure if this is where the problem comes from.
\nI tried run tensorrt-llm-aimo-wheels ns varlen notebook but still got an error.
\nWhen run this cell:
\n!pip wheel --no-verify --no-cache-dir -w /kaggle/working \\
\n /kaggle/input/tensorrt-llm/tensorrt_llm-0.17.0.dev2024121700-cp310-cp310-linux_x86_64.whl
Got an error:
\nWARNING: Requirement '/kaggle/input/tensorrt-llm/tensorrt_llm-0.17.0.dev2024121700-cp310-cp310-linux_x86_64.whl' looks like a filename, but the file does not exist
\nERROR: tensorrt_llm-0.17.0.dev2024121700-cp310-cp310-linux_x86_64.whl is not a supported wheel on this platform.
Then I tried replace tensorrt_llm-0.17.0 with 0.19.0 version that exists in the tensorrt-llm dataset but still got an error:
\nERROR: tensorrt_llm-0.19.0.dev2025032500-cp310-cp310-linux_x86_64.whl is not a supported wheel on this platform.
I have no idea how to run it without an error.
", "rawMarkdown": "@darraghdog Thanks for update! I tried again today. Unfortunately it still failed.\nWhen run this cell:\n\n%%bash\nexport LD_LIBRARY_PATH=\"/kaggle/working/nvidia/nvjitlink/lib:$LD_LIBRARY_PATH\"\npython -m tensorrt_llm.commands.build -h\n\nGot an error:\n/usr/bin/python3: Error while finding module specification for 'tensorrt_llm.commands.build' (ModuleNotFoundError: No module named 'tensorrt_llm')\n\nIn the previous cell in that notebook:\n!pip install --no-index --target=/kaggle/working --find-links file:///kaggle/input/tensorrt-llm-aimo-wheels-ns-varlen/ \\\n tensorrt_llm==0.17.0.dev2024121700 \\\n tensorrt-cu12_libs==10.7.0 \\\n tensorrt-cu12_bindings==10.7.0\n\nThen I check tensorrt-llm dataset. It contains tensorrt_llm-0.19.0 instead of tensorrt_llm-0.17.0. I am not sure if this is where the problem comes from. \nI tried run tensorrt-llm-aimo-wheels ns varlen notebook but still got an error.\nWhen run this cell:\n!pip wheel --no-verify --no-cache-dir -w /kaggle/working \\\n /kaggle/input/tensorrt-llm/tensorrt_llm-0.17.0.dev2024121700-cp310-cp310-linux_x86_64.whl\n\nGot an error:\nWARNING: Requirement '/kaggle/input/tensorrt-llm/tensorrt_llm-0.17.0.dev2024121700-cp310-cp310-linux_x86_64.whl' looks like a filename, but the file does not exist\nERROR: tensorrt_llm-0.17.0.dev2024121700-cp310-cp310-linux_x86_64.whl is not a supported wheel on this platform.\n\nThen I tried replace tensorrt_llm-0.17.0 with 0.19.0 version that exists in the tensorrt-llm dataset but still got an error:\nERROR: tensorrt_llm-0.19.0.dev2025032500-cp310-cp310-linux_x86_64.whl is not a supported wheel on this platform.\n\nI have no idea how to run it without an error.", "votes": 1 }, { "id": 3202326, "postDate": "2025-05-15T08:06:37.543Z", "content": "The lines below is from the linked tensorrt-llm installation script tensorrt-llm-aimo-wheels-ns-varlen,
\n%%bash\nexport LD_LIBRARY_PATH=\"/kaggle/working/nvidia/nvjitlink/lib:$LD_LIBRARY_PATH\"\npython -m tensorrt_llm.commands.build -h
\nYou are right, the linked tensorrt_llm pip wheel dataset has been upgraded to 0.19.0 in the meantime. Please give us a bit of time to get this sorted out for you.
In the meantime, if you want to run the main submission book without reinstalling tensorrt_llm, you should be able to just re-fork it (Copy and edit) in the drop down beside the edit button. This should just use the utility script we attached (the original v05 version is now pinned as default), and as long as L40 is turned on for GPU, this should run. See a test here - its running now, should be done in an hour.
\nThanks again for verifying this.
", "rawMarkdown": "The lines below is from the linked tensorrt-llm installation script [tensorrt-llm-aimo-wheels-ns-varlen](https://www.kaggle.com/code/darraghdog/tensorrt-llm-aimo-wheels-ns-varlen/notebook?scriptVersionId=239830711), \n```%%bash\nexport LD_LIBRARY_PATH=\"/kaggle/working/nvidia/nvjitlink/lib:$LD_LIBRARY_PATH\"\npython -m tensorrt_llm.commands.build -h```\nYou are right, the linked [tensorrt_llm pip wheel dataset](https://www.kaggle.com/datasets/sorokin/tensorrt-llm/data) has been upgraded to `0.19.0` in the meantime. Please give us a bit of time to get this sorted out for you.\n\nIn the meantime, if you want to run the main [submission book](https://www.kaggle.com/code/darraghdog/nemo-skills-qwen2-5-14b-mix48-trtllm-redrafter?scriptVersionId=228756964) without reinstalling tensorrt_llm, you should be able to just re-fork it (Copy and edit) in the drop down beside the edit button. This should just use the utility script we attached (the original v05 version is now pinned as default), and as long as L40 is turned on for GPU, this should run. See a test [here](https://www.kaggle.com/code/darraghdog/rerun-nemo-skills-qwen2-5-14b-mix48-trtllm-redra?scriptVersionId=239834279) - its running now, should be done in an hour. \n\nThanks again for verifying this. " }, { "id": 3202413, "postDate": "2025-05-15T10:29:51.657Z", "content": "Thank you for your help. I know that your utility works for running your notebook. But I want to be able to install tensorrt_llm in Kaggle by myself as well as other libraries used for your submission notebook so that any changes of your utility in the future will not affect my notebook.
", "rawMarkdown": "Thank you for your help. I know that your utility works for running your notebook. But I want to be able to install tensorrt_llm in Kaggle by myself as well as other libraries used for your submission notebook so that any changes of your utility in the future will not affect my notebook.", "votes": 1 }, { "id": 3202432, "postDate": "2025-05-15T11:04:49.807Z", "content": "So the levels we have are,
\n0.17.0.dev2024121700 version. pip-install-tensorrt-llm-ns-varlen2 - we made a new book to test it all works. Uses step 2 as input. f'export LD_LIBRARY_PATH=\"/kaggle/usr/lib/pip_install_tensorrt_llm_ns_varlen2/nvidia/nvjitlink/lib:$LD_LIBRARY_PATH\" &&'Also, if you have a new tensorrt_llm model, make sure it builds on an L4/L40/L40S for best performance.
\nAnd let us know how it goes.
Big thanks to @sorokin for helping through this! I forgot how complicated it is.
", "rawMarkdown": "So the levels we have are,\n1. Tensorrt_LLM [pip wheel](https://www.kaggle.com/datasets/sorokin/tensorrt-llm) which is used in step 2. This is a dataset. \n2. [tensorrt-llm-aimo-wheels ns varlen](https://www.kaggle.com/code/darraghdog/tensorrt-llm-aimo-wheels-ns-varlen/notebook) script. This builds wheel, dependencies, clones the correct branch of NeMo-Skills and builds wheels for all. **Important:** When you attach step 1, you need to use right-click and \"Pin to Version\" and choose the `0.17.0.dev2024121700` version. \n3. [Wheel installation utility script](https://www.kaggle.com/code/darraghdog/fork-of-pip-install-tensorrt-llm-ns-varlen2/notebook?scriptVersionId=239859898) - `pip-install-tensorrt-llm-ns-varlen2` - we made a new book to test it all works. Uses step 2 as input. \n4. Submission book. There is one running [now](https://www.kaggle.com/code/darraghdog/rerun2-nemo-skills-qwen2-5-14b-mix48-trtllm?scriptVersionId=239864263) with the new step 3 book. **Important:** If you use a new notebook from step 3; please make sure to rename the line below to point to your new step 3 utility script. \n` f'export LD_LIBRARY_PATH=\"/kaggle/usr/lib/pip_install_tensorrt_llm_ns_varlen2/nvidia/nvjitlink/lib:$LD_LIBRARY_PATH\" &&'`\n\nAlso, if you have a new `tensorrt_llm` model, make sure it builds on an L4/L40/L40S for best performance. \nAnd let us know how it goes. \n\nBig thanks to @sorokin for helping through this! I forgot how complicated it is. \n", "votes": 2 }, { "id": 3202834, "postDate": "2025-05-16T02:10:55.457Z", "content": "@darraghdog It works now. Thank you very much for your guidance.😀
\nI am wondering what reason for you to choose 0.17.0 version instead of 0.19.0? Is it possible to install new version such as 0.19.0 on Kaggle?
We did a lot of experiments on this 0.17.x version and had everything set up with it. I believe 0.19.0 was released towards the end or after the competition, but 0.18.0 was out there. We did do some experiments with it, but it would have meant rebuilding older models and the environment explained above with the new version - with more time we would have done it for sure.
If you are interest to try building the nvidia/OpenMathReasoning, or the Fast-OpenMath-Nemotron-14B models in the inference pipeline with the new pipeline it could be interesting. We work on docs for doing this, just to make a bit simpler, and could share. It would be without ReDrafter, but you could try building one of those models and the pipeline in 0.19.0. Without ReDrafter, it would probably make sense to try the Fast-OpenMath-Nemotron-14B.
Got it. I will definitely try it when I have time. Thank you very much for your patient guidance and I learned a lot from it. Congratulations on winning the first place in AIMO2.
", "rawMarkdown": "Got it. I will definitely try it when I have time. Thank you very much for your patient guidance and I learned a lot from it. Congratulations on winning the first place in AIMO2." } ] } ] }, { "id": 3189641, "postDate": "2025-04-29T14:23:37.553Z", "content": "Great job!
\nIt is a pleasure to read the magnum opus of such a large number of the best specialists!
", "rawMarkdown": "Great job! \n\nIt is a pleasure to read the magnum opus of such a large number of the best specialists!", "votes": 1 }, { "id": 3189079, "postDate": "2025-04-28T17:41:41.340Z", "content": "This is top notch. Congratulations and thanks a lot for the dataset released. 🙌
\nI am very very happy for noticing one small thing in your solution. You used both prompt+question in the user role. May I know the reason behind this? I even asked the second place team on why they didn't use it, they used both user and system role
\nDid your team find any improvements on doing this? I got some bumps in my LB by changing this. Of course I first saw this in many threads, posts on other communities and the model card on deepseek itself, but not a single public nb had this!
But I really want to know your thoughts on this 🤔
", "rawMarkdown": "This is top notch. Congratulations and thanks a lot for the dataset released. 🙌\n\nI am very very happy for noticing one small thing in your solution. You used both prompt+question in the user role. May I know the reason behind this? I even asked the second place team on why they didn't use it, they used both user and system role\nDid your team find any improvements on doing this? I got some bumps in my LB by changing this. Of course I first saw this in many threads, posts on other communities and the model card on deepseek itself, but not a single public nb had this!\n\nBut I really want to know your thoughts on this 🤔", "votes": 1, "replies": [ { "id": 3189857, "postDate": "2025-04-29T21:31:15.180Z", "content": "This is how we trained the model, to have both prompt and question inside the user part. When you do SFT, it's largely your own preference as the model will learn just fine whether you put everything in the system or the user part or split it. After all, it's just a few tokens difference, so as long as you match the setup between training and inference, it doesn't really matter
", "rawMarkdown": "This is how we trained the model, to have both prompt and question inside the user part. When you do SFT, it's largely your own preference as the model will learn just fine whether you put everything in the system or the user part or split it. After all, it's just a few tokens difference, so as long as you match the setup between training and inference, it doesn't really matter", "votes": 1 } ] }, { "id": 3187072, "postDate": "2025-04-25T14:15:57.167Z", "content": "Congratulations man!
", "rawMarkdown": "Congratulations man!", "votes": 1 }, { "id": 3186964, "postDate": "2025-04-25T11:00:35.607Z", "content": "Great job!
\nYour open data is a huge help to the open-source community.
\nIt’ll really benefit researchers who can’t build such large datasets themselves, just like the Numina team in the last competition.
Congratulations on First Prize in this competition. Can you please advise the running time of your notebook. Appreciate your observation on why your notebook did not solve correctly the remaining problems.
", "rawMarkdown": "Congratulations on First Prize in this competition. Can you please advise the running time of your notebook. Appreciate your observation on why your notebook did not solve correctly the remaining problems. ", "votes": 1, "replies": [ { "id": 3186956, "postDate": "2025-04-25T10:43:15.873Z", "content": "Thanks, you can check it in the Kaggle submission script. On LB, hard to tell due to runtime being obfuscated.
\nEach question has stats like below printed. For example total_num_generated_tokens is the LLM generated tokens for each of the 12 samples. cumulative time is time since start of script. We watched this quite a bit. Kaggle L4 was slower than our testing L4 for some reason (circa 10 to 20%). We also ran our CV sets in kaggle to check time - a big team luckily meant bigger gpu quota to help out 😃
88c219\n\n'<|im_start|>system<|im_end|><|im_start|>userSolve the following math problem. Make sure to put the answer (and only answer) inside boxed{}.For positive integers $x_1,ldots, x_n$ define (x_1, ldots, x_n)$ to be the sum of their $frac{n(n-1)}{2}$ pairwise greatest common divisors. We say that an integer $n geq 2$ is emph{artificial} if there exist $n$ different positive integers $a_1, ..., a_n$ such that [a_1 + cdots + a_n = G(a_1, ldots, a_n) +1.]Find the sum of all artificial integers $m$ in the range $2 leq m leq 40$.If the final answer is a number larger than 1000, take modulo 1000.<|im_end|><|im_start|>assistant'\n\nFinish times: 254 307 552 561 561 562 562 562 563 563 563 563 Cancelling @ 563\ntotal_num_generated_tokens [9550, 11381, 18992, 18112, 18444, 18243, 17759, 18565, 17498, 17945, 18892, 18746]\ngeneration_time [253, 306, 551, 561, 561, 562, 558, 561, 559, 562, 562, 563]\ncode_rounds_executed [0, 1, 0, 0, 0, 0, 3, 0, 4, 0, 0, 0]\ncode_execution_time [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n\nall extracted answers: [0, 6]\n\nSolved with answer 0; correct answer 810. Time taken 563; new buffer time 700; cumulative time 2504\nscore: 6/8\nvery interesting and clever investigation
", "rawMarkdown": "very interesting and clever investigation", "votes": 1 }, { "id": 3190158, "postDate": "2025-04-30T08:00:53.977Z", "content": "wow! its so easy
", "rawMarkdown": "wow! its so easy", "votes": 2 }, { "id": 3186392, "postDate": "2025-04-24T15:38:04.897Z", "content": "Congratulation for your great win and Thank you for your great sharing.
\nInitially thought of building on top of your CoT and TIR model with some merging ideas, but sadly those models was not release ☹ .
\nUnderstood that nVidia might have some legal reasons that prevent you from releasing certain things fully under open source, but can you consider at least open source the TIR and CoT model for the Kaggle related version.
\nIf releasing it on HF is not allowed, maybe you can discuss it with @friederrr to host it on Kaggle Model as part of AIMO full open-source open science reproducibility initiative.
\nThis is not only good for effort to reproduce or innovate on top of it, but also aid in apple-to-apple comparison with other’s open source release under non-merging setting.
\nNumina did it in AIMO1 : https://huggingface.co/collections/AI-MO/numinamath-6697df380293bcfdbc1d978c and attracted good open reproduce effort: https://www.kaggle.com/competitions/ai-mathematical-olympiad-prize/discussion/523210
\nHope the tradition can continue.
\nLast but not least, please do share to us the merging code as well, it is a good reference.
", "rawMarkdown": "Congratulation for your great win and Thank you for your great sharing.\n\nInitially thought of building on top of your CoT and TIR model with some merging ideas, but sadly those models was not release ☹ .\n\nUnderstood that nVidia might have some legal reasons that prevent you from releasing certain things fully under open source, but can you consider at least open source the TIR and CoT model for the Kaggle related version.\n\nIf releasing it on HF is not allowed, maybe you can discuss it with @friederrr to host it on Kaggle Model as part of AIMO full open-source open science reproducibility initiative.\n\nThis is not only good for effort to reproduce or innovate on top of it, but also aid in apple-to-apple comparison with other’s open source release under non-merging setting. \n\nNumina did it in AIMO1 : https://huggingface.co/collections/AI-MO/numinamath-6697df380293bcfdbc1d978c and attracted good open reproduce effort: https://www.kaggle.com/competitions/ai-mathematical-olympiad-prize/discussion/523210\n\nHope the tradition can continue.\n\nLast but not least, please do share to us the merging code as well, it is a good reference.\n", "votes": 2, "replies": [ { "id": 3186481, "postDate": "2025-04-24T18:09:30.610Z", "content": "The MIX model we used in kaggle, as well as the datasets for training are released. Check the model marked kaggle in the Collection of Math Reasoning models linked above.
\nTo rerun inference you need to convert to tensorrt_llm - see qwen conversion. This build would be without redrafter - but you could use the trtllm build and follow along what happens in the submission script.
\nWe plan to make some easy to follow tutorials to add redrafter but you can run without this; if you wish to quantise, maybe try the int8_wo (INT8 weight-only quantization) in the qwen conversion and you should get good speed up.
I refer to the before-merging TIR and CoT model (e.g model CoT & TIR in Table 10 of your paper), the one you released is after-merged final model.
\nYou can refer to Numina AIMO1 Winner release for better understanding:
\nTheir final model is NuminaMath-7B-TIR , but they also released NuminaMath-7B-CoT which is the intermediate trained model .
\nhttps://huggingface.co/collections/AI-MO/numinamath-6697df380293bcfdbc1d978c
There are few reasons from open science research point of view that this is important:
\nSome other open source released models is only trained on CoT or TIR without merging, apple-to-apple comparison can only been perform vs your non-merged CoT or TIR model, not vs the final merged model.
\nThe goal is to evaluate the impact of training dataset alone before the effect of merging.
Researcher can try out different merging ideas to see if better result can be achieve with your non-merged CoT/TIR model.
Some community member might just want to reproduce the model merging part.
We discussed internally and we will not release externally the intermediate model weights at the moment. There is some admin work for it, and it would mean more models on hf which need docs to explain they are from a specific part of the pipeline etc.; and kaggle when we host there. I know it was done last year, but we have a quite a different solution and architecture (trtllm based TIR/specdecoding) than last years winners. I could suggest you take a CoT model like deepseek-14b and mix with our TIR model; the details of the mixing are in the paper.
\nAlternatively you could try a small finetuning of a Math Reasoning CoT model (QwQ or dseek), using ~1k samples for the data we provide and create your own TIR model - and mix that with the original to reproduce the work.
\nI appreciate this is not the answer you are hoping for, but hopefully there is enough released between the new data, family of models that you can try some experiment on different parts.
@binbinai Thanks for raising these points, in particular 3. We are following this up with the NemoSkills team to assess if this is an impediment to reproducibility, and whether this is inherently unresolvable or not.
", "rawMarkdown": "@binbinai Thanks for raising these points, in particular 3. We are following this up with the NemoSkills team to assess if this is an impediment to reproducibility, and whether this is inherently unresolvable or not.", "votes": 1 }, { "id": 3201939, "postDate": "2025-05-14T16:17:16.123Z", "content": "@friederrr (fyi @binbiai ) as per our mail, we trained a CoT-only model, then finetuned it further on TIR data and then merged the two models. We released all the data we used as well as all details needed for training, so anyone can rerun SFT, get both models and merge them to reproduce the final checkpoint.
\nThanks again for all AIMO's effort here.
Good job man!
\nBut how?
I am here for such write-ups, truly a great work, thanks for sharing and congratulations!
", "rawMarkdown": "I am here for such write-ups, truly a great work, thanks for sharing and congratulations!" }, { "id": 3467809, "postDate": "2026-06-07T13:39:32.157Z", "content": "Type of writeups and people that I would love to get schooled by. Congratulations.
", "rawMarkdown": "Type of writeups and people that I would love to get schooled by. Congratulations." }, { "id": 3193128, "postDate": "2025-05-03T22:23:24.167Z", "content": "great job man!
", "rawMarkdown": "great job man!" }, { "id": 3187914, "postDate": "2025-04-26T17:41:13.130Z", "content": "great job man!
", "rawMarkdown": "great job man!" }, { "id": 3185887, "postDate": "2025-04-24T01:29:19.903Z", "rawMarkdown": "", "votes": 9, "isDeleted": true }, { "id": 3186990, "postDate": "2025-04-25T12:00:25.240Z", "content": "Great Work !
", "rawMarkdown": "Great Work ! ", "votes": 1 }, { "id": 3185956, "postDate": "2025-04-24T03:40:51.060Z", "content": "congratulations
", "rawMarkdown": "congratulations\n", "votes": 1 }, { "id": 3189733, "postDate": "2025-04-29T16:54:03.267Z", "content": "Congratulations!
", "rawMarkdown": "Congratulations!" }, { "id": 3186639, "postDate": "2025-04-25T01:50:27.520Z", "content": "Thank you. I learned more
", "rawMarkdown": "Thank you. I learned more" } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T06:42:25Z", "competition_id": 86023, "competition_slug": "ai-mathematical-olympiad-progress-prize-2", "team_id": 12938789, "solution_url": "/competitions/ai-mathematical-olympiad-progress-prize-2/writeups/nemoskills-1st-place-solution-nemoskills" }