![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F75976%2Ff4f759b7249fc2c56a507a65fee16fe6%2FScreenshot%202025-12-04%20at%2020-36-51%20ARC-AGI-2%20NVARC%20writeup%20-%20Google%20Docs.png?generation=1764877039515916&alt=media\")
The main components of our solution:
\nOur overall workflow can be summarized by this picture. We will review each part in the rest of this writeup.
\n
We build a Synthetic Data Generation (SDG) pipeline with 4 main stages:
\nThe majority of generated data is made with gpt-oss-120b model using the NeMo-Skills framework. This framework provides many tools for LLM development, including large-scale SDG pipelines on Slurm cluster. You can find more information how to use this framework for SDG here. Running this model on a single node with 8xH100 GPUs gives us 15k tokens/s generation throughput.
We collect the puzzle descriptions from two main sources. First, we found H-ARC, which was surprisingly underestimated in the ARC community. This dataset contains solution attempts from over 1700 humans on ARC problems, but more importantly it includes the natural-language solution descriptions. Second, we took 160 human written descriptions from BARC. After filtering and combining these descriptions we had descriptions for 716 training puzzles from ARC-AGI-2.
\nWe prompted LLM to join two puzzle summaries and produce a new more complex puzzle by combining elements from both. In total, we generated 266593 new puzzle summaries.
\nInstead of generating a full puzzle program with input/output logic, we decided to split this into two stages. First - generate a Python program for the input grid logic, second - generate a program of the output grid logic, i.e. transformation rules. To generate reliable and good input grid logic we prompt the LLM to produce additional unit test for the input grid. In total, we generated and filtered 126901 input grid programs in Python code.
\nWe prompt LLM with puzzle description and corresponding input grid program to generate the relevant output grid program. Here we apply a different validation strategy. Instead of validating Python code with unit tests, we prompt the LLM multiple times for each input grid logic and filter out solutions which are not consistent. After filtering we had 103253 puzzles, where each puzzle has 1 input grid program and multiple output grid programs, but all of these output programs produce consistent output grids.
\nFor this approach we try to use as many puzzles as possible. In addition to our synthetic data we also used a few real puzzles datasets. Our final dataset includes 3.2M augmented samples, where each sample has up to 7 input/output pairs.
\n| Source | \nUnique puzzles | \nAugmented samples \\ per puzzle | \nTotal samples | \nShare | \n
|---|---|---|---|---|
| MINI-ARC | \n147 | \n256 | \n37632 | \n1.2 | \n
| ConceptARC | \n160 | \n256 | \n40960 | \n1.3 | \n
| RE-ARC | \n400 | \n256 | \n102392 | \n3.2 | \n
| ARC-AGI-2 | \n609 | \n256 | \n155904 | \n4.8 | \n
| NVARC training | \n47337 | \n24 | \n1132633 | \n34.8 | \n
| NVARC full | \n55886 | \n32 | \n1785960 | \n54.9 | \n
| \n | 104539 | \n\n | 3255481 | \n\n |
ARC-AGI-2 training subset excludes RE-ARC puzzles. NVARC training subset based only on training puzzles from ARC-AGI-2. NVARC full subset uses both training and evaluation puzzles from ARC-AGI-2.
\nThe ARChitects approach has a simple representation of puzzles as a list of input/output grids. But we made it even simpler and used a dialog style template from Qwen3 model. For example, formatted message represents 1 pair of input/output grids looks like this:
\n<|im_start|>user\\n123\\n456<|im_end|>\n<|im_start|>assistant\\n78\\n90<|im_end|>\nThis simpler representation requires only 16 tokens: 10 - for digits, 1 - new line, “user” - start of input grid, “assistant” - start of output grid, 2 special tokens (<|im_start|>, <|im_end|>) and 1 for padding <|endoftext|>. We used the NeMo RL framework for the post-training stage. We run supervised fine-tuning with Megatron backend which allows us to efficiently utilize memory and computation resources of multiple nodes with H100 GPUs. For example, to do a full fine-tuning of the 4B model we used 4 nodes of 8xH100 for 27 hours.
We used LoRA test-time fine-tuning for each puzzle independently with r = 256 and alpha = 32. We removed gradient checkpointing and we removed 4-bit quantization too. Test-time fine-tuning was run with bfloat16 precision. We also used Flash Attention 2 with Unsloth Framework. You can find all hyperparameters in our source code.
\nThe main optimization we made in the ARChitects approach is for the decoding stage. We implemented the batch version of the Depth First Search (DFS) algorithm. However, there is one downside effect of batch implementation. It became nondeterministic. This phenomenon is well explained in a blog from Thinking Machines Lab. We used an open-sourced solution from Thinking Machines Lab and managed to make inference batch invariant. However, despite the better precisions, and better local validations scores, this version runs much longer in the Kaggle environment. About 17% slower, and we didn’t use it in the final submission.
\nThe ARChitects approach uses additional augmentations to rescore the candidates from DFS stage. We made a small change here. We used only 8 augmentations per candidate solution, but we used exactly the same augmentations for each candidate solution. This makes the scores of different solutions more comparable. After the competition deadline we tested the different re-scoring strategies and found the better method to select the right candidate. We calculate how many times the candidate solution was found during the DFS stage and adjust this with the geometric mean ensemble\nof log-probabilities from different augmentations.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F502311%2F061f66e26ce71349a1fab3a2d9adc778%2Fnvarc_evaluation.png?generation=1764873955755619&alt=media\")
During the competition we fine-tuned models on different portions of synthetic data. In the figure you can see the effect from adding more data on the loss function during the pretraining stage. The local validation loss measured on an augmented version of 120 evaluation puzzles, and we saw a good correlation with public leaderboard. Our best model scored 27.64% during the competition.
\nDuring the last 10 days or so of the competition we explored the use of the Tiny Recursive Models (TRM) by Alexia Jolicoeur-Martineau.
\nThe day before last we managed to get a working submission that ensembles TRM with the Qwen3 2B model. We lacked time to tune the ensemble and the TRM model itself, but we were glad to be able to test the idea still. We used a few late submissions to finish testing the ensemble and report on these as well. We only used models pretrained during the competition in these late submissions.
\nUsing the code, the ARC Prize organization got a score of 6.9 percents by adapting it to run on 4 nodes of 8 H100 GPUs. Using that amount of compute resources is impossible in Kaggle. To make it worse, we could only use 2 hours or so for TRM given we had to use the rest for the ARChitects approach. Therefore, we had to use a different method. We first pretrained the TRM model, then fine-tuned the TRM model using test data during submission.
\nUsing a batch size of 3072 instead of 768, and learning rate of 3e-4 instead of 1e-4 we could train TRM to the same accuracy as the original code in 24 hours on 8xH100. Using this in a submission yields a score of 2.08% only.
\nWe then tuned various parameters to get the best score while using at most 2 hours of running time on Kaggle. We ended up with these parameters: 4 H cycles instead of the default 3, and 10 halt max steps instead of 16. We used 2000 epochs and only 200 warmup steps. This makes TRM run in about 2 hours on Kaggle.
\nIn order to assess the strength of the TRM model we trained it twice. We trained it with the 4k puzzles and used that for submission. And we also trained it with the competition evaluation dataset removed. We then scored the latter on the evaluation data. We then looked at how many training steps were used for the checkpoint with best pass@2. We finally selected the checkpoint trained on all 4k samples with the closest number of training steps. Best pass@2 on evaluation with a model not trained on evaluation was 9.44%. When submitted the matching checkpoint trained with evaluation scored 7.5%. This is much better than the 2.08% we had before.
\nAfter the competition deadline we submitted our best TRM using 4k epochs instead of 2k. It yield a score of 10.0%, in less than 4 hours.
\nIn order to use TRM with the ARChitects approach we modified TRM so that it generates 10 attempts instead of 2 per puzzle. These attempts were then added to the attempts generated by The ARChitects procedure. They were then scored by the Qwen3 model like the other attempts.
\nThis worked fine but with mixed results. Most of the puzzles solved by TRM were solved by Qwen3, hence TRM added nothing there. However, about 2 or 3 puzzles solved by TRM were not solved by Qwen3. Unfortunately, these were not always picked by Qwen3 scoring.
\nWith two late submissions we found that we could improve the 21.53 score obtained with Qwen3 2B to 22.50 with a TRM ensemble. We also saw that when using a Qwen3 4B submission that uses 10 hours only, with a score 27.22, adding TRM yields the same 27.22 score.
\nWhile we think that there is some potential for improvement here, ensembling the Qwen3 4B model and TRM model would require more investigations and experiments than we could perform.
\nWe presented how we improved The ARChitects and TRM models to get the best score on the public leaderboard of ARC Prize 2025 competition. With limited resources during the evaluation stage we needed a good way to compress learned knowledge and skills. Scaling the pretraining stage with good synthetic puzzles was the right direction to succeed in this Kaggle competition. But we also believe that this synthetic data with python programs and reasoning traces could be used for other research directions.
", "comments": [ { "id": 3363444, "author_name": "Cody_Null", "author_url": "", "post_date": "2025-12-05T23:23:42.337000", "content": "Some of the most impressive work I have seen in a long time, can’t wait to review this more
", "votes": 3, "replies": [] }, { "id": 3363408, "author_name": "Giba", "author_url": "", "post_date": "2025-12-05T22:53:19.053000", "content": "Congratulations @sorokin and @cpmpml ! This is a incredible win!
\nGreat writeup, and not only a big gap to 2nd place, but real artificial intelligence applied to an extremely challenging task.
", "votes": 3, "replies": [] }, { "id": 3364576, "author_name": "Eduardo Rocha de Andrade", "author_url": "", "post_date": "2025-12-06T15:07:21.957000", "content": "Insane results, the LB gap is really impressive!! 🔥\nCongratulations and thanks for the write up!
", "votes": 1, "replies": [] }, { "id": 3363640, "author_name": "jojobagel", "author_url": "", "post_date": "2025-12-06T03:33:09.360000", "content": "Thanks for open-sourcing the training data and even the scripts generating them! They will be extremely valuable for future research on ARC AGI.
", "votes": 1, "replies": [] }, { "id": 3363222, "author_name": "Texxi", "author_url": "", "post_date": "2025-12-05T20:17:59.263000", "content": "No problem. Maybe next year I'll show you my Artifical Super Intelligence which can solve all publicly available 51'454'560 training tasks in milliseconds…
", "votes": -1, "replies": [] }, { "id": 3364624, "author_name": "Komal Mishra", "author_url": "", "post_date": "2025-12-06T15:39:00.010000", "content": "Instead of generating a full puzzle program with input/output logic, we decided to split this into two stages. First - generate a Python program for the input grid logic, second - generate a program of the output grid logic, i.e. transformation rules. To generate reliable and good input grid logic we prompt the LLM to produce additional unit test for the input grid. In total, we generated and filtered 126901 input grid programs in Python code.
", "votes": -2, "replies": [ { "id": 3367340, "author_name": "Navneet", "author_url": "", "post_date": "2025-12-08T11:16:36.877000", "content": "Generate a Python program for the input grid logic?
", "votes": 0, "replies": [] } ] }, { "id": 3376158, "author_name": "vialactea", "author_url": "", "post_date": "2025-12-13T19:07:09.403000", "content": "Thanks for sharing and congratulations for the 1st place and innovative solution. Well done!!!
\nI'd like to ask you a couple of questions:
\nThanks!
\nThank you for the ARChitects and TRM models!
", "votes": 0, "replies": [] }, { "id": 3369924, "author_name": "Tiberiu Musat", "author_url": "", "post_date": "2025-12-10T13:00:44.973000", "content": "Amazing work! How many tokens did it take you to generate one task?
", "votes": 0, "replies": [ { "id": 3370013, "author_name": "CPMP", "author_url": "", "post_date": "2025-12-10T14:19:27.443000", "content": "about one per cell in the grid plus one per row.
", "votes": 0, "replies": [ { "id": 3372137, "author_name": "Tiberiu Musat", "author_url": "", "post_date": "2025-12-11T20:30:51.303000", "content": "How about the synthetic tasks? How many gpt-oss tokens did you consume to generate those 50k synthetic tasks? Were you able to generate all that synthetic data using a single 8xH100 node?
We provide all generated texts here https://www.kaggle.com/datasets/sorokin/nvarc-artifacts-puzzles You can estimate tokens from it.
", "votes": 1, "replies": [] } ] } ] }, { "id": 3367566, "author_name": "Abhishek Awasthi", "author_url": "", "post_date": "2025-12-08T14:50:55.830000", "content": "thats impressive
", "votes": 0, "replies": [] }, { "id": 3366921, "author_name": "sayoulala", "author_url": "", "post_date": "2025-12-08T04:56:48.827000", "content": "Excellent and well-structured solution! Really appreciate you open-sourcing this. I learned a tremendous amount from it.
", "votes": 0, "replies": [] }, { "id": 3363073, "author_name": "xz", "author_url": "", "post_date": "2025-12-05T18:47:54.627000", "content": "Congrats on the achievement! I am a little bit surprised TRM wasn't a major contributing factor to your team's crazy climb during the last few days.
", "votes": 0, "replies": [ { "id": 3363457, "author_name": "CPMP", "author_url": "", "post_date": "2025-12-05T23:45:40.377000", "content": "\n\nI am a little bit surprised TRM wasn't a major contributing factor to your team's crazy climb
\n
Why? Arc prize team got less than 7% with TRM using 3x8 H100 for a day. We were pleased to see 10% in 2 hours or so with 4 L4.
\nModel ensembling is not about having lots of great models.
\nEnsembling is about having one great model (here Qwen 4B) and other models that can add a bit to it (TRM).
", "votes": 2, "replies": [ { "id": 3363498, "author_name": "xz", "author_url": "", "post_date": "2025-12-06T01:00:00.437000", "content": "I was under the wrong impression that the one would try to add the TRM style recurrent loops on top of their \"main model\". But upon further reading, it seems that having a pretrained natural language backbone is crucial to your approach?
", "votes": 0, "replies": [] }, { "id": 3364228, "author_name": "CPMP", "author_url": "", "post_date": "2025-12-06T10:53:34.717000", "content": "Recurring over the main model is expensive. We used all the 12 hours we got, and we could not run anything lower.
", "votes": 1, "replies": [] } ] } ] }, { "id": 3367313, "author_name": "Irakoze Ntawigenga Kelly", "author_url": "", "post_date": "2025-12-08T10:48:26.913000", "content": "Thank you for the write up!
", "votes": 2, "replies": [] } ], "comment_count": 21, "messages": [ { "id": 3362416, "postDate": "2025-12-04T17:21:29.607Z", "content": "The main components of our solution:
\nOur overall workflow can be summarized by this picture. We will review each part in the rest of this writeup.
\n
We build a Synthetic Data Generation (SDG) pipeline with 4 main stages:
\nThe majority of generated data is made with gpt-oss-120b model using the NeMo-Skills framework. This framework provides many tools for LLM development, including large-scale SDG pipelines on Slurm cluster. You can find more information how to use this framework for SDG here. Running this model on a single node with 8xH100 GPUs gives us 15k tokens/s generation throughput.
We collect the puzzle descriptions from two main sources. First, we found H-ARC, which was surprisingly underestimated in the ARC community. This dataset contains solution attempts from over 1700 humans on ARC problems, but more importantly it includes the natural-language solution descriptions. Second, we took 160 human written descriptions from BARC. After filtering and combining these descriptions we had descriptions for 716 training puzzles from ARC-AGI-2.
\nWe prompted LLM to join two puzzle summaries and produce a new more complex puzzle by combining elements from both. In total, we generated 266593 new puzzle summaries.
\nInstead of generating a full puzzle program with input/output logic, we decided to split this into two stages. First - generate a Python program for the input grid logic, second - generate a program of the output grid logic, i.e. transformation rules. To generate reliable and good input grid logic we prompt the LLM to produce additional unit test for the input grid. In total, we generated and filtered 126901 input grid programs in Python code.
\nWe prompt LLM with puzzle description and corresponding input grid program to generate the relevant output grid program. Here we apply a different validation strategy. Instead of validating Python code with unit tests, we prompt the LLM multiple times for each input grid logic and filter out solutions which are not consistent. After filtering we had 103253 puzzles, where each puzzle has 1 input grid program and multiple output grid programs, but all of these output programs produce consistent output grids.
\nFor this approach we try to use as many puzzles as possible. In addition to our synthetic data we also used a few real puzzles datasets. Our final dataset includes 3.2M augmented samples, where each sample has up to 7 input/output pairs.
\n| Source | \nUnique puzzles | \nAugmented samples \\ per puzzle | \nTotal samples | \nShare | \n
|---|---|---|---|---|
| MINI-ARC | \n147 | \n256 | \n37632 | \n1.2 | \n
| ConceptARC | \n160 | \n256 | \n40960 | \n1.3 | \n
| RE-ARC | \n400 | \n256 | \n102392 | \n3.2 | \n
| ARC-AGI-2 | \n609 | \n256 | \n155904 | \n4.8 | \n
| NVARC training | \n47337 | \n24 | \n1132633 | \n34.8 | \n
| NVARC full | \n55886 | \n32 | \n1785960 | \n54.9 | \n
| \n | 104539 | \n\n | 3255481 | \n\n |
ARC-AGI-2 training subset excludes RE-ARC puzzles. NVARC training subset based only on training puzzles from ARC-AGI-2. NVARC full subset uses both training and evaluation puzzles from ARC-AGI-2.
\nThe ARChitects approach has a simple representation of puzzles as a list of input/output grids. But we made it even simpler and used a dialog style template from Qwen3 model. For example, formatted message represents 1 pair of input/output grids looks like this:
\n<|im_start|>user\\n123\\n456<|im_end|>\n<|im_start|>assistant\\n78\\n90<|im_end|>\nThis simpler representation requires only 16 tokens: 10 - for digits, 1 - new line, “user” - start of input grid, “assistant” - start of output grid, 2 special tokens (<|im_start|>, <|im_end|>) and 1 for padding <|endoftext|>. We used the NeMo RL framework for the post-training stage. We run supervised fine-tuning with Megatron backend which allows us to efficiently utilize memory and computation resources of multiple nodes with H100 GPUs. For example, to do a full fine-tuning of the 4B model we used 4 nodes of 8xH100 for 27 hours.
We used LoRA test-time fine-tuning for each puzzle independently with r = 256 and alpha = 32. We removed gradient checkpointing and we removed 4-bit quantization too. Test-time fine-tuning was run with bfloat16 precision. We also used Flash Attention 2 with Unsloth Framework. You can find all hyperparameters in our source code.
\nThe main optimization we made in the ARChitects approach is for the decoding stage. We implemented the batch version of the Depth First Search (DFS) algorithm. However, there is one downside effect of batch implementation. It became nondeterministic. This phenomenon is well explained in a blog from Thinking Machines Lab. We used an open-sourced solution from Thinking Machines Lab and managed to make inference batch invariant. However, despite the better precisions, and better local validations scores, this version runs much longer in the Kaggle environment. About 17% slower, and we didn’t use it in the final submission.
\nThe ARChitects approach uses additional augmentations to rescore the candidates from DFS stage. We made a small change here. We used only 8 augmentations per candidate solution, but we used exactly the same augmentations for each candidate solution. This makes the scores of different solutions more comparable. After the competition deadline we tested the different re-scoring strategies and found the better method to select the right candidate. We calculate how many times the candidate solution was found during the DFS stage and adjust this with the geometric mean ensemble\nof log-probabilities from different augmentations.
\n
During the competition we fine-tuned models on different portions of synthetic data. In the figure you can see the effect from adding more data on the loss function during the pretraining stage. The local validation loss measured on an augmented version of 120 evaluation puzzles, and we saw a good correlation with public leaderboard. Our best model scored 27.64% during the competition.
\nDuring the last 10 days or so of the competition we explored the use of the Tiny Recursive Models (TRM) by Alexia Jolicoeur-Martineau.
\nThe day before last we managed to get a working submission that ensembles TRM with the Qwen3 2B model. We lacked time to tune the ensemble and the TRM model itself, but we were glad to be able to test the idea still. We used a few late submissions to finish testing the ensemble and report on these as well. We only used models pretrained during the competition in these late submissions.
\nUsing the code, the ARC Prize organization got a score of 6.9 percents by adapting it to run on 4 nodes of 8 H100 GPUs. Using that amount of compute resources is impossible in Kaggle. To make it worse, we could only use 2 hours or so for TRM given we had to use the rest for the ARChitects approach. Therefore, we had to use a different method. We first pretrained the TRM model, then fine-tuned the TRM model using test data during submission.
\nUsing a batch size of 3072 instead of 768, and learning rate of 3e-4 instead of 1e-4 we could train TRM to the same accuracy as the original code in 24 hours on 8xH100. Using this in a submission yields a score of 2.08% only.
\nWe then tuned various parameters to get the best score while using at most 2 hours of running time on Kaggle. We ended up with these parameters: 4 H cycles instead of the default 3, and 10 halt max steps instead of 16. We used 2000 epochs and only 200 warmup steps. This makes TRM run in about 2 hours on Kaggle.
\nIn order to assess the strength of the TRM model we trained it twice. We trained it with the 4k puzzles and used that for submission. And we also trained it with the competition evaluation dataset removed. We then scored the latter on the evaluation data. We then looked at how many training steps were used for the checkpoint with best pass@2. We finally selected the checkpoint trained on all 4k samples with the closest number of training steps. Best pass@2 on evaluation with a model not trained on evaluation was 9.44%. When submitted the matching checkpoint trained with evaluation scored 7.5%. This is much better than the 2.08% we had before.
\nAfter the competition deadline we submitted our best TRM using 4k epochs instead of 2k. It yield a score of 10.0%, in less than 4 hours.
\nIn order to use TRM with the ARChitects approach we modified TRM so that it generates 10 attempts instead of 2 per puzzle. These attempts were then added to the attempts generated by The ARChitects procedure. They were then scored by the Qwen3 model like the other attempts.
\nThis worked fine but with mixed results. Most of the puzzles solved by TRM were solved by Qwen3, hence TRM added nothing there. However, about 2 or 3 puzzles solved by TRM were not solved by Qwen3. Unfortunately, these were not always picked by Qwen3 scoring.
\nWith two late submissions we found that we could improve the 21.53 score obtained with Qwen3 2B to 22.50 with a TRM ensemble. We also saw that when using a Qwen3 4B submission that uses 10 hours only, with a score 27.22, adding TRM yields the same 27.22 score.
\nWhile we think that there is some potential for improvement here, ensembling the Qwen3 4B model and TRM model would require more investigations and experiments than we could perform.
\nWe presented how we improved The ARChitects and TRM models to get the best score on the public leaderboard of ARC Prize 2025 competition. With limited resources during the evaluation stage we needed a good way to compress learned knowledge and skills. Scaling the pretraining stage with good synthetic puzzles was the right direction to succeed in this Kaggle competition. But we also believe that this synthetic data with python programs and reasoning traces could be used for other research directions.
", "rawMarkdown": "The main components of our solution:\n- Multi-stage synthetic data generation pipeline\n- Improved version of the ARChitects solution\n- Improved version of the Tiny Recursive Model\n\nOur overall workflow can be summarized by this picture. We will review each part in the rest of this writeup.\n\n\n\n## Synthetic Puzzles\n\nWe build a Synthetic Data Generation (SDG) pipeline with 4 main stages:\n1. Collect puzzle descriptions and generate puzzle summaries\n2. Mix two summaries and generate new more complex summaries\n3. Generate python code for the input grid logic\n4. Generate python code for the output grid logic\n\nThe majority of generated data is made with `gpt-oss-120b` model using the NeMo-Skills framework. This framework provides many tools for LLM development, including large-scale SDG pipelines on Slurm cluster. You can find more information how to use this framework for SDG [here](https://nvidia-nemo.github.io/Skills/tutorials/2025/08/29/inference-with-gpt-oss-120b-using-stateful-python-code-execution/#synthetic-data-generation). Running this model on a single node with 8xH100 GPUs gives us 15k tokens/s generation throughput.\n\n### Puzzle summaries\n\nWe collect the puzzle descriptions from two main sources. First, we found [H-ARC](https://github.com/Le-Gris/h-arc), which was surprisingly underestimated in the ARC community. This dataset contains solution attempts from over 1700 humans on ARC problems, but more importantly it includes the natural-language solution descriptions. Second, we took 160 human written descriptions from [BARC](https://github.com/xu3kev/BARC). After filtering and combining these descriptions we had descriptions for 716 training puzzles from ARC-AGI-2.\n\n### Mix summaries\n\nWe prompted LLM to join two puzzle summaries and produce a new more complex puzzle by combining elements from both. In total, we generated 266593 new puzzle summaries.\n\n### Input grid programs\n\nInstead of generating a full puzzle program with input/output logic, we decided to split this into two stages. First - generate a Python program for the input grid logic, second - generate a program of the output grid logic, i.e. transformation rules. To generate reliable and good input grid logic we prompt the LLM to produce additional unit test for the input grid. In total, we generated and filtered 126901 input grid programs in Python code.\n\n### Output grid programs\n\nWe prompt LLM with puzzle description and corresponding input grid program to generate the relevant output grid program. Here we apply a different validation strategy. Instead of validating Python code with unit tests, we prompt the LLM multiple times for each input grid logic and filter out solutions which are not consistent. After filtering we had 103253 puzzles, where each puzzle has 1 input grid program and multiple output grid programs, but all of these output programs produce consistent output grids.\n\n## The ARChitects\n\nFor this approach we try to use as many puzzles as possible. In addition to our synthetic data we also used a few real puzzles datasets. Our final dataset includes 3.2M augmented samples, where each sample has up to 7 input/output pairs.\n\n| Source | Unique puzzles | Augmented samples \\\\ per puzzle | Total samples | Share |\n| ------- | ---------------: | ---------------------------------- | --------------: | ----- |\n| [MINI-ARC](https://github.com/KSB21ST/MINI-ARC) | 147 | 256 | 37632 | 1.2 |\n| [ConceptARC](https://github.com/victorvikram/ConceptARC) | 160 | 256 | 40960 | 1.3 |\n| [RE-ARC](https://github.com/michaelhodel/re-arc) | 400 | 256 | 102392 | 3.2 |\n| ARC-AGI-2 | 609 | 256 | 155904 | 4.8 |\n| NVARC training | 47337 | 24 | 1132633 | 34.8 |\n| NVARC full | 55886 | 32 | 1785960 | 54.9 |\n| | 104539 | | 3255481 | |\n\nARC-AGI-2 training subset excludes RE-ARC puzzles. NVARC training subset based only on training puzzles from ARC-AGI-2. NVARC full subset uses both training and evaluation puzzles from ARC-AGI-2.\n\nThe ARChitects approach has a simple representation of puzzles as a list of input/output grids. But we made it even simpler and used a dialog style template from Qwen3 model. For example, formatted message represents 1 pair of input/output grids looks like this:\n\n```\n<|im_start|>user\\n123\\n456<|im_end|>\n<|im_start|>assistant\\n78\\n90<|im_end|>\n```\n\nThis simpler representation requires only 16 tokens: 10 - for digits, 1 - new line, “user” - start of input grid, “assistant” - start of output grid, 2 special tokens (`<|im_start|>`, `<|im_end|>`) and 1 for padding `<|endoftext|>`. We used the [NeMo RL](https://github.com/NVIDIA-NeMo/RL) framework for the post-training stage. We run supervised fine-tuning with Megatron backend which allows us to efficiently utilize memory and computation resources of multiple nodes with H100 GPUs. For example, to do a full fine-tuning of the 4B model we used 4 nodes of 8xH100 for 27 hours.\n\nWe used LoRA test-time fine-tuning for each puzzle independently with r = 256 and alpha = 32. We removed gradient checkpointing and we removed 4-bit quantization too. Test-time fine-tuning was run with bfloat16 precision. We also used Flash Attention 2 with Unsloth Framework. You can find all hyperparameters in our [source code](https://github.com/1ytic/NVARC).\n\nThe main optimization we made in the ARChitects approach is for the decoding stage. We implemented the batch version of the Depth First Search (DFS) algorithm. However, there is one downside effect of batch implementation. It became nondeterministic. This phenomenon is well explained in a [blog](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference) from Thinking Machines Lab. We used an open-sourced [solution](https://github.com/thinking-machines-lab/batch_invariant_ops) from Thinking Machines Lab and managed to make inference batch invariant. However, despite the better precisions, and better local validations scores, this version runs much longer in the Kaggle environment. About 17% slower, and we didn’t use it in the final submission.\n\nThe ARChitects approach uses additional augmentations to rescore the candidates from DFS stage. We made a small change here. We used only 8 augmentations per candidate solution, but we used exactly the same augmentations for each candidate solution. This makes the scores of different solutions more comparable. After the competition deadline we tested the different re-scoring strategies and found the better method to select the right candidate. We calculate how many times the candidate solution was found during the DFS stage and adjust this with the geometric mean ensemble\nof log-probabilities from different augmentations.\n\n\n\nDuring the competition we fine-tuned models on different portions of synthetic data. In the figure you can see the effect from adding more data on the loss function during the pretraining stage. The local validation loss measured on an augmented version of 120 evaluation puzzles, and we saw a good correlation with public leaderboard. Our best model scored 27.64% during the competition.\n\n## Tiny Recursive Model\n\nDuring the last 10 days or so of the competition we explored the use of the Tiny Recursive Models ([TRM](https://github.com/SamsungSAILMontreal/TinyRecursiveModels)) by Alexia Jolicoeur-Martineau.\n\nThe day before last we managed to get a working submission that ensembles TRM with the Qwen3 2B model. We lacked time to tune the ensemble and the TRM model itself, but we were glad to be able to test the idea still. We used a few late submissions to finish testing the ensemble and report on these as well. We only used models pretrained during the competition in these late submissions.\n\nUsing the [code](https://github.com/SamsungSAILMontreal/TinyRecursiveModels), the ARC Prize organization got a score of 6.9 percents by adapting it to run on 4 nodes of 8 H100 GPUs. Using that amount of compute resources is impossible in Kaggle. To make it worse, we could only use 2 hours or so for TRM given we had to use the rest for the ARChitects approach. Therefore, we had to use a different method. We first pretrained the TRM model, then fine-tuned the TRM model using test data during submission.\n\nUsing a batch size of 3072 instead of 768, and learning rate of 3e-4 instead of 1e-4 we could train TRM to the same accuracy as the original code in 24 hours on 8xH100. Using this in a submission yields a score of 2.08% only.\n\nWe then tuned various parameters to get the best score while using at most 2 hours of running time on Kaggle. We ended up with these parameters: 4 H cycles instead of the default 3, and 10 halt max steps instead of 16. We used 2000 epochs and only 200 warmup steps. This makes TRM run in about 2 hours on Kaggle.\n\nIn order to assess the strength of the TRM model we trained it twice. We trained it with the 4k puzzles and used that for submission. And we also trained it with the competition evaluation dataset removed. We then scored the latter on the evaluation data. We then looked at how many training steps were used for the checkpoint with best pass@2. We finally selected the checkpoint trained on all 4k samples with the closest number of training steps. Best pass@2 on evaluation with a model not trained on evaluation was 9.44%. When submitted the matching checkpoint trained with evaluation scored 7.5%. This is much better than the 2.08% we had before.\n\nAfter the competition deadline we submitted our best TRM using 4k epochs instead of 2k. It yield a score of 10.0%, in less than 4 hours.\n\nIn order to use TRM with the ARChitects approach we modified TRM so that it generates 10 attempts instead of 2 per puzzle. These attempts were then added to the attempts generated by The ARChitects procedure. They were then scored by the Qwen3 model like the other attempts.\n\nThis worked fine but with mixed results. Most of the puzzles solved by TRM were solved by Qwen3, hence TRM added nothing there. However, about 2 or 3 puzzles solved by TRM were not solved by Qwen3. Unfortunately, these were not always picked by Qwen3 scoring.\n\nWith two late submissions we found that we could improve the 21.53 score obtained with Qwen3 2B to 22.50 with a TRM ensemble. We also saw that when using a Qwen3 4B submission that uses 10 hours only, with a score 27.22, adding TRM yields the same 27.22 score.\n\nWhile we think that there is some potential for improvement here, ensembling the Qwen3 4B model and TRM model would require more investigations and experiments than we could perform.\n\n## Conclusion\n\nWe presented how we improved The ARChitects and TRM models to get the best score on the public leaderboard of ARC Prize 2025 competition. With limited resources during the evaluation stage we needed a good way to compress learned knowledge and skills. Scaling the pretraining stage with good synthetic puzzles was the right direction to succeed in this Kaggle competition. But we also believe that this synthetic data with python programs and reasoning traces could be used for other research directions.", "votes": 123 }, { "id": 3363444, "postDate": "2025-12-05T23:23:42.337Z", "content": "Some of the most impressive work I have seen in a long time, can’t wait to review this more
", "rawMarkdown": "Some of the most impressive work I have seen in a long time, can’t wait to review this more", "votes": 3 }, { "id": 3363408, "postDate": "2025-12-05T22:53:19.053Z", "content": "Congratulations @sorokin and @cpmpml ! This is a incredible win!
\nGreat writeup, and not only a big gap to 2nd place, but real artificial intelligence applied to an extremely challenging task.
", "rawMarkdown": "Congratulations @sorokin and @cpmpml ! This is a incredible win! \n\nGreat writeup, and not only a big gap to 2nd place, but real artificial intelligence applied to an extremely challenging task. ", "votes": 3 }, { "id": 3364576, "postDate": "2025-12-06T15:07:21.957Z", "content": "Insane results, the LB gap is really impressive!! 🔥\nCongratulations and thanks for the write up!
", "rawMarkdown": "Insane results, the LB gap is really impressive!! 🔥\nCongratulations and thanks for the write up!", "votes": 1 }, { "id": 3363640, "postDate": "2025-12-06T03:33:09.360Z", "content": "Thanks for open-sourcing the training data and even the scripts generating them! They will be extremely valuable for future research on ARC AGI.
", "rawMarkdown": "Thanks for open-sourcing the training data and even the scripts generating them! They will be extremely valuable for future research on ARC AGI.", "votes": 1 }, { "id": 3363222, "postDate": "2025-12-05T20:17:59.263Z", "content": "No problem. Maybe next year I'll show you my Artifical Super Intelligence which can solve all publicly available 51'454'560 training tasks in milliseconds…
", "rawMarkdown": "No problem. Maybe next year I'll show you my Artifical Super Intelligence which can solve all publicly available 51'454'560 training tasks in milliseconds...", "votes": -1 }, { "id": 3364624, "postDate": "2025-12-06T15:39:00.010Z", "content": "Instead of generating a full puzzle program with input/output logic, we decided to split this into two stages. First - generate a Python program for the input grid logic, second - generate a program of the output grid logic, i.e. transformation rules. To generate reliable and good input grid logic we prompt the LLM to produce additional unit test for the input grid. In total, we generated and filtered 126901 input grid programs in Python code.
", "rawMarkdown": "Instead of generating a full puzzle program with input/output logic, we decided to split this into two stages. First - generate a Python program for the input grid logic, second - generate a program of the output grid logic, i.e. transformation rules. To generate reliable and good input grid logic we prompt the LLM to produce additional unit test for the input grid. In total, we generated and filtered 126901 input grid programs in Python code.", "votes": -2, "replies": [ { "id": 3367340, "postDate": "2025-12-08T11:16:36.877Z", "content": "Generate a Python program for the input grid logic?
", "rawMarkdown": "Generate a Python program for the input grid logic?" } ] }, { "id": 3376158, "postDate": "2025-12-13T19:07:09.403Z", "content": "Thanks for sharing and congratulations for the 1st place and innovative solution. Well done!!!
\nI'd like to ask you a couple of questions:
\nThanks!
\nThank you for the ARChitects and TRM models!
", "rawMarkdown": "Thank you for the ARChitects and TRM models!" }, { "id": 3369924, "postDate": "2025-12-10T13:00:44.973Z", "content": "Amazing work! How many tokens did it take you to generate one task?
", "rawMarkdown": "Amazing work! How many tokens did it take you to generate one task?", "replies": [ { "id": 3370013, "postDate": "2025-12-10T14:19:27.443Z", "content": "about one per cell in the grid plus one per row.
", "rawMarkdown": "about one per cell in the grid plus one per row.", "replies": [ { "id": 3372137, "postDate": "2025-12-11T20:30:51.303Z", "content": "How about the synthetic tasks? How many gpt-oss tokens did you consume to generate those 50k synthetic tasks? Were you able to generate all that synthetic data using a single 8xH100 node?
We provide all generated texts here https://www.kaggle.com/datasets/sorokin/nvarc-artifacts-puzzles You can estimate tokens from it.
", "rawMarkdown": "We provide all generated texts here https://www.kaggle.com/datasets/sorokin/nvarc-artifacts-puzzles You can estimate tokens from it.", "votes": 1 } ] } ] }, { "id": 3367566, "postDate": "2025-12-08T14:50:55.830Z", "content": "thats impressive
", "rawMarkdown": "thats impressive " }, { "id": 3366921, "postDate": "2025-12-08T04:56:48.827Z", "content": "Excellent and well-structured solution! Really appreciate you open-sourcing this. I learned a tremendous amount from it.
", "rawMarkdown": "Excellent and well-structured solution! Really appreciate you open-sourcing this. I learned a tremendous amount from it." }, { "id": 3363073, "postDate": "2025-12-05T18:47:54.627Z", "content": "Congrats on the achievement! I am a little bit surprised TRM wasn't a major contributing factor to your team's crazy climb during the last few days.
", "rawMarkdown": "Congrats on the achievement! I am a little bit surprised TRM wasn't a major contributing factor to your team's crazy climb during the last few days.", "replies": [ { "id": 3363457, "postDate": "2025-12-05T23:45:40.377Z", "content": "\n\nI am a little bit surprised TRM wasn't a major contributing factor to your team's crazy climb
\n
Why? Arc prize team got less than 7% with TRM using 3x8 H100 for a day. We were pleased to see 10% in 2 hours or so with 4 L4.
\nModel ensembling is not about having lots of great models.
\nEnsembling is about having one great model (here Qwen 4B) and other models that can add a bit to it (TRM).
", "rawMarkdown": "> I am a little bit surprised TRM wasn't a major contributing factor to your team's crazy climb \n\nWhy? Arc prize team got less than 7% with TRM using 3x8 H100 for a day. We were pleased to see 10% in 2 hours or so with 4 L4.\n\nModel ensembling is not about having lots of great models.\n\nEnsembling is about having one great model (here Qwen 4B) and other models that can add a bit to it (TRM).", "votes": 2, "replies": [ { "id": 3363498, "postDate": "2025-12-06T01:00:00.437Z", "content": "I was under the wrong impression that the one would try to add the TRM style recurrent loops on top of their \"main model\". But upon further reading, it seems that having a pretrained natural language backbone is crucial to your approach?
", "rawMarkdown": "I was under the wrong impression that the one would try to add the TRM style recurrent loops on top of their \"main model\". But upon further reading, it seems that having a pretrained natural language backbone is crucial to your approach? " }, { "id": 3364228, "postDate": "2025-12-06T10:53:34.717Z", "content": "Recurring over the main model is expensive. We used all the 12 hours we got, and we could not run anything lower.
", "rawMarkdown": "Recurring over the main model is expensive. We used all the 12 hours we got, and we could not run anything lower.", "votes": 1 } ] } ] }, { "id": 3367313, "postDate": "2025-12-08T10:48:26.913Z", "content": "Thank you for the write up!
", "rawMarkdown": "Thank you for the write up!", "votes": 2 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T21:00:30Z", "competition_id": 91496, "competition_slug": "arc-prize-2025", "team_id": 14425518, "solution_url": "/competitions/arc-prize-2025/writeups/nvarc" }