![\"The](\"https://lambdalabsml.github.io/ARC2025_Solution_by_the_ARChitects/The%20ARChitects%20-%20Technical%20Report/image%2012.png\")
\n ![\"Photo](\"https://lambdalabsml.github.io/ARC2025_Solution_by_the_ARChitects/The%20ARChitects%20-%20Technical%20Report/arcfront.jpg\")
\nHey Kaggle & ARC community!
\nThe competition has been an exciting ride, especially in the last few intense weeks.
\nWe’re excited to give you two compact “solution cards” below that summarize the core ideas of our submissions.
\nOr, check out our full technical report here: lambdalabsml.github.io/ARC2025_Solution_by_the_ARChitects
In the technical report we talk about the full story and entire journey: our approaches, our total compute budget, what else we’ve been working on, what surprised us by working really well, what didn’t quite work (yet!?) and what our next steps would have been if we had another week or two.
\nARC was the perfect project to pour a lot of energy and passion into, spending nearly every waking hour on the home stretch for fine-tuning and experimenting.
\nSo: we’re all looking forward to what comes next!!
The ARChitects — Daniel, Jan & David 👋
\n\n \n| Aspect | \nSummary | \n
|---|---|
| Base model | \nUsed a masked-diffusion LLM: LLaDA-8B fine-tuned for ARC-style tasks. | \n
| Masking & Loss Strategy | \nStandard cross-entropy loss to reconstruct masked positions. | \n
| Soft Masking + Recursive Latent Sampling | \nWe found that adding the mask tokens to every token let’s the model refine it’s own pedictions. After an initial guess, feed model output back in (with soft-masking), iteratively refine predictions in a loop until convergence or stability. This effectively turns the model into a continuous, recursive solver. | \n
| Why Recursion Helps | \nThe models token embeddings are continuous. We found that you can blend tokens (token algebra) and feed back outputs. Soft-masking seems to tell the model “this position needs improvement.” This enables iterative self-improvement, rather than one-shot generation. | \n
| Adaptations for ARC | \nReplaced original 1D positional encoding with a 2D positional encoding (based on “Golden Gate RoPE”) to better match the 2D grid nature of ARC tasks. | \n
| Datasets & Training | \nSame data pipeline as AR model from last year: used datasets like ReARC, ARC-GEN-100K, official ARC1 & ARC2, ARC-Heavy, ConceptARC. Pretraining: ~175k steps (batch size 8) on 8×H100 GPUs. Test-time fine-tuning: for each task, 128 steps on L4 GPU (batch size 1). | \n
| Shape Prediction | \nBecause output grids may vary in size, we trained a second LLaDA model (finetuned) to predict the correct output “shape” (grid size) given input examples, i.e. where delimiters go. | \n
| Final Submission Performance | \nWith known shape: ~30.5% ±1% on held-out eval set (after 102 inference steps, that is two rounds of 51 with cold restart). Shape predictor ≈ 85% ±2% correct on eval set. Combined we expected ~26%, but real public score was 21.67%, private score was 16.53%. Our best submission (using a different hyperparameter selection mechanism) achieved a public score of 19.17% and a private score of 19.17%. We didn’t choose that submission due to the lower public leaderboard score. ;D | \n
| Aspect | \nSummary | \n\n |
|---|---|---|
| Summary: Improvements vs prior year | \nTask-specific test-time fine-tuning rather than multi-task fine-tuning, speculative decoding + prefix caching, lowering probability cutoff (from 17% → 7%) to explore more candidates, increased scoring augmentations (8 → 32). | \n\n |
| Base Model | \nUsed an autoregressive model (from previous year) built on Mistral‑NeMo‑Minitron‑8B‑Base. We had a better pre-trained model from a late submission last year that we didn’t publish, which included Arc-Heavy in its training data. | \n\n |
| Sampling Method | \nFaster depth-first search (DFS) using speculative decoding (predicting 16 to 32 tokens ahead) to generate even more candidate solutions to select from. | \n\n |
| Scoring / Selection | \nStill “Product-of-Experts” scoring: apply multiple augmentations and aggregate neg-log-likelihoods across them to choose the most probable candidate. | \n\n |
| Performance / Result | \nMaxed-out performance at 16.94% on public ARC2 leaderboard (as of Aug 11). After that, we transitioned to the masked diffusion model. | \n\n |
| Limitations Observed | \nAR model struggled with puzzle-type tasks or tasks requiring global restructuring (e.g., diagonal lines, simulation tasks, etc). Augmentations lacked expressiveness to fully capture global constraints. | \n\n |
Congrats and good stuff! 1. The current released notebook is private. 2. will you release anything from the “First half of competition” (ie nemo model, notebook, training code/dataset)?
", "votes": 1, "replies": [] }, { "id": 3370213, "author_name": "Mani Ratan", "author_url": "", "post_date": "2025-12-10T16:25:45.087000", "content": "Congrats dude, it's great that you achieved 2nd place.It's Amazing, keep it up.👍
", "votes": 1, "replies": [] }, { "id": 3369843, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2025-12-10T11:46:22.487000", "content": "Congrats on the second place.
", "votes": 1, "replies": [] }, { "id": 3378378, "author_name": "vialactea", "author_url": "", "post_date": "2025-12-18T01:35:02.333000", "content": "Congratulations for the 2nd place and thanks for sharing. You came up with several very interesting ideas!
\nI'd like to ask you a couple of questions:
\nYour experiments suggest that your new approach is superior to last years'. That being the case, it seems to have the potential to do even better than the wining solution if your models were pretrained with a similar dataset (and number of steps). Do you have any thoughts on that? Would the best of both words be to use your approach with their synthetic data?
When I look at your logits normalization I sense (perhaps incorrectly) several objectives: convert logits to the range 0..1, making them more of a probability to ensemble the corresponding embeddings; initially zero most of the logits and gradually become less restrictive, possibly encouraging more exploration; approximate the logits distribution for the various grid positions. Is my interpretation correct? Does the model get better overtime (as T_mask goes down) or it simply performs more exploration (or neither)? Did you experiment with other approaches to normalization and found this one to be superior or just tried it first and stuck to it because it worked?
Thanks a lot for the congratulations, and for the thoughtful questions!
\nIn our experiments, we found that the diffusion-based approach works better on ARC-2. Interestingly, this was not the case for ARC-1, where diffusion performed roughly the same as our autoregressive method. We suspect this difference comes from a different selection of tasks in the ARC-2 dataset.
\nIn particular, ARC-2 seems to include more tasks that require multiple solving steps, where some pixels’ values depend on pixels that are effectively generated earlier in the solution. Diffusion models can handle such interdependencies better, as they refine the whole output jointly. Autoregressive models, on the other hand, can only capture these dependencies when the generation order aligns with the dependency structure. Data augmentation helps a bit, but it cannot fully resolve this limitation.
\nAs for pretraining, we agree that using more high-quality data would likely further improve performance. We also explored generating additional training data ourselves, but despite some promising ideas, we were not able to make this process efficient enough in practice.
Yes, your interpretation is correct. First, normalizing the logits to the range 0..1 is necessary for the model to handle the inputs. Naturally, our initial approach was to just use the softmax probabilities. But we quickly noticed that this behaves almost like discretization, since the model is usually extremely confident (for most pixels, the softmax probabilities are well above 99%). So we searched for alternatives and found that a linear rescaling of the logits (followed by clipping to 0..1) works well in practice. That said, in some cases the model would still snap in to an incorrect solution very early and not be able to fix it later on.
\nTo address this, we tried to mimic the process that is generally used in discrete diffusion, where only the most confident tokens/pixels are fixed first, and the rest are filled in over multiple iterations. Our temperature schedule has a similar effect: Early on, when the temperature is close to 1, only the outputs for which the model is most confident are passed as input for the next iteration. This is then gradually relaxed as the temperature decreases. In practice, this means the model often needs more iterations to converge, but we found it encourages more exploration early on, allowing the model to hopefully \"snap\" in to the correct solution as the temperature decreases.
\nFinally, note that we didn't have time to thoroughly study our recursive inference method, as we only came up with it in the very last days of the competition. So while it worked well for us, there's likely room for further improvement.
We hope this answers your questions. Please feel free to follow up if anything is unclear or if you want to discuss additional ideas.
", "votes": 1, "replies": [] } ] } ], "comment_count": 5, "messages": [ { "id": 3369299, "postDate": "2025-12-09T23:00:17.160Z", "content": "Hey Kaggle & ARC community!
\nThe competition has been an exciting ride, especially in the last few intense weeks.
\nWe’re excited to give you two compact “solution cards” below that summarize the core ideas of our submissions.
\nOr, check out our full technical report here: lambdalabsml.github.io/ARC2025_Solution_by_the_ARChitects
In the technical report we talk about the full story and entire journey: our approaches, our total compute budget, what else we’ve been working on, what surprised us by working really well, what didn’t quite work (yet!?) and what our next steps would have been if we had another week or two.
\nARC was the perfect project to pour a lot of energy and passion into, spending nearly every waking hour on the home stretch for fine-tuning and experimenting.
\nSo: we’re all looking forward to what comes next!!
The ARChitects — Daniel, Jan & David 👋
\n\n \n| Aspect | \nSummary | \n
|---|---|
| Base model | \nUsed a masked-diffusion LLM: LLaDA-8B fine-tuned for ARC-style tasks. | \n
| Masking & Loss Strategy | \nStandard cross-entropy loss to reconstruct masked positions. | \n
| Soft Masking + Recursive Latent Sampling | \nWe found that adding the mask tokens to every token let’s the model refine it’s own pedictions. After an initial guess, feed model output back in (with soft-masking), iteratively refine predictions in a loop until convergence or stability. This effectively turns the model into a continuous, recursive solver. | \n
| Why Recursion Helps | \nThe models token embeddings are continuous. We found that you can blend tokens (token algebra) and feed back outputs. Soft-masking seems to tell the model “this position needs improvement.” This enables iterative self-improvement, rather than one-shot generation. | \n
| Adaptations for ARC | \nReplaced original 1D positional encoding with a 2D positional encoding (based on “Golden Gate RoPE”) to better match the 2D grid nature of ARC tasks. | \n
| Datasets & Training | \nSame data pipeline as AR model from last year: used datasets like ReARC, ARC-GEN-100K, official ARC1 & ARC2, ARC-Heavy, ConceptARC. Pretraining: ~175k steps (batch size 8) on 8×H100 GPUs. Test-time fine-tuning: for each task, 128 steps on L4 GPU (batch size 1). | \n
| Shape Prediction | \nBecause output grids may vary in size, we trained a second LLaDA model (finetuned) to predict the correct output “shape” (grid size) given input examples, i.e. where delimiters go. | \n
| Final Submission Performance | \nWith known shape: ~30.5% ±1% on held-out eval set (after 102 inference steps, that is two rounds of 51 with cold restart). Shape predictor ≈ 85% ±2% correct on eval set. Combined we expected ~26%, but real public score was 21.67%, private score was 16.53%. Our best submission (using a different hyperparameter selection mechanism) achieved a public score of 19.17% and a private score of 19.17%. We didn’t choose that submission due to the lower public leaderboard score. ;D | \n
| Aspect | \nSummary | \n\n |
|---|---|---|
| Summary: Improvements vs prior year | \nTask-specific test-time fine-tuning rather than multi-task fine-tuning, speculative decoding + prefix caching, lowering probability cutoff (from 17% → 7%) to explore more candidates, increased scoring augmentations (8 → 32). | \n\n |
| Base Model | \nUsed an autoregressive model (from previous year) built on Mistral‑NeMo‑Minitron‑8B‑Base. We had a better pre-trained model from a late submission last year that we didn’t publish, which included Arc-Heavy in its training data. | \n\n |
| Sampling Method | \nFaster depth-first search (DFS) using speculative decoding (predicting 16 to 32 tokens ahead) to generate even more candidate solutions to select from. | \n\n |
| Scoring / Selection | \nStill “Product-of-Experts” scoring: apply multiple augmentations and aggregate neg-log-likelihoods across them to choose the most probable candidate. | \n\n |
| Performance / Result | \nMaxed-out performance at 16.94% on public ARC2 leaderboard (as of Aug 11). After that, we transitioned to the masked diffusion model. | \n\n |
| Limitations Observed | \nAR model struggled with puzzle-type tasks or tasks requiring global restructuring (e.g., diagonal lines, simulation tasks, etc). Augmentations lacked expressiveness to fully capture global constraints. | \n\n |
Congrats and good stuff! 1. The current released notebook is private. 2. will you release anything from the “First half of competition” (ie nemo model, notebook, training code/dataset)?
", "rawMarkdown": "Congrats and good stuff! 1. The current released notebook is private. 2. will you release anything from the “First half of competition” (ie nemo model, notebook, training code/dataset)? ", "votes": 1 }, { "id": 3370213, "postDate": "2025-12-10T16:25:45.087Z", "content": "Congrats dude, it's great that you achieved 2nd place.It's Amazing, keep it up.👍
", "rawMarkdown": "Congrats dude, it's great that you achieved 2nd place.It's Amazing, keep it up.👍", "votes": 1 }, { "id": 3369843, "postDate": "2025-12-10T11:46:22.487Z", "content": "Congrats on the second place.
", "rawMarkdown": "Congrats on the second place.", "votes": 1 }, { "id": 3378378, "postDate": "2025-12-18T01:35:02.333Z", "content": "Congratulations for the 2nd place and thanks for sharing. You came up with several very interesting ideas!
\nI'd like to ask you a couple of questions:
\nYour experiments suggest that your new approach is superior to last years'. That being the case, it seems to have the potential to do even better than the wining solution if your models were pretrained with a similar dataset (and number of steps). Do you have any thoughts on that? Would the best of both words be to use your approach with their synthetic data?
When I look at your logits normalization I sense (perhaps incorrectly) several objectives: convert logits to the range 0..1, making them more of a probability to ensemble the corresponding embeddings; initially zero most of the logits and gradually become less restrictive, possibly encouraging more exploration; approximate the logits distribution for the various grid positions. Is my interpretation correct? Does the model get better overtime (as T_mask goes down) or it simply performs more exploration (or neither)? Did you experiment with other approaches to normalization and found this one to be superior or just tried it first and stuck to it because it worked?
Thanks a lot for the congratulations, and for the thoughtful questions!
\nIn our experiments, we found that the diffusion-based approach works better on ARC-2. Interestingly, this was not the case for ARC-1, where diffusion performed roughly the same as our autoregressive method. We suspect this difference comes from a different selection of tasks in the ARC-2 dataset.
\nIn particular, ARC-2 seems to include more tasks that require multiple solving steps, where some pixels’ values depend on pixels that are effectively generated earlier in the solution. Diffusion models can handle such interdependencies better, as they refine the whole output jointly. Autoregressive models, on the other hand, can only capture these dependencies when the generation order aligns with the dependency structure. Data augmentation helps a bit, but it cannot fully resolve this limitation.
\nAs for pretraining, we agree that using more high-quality data would likely further improve performance. We also explored generating additional training data ourselves, but despite some promising ideas, we were not able to make this process efficient enough in practice.
Yes, your interpretation is correct. First, normalizing the logits to the range 0..1 is necessary for the model to handle the inputs. Naturally, our initial approach was to just use the softmax probabilities. But we quickly noticed that this behaves almost like discretization, since the model is usually extremely confident (for most pixels, the softmax probabilities are well above 99%). So we searched for alternatives and found that a linear rescaling of the logits (followed by clipping to 0..1) works well in practice. That said, in some cases the model would still snap in to an incorrect solution very early and not be able to fix it later on.
\nTo address this, we tried to mimic the process that is generally used in discrete diffusion, where only the most confident tokens/pixels are fixed first, and the rest are filled in over multiple iterations. Our temperature schedule has a similar effect: Early on, when the temperature is close to 1, only the outputs for which the model is most confident are passed as input for the next iteration. This is then gradually relaxed as the temperature decreases. In practice, this means the model often needs more iterations to converge, but we found it encourages more exploration early on, allowing the model to hopefully \"snap\" in to the correct solution as the temperature decreases.
\nFinally, note that we didn't have time to thoroughly study our recursive inference method, as we only came up with it in the very last days of the competition. So while it worked well for us, there's likely room for further improvement.
We hope this answers your questions. Please feel free to follow up if anything is unclear or if you want to discuss additional ideas.
", "rawMarkdown": "Thanks a lot for the congratulations, and for the thoughtful questions!\n\n1. In our experiments, we found that the diffusion-based approach works better on ARC-2. Interestingly, this was not the case for ARC-1, where diffusion performed roughly the same as our autoregressive method. We suspect this difference comes from a different selection of tasks in the ARC-2 dataset.\n\n In particular, ARC-2 seems to include more tasks that require multiple solving steps, where some pixels’ values depend on pixels that are effectively generated *earlier* in the solution. Diffusion models can handle such interdependencies better, as they refine the whole output jointly. Autoregressive models, on the other hand, can only capture these dependencies when the generation order aligns with the dependency structure. Data augmentation helps a bit, but it cannot fully resolve this limitation.\n\n As for pretraining, we agree that using more high-quality data would likely further improve performance. We also explored generating additional training data ourselves, but despite some promising ideas, we were not able to make this process efficient enough in practice.\n\n2. Yes, your interpretation is correct. First, normalizing the logits to the range 0..1 is necessary for the model to handle the inputs. Naturally, our initial approach was to just use the softmax probabilities. But we quickly noticed that this behaves almost like discretization, since the model is usually extremely confident (for most pixels, the softmax probabilities are well above 99%). So we searched for alternatives and found that a linear rescaling of the logits (followed by clipping to 0..1) works well in practice. That said, in some cases the model would still snap in to an incorrect solution very early and not be able to fix it later on.\n\n To address this, we tried to mimic the process that is generally used in discrete diffusion, where only the most confident tokens/pixels are fixed first, and the rest are filled in over multiple iterations. Our temperature schedule has a similar effect: Early on, when the temperature is close to 1, only the outputs for which the model is most confident are passed as input for the next iteration. This is then gradually relaxed as the temperature decreases. In practice, this means the model often needs more iterations to converge, but we found it encourages more exploration early on, allowing the model to hopefully \"snap\" in to the correct solution as the temperature decreases.\n\n Finally, note that we didn't have time to thoroughly study our recursive inference method, as we only came up with it in the very last days of the competition. So while it worked well for us, there's likely room for further improvement.\n\nWe hope this answers your questions. Please feel free to follow up if anything is unclear or if you want to discuss additional ideas.", "votes": 1 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T21:00:35Z", "competition_id": 91496, "competition_slug": "arc-prize-2025", "team_id": 13563111, "solution_url": "/competitions/arc-prize-2025/writeups/the-architects-solution-2025" }