Our solution is a modified version of the 2024 winning solution by the ARChitects, with the following tweaks:
\nFAKE = True in my solution notebook, you will be using the 30 holdout tasks(The motivation behind our \"2024 solution but even simpler\" kind of approach is that we were trying to work more on the local finetuning part. Unfortunately there hasn't been much great result that can be reported.)
\nTo be fair, our solution is not optimized at all. We discarded the more complex inference method that could help. Inference is done with slower library (on the private test set, we estimate ~1.5 hours being spent on inference). We even attribute tasks to 4 gpus simply by a random split. However, our notebook still finishes within ~5 hours on the private test set while achieving a LB score of 8.33, thanks to the smaller model.
\nAlthough our LB score is not very high, we hope the info reported here can be helpful to the community. Our main takeaways:
\nFinally we want to thank arcprize.org for hosting this competition and the 2024 winners (and front runners) for sharing their solutions!
", "comments": [ { "id": 3313849, "author_name": "Richard Emslie", "author_url": "", "post_date": "2025-11-09T22:48:43.513000", "content": "Hi - thanks for write-up and model, good job with training a small model.\nI also had most success with 8B ARCitects model doing singe-task test time training, fwiw.
\nIs it possible to provide the 30 eval tasks you held back from your model please? I think with that info, it would make the model much more useful for community to experiment.
", "votes": 0, "replies": [ { "id": 3313858, "author_name": "jojobagel", "author_url": "", "post_date": "2025-11-09T23:26:28.540000", "content": "Hi rxe,
\nThanks so much!
", "votes": 0, "replies": [] } ] } ] } ], "comment_count": 3, "messages": [ { "id": 3313787, "postDate": "2025-11-09T20:01:32.320Z", "content": "Our solution is a modified version of the 2024 winning solution by the ARChitects, with the following tweaks:
\nFAKE = True in my solution notebook, you will be using the 30 holdout tasks(The motivation behind our \"2024 solution but even simpler\" kind of approach is that we were trying to work more on the local finetuning part. Unfortunately there hasn't been much great result that can be reported.)
\nTo be fair, our solution is not optimized at all. We discarded the more complex inference method that could help. Inference is done with slower library (on the private test set, we estimate ~1.5 hours being spent on inference). We even attribute tasks to 4 gpus simply by a random split. However, our notebook still finishes within ~5 hours on the private test set while achieving a LB score of 8.33, thanks to the smaller model.
\nAlthough our LB score is not very high, we hope the info reported here can be helpful to the community. Our main takeaways:
\nFinally we want to thank arcprize.org for hosting this competition and the 2024 winners (and front runners) for sharing their solutions!
", "rawMarkdown": "## **Huge shoutout to [lambda.ai](https://lambda.ai) for giving us compute credit, allowing us to finetune our own model!**\n\nOur solution is a modified version of [the 2024 winning solution by the ARChitects](https://www.kaggle.com/code/dfranzen/arc-prize-2024-solution-by-the-architects), with the following tweaks:\n- We use a much smaller model. We finetuned on [Qwen2.5-0.5B](https://huggingface.co/Qwen/Qwen2.5-0.5B), using the scripts [open-sourced by ARChitects](https://github.com/da-fr/arc-prize-2024), with ARC 2 data instead of ARC 1 evaluation set. We used 90 tasks from ARC 2 evaluation set so that we have 30 tasks to experiment with test time training (TTT)\n- Set `FAKE = True` in my solution notebook, you will be using the 30 holdout tasks\n- Do single task finetuning. Same finding as [Guillermo Barbadillo](https://www.kaggle.com/code/ironbar/the-architects-single-task-ttt), [ippeiogawa](https://www.kaggle.com/code/ippeiogawa/lb10-00-with-tweak-to-2024-1st-solution) and [rxe](https://www.kaggle.com/code/albartrose/10-4-notebook-arcitects-tweaked) using a 8B model. We also tried TTT on *60 tasks together* per gpu, but never achieved a LB score more than 4.\n- During TTT, the ARChitects data augmentation approach is used to generate 560 augmented samples for each task, and then finetune for 140 steps (effective batch size 4). One more tweak: during test time, we don't train on token embedding layer or output projection. \n- We observe double descent as test time training steps increases\n- During inference, we simply use greedy sampling with 16 augmented samples (and we just use the transformers library)\n\n(The motivation behind our \"2024 solution but even simpler\" kind of approach is that we were trying to work more on the local finetuning part. Unfortunately there hasn't been much great result that can be reported.)\n\nTo be fair, our solution is not optimized at all. We discarded the more complex inference method that could help. Inference is done with slower library (on the private test set, we estimate ~1.5 hours being spent on inference). We even attribute tasks to 4 gpus simply by a random split. **However, our notebook still finishes within ~5 hours on the private test set while achieving a LB score of 8.33, thanks to the smaller model.**\n\nAlthough our LB score is not very high, we hope the info reported here can be helpful to the community. Our main takeaways:\n1. With a 0.5B decoder-only language model, you also want to do singe-task test time training (same as with a 8B model)\n2. A 0.5B model can get close in terms of performances\n3. We want to share our 0.5B model so that future ARC-AGI players can run easy and fast experiments using it\n\nFinally we want to thank [arcprize.org](https://arcprize.org) for hosting this competition and the 2024 winners (and front runners) for sharing their solutions!", "votes": 19 }, { "id": 3313849, "postDate": "2025-11-09T22:48:43.513Z", "content": "Hi - thanks for write-up and model, good job with training a small model.\nI also had most success with 8B ARCitects model doing singe-task test time training, fwiw.
\nIs it possible to provide the 30 eval tasks you held back from your model please? I think with that info, it would make the model much more useful for community to experiment.
", "rawMarkdown": "Hi - thanks for write-up and model, good job with training a small model.\nI also had most success with 8B ARCitects model doing singe-task test time training, fwiw.\n\nIs it possible to provide the 30 eval tasks you held back from your model please? I think with that info, it would make the model much more useful for community to experiment.", "replies": [ { "id": 3313858, "postDate": "2025-11-09T23:26:28.540Z", "content": "Hi rxe,
\nThanks so much!
", "rawMarkdown": "Thanks so much!" } ] } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T21:00:17Z", "competition_id": 91496, "competition_slug": "arc-prize-2025", "team_id": 13937287, "solution_url": "/competitions/arc-prize-2025/writeups/lb-8-33-with-architects-approach-and-a-0-5b-model" }