# Lb 8.33 with ARChitects Approach and a 0.5B model (Private Runtime ~5 hours)

Huge shoutout to
[lambda.ai](https://lambda.ai)
for giving us compute credit, allowing us to finetune our own model!
Our solution is a modified version of
[the 2024 winning solution by the ARChitects](https://www.kaggle.com/code/dfranzen/arc-prize-2024-solution-by-the-architects)
, with the following tweaks:
We use a much smaller model. We finetuned on
[Qwen2.5-0.5B](https://huggingface.co/Qwen/Qwen2.5-0.5B)
, using the scripts
[open-sourced by ARChitects](https://github.com/da-fr/arc-prize-2024)
, with ARC 2 data instead of ARC 1 evaluation set. We used 90 tasks from ARC 2 evaluation set so that we have 30 tasks to experiment with test time training (TTT)
Set
FAKE = True
in my solution notebook, you will be using the 30 holdout tasks
Do single task finetuning. Same finding as
[Guillermo Barbadillo](https://www.kaggle.com/code/ironbar/the-architects-single-task-ttt)
,
[ippeiogawa](https://www.kaggle.com/code/ippeiogawa/lb10-00-with-tweak-to-2024-1st-solution)
and
[rxe](https://www.kaggle.com/code/albartrose/10-4-notebook-arcitects-tweaked)
using a 8B model. We also tried TTT on
60 tasks together
per gpu, but never achieved a LB score more than 4.
During TTT, the ARChitects data augmentation approach is used to generate 560 augmented samples for each task, and then finetune for 140 steps (effective batch size 4). One more tweak: during test time, we don't train on token embedding layer or output projection.
We observe double descent as test time training steps increases
During inference, we simply use greedy sampling with 16 augmented samples (and we just use the transformers library)
(The motivation behind our "2024 solution but even simpler" kind of approach is that we were trying to work more on the local finetuning part. Unfortunately there hasn't been much great result that can be reported.)
To be fair, our solution is not optimized at all. We discarded the more complex inference method that could help. Inference is done with slower library (on the private test set, we estimate ~1.5 hours being spent on inference). We even attribute tasks to 4 gpus simply by a random split.
However, our notebook still finishes within ~5 hours on the private test set while achieving a LB score of 8.33, thanks to the smaller model.
Although our LB score is not very high, we hope the info reported here can be helpful to the community. Our main takeaways:
With a 0.5B decoder-only language model, you also want to do singe-task test time training (same as with a 8B model)
A 0.5B model can get close in terms of performances
We want to share our 0.5B model so that future ARC-AGI players can run easy and fast experiments using it
Finally we want to thank
[arcprize.org](https://arcprize.org)
for hosting this competition and the 2024 winners (and front runners) for sharing their solutions!