![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F264656%2F1872ebd8b23eeb052b5d0de5a20a5c9a%2F1.png?generation=1731337058067434&alt=media\")
Any feedback to improve this post or the paper is very welcome 👍!
\nThis post is almost a duplicate of the paper, I would recommend to go to the paper directly because it has better formatting.
\nMy solution is an implementation of the MindsAI team approach (test-time fine-tuning) that in addition to train the model to generate the test outputs (the original ARC task) it also trains the model to do other tasks such as learning the input distribution by generating new inputs.
\nIn the ARC challenge we have to learn a transformation rule given a few high-dimensional pairs of input and output images. The images can have a size of up to 30x30 pixels and each pixel can take 10 different colors. The images are not as complex as real world images, but nevertheless they are high dimensional data.
\n
To solve each ARC problem we have to find the right representation of the data. When humans solve the tasks, the biggest challenge is to find the right perspective to look at the problem. Once we have the right perspective of the data the ARC problems are trivial to solve.
\nThe right representation of the data allows to decrease the dimensionality of the data and makes possible to learn the transformation from very few examples.
\nIf we train a model to do tasks that require a good representation of the data, it's likely that the model will internally develop the required representation.
\nMy insight was that we could use the ARC problems in many different ways to learn that representation, not just in the original proposed task that asks to generate the output for an image given a few input-output pairs.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F264656%2F0cfb88d56cdfe4d4448745507699d961%2F2.png?generation=1731337090400140&alt=media\")
examples + input -> output. The original task of the ARC dataset.inputs -> input. Generating new inputs requires to understand the distribution of the grids. It could also be done with the outputs, that should also follow some distribution.examples -> code. This is the approach used by Ryan Greenblat with GPT-4ocode + input -> output. This is equivalent to the first task, but instead of giving examples as input, it gives the code definition of the problem.code -> inputs. Each input to a task follows some distribution, given a description of theinputs -> code. We could also do the opposite task, given some inputs write code to generate that distribution.examples + input + output -> is the output correct?. It is possible to train the model to verify wether a proposed output is correct.examples + input + output options-> select the correct output. We can train a model to select the correct output between multiple options.All the listed tasks require that the model learns some useful representation of the ARC image. The idea behind the Omni-ARC approach is to train a single model to do all the tasks, with the expectation that a shared representation across all the tasks will generalize better than training the model to do a single task.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F264656%2Fd875023e376c6b32bfec4fcd9accdc46%2F3.png?generation=1731337101860834&alt=media\")
Omni-ARC, a single model that does all the ARC-related tasks (and it has a very cool logo)
\nThe solution on a nutshell:
\nQwen2.5-0.5B-Instruct and fine-tune it on publicly available ARC datasets. The model was fine-tuned to:![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F264656%2F6582e5414c30bcabeae4b4835b80a273%2F4.png?generation=1731337122415869&alt=media\")
I used the following publicly available datasets for training:
\n| dataset | \nnumber of unique tasks | \n
|---|---|
| original ARC dataset | \n800 | \n
| Michael Hodel's RE-ARC dataset | \n400 | \n
| Simon Strandgaard's PQA dataset | \n7 | \n
| Simon Strandgaard's Tama dataset | \n50 | \n
| Mini-ARC | \n149 | \n
| nosound's hand crafted ARC tasks | \n9 | \n
| Andy Penrose's tasks | \n5 | \n
| TOTAL | \n1420 | \n
For all the datasets I trained the model to do two tasks:
\nexamples + input -> output. The original task of the ARC dataset.inputs -> input. Generating new inputs requires to understand the distribution of the grids. It could also be done with the outputs, that should also follow some distribution.Examples of newly generated inputs
\nFor each problem the same data augmentation was applied to all the inputs and outputs. Data augmentation was a composition of the following augmentations:
\nIn addition to the data augmentation I also did problem augmentation by applying a transformation only to the inputs or to the outputs. This transformation created new ARC problems by composing the original ARC transformation with randomly chosen new ones.
\nThese new transformations needed to be reversible, otherwise the new generated problems might not be solvable. I used the following additional transformations:
\nExamples of problem augmentation in the paper
\nI used a very simple text representation of the ARC grids as an input to the LLMs. The grid was enclosed on a Markdown code snippet, the shape was defined at the first line and each row was numbered.
\n```grid shape: x3\n \n \n \n```\nThe model was fine-tuned using LoRA. No significative improvement was found when doing full model fine-tuning and also on test-time fine-tuning it seemed to be beneficial to just fine-tune the already trained LoRA adapter instead of creating a fresh new adapter.
\nQwen2.5-0.5B-InstructI used huggingface's trl and accelerate libraries for the training.
\nFine-tuning a model on ARC tasks is not enough to do well on the private test set. By applying test-time fine-tuning we could improve the number of solved problems from 11 to 33 for one of the models that I trained along the challenge.
\nThis is my interpretation of the test-time fine-tuning:
\nn train samples, I fine-tuned the model using n-1 train samples and using the remaining sample as a test sample. The selection of the test sample was done randomly on the fly during training.Due to the limited submission time test-time fine-tuning was only applied to the canonical ARC task of predicting the test outputs. But it could also be applied to the task of generating new inputs, or to the tasks of verifying the correctness of the outputs.
\nThe unusual configuration of training a single model for each task with batch size 1 arose due to the limitations of compute and submission time. It was the configuration that allowed to learn faster the new test problems.
\nData augmentation was applied also at inference, and the data augmentation was reverted from the prediction to get the original output. 96 predictions were done for each problem and voting was used to select the most promising predictions. So just like MindsAI's AIRV (augment, inference, reverse augmentation, and vote).
\nInference was done using a temperature of 0.
\nVLLM was used to generate the predictions. Each fine-tuned model was used to generate predictions for its problem.
\nI ensembled my model predictions with the 2020 solution. Since the 2020 solution only requires CPU, I managed to run it on the background while I used the GPU for fine-tuning and inference with my model. I only had to be careful with the RAM usage because both jobs had to share the same memory.
\nThe ensemble strategy was very simple, just take the first attempt from each solution.
\nThis approach scored 40 on the ARC leaderboard.
\n
The same approach (without test-time fine-tuning) could solve 32% of the evaluation tasks, and when using voting with 32 predictions it achieved a top_2 accuracy of 22%. Due to limited hardware resources I didn't usually evaluated the models with test-time fine-tuning on the evaluation dataset. Kaggle provides 30 hours of GPU each week, but we could make 3 submissions a day which is equivalent to 36 hours of compute. Thus it was much cheaper to use the submissions to see the performance of the test-time fine-tuning where we had 7 times more compute available per week.
\nClearly this competition has shown that LLMs need test-time fine-tuning to do new tasks. Few-shot prompting is not enough for the model to learn novel tasks.
\nDuring the last weeks of the challenge I tried to continue with the Omni-ARC approach and train the model to:
\nThe idea was that we could improve the LB score if we replaced the voting selection mechanism by a more accurate one. Using trained models I generated wrong predictions for the original ARC dataset using a sampling temperature close to 1.
\n| method | \ntop 1 accuracy | \ntop 2 accuracy | \n
|---|---|---|
| voting | \n60.0% | \n70.0% | \n
| verification | \n59.3% | \n77.4% | \n
| selection | \n68.7% | \n81.0% | \n
As the table above shows I was able to achieve promising results on the evaluation dataset. Those numbers are for 32 predictions.
\nHowever I was not able to improve the LB score using this approach. My hypothesis is that the distribution of the predictions of a test-time fine-tuned model is different from the distribution of a frozen model. Thus the accuracy of voting for a test-time fine-tuned model might be much higher than the shown in the table for a frozen model.
\nThis verifier models could benefit from test-time fine-tuning, but I could not test the hypothesis due to the limited submission time.
\nMore information on Iteration 47 and Iteration 45.
\nI also tried to expand on the Omni-ARC approach by training the model to do the additional tasks:
\nexamples -> code. This is the approach used by Ryan Greenblat with GPT-4ocode + input -> output. This is equivalent to the first task, but instead of giving examples as input, it gives the code definition of the problem.To do so I build a small domain specific language (DSL) and recreated 285 of the ARC training tasks with python code. This was a laborious process that took around 3 weeks.
\nUnfortunately the model did not generalize well. It could only solve 5% of the evaluation tasks, and those tasks were very similar to the training tasks. On the private test set a lucky submission was able to solve 1 task.
\nI believe this approach has great potential, but I had to change to other approaches because the end of the challenge was close and other teams were improving in the leaderboard.
\nMore info on Iteration 26, Iteration 40 and Iteration 42.
\nOn the last weeks of the challenge I tried adding the BARC datasets to the training data. Surprisingly despite the enormous claimed number of different tasks (400k) I did not see any significative improvement either on the evaluation dataset or in the leaderboard. More information on Iteration 48. More external data.
\nThis is surprising because the original ARC dataset shows a clear trend when increasing the number of training tasks:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F264656%2F52306f5322b2d0d43fabe1f752f44c7a%2F5.png?generation=1731337162581753&alt=media\")
My guess is that the automatically generated tasks by GPT4 did not have too much novelty respect to the original ARC tasks.
\nQwen2.5-0.5B was the right model size for my approach and the available compute for submission.
On a first step I tried smaller models such as SmolLM2-135M and NanoLM-0.3B but they did not achieve the same accuracy as Qwen2.5-0.5B. More on Iteration 46. Revisit small LLMs
On my final attempt I also tried bigger models such as Qwen2.5-1.5B and Qwen2.5-7B. These models exhibit a higher data efficiency, they reach a smaller training loss for the same amount
\nof training steps. The problem with these models is that they are slower to fine-tune and inference at submission. Moreover due to VRAM requirements we have to decrease the length of the training samples. It's very likely that LB score could be improved with this bigger models if better hardware and more submission time is given.
So many fascinating ideas - well done! By teaching the model through code, you’re guiding it to reason more effectively and become \"transductive\", capable of generating numerous correct solutions. It’s remarkable how much a relatively small model can accomplish! 👏
", "votes": 2, "replies": [ { "id": 3162091, "author_name": "Derrick Donkor", "author_url": "", "post_date": "2025-03-28T19:37:11.190000", "content": "Algorithm on monotonic sequencing, can be found here. It is a new form of model code extraction latest upgrades will be released soon.
", "votes": 1, "replies": [] } ] }, { "id": 3048847, "author_name": "Francisco Zamora-Martinez", "author_url": "", "post_date": "2024-11-18T12:13:08.330000", "content": "Congrats @ironbar . The summary of the job you have done is amazing, I liked reading it a lot.
", "votes": 3, "replies": [] }, { "id": 3048026, "author_name": "JM", "author_url": "", "post_date": "2024-11-17T13:22:11.710000", "content": "Amazing work, thanks for sharing everything!
", "votes": 3, "replies": [] }, { "id": 3042556, "author_name": "CPMP", "author_url": "", "post_date": "2024-11-11T15:39:35.883000", "content": "Great writeup/paper. Congrats on the result.
", "votes": 2, "replies": [] }, { "id": 3065320, "author_name": "raconion", "author_url": "", "post_date": "2024-12-06T17:18:42.370000", "content": "The MindsAI team disappeared from lb. You are 2nd place now. Congrats!
", "votes": 1, "replies": [ { "id": 3065420, "author_name": "", "author_url": "", "post_date": "2024-12-06T19:18:50.760000", "content": "\"c-click!\" that was easy! (congrats, irregardlessly!)
", "votes": 0, "replies": [] }, { "id": 3065633, "author_name": "Guillermo Barbadillo", "author_url": "", "post_date": "2024-12-07T04:56:24.453000", "content": "Thanks, updating the title to avoid confusions.
", "votes": 0, "replies": [] } ] }, { "id": 3056607, "author_name": "Patrick Chan", "author_url": "", "post_date": "2024-11-27T07:01:44.717000", "content": "Congratulations and well done!
", "votes": 1, "replies": [] }, { "id": 3055633, "author_name": "RogerTaylor", "author_url": "", "post_date": "2024-11-25T23:59:58.727000", "content": "Congratulations @ironbar - excellent summary and all your hard work has paid off. Hope you continue to improve your solution.
", "votes": 1, "replies": [] }, { "id": 3049345, "author_name": "Randy Verdian", "author_url": "", "post_date": "2024-11-19T02:05:42.647000", "content": "very cool!!
", "votes": 1, "replies": [] }, { "id": 3046684, "author_name": "Xander", "author_url": "", "post_date": "2024-11-15T18:22:56.820000", "content": "nicely done!
", "votes": 1, "replies": [] }, { "id": 3046020, "author_name": "Guillermo Perez G", "author_url": "", "post_date": "2024-11-15T04:21:47.067000", "content": "Congratulations! This approach is new to me. I wonder if there is some basic theoretical information to get me started?
", "votes": 1, "replies": [ { "id": 3047068, "author_name": "Guillermo Barbadillo", "author_url": "", "post_date": "2024-11-16T08:04:26.537000", "content": "I don't understand your question, could you elaborate more?
", "votes": 1, "replies": [ { "id": 3065369, "author_name": "Guillermo Perez G", "author_url": "", "post_date": "2024-12-06T17:59:55.447000", "content": "Congratulations again on your progress to 2nd place, and for accepting public scrutiny… I'm interested in your strategy in assembling solutions and you give more information in the attached paper. I wonder what would happen if you took a second attempt. What's the difference from the first attempt?
\nHave a nice day!
", "votes": 0, "replies": [] } ] } ] }, { "id": 3045091, "author_name": "Sujay Vivek", "author_url": "", "post_date": "2024-11-14T06:44:43.197000", "content": "Excellent approach, got to learn a lot from this.
", "votes": 1, "replies": [] }, { "id": 3044491, "author_name": "Humayra Khanom Rime", "author_url": "", "post_date": "2024-11-13T13:10:22.457000", "content": "Excellent work! There is so much to learn here.
", "votes": 1, "replies": [] }, { "id": 3044219, "author_name": "Muhammad Irfan", "author_url": "", "post_date": "2024-11-13T06:56:21.037000", "content": "Many Congratulation! on your position.
", "votes": 1, "replies": [] }, { "id": 3044036, "author_name": "Bouh", "author_url": "", "post_date": "2024-11-12T23:32:45.213000", "content": "Great work ! And congrats with the 3rd place ! 💪
", "votes": 1, "replies": [] }, { "id": 3043968, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2024-11-12T21:22:50.870000", "content": "What an amazing write-up, thanks for the paper as well. 🥳
", "votes": 1, "replies": [] }, { "id": 3043760, "author_name": "Veniamin Nelin", "author_url": "", "post_date": "2024-11-12T16:54:36.827000", "content": "Impressive work! Thank you for sharing your approach, and congratulations on 3rd place!
", "votes": 1, "replies": [] }, { "id": 3043190, "author_name": "TuMinhDang", "author_url": "", "post_date": "2024-11-12T06:35:18.980000", "content": "Congratulations on your 3rd leaderboard.
", "votes": 1, "replies": [] }, { "id": 3043151, "author_name": "Carlos Schenone", "author_url": "", "post_date": "2024-11-12T05:54:58.813000", "content": "Congrats on your 3rd place! great work!
", "votes": 1, "replies": [] }, { "id": 3043132, "author_name": "Guillermo Barbadillo", "author_url": "", "post_date": "2024-11-12T05:40:17.003000", "content": "I'm updating the title to 3rd position because that is the new leaderboard standing and to avoid confusions with @alijs1 write up.
", "votes": 1, "replies": [] }, { "id": 3043068, "author_name": "Natapong Nitarach (Schwyter)", "author_url": "", "post_date": "2024-11-12T04:29:29.010000", "content": "Great work, fantastic omni-arc 🔥
", "votes": 1, "replies": [] }, { "id": 3042656, "author_name": "", "author_url": "", "post_date": "2024-11-11T16:45:21", "content": "Awesome solution. Congratulations for your 4st place.
\nI have failed in my attempts to fintune LLM for ARC. Honestly, your approach is much more novel compared to my part.
", "votes": 1, "replies": [] }, { "id": 3048065, "author_name": "Sumit_08", "author_url": "", "post_date": "2024-11-17T14:16:13.817000", "content": "Congratulations Bro. I love the way of your logical thinking that the path you make to execute this process.
", "votes": 2, "replies": [] }, { "id": 3044428, "author_name": "Sinan Calisir", "author_url": "", "post_date": "2024-11-13T12:03:05.343000", "content": "Amazing work!! Thank you for the detailed write up. So much to dig into.
", "votes": 2, "replies": [] }, { "id": 3044285, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2024-11-13T08:16:04.850000", "content": "Great work! So many things to learn here.
\nOne question: I am intrigued by the fact that you are fine-tuning a separate model for every test task. Is the intuition that too many different types of test tasks confuse the model? In that case, can we really talk about \"general\" AI here?
", "votes": 2, "replies": [ { "id": 3044313, "author_name": "Guillermo Barbadillo", "author_url": "", "post_date": "2024-11-13T08:48:10.970000", "content": "It simply gave better results.
\nMy intuition is that since all the test problems are unique, it is not helpful to fine-tune the model to do all the test problems. Furthermore, the different problems could be contradictory and confusing.
\nBear in mind that submission time is limited, and we have to use it as best as possible. It is very likely that without time constraints, a single model could learn to do all the test problems as well as the current solution.
", "votes": 5, "replies": [ { "id": 3044408, "author_name": "narsil (jobs-in-data.com)", "author_url": "", "post_date": "2024-11-13T11:24:32.510000", "content": "Makes sense, thank you.
", "votes": 1, "replies": [] } ] } ] }, { "id": 3043187, "author_name": "Arman Mehdi Kazmi", "author_url": "", "post_date": "2024-11-12T06:33:17.253000", "content": "The paper is written with great effort,
\nSO Insightfull ,
\nCongratulations on your 3rd place omni-arc.
Can you share some performance data in terms of the speed on kaggle? Assuming you used T4x2.
\nHow long does test time learning take for each epoch, how many epochs, how long does token generations take and the batch size? It has to be for a specific task.
\nI started the competition in the last month, did not have time to train a larger model or larger context window. Mine has 160M parameters and trained with around 2500 tokens. So I am curious about your numbers and I’d like to change my model and context window.
\nI can share mine to compare.
", "votes": 2, "replies": [ { "id": 3043204, "author_name": "Guillermo Barbadillo", "author_url": "", "post_date": "2024-11-12T07:03:39.163000", "content": "Test-time fine-tuning uses almost all the time of the submission. Let's assume that 10 hours are used for test-time fine-tuning. So that is 360 seconds per problem. I trained for around 320 steps, so that would roughly be 1 step per second on average. The batch size is 1. Regarding model size, the LoRA adapter has around 17M parameters.
", "votes": 1, "replies": [ { "id": 3043229, "author_name": "Guboken", "author_url": "", "post_date": "2024-11-12T07:55:46.450000", "content": "Interesting approach with the even distribution of training time. Have you considered allocating the training time dynamically across problems? Since ARC problems vary significantly in complexity, some might benefit from more than 320 steps while others might converge much earlier. You might be able to improve beyond 40% accuracy by implementing early stopping on simpler problems and reallocating that compute to more complex ones that need additional training steps. This could potentially give you more efficient use of the 10-hour training budget. What metrics would you use to determine which problems deserve more training time? Thanks for the article! 🥰
", "votes": 3, "replies": [] }, { "id": 3043290, "author_name": "Guillermo Barbadillo", "author_url": "", "post_date": "2024-11-12T09:39:56.067000", "content": "Yes, that is something totally worth exploring. I had that idea in the middle of the challenge, but some quick experiment did not show a clear relation between training loss and the accuracy of inference.
\nI should have devoted more time exploring that direction and optimizing the test-time fine-tuning configuration.
", "votes": 2, "replies": [] }, { "id": 3044108, "author_name": "Nikola Hu", "author_url": "", "post_date": "2024-11-13T03:30:17.670000", "content": "I did the experiment @guboken suggested - it did not help the private test score. I hypothesize that my model did not perform well on the very large tasks anyway.
\nOne of the tests:
\nSort by the challenge size. Start from easy ones.
\nAllocate budget based on the average remaining time (total time / remaining count) * factor
I tested factor = 1 and factor = 1.5. no difference.
", "votes": 0, "replies": [] }, { "id": 3044111, "author_name": "Nikola Hu", "author_url": "", "post_date": "2024-11-13T03:38:17.817000", "content": "320 steps are in the same magnitude as mine ( my total parameter size is smaller than yours, at 160M. Lora did not apply to my case, it is a start-from-scratch model )
\nOur definition of steps might be different. Mine is batches, each batch has 1-6 samples, about 6-10 batches are one epoch, I run 16 epochs if possible.
\nIn my case, inference could take slightly longer than the training part because my model has to generate tokens one by one. I did batch inference, which generates multiple answers in parallel.
\nFor training, the time is like this:
\nBatch count * transformer computation time * 4 (backpropagation overhead)
For inference, the complexity is like this:
\nNumber of tokens (batch parallel does not count) * transformer computation time
Your algorithm inference 96 times. Mine is at a similar level.
\nHow did you save the inference time? You did some excellent inference optimization!
", "votes": 0, "replies": [] }, { "id": 3044133, "author_name": "Guillermo Barbadillo", "author_url": "", "post_date": "2024-11-13T04:33:26.867000", "content": "My only trick to speedup inference is to use VLLM. From previous competitions, I have observed that is much faster than doing inference with the transformers library.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3043046, "author_name": "Olayinka", "author_url": "", "post_date": "2024-11-12T03:36:44.913000", "content": "The paper is written with great skill. Congratulations on your 3rd place my friend.
", "votes": 2, "replies": [] }, { "id": 3042757, "author_name": "Richard Emslie", "author_url": "", "post_date": "2024-11-11T18:11:55.267000", "content": "What a fantastic write up, especially those 50 work iterations! Look forward to reading through them all. Thank you and congrats!
", "votes": 2, "replies": [] }, { "id": 3042548, "author_name": "ivan", "author_url": "", "post_date": "2024-11-11T15:36:08.797000", "content": "Great job! Running 2020 solution in background - excellent way to manage resources. I had no idea this could be done. Btw, why you didn't used ConceptARC dataset?
", "votes": 2, "replies": [ { "id": 3042689, "author_name": "Guillermo Barbadillo", "author_url": "", "post_date": "2024-11-11T17:14:35.737000", "content": "If I remember well, I tried it but didn't give any significative improvement.
", "votes": 1, "replies": [ { "id": 3042716, "author_name": "Guillermo Barbadillo", "author_url": "", "post_date": "2024-11-11T17:40:18.250000", "content": "", "votes": 2, "replies": [] }, { "id": 3042721, "author_name": "ivan", "author_url": "", "post_date": "2024-11-11T17:47:38.763000", "content": "Wow! I missed that you kept a diary for each iteration! Thanks
", "votes": 2, "replies": [] }, { "id": 3042906, "author_name": "", "author_url": "", "post_date": "2024-11-11T22:03:52.210000", "content": "Congratulations on making it to the money final! Your focus, your success, and your impeccable attention to detail are a joyful inspiration!
", "votes": 1, "replies": [] } ] } ] }, { "id": 3240165, "author_name": "", "author_url": "", "post_date": "2025-07-03T14:20:11.237000", "content": "", "votes": 0, "replies": [] }, { "id": 3206069, "author_name": "", "author_url": "", "post_date": "2025-05-20T19:27:55.110000", "content": "", "votes": 0, "replies": [] }, { "id": 3175696, "author_name": "", "author_url": "", "post_date": "2025-04-10T13:42:20.687000", "content": "", "votes": 0, "replies": [] }, { "id": 3042679, "author_name": "", "author_url": "", "post_date": "2024-11-11T17:03:31.517000", "content": "", "votes": 0, "replies": [ { "id": 3042692, "author_name": "", "author_url": "", "post_date": "2024-11-11T17:19:07.560000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 3049674, "author_name": "", "author_url": "", "post_date": "2024-11-19T12:05:56.100000", "content": "", "votes": 0, "replies": [] }, { "id": 3047458, "author_name": "", "author_url": "", "post_date": "2024-11-16T17:42:42.960000", "content": "", "votes": 1, "replies": [] }, { "id": 3044496, "author_name": "", "author_url": "", "post_date": "2024-11-13T13:17:32.683000", "content": "", "votes": 1, "replies": [] }, { "id": 3046439, "author_name": "", "author_url": "", "post_date": "2024-11-15T13:30:43.603000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 55, "messages": [ { "id": 3042498, "postDate": "2024-11-11T15:04:54.093Z", "content": "Any feedback to improve this post or the paper is very welcome 👍!
\nThis post is almost a duplicate of the paper, I would recommend to go to the paper directly because it has better formatting.
\nMy solution is an implementation of the MindsAI team approach (test-time fine-tuning) that in addition to train the model to generate the test outputs (the original ARC task) it also trains the model to do other tasks such as learning the input distribution by generating new inputs.
\nIn the ARC challenge we have to learn a transformation rule given a few high-dimensional pairs of input and output images. The images can have a size of up to 30x30 pixels and each pixel can take 10 different colors. The images are not as complex as real world images, but nevertheless they are high dimensional data.
\n
To solve each ARC problem we have to find the right representation of the data. When humans solve the tasks, the biggest challenge is to find the right perspective to look at the problem. Once we have the right perspective of the data the ARC problems are trivial to solve.
\nThe right representation of the data allows to decrease the dimensionality of the data and makes possible to learn the transformation from very few examples.
\nIf we train a model to do tasks that require a good representation of the data, it's likely that the model will internally develop the required representation.
\nMy insight was that we could use the ARC problems in many different ways to learn that representation, not just in the original proposed task that asks to generate the output for an image given a few input-output pairs.
\n
examples + input -> output. The original task of the ARC dataset.inputs -> input. Generating new inputs requires to understand the distribution of the grids. It could also be done with the outputs, that should also follow some distribution.examples -> code. This is the approach used by Ryan Greenblat with GPT-4ocode + input -> output. This is equivalent to the first task, but instead of giving examples as input, it gives the code definition of the problem.code -> inputs. Each input to a task follows some distribution, given a description of theinputs -> code. We could also do the opposite task, given some inputs write code to generate that distribution.examples + input + output -> is the output correct?. It is possible to train the model to verify wether a proposed output is correct.examples + input + output options-> select the correct output. We can train a model to select the correct output between multiple options.All the listed tasks require that the model learns some useful representation of the ARC image. The idea behind the Omni-ARC approach is to train a single model to do all the tasks, with the expectation that a shared representation across all the tasks will generalize better than training the model to do a single task.
\n
Omni-ARC, a single model that does all the ARC-related tasks (and it has a very cool logo)
\nThe solution on a nutshell:
\nQwen2.5-0.5B-Instruct and fine-tune it on publicly available ARC datasets. The model was fine-tuned to:
I used the following publicly available datasets for training:
\n| dataset | \nnumber of unique tasks | \n
|---|---|
| original ARC dataset | \n800 | \n
| Michael Hodel's RE-ARC dataset | \n400 | \n
| Simon Strandgaard's PQA dataset | \n7 | \n
| Simon Strandgaard's Tama dataset | \n50 | \n
| Mini-ARC | \n149 | \n
| nosound's hand crafted ARC tasks | \n9 | \n
| Andy Penrose's tasks | \n5 | \n
| TOTAL | \n1420 | \n
For all the datasets I trained the model to do two tasks:
\nexamples + input -> output. The original task of the ARC dataset.inputs -> input. Generating new inputs requires to understand the distribution of the grids. It could also be done with the outputs, that should also follow some distribution.Examples of newly generated inputs
\nFor each problem the same data augmentation was applied to all the inputs and outputs. Data augmentation was a composition of the following augmentations:
\nIn addition to the data augmentation I also did problem augmentation by applying a transformation only to the inputs or to the outputs. This transformation created new ARC problems by composing the original ARC transformation with randomly chosen new ones.
\nThese new transformations needed to be reversible, otherwise the new generated problems might not be solvable. I used the following additional transformations:
\nExamples of problem augmentation in the paper
\nI used a very simple text representation of the ARC grids as an input to the LLMs. The grid was enclosed on a Markdown code snippet, the shape was defined at the first line and each row was numbered.
\n```grid shape: x3\n \n \n \n```\nThe model was fine-tuned using LoRA. No significative improvement was found when doing full model fine-tuning and also on test-time fine-tuning it seemed to be beneficial to just fine-tune the already trained LoRA adapter instead of creating a fresh new adapter.
\nQwen2.5-0.5B-InstructI used huggingface's trl and accelerate libraries for the training.
\nFine-tuning a model on ARC tasks is not enough to do well on the private test set. By applying test-time fine-tuning we could improve the number of solved problems from 11 to 33 for one of the models that I trained along the challenge.
\nThis is my interpretation of the test-time fine-tuning:
\nn train samples, I fine-tuned the model using n-1 train samples and using the remaining sample as a test sample. The selection of the test sample was done randomly on the fly during training.Due to the limited submission time test-time fine-tuning was only applied to the canonical ARC task of predicting the test outputs. But it could also be applied to the task of generating new inputs, or to the tasks of verifying the correctness of the outputs.
\nThe unusual configuration of training a single model for each task with batch size 1 arose due to the limitations of compute and submission time. It was the configuration that allowed to learn faster the new test problems.
\nData augmentation was applied also at inference, and the data augmentation was reverted from the prediction to get the original output. 96 predictions were done for each problem and voting was used to select the most promising predictions. So just like MindsAI's AIRV (augment, inference, reverse augmentation, and vote).
\nInference was done using a temperature of 0.
\nVLLM was used to generate the predictions. Each fine-tuned model was used to generate predictions for its problem.
\nI ensembled my model predictions with the 2020 solution. Since the 2020 solution only requires CPU, I managed to run it on the background while I used the GPU for fine-tuning and inference with my model. I only had to be careful with the RAM usage because both jobs had to share the same memory.
\nThe ensemble strategy was very simple, just take the first attempt from each solution.
\nThis approach scored 40 on the ARC leaderboard.
\n
The same approach (without test-time fine-tuning) could solve 32% of the evaluation tasks, and when using voting with 32 predictions it achieved a top_2 accuracy of 22%. Due to limited hardware resources I didn't usually evaluated the models with test-time fine-tuning on the evaluation dataset. Kaggle provides 30 hours of GPU each week, but we could make 3 submissions a day which is equivalent to 36 hours of compute. Thus it was much cheaper to use the submissions to see the performance of the test-time fine-tuning where we had 7 times more compute available per week.
\nClearly this competition has shown that LLMs need test-time fine-tuning to do new tasks. Few-shot prompting is not enough for the model to learn novel tasks.
\nDuring the last weeks of the challenge I tried to continue with the Omni-ARC approach and train the model to:
\nThe idea was that we could improve the LB score if we replaced the voting selection mechanism by a more accurate one. Using trained models I generated wrong predictions for the original ARC dataset using a sampling temperature close to 1.
\n| method | \ntop 1 accuracy | \ntop 2 accuracy | \n
|---|---|---|
| voting | \n60.0% | \n70.0% | \n
| verification | \n59.3% | \n77.4% | \n
| selection | \n68.7% | \n81.0% | \n
As the table above shows I was able to achieve promising results on the evaluation dataset. Those numbers are for 32 predictions.
\nHowever I was not able to improve the LB score using this approach. My hypothesis is that the distribution of the predictions of a test-time fine-tuned model is different from the distribution of a frozen model. Thus the accuracy of voting for a test-time fine-tuned model might be much higher than the shown in the table for a frozen model.
\nThis verifier models could benefit from test-time fine-tuning, but I could not test the hypothesis due to the limited submission time.
\nMore information on Iteration 47 and Iteration 45.
\nI also tried to expand on the Omni-ARC approach by training the model to do the additional tasks:
\nexamples -> code. This is the approach used by Ryan Greenblat with GPT-4ocode + input -> output. This is equivalent to the first task, but instead of giving examples as input, it gives the code definition of the problem.To do so I build a small domain specific language (DSL) and recreated 285 of the ARC training tasks with python code. This was a laborious process that took around 3 weeks.
\nUnfortunately the model did not generalize well. It could only solve 5% of the evaluation tasks, and those tasks were very similar to the training tasks. On the private test set a lucky submission was able to solve 1 task.
\nI believe this approach has great potential, but I had to change to other approaches because the end of the challenge was close and other teams were improving in the leaderboard.
\nMore info on Iteration 26, Iteration 40 and Iteration 42.
\nOn the last weeks of the challenge I tried adding the BARC datasets to the training data. Surprisingly despite the enormous claimed number of different tasks (400k) I did not see any significative improvement either on the evaluation dataset or in the leaderboard. More information on Iteration 48. More external data.
\nThis is surprising because the original ARC dataset shows a clear trend when increasing the number of training tasks:
\n
My guess is that the automatically generated tasks by GPT4 did not have too much novelty respect to the original ARC tasks.
\nQwen2.5-0.5B was the right model size for my approach and the available compute for submission.
On a first step I tried smaller models such as SmolLM2-135M and NanoLM-0.3B but they did not achieve the same accuracy as Qwen2.5-0.5B. More on Iteration 46. Revisit small LLMs
On my final attempt I also tried bigger models such as Qwen2.5-1.5B and Qwen2.5-7B. These models exhibit a higher data efficiency, they reach a smaller training loss for the same amount
\nof training steps. The problem with these models is that they are slower to fine-tune and inference at submission. Moreover due to VRAM requirements we have to decrease the length of the training samples. It's very likely that LB score could be improved with this bigger models if better hardware and more submission time is given.
So many fascinating ideas - well done! By teaching the model through code, you’re guiding it to reason more effectively and become \"transductive\", capable of generating numerous correct solutions. It’s remarkable how much a relatively small model can accomplish! 👏
", "rawMarkdown": "So many fascinating ideas - well done! By teaching the model through code, you’re guiding it to reason more effectively and become \"transductive\", capable of generating numerous correct solutions. It’s remarkable how much a relatively small model can accomplish! 👏", "votes": 2, "replies": [ { "id": 3162091, "postDate": "2025-03-28T19:37:11.190Z", "content": "Algorithm on monotonic sequencing, can be found here. It is a new form of model code extraction latest upgrades will be released soon.
", "rawMarkdown": "Algorithm on monotonic sequencing, can be found [here](https://drive.google.com/file/d/1EHsfGPonE_enXndZaaDQx4O18InP49dD/view?usp=sharing&usp=embed_facebook). It is a new form of model code extraction latest upgrades will be released soon.", "votes": 1 } ] }, { "id": 3048847, "postDate": "2024-11-18T12:13:08.330Z", "content": "Congrats @ironbar . The summary of the job you have done is amazing, I liked reading it a lot.
", "rawMarkdown": "Congrats @ironbar . The summary of the job you have done is amazing, I liked reading it a lot.", "votes": 3 }, { "id": 3048026, "postDate": "2024-11-17T13:22:11.710Z", "content": "Amazing work, thanks for sharing everything!
", "rawMarkdown": "Amazing work, thanks for sharing everything!", "votes": 3 }, { "id": 3042556, "postDate": "2024-11-11T15:39:35.883Z", "content": "Great writeup/paper. Congrats on the result.
", "rawMarkdown": "Great writeup/paper. Congrats on the result.", "votes": 2 }, { "id": 3065320, "postDate": "2024-12-06T17:18:42.370Z", "content": "The MindsAI team disappeared from lb. You are 2nd place now. Congrats!
", "rawMarkdown": "The MindsAI team disappeared from lb. You are 2nd place now. Congrats!", "votes": 1, "replies": [ { "id": 3065420, "postDate": "2024-12-06T19:18:50.760Z", "content": "\"c-click!\" that was easy! (congrats, irregardlessly!)
", "rawMarkdown": "\"c-click!\" that was easy! (congrats, irregardlessly!)", "isDeleted": true }, { "id": 3065633, "postDate": "2024-12-07T04:56:24.453Z", "content": "Thanks, updating the title to avoid confusions.
", "rawMarkdown": "Thanks, updating the title to avoid confusions." } ] }, { "id": 3056607, "postDate": "2024-11-27T07:01:44.717Z", "content": "Congratulations and well done!
", "rawMarkdown": "Congratulations and well done!", "votes": 1 }, { "id": 3055633, "postDate": "2024-11-25T23:59:58.727Z", "content": "Congratulations @ironbar - excellent summary and all your hard work has paid off. Hope you continue to improve your solution.
", "rawMarkdown": "Congratulations @ironbar - excellent summary and all your hard work has paid off. Hope you continue to improve your solution.", "votes": 1 }, { "id": 3049345, "postDate": "2024-11-19T02:05:42.647Z", "content": "very cool!!
", "rawMarkdown": "very cool!!", "votes": 1 }, { "id": 3046684, "postDate": "2024-11-15T18:22:56.820Z", "content": "nicely done!
", "rawMarkdown": "nicely done!", "votes": 1 }, { "id": 3046020, "postDate": "2024-11-15T04:21:47.067Z", "content": "Congratulations! This approach is new to me. I wonder if there is some basic theoretical information to get me started?
", "rawMarkdown": "Congratulations! This approach is new to me. I wonder if there is some basic theoretical information to get me started?", "votes": 1, "replies": [ { "id": 3047068, "postDate": "2024-11-16T08:04:26.537Z", "content": "I don't understand your question, could you elaborate more?
", "rawMarkdown": "I don't understand your question, could you elaborate more?", "votes": 1, "replies": [ { "id": 3065369, "postDate": "2024-12-06T17:59:55.447Z", "content": "Congratulations again on your progress to 2nd place, and for accepting public scrutiny… I'm interested in your strategy in assembling solutions and you give more information in the attached paper. I wonder what would happen if you took a second attempt. What's the difference from the first attempt?
\nHave a nice day!
", "rawMarkdown": "Congratulations again on your progress to 2nd place, and for accepting public scrutiny... I'm interested in your strategy in assembling solutions and you give more information in the attached paper. I wonder what would happen if you took a second attempt. What's the difference from the first attempt?\n\nHave a nice day!" } ] } ] }, { "id": 3045091, "postDate": "2024-11-14T06:44:43.197Z", "content": "Excellent approach, got to learn a lot from this.
", "rawMarkdown": "Excellent approach, got to learn a lot from this.", "votes": 1 }, { "id": 3044491, "postDate": "2024-11-13T13:10:22.457Z", "content": "Excellent work! There is so much to learn here.
", "rawMarkdown": "Excellent work! There is so much to learn here.", "votes": 1 }, { "id": 3044219, "postDate": "2024-11-13T06:56:21.037Z", "content": "Many Congratulation! on your position.
", "rawMarkdown": "Many Congratulation! on your position.", "votes": 1 }, { "id": 3044036, "postDate": "2024-11-12T23:32:45.213Z", "content": "Great work ! And congrats with the 3rd place ! 💪
", "rawMarkdown": "Great work ! And congrats with the 3rd place ! 💪", "votes": 1 }, { "id": 3043968, "postDate": "2024-11-12T21:22:50.870Z", "content": "What an amazing write-up, thanks for the paper as well. 🥳
", "rawMarkdown": "What an amazing write-up, thanks for the paper as well. 🥳", "votes": 1 }, { "id": 3043760, "postDate": "2024-11-12T16:54:36.827Z", "content": "Impressive work! Thank you for sharing your approach, and congratulations on 3rd place!
", "rawMarkdown": "Impressive work! Thank you for sharing your approach, and congratulations on 3rd place!", "votes": 1 }, { "id": 3043190, "postDate": "2024-11-12T06:35:18.980Z", "content": "Congratulations on your 3rd leaderboard.
", "rawMarkdown": "Congratulations on your 3rd leaderboard.", "votes": 1 }, { "id": 3043151, "postDate": "2024-11-12T05:54:58.813Z", "content": "Congrats on your 3rd place! great work!
", "rawMarkdown": "Congrats on your 3rd place! great work!", "votes": 1 }, { "id": 3043132, "postDate": "2024-11-12T05:40:17.003Z", "content": "I'm updating the title to 3rd position because that is the new leaderboard standing and to avoid confusions with @alijs1 write up.
", "rawMarkdown": "I'm updating the title to 3rd position because that is the new leaderboard standing and to avoid confusions with @alijs1 write up.", "votes": 1 }, { "id": 3043068, "postDate": "2024-11-12T04:29:29.010Z", "content": "Great work, fantastic omni-arc 🔥
", "rawMarkdown": "Great work, fantastic omni-arc 🔥", "votes": 1 }, { "id": 3042656, "postDate": "2024-11-11T16:45:21Z", "content": "Awesome solution. Congratulations for your 4st place.
\nI have failed in my attempts to fintune LLM for ARC. Honestly, your approach is much more novel compared to my part.
", "rawMarkdown": "Awesome solution. Congratulations for your 4st place.\n\nI have failed in my attempts to fintune LLM for ARC. Honestly, your approach is much more novel compared to my part.", "votes": 1 }, { "id": 3048065, "postDate": "2024-11-17T14:16:13.817Z", "content": "Congratulations Bro. I love the way of your logical thinking that the path you make to execute this process.
", "rawMarkdown": "Congratulations Bro. I love the way of your logical thinking that the path you make to execute this process.", "votes": 2 }, { "id": 3044428, "postDate": "2024-11-13T12:03:05.343Z", "content": "Amazing work!! Thank you for the detailed write up. So much to dig into.
", "rawMarkdown": "Amazing work!! Thank you for the detailed write up. So much to dig into. ", "votes": 2 }, { "id": 3044285, "postDate": "2024-11-13T08:16:04.850Z", "content": "Great work! So many things to learn here.
\nOne question: I am intrigued by the fact that you are fine-tuning a separate model for every test task. Is the intuition that too many different types of test tasks confuse the model? In that case, can we really talk about \"general\" AI here?
", "rawMarkdown": "Great work! So many things to learn here.\n\nOne question: I am intrigued by the fact that you are fine-tuning a separate model for every test task. Is the intuition that too many different types of test tasks confuse the model? In that case, can we really talk about \"general\" AI here?", "votes": 2, "replies": [ { "id": 3044313, "postDate": "2024-11-13T08:48:10.970Z", "content": "It simply gave better results.
\nMy intuition is that since all the test problems are unique, it is not helpful to fine-tune the model to do all the test problems. Furthermore, the different problems could be contradictory and confusing.
\nBear in mind that submission time is limited, and we have to use it as best as possible. It is very likely that without time constraints, a single model could learn to do all the test problems as well as the current solution.
", "rawMarkdown": "It simply gave better results. \n\nMy intuition is that since all the test problems are unique, it is not helpful to fine-tune the model to do all the test problems. Furthermore, the different problems could be contradictory and confusing.\n\nBear in mind that submission time is limited, and we have to use it as best as possible. It is very likely that without time constraints, a single model could learn to do all the test problems as well as the current solution. ", "votes": 5, "replies": [ { "id": 3044408, "postDate": "2024-11-13T11:24:32.510Z", "content": "Makes sense, thank you.
", "rawMarkdown": "Makes sense, thank you.", "votes": 1 } ] } ] }, { "id": 3043187, "postDate": "2024-11-12T06:33:17.253Z", "content": "The paper is written with great effort,
\nSO Insightfull ,
\nCongratulations on your 3rd place omni-arc.
Can you share some performance data in terms of the speed on kaggle? Assuming you used T4x2.
\nHow long does test time learning take for each epoch, how many epochs, how long does token generations take and the batch size? It has to be for a specific task.
\nI started the competition in the last month, did not have time to train a larger model or larger context window. Mine has 160M parameters and trained with around 2500 tokens. So I am curious about your numbers and I’d like to change my model and context window.
\nI can share mine to compare.
", "rawMarkdown": "Can you share some performance data in terms of the speed on kaggle? Assuming you used T4x2.\n\nHow long does test time learning take for each epoch, how many epochs, how long does token generations take and the batch size? It has to be for a specific task.\n\nI started the competition in the last month, did not have time to train a larger model or larger context window. Mine has 160M parameters and trained with around 2500 tokens. So I am curious about your numbers and I’d like to change my model and context window.\n\nI can share mine to compare.", "votes": 2, "replies": [ { "id": 3043204, "postDate": "2024-11-12T07:03:39.163Z", "content": "Test-time fine-tuning uses almost all the time of the submission. Let's assume that 10 hours are used for test-time fine-tuning. So that is 360 seconds per problem. I trained for around 320 steps, so that would roughly be 1 step per second on average. The batch size is 1. Regarding model size, the LoRA adapter has around 17M parameters.
", "rawMarkdown": "Test-time fine-tuning uses almost all the time of the submission. Let's assume that 10 hours are used for test-time fine-tuning. So that is 360 seconds per problem. I trained for around 320 steps, so that would roughly be 1 step per second on average. The batch size is 1. Regarding model size, the LoRA adapter has around 17M parameters.", "votes": 1, "replies": [ { "id": 3043229, "postDate": "2024-11-12T07:55:46.450Z", "content": "Interesting approach with the even distribution of training time. Have you considered allocating the training time dynamically across problems? Since ARC problems vary significantly in complexity, some might benefit from more than 320 steps while others might converge much earlier. You might be able to improve beyond 40% accuracy by implementing early stopping on simpler problems and reallocating that compute to more complex ones that need additional training steps. This could potentially give you more efficient use of the 10-hour training budget. What metrics would you use to determine which problems deserve more training time? Thanks for the article! 🥰
", "rawMarkdown": "Interesting approach with the even distribution of training time. Have you considered allocating the training time dynamically across problems? Since ARC problems vary significantly in complexity, some might benefit from more than 320 steps while others might converge much earlier. You might be able to improve beyond 40% accuracy by implementing early stopping on simpler problems and reallocating that compute to more complex ones that need additional training steps. This could potentially give you more efficient use of the 10-hour training budget. What metrics would you use to determine which problems deserve more training time? Thanks for the article! 🥰", "votes": 3 }, { "id": 3043290, "postDate": "2024-11-12T09:39:56.067Z", "content": "Yes, that is something totally worth exploring. I had that idea in the middle of the challenge, but some quick experiment did not show a clear relation between training loss and the accuracy of inference.
\nI should have devoted more time exploring that direction and optimizing the test-time fine-tuning configuration.
", "rawMarkdown": "Yes, that is something totally worth exploring. I had that idea in the middle of the challenge, but some quick experiment did not show a clear relation between training loss and the accuracy of inference. \n\nI should have devoted more time exploring that direction and optimizing the test-time fine-tuning configuration.", "votes": 2 }, { "id": 3044108, "postDate": "2024-11-13T03:30:17.670Z", "content": "I did the experiment @guboken suggested - it did not help the private test score. I hypothesize that my model did not perform well on the very large tasks anyway.
\nOne of the tests:
\nSort by the challenge size. Start from easy ones.
\nAllocate budget based on the average remaining time (total time / remaining count) * factor
I tested factor = 1 and factor = 1.5. no difference.
", "rawMarkdown": "I did the experiment @guboken suggested - it did not help the private test score. I hypothesize that my model did not perform well on the very large tasks anyway.\n\nOne of the tests:\n\nSort by the challenge size. Start from easy ones.\nAllocate budget based on the average remaining time (total time / remaining count) * factor\n\nI tested factor = 1 and factor = 1.5. no difference." }, { "id": 3044111, "postDate": "2024-11-13T03:38:17.817Z", "content": "320 steps are in the same magnitude as mine ( my total parameter size is smaller than yours, at 160M. Lora did not apply to my case, it is a start-from-scratch model )
\nOur definition of steps might be different. Mine is batches, each batch has 1-6 samples, about 6-10 batches are one epoch, I run 16 epochs if possible.
\nIn my case, inference could take slightly longer than the training part because my model has to generate tokens one by one. I did batch inference, which generates multiple answers in parallel.
\nFor training, the time is like this:
\nBatch count * transformer computation time * 4 (backpropagation overhead)
For inference, the complexity is like this:
\nNumber of tokens (batch parallel does not count) * transformer computation time
Your algorithm inference 96 times. Mine is at a similar level.
\nHow did you save the inference time? You did some excellent inference optimization!
", "rawMarkdown": "320 steps are in the same magnitude as mine ( my total parameter size is smaller than yours, at 160M. Lora did not apply to my case, it is a start-from-scratch model )\n\nOur definition of steps might be different. Mine is batches, each batch has 1-6 samples, about 6-10 batches are one epoch, I run 16 epochs if possible. \n\nIn my case, inference could take slightly longer than the training part because my model has to generate tokens one by one. I did batch inference, which generates multiple answers in parallel.\n\nFor training, the time is like this:\nBatch count * transformer computation time * 4 (backpropagation overhead)\n\nFor inference, the complexity is like this:\nNumber of tokens (batch parallel does not count) * transformer computation time \n\nYour algorithm inference 96 times. Mine is at a similar level.\n\nHow did you save the inference time? You did some excellent inference optimization!" }, { "id": 3044133, "postDate": "2024-11-13T04:33:26.867Z", "content": "My only trick to speedup inference is to use VLLM. From previous competitions, I have observed that is much faster than doing inference with the transformers library.
", "rawMarkdown": "My only trick to speedup inference is to use VLLM. From previous competitions, I have observed that is much faster than doing inference with the transformers library." } ] } ] }, { "id": 3043046, "postDate": "2024-11-12T03:36:44.913Z", "content": "The paper is written with great skill. Congratulations on your 3rd place my friend.
", "rawMarkdown": "The paper is written with great skill. Congratulations on your 3rd place my friend.", "votes": 2 }, { "id": 3042757, "postDate": "2024-11-11T18:11:55.267Z", "content": "What a fantastic write up, especially those 50 work iterations! Look forward to reading through them all. Thank you and congrats!
", "rawMarkdown": "What a fantastic write up, especially those 50 work iterations! Look forward to reading through them all. Thank you and congrats! ", "votes": 2 }, { "id": 3042548, "postDate": "2024-11-11T15:36:08.797Z", "content": "Great job! Running 2020 solution in background - excellent way to manage resources. I had no idea this could be done. Btw, why you didn't used ConceptARC dataset?
", "rawMarkdown": "Great job! Running 2020 solution in background - excellent way to manage resources. I had no idea this could be done. Btw, why you didn't used ConceptARC dataset?", "votes": 2, "replies": [ { "id": 3042689, "postDate": "2024-11-11T17:14:35.737Z", "content": "If I remember well, I tried it but didn't give any significative improvement.
", "rawMarkdown": "If I remember well, I tried it but didn't give any significative improvement.", "votes": 1, "replies": [ { "id": 3042716, "postDate": "2024-11-11T17:40:18.250Z", "content": "", "rawMarkdown": "Source: https://ironbar.github.io/arc24/modeling/Iteration_17_external_data/#what-is-the-best-data-for-training", "votes": 2 }, { "id": 3042721, "postDate": "2024-11-11T17:47:38.763Z", "content": "Wow! I missed that you kept a diary for each iteration! Thanks
", "rawMarkdown": "Wow! I missed that you kept a diary for each iteration! Thanks", "votes": 2 }, { "id": 3042906, "postDate": "2024-11-11T22:03:52.210Z", "content": "Congratulations on making it to the money final! Your focus, your success, and your impeccable attention to detail are a joyful inspiration!
", "rawMarkdown": "Congratulations on making it to the money final! Your focus, your success, and your impeccable attention to detail are a joyful inspiration!", "votes": 1, "isDeleted": true } ] } ] }, { "id": 3240165, "postDate": "2025-07-03T14:20:11.237Z", "content": "I wonder why it‘s called Omni-ARC?
", "rawMarkdown": "I wonder why it‘s called Omni-ARC?" }, { "id": 3206069, "postDate": "2025-05-20T19:27:55.110Z", "content": "Congratulations! merci
", "rawMarkdown": "Congratulations! merci" }, { "id": 3175696, "postDate": "2025-04-10T13:42:20.687Z", "content": "congratulations, neat explanations.
", "rawMarkdown": "congratulations, neat explanations." }, { "id": 3042679, "postDate": "2024-11-11T17:03:31.517Z", "content": "Congrats on your achievement! And thanks for sharing all the details. I'm just curious as a GPU poor, what could be a way to access to no-cost computes if it was. I'm so desperate if there's any way. Appreciate if you could share.
", "rawMarkdown": "Congrats on your achievement! And thanks for sharing all the details. I'm just curious as a GPU poor, what could be a way to access to no-cost computes if it was. I'm so desperate if there's any way. Appreciate if you could share.", "replies": [ { "id": 3042692, "postDate": "2024-11-11T17:19:07.560Z", "content": "Some companies/organizations such as StrongCompute provide free compute if your project is interesting. I would search for compute grants, publish the work online, and maybe you are lucky to receive the free compute.
", "rawMarkdown": "Some companies/organizations such as [StrongCompute](https://strongcompute.com/research-grants) provide free compute if your project is interesting. I would search for compute grants, publish the work online, and maybe you are lucky to receive the free compute.", "votes": 2 } ] }, { "id": 3049674, "postDate": "2024-11-19T12:05:56.100Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3047458, "postDate": "2024-11-16T17:42:42.960Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 3044496, "postDate": "2024-11-13T13:17:32.683Z", "content": "Great work!
", "rawMarkdown": "Great work!", "votes": 1 }, { "id": 3046439, "postDate": "2024-11-15T13:30:43.603Z", "rawMarkdown": "", "isDeleted": true } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T22:21:40Z", "competition_id": 67357, "competition_slug": "arc-prize-2024", "team_id": 12243942, "solution_url": "/competitions/arc-prize-2024/writeups/guillermo-barbadillo-2nd-place-solution-for-the-ar" }