Please see the full write-up with ablations: https://github.com/jcole75/arc_2025_mindsai/blob/main/MindsAI_Tufa_Labs_2025_Solution.pdf
\nCompetition: ARC Prize 2025
\nTeam Name: MindsAI & Tufa Labs
\nPrivate Leaderboard Score: 15.42%
\nPrivate Leaderboard Place: 3rd
Team Members:
\nOur 3rd-place solution is built around a pruned 660M-parameter encoder–decoder model derived from Salesforce CodeT5-Large, trained for many months on >100 million reasoning examples (~70M of them ARC-style tasks). Almost all submitted performance comes from heavy test-time adaptation: Test-Time Training (TTT) with permutation-based labeling (~45k steps) and Augment–Inference–Reverse–Vote (AIRV) using 10k augmented inferences per task — techniques introduced by our team in 2023 and now often used across the leaderboard. These two methods combine almost perfectly additively, delivering 8–12× gains over zero-shot in ablations. Additional improvements come from new mixup/combine augmentations, reversal augmentations, tokenizer BPE dropout, T5 span-corruption, and ensembling two checkpoints. Full code, models, and the complete 100M+ example training corpus are publicly released.
\nNo traditional features — performance is driven entirely by data augmentations (the ARC equivalent of feature engineering):
\n| Rank | \nAugmentation | \nImpact | \n
|---|---|---|
| 1 | \nGeometric (rotations/flips) + color permutations | \nBaseline essential | \n
| 2 | \nNew 2025: Mixup, Combine, Combine-mixup | \n+6.3% top-2 on ARC 1.5 | \n
| 3 | \nInput/output swap (30% of training) | \nSmall score improvement | \n
| 4 | \nPrompt/answer reversals (training only) | \nModel flexibility | \n
| 5 | \nBPE tokenizer dropout (TTT & inference) | \nSmall score improvement | \n
Ensembling: combining of predictions from two strong checkpoints of the same architecture.
\nA dramatically simplified yet still strong version (90–95% of the full relative gain -- linked in resources):
\nThis runs in minutes to a few hours on a single consumer GPU and is ideal for research iteration.
\n| Component | \nHardware | \nApproximate Time | \n
|---|---|---|
| Full 660M training | \nGoogle TPUs | \nMany months to 2.5 years cumulative | \n
| 77M ablation model training | \nTPU v2-8 / v3-8 + v4-64 | \n~2 years + 7 days | \n
| Final submission inference (2× ensemble) | \n4× L4 GPUs (Kaggle) | \n~11 hours | \n
| Simplified 77M single-run inference | \nSingle P100 | \n10–60 minutes | \n
Complete code, training scripts, models, and 100M+ example dataset:
\n→ https://github.com/jcole75/arc_2025_mindsai
\n→ Full Write-up with Ablations: https://github.com/jcole75/arc_2025_mindsai/blob/main/MindsAI_Tufa_Labs_2025_Solution.pdf\n→ Dataset: https://huggingface.co/datasets/mindware/arc-agi-mega
\n→ Models: https://huggingface.co/mindware
Cole & Osman (2025). Don’t throw the baby out with the bathwater: How and why deep learning for ARC. arXiv:2506.14276 (https://arxiv.org/abs/2506.14276)
Hodel (2024). RE-ARC procedural generation. arXiv:2404.07353
Everything is released under permissive licenses to help the community push toward AGI.
", "comments": [ { "id": 3367469, "author_name": "Khushi Yadav", "author_url": "", "post_date": "2025-12-08T13:35:28.120000", "content": "MindsAI & Tufa Labs
", "votes": -2, "replies": [] }, { "id": 3362921, "author_name": "", "author_url": "", "post_date": "2025-12-05T15:08:20.060000", "content": "", "votes": 3, "replies": [] } ], "comment_count": 2, "messages": [ { "id": 3335288, "postDate": "2025-11-17T22:36:10.967Z", "content": "Please see the full write-up with ablations: https://github.com/jcole75/arc_2025_mindsai/blob/main/MindsAI_Tufa_Labs_2025_Solution.pdf
\nCompetition: ARC Prize 2025
\nTeam Name: MindsAI & Tufa Labs
\nPrivate Leaderboard Score: 15.42%
\nPrivate Leaderboard Place: 3rd
Team Members:
\nOur 3rd-place solution is built around a pruned 660M-parameter encoder–decoder model derived from Salesforce CodeT5-Large, trained for many months on >100 million reasoning examples (~70M of them ARC-style tasks). Almost all submitted performance comes from heavy test-time adaptation: Test-Time Training (TTT) with permutation-based labeling (~45k steps) and Augment–Inference–Reverse–Vote (AIRV) using 10k augmented inferences per task — techniques introduced by our team in 2023 and now often used across the leaderboard. These two methods combine almost perfectly additively, delivering 8–12× gains over zero-shot in ablations. Additional improvements come from new mixup/combine augmentations, reversal augmentations, tokenizer BPE dropout, T5 span-corruption, and ensembling two checkpoints. Full code, models, and the complete 100M+ example training corpus are publicly released.
\nNo traditional features — performance is driven entirely by data augmentations (the ARC equivalent of feature engineering):
\n| Rank | \nAugmentation | \nImpact | \n
|---|---|---|
| 1 | \nGeometric (rotations/flips) + color permutations | \nBaseline essential | \n
| 2 | \nNew 2025: Mixup, Combine, Combine-mixup | \n+6.3% top-2 on ARC 1.5 | \n
| 3 | \nInput/output swap (30% of training) | \nSmall score improvement | \n
| 4 | \nPrompt/answer reversals (training only) | \nModel flexibility | \n
| 5 | \nBPE tokenizer dropout (TTT & inference) | \nSmall score improvement | \n
Ensembling: combining of predictions from two strong checkpoints of the same architecture.
\nA dramatically simplified yet still strong version (90–95% of the full relative gain -- linked in resources):
\nThis runs in minutes to a few hours on a single consumer GPU and is ideal for research iteration.
\n| Component | \nHardware | \nApproximate Time | \n
|---|---|---|
| Full 660M training | \nGoogle TPUs | \nMany months to 2.5 years cumulative | \n
| 77M ablation model training | \nTPU v2-8 / v3-8 + v4-64 | \n~2 years + 7 days | \n
| Final submission inference (2× ensemble) | \n4× L4 GPUs (Kaggle) | \n~11 hours | \n
| Simplified 77M single-run inference | \nSingle P100 | \n10–60 minutes | \n
Complete code, training scripts, models, and 100M+ example dataset:
\n→ https://github.com/jcole75/arc_2025_mindsai
\n→ Full Write-up with Ablations: https://github.com/jcole75/arc_2025_mindsai/blob/main/MindsAI_Tufa_Labs_2025_Solution.pdf\n→ Dataset: https://huggingface.co/datasets/mindware/arc-agi-mega
\n→ Models: https://huggingface.co/mindware
Cole & Osman (2025). Don’t throw the baby out with the bathwater: How and why deep learning for ARC. arXiv:2506.14276 (https://arxiv.org/abs/2506.14276)
Hodel (2024). RE-ARC procedural generation. arXiv:2404.07353
Everything is released under permissive licenses to help the community push toward AGI.
", "rawMarkdown": "# MindsAI & Tufa Labs – ARC Prize 2025 (3rd Place)\n\nPlease see the full write-up with ablations: https://github.com/jcole75/arc_2025_mindsai/blob/main/MindsAI_Tufa_Labs_2025_Solution.pdf\n\n## A1. Background on You / Your Team\n**Competition:** ARC Prize 2025 \n**Team Name:** MindsAI & Tufa Labs \n**Private Leaderboard Score:** 15.42% \n**Private Leaderboard Place:** 3rd\n\n**Team Members:** \n- Jack Cole – Olney, Illinois, USA – jackcole@mindware.mobi \n- Dries Smit – Somerset West, South Africa – dries.epos@gmail.com \n- Isaiah Pressman – Cleveland Heights, Ohio, USA – isaiahpressman16@gmail.com \n- Mohamed Osman – Calgary, Alberta, Canada – mothman198@outlook.com \n- Michael Hodel – Switzerland – hodelmichi@gmail.com \n\n---\n\n## A2. Background on You / Your Team (Full Bios)\n\n### Jack Cole (team lead & primary contributor)\n- **Academic/Professional Background:** PhD in Clinical Psychology; part-time private practice psychologist; parallel career as an AI researcher and mobile app developer (author of Mind Games – 30M+ downloads). \n- **Prior Experience:** Led the 1st-place team in ARCathon 2023; held state-of-the-art scores on ARC-AGI-1 through 2024; pioneered Test-Time Training (TTT) and AIRV techniques that are now used by nearly all top ARC solutions. \n- **Why ARC Prize:** ARC has been the central focus of my research program since mid-2022. \n- **Time Spent:** Approximately 3.5 years of near full-time work on ARC (minus ~6 months off at the end of 2024 / early 2025); hundreds of experiments and training runs lasting up to 2.5 years on TPUs. \n- **2025 Role:** Designed, trained, and submitted the solution; code, datasets, ablations, and the final submission were produced by me. \n\n### Dries Smit\n- **Academic/Professional Background:** PhD in Electrical & Electronic Engineering; specialist in reinforcement learning, multi-agent systems, distributed training, and large language models. \n- **Prior Experience:** Led the winning solution in the ARC-AGI-3 Preview competition; developed Laila (biology-assistant fine-tune of Llama 3.1); extensive work scaling RL and reasoning systems. \n- **Why ARC Prize:** ARC is a core reasoning benchmark that aligns perfectly with research on adaptive, test-time reasoning systems. \n- **Time Spent:** Part-time contributions through part of the season. \n- **2025 Role:** Provided refinement-based training experiments and strategic insights; some of those ideas were explored but ultimately not additive with the TTT/AIRV core. \n\n### Isaiah Pressman\n- **Academic/Professional Background:** AI researcher/engineer since 2019; previous work in computer vision for histopathology, data/ML pipelines for startups, four top-2 Kaggle finishes. \n- **Prior Experience:** Deep experience improving LLM training and reasoning capabilities at Tufa Labs. \n- **Why ARC Prize:** Fascination with pure reasoning challenges that are hard for current foundation models. \n- **Time Spent:** ~20 hours/week for one month, then full-time for another month toward the end of the competition. \n- **2025 Role:** Explored a promising diffusion-based language modeling approach for ARC (not included in the final submission but deepened architectural understanding). \n\n### Mohamed Osman\n- **Academic/Professional Background:** Master’s in Electrical Engineering; 5+ years as an ML practitioner and researcher. \n- **Prior Experience:** Co-developed early versions of TTT for ARC in previous seasons; strong ML engineering background. \n- **Why ARC Prize:** Strong agreement with the ARC Prize definition of intelligence and interest in the core problem. \n- **Time Spent:** Very limited in 2025 due to external commitments. \n- **2025 Role:** Minimal direct involvement this year. \n\n### Michael Hodel\n- **Academic/Professional Background:** AI researcher; creator of RE-ARC and ARC-DSL.\n- **Prior Experience:** Machine learning experience, Prior work on ARC. \n- **Why ARC Prize:** Long-standing passion for the ARC challenge and building tools/datasets for the community. \n- **Time Spent:** Limited in 2025. \n- **2025 Role:** Provided foundational datasets (ARC 1.5, RE-ARC variants) that were used for ablations and portions of training data. \n\n---\n\n## A3. Summary\n\nOur 3rd-place solution is built around a pruned 660M-parameter encoder–decoder model derived from Salesforce CodeT5-Large, trained for many months on >100 million reasoning examples (~70M of them ARC-style tasks). Almost all submitted performance comes from heavy test-time adaptation: Test-Time Training (TTT) with permutation-based labeling (~45k steps) and Augment–Inference–Reverse–Vote (AIRV) using 10k augmented inferences per task — techniques introduced by our team in 2023 and now often used across the leaderboard. These two methods combine almost perfectly additively, delivering 8–12× gains over zero-shot in ablations. Additional improvements come from new mixup/combine augmentations, reversal augmentations, tokenizer BPE dropout, T5 span-corruption, and ensembling two checkpoints. Full code, models, and the complete 100M+ example training corpus are publicly released.\n\n---\n\n## A4. Feature Selection / Engineering\n\nNo traditional features — performance is driven entirely by data augmentations (the ARC equivalent of feature engineering):\n\n| Rank | Augmentation | Impact |\n|------|--------------------------------------------------|--------|\n| 1 | Geometric (rotations/flips) + color permutations| Baseline essential |\n| 2 | New 2025: Mixup, Combine, Combine-mixup | +6.3% top-2 on ARC 1.5 |\n| 3 | Input/output swap (30% of training) | Small score improvement |\n| 4 | Prompt/answer reversals (training only) | Model flexibility |\n| 5 | BPE tokenizer dropout (TTT & inference) | Small score improvement |\n\n---\n\n## A5. Training Method(s)\n\n- 660M CodeT5-Large → encoder kept at 24 layers, decoder pruned to 16 layers \n- Supervised training on >100M examples (Google TPUs, some runs >2 years cumulative) \n- T5 span-corruption + reversal augmentations + BPE dropout \n- Test-time: TTT (~45k examples) + AIRV (10k augmentations) + self-ensembling two checkpoints \n\nEnsembling: combining of predictions from two strong checkpoints of the same architecture.\n\n---\n\n## A6. Interesting Findings\n\n- TTT + AIRV gains are nearly perfectly additive (~812% combined vs. ~410–430% each alone). \n- Self-ensembling with different seeds beats using 2× more TTT/AIRV samples (compute-matched +6.2%). \n- Encoder depth >> decoder depth (removing encoder layers hurts badly; decoder can be heavily pruned). \n- Augmentation-driven data expansion can break multi-month training plateaus. \n- Several otherwise strong methods (refinement, DPO on beam pairs, targeted ARC-2 data) were non-additive with TTT+AIRV. \n- ARC-AGI-2 appears partially adversarial to the current TTT/AIRV paradigm (at least with models this small) — new ideas are needed.\n\n---\n\n## A7. Simple Features and Methods\n\nA dramatically simplified yet still strong version (90–95% of the full relative gain -- linked in resources):\n\n- Only classic geometric + color permutation augmentations \n- TTT with 20–30k examples + AIRV with 5–10k samples (single run, no ensembling, no BPE dropout) \n\nThis runs in minutes to a few hours on a single consumer GPU and is ideal for research iteration.\n\n---\n\n## A8. Model Execution Time\n\n| Component | Hardware | Approximate Time |\n|---------------------------------------|-------------------------|-----------------------------------|\n| Full 660M training | Google TPUs | Many months to 2.5 years cumulative |\n| 77M ablation model training | TPU v2-8 / v3-8 + v4-64 | ~2 years + 7 days |\n| Final submission inference (2× ensemble) | 4× L4 GPUs (Kaggle) | ~11 hours |\n| Simplified 77M single-run inference | Single P100 | 10–60 minutes |\n\n---\n\n## A9. References & Public Resources\n\n- Complete code, training scripts, models, and 100M+ example dataset: \n → **https://github.com/jcole75/arc_2025_mindsai** \n → Full Write-up with Ablations: https://github.com/jcole75/arc_2025_mindsai/blob/main/MindsAI_Tufa_Labs_2025_Solution.pdf\n → Dataset: https://huggingface.co/datasets/mindware/arc-agi-mega \n → Models: https://huggingface.co/mindware \n\n- Cole & Osman (2025). Don’t throw the baby out with the bathwater: How and why deep learning for ARC. arXiv:2506.14276 (https://arxiv.org/abs/2506.14276)\n- Hodel (2024). RE-ARC procedural generation. arXiv:2404.07353 \n\nEverything is released under permissive licenses to help the community push toward AGI.", "votes": 16 }, { "id": 3367469, "postDate": "2025-12-08T13:35:28.120Z", "content": "MindsAI & Tufa Labs
", "rawMarkdown": "MindsAI & Tufa Labs ", "votes": -2 }, { "id": 3362921, "postDate": "2025-12-05T15:08:20.060Z", "rawMarkdown": "", "votes": 3, "isDeleted": true } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T21:00:23Z", "competition_id": 91496, "competition_slug": "arc-prize-2025", "team_id": 13573137, "solution_url": "/competitions/arc-prize-2025/writeups/mindsai-and-tufa-labs-arc-prize-2025-solution" }