{
  "id": 617939,
  "title": "ARC Prize 2025 Competition Writeup: 5th Place",
  "content": "<h1>Overview</h1>\n<p>This submission forks the notebook ~<a href=\"https://www.kaggle.com/code/sanyul/lb-5-83-baseline-from-1st-place-of-2024\" target=\"_blank\">LB 5.83 baseline from 1st place of 2024</a>~ by <a href=\"https://www.kaggle.com/sanyul\" target=\"_blank\">@sanyul</a> and the ~<a href=\"https://www.kaggle.com/code/boristown/baseline-from-1st-place-of-2024\" target=\"_blank\">baseline from 1st place of 2024</a>~ by <a href=\"https://www.kaggle.com/boristown\" target=\"_blank\">@boristown</a>, which originates from the first-place solution from ARC Prize 2024 by the ARChitects team (Daniel Franzen and Jan Disselhoff). Full credits to all of them for their exceptional work.\nMy submission notebook can be found here: <a href=\"https://www.kaggle.com/code/lonnieqin/lb-5-83-baseline-from-1st-place-of-2024\" target=\"_blank\">https://www.kaggle.com/code/lonnieqin/lb-5-83-baseline-from-1st-place-of-2024</a></p>\n<h1>Performance</h1>\n<p><strong>Public Leaderboard</strong>: 344th place</p>\n<p><strong>Private Leaderboard</strong>: 5th place</p>\n<p>The dramatic improvement from public to private leaderboard rankings highlights the importance of model generalization and the critical role of hyperparameter optimization in competitive machine learning, especially when working with small evaluation sets.</p>\n<h1>Structure of the Solution</h1>\n<p>The solution consists of several key components organized as follows:</p>\n<h2>1. Core Infrastructure Files:</h2>\n<ul>\n<li>model_runner.py: Handles model loading, training, inference, and vocabulary shrinking</li>\n<li>selection.py: Implements various algorithms for selecting the best predictions</li>\n<li>async_tools.py: Manages parallel execution across multiple GPUs</li>\n<li>common_stuff.py: Central configuration hub containing all hyperparameters and orchestration logic</li>\n</ul>\n<h2>2. Training Pipeline:</h2>\n<ul>\n<li>Multi-GPU distributed training (4 GPUs) with different data splits</li>\n<li>LoRA fine-tuning on a quantized base model (Nemo Mini)</li>\n<li>Data augmentation (transposition, rotation, color permutation)</li>\n<li>Task-specific priming for improved predictions</li>\n</ul>\n<h2>3. Inference Pipeline:</h2>\n<ul>\n<li>Turbo DFS (Depth-First Search) beam search for efficient decoding</li>\n<li>Augmented scoring mechanism for prediction selection</li>\n<li>Parallel inference across multiple GPUs</li>\n<li>Result caching and incremental processing</li>\n</ul>\n<h2>4. Execution Flow:</h2>\n<ul>\n<li>Parallel training on 4 GPUs → Wait for completion → Parallel inference on 4 GPUs → Aggregate results → Generate submission</li>\n</ul>\n<h1>Methodology</h1>\n<p>The solution employs a <strong>fine-tuned language model approach</strong> to the ARC-AGI challenge:</p>\n<h2>Base Architecture</h2>\n<ul>\n<li><strong>Model</strong>: Nemo Mini (fine-tuned on the ARC 2024 dataset – the best solution of ARC 2024)</li>\n<li><strong>Fine-tuning</strong>: LoRA (Low-Rank Adaptation) with rank 32</li>\n<li><strong>Active layers</strong>: Only the first 32 layers trainable to reduce overfitting</li>\n</ul>\n<h2>Data Representation</h2>\n<ul>\n<li>Tasks are converted to text format using a specialized formatter (ArcFormatter_premix_3)</li>\n<li>Input/output grids are serialized as token sequences</li>\n<li>Maximum sequence length: 4,224 tokens (training), 8,192 tokens (inference)</li>\n</ul>\n<h2>Training Strategy</h2>\n<ul>\n<li><strong>Augmentation</strong>: Transposition, rotation, random color permutation, example shuffling</li>\n<li><strong>Epochs</strong>: 4 epochs with 240 training steps</li>\n<li><strong>Multi-GPU</strong>: Data distributed across 4 GPUs (each processes ~1/4 of tasks)</li>\n<li><strong>Learning rates</strong>: 1e-4 (main), 1e-5 (embeddings)</li>\n</ul>\n<h2>Inference Strategy</h2>\n<ul>\n<li><strong>Task-specific priming</strong>: Model is briefly retrained on each test task's examples before prediction</li>\n<li><strong>Turbo DFS beam search</strong>: Efficiently explores multiple prediction candidates</li>\n<li><strong>Augmented scoring</strong>: Predictions evaluated across multiple augmentations to select the most robust solution</li>\n<li><strong>Probability tracking</strong>: Maintains cumulative probabilities to identify best candidates</li>\n</ul>\n<h1>My Contribution: Treating Random Seed as a Hyperparameter</h1>\n<p>While this solution builds upon the strong 2024 first-place baseline, I made one key modification that proved surprisingly impactful:\n<strong>Changed the random seed to 19920627</strong> , selected based on empirical exploration of seed values.</p>\n<h2>Random Seed as a Legitimate Hyperparameter</h2>\n<p>In scenarios with <strong>small sample sizes</strong> (like this competition's 240 evaluation tasks), random seed should be treated as a <strong>first-class hyperparameter</strong> rather than an afterthought. This is because:</p>\n<ul>\n<li><p><strong>High variance in small samples</strong>: With only 120 tasks per leaderboard split, solving just 1-2 additional tasks can dramatically shift rankings</p></li>\n<li><p><strong>Stochastic model behavior</strong>: Random seeds influence multiple pipeline stages, creating meaningfully different model behaviors</p></li>\n<li><p><strong>Task-specific interactions</strong>: Different seeds may produce augmentation patterns or training dynamics better suited to specific task types</p></li>\n</ul>\n<p>This seed change creates cascading effects throughout the entire machine learning pipeline:</p>\n<ul>\n<li><strong>Data shuffling</strong>: Alters the order in which training examples are presented</li>\n<li><strong>Augmentation patterns</strong>: Modifies rotations, transpositions, and color permutations applied to training data</li>\n<li><strong>Model training</strong>: Influences weight initialization in LoRA layers and training dynamics</li>\n<li><strong>Inference behavior</strong>: Changes stochastic elements in beam search decoding</li>\n<li><strong>Convergence paths</strong>: Leads to different optimization trajectories and final model states</li>\n</ul>\n<h1>The Hyperparameter Landscape: Beyond Random Seeds</h1>\n<p>While random seed optimization proved effective in this competition, it's important to recognize that this solution space contains <strong>numerous other hyperparameters</strong> worthy of systematic exploration:</p>\n<h2>1. Model Selection and Architecture Hyperparameters</h2>\n<p><strong>Base Model Choice</strong> - One of the most impactful decisions:</p>\n<ul>\n<li><strong>Nemo Mini</strong> (current choice): Efficient, fast inference, good balance</li>\n<li><strong>Llama 3.1 variants</strong>: 8B, 70B models with different reasoning capabilities</li>\n<li><strong>Qwen 2.5</strong>: Strong performance on reasoning tasks</li>\n<li><strong>Gemma 2</strong>: Google's open models with different architectural choices</li>\n<li><strong>Phi-3/Phi-4</strong>: Microsoft's small language models optimized for reasoning</li>\n<li><strong>Mistral variants</strong>: 7B, Mixtral MoE models with different trade-offs</li>\n<li><strong>DeepSeek models</strong>: Strong coding and reasoning abilities</li>\n</ul>\n<p>⠀Different base models have fundamentally different:</p>\n<ul>\n<li>Vocabulary and tokenization strategies (affecting grid representation)</li>\n<li>Architectural inductive biases (attention patterns, layer depths)</li>\n<li>Pre-training distributions (some models see more code/structured data)</li>\n<li>Context window capabilities (affecting maximum task size handling)</li>\n<li>Inference speed and memory requirements (computational constraints)</li>\n</ul>\n<p>⠀<strong>Architecture Configuration</strong>:</p>\n<ul>\n<li><strong>LoRA rank</strong>: Currently set to 32, but values from 8 to 128+ could be explored</li>\n<li><strong>LoRA alpha</strong>: Scaling factor for LoRA updates</li>\n<li><strong>LoRA target modules</strong>: Which layers to apply LoRA (attention, MLP, both)</li>\n<li><strong>Trainable layers</strong>: Currently first 32 layers; different layer selections might capture different abstractions</li>\n<li><strong>Quantization level</strong>: 4-bit vs 8-bit vs full precision trade-offs</li>\n<li><strong>Model size within family</strong>: Different parameter counts capture different complexity levels</li>\n</ul>\n<h2>2. Training Hyperparameters</h2>\n<ul>\n<li><strong>Learning rates</strong>: Main (1e-4) and embedding (1e-5) rates could be tuned independently</li>\n<li><strong>Number of epochs</strong>: Currently 4; more epochs might improve convergence or cause overfitting</li>\n<li><strong>Training steps</strong>: 240 steps per epoch is another tunable parameter</li>\n<li><strong>Batch size</strong>: Affects training stability and generalization</li>\n<li><strong>Gradient accumulation steps</strong>: For effective larger batch sizes</li>\n<li><strong>Optimizer choice</strong>: Adam, AdamW, SGD with different momentum values</li>\n<li><strong>Learning rate schedule</strong>: Constant vs cosine decay vs linear warmup strategies</li>\n<li><strong>Weight decay</strong>: Regularization strength</li>\n<li><strong>Dropout rates</strong>: If applicable to the fine-tuning strategy</li>\n</ul>\n<h2>3. Data Augmentation Hyperparameters</h2>\n<ul>\n<li><strong>Augmentation probability</strong>: How frequently each augmentation is applied</li>\n<li><strong>Augmentation types</strong>: Which transformations to include/exclude (transposition, rotation, color permutation)</li>\n<li><strong>Example shuffling strategy</strong>: Different orderings of demonstration examples</li>\n<li><strong>Color mapping schemes</strong>: Alternative color permutation strategies</li>\n<li><strong>Rotation angles</strong>: 90°, 180°, 270° vs other transformations</li>\n<li><strong>Augmentation composition</strong>: Sequential vs parallel application</li>\n</ul>\n<h2>4. Inference Hyperparameters</h2>\n<ul>\n<li><strong>Beam search width</strong>: Number of candidates to explore simultaneously</li>\n<li><strong>DFS depth limit</strong>: How deep to search in the solution space</li>\n<li><strong>Temperature</strong>: Sampling temperature for generation (typically 0.1 to 1.5)</li>\n<li><strong>Top-k/top-p values</strong>: Nucleus sampling parameters</li>\n<li><strong>Repetition penalty</strong>: To discourage repetitive patterns</li>\n<li><strong>Priming epochs</strong>: How long to fine-tune on test task examples</li>\n<li><strong>Number of augmentations</strong>: How many augmented versions to score</li>\n<li><strong>Maximum generation length</strong>: Token limits for output</li>\n</ul>\n<h2>5. Selection Algorithm Hyperparameters</h2>\n<ul>\n<li><strong>Scoring weights</strong>: How to balance different scoring criteria</li>\n<li><strong>Ensemble strategies</strong>: How to combine predictions from multiple runs</li>\n<li><strong>Confidence thresholds</strong>: When to trust a prediction vs explore alternatives</li>\n<li><strong>Voting mechanisms</strong>: Majority vote vs weighted vote vs consensus strategies</li>\n</ul>\n<h2>The Small Sample Size Advantage</h2>\n<p>In competitions with small evaluation sets (100-500 samples), these hyperparameters become <strong>especially influential</strong> because:</p>\n<ul>\n<li><strong>Lower statistical power</strong>: Individual hyperparameter choices create measurable differences in final scores</li>\n<li><strong>Higher variance sensitivity</strong>: Small improvements (1-3 additional correct predictions) translate to significant rank changes</li>\n<li><strong>Exploration feasibility</strong>: With limited test data, systematic grid search or random search over hyperparameters becomes computationally tractable</li>\n<li><strong>Overfitting opportunities</strong>: Hyperparameters can be tuned to exploit specific characteristics of the small evaluation distribution</li>\n</ul>\n<p>This doesn't diminish the achievement—rather, it highlights that <strong>comprehensive hyperparameter optimization is a valuable and legitimate competition strategy</strong>, especially when the evaluation set is constrained.</p>\n<h1>Motivation: The Statistics Behind Random Seed Optimization</h1>\n<p>My approach was motivated by understanding the <strong>statistical landscape of the competition</strong>:</p>\n<h3>Competition Structure</h3>\n<ul>\n<li><strong>Public LB</strong>: 120 samples (scores reflected on public leaderboard during competition; samples not visible to participants)</li>\n<li><strong>Private LB</strong>: 120 samples (used to compute final private scores; revealed after December 5, 2025)</li>\n</ul>\n<h2>The Random Seed Strategy</h2>\n<p>Given that:</p>\n<ul>\n<li>Most competitive solutions were variations of the same baseline notebook</li>\n<li>The same model can produce different scores across runs due to stochasticity</li>\n<li>There are only 120 samples in both public LB and private LB</li>\n</ul>\n<p><strong>This creates an opportunity</strong>: systematically exploring hyperparameters (including random seeds) can yield significantly different rankings even without fundamental algorithmic improvements.</p>\n<h3>Score Variance Analysis</h3>\n<p><strong>My observed performance across different random seeds</strong>:</p>\n<ul>\n<li>Best public LB score: <strong>4.17%</strong> (5/120 tasks)</li>\n<li>Estimated private score: <strong>~10%</strong> (12/120 tasks)</li>\n<li><strong>This notebook's score range: 3.33 % to 6.67 %(with the upper bound potentially higher)</strong> across different random seed configurations</li>\n<li>Score variance: ±3.34 percentage points (equivalent to ±4 tasks out of 120)</li>\n<li><strong>Pixel-Level Sensitivity Amplification</strong>: The all-or-nothing scoring (one wrong pixel = full failure) magnifies this variance; hyperparameter tweaks that nudge predictions toward exact matches on borderline tasks can swing 1-2 full solves, equivalent to 1-2% score jumps.</li>\n</ul>\n<p>This <strong>2x variation in performance</strong> (from 3.33 % to 6.67 %) demonstrates the substantial impact random seed can have as a hyperparameter in small-sample scenarios.</p>\n<h2>Public LB landscape:</h2>\n<ul>\n<li>8 teams achieved &gt;10% score (&gt;12 tasks solved)</li>\n<li>Significant variance exists even among top solutions</li>\n<li>Top solutions likely benefited from both strong architectures AND favorable hyperparameter configurations</li>\n</ul>\n<h2>Probability Calculation</h2>\n<p><strong>Question</strong>: Under the simplifying assumption of a model with 4% base accuracy on independent tasks (which isn't the actual case, as task accuracies vary and dependencies exist), what's the probability of achieving a 10% score (12/120 tasks)?</p>\n<p>Assuming each task is independent with success probability p = 0.04:</p>\n<ul>\n<li>P(X ≥ 12) where X ~ Binomial(n=120, p=0.04)</li>\n<li>Expected value: E[X] = 120 × 0.04 = 4.8 tasks</li>\n<li>Standard deviation: σ = √(120 × 0.04 × 0.96) ≈ 2.15</li>\n<li>P(X ≥ 12) ≈ P(Z ≥ (12 - 4.8) / 2.15) ≈ P(Z ≥ 3.35) ≈ 0.04 %</li>\n</ul>\n<p>This is extremely unlikely (~1 in 2,500 chance) if purely random!</p>\n<h3>The Real Insight</h3>\n<p>However, the actual mechanism is more nuanced:</p>\n<ul>\n<li><strong>Model capability varies by task type</strong>: The model may have 10-15% accuracy on certain task patterns and 0-2% on others</li>\n<li><strong>Hyperparameters affect task-specific performance</strong>: Different hyperparameter configurations (including seeds, base models, and training strategies) produce different augmentation patterns and model behaviors, leading to varying performance on specific tasks</li>\n<li><strong>Small sample size amplification</strong>: With only 120 tasks per split, small improvements (solving 1-2 more tasks) create large ranking jumps</li>\n<li><strong>Model selection matters significantly</strong>: Different base models have varying strengths across task categories—some excel at spatial reasoning, others at pattern completion or color transformations\n<strong>Strategic implication</strong>:</li>\n<li>While LLMs struggle with ARC-AGI reasoning (fundamental limitation), Achieving competitive results is possible through:<ul>\n<li>Starting from strong baselines</li>\n<li>Systematic hyperparameter exploration (seeds, learning rates, augmentation strategies, etc.)</li>\n<li><strong>Exploring different base model architectures</strong> (Llama, Qwen, Gemma, Phi, etc.)</li>\n<li>Understanding which task types benefit from different configurations</li>\n<li>Leveraging the competition's small evaluation set for targeted optimization</li></ul></li>\n</ul>\n<h3>Competitive Reality</h3>\n<p>This is <strong>not about gaming the system</strong>—it's about:</p>\n<ul>\n<li>Recognizing that <strong>all stochastic elements are legitimate hyperparameters</strong> in small-sample scenarios</li>\n<li>Understanding that competition rankings reflect both model quality AND optimal hyperparameter selection</li>\n<li>Acknowledging that with 240 evaluation samples, hyperparameter optimization matters significantly</li>\n<li>Appreciating that hyperparameter tuning is a <strong>core machine learning skill</strong>, not a shortcut</li>\n<li><strong>Recognizing that base model selection is itself a critical hyperparameter choice</strong></li>\n</ul>\n<p>My 5th place finish validates that while <strong>solving ARC-AGI fundamentally remains extremely difficult</strong>, competitive performance in this format is achievable through careful exploration of the hyperparameter space, including often-overlooked parameters like random seeds and the foundational choice of base model architecture.</p>\n<h1>What Worked</h1>\n<ul>\n<li><strong>Strong baseline architecture</strong>: The ARChitects' original design proved highly effective as a foundation</li>\n<li><strong>Multi-GPU distribution</strong>: Enabled processing all tasks within the competition's tight time constraints</li>\n<li><strong>Task-specific priming</strong>: Improved predictions by conditioning on individual challenge characteristics</li>\n<li><strong>Augmented scoring</strong>: Enhanced prediction selection through more robust evaluation metrics</li>\n<li><strong>Hyperparameter optimization</strong>: Treating random seed as a tunable hyperparameter achieved competitive results, with score variance of 3.33 % to 6.67 % demonstrating significant impact</li>\n<li><strong>Nemo Mini as base model</strong>: Balanced efficiency and performance well for this specific task distribution</li>\n</ul>\n<h1>What Could Be Improved</h1>\n<ul>\n<li><strong>Systematic hyperparameter search</strong>: A principled grid search or Bayesian optimization over multiple hyperparameters (not just random seed) could identify even better configurations</li>\n<li><strong>Base model exploration</strong>: Testing alternative foundation models (Llama 3.2, Qwen 3, Gemma 3, Phi-3/4, Mistral, DeepSeek) could reveal architectures better suited to ARC-AGI reasoning patterns</li>\n<li><strong>Model ensemble methods</strong>: Combining predictions from multiple base models with different architectural biases could improve robustness and coverage across diverse task types</li>\n<li><strong>Seed ensemble strategies</strong>: Rather than selecting a single optimal seed, averaging or voting across multiple seed configurations could reduce variance</li>\n<li><strong>Public-private gap analysis</strong>: Deeper investigation into the score discrepancy would provide valuable insights for future competitions</li>\n<li><strong>Selection algorithms</strong>: More sophisticated prediction selection methods could improve consistency</li>\n<li><strong>Architectural enhancements</strong>: Exploring modifications beyond LoRA fine-tuning (full fine-tuning, adapter layers, prefix tuning) might unlock further gains</li>\n<li><strong>Edge case handling</strong>: Better strategies for tasks with unusual or rare patterns</li>\n<li><strong>Cross-validation on small samples</strong>: Developing robust validation strategies for competitions with limited evaluation data</li>\n<li><strong>Multi-stage hyperparameter optimization</strong>: Sequential optimization (model selection → training hyperparameters → inference hyperparameters) could be more efficient than random search</li>\n<li><strong>Pixel-Precision Robustness</strong>: Develop tolerance-aware scoring proxies during hyperparameter search to better handle the one-pixel-failure risk, e.g., via soft-match metrics in validation to guide toward exact predictions without overfitting to noise.</li>\n<li><strong>Symbolic Function Generation</strong>: Rather than directly predicting output grids pixel-by-pixel, explore generating interpretable transformation functions (e.g., via program synthesis or rule induction) from diverse training samples. This approach could enhance generalization across unseen task variations by capturing underlying patterns more abstractly, mitigating the fragility of raw pixel outputs.</li>\n</ul>\n<h1>Conclusion</h1>\n<p>This competition experience has been both humbling and inspiring. Achieving 5th place on the private leaderboard demonstrates that in competitions with small evaluation sets, <strong>comprehensive hyperparameter optimization is not just beneficial—it's essential</strong>. The lesson extends beyond random seeds: base model selection, learning rate schedules, augmentation strategies, model architecture choices, and inference parameters all constitute a rich landscape worthy of systematic exploration.</p>\n<p>The observed score variance (3.33 % to 6.67 %) from random seed changes alone illustrates how impactful these seemingly minor decisions can be. When we consider the additional dimensions of model selection (Llama vs Qwen vs Gemma vs Phi vs Mistral vs DeepSeek) and the interactions between these choices and other hyperparameters, the potential solution space becomes even more expansive.</p>\n<p>The key insight is recognizing when hyperparameter variance becomes comparable to or exceeds model variance. In such scenarios, treating traditionally \"fixed\" parameters (like random seeds and base model choices) as first-class hyperparameters becomes a legitimate and powerful strategy.</p>\n<p>I want to express my sincere gratitude to <strong>Kaggle and the entire ARC Prize community</strong> for creating this challenging and thought-provoking competition. The opportunity to engage with abstract reasoning tasks and learn from world-class competitors has been invaluable. Special thanks to the ARChitects team for open-sourcing their solution, and to <a href=\"https://www.kaggle.com/sanyul\" target=\"_blank\">@sanyul</a> and <a href=\"https://www.kaggle.com/boristown\" target=\"_blank\">@boristown</a> for their excellent baseline implementations.</p>\n<p>This competition has motivated me to become more deeply involved in future ARC challenges. The problem of abstract reasoning represents one of the most fascinating frontiers in AI, and I'm excited to contribute to advancing this field. For future iterations, I plan to systematically explore the base model selection space alongside other hyperparameters to better understand which architectural choices align best with different types of abstract reasoning tasks.</p>\n<p><strong>To anyone reading this writeup</strong>: In small-sample competitions, never underestimate the importance of exploring every hyperparameter systematically, starting with the fundamental choice of base model architecture. What seems like a minor configuration choice—whether it's a random seed, a learning rate, or a model selection—may be the difference between a good solution and a winning one. The 2x performance variance I observed from seed changes alone (3.33 % to 6.67 %) suggests that comprehensive exploration of the full hyperparameter space could yield even more dramatic improvements.</p>",
  "comments": [
    {
      "id": 3368284,
      "author_name": "sanyul",
      "author_url": "",
      "post_date": "2025-12-09T05:41:57.103000",
      "content": "<p>I thought the only way to improve the notebook performance was to change model or training settings, but the idea of ​​exploring the upper limit within a single training setting in this way is quite amazing.\nCongratulations on your 4th place!🎉</p>",
      "votes": 1,
      "replies": [
        {
          "id": 3368911,
          "author_name": "Lonnie",
          "author_url": "",
          "post_date": "2025-12-09T15:47:52.920000",
          "content": "<p><a href=\"https://www.kaggle.com/sanyul\" target=\"_blank\">@sanyul</a> Thank you again, your solution is excellent, keep up the good work.</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 3378314,
      "author_name": "Bananan",
      "author_url": "",
      "post_date": "2025-12-17T22:06:24.683000",
      "content": "<p>Thanks for the detailed writeup of your angle on this competition, it's really interesting to see.</p>\n<p>I do have a couple question. I understand that you treated the random seed as a first-class hyperparameter, and selected the final value based on empirical exploration of seed values. Could you help understand:</p>\n<ul>\n<li>how you went about this exploration of values?</li>\n<li>whether you optimized for additional parameters other than seed?</li>\n</ul>\n<p>Once again, thanks for sharing. It's very cool to see someone boil down a competition like this to such factors, and achieve such a high final ranking. Congratulations on the result!</p>",
      "votes": 2,
      "replies": [
        {
          "id": 3378549,
          "author_name": "Lonnie",
          "author_url": "",
          "post_date": "2025-12-18T13:20:17.510000",
          "content": "<p>Thanks so much for the thoughtful questions and kind words! To be honest, my approach was pretty straightforward given the time constraints. I tested several different seed values and tracked their public LB scores. When I found one that gave me the best public result, I went with it for my final submission. \n I really wasn't sure how this would translate to private LB—optimizing on public scores is always risky. But I believed the change would matter, so I took the bet and submitted the best-performing version. Looking back, the better approach would've been running evaluations on the training set first to get more reliable signals about generalization. But sometimes in competitions you just have to commit and keep trying.\nFor this submission, I mainly focused on the seed while keeping the baseline largely intact. I had ideas for exploring other things—different base models, training strategies, ensembles—but with limited compute and tight deadlines, I couldn't fully develop them.\nThe jump from 344th to 4th place really validated that systematic exploration matters in small-sample competitions, even if it's just one parameter. It's definitely motivated me to be more comprehensive in future competitions—better validation strategies, more systematic searches, and hopefully more compute resources!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 3363198,
      "author_name": "parthenos",
      "author_url": "",
      "post_date": "2025-12-05T20:04:05.580000",
      "content": "<p>Congrats on the 4th place!!  🎉🎉</p>\n<p>Btw ur submitted notebook is not public yet</p>",
      "votes": 2,
      "replies": [
        {
          "id": 3363346,
          "author_name": "Lonnie",
          "author_url": "",
          "post_date": "2025-12-05T22:01:11.300000",
          "content": "<p><a href=\"https://www.kaggle.com/nihilisticneuralnet\" target=\"_blank\">@nihilisticneuralnet</a>  Thank you so much! 🎉 And oh — thanks for the heads-up about the notebook! I've just made it public. Appreciate you letting me know!</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 3320705,
      "author_name": "Bryan Landers",
      "author_url": "",
      "post_date": "2025-11-12T18:15:56.577000",
      "content": "<p>Quick note from the competition hosts: final leaderboard places have not yet been awarded. Winners and prizes will be announced December 5, 2025.</p>",
      "votes": 1,
      "replies": []
    }
  ],
  "comment_count": 7,
  "messages": [
    {
      "id": 3320678,
      "postDate": "2025-11-12T17:53:40.273Z",
      "content": "<h1>Overview</h1>\n<p>This submission forks the notebook ~<a href=\"https://www.kaggle.com/code/sanyul/lb-5-83-baseline-from-1st-place-of-2024\" target=\"_blank\">LB 5.83 baseline from 1st place of 2024</a>~ by <a href=\"https://www.kaggle.com/sanyul\" target=\"_blank\">@sanyul</a> and the ~<a href=\"https://www.kaggle.com/code/boristown/baseline-from-1st-place-of-2024\" target=\"_blank\">baseline from 1st place of 2024</a>~ by <a href=\"https://www.kaggle.com/boristown\" target=\"_blank\">@boristown</a>, which originates from the first-place solution from ARC Prize 2024 by the ARChitects team (Daniel Franzen and Jan Disselhoff). Full credits to all of them for their exceptional work.\nMy submission notebook can be found here: <a href=\"https://www.kaggle.com/code/lonnieqin/lb-5-83-baseline-from-1st-place-of-2024\" target=\"_blank\">https://www.kaggle.com/code/lonnieqin/lb-5-83-baseline-from-1st-place-of-2024</a></p>\n<h1>Performance</h1>\n<p><strong>Public Leaderboard</strong>: 344th place</p>\n<p><strong>Private Leaderboard</strong>: 5th place</p>\n<p>The dramatic improvement from public to private leaderboard rankings highlights the importance of model generalization and the critical role of hyperparameter optimization in competitive machine learning, especially when working with small evaluation sets.</p>\n<h1>Structure of the Solution</h1>\n<p>The solution consists of several key components organized as follows:</p>\n<h2>1. Core Infrastructure Files:</h2>\n<ul>\n<li>model_runner.py: Handles model loading, training, inference, and vocabulary shrinking</li>\n<li>selection.py: Implements various algorithms for selecting the best predictions</li>\n<li>async_tools.py: Manages parallel execution across multiple GPUs</li>\n<li>common_stuff.py: Central configuration hub containing all hyperparameters and orchestration logic</li>\n</ul>\n<h2>2. Training Pipeline:</h2>\n<ul>\n<li>Multi-GPU distributed training (4 GPUs) with different data splits</li>\n<li>LoRA fine-tuning on a quantized base model (Nemo Mini)</li>\n<li>Data augmentation (transposition, rotation, color permutation)</li>\n<li>Task-specific priming for improved predictions</li>\n</ul>\n<h2>3. Inference Pipeline:</h2>\n<ul>\n<li>Turbo DFS (Depth-First Search) beam search for efficient decoding</li>\n<li>Augmented scoring mechanism for prediction selection</li>\n<li>Parallel inference across multiple GPUs</li>\n<li>Result caching and incremental processing</li>\n</ul>\n<h2>4. Execution Flow:</h2>\n<ul>\n<li>Parallel training on 4 GPUs → Wait for completion → Parallel inference on 4 GPUs → Aggregate results → Generate submission</li>\n</ul>\n<h1>Methodology</h1>\n<p>The solution employs a <strong>fine-tuned language model approach</strong> to the ARC-AGI challenge:</p>\n<h2>Base Architecture</h2>\n<ul>\n<li><strong>Model</strong>: Nemo Mini (fine-tuned on the ARC 2024 dataset – the best solution of ARC 2024)</li>\n<li><strong>Fine-tuning</strong>: LoRA (Low-Rank Adaptation) with rank 32</li>\n<li><strong>Active layers</strong>: Only the first 32 layers trainable to reduce overfitting</li>\n</ul>\n<h2>Data Representation</h2>\n<ul>\n<li>Tasks are converted to text format using a specialized formatter (ArcFormatter_premix_3)</li>\n<li>Input/output grids are serialized as token sequences</li>\n<li>Maximum sequence length: 4,224 tokens (training), 8,192 tokens (inference)</li>\n</ul>\n<h2>Training Strategy</h2>\n<ul>\n<li><strong>Augmentation</strong>: Transposition, rotation, random color permutation, example shuffling</li>\n<li><strong>Epochs</strong>: 4 epochs with 240 training steps</li>\n<li><strong>Multi-GPU</strong>: Data distributed across 4 GPUs (each processes ~1/4 of tasks)</li>\n<li><strong>Learning rates</strong>: 1e-4 (main), 1e-5 (embeddings)</li>\n</ul>\n<h2>Inference Strategy</h2>\n<ul>\n<li><strong>Task-specific priming</strong>: Model is briefly retrained on each test task's examples before prediction</li>\n<li><strong>Turbo DFS beam search</strong>: Efficiently explores multiple prediction candidates</li>\n<li><strong>Augmented scoring</strong>: Predictions evaluated across multiple augmentations to select the most robust solution</li>\n<li><strong>Probability tracking</strong>: Maintains cumulative probabilities to identify best candidates</li>\n</ul>\n<h1>My Contribution: Treating Random Seed as a Hyperparameter</h1>\n<p>While this solution builds upon the strong 2024 first-place baseline, I made one key modification that proved surprisingly impactful:\n<strong>Changed the random seed to 19920627</strong> , selected based on empirical exploration of seed values.</p>\n<h2>Random Seed as a Legitimate Hyperparameter</h2>\n<p>In scenarios with <strong>small sample sizes</strong> (like this competition's 240 evaluation tasks), random seed should be treated as a <strong>first-class hyperparameter</strong> rather than an afterthought. This is because:</p>\n<ul>\n<li><p><strong>High variance in small samples</strong>: With only 120 tasks per leaderboard split, solving just 1-2 additional tasks can dramatically shift rankings</p></li>\n<li><p><strong>Stochastic model behavior</strong>: Random seeds influence multiple pipeline stages, creating meaningfully different model behaviors</p></li>\n<li><p><strong>Task-specific interactions</strong>: Different seeds may produce augmentation patterns or training dynamics better suited to specific task types</p></li>\n</ul>\n<p>This seed change creates cascading effects throughout the entire machine learning pipeline:</p>\n<ul>\n<li><strong>Data shuffling</strong>: Alters the order in which training examples are presented</li>\n<li><strong>Augmentation patterns</strong>: Modifies rotations, transpositions, and color permutations applied to training data</li>\n<li><strong>Model training</strong>: Influences weight initialization in LoRA layers and training dynamics</li>\n<li><strong>Inference behavior</strong>: Changes stochastic elements in beam search decoding</li>\n<li><strong>Convergence paths</strong>: Leads to different optimization trajectories and final model states</li>\n</ul>\n<h1>The Hyperparameter Landscape: Beyond Random Seeds</h1>\n<p>While random seed optimization proved effective in this competition, it's important to recognize that this solution space contains <strong>numerous other hyperparameters</strong> worthy of systematic exploration:</p>\n<h2>1. Model Selection and Architecture Hyperparameters</h2>\n<p><strong>Base Model Choice</strong> - One of the most impactful decisions:</p>\n<ul>\n<li><strong>Nemo Mini</strong> (current choice): Efficient, fast inference, good balance</li>\n<li><strong>Llama 3.1 variants</strong>: 8B, 70B models with different reasoning capabilities</li>\n<li><strong>Qwen 2.5</strong>: Strong performance on reasoning tasks</li>\n<li><strong>Gemma 2</strong>: Google's open models with different architectural choices</li>\n<li><strong>Phi-3/Phi-4</strong>: Microsoft's small language models optimized for reasoning</li>\n<li><strong>Mistral variants</strong>: 7B, Mixtral MoE models with different trade-offs</li>\n<li><strong>DeepSeek models</strong>: Strong coding and reasoning abilities</li>\n</ul>\n<p>⠀Different base models have fundamentally different:</p>\n<ul>\n<li>Vocabulary and tokenization strategies (affecting grid representation)</li>\n<li>Architectural inductive biases (attention patterns, layer depths)</li>\n<li>Pre-training distributions (some models see more code/structured data)</li>\n<li>Context window capabilities (affecting maximum task size handling)</li>\n<li>Inference speed and memory requirements (computational constraints)</li>\n</ul>\n<p>⠀<strong>Architecture Configuration</strong>:</p>\n<ul>\n<li><strong>LoRA rank</strong>: Currently set to 32, but values from 8 to 128+ could be explored</li>\n<li><strong>LoRA alpha</strong>: Scaling factor for LoRA updates</li>\n<li><strong>LoRA target modules</strong>: Which layers to apply LoRA (attention, MLP, both)</li>\n<li><strong>Trainable layers</strong>: Currently first 32 layers; different layer selections might capture different abstractions</li>\n<li><strong>Quantization level</strong>: 4-bit vs 8-bit vs full precision trade-offs</li>\n<li><strong>Model size within family</strong>: Different parameter counts capture different complexity levels</li>\n</ul>\n<h2>2. Training Hyperparameters</h2>\n<ul>\n<li><strong>Learning rates</strong>: Main (1e-4) and embedding (1e-5) rates could be tuned independently</li>\n<li><strong>Number of epochs</strong>: Currently 4; more epochs might improve convergence or cause overfitting</li>\n<li><strong>Training steps</strong>: 240 steps per epoch is another tunable parameter</li>\n<li><strong>Batch size</strong>: Affects training stability and generalization</li>\n<li><strong>Gradient accumulation steps</strong>: For effective larger batch sizes</li>\n<li><strong>Optimizer choice</strong>: Adam, AdamW, SGD with different momentum values</li>\n<li><strong>Learning rate schedule</strong>: Constant vs cosine decay vs linear warmup strategies</li>\n<li><strong>Weight decay</strong>: Regularization strength</li>\n<li><strong>Dropout rates</strong>: If applicable to the fine-tuning strategy</li>\n</ul>\n<h2>3. Data Augmentation Hyperparameters</h2>\n<ul>\n<li><strong>Augmentation probability</strong>: How frequently each augmentation is applied</li>\n<li><strong>Augmentation types</strong>: Which transformations to include/exclude (transposition, rotation, color permutation)</li>\n<li><strong>Example shuffling strategy</strong>: Different orderings of demonstration examples</li>\n<li><strong>Color mapping schemes</strong>: Alternative color permutation strategies</li>\n<li><strong>Rotation angles</strong>: 90°, 180°, 270° vs other transformations</li>\n<li><strong>Augmentation composition</strong>: Sequential vs parallel application</li>\n</ul>\n<h2>4. Inference Hyperparameters</h2>\n<ul>\n<li><strong>Beam search width</strong>: Number of candidates to explore simultaneously</li>\n<li><strong>DFS depth limit</strong>: How deep to search in the solution space</li>\n<li><strong>Temperature</strong>: Sampling temperature for generation (typically 0.1 to 1.5)</li>\n<li><strong>Top-k/top-p values</strong>: Nucleus sampling parameters</li>\n<li><strong>Repetition penalty</strong>: To discourage repetitive patterns</li>\n<li><strong>Priming epochs</strong>: How long to fine-tune on test task examples</li>\n<li><strong>Number of augmentations</strong>: How many augmented versions to score</li>\n<li><strong>Maximum generation length</strong>: Token limits for output</li>\n</ul>\n<h2>5. Selection Algorithm Hyperparameters</h2>\n<ul>\n<li><strong>Scoring weights</strong>: How to balance different scoring criteria</li>\n<li><strong>Ensemble strategies</strong>: How to combine predictions from multiple runs</li>\n<li><strong>Confidence thresholds</strong>: When to trust a prediction vs explore alternatives</li>\n<li><strong>Voting mechanisms</strong>: Majority vote vs weighted vote vs consensus strategies</li>\n</ul>\n<h2>The Small Sample Size Advantage</h2>\n<p>In competitions with small evaluation sets (100-500 samples), these hyperparameters become <strong>especially influential</strong> because:</p>\n<ul>\n<li><strong>Lower statistical power</strong>: Individual hyperparameter choices create measurable differences in final scores</li>\n<li><strong>Higher variance sensitivity</strong>: Small improvements (1-3 additional correct predictions) translate to significant rank changes</li>\n<li><strong>Exploration feasibility</strong>: With limited test data, systematic grid search or random search over hyperparameters becomes computationally tractable</li>\n<li><strong>Overfitting opportunities</strong>: Hyperparameters can be tuned to exploit specific characteristics of the small evaluation distribution</li>\n</ul>\n<p>This doesn't diminish the achievement—rather, it highlights that <strong>comprehensive hyperparameter optimization is a valuable and legitimate competition strategy</strong>, especially when the evaluation set is constrained.</p>\n<h1>Motivation: The Statistics Behind Random Seed Optimization</h1>\n<p>My approach was motivated by understanding the <strong>statistical landscape of the competition</strong>:</p>\n<h3>Competition Structure</h3>\n<ul>\n<li><strong>Public LB</strong>: 120 samples (scores reflected on public leaderboard during competition; samples not visible to participants)</li>\n<li><strong>Private LB</strong>: 120 samples (used to compute final private scores; revealed after December 5, 2025)</li>\n</ul>\n<h2>The Random Seed Strategy</h2>\n<p>Given that:</p>\n<ul>\n<li>Most competitive solutions were variations of the same baseline notebook</li>\n<li>The same model can produce different scores across runs due to stochasticity</li>\n<li>There are only 120 samples in both public LB and private LB</li>\n</ul>\n<p><strong>This creates an opportunity</strong>: systematically exploring hyperparameters (including random seeds) can yield significantly different rankings even without fundamental algorithmic improvements.</p>\n<h3>Score Variance Analysis</h3>\n<p><strong>My observed performance across different random seeds</strong>:</p>\n<ul>\n<li>Best public LB score: <strong>4.17%</strong> (5/120 tasks)</li>\n<li>Estimated private score: <strong>~10%</strong> (12/120 tasks)</li>\n<li><strong>This notebook's score range: 3.33 % to 6.67 %(with the upper bound potentially higher)</strong> across different random seed configurations</li>\n<li>Score variance: ±3.34 percentage points (equivalent to ±4 tasks out of 120)</li>\n<li><strong>Pixel-Level Sensitivity Amplification</strong>: The all-or-nothing scoring (one wrong pixel = full failure) magnifies this variance; hyperparameter tweaks that nudge predictions toward exact matches on borderline tasks can swing 1-2 full solves, equivalent to 1-2% score jumps.</li>\n</ul>\n<p>This <strong>2x variation in performance</strong> (from 3.33 % to 6.67 %) demonstrates the substantial impact random seed can have as a hyperparameter in small-sample scenarios.</p>\n<h2>Public LB landscape:</h2>\n<ul>\n<li>8 teams achieved &gt;10% score (&gt;12 tasks solved)</li>\n<li>Significant variance exists even among top solutions</li>\n<li>Top solutions likely benefited from both strong architectures AND favorable hyperparameter configurations</li>\n</ul>\n<h2>Probability Calculation</h2>\n<p><strong>Question</strong>: Under the simplifying assumption of a model with 4% base accuracy on independent tasks (which isn't the actual case, as task accuracies vary and dependencies exist), what's the probability of achieving a 10% score (12/120 tasks)?</p>\n<p>Assuming each task is independent with success probability p = 0.04:</p>\n<ul>\n<li>P(X ≥ 12) where X ~ Binomial(n=120, p=0.04)</li>\n<li>Expected value: E[X] = 120 × 0.04 = 4.8 tasks</li>\n<li>Standard deviation: σ = √(120 × 0.04 × 0.96) ≈ 2.15</li>\n<li>P(X ≥ 12) ≈ P(Z ≥ (12 - 4.8) / 2.15) ≈ P(Z ≥ 3.35) ≈ 0.04 %</li>\n</ul>\n<p>This is extremely unlikely (~1 in 2,500 chance) if purely random!</p>\n<h3>The Real Insight</h3>\n<p>However, the actual mechanism is more nuanced:</p>\n<ul>\n<li><strong>Model capability varies by task type</strong>: The model may have 10-15% accuracy on certain task patterns and 0-2% on others</li>\n<li><strong>Hyperparameters affect task-specific performance</strong>: Different hyperparameter configurations (including seeds, base models, and training strategies) produce different augmentation patterns and model behaviors, leading to varying performance on specific tasks</li>\n<li><strong>Small sample size amplification</strong>: With only 120 tasks per split, small improvements (solving 1-2 more tasks) create large ranking jumps</li>\n<li><strong>Model selection matters significantly</strong>: Different base models have varying strengths across task categories—some excel at spatial reasoning, others at pattern completion or color transformations\n<strong>Strategic implication</strong>:</li>\n<li>While LLMs struggle with ARC-AGI reasoning (fundamental limitation), Achieving competitive results is possible through:<ul>\n<li>Starting from strong baselines</li>\n<li>Systematic hyperparameter exploration (seeds, learning rates, augmentation strategies, etc.)</li>\n<li><strong>Exploring different base model architectures</strong> (Llama, Qwen, Gemma, Phi, etc.)</li>\n<li>Understanding which task types benefit from different configurations</li>\n<li>Leveraging the competition's small evaluation set for targeted optimization</li></ul></li>\n</ul>\n<h3>Competitive Reality</h3>\n<p>This is <strong>not about gaming the system</strong>—it's about:</p>\n<ul>\n<li>Recognizing that <strong>all stochastic elements are legitimate hyperparameters</strong> in small-sample scenarios</li>\n<li>Understanding that competition rankings reflect both model quality AND optimal hyperparameter selection</li>\n<li>Acknowledging that with 240 evaluation samples, hyperparameter optimization matters significantly</li>\n<li>Appreciating that hyperparameter tuning is a <strong>core machine learning skill</strong>, not a shortcut</li>\n<li><strong>Recognizing that base model selection is itself a critical hyperparameter choice</strong></li>\n</ul>\n<p>My 5th place finish validates that while <strong>solving ARC-AGI fundamentally remains extremely difficult</strong>, competitive performance in this format is achievable through careful exploration of the hyperparameter space, including often-overlooked parameters like random seeds and the foundational choice of base model architecture.</p>\n<h1>What Worked</h1>\n<ul>\n<li><strong>Strong baseline architecture</strong>: The ARChitects' original design proved highly effective as a foundation</li>\n<li><strong>Multi-GPU distribution</strong>: Enabled processing all tasks within the competition's tight time constraints</li>\n<li><strong>Task-specific priming</strong>: Improved predictions by conditioning on individual challenge characteristics</li>\n<li><strong>Augmented scoring</strong>: Enhanced prediction selection through more robust evaluation metrics</li>\n<li><strong>Hyperparameter optimization</strong>: Treating random seed as a tunable hyperparameter achieved competitive results, with score variance of 3.33 % to 6.67 % demonstrating significant impact</li>\n<li><strong>Nemo Mini as base model</strong>: Balanced efficiency and performance well for this specific task distribution</li>\n</ul>\n<h1>What Could Be Improved</h1>\n<ul>\n<li><strong>Systematic hyperparameter search</strong>: A principled grid search or Bayesian optimization over multiple hyperparameters (not just random seed) could identify even better configurations</li>\n<li><strong>Base model exploration</strong>: Testing alternative foundation models (Llama 3.2, Qwen 3, Gemma 3, Phi-3/4, Mistral, DeepSeek) could reveal architectures better suited to ARC-AGI reasoning patterns</li>\n<li><strong>Model ensemble methods</strong>: Combining predictions from multiple base models with different architectural biases could improve robustness and coverage across diverse task types</li>\n<li><strong>Seed ensemble strategies</strong>: Rather than selecting a single optimal seed, averaging or voting across multiple seed configurations could reduce variance</li>\n<li><strong>Public-private gap analysis</strong>: Deeper investigation into the score discrepancy would provide valuable insights for future competitions</li>\n<li><strong>Selection algorithms</strong>: More sophisticated prediction selection methods could improve consistency</li>\n<li><strong>Architectural enhancements</strong>: Exploring modifications beyond LoRA fine-tuning (full fine-tuning, adapter layers, prefix tuning) might unlock further gains</li>\n<li><strong>Edge case handling</strong>: Better strategies for tasks with unusual or rare patterns</li>\n<li><strong>Cross-validation on small samples</strong>: Developing robust validation strategies for competitions with limited evaluation data</li>\n<li><strong>Multi-stage hyperparameter optimization</strong>: Sequential optimization (model selection → training hyperparameters → inference hyperparameters) could be more efficient than random search</li>\n<li><strong>Pixel-Precision Robustness</strong>: Develop tolerance-aware scoring proxies during hyperparameter search to better handle the one-pixel-failure risk, e.g., via soft-match metrics in validation to guide toward exact predictions without overfitting to noise.</li>\n<li><strong>Symbolic Function Generation</strong>: Rather than directly predicting output grids pixel-by-pixel, explore generating interpretable transformation functions (e.g., via program synthesis or rule induction) from diverse training samples. This approach could enhance generalization across unseen task variations by capturing underlying patterns more abstractly, mitigating the fragility of raw pixel outputs.</li>\n</ul>\n<h1>Conclusion</h1>\n<p>This competition experience has been both humbling and inspiring. Achieving 5th place on the private leaderboard demonstrates that in competitions with small evaluation sets, <strong>comprehensive hyperparameter optimization is not just beneficial—it's essential</strong>. The lesson extends beyond random seeds: base model selection, learning rate schedules, augmentation strategies, model architecture choices, and inference parameters all constitute a rich landscape worthy of systematic exploration.</p>\n<p>The observed score variance (3.33 % to 6.67 %) from random seed changes alone illustrates how impactful these seemingly minor decisions can be. When we consider the additional dimensions of model selection (Llama vs Qwen vs Gemma vs Phi vs Mistral vs DeepSeek) and the interactions between these choices and other hyperparameters, the potential solution space becomes even more expansive.</p>\n<p>The key insight is recognizing when hyperparameter variance becomes comparable to or exceeds model variance. In such scenarios, treating traditionally \"fixed\" parameters (like random seeds and base model choices) as first-class hyperparameters becomes a legitimate and powerful strategy.</p>\n<p>I want to express my sincere gratitude to <strong>Kaggle and the entire ARC Prize community</strong> for creating this challenging and thought-provoking competition. The opportunity to engage with abstract reasoning tasks and learn from world-class competitors has been invaluable. Special thanks to the ARChitects team for open-sourcing their solution, and to <a href=\"https://www.kaggle.com/sanyul\" target=\"_blank\">@sanyul</a> and <a href=\"https://www.kaggle.com/boristown\" target=\"_blank\">@boristown</a> for their excellent baseline implementations.</p>\n<p>This competition has motivated me to become more deeply involved in future ARC challenges. The problem of abstract reasoning represents one of the most fascinating frontiers in AI, and I'm excited to contribute to advancing this field. For future iterations, I plan to systematically explore the base model selection space alongside other hyperparameters to better understand which architectural choices align best with different types of abstract reasoning tasks.</p>\n<p><strong>To anyone reading this writeup</strong>: In small-sample competitions, never underestimate the importance of exploring every hyperparameter systematically, starting with the fundamental choice of base model architecture. What seems like a minor configuration choice—whether it's a random seed, a learning rate, or a model selection—may be the difference between a good solution and a winning one. The 2x performance variance I observed from seed changes alone (3.33 % to 6.67 %) suggests that comprehensive exploration of the full hyperparameter space could yield even more dramatic improvements.</p>",
      "rawMarkdown": "# Overview\nThis submission forks the notebook ~[LB 5.83 baseline from 1st place of 2024](https://www.kaggle.com/code/sanyul/lb-5-83-baseline-from-1st-place-of-2024)~ by @sanyul and the ~[baseline from 1st place of 2024](https://www.kaggle.com/code/boristown/baseline-from-1st-place-of-2024)~ by @boristown, which originates from the first-place solution from ARC Prize 2024 by the ARChitects team (Daniel Franzen and Jan Disselhoff). Full credits to all of them for their exceptional work.\nMy submission notebook can be found here: https://www.kaggle.com/code/lonnieqin/lb-5-83-baseline-from-1st-place-of-2024\n# Performance\n**Public Leaderboard**: 344th place\n\n**Private Leaderboard**: 5th place\n\nThe dramatic improvement from public to private leaderboard rankings highlights the importance of model generalization and the critical role of hyperparameter optimization in competitive machine learning, especially when working with small evaluation sets.\n\n# Structure of the Solution\nThe solution consists of several key components organized as follows:\n\n## 1. Core Infrastructure Files:\n* model_runner.py: Handles model loading, training, inference, and vocabulary shrinking\n* selection.py: Implements various algorithms for selecting the best predictions\n* async_tools.py: Manages parallel execution across multiple GPUs\n* common_stuff.py: Central configuration hub containing all hyperparameters and orchestration logic\n\n## 2. Training Pipeline:\n* Multi-GPU distributed training (4 GPUs) with different data splits\n* LoRA fine-tuning on a quantized base model (Nemo Mini)\n* Data augmentation (transposition, rotation, color permutation)\n* Task-specific priming for improved predictions\n\n## 3. Inference Pipeline:\n* Turbo DFS (Depth-First Search) beam search for efficient decoding\n* Augmented scoring mechanism for prediction selection\n* Parallel inference across multiple GPUs\n* Result caching and incremental processing\n\n## 4. Execution Flow:\n* Parallel training on 4 GPUs → Wait for completion → Parallel inference on 4 GPUs → Aggregate results → Generate submission\n\n# Methodology\nThe solution employs a **fine-tuned language model approach** to the ARC-AGI challenge:\n## Base Architecture\n* **Model**: Nemo Mini (fine-tuned on the ARC 2024 dataset – the best solution of ARC 2024)\n* **Fine-tuning**: LoRA (Low-Rank Adaptation) with rank 32\n* **Active layers**: Only the first 32 layers trainable to reduce overfitting\n\n## Data Representation\n* Tasks are converted to text format using a specialized formatter (ArcFormatter_premix_3)\n* Input/output grids are serialized as token sequences\n* Maximum sequence length: 4,224 tokens (training), 8,192 tokens (inference)\n\n## Training Strategy\n* **Augmentation**: Transposition, rotation, random color permutation, example shuffling\n* **Epochs**: 4 epochs with 240 training steps\n* **Multi-GPU**: Data distributed across 4 GPUs (each processes ~1/4 of tasks)\n* **Learning rates**: 1e-4 (main), 1e-5 (embeddings)\n\n## Inference Strategy\n* **Task-specific priming**: Model is briefly retrained on each test task's examples before prediction\n* **Turbo DFS beam search**: Efficiently explores multiple prediction candidates\n* **Augmented scoring**: Predictions evaluated across multiple augmentations to select the most robust solution\n* **Probability tracking**: Maintains cumulative probabilities to identify best candidates\n\n# My Contribution: Treating Random Seed as a Hyperparameter\nWhile this solution builds upon the strong 2024 first-place baseline, I made one key modification that proved surprisingly impactful:\n**Changed the random seed to 19920627** , selected based on empirical exploration of seed values.\n## Random Seed as a Legitimate Hyperparameter\nIn scenarios with **small sample sizes** (like this competition's 240 evaluation tasks), random seed should be treated as a **first-class hyperparameter** rather than an afterthought. This is because:\n\n* **High variance in small samples**: With only 120 tasks per leaderboard split, solving just 1-2 additional tasks can dramatically shift rankings\n\n* **Stochastic model behavior**: Random seeds influence multiple pipeline stages, creating meaningfully different model behaviors\n\n* **Task-specific interactions**: Different seeds may produce augmentation patterns or training dynamics better suited to specific task types\n\nThis seed change creates cascading effects throughout the entire machine learning pipeline:\n* **Data shuffling**: Alters the order in which training examples are presented\n* **Augmentation patterns**: Modifies rotations, transpositions, and color permutations applied to training data\n* **Model training**: Influences weight initialization in LoRA layers and training dynamics\n* **Inference behavior**: Changes stochastic elements in beam search decoding\n* **Convergence paths**: Leads to different optimization trajectories and final model states\n\n# The Hyperparameter Landscape: Beyond Random Seeds\nWhile random seed optimization proved effective in this competition, it's important to recognize that this solution space contains **numerous other hyperparameters** worthy of systematic exploration:\n## 1. Model Selection and Architecture Hyperparameters\n**Base Model Choice** - One of the most impactful decisions:\n* **Nemo Mini** (current choice): Efficient, fast inference, good balance\n* **Llama 3.1 variants**: 8B, 70B models with different reasoning capabilities\n* **Qwen 2.5**: Strong performance on reasoning tasks\n* **Gemma 2**: Google's open models with different architectural choices\n* **Phi-3/Phi-4**: Microsoft's small language models optimized for reasoning\n* **Mistral variants**: 7B, Mixtral MoE models with different trade-offs\n* **DeepSeek models**: Strong coding and reasoning abilities\n\n⠀Different base models have fundamentally different:\n* Vocabulary and tokenization strategies (affecting grid representation)\n* Architectural inductive biases (attention patterns, layer depths)\n* Pre-training distributions (some models see more code/structured data)\n* Context window capabilities (affecting maximum task size handling)\n* Inference speed and memory requirements (computational constraints)\n\n⠀**Architecture Configuration**:\n* **LoRA rank**: Currently set to 32, but values from 8 to 128+ could be explored\n* **LoRA alpha**: Scaling factor for LoRA updates\n* **LoRA target modules**: Which layers to apply LoRA (attention, MLP, both)\n* **Trainable layers**: Currently first 32 layers; different layer selections might capture different abstractions\n* **Quantization level**: 4-bit vs 8-bit vs full precision trade-offs\n* **Model size within family**: Different parameter counts capture different complexity levels\n\n## 2. Training Hyperparameters\n* **Learning rates**: Main (1e-4) and embedding (1e-5) rates could be tuned independently\n* **Number of epochs**: Currently 4; more epochs might improve convergence or cause overfitting\n* **Training steps**: 240 steps per epoch is another tunable parameter\n* **Batch size**: Affects training stability and generalization\n* **Gradient accumulation steps**: For effective larger batch sizes\n* **Optimizer choice**: Adam, AdamW, SGD with different momentum values\n* **Learning rate schedule**: Constant vs cosine decay vs linear warmup strategies\n* **Weight decay**: Regularization strength\n* **Dropout rates**: If applicable to the fine-tuning strategy\n\n## 3. Data Augmentation Hyperparameters\n* **Augmentation probability**: How frequently each augmentation is applied\n* **Augmentation types**: Which transformations to include/exclude (transposition, rotation, color permutation)\n* **Example shuffling strategy**: Different orderings of demonstration examples\n* **Color mapping schemes**: Alternative color permutation strategies\n* **Rotation angles**: 90°, 180°, 270° vs other transformations\n* **Augmentation composition**: Sequential vs parallel application\n\n## 4. Inference Hyperparameters\n* **Beam search width**: Number of candidates to explore simultaneously\n* **DFS depth limit**: How deep to search in the solution space\n* **Temperature**: Sampling temperature for generation (typically 0.1 to 1.5)\n* **Top-k/top-p values**: Nucleus sampling parameters\n* **Repetition penalty**: To discourage repetitive patterns\n* **Priming epochs**: How long to fine-tune on test task examples\n* **Number of augmentations**: How many augmented versions to score\n* **Maximum generation length**: Token limits for output\n\n## 5. Selection Algorithm Hyperparameters\n* **Scoring weights**: How to balance different scoring criteria\n* **Ensemble strategies**: How to combine predictions from multiple runs\n* **Confidence thresholds**: When to trust a prediction vs explore alternatives\n* **Voting mechanisms**: Majority vote vs weighted vote vs consensus strategies\n\n## The Small Sample Size Advantage\nIn competitions with small evaluation sets (100-500 samples), these hyperparameters become **especially influential** because:\n* **Lower statistical power**: Individual hyperparameter choices create measurable differences in final scores\n* **Higher variance sensitivity**: Small improvements (1-3 additional correct predictions) translate to significant rank changes\n* **Exploration feasibility**: With limited test data, systematic grid search or random search over hyperparameters becomes computationally tractable\n* **Overfitting opportunities**: Hyperparameters can be tuned to exploit specific characteristics of the small evaluation distribution\n\nThis doesn't diminish the achievement—rather, it highlights that **comprehensive hyperparameter optimization is a valuable and legitimate competition strategy**, especially when the evaluation set is constrained.\n\n# Motivation: The Statistics Behind Random Seed Optimization\nMy approach was motivated by understanding the **statistical landscape of the competition**:\n### Competition Structure\n* **Public LB**: 120 samples (scores reflected on public leaderboard during competition; samples not visible to participants)\n* **Private LB**: 120 samples (used to compute final private scores; revealed after December 5, 2025)\n\n## The Random Seed Strategy\nGiven that:\n* Most competitive solutions were variations of the same baseline notebook\n* The same model can produce different scores across runs due to stochasticity\n* There are only 120 samples in both public LB and private LB\n\n**This creates an opportunity**: systematically exploring hyperparameters (including random seeds) can yield significantly different rankings even without fundamental algorithmic improvements.\n### Score Variance Analysis\n**My observed performance across different random seeds**:\n* Best public LB score: **4.17%** (5/120 tasks)\n* Estimated private score: **~10%** (12/120 tasks)\n* **This notebook's score range: 3.33 % to 6.67 %(with the upper bound potentially higher)** across different random seed configurations\n* Score variance: ±3.34 percentage points (equivalent to ±4 tasks out of 120)\n* **Pixel-Level Sensitivity Amplification**: The all-or-nothing scoring (one wrong pixel = full failure) magnifies this variance; hyperparameter tweaks that nudge predictions toward exact matches on borderline tasks can swing 1-2 full solves, equivalent to 1-2% score jumps.\n\nThis **2x variation in performance** (from 3.33 % to 6.67 %) demonstrates the substantial impact random seed can have as a hyperparameter in small-sample scenarios.\n\n## Public LB landscape:\n* 8 teams achieved >10% score (>12 tasks solved)\n* Significant variance exists even among top solutions\n* Top solutions likely benefited from both strong architectures AND favorable hyperparameter configurations\n\n## Probability Calculation\n**Question**: Under the simplifying assumption of a model with 4% base accuracy on independent tasks (which isn't the actual case, as task accuracies vary and dependencies exist), what's the probability of achieving a 10% score (12/120 tasks)?\n\nAssuming each task is independent with success probability p = 0.04:\n* P(X ≥ 12) where X ~ Binomial(n=120, p=0.04)\n* Expected value: E[X] = 120 × 0.04 = 4.8 tasks\n* Standard deviation: σ = √(120 × 0.04 × 0.96) ≈ 2.15\n* P(X ≥ 12) ≈ P(Z ≥ (12 - 4.8) / 2.15) ≈ P(Z ≥ 3.35) ≈ 0.04 %\n\nThis is extremely unlikely (~1 in 2,500 chance) if purely random!\n### The Real Insight\nHowever, the actual mechanism is more nuanced:\n* **Model capability varies by task type**: The model may have 10-15% accuracy on certain task patterns and 0-2% on others\n* **Hyperparameters affect task-specific performance**: Different hyperparameter configurations (including seeds, base models, and training strategies) produce different augmentation patterns and model behaviors, leading to varying performance on specific tasks\n* **Small sample size amplification**: With only 120 tasks per split, small improvements (solving 1-2 more tasks) create large ranking jumps\n* **Model selection matters significantly**: Different base models have varying strengths across task categories—some excel at spatial reasoning, others at pattern completion or color transformations\n**Strategic implication**:\n* While LLMs struggle with ARC-AGI reasoning (fundamental limitation), Achieving competitive results is possible through:\n  * Starting from strong baselines\n  * Systematic hyperparameter exploration (seeds, learning rates, augmentation strategies, etc.)\n  * **Exploring different base model architectures** (Llama, Qwen, Gemma, Phi, etc.)\n  * Understanding which task types benefit from different configurations\n  * Leveraging the competition's small evaluation set for targeted optimization\n\n### Competitive Reality\nThis is **not about gaming the system**—it's about:\n* Recognizing that **all stochastic elements are legitimate hyperparameters** in small-sample scenarios\n* Understanding that competition rankings reflect both model quality AND optimal hyperparameter selection\n* Acknowledging that with 240 evaluation samples, hyperparameter optimization matters significantly\n* Appreciating that hyperparameter tuning is a **core machine learning skill**, not a shortcut\n* **Recognizing that base model selection is itself a critical hyperparameter choice**\n\nMy 5th place finish validates that while **solving ARC-AGI fundamentally remains extremely difficult**, competitive performance in this format is achievable through careful exploration of the hyperparameter space, including often-overlooked parameters like random seeds and the foundational choice of base model architecture.\n\n# What Worked\n* **Strong baseline architecture**: The ARChitects' original design proved highly effective as a foundation\n* **Multi-GPU distribution**: Enabled processing all tasks within the competition's tight time constraints\n* **Task-specific priming**: Improved predictions by conditioning on individual challenge characteristics\n* **Augmented scoring**: Enhanced prediction selection through more robust evaluation metrics\n* **Hyperparameter optimization**: Treating random seed as a tunable hyperparameter achieved competitive results, with score variance of 3.33 % to 6.67 % demonstrating significant impact\n* **Nemo Mini as base model**: Balanced efficiency and performance well for this specific task distribution\n\n# What Could Be Improved\n* **Systematic hyperparameter search**: A principled grid search or Bayesian optimization over multiple hyperparameters (not just random seed) could identify even better configurations\n* **Base model exploration**: Testing alternative foundation models (Llama 3.2, Qwen 3, Gemma 3, Phi-3/4, Mistral, DeepSeek) could reveal architectures better suited to ARC-AGI reasoning patterns\n* **Model ensemble methods**: Combining predictions from multiple base models with different architectural biases could improve robustness and coverage across diverse task types\n* **Seed ensemble strategies**: Rather than selecting a single optimal seed, averaging or voting across multiple seed configurations could reduce variance\n* **Public-private gap analysis**: Deeper investigation into the score discrepancy would provide valuable insights for future competitions\n* **Selection algorithms**: More sophisticated prediction selection methods could improve consistency\n* **Architectural enhancements**: Exploring modifications beyond LoRA fine-tuning (full fine-tuning, adapter layers, prefix tuning) might unlock further gains\n* **Edge case handling**: Better strategies for tasks with unusual or rare patterns\n* **Cross-validation on small samples**: Developing robust validation strategies for competitions with limited evaluation data\n* **Multi-stage hyperparameter optimization**: Sequential optimization (model selection → training hyperparameters → inference hyperparameters) could be more efficient than random search\n* **Pixel-Precision Robustness**: Develop tolerance-aware scoring proxies during hyperparameter search to better handle the one-pixel-failure risk, e.g., via soft-match metrics in validation to guide toward exact predictions without overfitting to noise.\n* **Symbolic Function Generation**: Rather than directly predicting output grids pixel-by-pixel, explore generating interpretable transformation functions (e.g., via program synthesis or rule induction) from diverse training samples. This approach could enhance generalization across unseen task variations by capturing underlying patterns more abstractly, mitigating the fragility of raw pixel outputs.\n\n# Conclusion\nThis competition experience has been both humbling and inspiring. Achieving 5th place on the private leaderboard demonstrates that in competitions with small evaluation sets, **comprehensive hyperparameter optimization is not just beneficial—it's essential**. The lesson extends beyond random seeds: base model selection, learning rate schedules, augmentation strategies, model architecture choices, and inference parameters all constitute a rich landscape worthy of systematic exploration.\n\nThe observed score variance (3.33 % to 6.67 %) from random seed changes alone illustrates how impactful these seemingly minor decisions can be. When we consider the additional dimensions of model selection (Llama vs Qwen vs Gemma vs Phi vs Mistral vs DeepSeek) and the interactions between these choices and other hyperparameters, the potential solution space becomes even more expansive.\n\nThe key insight is recognizing when hyperparameter variance becomes comparable to or exceeds model variance. In such scenarios, treating traditionally \"fixed\" parameters (like random seeds and base model choices) as first-class hyperparameters becomes a legitimate and powerful strategy.\n\nI want to express my sincere gratitude to **Kaggle and the entire ARC Prize community** for creating this challenging and thought-provoking competition. The opportunity to engage with abstract reasoning tasks and learn from world-class competitors has been invaluable. Special thanks to the ARChitects team for open-sourcing their solution, and to @sanyul and @boristown for their excellent baseline implementations.\n\nThis competition has motivated me to become more deeply involved in future ARC challenges. The problem of abstract reasoning represents one of the most fascinating frontiers in AI, and I'm excited to contribute to advancing this field. For future iterations, I plan to systematically explore the base model selection space alongside other hyperparameters to better understand which architectural choices align best with different types of abstract reasoning tasks.\n\n**To anyone reading this writeup**: In small-sample competitions, never underestimate the importance of exploring every hyperparameter systematically, starting with the fundamental choice of base model architecture. What seems like a minor configuration choice—whether it's a random seed, a learning rate, or a model selection—may be the difference between a good solution and a winning one. The 2x performance variance I observed from seed changes alone (3.33 % to 6.67 %) suggests that comprehensive exploration of the full hyperparameter space could yield even more dramatic improvements.",
      "votes": 13
    },
    {
      "id": 3368284,
      "postDate": "2025-12-09T05:41:57.103Z",
      "content": "<p>I thought the only way to improve the notebook performance was to change model or training settings, but the idea of ​​exploring the upper limit within a single training setting in this way is quite amazing.\nCongratulations on your 4th place!🎉</p>",
      "rawMarkdown": "I thought the only way to improve the notebook performance was to change model or training settings, but the idea of ​​exploring the upper limit within a single training setting in this way is quite amazing.\nCongratulations on your 4th place!🎉",
      "votes": 1,
      "replies": [
        {
          "id": 3368911,
          "postDate": "2025-12-09T15:47:52.920Z",
          "content": "<p><a href=\"https://www.kaggle.com/sanyul\" target=\"_blank\">@sanyul</a> Thank you again, your solution is excellent, keep up the good work.</p>",
          "rawMarkdown": "@sanyul Thank you again, your solution is excellent, keep up the good work.",
          "votes": 1
        }
      ]
    },
    {
      "id": 3378314,
      "postDate": "2025-12-17T22:06:24.683Z",
      "content": "<p>Thanks for the detailed writeup of your angle on this competition, it's really interesting to see.</p>\n<p>I do have a couple question. I understand that you treated the random seed as a first-class hyperparameter, and selected the final value based on empirical exploration of seed values. Could you help understand:</p>\n<ul>\n<li>how you went about this exploration of values?</li>\n<li>whether you optimized for additional parameters other than seed?</li>\n</ul>\n<p>Once again, thanks for sharing. It's very cool to see someone boil down a competition like this to such factors, and achieve such a high final ranking. Congratulations on the result!</p>",
      "rawMarkdown": "Thanks for the detailed writeup of your angle on this competition, it's really interesting to see.\n\nI do have a couple question. I understand that you treated the random seed as a first-class hyperparameter, and selected the final value based on empirical exploration of seed values. Could you help understand:\n- how you went about this exploration of values?\n- whether you optimized for additional parameters other than seed?\n\nOnce again, thanks for sharing. It's very cool to see someone boil down a competition like this to such factors, and achieve such a high final ranking. Congratulations on the result!",
      "votes": 2,
      "replies": [
        {
          "id": 3378549,
          "postDate": "2025-12-18T13:20:17.510Z",
          "content": "<p>Thanks so much for the thoughtful questions and kind words! To be honest, my approach was pretty straightforward given the time constraints. I tested several different seed values and tracked their public LB scores. When I found one that gave me the best public result, I went with it for my final submission. \n I really wasn't sure how this would translate to private LB—optimizing on public scores is always risky. But I believed the change would matter, so I took the bet and submitted the best-performing version. Looking back, the better approach would've been running evaluations on the training set first to get more reliable signals about generalization. But sometimes in competitions you just have to commit and keep trying.\nFor this submission, I mainly focused on the seed while keeping the baseline largely intact. I had ideas for exploring other things—different base models, training strategies, ensembles—but with limited compute and tight deadlines, I couldn't fully develop them.\nThe jump from 344th to 4th place really validated that systematic exploration matters in small-sample competitions, even if it's just one parameter. It's definitely motivated me to be more comprehensive in future competitions—better validation strategies, more systematic searches, and hopefully more compute resources!</p>",
          "rawMarkdown": "Thanks so much for the thoughtful questions and kind words! To be honest, my approach was pretty straightforward given the time constraints. I tested several different seed values and tracked their public LB scores. When I found one that gave me the best public result, I went with it for my final submission. \n I really wasn't sure how this would translate to private LB—optimizing on public scores is always risky. But I believed the change would matter, so I took the bet and submitted the best-performing version. Looking back, the better approach would've been running evaluations on the training set first to get more reliable signals about generalization. But sometimes in competitions you just have to commit and keep trying.\nFor this submission, I mainly focused on the seed while keeping the baseline largely intact. I had ideas for exploring other things—different base models, training strategies, ensembles—but with limited compute and tight deadlines, I couldn't fully develop them.\nThe jump from 344th to 4th place really validated that systematic exploration matters in small-sample competitions, even if it's just one parameter. It's definitely motivated me to be more comprehensive in future competitions—better validation strategies, more systematic searches, and hopefully more compute resources!",
          "votes": 1
        }
      ]
    },
    {
      "id": 3363198,
      "postDate": "2025-12-05T20:04:05.580Z",
      "content": "<p>Congrats on the 4th place!!  🎉🎉</p>\n<p>Btw ur submitted notebook is not public yet</p>",
      "rawMarkdown": "Congrats on the 4th place!!  🎉🎉\n\nBtw ur submitted notebook is not public yet",
      "votes": 2,
      "replies": [
        {
          "id": 3363346,
          "postDate": "2025-12-05T22:01:11.300Z",
          "content": "<p><a href=\"https://www.kaggle.com/nihilisticneuralnet\" target=\"_blank\">@nihilisticneuralnet</a>  Thank you so much! 🎉 And oh — thanks for the heads-up about the notebook! I've just made it public. Appreciate you letting me know!</p>",
          "rawMarkdown": "@nihilisticneuralnet  Thank you so much! 🎉 And oh — thanks for the heads-up about the notebook! I've just made it public. Appreciate you letting me know!",
          "votes": 1
        }
      ]
    },
    {
      "id": 3320705,
      "postDate": "2025-11-12T18:15:56.577Z",
      "content": "<p>Quick note from the competition hosts: final leaderboard places have not yet been awarded. Winners and prizes will be announced December 5, 2025.</p>",
      "rawMarkdown": "Quick note from the competition hosts: final leaderboard places have not yet been awarded. Winners and prizes will be announced December 5, 2025.",
      "votes": 1
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T21:00:21Z",
  "competition_id": 91496,
  "competition_slug": "arc-prize-2025",
  "team_id": 13591801,
  "solution_url": "/competitions/arc-prize-2025/writeups/arc-prize-2025-competition-writeup-5th-place"
}