![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F22580177%2F1882f27d117f014870751059d4ad1aba%2FSelection_001.png?generation=1731888674383323&alt=media\")
https://github.com/zoenguyenramirez/arc-prize-2024
\nhttps://www.kaggle.com/code/zoenguyenramirez/fs-1-inherted-newzoe-v7
Hi Kagglers! I'm excited to share my solution that scored 31 points in the ARC Prize 2024 competition. Instead of using existing language models, I took a \"from-scratch\" approach with custom transformer architectures, leading to some interesting findings.
\ngrid_encoding that outperforms traditional NLP positional encodingThe code is organized into two main components:
\ndev_folder: Development and training codesubmission_folder: Final submission codeSpecial thanks to:
\nFeel free to explore the code, provide feedback, or reach out with questions. Let's learn from each other and push the boundaries of what's possible with transformer architectures!
", "comments": [ { "id": 3046045, "author_name": "CPMP", "author_url": "", "post_date": "2024-11-15T04:51:02.787000", "content": "Thanks for sharing, will study your git repo. A score of 31 form scratch is impressive.
", "votes": 5, "replies": [] }, { "id": 3045996, "author_name": "Nikola Hu", "author_url": "", "post_date": "2024-11-15T03:40:05.653000", "content": " enum Enum, auto\n\n ():\n CELL_TOKEN_SIZE = \n PAD = auto() \n START = auto() \n START_INPUT = auto() \n END_INPUT = auto() \n START_OUTPUT = auto() \n END_OUTPUT = auto() \n END = auto() \n ROW_SEPARATOR = auto() \n COUNT_OF_TOKENS = auto() \n\nVOCAB_SIZE = SpecialToken.COUNT_OF_TOKENS.value\nThe grid encoding function combines four key positional elements:
\n ():\n use_grid_encoder:\n \n torch.(x[:, :, :] >= -), \n torch.(x[:, :, :] < max_grid_size), \n\n \n row_encodings = grid_encoding[:, x[:, :, ], :]\n col_encodings = grid_encoding[:, x[:, :, ], :]\n top_encodings = grid_encoding[:, x[:, :, ], :] \n sample_encoding = grid_encoding[:, x[:, :, ], :] \n\n \n grid_encodings = torch.cat([\n row_encodings * grid_scale[],\n col_encodings * grid_scale[],\n top_encodings * grid_scale[],\n sample_encoding\n ], dim=-) \n\n grid_encodings.squeeze()\n\n positional_encoding[:, :seq_length, :]\n ():\n input_ids.dim() == \n batch_size, seq_length, *_ = input_ids.shape\n batch_size == (end_of_example_indices)\n\n \n mask = torch.triu(torch.ones((seq_length, seq_length), device=device), diagonal=).()\n mask = mask.unsqueeze().repeat(batch_size, , )\n\n \n mask_hack:\n batch_index, end_index (end_of_example_indices):\n end_index > \n mask[batch_index, :, :end_index] = \n\n mask\nNote: The mask_hack option, which allows input tokens to see all other input tokens, did not improve model performance.
The loss function only considers output tokens:
\n ():\n prefix_length = torch.tensor(end_of_examples, device=target.device) \n mask = torch.arange(seq_length, device=target.device).unsqueeze() < prefix_length.unsqueeze() - \n target[mask] = SpecialToken.PAD.value\n\n target\nThis implementation enables a decoder-only transformer architecture, eliminating the need for a separate encoder component.
", "votes": 5, "replies": [] }, { "id": 3048440, "author_name": "Nikola Hu", "author_url": "", "post_date": "2024-11-18T00:14:00.133000", "content": "
This was the training trajectory, captured from the tensorboard.
\nI used 3x 4090 in the beginning and 2x 4090 since Nov 6.
", "votes": 3, "replies": [] }, { "id": 3048297, "author_name": "Guillermo Perez G", "author_url": "", "post_date": "2024-11-17T18:08:35.360000", "content": "Again, thanks for sharing and good luck for the next ARC contest! I hope the other kagglers do the same, since public scrutiny gives strength.
\nI like this original solution, and I will try to analyze if there are other variants that help improve this solution, and I will let you know. I think that to reach 85% an efficient, simple and elegant solution should be used.
\nI have seen other solutions and I donยดt believe that training a model for each problem and then assembling them is an efficient solution, because how is it possible to assemble 100 models of 100 problems. Itยดs like training 100 people to solve 100 problems, when one person could solve 70 or 80 math problems.
\nAll the best
", "votes": 1, "replies": [ { "id": 3048334, "author_name": "Nikola Hu", "author_url": "", "post_date": "2024-11-17T19:07:02.370000", "content": "\"Regarding 'assembling them,' could you provide a reference for this approach?
\nFrom my research, I've observed two main methods:
\nhttps://www.kaggle.com/code/ironbar/single-task-test-time-fine-tuning-for-arc24
\nhttps://www.kaggle.com/competitions/arc-prize-2024/discussion/545671
\n:)
", "votes": 0, "replies": [] }, { "id": 3048442, "author_name": "Nikola Hu", "author_url": "", "post_date": "2024-11-18T00:21:27.517000", "content": "It should still be the first method I listed. It's not so much like \"training 100 people to solve 100 problems\"; rather, it's more like training one person (pre-training) who then learns 100 tasks and solves them individually. This is more similar to how humans actually work - when a human faces a challenge, they think and modify their neural connections while solving it.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3045985, "author_name": "", "author_url": "", "post_date": "2024-11-15T03:06:32.947000", "content": "", "votes": 0, "replies": [] } ], "comment_count": 8, "messages": [ { "id": 3045948, "postDate": "2024-11-15T00:53:20.330Z", "content": "https://github.com/zoenguyenramirez/arc-prize-2024
\nhttps://www.kaggle.com/code/zoenguyenramirez/fs-1-inherted-newzoe-v7
Hi Kagglers! I'm excited to share my solution that scored 31 points in the ARC Prize 2024 competition. Instead of using existing language models, I took a \"from-scratch\" approach with custom transformer architectures, leading to some interesting findings.
\ngrid_encoding that outperforms traditional NLP positional encodingThe code is organized into two main components:
\ndev_folder: Development and training codesubmission_folder: Final submission codeSpecial thanks to:
\nFeel free to explore the code, provide feedback, or reach out with questions. Let's learn from each other and push the boundaries of what's possible with transformer architectures!
", "rawMarkdown": "https://github.com/zoenguyenramirez/arc-prize-2024\nhttps://www.kaggle.com/code/zoenguyenramirez/fs-1-inherted-newzoe-v7\n\nHi Kagglers! I'm excited to share my solution that scored 31 points in the ARC Prize 2024 competition. Instead of using existing language models, I took a \"from-scratch\" approach with custom transformer architectures, leading to some interesting findings.\n\n## ๐ Key Approaches & Findings\n1. **Active Inference**: Implemented based on insights from MindAI/Jack Cole's team (a game-changer when I was stuck at 4 points!)\n2. **Reverse Augmentation**: Originally implemented as \"consistency\", later discovered to align with MindAI's approach\n3. **Custom Positional Encoding**: Developed a specialized `grid_encoding` that outperforms traditional NLP positional encoding\n4. **Architectural Experiments**:\n - Transformer Mask Hack (disabled in final submission due to performance impact)\n - Progressive Head (impact uncertain due to later-discovered bugs)\n5. **Minimal Vocabulary**: Efficient 19-token representation\n\n## ๐ ๏ธ Technical Details\n### Environment\n- Python 3.11.9\n- PyTorch 2.2.1\n\n### Repository Structure\nThe code is organized into two main components:\n- `dev_folder`: Development and training code\n- `submission_folder`: Final submission code\n\n## ๐ Performance Journey\n- **Initial Stage**: 30-40% accuracy on public evaluation set with ~4 points on private test\n (Note: This early version is not in the GitHub repo, which contains only the final submission)\n- **Final Result**: 31 points achieved using ensemble approach\n\n## ๐ค Open Questions for the Community\n1. Could this transformer-based approach reach the top of the leaderboard with further optimization? (I joined in the final month and couldn't explore larger models)\n2. Is the 85% grand prize threshold achievable with this architecture?\n3. What are your thoughts on specialized transformers versus general language models for this task?\n\n## ๐ Acknowledgments\nSpecial thanks to:\n- Competition organizers\n- Mehran Kazeminia for the comprehensive previous solutions notebook\n- MindAI's interview which provided crucial insights when I was stuck\n- [michaelhodel/re-arc](https://github.com/michaelhodel/re-arc) and [xu3kev/BARC](https://github.com/xu3kev/BARC) for their datasets\n\n## ๐ Links\n- [GitHub Repository](https://github.com/zoenguyenramirez/arc-prize-2024)\n- [MindAI Interview](https://www.youtube.com/watch?v=jSAT_RuJ_Cg)\n\nFeel free to explore the code, provide feedback, or reach out with questions. Let's learn from each other and push the boundaries of what's possible with transformer architectures!", "votes": 26 }, { "id": 3046045, "postDate": "2024-11-15T04:51:02.787Z", "content": "Thanks for sharing, will study your git repo. A score of 31 form scratch is impressive.
", "rawMarkdown": "Thanks for sharing, will study your git repo. A score of 31 form scratch is impressive.", "votes": 5 }, { "id": 3045996, "postDate": "2024-11-15T03:40:05.653Z", "content": " enum Enum, auto\n\n ():\n CELL_TOKEN_SIZE = \n PAD = auto() \n START = auto() \n START_INPUT = auto() \n END_INPUT = auto() \n START_OUTPUT = auto() \n END_OUTPUT = auto() \n END = auto() \n ROW_SEPARATOR = auto() \n COUNT_OF_TOKENS = auto() \n\nVOCAB_SIZE = SpecialToken.COUNT_OF_TOKENS.value\nThe grid encoding function combines four key positional elements:
\n ():\n use_grid_encoder:\n \n torch.(x[:, :, :] >= -), \n torch.(x[:, :, :] < max_grid_size), \n\n \n row_encodings = grid_encoding[:, x[:, :, ], :]\n col_encodings = grid_encoding[:, x[:, :, ], :]\n top_encodings = grid_encoding[:, x[:, :, ], :] \n sample_encoding = grid_encoding[:, x[:, :, ], :] \n\n \n grid_encodings = torch.cat([\n row_encodings * grid_scale[],\n col_encodings * grid_scale[],\n top_encodings * grid_scale[],\n sample_encoding\n ], dim=-) \n\n grid_encodings.squeeze()\n\n positional_encoding[:, :seq_length, :]\n ():\n input_ids.dim() == \n batch_size, seq_length, *_ = input_ids.shape\n batch_size == (end_of_example_indices)\n\n \n mask = torch.triu(torch.ones((seq_length, seq_length), device=device), diagonal=).()\n mask = mask.unsqueeze().repeat(batch_size, , )\n\n \n mask_hack:\n batch_index, end_index (end_of_example_indices):\n end_index > \n mask[batch_index, :, :end_index] = \n\n mask\nNote: The mask_hack option, which allows input tokens to see all other input tokens, did not improve model performance.
The loss function only considers output tokens:
\n ():\n prefix_length = torch.tensor(end_of_examples, device=target.device) \n mask = torch.arange(seq_length, device=target.device).unsqueeze() < prefix_length.unsqueeze() - \n target[mask] = SpecialToken.PAD.value\n\n target\nThis implementation enables a decoder-only transformer architecture, eliminating the need for a separate encoder component.
", "rawMarkdown": "# Some Key Components of the Implementation\n\n## 1. Token Definitions\n```python\nfrom enum import Enum, auto\n\nclass SpecialToken(Enum):\n CELL_TOKEN_SIZE = 10 # Tokens 0-9 reserved for cell values\n PAD = auto() # Token 11: Padding token\n START = auto() # Token 12: Sequence start\n START_INPUT = auto() # Token 13: Input section start\n END_INPUT = auto() # Token 14: Input section end\n START_OUTPUT = auto() # Token 15: Output section start\n END_OUTPUT = auto() # Token 16: Output section end\n END = auto() # Token 17: Sequence end\n ROW_SEPARATOR = auto() # Token 18: Separates rows in the grid\n COUNT_OF_TOKENS = auto() # Token 19: Total token count\n\nVOCAB_SIZE = SpecialToken.COUNT_OF_TOKENS.value\n```\n\n## 2. Grid Encoding\nThe grid encoding function combines four key positional elements:\n```python\ndef combine_encoding(x: torch.Tensor, batch_size: int, use_grid_encoder: bool, \n seq_length: int, max_grid_size: int, grid_scale: torch.Tensor, \n grid_encoding: torch.Tensor, positional_encoding: torch.Tensor):\n if use_grid_encoder:\n # Validate input ranges\n assert torch.all(x[:, :, 1:] >= -1), \"Indices out of range\"\n assert torch.all(x[:, :, 1:] < max_grid_size), \"Indices out of range\"\n \n # Extract positional encodings\n row_encodings = grid_encoding[:, x[:, :, 1], :]\n col_encodings = grid_encoding[:, x[:, :, 2], :]\n top_encodings = grid_encoding[:, x[:, :, 3], :] # Topology/chart information\n sample_encoding = grid_encoding[:, x[:, :, 4], :] # Grid index\n \n # Combine encodings with scaling\n grid_encodings = torch.cat([\n row_encodings * grid_scale[0],\n col_encodings * grid_scale[1],\n top_encodings * grid_scale[2],\n sample_encoding\n ], dim=-1) # Shape: (N, seq_length, embed_size // 2)\n \n return grid_encodings.squeeze(0)\n \n return positional_encoding[:, :seq_length, :]\n```\n\n## 3. Attention Masking\n```python\ndef create_mask(input_ids, device, end_of_example_indices: List[int], mask_hack=False):\n assert input_ids.dim() == 3\n batch_size, seq_length, *_ = input_ids.shape\n assert batch_size == len(end_of_example_indices)\n \n # Create upper triangular mask\n mask = torch.triu(torch.ones((seq_length, seq_length), device=device), diagonal=1).bool()\n mask = mask.unsqueeze(0).repeat(batch_size, 1, 1)\n \n # Optional: Allow full visibility of input tokens\n if mask_hack:\n for batch_index, end_index in enumerate(end_of_example_indices):\n assert end_index > 0\n mask[batch_index, :, :end_index] = 0\n \n return mask\n```\nNote: The `mask_hack` option, which allows input tokens to see all other input tokens, did not improve model performance.\n\n## 4. Loss Masking\nThe loss function only considers output tokens:\n```python\ndef prefix_mask(target, seq_length, end_of_examples: List[int]):\n prefix_length = torch.tensor(end_of_examples, device=target.device) \n mask = torch.arange(seq_length, device=target.device).unsqueeze(0) < prefix_length.unsqueeze(1) - 1\n target[mask] = SpecialToken.PAD.value\n \n return target\n```\n\nThis implementation enables a decoder-only transformer architecture, eliminating the need for a separate encoder component.", "votes": 5 }, { "id": 3048440, "postDate": "2024-11-18T00:14:00.133Z", "content": "
This was the training trajectory, captured from the tensorboard.
\nI used 3x 4090 in the beginning and 2x 4090 since Nov 6.
", "rawMarkdown": "\n\nThis was the training trajectory, captured from the tensorboard.\n\nI used 3x 4090 in the beginning and 2x 4090 since Nov 6.", "votes": 3 }, { "id": 3048297, "postDate": "2024-11-17T18:08:35.360Z", "content": "Again, thanks for sharing and good luck for the next ARC contest! I hope the other kagglers do the same, since public scrutiny gives strength.
\nI like this original solution, and I will try to analyze if there are other variants that help improve this solution, and I will let you know. I think that to reach 85% an efficient, simple and elegant solution should be used.
\nI have seen other solutions and I donยดt believe that training a model for each problem and then assembling them is an efficient solution, because how is it possible to assemble 100 models of 100 problems. Itยดs like training 100 people to solve 100 problems, when one person could solve 70 or 80 math problems.
\nAll the best
", "rawMarkdown": "Again, thanks for sharing and good luck for the next ARC contest! I hope the other kagglers do the same, since public scrutiny gives strength.\n\nI like this original solution, and I will try to analyze if there are other variants that help improve this solution, and I will let you know. I think that to reach 85% an efficient, simple and elegant solution should be used.\n\nI have seen other solutions and I donยดt believe that training a model for each problem and then assembling them is an efficient solution, because how is it possible to assemble 100 models of 100 problems. Itยดs like training 100 people to solve 100 problems, when one person could solve 70 or 80 math problems.\n\nAll the best", "votes": 1, "replies": [ { "id": 3048334, "postDate": "2024-11-17T19:07:02.370Z", "content": "\"Regarding 'assembling them,' could you provide a reference for this approach?
\nFrom my research, I've observed two main methods:
\nhttps://www.kaggle.com/code/ironbar/single-task-test-time-fine-tuning-for-arc24
\nhttps://www.kaggle.com/competitions/arc-prize-2024/discussion/545671
\n:)
", "rawMarkdown": "https://www.kaggle.com/code/ironbar/single-task-test-time-fine-tuning-for-arc24\n\nhttps://www.kaggle.com/competitions/arc-prize-2024/discussion/545671\n\n:) \n" }, { "id": 3048442, "postDate": "2024-11-18T00:21:27.517Z", "content": "It should still be the first method I listed. It's not so much like \"training 100 people to solve 100 problems\"; rather, it's more like training one person (pre-training) who then learns 100 tasks and solves them individually. This is more similar to how humans actually work - when a human faces a challenge, they think and modify their neural connections while solving it.
", "rawMarkdown": "It should still be the first method I listed. It's not so much like \"training 100 people to solve 100 problems\"; rather, it's more like training one person (pre-training) who then learns 100 tasks and solves them individually. This is more similar to how humans actually work - when a human faces a challenge, they think and modify their neural connections while solving it." } ] } ] }, { "id": 3045985, "postDate": "2024-11-15T03:06:32.947Z", "rawMarkdown": "", "isDeleted": true } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T22:22:27Z", "competition_id": 67357, "competition_slug": "arc-prize-2024", "team_id": 12633121, "solution_url": "/competitions/arc-prize-2024/writeups/nikola-hu-sharing-my-arc-prize-2024-code-31-points" }