{ "id": 545917, "title": "Enhance 400k BARC models with 850k VLM", "content": "

Enhance Abstract Reasoning Corpus representation with Vision Language Model

\n

In this solution, I explore enhanced representations of 2D puzzles in the ARC-AGI Challenge.

\n

Assumptions:

\n\n

Based on these assumptions, I propose two approaches to build a better perception model:

\n\n

Motivation

\n

The importance of object-based representation is well explained in [1]. However, rather than utilizing the object-centric framework (ARGA) [2], I decided to fine-tune VLM on synthetic data.

\n

Synthetic data

\n

Implementing a single 2D grid logic is much easier than implementing an entire puzzle with multiple grid pairs. I selected several tasks from both training and evaluation sets that could be easily represented using simple objects. Ultimately, I chose 42 tasks from the training set and 31 tasks from the evaluation set. It is worth noting that puzzles may use similar objects, making them difficult to distinguish from one another. For each of these grids, I generated approximately 11,000 random samples, resulting in roughly 850,000 training samples in total.

\n

Objects

\n

The following object was chosen to represent a 2D grid:

\n\n

All elements (Lines, Boxes, Frames, and Polygons) can be represented with single or multiple colors.

\n

VLM

\n

For this competition, I selected the Qwen2-VL-2B-Instruct model and modified its original architecture. I replaced the visual patch embedding matrix with 11×1280 embeddings, where the 11-dimensional covers 10 distinct colors plus one padding value.

\n
 ():\n     ():\n        ().__init__(config)\n        .patch_embed = nn.Embedding(, config.embed_dim)\n
\n

Additionally, I rearranged the 2D grid input to align with the 2D Rotary Position Embeddings.

\n

Chat template

\n

I implemented a multi-turn dialogue schema. The initial idea was to use VLM in interactive mode. For example, ask VLM to represent grid with different set of objects.

\n
<|im_start|>system\nYou are an expert in the Abstraction and Reasoning Corpus.<|im_end|>\n<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>\n{pixels}\nImage(height=10, width=10)<|im_end|>\n<|im_start|>assistant\ngrid,box,frame<|im_end|>\n<|im_start|>user\nDescribe grid<|im_end|>\n<|im_start|>assistant\n<|object_ref_start|>grid<|object_ref_end|> = Grid(color=0)<|im_end|>\n<|im_start|>user\nDescribe box<|im_end|>\n<|im_start|>assistant\n<|object_ref_start|>box0<|object_ref_end|> = Box(color=1, vertices=[<|box_start|>(1,1),(3,4)<|box_end|>])\n<|object_ref_start|>box1<|object_ref_end|> = Box(color=[2,3], vertices=[<|box_start|>(2,1),(5,4)<|box_end|>])<|im_end|>\n<|im_start|>user\nDescribe frame<|im_end|>\n<|im_start|>assistant\n<|object_ref_start|>frame0<|object_ref_end|> = Frame(color=1, vertices=[<|box_start|>(3,7),(7,3)<|box_end|>])<|im_end|>\n<|im_start|>user\nReorder box0,box1,frame0<|im_end|>\n<|im_start|>assistant\n# Layer 0\nbox1\nframe0\n# Layer 1\nbox0<|im_end|>\n
\n

In addition to feeding the raw grid, I use a simple text representation:

\n
pixels = .join([.join((, r))  r  image])\n
\n

As demonstrated, I also use special Qwen tokens <|object_ref_start|>, <|object_ref_end|>, <|box_start|>, and <|box_end|> to enhance the representation.

\n

Perception result

\n

I evaluated the VLM using a reconstruction accuracy, which measures the number of grids that can be successfully reconstructed based on recognized objects.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
modeldevConceptARCtrainingevaluation
VLM97.69%75.49%63.05%54.62%
\n

After independently recognizing each grid, I combine the recognitions from one puzzle and convert them into a single Python program.

\n

Puzzle solver

\n

To evaluate the VLM's performance in puzzle-solving scenarios, I fine-tuned induction and transduction models using the recently released BARC dataset with 400,000 puzzles [3]. Each puzzle was processed through the VLM to generate corresponding Python representation programs. After cleaning and filtering the results, I obtained a dataset of 350,000 puzzles for training and 8,000 puzzles for development purposes. The dataset was split according to different skills/concepts.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
typemodeltraining sizesamples per taskbeam widthevaluation accuracy
induction(BARC) Llama3.1-8B-instruct100k2048-18.78%
transduction(BARC) Llama3.1-8B-instruct100k-2015.25%
induction(BARC) Llama3.1-8B-instruct200k10k-30.50%
induction(BARC) Llama3.1-8B-instruct400k20k-38.00%
inductionQwen2-VL-2B-Instruct + Llama-3.2-3B350k3k-29.00%*
transductionQwen2-VL-2B-Instruct + Llama-3.2-3B350k100-7.25%
\n\n

By combining several models together, I achieved a score of 8 points on the hidden test set here. When merging this with the publicly available notebook that scored 26 points, I managed to achieve a score of 27 here.

\n

Conclusion

\n

Several experiments need to be conducted to properly compare this approach with alternative solutions. Nevertheless, even these preliminary results demonstrate promising potential for ARC-AGI object-based representation using VLM. Specifically, using the proposed representation, a 3.2B model with just 3k samples per task achieved 29% accuracy on the evaluation test set. It is worth noting that only 73 seed grids were utilized for the synthetic dataset. Based on these findings, I believe that leveraging VLM and additional synthetic data is a promising direction for developing an effective grid representation approach for 2D ARC-AGI puzzles.

\n

[1] https://arxiv.org/pdf/2305.18354
\nLLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations

\n

[2] https://ssanner.github.io/papers/aaai23_arga.pdf
\nGraphs, Constraints, and Search for the Abstraction and Reasoning Corpus

\n

[3] https://www.cs.cornell.edu/~ellisk/documents/arc_induction_vs_transduction.pdf
\nCOMBINING INDUCTION AND TRANSDUCTION FOR ABSTRACT REASONING

", "comments": [], "comment_count": 0, "messages": [ { "id": 3043857, "postDate": "2024-11-12T18:52:31.493Z", "content": "

Enhance Abstract Reasoning Corpus representation with Vision Language Model

\n

In this solution, I explore enhanced representations of 2D puzzles in the ARC-AGI Challenge.

\n

Assumptions:

\n\n

Based on these assumptions, I propose two approaches to build a better perception model:

\n\n

Motivation

\n

The importance of object-based representation is well explained in [1]. However, rather than utilizing the object-centric framework (ARGA) [2], I decided to fine-tune VLM on synthetic data.

\n

Synthetic data

\n

Implementing a single 2D grid logic is much easier than implementing an entire puzzle with multiple grid pairs. I selected several tasks from both training and evaluation sets that could be easily represented using simple objects. Ultimately, I chose 42 tasks from the training set and 31 tasks from the evaluation set. It is worth noting that puzzles may use similar objects, making them difficult to distinguish from one another. For each of these grids, I generated approximately 11,000 random samples, resulting in roughly 850,000 training samples in total.

\n

Objects

\n

The following object was chosen to represent a 2D grid:

\n\n

All elements (Lines, Boxes, Frames, and Polygons) can be represented with single or multiple colors.

\n

VLM

\n

For this competition, I selected the Qwen2-VL-2B-Instruct model and modified its original architecture. I replaced the visual patch embedding matrix with 11×1280 embeddings, where the 11-dimensional covers 10 distinct colors plus one padding value.

\n
 ():\n     ():\n        ().__init__(config)\n        .patch_embed = nn.Embedding(, config.embed_dim)\n
\n

Additionally, I rearranged the 2D grid input to align with the 2D Rotary Position Embeddings.

\n

Chat template

\n

I implemented a multi-turn dialogue schema. The initial idea was to use VLM in interactive mode. For example, ask VLM to represent grid with different set of objects.

\n
<|im_start|>system\nYou are an expert in the Abstraction and Reasoning Corpus.<|im_end|>\n<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>\n{pixels}\nImage(height=10, width=10)<|im_end|>\n<|im_start|>assistant\ngrid,box,frame<|im_end|>\n<|im_start|>user\nDescribe grid<|im_end|>\n<|im_start|>assistant\n<|object_ref_start|>grid<|object_ref_end|> = Grid(color=0)<|im_end|>\n<|im_start|>user\nDescribe box<|im_end|>\n<|im_start|>assistant\n<|object_ref_start|>box0<|object_ref_end|> = Box(color=1, vertices=[<|box_start|>(1,1),(3,4)<|box_end|>])\n<|object_ref_start|>box1<|object_ref_end|> = Box(color=[2,3], vertices=[<|box_start|>(2,1),(5,4)<|box_end|>])<|im_end|>\n<|im_start|>user\nDescribe frame<|im_end|>\n<|im_start|>assistant\n<|object_ref_start|>frame0<|object_ref_end|> = Frame(color=1, vertices=[<|box_start|>(3,7),(7,3)<|box_end|>])<|im_end|>\n<|im_start|>user\nReorder box0,box1,frame0<|im_end|>\n<|im_start|>assistant\n# Layer 0\nbox1\nframe0\n# Layer 1\nbox0<|im_end|>\n
\n

In addition to feeding the raw grid, I use a simple text representation:

\n
pixels = .join([.join((, r))  r  image])\n
\n

As demonstrated, I also use special Qwen tokens <|object_ref_start|>, <|object_ref_end|>, <|box_start|>, and <|box_end|> to enhance the representation.

\n

Perception result

\n

I evaluated the VLM using a reconstruction accuracy, which measures the number of grids that can be successfully reconstructed based on recognized objects.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
modeldevConceptARCtrainingevaluation
VLM97.69%75.49%63.05%54.62%
\n

After independently recognizing each grid, I combine the recognitions from one puzzle and convert them into a single Python program.

\n

Puzzle solver

\n

To evaluate the VLM's performance in puzzle-solving scenarios, I fine-tuned induction and transduction models using the recently released BARC dataset with 400,000 puzzles [3]. Each puzzle was processed through the VLM to generate corresponding Python representation programs. After cleaning and filtering the results, I obtained a dataset of 350,000 puzzles for training and 8,000 puzzles for development purposes. The dataset was split according to different skills/concepts.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
typemodeltraining sizesamples per taskbeam widthevaluation accuracy
induction(BARC) Llama3.1-8B-instruct100k2048-18.78%
transduction(BARC) Llama3.1-8B-instruct100k-2015.25%
induction(BARC) Llama3.1-8B-instruct200k10k-30.50%
induction(BARC) Llama3.1-8B-instruct400k20k-38.00%
inductionQwen2-VL-2B-Instruct + Llama-3.2-3B350k3k-29.00%*
transductionQwen2-VL-2B-Instruct + Llama-3.2-3B350k100-7.25%
\n\n

By combining several models together, I achieved a score of 8 points on the hidden test set here. When merging this with the publicly available notebook that scored 26 points, I managed to achieve a score of 27 here.

\n

Conclusion

\n

Several experiments need to be conducted to properly compare this approach with alternative solutions. Nevertheless, even these preliminary results demonstrate promising potential for ARC-AGI object-based representation using VLM. Specifically, using the proposed representation, a 3.2B model with just 3k samples per task achieved 29% accuracy on the evaluation test set. It is worth noting that only 73 seed grids were utilized for the synthetic dataset. Based on these findings, I believe that leveraging VLM and additional synthetic data is a promising direction for developing an effective grid representation approach for 2D ARC-AGI puzzles.

\n

[1] https://arxiv.org/pdf/2305.18354
\nLLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations

\n

[2] https://ssanner.github.io/papers/aaai23_arga.pdf
\nGraphs, Constraints, and Search for the Abstraction and Reasoning Corpus

\n

[3] https://www.cs.cornell.edu/~ellisk/documents/arc_induction_vs_transduction.pdf
\nCOMBINING INDUCTION AND TRANSDUCTION FOR ABSTRACT REASONING

", "rawMarkdown": "# Enhance Abstract Reasoning Corpus representation with Vision Language Model\n\nIn this solution, I explore enhanced representations of 2D puzzles in the ARC-AGI Challenge.\n\nAssumptions:\n- Each grid can be represented using simple objects (lines, squares, etc.)\n- Most puzzles have only one way to represent the grid\n\nBased on these assumptions, I propose two approaches to build a better perception model:\n\n- Generate synthetic grids independently from puzzle logic\n- Utilize Vision Language Model (VLM) to convert 2D grid into simple Python code\n\n## Motivation\n\nThe importance of object-based representation is well explained in [1]. However, rather than utilizing the object-centric framework (ARGA) [2], I decided to fine-tune VLM on synthetic data.\n\n## Synthetic data\n\nImplementing a single 2D grid logic is much easier than implementing an entire puzzle with multiple grid pairs. I selected several tasks from both training and evaluation sets that could be easily represented using simple objects. Ultimately, I chose 42 tasks from the training set and 31 tasks from the evaluation set. It is worth noting that puzzles may use similar objects, making them difficult to distinguish from one another. For each of these grids, I generated approximately 11,000 random samples, resulting in roughly 850,000 training samples in total.\n\n### Objects\n\nThe following object was chosen to represent a 2D grid:\n\n- **Grid**: A container for other objects\n- **Pixel**: The smallest discrete unit of the grid\n- **Line**: I only implement Vertical and Horizontal lines\n- **Box**: A solid rectangular shape\n- **Frame**: A rectangular border structure with an empty interior\n- **Polygon**: Any irregular geometric shape not classified above\n- **Noise**: Random, unstructured patterns (Note: Future implementations may include additional structured patterns such as mosaics and line lattices)\n\nAll elements (Lines, Boxes, Frames, and Polygons) can be represented with single or multiple colors.\n\n## VLM\n\nFor this competition, I selected the [Qwen2-VL-2B-Instruct](https://huggingface.co/Qwen/Qwen2-VL-2B-Instruct) model and modified its original architecture. I replaced the visual patch embedding matrix with `11×1280` embeddings, where the 11-dimensional covers 10 distinct colors plus one padding value.\n\n```python\nclass Qwen2ARCTransformerPretrainedModel(Qwen2VisionTransformerPretrainedModel):\n def __init__(self, config):\n super().__init__(config)\n self.patch_embed = nn.Embedding(11, config.embed_dim)\n```\n\nAdditionally, I rearranged the 2D grid input to align with the 2D Rotary Position Embeddings.\n\n### Chat template\n\nI implemented a multi-turn dialogue schema. The initial idea was to use VLM in interactive mode. For example, ask VLM to represent grid with different set of objects.\n\n```xml\n<|im_start|>system\nYou are an expert in the Abstraction and Reasoning Corpus.<|im_end|>\n<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>\n{pixels}\nImage(height=10, width=10)<|im_end|>\n<|im_start|>assistant\ngrid,box,frame<|im_end|>\n<|im_start|>user\nDescribe grid<|im_end|>\n<|im_start|>assistant\n<|object_ref_start|>grid<|object_ref_end|> = Grid(color=0)<|im_end|>\n<|im_start|>user\nDescribe box<|im_end|>\n<|im_start|>assistant\n<|object_ref_start|>box0<|object_ref_end|> = Box(color=1, vertices=[<|box_start|>(1,1),(3,4)<|box_end|>])\n<|object_ref_start|>box1<|object_ref_end|> = Box(color=[2,3], vertices=[<|box_start|>(2,1),(5,4)<|box_end|>])<|im_end|>\n<|im_start|>user\nDescribe frame<|im_end|>\n<|im_start|>assistant\n<|object_ref_start|>frame0<|object_ref_end|> = Frame(color=1, vertices=[<|box_start|>(3,7),(7,3)<|box_end|>])<|im_end|>\n<|im_start|>user\nReorder box0,box1,frame0<|im_end|>\n<|im_start|>assistant\n# Layer 0\nbox1\nframe0\n# Layer 1\nbox0<|im_end|>\n```\n\nIn addition to feeding the raw grid, I use a simple text representation:\n```python\npixels = \"\\n\".join([\" \".join(map(str, r)) for r in image])\n```\n\nAs demonstrated, I also use special `Qwen` tokens `<|object_ref_start|>`, `<|object_ref_end|>`, `<|box_start|>`, and `<|box_end|>` to enhance the representation.\n\n### Perception result\n\nI evaluated the VLM using a `reconstruction accuracy`, which measures the number of grids that can be successfully reconstructed based on recognized objects.\n\n|model|dev|ConceptARC|training|evaluation|\n|-----|---|--------|----------|----------|\n|VLM|97.69%|75.49%|63.05%|54.62%|\n\nAfter independently recognizing each grid, I combine the recognitions from one puzzle and convert them into a single Python program.\n\n## Puzzle solver\n\nTo evaluate the VLM's performance in puzzle-solving scenarios, I fine-tuned induction and transduction models using the recently released BARC dataset with 400,000 puzzles [3]. Each puzzle was processed through the VLM to generate corresponding Python representation programs. After cleaning and filtering the results, I obtained a dataset of 350,000 puzzles for training and 8,000 puzzles for development purposes. The dataset was split according to different skills/concepts.\n\n|type|model|training size|samples per task|beam width|evaluation accuracy|\n|-----|-----|-------------|----------------|----------|----------|\n| induction | (BARC) Llama3.1-8B-instruct | 100k | 2048 | - | 18.78% |\n| transduction | (BARC) Llama3.1-8B-instruct | 100k | - | 20 | 15.25% |\n| induction | (BARC) Llama3.1-8B-instruct | 200k | 10k | - | 30.50% |\n| induction | (BARC) Llama3.1-8B-instruct | 400k | 20k | - | 38.00% |\n| induction | Qwen2-VL-2B-Instruct + Llama-3.2-3B | 350k | 3k | - | 29.00%* |\n| transduction | Qwen2-VL-2B-Instruct + Llama-3.2-3B | 350k | 100 | - | 7.25% |\n\n* I checked correctly recognized puzzles and found that only two of them intersected with the manually labeled grids used in the synthetic data generation stage.\n\nBy combining several models together, I achieved a score of 8 points on the hidden test set [here](https://www.kaggle.com/code/sorokin/barc-vlm-itlms?scriptVersionId=206209197). When merging this with the publicly available notebook that scored 26 points, I managed to achieve a score of 27 [here](https://www.kaggle.com/code/sorokin/arc-ensemble-v3?scriptVersionId=206314503).\n\n## Conclusion\n\nSeveral experiments need to be conducted to properly compare this approach with alternative solutions. Nevertheless, even these preliminary results demonstrate promising potential for ARC-AGI object-based representation using VLM. Specifically, using the proposed representation, a 3.2B model with just 3k samples per task achieved 29% accuracy on the evaluation test set. It is worth noting that only 73 seed grids were utilized for the synthetic dataset. Based on these findings, I believe that leveraging VLM and additional synthetic data is a promising direction for developing an effective grid representation approach for 2D ARC-AGI puzzles.\n\n\n[1] https://arxiv.org/pdf/2305.18354\nLLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations\n\n[2] https://ssanner.github.io/papers/aaai23_arga.pdf\nGraphs, Constraints, and Search for the Abstraction and Reasoning Corpus\n\n[3] https://www.cs.cornell.edu/~ellisk/documents/arc_induction_vs_transduction.pdf\nCOMBINING INDUCTION AND TRANSDUCTION FOR ABSTRACT REASONING", "votes": 13 } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T22:22:09Z", "competition_id": 67357, "competition_slug": "arc-prize-2024", "team_id": 12809968, "solution_url": "/competitions/arc-prize-2024/writeups/ivan-enhance-400k-barc-models-with-850k-vlm" }