# LLM-Guided Evolutionary Visual Transformation (0-Score but Interesting)

Congratulations to the winners and special thanks the organizers!
Although the one that I post below is currently a zero-scored solution/approach I tend to believe the idea is not far away from the top-scoring solutions and I really enjoied the challenge as I've learned a lot from it.
[https://www.kaggle.com/code/lyubomirklyambarski/v26-script](url)
Overview
The solution build is a type of hybrid neuro-symbolic system that aims at learning to generate visual transformation programs which transform an input image into a target image.
The system combines evolutionary search with a fine-tuned LLM both for pushing the evolution in the desired direction and for inference of test cases.
System Architecture
1. Program Representation
Each program is a chain of operations (primitive transformations) applied sequentially to a grid (image).
Implemented as a ProgramNode tree structure:
op_name: the primitive operation (e.g., flip_horizontal, stack_xy, set_pixel_np).
params: operation parameters (randomized or mutated).
children: for chaining operations; instances of ProgramNode or None.
in_grid, out_grid: cache of input/output grids per node.
level: indication of the element's stage in the process
2. Primitive Operations
A dictionary PRIMITIVES maps each operation name to its implementation.
15 generic operations.
Examples:
stack_xy, rotate90, flip_horizontal, set_pixel_np, etc.
Each primitive modifies the grid in deterministic way.
Most of the primitives are applicable on either part or the whole image (using region_wraprer decorator). Region coordinates are function arguments.
3. Evolutionary Search
QD-style evolutionary search. Instead of optimizing for a single best program, the goal is to explore a wide set of diverse high-quality solutions that achieve similar goals through different strategies.
The evolution and the LLM training described below are run individually for each image index in the train set of the tasks. The inference is done on the test image input.
As initialisation, randomly generated individual programs form the population. The zero-generation is different from the following - it uses only a subset of the primitives that change the shape of the image (i.e. it aims to find the output shape). Then, the random generation focuses on single-Node programs postponing the more complex programs gradually for the next generations. Each following generation's population is built considering:
Elites - adding elites part of the previous population
Elite mutation – adding random mutations (of parameters only or on the entire operation Node) of elites
Elites extension – adding new random nodes to the elites programs
LLM proposals – a language model suggests new programs conditioned on prior training;
Random injection – occasional random programs to prevent premature convergence and/or to fill the gap to predefined population size
These candidates are then checked for novelty (i.e. the seen programs are removed). 
Then some local optimisation techniques are applited:
Try to evolve best - getting the best program and passing it through a range of random extensions/mutations to try achieve better score
Try to evolve a set of historically best elements (top elites*2 length) - this is to avoid that some solution has been skipped from evolving just because the curent best solution includes for example an individual pixel better than the one in the previous step (i.e. evolving sub-opptimal solutions)
Append good nodes on best programs - appends executable programs at the end of the current best solution (i.e. try to limit the random-next to random-executable solutions)
Append best next node - loops over all the available primitives functions and tries fixed times to append the respective function with (random) parameres so that the best score is improved. As result, it takes the operation that provides highest improvemet of the best score.
These mechanisms balance exploration (diversity through novelty and randomness) and exploitation (focused search around good solutions).
The process resembles a MAP-Elites–style algorithm, maintaining a set of programs that are both high-performing and behaviorally distinct.
The resulting top programs of each generation form a collection of programs, gradually leading to the optimal solution, later used to construct the fine-tuning dataset for the LLM.
4. LLM training
Once evolution finds solution that meets the target, the training set for the LLM proposer is constructed from the archive of the best programs.
Some tests with the trainable parameters have been done.
4.1. TinyGridCNN
Small convolutional neural network used for feature extraction. Pretrained on all ARC 2025 images based on coisine similarity of randomly augmented views of an image. The augmentations were set to random flips only. 
Outputs a dense vector representation that captures visual features. 
Models like DINO might help here.
4.2. Prompts
Each program that will participate in the LLM training dataset forms a multimodal prompt. The prompts contain:
score - the result of the fitness function for the program that forms the prompt
text - pure text "Task: transform input image to match target Reward:  {';'.join(tokens)} ", where tokens is the textual representation of the program
input_emb - visual embeddings of the input image extracted using TinyGridCNN
intermediate_embeds - visual embeddings of each intermediate image of the program extracted using TinyGridCNN
target_emb - visual embeddings of the target image extracted using TinyGridCNN
4.3. ProgramDataset
Torch dataset built from the prompts - contains all the prompt elements except score.
4.4. DataCollatorForMultimodal
Batches and pads inputs from ProgramDataset
4.5. MultimodalLLMWrapper
Processes text and image embeddings.
Contains:
base_lm - DistilGPT2 (small one to enable quick tests also on local machine - 12GB GTX3060).
Projects image embeddings into the same space as text embeddings
Concatenates input text, input, intermediate and target embeddings and feeds them to the base_lm for next-token prediction.
4.6. LLMProposer
Coordinates fine-tuning and inference.
Contains:
model - instance of MultimodalLLMWrapper (see below for details)
tokenizer
Several special tokens have been added to the model and the tokenizer, like PROG_END, STEP, ENDSTEP, END and keywords related to the primitive functions and their region representation.
The Trainer is initialised during training with:
MultimodalLLMWrapper as model
ProgramDataset as data
DataCollatorForMultimodal as collator
In inference proposes new program sequences given input embeddings.
4.7. Training
Run training (on the same dataset formed from the best programs) with low temperature, early stopping, until the model is able to generate successful program (i.e. one with maximum score; training+inference step). Once the model generates successfuly a program for that image, the training stops and the process moves to step 3 where the Evolutionary Search continue with the next image index for the current task.
5. Inference
The inference is performed after all the images in the training set of the current task have passed through the Evolutionary Search+LLM training process. Using the trained LLM in the task context, the inference is performed still in sample mode and then the final program selected is the one that is generated most frequently in the sample. The decoded program is used to generate output image and it is stored for submission. If the final program runs on the test image successfully it is accepted for submission. Tf not, the challenge is restarted for fixed number of itereations.
Observations
practically all the challenges should be solvable by programs constructed from sequences of only 2 primitives: one that finds the correct output size and one that changes an individual pixel colour
some tests on the idea above on some easy challenges shown the idea is plausible, but the resulting programs turn out to be quite long. This in turn is a challenge for the LLM training. Here, further tokenisation of frequent patterns might help (i.e. adding new primitives).
an observation of the LLM proposals' role in the evolution step is needed - it was envisaged for the next steps
the current trainied params (around 40% of the total params in DistilGPT2) are maybe too much, allowing the model to be distorted after training just some images. Gradual increasing might work better.
the approach seems too slow, although this observation is based on just 2-3 challenges passed. The idea of the setup is that the LLM accumulates knowledge and helps the evolution to solve the next challenge quicker.
This attempt was developed entirely from scratch as a way to learn and experiment with combining evolution, perception, and language models for ARC reasoning.
Feedback, ideas, and constructive criticism are very welcome!
Looking forward to the 2026 edition!