# 5th place solution - VTracer and DiffVG optimization

I want to thank the organizers for providing such an interesting and challenging competition, especially since the metric is really tricky and so much thought had to be put into it.
My final solution is based on generating a bunch of images with SDXL-turbo, converting them to SVGs, and then adding an optimized patch to significantly improve the aesthetic score. After that, I use a proxy function for the VQA model to find the best SVG to use in the end.
Adding this to the end of your SVG file will improve the
average
aesthetic score to around
0.7
, almost independently of the background:
![Aesthetic patch - 0.7 score](assets/07a227089900d8f5c22da5e1.png)
The VQA score will suffer by adding this patch, partly because it uses around 3.5k bytes which cannot be used further in the original image conversion and also because it is a big chunk of noise in your image. However, the overall final score will be higher by quite a bit, especially when you can select which image to use in order to not hurt the VQA score too much.
Final solution
[here](https://www.kaggle.com/code/tomirol/fork-of-sdxl-turbo-vtracer-multi-gpu-10b-448-24x?scriptVersionId=242123343)
and Github with all the codebase
[here](https://github.com/matheuspf/draw)
.
Example:
![With and without aesthetic patch score comparison](assets/4d95aa4dc988ed74609da9f6.png)
Generating images
Nothing special here, I just used SDXL-turbo to generate a bunch of images. I tried multiple other models, finetuning SDXL-turbo to a preference dataset using the competition score, finetuning using REFL with
[image-reward](https://github.com/THUDM/ImageReward)
, DPO with VQA as reward, etc.
The best overall generation I could get was using SDXL-turbo with a set of prompts I selected via beam search in a generated dataset. Basically, I ran around 200 different prompts in a dataset and picked the subset that generated the best images when ensembled together. The more prompts I added, the better the results, even when using 20+ prompts.
Most prompts focus on simple, SVG-like images that can be readily converted to SVG using Vtracer. Also, explicitly adding
no text
and
no other colors
to my negative prompts consistently improved the results a little bit.
Raster to vector
Initially, I was using
[primitive](https://github.com/fogleman/primitive)
to convert the images to SVGs. I still believe the images generated by primitive can be much better than Vtracer when tuned properly, but the CPUs on Kaggle are not good enough to run it on time: it took over 30 seconds to generate 50 SVG
<paths>
for a single image using
all cores
(
primitive
is written in GO and is heavily parallelized).
On the other hand, Vtracer was able to convert an image using a
single core
in less than 2 seconds, while the results were almost as good. After adding some polygon simplification, size clipping, and tuning the parameters a bit, I was able to generate good enough SVGs with around 6k bytes per image. Finally, I added a small white OCR decoy with a black background, so it could not be missed.
Selecting the best image
After generating all the SVGs, the selection of the best one is done using a proxy function over the VQA score. I simply followed the method described in
[this paper](https://arxiv.org/abs/2404.01291)
and used the
"Yes"
vs
"No"
probability for this question:
"f<image>answer en Question: Does this image contain {description}? Answer Yes or No.\nAnswer:"
.
This actually turned out to work better than generating questions, choices, and answers using Qwen-2.5 and running the original VQA score. I believe this is because the proxy function is much more robust when we can only evaluate a few questions, but I could be wrong and my question generation model was just bad.
Although using other model versions (3b 224, 10b 224 and 3b 448) also correlated with the main metric, the original VQA model (10b 448) was always better, even if much slower and using less prompts because of that.
Optimizing the pipeline
After I noticed ensembling would be quite powerful, I had to find a way to efficiently use all Kaggle resources. I was heavily underusing GPUs because Vtracer is CPU bound, and Kaggle machines have
really
slow CPUs. For example, generating images with SDXL-turbo was actually faster than running Vtracer on a single thread.
In the end, I was able to use almost 100% of the resources by running three processes in the background:
Diffusion model on
cuda:0
Vtracer on
cpu
Proxy function on
cuda:1
and then controlling the flow within the main notebook. For example, whenever a few images are ready from SDXL, we can start converting while we keep generating more images in parallel. The same for scoring when a few SVGs are ready.
This way, I was able to run
28
different SVGs, but I believe I could push this a bit more because my solution was running in around 55 seconds per example on average.
Finding the aesthetic patch
In order to find the patch, I built a fully differentiable version of every step of the preprocessing, aesthetic model, and VQA. With this differentiable pipeline, I tried so many things:
Optimize full score directly with DiffVG (SVG -> score directly)
Add text to the image and
reduce
the VQA OCR score in order to
hide
it
Find a small subset of paths which will optimize the aesthetic score alongside a dataset
Optimize a proxy function (yes probability) with the VQA model to find the best SVG to use
Plug the full function (VQA + aesthetic) into a method like
[ReNO](https://arxiv.org/abs/2406.04312)
to optimize images from SDXL-turbo directly
Directly finetune a Diffusion model using a similar method to
[image-reward](https://github.com/THUDM/ImageReward)
but with the differentiable VQA loss
All of those solutions
do work
, but they are either too slow, require too much GPU memory to run on Kaggle, generate SVGs way over 10kb, or are just too unstable to make sure they will work every time. In the end, I tried to do something similar to what other people did in optimizing directly over the generated SVG, but could not get a proxy function over the VQA score to work well unfortunately =/
The best solution I could find was to just search a small subset of paths that, when added over the background of the actual scene, will improve the average aesthetic score in a dataset. Due to all the preprocessing and especially cropping, the aesthetic score varies quite a bit from image to image, but the average over a significant amount of images will improve measurably. Also, because of the cropping, the position of the patch cannot be so close to the edges; otherwise, the score will be much lower.
Because the patch uses a lot of bytes, I had to balance patch size and VQA score. The bigger the patch, the less information VTracer could use to convert the SVG. I was able to get patches with over
0.8
aesthetic score by using much more bytes, but then the VQA hit would be too high.
My final patch has around 100x100 pixels and was found using a simple gradient ascent over the aesthetic score on a small dataset of 150 images. I selected a bunch of background from a previous submission, started the SVG from scratch in a grid like pattern of 96x96 pixels, and run gradient ascent. This
[notebook](https://www.kaggle.com/code/tomirol/sdxl-turbo-vtracer-aest-opt?scriptVersionId=242366685)
shows the process, it took a few hours to find a good aesthetic patch and it is extremely sensitive on where in the image it is.
Bonus
I think the nicest thing I tried was to plug the differentiable pipeline into
[ReNO](https://arxiv.org/abs/2406.04312)
and optimize the diffusion process directly against the full loss. It generated some really nice images and was actually significantly better than a single run of SDXL-turbo, but it was very inefficient and did not work as well as the simple ensembling I did in the end. Also, I could only use the 3B-224 version of the VQA, since the bigger one would not run with gradients enabled on a T4 with 16GB of memory, so the loss did not perfectly correlate with the competition score.
[This notebook](https://www.kaggle.com/code/tomirol/fork-of-sdxl-turbo-reno-vqa?scriptVersionId=242363815)
shows the improvement from the first generated image from SDXL and the images in between, optimizing the metric alongside the way on the diffusion process itself.