# 3rd place solution: VQA/AES=0.81/0.64 Diffusion model + differentiable SVG optimization

We (
[@cnumber](https://www.kaggle.com/cnumber)
and
[@daiwakun](https://www.kaggle.com/daiwakun)
) would like to thank the hosts who continually worked hard to improve the metric, making this competition interesting and meaningful. We learned a lot about image generation, which we knew little before entering this competition.
[
[Final Submission](https://www.kaggle.com/code/cnumber/fork-of-flux-baseline-75c113?scriptVersionId=242056669)
]
Description of the solution is below
Images at each step
Original Image generated by FLUX.1-schnell
![](assets/4eda83d4a70295eebd0a726a.png)
Original SVG image generated by vtracer
![](assets/9764bec534300c2cbc8722cd.png)
SVG image after iterative optimization
![](assets/135833e7d113e6c1d5360d14.png)
SVG image after iterative optimization, processed with ImageProcessor (final image used for VQA evaluation)
![](assets/f38dd41f88e415817b937efb.png)
We are excited to share our approach that secured 3rd place in the "Drawing with LLMs" competition. This competition challenged us to generate Scalable Vector Graphics (SVG) from text prompts, adhering to strict constraints on size and content, while maximizing visual fidelity and aesthetic appeal. Our solution is a multi-stage pipeline that combines the strengths of powerful generative models, vectorization tools, and differentiable graphics rendering. We’re proud to be able to present our solution that DOES NOT hack the VQA evaluator using text embedding!
Overall Pipeline
Our core idea was to first generate a high-quality bitmap image that captures the essence of the prompt, convert it into an initial SVG, and then iteratively refine this SVG to improve its faithfulness to the prompt, its aesthetic quality, and its adherence to the competition's VQA metrics, all while respecting the output constraints.
Here's a breakdown of the stages:
Bitmap Generation (Flux):
We used the
Flux
model (
flux.1-schnell
) to generate candidate bitmap images.
Prompt Engineering:
To increase diversity and quality, we employed a set of prompt templates:
'{description}'
(the raw prompt)
'Beautiful illustration of "{description}"'
'Simple svg illustration of "{description}", with colorful background, beautiful background.'
'Simple svg illustration of "{description}", with background of minimal flat vector illustration of "colorful layered big mountains"'
We generated 2 images per prompt template, resulting in 8 initial candidate bitmaps for each input description.
Initial SVG Conversion (
vtracer
):
Each generated bitmap was converted to an SVG using
vtracer
. We found
vtracer
to be effective in producing clean polygonal vector graphics.
Since vtracer doesn't provide direct control over SVG size through parameters, we implemented a binary search approach to find the optimal input PNG resolution that would result in an SVG within the size constraints.
Candidate Selection & Pre-Optimization:
The initial SVGs were rendered back to PNGs.
These PNGs were then ranked using
SigLIP
(
google/siglip-so400m-patch14-384
) to select the images most similar to the original text prompt; the same model used as the image encoder the vqa scorer PaliGemma2. We selected the top 2 candidates from the initial 8.
These top 2 (SVG, original bitmap) pairs underwent a
short optimization
(
iterations = 10
) using our
pydiffvg
-based optimizer (details below).
The candidate that achieved the highest aesthetic score after this short optimization was chosen for the final, more intensive optimization stage.
Main SVG Optimization (
pydiffvg
& Custom Losses):
This is the core of our refinement process. The selected SVG's parameters (control points of paths, fill colors, stroke colors, and stroke widths) were optimized using
pydiffvg
.
Optimizer:
Adam optimizer.
Learning Rate Schedule:
Cosine annealing with warmup.
Loss Functions:
The total loss was a weighted sum of:
Aesthetic Loss:
Negative score from the Aesthetic Predictor. We aimed to maximize this score.
SigLIP Similarity Loss:
Negative cosine similarity between the SigLIP (
google/siglip2-base-patch16-224
) embedding of the rendered SVG (after processing with a differentiable
ImageProcessor
detailed below) and the SigLIP embedding of the
target png image
.
MSE Loss:
Mean Squared Error between the rendered SVG (after processing with
ImageProcessorTorch
) and the
target bitmap image
(after processing with a reference
ImageProcessor
with the same random seed for cropping). This helps maintain structural and color fidelity to the initial high-quality bitmap`.
The optimization ran for
iterations = 200
.
ImageProcessorTorch
- Differentiable Preprocessing:
To bridge the gap between our optimization process and the competition's evaluation, we implemented
ImageProcessorTorch
. This PyTorch module applies a sequence of image transformations similar to those described in the evaluation details.
Crucially, these operations were implemented to be (mostly) differentiable or use straight-through estimators where necessary, allowing gradients to flow back to the SVG parameters.
Transformations included: random crop & resize, JPEG compression (simulated via
diff_jpeg
), median filter, FFT low-pass filter, and bilateral filter.
The rendered SVG was passed through this processor before loss calculation. The target bitmap for the MSE loss was also processed using the random crop & resize part of this module (with a synchronized random seed) to ensure fair comparison.
Final SVG Compression & Formatting:
After optimization, the SVG underwent several post-processing steps to meet the 10KB size constraint and other formatting requirements.
This involved:
Removing opacity attributes.
Converting RGB colors to hex.
Rounding numerical values.
Merging paths with the same fill color.
Optimizing path commands (e.g., using shorter relative commands, H/V lines).
A binary search-like approach was used to find the optimal rendering size for
pydiffvg
output /
vtracer
input that, after compression, would fit within the 10KB limit. If necessary, paths were aggressively pruned.
A small, consistent watermark-like signature was added to the SVGs as per the observed pattern in the solution (
<path d="M167 200h20l-20 30h20"...
).