Submissions are evaluated by the **SVG Image Fidelity Score** between the given descriptions and the submitted SVG code.

For each given description in the test set, your model must produce an SVG image depicting the scene described. 

The evaluation proceeds through several stages.

First, in order to ensure the generated images adhere to the spirit of the competition, we check that the SVG code satisfies a number of constraints:
1. No SVG may more than 10,000 bytes long.
2. Each SVG may only include elements and attributes from an allowlist. Note that CSS `style` elements are not allowed.
3. No SVG may include any rasterized image data or data from external sources.

Second, we convert each SVG to a PNG image with the [`cairosvg` Python library](https://cairosvg.org/documentation/). To this PNG image we apply a number of preprocessing filters to prevent noise attacks and encourage faithfulness at the level of overall image composition.

Third, a VQA task is applied to the processed PNG with a PaliGemma model. To each description, there are four image-specific questions with either yes-or-no or multiple choice answers. The questions are meant to check how well the rendered image conforms to specific parts of the given description. See [TIFA: Text-to-Image Faithfulness Evaluation with Question Answering](https://tifa-benchmark.github.io/) for more details on the method.

Fourth, we apply OCR text detection and score adjustment. We also use PaliGemma's OCR capabilities to read any text out of a (lightly processed) PNG image. If more than four characters are detected, an exponential penalty is applied to the overall score for that image. The four "free" characters are meant to give some buffer against hallucinated text.The overall VQA score is the product of the OCR penalty and the TIFA task score.

Fourth, we compute a CLIP-based Aesthetic Score. See the [CLIP Aesthetic Predictor](https://github.com/christophschuhmann/improved-aesthetic-predictor) repo for details.

The final overall score is the harmonic mean of the VQA score and the Aesthetic score, with a \(\beta=2\) weight favoring the VQA score.

In addition, the evaluation system requires that:
1. Your model returns an SVG within 5 minutes of being passed a description.
2. All SVGs are generated in under 9 hours.

You may review the implementation of the metric here: [**SVG Image Fidelity**](https://www.kaggle.com/code/metric/svg-image-fidelity/notebook). This metric imports the `svg_constraints` package defined here: [**SVG Constraints**](https://www.kaggle.com/code/metric/svg-constraints). You may wish to use this constraints package to help ensure your submissions are valid.