{ "id": 581061, "title": "9th place - SDXL LoRA fine-tune", "content": "

My general approach was:

\n\n

Notebook: https://www.kaggle.com/code/linrock/9th-place-sdxl-lora-vqa-boost-aesthetics

\n

Example outputs:

\n

Rotations in gradient direction and variations in frame color were meant to improve aesthetic score with minimal effect on VQA.

\n

\"\"

\n

Diffusion model config:

\n\n

LoRA training:

\n

I generated 1.5k descriptions with LLMs, and images with FLUX dev and Hyper SDXL, using prompt variations to increase variety. Images were quantized to 24 colors, with median and gaussian filtering applied to remove details. Datasets were composed of the top scoring image for any given base prompt, based on paligemma2 VQA and aesthetic scores.

\n

Example training images:
\n\"\"

\n

The goal was to steer image generation towards compositions and color palettes with high scores, given generic text prompts.

\n

Trained with diffusers train_text_to_image_lora_sdxl.py with an offset noise of 0.0357 to enable high-contrast images that base SDXL is otherwise unable to generate.

\n

This approach yielded better LB scores than all combinations I tried of base models (ie. SDXL, SD 1.5, Flux), community models and LoRAs, and prompting.

\n

Basic VQA questions:

\n\n

These were used both for VQA score filtering of LoRA datasets and ranking candidate images during image prediction. In submissions, I tried creating custom-tailored VQA questions out of prompts using Qwen, but didn’t find anything better than combos of basic questions.

\n

Misc:

\n", "comments": [ { "id": 3211580, "author_name": "Ziyi777", "author_url": "", "post_date": "2025-05-28T15:13:38.270000", "content": "

Congratulations! Great work! I really like your drawings!

", "votes": 1, "replies": [] }, { "id": 3211231, "author_name": "WOOSUNG YOON", "author_url": "", "post_date": "2025-05-28T08:02:12.900000", "content": "

Congratulations.
\nThis is truly impressive. It's such a clean solution.

\n

There’s something I genuinely want to learn from you.

\n
    \n
  1. Do you approach it by creating a sequential schedule and plan?
    2. \n
  2. If the plan doesn't go as expected, how do you manage the risks?
    3. \n
\n

All Kaggle participants have their own unique strengths, and there's so much to learn from each of them.

", "votes": 2, "replies": [ { "id": 3211785, "author_name": "linrock", "author_url": "", "post_date": "2025-05-28T20:51:08.243000", "content": "

Thank you!

\n
    \n

  1. For the most part, I made lists of ideas and worked on ones I found fun, even if not always practical for the competition. I spent a while exploring diffusion models, LLMs, and fine-tuning to improve image aesthetics. If ideas didn't improve LB score within some noise range over a few submissions, I abandoned them for other ideas.

    2. \n

  2. Towards the end, I thought the top 2 teams were not using diffusion models, hacked the metrics somehow, and that their approach must be optimal for maximizing competition score. So I stopped submitting for 1.5 weeks to go a completely different direction… and did not find their clever ways of sneaking past the OCR detector. Likely I would've done better with a principled plan of improving the raster-to-SVG conversion where I had spent little time, ie. with VTracer/diffvg. It just wasn't as interesting as finding a moonshot. I took a high-risk approach that did not pay off, but it's more my style.

    3. \n
", "votes": 3, "replies": [] } ] } ], "comment_count": 3, "messages": [ { "id": 3211189, "postDate": "2025-05-28T07:05:55.067Z", "content": "

My general approach was:

\n\n

Notebook: https://www.kaggle.com/code/linrock/9th-place-sdxl-lora-vqa-boost-aesthetics

\n

Example outputs:

\n

Rotations in gradient direction and variations in frame color were meant to improve aesthetic score with minimal effect on VQA.

\n

\"\"

\n

Diffusion model config:

\n\n

LoRA training:

\n

I generated 1.5k descriptions with LLMs, and images with FLUX dev and Hyper SDXL, using prompt variations to increase variety. Images were quantized to 24 colors, with median and gaussian filtering applied to remove details. Datasets were composed of the top scoring image for any given base prompt, based on paligemma2 VQA and aesthetic scores.

\n

Example training images:
\n\"\"

\n

The goal was to steer image generation towards compositions and color palettes with high scores, given generic text prompts.

\n

Trained with diffusers train_text_to_image_lora_sdxl.py with an offset noise of 0.0357 to enable high-contrast images that base SDXL is otherwise unable to generate.

\n

This approach yielded better LB scores than all combinations I tried of base models (ie. SDXL, SD 1.5, Flux), community models and LoRAs, and prompting.

\n

Basic VQA questions:

\n\n

These were used both for VQA score filtering of LoRA datasets and ranking candidate images during image prediction. In submissions, I tried creating custom-tailored VQA questions out of prompts using Qwen, but didn’t find anything better than combos of basic questions.

\n

Misc:

\n", "rawMarkdown": "My general approach was:\n- Generate images with a diffusion model\n- Add an image frame and gradient background to increase aesthetic score\n- VQA with basic questions to choose a candidate image\n- Try slight variations to the image to boost aesthetic score further\n\nNotebook: https://www.kaggle.com/code/linrock/9th-place-sdxl-lora-vqa-boost-aesthetics\n\n### Example outputs:\nRotations in gradient direction and variations in frame color were meant to improve aesthetic score with minimal effect on VQA.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12982997%2F46827eab20190a3e01658abf3bb208b7%2FScreenshot%202025-05-28%20at%202.18.21AM.png?generation=1748415283758984&alt=media)\n\n### Diffusion model config:\n- SDXL base 1.0\n- TinyVAE to speed up image generation\n- LoRA trained on quantized-color images, filtered for high aesthetics and prompt alignment\n- sampler: DPMSolver++ Karras 2M\n- guidance scale: 5\n- num inference steps: 8\n- image size: 768x768\n\n### LoRA training:\n\nI generated 1.5k descriptions with LLMs, and images with FLUX dev and Hyper SDXL, using prompt variations to increase variety. Images were quantized to 24 colors, with median and gaussian filtering applied to remove details. Datasets were composed of the top scoring image for any given base prompt, based on paligemma2 VQA and aesthetic scores.\n\nExample training images:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12982997%2F22996a349690a24ffefa6d0d4058bae0%2FScreenshot%202025-05-28%20at%202.48.04AM.png?generation=1748414922197235&alt=media)\n\nThe goal was to steer image generation towards compositions and color palettes with high scores, given generic text prompts.\n\nTrained with diffusers `train_text_to_image_lora_sdxl.py` with an offset noise of 0.0357 to enable high-contrast images that base SDXL is otherwise unable to generate.\n\nThis approach yielded better LB scores than all combinations I tried of base models (ie. SDXL, SD 1.5, Flux), community models and LoRAs, and prompting.\n\n\n### Basic VQA questions:\n\n- Does the image show ``? yes / no\n- Are there ``? yes / no\n- Does `` exist in the image? yes / no\n\nThese were used both for VQA score filtering of LoRA datasets and ranking candidate images during image prediction. In submissions, I tried creating custom-tailored VQA questions out of prompts using Qwen, but didn’t find anything better than combos of basic questions.\n\n\n### Misc:\n\n- I used differential evolution to search for image modifications to improve aesthetic scores, and found that adding image frames on green backgrounds tended to do well\n- Other than data generation, I didn't find any useful way to involve LLMs in the image generation process\n- PNG to SVG code was incrementally improved upon Rich Olson’s notebook, and later on the OCR decoy, as it looked better than the signature I previously used\n- The highest-aesthetic score post-blur images were consistently vividly-colored landscapes:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F12982997%2F62b342873f2500dfda11ed8c83d4a8cf%2FScreenshot%202025-05-28%20at%202.18.41AM.png?generation=1748415151159269&alt=media)", "votes": 23 }, { "id": 3211580, "postDate": "2025-05-28T15:13:38.270Z", "content": "

Congratulations! Great work! I really like your drawings!

", "rawMarkdown": "Congratulations! Great work! I really like your drawings! ", "votes": 1 }, { "id": 3211231, "postDate": "2025-05-28T08:02:12.900Z", "content": "

Congratulations.
\nThis is truly impressive. It's such a clean solution.

\n

There’s something I genuinely want to learn from you.

\n
    \n
  1. Do you approach it by creating a sequential schedule and plan?
    2. \n
  2. If the plan doesn't go as expected, how do you manage the risks?
    3. \n
\n

All Kaggle participants have their own unique strengths, and there's so much to learn from each of them.

", "rawMarkdown": "Congratulations. \nThis is truly impressive. It's such a clean solution.\n\nThere’s something I genuinely want to learn from you.\n1. Do you approach it by creating a sequential schedule and plan?\n2. If the plan doesn't go as expected, how do you manage the risks?\n\nAll Kaggle participants have their own unique strengths, and there's so much to learn from each of them.", "votes": 2, "replies": [ { "id": 3211785, "postDate": "2025-05-28T20:51:08.243Z", "content": "

Thank you!

\n
    \n

  1. For the most part, I made lists of ideas and worked on ones I found fun, even if not always practical for the competition. I spent a while exploring diffusion models, LLMs, and fine-tuning to improve image aesthetics. If ideas didn't improve LB score within some noise range over a few submissions, I abandoned them for other ideas.

    2. \n

  2. Towards the end, I thought the top 2 teams were not using diffusion models, hacked the metrics somehow, and that their approach must be optimal for maximizing competition score. So I stopped submitting for 1.5 weeks to go a completely different direction… and did not find their clever ways of sneaking past the OCR detector. Likely I would've done better with a principled plan of improving the raster-to-SVG conversion where I had spent little time, ie. with VTracer/diffvg. It just wasn't as interesting as finding a moonshot. I took a high-risk approach that did not pay off, but it's more my style.

    3. \n
", "rawMarkdown": "Thank you!\n\n1. For the most part, I made lists of ideas and worked on ones I found fun, even if not always practical for the competition. I spent a while exploring diffusion models, LLMs, and fine-tuning to improve image aesthetics. If ideas didn't improve LB score within some noise range over a few submissions, I abandoned them for other ideas.\n\n2. Towards the end, I thought the top 2 teams were not using diffusion models, hacked the metrics somehow, and that their approach must be optimal for maximizing competition score. So I stopped submitting for 1.5 weeks to go a completely different direction... and did not find their clever ways of sneaking past the OCR detector. Likely I would've done better with a principled plan of improving the raster-to-SVG conversion where I had spent little time, ie. with VTracer/diffvg. It just wasn't as interesting as finding a moonshot. I took a high-risk approach that did not pay off, but it's more my style.", "votes": 3 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T09:09:14Z", "competition_id": 89659, "competition_slug": "drawing-with-llms", "team_id": 13413163, "solution_url": "/competitions/drawing-with-llms/writeups/linrock-9th-place-sdxl-lora-fine-tune" }