{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.17","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"tpu1vmV38","dataSources":[{"sourceId":21755,"databundleVersionId":1475600,"sourceType":"competition"}],"dockerImageVersionId":31040,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# GAN Art: Monet\n\nThe goal of this project is to build a model that can generate Monet-style images, evaluated using the Memorization-informed Fréchet Inception Distance (MiFID).\n\nThis project was developed as a submission to the [Kaggle competition](https://www.kaggle.com/competitions/gan-getting-started).","metadata":{}},{"cell_type":"markdown","source":"## Setup","metadata":{}},{"cell_type":"code","source":"import os\nimport matplotlib.pyplot as plt\nimport tensorflow as tf\nimport numpy as np\n\nfrom collections import Counter\nfrom sklearn.cluster import KMeans\nfrom tensorflow.keras import layers, Model\n\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='local')\n    print('Device:', tpu.master())\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nexcept:\n    strategy = tf.distribute.get_strategy()\n    \nprint('Number of replicas:', strategy.num_replicas_in_sync)\nAUTOTUNE = tf.data.experimental.AUTOTUNE\nprint(tf.__version__)\n\nDATA_PATH = \"/kaggle/input/gan-getting-started\"\nmonet_dir = os.path.join(DATA_PATH, \"monet_tfrec\")\nphoto_dir = os.path.join(DATA_PATH, \"photo_tfrec\")","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Problem Analysis\n\nThe dataset contains four directories: `monet_tfrec`, `photo_tfrec`, `monet_jpg`, and `photo_jpg`. The Monet directories contain Monet paintings, while the photo directories contain photographs. We are instructed to apply Monet-style transformations to the photos and submit the generated JPEG images as a ZIP file.\n\nThe submission is a ZIP file containing 7,000 to 10,000 images sized `256 x 256`. The images are evaluated using MiFID (Memorization-informed Fréchet Inception Distance), a modification of the [Fréchet Inception Distance (FID)](https://github.com/bioinf-jku/TTUR). A lower MiFID score indicates greater similarity between the generated images and real Monet paintings.\n\nTo accomplish this, we will build and iterate on a Generative Adversarial Network (GAN), which uses two deep learning models—a generator and a discriminator—to generate new images within the distribution of the training data.","metadata":{}},{"cell_type":"markdown","source":"## EDA","metadata":{}},{"cell_type":"markdown","source":"### Dataset Size and Balance","metadata":{}},{"cell_type":"code","source":"def sample_image_info(tfrecord_dir):\n    tfrec_files = [os.path.join(tfrecord_dir, f) for f in os.listdir(tfrecord_dir) if f.endswith(\".tfrec\")]\n    sizes = []\n    dimensions = None\n\n    for raw in tf.data.TFRecordDataset(tfrec_files):\n        ex = tf.io.parse_single_example(raw, {\n            'image': tf.io.FixedLenFeature([], tf.string),\n        })\n        img_bytes = ex['image'].numpy()\n        sizes.append(len(img_bytes) / 1024)  # KB\n\n        if dimensions is None:\n            img = tf.image.decode_image(img_bytes, channels=3)\n            dimensions = (img.shape[1], img.shape[0])  # width, height\n\n    return {\n        \"count\": len(sizes),\n        \"dimensions\": dimensions,\n        \"avg_kb\": np.mean(sizes),\n        \"median_kb\": np.median(sizes)\n    }\n\nmonet_info = sample_image_info(monet_dir)\nphoto_info = sample_image_info(photo_dir)\n\nfor name, info in [('monet_tfrec', monet_info), ('photo_tfrec', photo_info)]:\n    print(f\"{name}:\")\n    print(f\"  Dimensions: {info['dimensions'][0]}×{info['dimensions'][1]}\")\n    print(f\"  Avg size:   {info['avg_kb']:.1f} KB\")\n    print(f\"  Median size:{info['median_kb']:.1f} KB\\n\")\n\nlabels = ['monet_tfrec', 'photo_tfrec']\ncounts = [monet_info['count'], photo_info['count']]\nplt.bar(labels, counts, color=['blue', 'green'])\nplt.ylabel('Number of Images')\nplt.title('Image Counts')\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We can see that this dataset is very clean and sets us up for success.\n\n- All images in both datasets are 256×256 pixels.\n- Both datasets have small, consistent file sizes and are in the optimized TFRecord format.\n\nHowever, there's a significant difference in sample quantity. This imbalance is the main challenge and may require data augmentation or oversampling of the Monet set to prevent overfitting or training instability.","metadata":{}},{"cell_type":"markdown","source":"### Inspect Sample Images","metadata":{}},{"cell_type":"code","source":"def load_images(tfrecord_dir, n=3):\n    files = [os.path.join(tfrecord_dir, f) for f in os.listdir(tfrecord_dir) if f.endswith('.tfrec')]\n    ds = tf.data.TFRecordDataset(files).map(\n        lambda x: tf.io.parse_single_example(x, {'image': tf.io.FixedLenFeature([], tf.string)})\n    )\n    return [tf.image.decode_image(ex['image'].numpy(), channels=3) for ex in ds.take(n)]\n\nmonet_imgs = load_images(monet_dir)\nphoto_imgs = load_images(photo_dir)\n\nfig, axes = plt.subplots(2, 3, figsize=(9, 6))\nfor i in range(3):\n    axes[0, i].imshow(monet_imgs[i].numpy())\n    axes[1, i].imshow(photo_imgs[i].numpy())\n    axes[0, i].set_title(f\"Monet {i+1}\")\n    axes[1, i].set_title(f\"Photo {i+1}\")\n    for ax in [axes[0, i], axes[1, i]]:\n        ax.axis('off')\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Visually inspecting a sample of images, we can identify several potential challenges:\n\n- There is a significant stylistic gap between the photos and Monet paintings. The GAN will need to learn a substantial transformation to bridge this difference.\n- The content of both datasets is diverse. While this supports generalization, it may be difficult to find 1:1 matches in the Monet dataset. For example, Photo 2 above likely has no direct Monet equivalent.\n- Watermarks and text in photos will need to be removed when generating Monet-style images.","metadata":{}},{"cell_type":"markdown","source":"### Color Space Analysis","metadata":{}},{"cell_type":"code","source":"monet_imgs = load_images(monet_dir, n=100)\nphoto_imgs = load_images(photo_dir, n=100)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### Color Palette","metadata":{}},{"cell_type":"code","source":"def extract_palette(images, k=5, title=''):\n    pixels = tf.concat([tf.reshape(img, [-1, 3]) for img in images], axis=0).numpy()\n    kmeans = KMeans(n_clusters=k, random_state=0, n_init=10).fit(pixels)\n    colors = kmeans.cluster_centers_.astype(int)\n\n    plt.figure(figsize=(k, 1))\n    for i, color in enumerate(colors):\n        plt.subplot(1, k, i + 1)\n        plt.imshow(np.ones((10, 10, 3), dtype=np.uint8) * color)\n        plt.axis('off')\n    plt.suptitle(f\"{title} Palette\")\n    plt.tight_layout()\n    plt.show()\n\nextract_palette(monet_imgs, k=5, title='Monet')\nextract_palette(photo_imgs, k=5, title='Photo')","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### Color Stats","metadata":{}},{"cell_type":"code","source":"def compute_stats(images):\n    pixels = tf.concat([tf.reshape(img, [-1, 3]) for img in images], axis=0).numpy()\n    return {\n        'mean': np.mean(pixels, axis=0),\n        'median': np.median(pixels, axis=0)\n    }\n\nfor name, imgs in [('Monet', monet_imgs), ('Photo', photo_imgs)]:\n    stats = compute_stats(imgs)\n    print(f\"{name} Mean RGB:   {stats['mean'].astype(int)}\")\n    print(f\"{name} Median RGB: {stats['median'].astype(int)}\")\n\nfig, axes = plt.subplots(1, 2, figsize=(12, 4))\nfor ax, images, title in zip(axes, [monet_imgs, photo_imgs], ['Monet', 'Photo']):\n    pixels = tf.concat([tf.reshape(img, [-1, 3]) for img in images], axis=0).numpy() / 255.0\n    for i, c in enumerate(['r', 'g', 'b']):\n        ax.hist(pixels[:, i], bins=50, alpha=0.6, color=c, label=c.upper())\n    ax.set_title(f'{title} - RGB Histogram')\n    ax.legend()\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The primary challenge will be shifting the extreme dark and light color values in real photos to a more neutral and artistically balanced color range—avoiding extremes and focusing on brighter tones and hues. This will likely be difficult for our model, as these transformations are non-linear, resembling texture and atmospheric changes. The model will need to learn subtle variations in lighting and atmosphere.\n\nTo address these challenges, we can consider adding additional loss components to our generative model that specifically target lighting, color, and atmospheric transformations.","metadata":{}},{"cell_type":"markdown","source":"### Summary\n\nOur EDA identified several significant challenges:\n\n- The dataset is imbalanced, with significantly more photos than Monet examples.\n- The color space differs greatly between photos and Monet paintings. Our model will need to learn nuanced and sophisticated lighting and atmospheric transformations to capture the variety of Monet’s style and apply it in novel situations (like \"Photo 2\" in the visualization section).\n- Some photo examples contain watermarks and text that will need to be removed.","metadata":{}},{"cell_type":"markdown","source":"## Model Architecture","metadata":{}},{"cell_type":"markdown","source":"### Baseline Model and Validation Pipeline\n\nOur input pipeline will need to:\n\n- Parse and decode images from the TFRecord format.\n- Normalize pixel values from `[0, 255]` to `[-1, 1]`, which is crucial for the generator’s output layer using a `tanh` activation.\n- Batch record loading to maximize training throughput on the system.","metadata":{}},{"cell_type":"code","source":"def parse_image(example_proto):\n    features = {\n        'image': tf.io.FixedLenFeature([], tf.string)\n    }\n    parsed = tf.io.parse_single_example(example_proto, features)\n    img = tf.image.decode_image(parsed['image'], channels=3)\n    img = tf.image.convert_image_dtype(img, tf.float32)\n    img = (img * 2.0) - 1.0\n    img.set_shape([None, None, 3])\n    return img\n\ndef load_dataset(tfrecord_dir, batch_size=32, shuffle=True, buffer_size=1000):\n    files = tf.io.gfile.glob(os.path.join(tfrecord_dir, '*.tfrec'))\n    ds = tf.data.TFRecordDataset(files, num_parallel_reads=tf.data.AUTOTUNE)\n    ds = ds.map(parse_image, num_parallel_calls=tf.data.AUTOTUNE)\n    if shuffle:\n        ds = ds.shuffle(buffer_size)\n    ds = ds.batch(batch_size).prefetch(tf.data.AUTOTUNE)\n    return ds\n\nmonet_ds = load_dataset(monet_dir, batch_size=16)\nphoto_ds = load_dataset(photo_dir, batch_size=16)\n\nexample_monet = next(iter(monet_ds))\nexample_photo = next(iter(photo_ds))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### Generator\n\nWe will build a `U-Net` style generator. It will use `downsample` as the encoder and `upsample` as the decoder. Notable features include layer normalization to stabilize the GAN and Leaky ReLU to keep all neurons active.","metadata":{}},{"cell_type":"code","source":"def downsample(filters, size, apply_instancenorm=True):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    result = tf.keras.Sequential()\n    result.add(layers.Conv2D(filters, size, strides=2, padding='same',\n                             kernel_initializer=initializer, use_bias=False))\n    if apply_instancenorm:\n        result.add(layers.LayerNormalization())\n    result.add(layers.LeakyReLU())\n    return result\n\ndef upsample(filters, size, apply_dropout=False):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    result = tf.keras.Sequential()\n    result.add(layers.Conv2DTranspose(filters, size, strides=2, padding='same',\n                                      kernel_initializer=initializer, use_bias=False))\n    result.add(layers.LayerNormalization())\n    if apply_dropout:\n        result.add(layers.Dropout(0.5))\n    result.add(layers.ReLU())\n    return result\n\ndef build_generator():\n    inputs = layers.Input(shape=[256, 256, 3])\n    \n    down_stack = [\n        downsample(64, 4, apply_instancenorm=False),\n        downsample(128, 4),\n        downsample(256, 4),\n        downsample(512, 4)\n    ]\n\n    up_stack = [\n        upsample(256, 4, apply_dropout=True),\n        upsample(128, 4, apply_dropout=True),\n        upsample(64, 4),\n    ]\n\n    initializer = tf.random_normal_initializer(0., 0.02)\n    last = layers.Conv2DTranspose(3, 4, strides=2,\n                                  padding='same', kernel_initializer=initializer,\n                                  activation='tanh')\n\n    x = inputs\n    skips = []\n    for down in down_stack:\n        x = down(x)\n        skips.append(x)\n    skips = reversed(skips[:-1])\n    \n    for up, skip in zip(up_stack, skips):\n        x = up(x)\n        x = layers.Concatenate()([x, skip])\n\n    x = last(x)\n    return Model(inputs=inputs, outputs=x)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### Discriminator\n\nWe will use a `PatchGAN` architecture for the discriminator, incorporating many of the same layer choices as the generator.","metadata":{}},{"cell_type":"code","source":"def build_discriminator():\n    initializer = tf.random_normal_initializer(0., 0.02)\n    inp = layers.Input(shape=[256, 256, 3], name='input_image')\n    \n    x = inp\n    x = layers.Conv2D(64, 4, strides=2, padding='same',\n                      kernel_initializer=initializer)(x)\n    x = layers.LeakyReLU()(x)\n\n    x = layers.Conv2D(128, 4, strides=2, padding='same',\n                      kernel_initializer=initializer, use_bias=False)(x)\n    x = layers.LayerNormalization()(x)\n    x = layers.LeakyReLU()(x)\n\n    x = layers.Conv2D(256, 4, strides=2, padding='same',\n                      kernel_initializer=initializer, use_bias=False)(x)\n    x = layers.LayerNormalization()(x)\n    x = layers.LeakyReLU()(x)\n\n    x = layers.Conv2D(1, 4, strides=1, padding='same',\n                      kernel_initializer=initializer)(x)\n    return Model(inputs=inp, outputs=x)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### Loss Functions\n\nWe use `L1` loss and discriminator loss to ensure the generated images fool the discriminator. Cycle loss ensures we can recover the original image. Identity loss encourages the model to preserve real Monet paintings without alteration.","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    bce = tf.keras.losses.BinaryCrossentropy(from_logits=True)\n    mae = tf.keras.losses.MeanAbsoluteError()\n    \n    def generator_loss(fake_output):\n        return bce(tf.ones_like(fake_output), fake_output)\n    \n    def discriminator_loss(real_output, fake_output):\n        real_loss = bce(tf.ones_like(real_output), real_output)\n        fake_loss = bce(tf.zeros_like(fake_output), fake_output)\n        return (real_loss + fake_loss) * 0.5\n    \n    def cycle_loss(real_image, cycled_image, lambda_cycle=10.0):\n        return lambda_cycle * mae(real_image, cycled_image)\n    \n    def identity_loss(real_image, same_image, lambda_identity=5.0):\n        return lambda_identity * mae(real_image, same_image)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### Optimizers and Visualization\n\nEach model will have its own optimizer to update parameters independently.","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    gen_G = build_generator()  # Monet → Photo\n    gen_F = build_generator()  # Photo → Monet\n    disc_X = build_discriminator()\n    disc_Y = build_discriminator()\n    \n    generator_G_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    generator_F_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    discriminator_X_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    discriminator_Y_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def plot_first_example():\n    to_monet = gen_F(example_photo)\n    \n    plt.figure(figsize=(10, 5))\n    \n    for i, (title, img) in enumerate([\n        (\"Original Photo\", example_photo[0]),\n        (\"Monet-esque Photo\", to_monet[0])\n    ]):\n        plt.subplot(1, 2, i + 1)\n        plt.imshow((img + 1) / 2.0)\n        plt.title(title, fontsize=14)\n        plt.axis('off')\n    \n    plt.tight_layout()\n    plt.show()\n\nplot_first_example()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Training","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    @tf.function\n    def train_step(real_x, real_y):\n        with tf.GradientTape(persistent=True) as tape:\n            fake_y = gen_G(real_x, training=True)\n            cycled_x = gen_F(fake_y, training=True)\n            fake_x = gen_F(real_y, training=True)\n            cycled_y = gen_G(fake_x, training=True)\n\n            # Identity mapping\n            same_x = gen_F(real_x, training=True)\n            same_y = gen_G(real_y, training=True)\n\n            # Discriminator outputs\n            disc_real_x = disc_X(real_x, training=True)\n            disc_fake_x = disc_X(fake_x, training=True)\n\n            disc_real_y = disc_Y(real_y, training=True)\n            disc_fake_y = disc_Y(fake_y, training=True)\n\n            # Losses\n            gen_G_loss = generator_loss(disc_fake_y)\n            gen_F_loss = generator_loss(disc_fake_x)\n            total_cycle_loss = cycle_loss(real_x, cycled_x) + cycle_loss(real_y, cycled_y)\n            total_gen_G_loss = gen_G_loss + total_cycle_loss + identity_loss(real_y, same_y)\n            total_gen_F_loss = gen_F_loss + total_cycle_loss + identity_loss(real_x, same_x)\n            disc_X_loss = discriminator_loss(disc_real_x, disc_fake_x)\n            disc_Y_loss = discriminator_loss(disc_real_y, disc_fake_y)\n\n        # Calculate and apply gradients\n        generator_G_gradients = tape.gradient(total_gen_G_loss, gen_G.trainable_variables)\n        generator_F_gradients = tape.gradient(total_gen_F_loss, gen_F.trainable_variables)\n\n        discriminator_X_gradients = tape.gradient(disc_X_loss, disc_X.trainable_variables)\n        discriminator_Y_gradients = tape.gradient(disc_Y_loss, disc_Y.trainable_variables)\n\n        generator_G_optimizer.apply_gradients(zip(generator_G_gradients, gen_G.trainable_variables))\n        generator_F_optimizer.apply_gradients(zip(generator_F_gradients, gen_F.trainable_variables))\n\n        discriminator_X_optimizer.apply_gradients(zip(discriminator_X_gradients, disc_X.trainable_variables))\n        discriminator_Y_optimizer.apply_gradients(zip(discriminator_Y_gradients, disc_Y.trainable_variables))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"with strategy.scope():\n    @tf.function\n    def distributed_train_step(monets, photos):\n        strategy.run(train_step, args=(monets, photos))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import time\n\ndef train(dataset_monet, dataset_photo, epochs):\n    for epoch in range(epochs):\n        start = time.time()\n\n        for monets, photos in tf.data.Dataset.zip((dataset_monet, dataset_photo)):\n            distributed_train_step(monets, photos)\n\n        print(f'Epoch {epoch + 1} completed in {time.time() - start:.2f} sec')","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train(monet_ds, photo_ds, epochs=50)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#### Evaluation","metadata":{}},{"cell_type":"code","source":"plot_first_example()","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Iteration and Improvement\n\nWe have developed a good baseline model, but it looks very grid-like, has colors that are too bright, and does not remove any watermarks. To improve results, we should address the challenges identified in the EDA.\n\n- **Addressing Dataset Imbalance & Generalization**  \n  This is our biggest area for improvement, as GANs typically perform best with balanced datasets.\n\n- **Perceptual Loss (VGG Loss)**  \n  VGG Loss helps the model capture higher-level stylistic features, which could smooth out the grid-like artifacts seen in the baseline.\n\n- **Addressing Watermarks**  \n  Given more time, we could develop an algorithm to remove watermarks entirely, though this may be outside the project scope.","metadata":{}},{"cell_type":"markdown","source":"#### Dataset Imbalance\n\nWe will modify the dataset with aggressive augmentation and oversampling to expose the model to more \"Monet paintings\" by creating variations of real paintings.","metadata":{}},{"cell_type":"code","source":"def augment_image(image, aggressive=False):\n    image = tf.image.random_flip_left_right(image)\n    image = tf.image.resize_with_crop_or_pad(image, 286, 286)\n    image = tf.image.random_crop(image, size=[256, 256, 3])\n\n    if aggressive:\n        # Stronger augmentation for Monet (small dataset)\n        image = tf.image.random_brightness(image, max_delta=0.2)\n        image = tf.image.random_contrast(image, lower=0.8, upper=1.2)\n        image = tf.image.random_saturation(image, lower=0.8, upper=1.2)\n        image = tf.image.random_hue(image, max_delta=0.05)\n    else:\n        # Milder for Photos (to avoid photo → Monet leakage)\n        image = tf.image.random_brightness(image, max_delta=0.1)\n        image = tf.image.random_contrast(image, lower=0.9, upper=1.1)\n\n    return tf.clip_by_value(image, -1.0, 1.0)\n\ndef load_dataset(tfrecord_dir, batch_size=32, shuffle=True, buffer_size=1000, augment=False, aggressive=False, oversample_factor=1):\n    files = tf.io.gfile.glob(os.path.join(tfrecord_dir, '*.tfrec'))\n    ds = tf.data.TFRecordDataset(files, num_parallel_reads=tf.data.AUTOTUNE)\n\n    def _parse_and_augment(example_proto):\n        image = parse_image(example_proto)\n        if augment:\n            image = augment_image(image, aggressive=aggressive)\n        return image\n\n    ds = ds.map(_parse_and_augment, num_parallel_calls=tf.data.AUTOTUNE)\n    \n    if shuffle:\n        ds = ds.shuffle(buffer_size)\n    ds = ds.repeat(oversample_factor)\n    ds = ds.batch(batch_size).prefetch(tf.data.AUTOTUNE)\n    return ds\n\nmonet_ds = load_dataset(monet_dir, batch_size=16, augment=True, aggressive=True, oversample_factor=5)\nphoto_ds = load_dataset(photo_dir, batch_size=16, augment=True, aggressive=False)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train(monet_ds, photo_ds, epochs=50)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def show_translation_samples(dataset, generator, num_samples=5, title_input=\"Input Photo\", title_output=\"Monet-esque\"):\n    fig, axes = plt.subplots(num_samples, 2, figsize=(7.5, num_samples * 2.2),\n                             constrained_layout=True)\n\n    for i, images in enumerate(dataset.take(num_samples)):\n        input_img = images[0]\n        translated_img = generator(tf.expand_dims(input_img, 0), training=False)[0]\n        input_img = ((input_img + 1.0) * 127.5).numpy().astype(np.uint8)\n        translated_img = ((translated_img + 1.0) * 127.5).numpy().astype(np.uint8)\n        axes[i, 0].imshow(input_img)\n        axes[i, 0].axis('off')\n        if i == 0:\n            axes[i, 0].set_title(title_input, fontsize=14)\n        axes[i, 1].imshow(translated_img)\n        axes[i, 1].axis('off')\n        if i == 0:\n            axes[i, 1].set_title(title_output, fontsize=14)\n\n    plt.show()\n\nshow_translation_samples(photo_ds, gen_F, num_samples=3)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"You can see this significantly improved our color palette by exposing the model to more muted colors from painting examples. However, the model may be suffering from **mode collapse** on some colors, allowing darker and muddier colors to sneak past the discriminator more often than it should.","metadata":{}},{"cell_type":"markdown","source":"#### Perceptual Loss (VGG Loss)\n\n[VGG loss](https://paperswithcode.com/method/vgg-loss) is a loss function that optimizes for larger structural differences in image styles. This could significantly improve our model, which currently focuses on optimizing individual pixels or small groups rather than capturing broader stylistic changes.","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.applications import VGG19\nfrom tensorflow.keras.applications.vgg19 import preprocess_input\n\nvgg = VGG19(include_top=False, weights='imagenet', input_shape=(256, 256, 3))\nvgg.trainable = False\nvgg_model = tf.keras.Model(inputs=vgg.input, outputs=vgg.get_layer('block5_conv2').output)\n\ndef preprocess_vgg(image):\n    image = (image + 1.0) * 127.5\n    return preprocess_input(image)\n    \ndef perceptual_loss(real_image, generated_image):\n    real = preprocess_vgg(real_image)\n    generated = preprocess_vgg(generated_image)\n\n    real_features = vgg_model(real)\n    generated_features = vgg_model(generated)\n\n    return tf.reduce_mean(tf.abs(real_features - generated_features))\n\ndef cycle_loss(real_image, cycled_image, lambda_cycle=10.0, lambda_perceptual=1.0):\n    l1_loss = mae(real_image, cycled_image)\n    vgg_loss = perceptual_loss(real_image, cycled_image)\n    return lambda_cycle * l1_loss + lambda_perceptual * vgg_loss\n\nwith strategy.scope():\n    @tf.function\n    def train_step(real_x, real_y):\n        with tf.GradientTape(persistent=True) as tape:\n            fake_y = gen_G(real_x, training=True)\n            cycled_x = gen_F(fake_y, training=True)\n            fake_x = gen_F(real_y, training=True)\n            cycled_y = gen_G(fake_x, training=True)\n\n            # Identity mapping\n            same_x = gen_F(real_x, training=True)\n            same_y = gen_G(real_y, training=True)\n\n            # Discriminator outputs\n            disc_real_x = disc_X(real_x, training=True)\n            disc_fake_x = disc_X(fake_x, training=True)\n\n            disc_real_y = disc_Y(real_y, training=True)\n            disc_fake_y = disc_Y(fake_y, training=True)\n\n            # Losses\n            gen_G_loss = generator_loss(disc_fake_y)\n            gen_F_loss = generator_loss(disc_fake_x)\n\n            total_cycle_loss = cycle_loss(real_x, cycled_x) + cycle_loss(real_y, cycled_y)\n\n            total_gen_G_loss = gen_G_loss + total_cycle_loss + identity_loss(real_y, same_y)\n            total_gen_F_loss = gen_F_loss + total_cycle_loss + identity_loss(real_x, same_x)\n\n            disc_X_loss = discriminator_loss(disc_real_x, disc_fake_x)\n            disc_Y_loss = discriminator_loss(disc_real_y, disc_fake_y)\n\n        # Calculate and apply gradients\n        generator_G_gradients = tape.gradient(total_gen_G_loss, gen_G.trainable_variables)\n        generator_F_gradients = tape.gradient(total_gen_F_loss, gen_F.trainable_variables)\n\n        discriminator_X_gradients = tape.gradient(disc_X_loss, disc_X.trainable_variables)\n        discriminator_Y_gradients = tape.gradient(disc_Y_loss, disc_Y.trainable_variables)\n\n        generator_G_optimizer.apply_gradients(zip(generator_G_gradients, gen_G.trainable_variables))\n        generator_F_optimizer.apply_gradients(zip(generator_F_gradients, gen_F.trainable_variables))\n\n        discriminator_X_optimizer.apply_gradients(zip(discriminator_X_gradients, disc_X.trainable_variables))\n        discriminator_Y_optimizer.apply_gradients(zip(discriminator_Y_gradients, disc_Y.trainable_variables))","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"train(monet_ds, photo_ds, epochs=50)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"show_translation_samples(photo_ds, gen_F, num_samples=3)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Results and Analysis","metadata":{}},{"cell_type":"code","source":"import PIL\n! mkdir ../images\n\nphoto_ds = load_dataset(photo_dir, batch_size=1, shuffle=False)\n\ni = 1\nfor img_batch in photo_ds:\n    img = img_batch[0]\n    prediction = gen_F(tf.expand_dims(img, 0), training=False)[0].numpy()\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    im = PIL.Image.fromarray(prediction)\n    im.save(f\"../images/{i}.jpg\")\n    i += 1\n\nimport shutil\nshutil.make_archive(\"/kaggle/working/images\", 'zip', \"../images\")","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"VGG loss significantly improved the model performance, and after reviewing some samples I decided visually it was good enough to submit. It would be great to implement a scoring function here so we could validate on iterations moving forward.\n","metadata":{}},{"cell_type":"markdown","source":"## Conclusion","metadata":{}},{"cell_type":"markdown","source":"Our GAN model made good progress generating Monet-style images after some key changes.\n\n---\n\n### Key Improvements:\n\n* **Color Palette:** We used more data and oversampling to fix dataset imbalance. This helped the model learn a wider range of muted colors, leading to a more balanced and less bright color palette, unlike the earlier grid-like artifacts.\n* **Stylistic Fidelity:** Adding **perceptual loss (VGG loss)** was crucial. It optimized for larger structural and textural features, allowing the model to better capture Monet's brushwork and lighting, making the images more stylistically authentic than with just pixel-wise losses.\n\n---\n\n### Remaining Challenges:\n\n* We still saw some **mode collapse**, especially with darker colors, which let less realistic textures slip past the discriminator.\n* **Watermark removal** wasn't fully solved and is something we can work on in the future.","metadata":{}}]}