{"metadata":{"kernelspec":{"display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":21755,"databundleVersionId":1475600,"sourceType":"competition"}],"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false},"colab":{"provenance":[],"toc_visible":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Week 5: GAN Mini-Project\n\nThis is the mini-project for Week 5 of CSCA 5642: Introduction to Deep Learning.\n\nAs part of this project, we will compete in the [Monet Style Image Generation](https://www.kaggle.com/competitions/gan-getting-started) competition. The primary goal of the competitoin is to build a model that will produce Monet-style images based on \"normal\" input images.\n\nThis type of problem is typically addressed using a GAN (Genrative Adversarial Network) which consists of at least two neural networks: a generator and a discriminator. The generator is used to generate the images, while the discriminator is used to train the generator during model training.","metadata":{"id":"QJjKT3-UqMKM"}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\n# import os\n# for dirname, _, filenames in os.walk('/kaggle/input'):\n#     for filename in filenames:\n#         print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\"\n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session\n\nimport tensorflow as tf\nimport keras\nfrom keras import layers\nfrom keras import ops","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2025-08-20T02:15:00.385721Z","iopub.execute_input":"2025-08-20T02:15:00.38598Z","iopub.status.idle":"2025-08-20T02:15:18.854199Z","shell.execute_reply.started":"2025-08-20T02:15:00.385962Z","shell.execute_reply":"2025-08-20T02:15:18.853313Z"},"id":"sHeoYtBwqMKN"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Competition Data\n\nThe [competition data](https://www.kaggle.com/competitions/gan-getting-started/data) consists of 300 images of Monet paintings, and 7028 regular photos. All images are color with dimensions 256x256. Each set of images is provided in both JPEG or TFRecord format - either may be used for the competiton.","metadata":{"id":"R8PjuK-4qMKN"}},{"cell_type":"markdown","source":"# Exploratory Data Analysis (EDA)\n\nWe will now load the competion data and perform EDA to get an idea of out inputs, and to ensure consistency as far as image sizes.\n\n## Data Loading\n\nAs mention above both TFRecord and JPEG formats are provided. We'll use the JPEG files and leverage Keras utilities to load and process. Using [image_dataset_from_directory](https://keras.io/api/data_loading/image/) we can get a TensorFlow Dataset object, which can be easier to use for processing and as input to TF/Keras pipelines.","metadata":{"id":"RriDHV4QqMKO"}},{"cell_type":"code","source":"# ROOT='/root/.cache/kagglehub/competitions/gan-getting-started'\nROOT='/kaggle/input/gan-getting-started'\n\nmonet_ds = keras.utils.image_dataset_from_directory(\n    ROOT + '/monet_jpg',\n    label_mode=None,\n    batch_size=1,\n    image_size=(256, 256),\n)\n\nphoto_ds = keras.utils.image_dataset_from_directory(\n    ROOT + '/photo_jpg',\n    label_mode=None,\n    batch_size=1,\n    image_size=(256, 256),\n)\n\nmonet_ds = monet_ds.cache()\nphoto_ds = photo_ds.cache()\nmonet_ds = monet_ds.prefetch(tf.data.AUTOTUNE)\nphoto_ds = photo_ds.prefetch(tf.data.AUTOTUNE)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-20T02:15:34.69484Z","iopub.execute_input":"2025-08-20T02:15:34.695497Z","iopub.status.idle":"2025-08-20T02:15:40.176921Z","shell.execute_reply.started":"2025-08-20T02:15:34.695442Z","shell.execute_reply":"2025-08-20T02:15:40.175556Z"},"id":"BbhRlFuoqMKO"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Data Exploration\n\nWith the data loaded into Dataset objects, we can start exploring what we're given.\n\nTo start with, we can view a sample of both the Monet paintings and the photos:","metadata":{"id":"AX1NwJQRqMKP"}},{"cell_type":"code","source":"# Modified helper from https://www.kaggle.com/code/dimitreoliveira/introduction-to-cyclegan-monet-paintings\ndef display_samples(ds, row, col):\n    ds_iter = iter(ds)\n    plt.figure(figsize=(15, int(15*row/col)))\n    for j in range(row*col):\n        example_sample = next(ds_iter)\n        plt.subplot(row,col,j+1)\n        plt.axis('off')\n        plt.imshow(example_sample.numpy().astype('int32')[0])\n    plt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-20T02:29:52.595391Z","iopub.execute_input":"2025-08-20T02:29:52.596428Z","iopub.status.idle":"2025-08-20T02:29:52.601837Z","shell.execute_reply.started":"2025-08-20T02:29:52.596394Z","shell.execute_reply":"2025-08-20T02:29:52.600808Z"},"id":"2yXl4CO_qMKP"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Monet Painting Samples","metadata":{"id":"gUDHyrTEqMKQ"}},{"cell_type":"code","source":"display_samples(monet_ds, 2, 3)","metadata":{"trusted":true,"id":"R0jAzWTCqMKQ"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Photo Samples","metadata":{"id":"dvNytgFyqMKQ"}},{"cell_type":"code","source":"display_samples(photo_ds, 2, 3)","metadata":{"trusted":true,"id":"EpoLtK1mqMKR"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The paintings definitely look like Monet, and the photos appear to include at least landscapes and humans.\n\nNext, we'll verify our understand of the data shape based on the TensorFlow dataset.","metadata":{"id":"7kSeTCpSqMKR"}},{"cell_type":"code","source":"# Monet dataset\nprint(f'Monet shape: {iter(monet_ds).next().shape}')\n\n# Photo dataset\nprint(f'Photo shape: {iter(photo_ds).next().shape}')","metadata":{"trusted":true,"id":"kirwOVHZqMKR"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"As anticipated, we have 32 photos per batch (the default for `image_dataset_from_directory`), images with dimensions 256 x 256, and they're color images as indicated by the final `3` in the shape, representing the RGB color (channels).","metadata":{"id":"mTvKZnlDqMKR"}},{"cell_type":"markdown","source":"## Data Preprocessing\n\nThe datasets provide color images, so we need to rescale the possible value range of [0, 255] down to [-1, 1] for training. This normalizing step is required for training to be more stable and generally have faster convergence. If we left the full values in place, there could be very large gradients that would be difficult for backprop to handle. Additionally, if we have any sigmoid activation functions in the model(s), they do not work well with values much greater than 1 or smaller than -1.\n\nBeyond normalization, we will not perform any image augmentation since that could adversly affect training by introducing image transformations that we do not want the model to learn.","metadata":{"id":"iGbKAPV6qMKR"}},{"cell_type":"code","source":"monet_ds = monet_ds.map(lambda x: (x / 127.5) - 1.0)\nphoto_ds = photo_ds.map(lambda x: (x / 127.5) - 1.0)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-20T02:16:07.75314Z","iopub.execute_input":"2025-08-20T02:16:07.75364Z","iopub.status.idle":"2025-08-20T02:16:07.807788Z","shell.execute_reply.started":"2025-08-20T02:16:07.753606Z","shell.execute_reply":"2025-08-20T02:16:07.806864Z"},"id":"T31AoKxeqMKR"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Plan of Analysis\n\nBased on our consistent data shape, goal of the competition, and subtle hint from the challenge description, we'll build CycleGAN and then expand on architecture/hyperparameter changes to try to improve model performance.","metadata":{"id":"ZJy5GpagqMKS"}},{"cell_type":"markdown","source":"# Model Architecture\n\n## Basic CycleGAN\n\nAs noted above in `Plan of Analysis`, we'll begin with a baseline CycleGAN for comparison of model performance in the competition.\n\nA vanilla GAN typically consists of a `generator` and `discriminator`. The `generator` part of the GAN is trained to generate images that the `discriminator` labels as \"real\" or \"fake\". Input to the `generator` is \"noise\" from latent space, and the `discriminator` compares the generated images with real images to determine whether the generated images are \"real\".\n\nOur challenge has a slightly different goal - we need to build and train a model to generate a given image with a style that it's trained to use - in this case, Monet style paintings. As hinted at in the challenge description, a CycleGAN can be used to solve this type of problem.\n\nA CycleGAN is similar to a traditional GAN in that is has the concept of `generator` and `discriminator`, but the CycleGAN uses two of each.\n\n* Generator A: takes input images from domain X and translates them into domain Y\n* Generator B: takes input images from domain Y and translates them into domain X\n* discriminator A: decides on fake vs. real for images from domain X and images generated by Generator B\n* discriminator B: decides on fake vs. real for images from domain Y and images generated by Generator A\n\nThis is where the \"cycle\" part of CycleGAN comes from - images are sort of going in a cycle between generators and discriminators for each domain. In our case for the Monet challengs, domain X consists of the photo images, and domain Y consists of the Monet painting images.\n\nThe `generator` part of the model is generally based on the [U-Net](https://www.geeksforgeeks.org/machine-learning/u-net-architecture-explained/) architecture, which has the Encoder->Bottleneck->Decoder shape. The Encoder continually downsamples the images and extracts features, the Bottleneck typically consists of convolutional layers where the image abstraction/latent space is learned, and the Decoder continually upsamples latent space data back to the original images size. There may also be skip connections between some of the Encoder and Decoder layers.\n\nOn the 'discriminator` side, a [PatchGAN architecture](https://www.researchgate.net/figure/Architecture-of-the-PatchGAN-discriminator-The-discriminator-takes-two-channel-wise_fig2_358603007) is used due to its ability to extract more accurate real/fake signals based on localized features since it's looking at patches of the image vs. the entire image all at once.","metadata":{"id":"HZd2dDR0qMKS"}},{"cell_type":"markdown","source":"## Basic CycleGAN - Build the Model\n\nWe'll now build the `Generator` part of the model as a basic U-Net architecture as noted above. It will use standard Conv2D and batch-norm blocks combined with MaxPooling2D for downsampling, and standard Conv2D + batch-norm blocks with UpSampling2D for the upsampling side. No other special architecture features will be used for this baseline model.","metadata":{"id":"l97bev4GqMKS"}},{"cell_type":"code","source":"from keras.layers import Input, Conv2D, MaxPooling2D, UpSampling2D, BatchNormalization, Activation\nfrom keras.models import Model\n\ndef conv2d_block(i, n):\n    x = Conv2D(n, (3, 3), padding='same')(i)\n    x = BatchNormalization()(x)\n    x = Activation('relu')(x)\n    # try LeakyReLU(negative_slope=0.2)\n    x = Conv2D(n, (3, 3), padding='same')(x)\n    x = BatchNormalization()(x)\n    x = Activation('relu')(x)\n    return x\n\ndef downsample(i, n):\n    x = conv2d_block(i, n)\n    return MaxPooling2D(pool_size=(2, 2))(x)\n\ndef upsample(i, n):\n    x = UpSampling2D(size=(2,2))(i)\n    return conv2d_block(x, n)\n\ndef build_generator(input_shape=(256,256,3)):\n    inputs = Input(input_shape)\n\n    # Encoder\n    x = downsample(inputs, 64)\n    x = downsample(x, 256)\n    x = downsample(x, 512)\n\n    # Bottleneck\n    x = conv2d_block(x, 1024)\n\n    # Decoder\n    x = upsample(x, 512)\n    x = upsample(x, 256)\n    x = upsample(x, 128)\n    x = upsample(x, 64)\n\n    outputs = Conv2D(3, 4, strides=2, padding='same', activation='sigmoid')(x)\n\n    return Model(inputs=inputs, outputs=outputs, name='generator')\n\n# g = build_generator()\n# g.summary()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-20T02:16:22.049969Z","iopub.execute_input":"2025-08-20T02:16:22.050266Z","iopub.status.idle":"2025-08-20T02:16:22.060904Z","shell.execute_reply.started":"2025-08-20T02:16:22.050245Z","shell.execute_reply":"2025-08-20T02:16:22.059941Z"},"id":"rDg3yyLZqMKS"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Now for the `discriminator` part of the model. This is where we use the `PatchGAN` architecture to downsample patches of the original image to output a real/fake matrix with a value for each patch.","metadata":{"id":"NwQRBhrwqMKS"}},{"cell_type":"code","source":"def build_discriminator(input_shape=(256,256,3)):\n    inputs = Input(input_shape)\n\n    x = downsample(inputs, 64)\n    x = downsample(x, 128)\n    x = downsample(x, 256)\n\n    x = Conv2D(512, 4, strides=1)(x)\n    out = Conv2D(1, 4, strides=1)(x)\n\n    return Model(inputs=inputs, outputs=out, name='discriminator')\n\n# d = build_discriminator()\n# d.summary()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-20T02:16:57.334126Z","iopub.execute_input":"2025-08-20T02:16:57.334403Z","iopub.status.idle":"2025-08-20T02:16:57.340127Z","shell.execute_reply.started":"2025-08-20T02:16:57.334384Z","shell.execute_reply":"2025-08-20T02:16:57.339131Z"},"id":"2UgdsUI1qMKS"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Instantiate generators and discriminators.\nmonet_generator = build_generator() # convert photos to Monet\nphoto_generator = build_generator() # convert Monet to photos\n\nmonet_discriminator = build_discriminator() # determines real Monet vs generated Monet (real vs fake)\nphoto_discriminator = build_discriminator() # determines real photos vs generated photos (real vs fake)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-20T02:17:16.92207Z","iopub.execute_input":"2025-08-20T02:17:16.922369Z","iopub.status.idle":"2025-08-20T02:17:18.26547Z","shell.execute_reply.started":"2025-08-20T02:17:16.922348Z","shell.execute_reply":"2025-08-20T02:17:18.264553Z"},"id":"I6cjeTMgqMKT"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Now, we need to provide our own loss functions for the generator, discriminator, cycled photo, and \"self generated\" photo.\n\nThe generator loss is determined by comparing the generated image with all 1s. All 1s would represent a generated image that the discriminator thinks is 100% real, so we want to see the difference between current and desired state.\n\nThe discriminator loss is determined by comparing the real image to all 1s and the fake image to all 0s and returning the average. Since the discriminator is training toward always being correct (i.e., correctly identifying real vs fake images), this loss represents how well it's doing.\n\nThe \"cycle photo\" loss measures how well a photo is converted to Monet style and then back to the real photo (as well as the opposite path).\n\nFinally, the \"self generated\" loss measures the similarity between an original photo and the same photo fed through the photo generator.","metadata":{"id":"sRZjY6axqMKT"}},{"cell_type":"code","source":"# Loss functions (ref: https://www.kaggle.com/code/amyjang/monet-cyclegan-tutorial)\ndef discriminator_loss(real, generated):\n    real_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(real), real)\n\n    generated_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.zeros_like(generated), generated)\n\n    total_disc_loss = real_loss + generated_loss\n\n    return total_disc_loss * 0.5\n\ndef generator_loss(generated):\n    return tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(generated), generated)\n\n# with strategy.scope():\ndef calc_cycle_loss(real_image, cycled_image, LAMBDA):\n    loss1 = tf.reduce_mean(tf.abs(real_image - cycled_image))\n\n    return LAMBDA * loss1\n\n# with strategy.scope():\ndef identity_loss(real_image, same_image, LAMBDA):\n    loss = tf.reduce_mean(tf.abs(real_image - same_image))\n    return LAMBDA * 0.5 * loss","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-20T02:17:30.653154Z","iopub.execute_input":"2025-08-20T02:17:30.654134Z","iopub.status.idle":"2025-08-20T02:17:30.660475Z","shell.execute_reply.started":"2025-08-20T02:17:30.654094Z","shell.execute_reply":"2025-08-20T02:17:30.659484Z"},"id":"T1Vo4GgKqMKT"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"With loss functions defined, we can now build the `CycleGAN` model by extending `keras.Model` to override `compile()` and `train_step()`. The latter is called by `fit()` during training.\n\nThe model implementation was heavily adopted from https://www.kaggle.com/code/amyjang/monet-cyclegan-tutorial and https://keras.io/examples/generative/cyclegan/. New comments have been added in-line to expand on describing the functionality and progression of using the different models and loss functions during a training step.","metadata":{"id":"QxBlw2t5qMKT"}},{"cell_type":"code","source":"# CycleGAN model.\nclass CycleGAN(keras.Model):\n    def __init__(\n        self,\n        monet_generator,\n        photo_generator,\n        monet_discriminator,\n        photo_discriminator,\n        lambda_cycle=10,\n    ):\n        super(CycleGAN, self).__init__()\n        self.m_gen = monet_generator\n        self.p_gen = photo_generator\n        self.m_disc = monet_discriminator\n        self.p_disc = photo_discriminator\n        self.lambda_cycle = lambda_cycle\n\n    def compile(\n        self,\n        m_gen_optimizer,\n        p_gen_optimizer,\n        m_disc_optimizer,\n        p_disc_optimizer,\n        gen_loss_fn,\n        disc_loss_fn,\n        cycle_loss_fn,\n        identity_loss_fn\n    ):\n        super(CycleGAN, self).compile()\n        self.m_gen_optimizer = m_gen_optimizer\n        self.p_gen_optimizer = p_gen_optimizer\n        self.m_disc_optimizer = m_disc_optimizer\n        self.p_disc_optimizer = p_disc_optimizer\n        self.gen_loss_fn = gen_loss_fn\n        self.disc_loss_fn = disc_loss_fn\n        self.cycle_loss_fn = cycle_loss_fn\n        self.identity_loss_fn = identity_loss_fn\n\n    def train_step(self, batch_data):\n        # Extract the real Monet and photo tensors from provided input.\n        real_monet, real_photo = batch_data\n\n        # Use auto-differentiation! https://www.tensorflow.org/guide/autodiff\n        with tf.GradientTape(persistent=True) as tape:\n            # First, we convert a real photo to a Monet, then back again. This is the\n            # \"cycle photo\" mentioned in the loss function description above.\n            fake_monet = self.m_gen(real_photo, training=True)\n            cycled_photo = self.p_gen(fake_monet, training=True)\n\n            # Next, convert a real Monet to a photo, then back again. This is another\n            # \"cycle photo\", but in the opposite direction as above.\n            fake_photo = self.p_gen(real_monet, training=True)\n            cycled_monet = self.m_gen(fake_photo, training=True)\n\n            # Next, create the \"self generated\" Monet image and photo. Each of these\n            # will be compared to the original \"real\" version for part of the\n            # generator loss measurement.\n            same_monet = self.m_gen(real_monet, training=True)\n            same_photo = self.p_gen(real_photo, training=True)\n\n            # Next, we measure each discriminator's response to a real Monet and\n            # a real photo for use by the discriminator loss function.\n            disc_real_monet = self.m_disc(real_monet, training=True)\n            disc_real_photo = self.p_disc(real_photo, training=True)\n\n            # Similarly, measure each discriminator's response to a fake Monet and\n            # a fake photo, but for use by the generator loss function.\n            disc_fake_monet = self.m_disc(fake_monet, training=True)\n            disc_fake_photo = self.p_disc(fake_photo, training=True)\n\n            # Provide the discriminator responses to the generator loss function\n            # for calculating generator loss.\n            monet_gen_loss = self.gen_loss_fn(disc_fake_monet)\n            photo_gen_loss = self.gen_loss_fn(disc_fake_photo)\n\n            # Calculate the sum of the \"cycled\" loss for both directions\n            # (Photo->Monet->Photo and Monet->Photo->Monet)\n            total_cycle_loss = self.cycle_loss_fn(real_monet, cycled_monet, self.lambda_cycle) + self.cycle_loss_fn(real_photo, cycled_photo, self.lambda_cycle)\n\n            # Sum the generator losses for use in backprop.\n            total_monet_gen_loss = monet_gen_loss + total_cycle_loss + self.identity_loss_fn(real_monet, same_monet, self.lambda_cycle)\n            total_photo_gen_loss = photo_gen_loss + total_cycle_loss + self.identity_loss_fn(real_photo, same_photo, self.lambda_cycle)\n\n            # Get the discriminator losses for use in backprop.\n            monet_disc_loss = self.disc_loss_fn(disc_real_monet, disc_fake_monet)\n            photo_disc_loss = self.disc_loss_fn(disc_real_photo, disc_fake_photo)\n\n        # Calculate the gradients for generator and discriminator.\n        monet_generator_gradients = tape.gradient(total_monet_gen_loss,\n                                                  self.m_gen.trainable_variables)\n        photo_generator_gradients = tape.gradient(total_photo_gen_loss,\n                                                  self.p_gen.trainable_variables)\n\n        monet_discriminator_gradients = tape.gradient(monet_disc_loss,\n                                                      self.m_disc.trainable_variables)\n        photo_discriminator_gradients = tape.gradient(photo_disc_loss,\n                                                      self.p_disc.trainable_variables)\n\n        # Apply the gradients to the optimizer.\n        # print(f'm_gen_optimizer.apply_gradients()')\n        self.m_gen_optimizer.apply_gradients(zip(monet_generator_gradients,\n                                                 self.m_gen.trainable_variables))\n\n        # print(f'p_gen_optimizer.apply_gradients()')\n        self.p_gen_optimizer.apply_gradients(zip(photo_generator_gradients,\n                                                 self.p_gen.trainable_variables))\n\n        # print(f'm_disc_optimizer.apply_gradients()')\n        self.m_disc_optimizer.apply_gradients(zip(monet_discriminator_gradients,\n                                                  self.m_disc.trainable_variables))\n\n        # print(f'p_disc_optimizer.apply_gradients()')\n        self.p_disc_optimizer.apply_gradients(zip(photo_discriminator_gradients,\n                                                  self.p_disc.trainable_variables))\n\n        # print(f'After p_disc_optimizer.apply_gradients()\\n')\n        return {\n            \"monet_gen_loss\": total_monet_gen_loss,\n            \"photo_gen_loss\": total_photo_gen_loss,\n            \"monet_disc_loss\": monet_disc_loss,\n            \"photo_disc_loss\": photo_disc_loss\n        }","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-20T02:17:41.14297Z","iopub.execute_input":"2025-08-20T02:17:41.14353Z","iopub.status.idle":"2025-08-20T02:17:41.165314Z","shell.execute_reply.started":"2025-08-20T02:17:41.143493Z","shell.execute_reply":"2025-08-20T02:17:41.16373Z"},"id":"xQyZmdxlqMKT"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Define the optimizers for each submodel.\nmonet_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\nphoto_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n\nmonet_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\nphoto_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-20T02:17:54.16448Z","iopub.execute_input":"2025-08-20T02:17:54.164766Z","iopub.status.idle":"2025-08-20T02:17:54.258127Z","shell.execute_reply.started":"2025-08-20T02:17:54.164744Z","shell.execute_reply":"2025-08-20T02:17:54.257185Z"},"id":"TICUJpBAqMKU"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Instantiate and compile the CycleGAN model.\ncycle_gan_model = CycleGAN(\n    monet_generator, photo_generator, monet_discriminator, photo_discriminator\n)\n\ncycle_gan_model.compile(\n    m_gen_optimizer = monet_generator_optimizer,\n    p_gen_optimizer = photo_generator_optimizer,\n    m_disc_optimizer = monet_discriminator_optimizer,\n    p_disc_optimizer = photo_discriminator_optimizer,\n    gen_loss_fn = generator_loss,\n    disc_loss_fn = discriminator_loss,\n    cycle_loss_fn = calc_cycle_loss,\n    identity_loss_fn = identity_loss\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-20T02:18:17.465909Z","iopub.execute_input":"2025-08-20T02:18:17.466221Z","iopub.status.idle":"2025-08-20T02:18:17.480844Z","shell.execute_reply.started":"2025-08-20T02:18:17.466198Z","shell.execute_reply":"2025-08-20T02:18:17.480148Z"},"id":"btH8ee_6qMKU"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Train/fit the model using the input Dataset objects.\nh = cycle_gan_model.fit(\n    tf.data.Dataset.zip((monet_ds, photo_ds)),\n    epochs=1,\n    verbose=2,\n)","metadata":{"trusted":true,"id":"A-XlFkLAqMKU"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Visualize Results","metadata":{"id":"ZODygDUzqMKU"}},{"cell_type":"code","source":"fig, ax = plt.subplots(5, 2, figsize=(12, 12))\nfor i, img in enumerate(photo_ds.take(5)):\n    prediction = monet_generator(img, training=False)[0].numpy()\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    img = (img[0] * 127.5 + 127.5).numpy().astype(np.uint8)\n\n    ax[i, 0].imshow(img)\n    ax[i, 1].imshow(prediction)\n    ax[i, 0].set_title(\"Input Photo\")\n    ax[i, 1].set_title(\"Transformed\")\n    ax[i, 0].axis(\"off\")\n    ax[i, 1].axis(\"off\")\nfig.suptitle(\"Results: 1 epoch\", fontsize=16)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-20T02:21:06.756681Z","iopub.execute_input":"2025-08-20T02:21:06.756982Z","iopub.status.idle":"2025-08-20T02:21:27.052854Z","shell.execute_reply.started":"2025-08-20T02:21:06.756958Z","shell.execute_reply":"2025-08-20T02:21:27.051933Z"},"id":"-48_oDOsqMKU"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"From a single training run, we can see that the transformation is approaching a very basic shape of the original image.","metadata":{"id":"ReKOL_eIGoeo"}},{"cell_type":"markdown","source":"## Create Competition Submission","metadata":{"id":"9IKmHiOWqMKU"}},{"cell_type":"code","source":"import PIL\n! mkdir /kaggle/working/images_stage","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-20T02:22:57.100743Z","iopub.execute_input":"2025-08-20T02:22:57.101063Z","iopub.status.idle":"2025-08-20T02:22:57.264476Z","shell.execute_reply.started":"2025-08-20T02:22:57.10104Z","shell.execute_reply":"2025-08-20T02:22:57.263532Z"},"id":"4fHcjGovqMKV"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import shutil\ndef create_submission():\n    # shutil.rmtree('/kaggle/images/')\n    for i, img in enumerate(photo_ds):\n        prediction = monet_generator(img, training=False)[0].numpy()\n        prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n        im = PIL.Image.fromarray(prediction)\n        im.save(\"/kaggle/working/images_stage/\" + str(i) + \".jpg\")\n    shutil.make_archive(\"/kaggle/working/images\", 'zip', \"/kaggle/working/images_stage\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-20T02:24:03.922694Z","iopub.execute_input":"2025-08-20T02:24:03.92316Z","iopub.status.idle":"2025-08-20T02:24:03.929184Z","shell.execute_reply.started":"2025-08-20T02:24:03.923136Z","shell.execute_reply":"2025-08-20T02:24:03.927801Z"},"id":"KNFzyFweqMKV"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Baseline - 1 epoch\n# create_submission()","metadata":{"trusted":true,"id":"OiFW6UmEqMKV"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Score for baseline (1 epoch): 232.49346","metadata":{"id":"TtrZxWMKqMKV"}},{"cell_type":"markdown","source":"## Additional Training\n\nWe'll now train for 10 epochs and see how our results compare visually, and submit to the competition for scoring.","metadata":{"id":"Husyu9zGqMKb"}},{"cell_type":"code","source":"# Rebuild generators, discriminators, optimizers, and model for new training.","metadata":{"trusted":true,"id":"znLukIVSqMKb"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Instantiate generators and discriminators.\nmonet_generator = build_generator() # convert photos to Monet\nphoto_generator = build_generator() # convert Monet to photos\n\nmonet_discriminator = build_discriminator() # determines real Monet vs generated Monet (real vs fake)\nphoto_discriminator = build_discriminator() # determines real photos vs generated photos (real vs fake)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-20T02:17:16.92207Z","iopub.execute_input":"2025-08-20T02:17:16.922369Z","iopub.status.idle":"2025-08-20T02:17:18.26547Z","shell.execute_reply.started":"2025-08-20T02:17:16.922348Z","shell.execute_reply":"2025-08-20T02:17:18.264553Z"},"id":"PkTU87dWqMKc"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Define the optimizers for each submodel.\nmonet_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\nphoto_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n\nmonet_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\nphoto_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-20T02:17:54.16448Z","iopub.execute_input":"2025-08-20T02:17:54.164766Z","iopub.status.idle":"2025-08-20T02:17:54.258127Z","shell.execute_reply.started":"2025-08-20T02:17:54.164744Z","shell.execute_reply":"2025-08-20T02:17:54.257185Z"},"id":"tsl2AVR7qMKc"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Instantiate and compile the CycleGAN model.\ncycle_gan_model = CycleGAN(\n    monet_generator, photo_generator, monet_discriminator, photo_discriminator\n)\n\ncycle_gan_model.compile(\n    m_gen_optimizer = monet_generator_optimizer,\n    p_gen_optimizer = photo_generator_optimizer,\n    m_disc_optimizer = monet_discriminator_optimizer,\n    p_disc_optimizer = photo_discriminator_optimizer,\n    gen_loss_fn = generator_loss,\n    disc_loss_fn = discriminator_loss,\n    cycle_loss_fn = calc_cycle_loss,\n    identity_loss_fn = identity_loss\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-08-20T02:18:17.465909Z","iopub.execute_input":"2025-08-20T02:18:17.466221Z","iopub.status.idle":"2025-08-20T02:18:17.480844Z","shell.execute_reply.started":"2025-08-20T02:18:17.466198Z","shell.execute_reply":"2025-08-20T02:18:17.480148Z"},"id":"W4tzc60rqMKc"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Train/fit the model using the input Dataset objects.\nh = cycle_gan_model.fit(\n    tf.data.Dataset.zip((monet_ds, photo_ds)),\n    epochs=10,\n    verbose=2,\n)","metadata":{"trusted":true,"id":"CUUK64H4qMKc"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"fig, ax = plt.subplots(5, 2, figsize=(12, 12))\nfor i, img in enumerate(photo_ds.take(5)):\n    prediction = monet_generator(img, training=False)[0].numpy()\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    img = (img[0] * 127.5 + 127.5).numpy().astype(np.uint8)\n\n    ax[i, 0].imshow(img)\n    ax[i, 1].imshow(prediction)\n    ax[i, 0].set_title(\"Input Photo\")\n    ax[i, 1].set_title(\"Transformed\")\n    ax[i, 0].axis(\"off\")\n    ax[i, 1].axis(\"off\")\nfig.suptitle(\"Results: 10 epochs\", fontsize=16)\nplt.show()","metadata":{"trusted":true,"id":"fGcQEq4yqMKc"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Baseline - 10 epochs\n# create_submission()","metadata":{"trusted":true,"id":"AssBGy8PqMKc"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Visually, we can see that the 10 epoch model performed much worse! Based on just the `generator` loss values from the training run, they continued to hover in the 20s and never quite improved. Sadly, submitting this version to the competition repeatedly failed with `invalid JSON` errors, but did finally get a valid submission after chewing through 2 days of submit quota.\n\nThe Kaggle score was: 492.09268","metadata":{"id":"OBExCfNKqMKc"}},{"cell_type":"markdown","source":"## Model Architecture Change\n\nFrom the 10 epoch `fit()` progression above, we can notice that the loss for the `generator`s didn't improve much, hovering in the 20s. However, the `discriminator` losses moved to well below zero. This may indicate overfitting on the `discriminator` side. We'll next update the architecture for the PatchGAN and U-Net models with better regularization and initialization as was used by the [tutorial notebook](https://www.kaggle.com/code/amyjang/monet-cyclegan-tutorial).\n\nFor the downsample blocks, they use random initializers combined with `InstanceNormalization` instead of `BatchNormalization`. Additionally, they're using a stock `LeakyReLU` vs the standard `ReLU`.\n\nFor upsample blocks, they also used random initializers with `InstanceNormalization`. Instead of `UpSampling2D`, they use `Conv2DTranspose` and also add an additional `Dropout`.\n\nThese changes at the \"block\" level should improve regularization overall. As an added improvement, we'll add the skip connections common for U-Net architectures to the `generator`, which can help the model recover details on the upsampling steps that were lost during downsampling.\n\nNOTE: `InstanceNormalization` from TensorFlow addons is no longer in development, but we can achieve the same effect using `GroupNormalization` across the channels.","metadata":{"id":"PCM0Tny3um4J"}},{"cell_type":"code","source":"# New upsample/downsample blocks and generator/discriminator from\n# https://www.kaggle.com/code/amyjang/monet-cyclegan-tutorial#Build-the-generator\n# with modifications for EOL layer(s).\nfrom keras.layers import GroupNormalization\n\ndef downsample(filters, size, apply_instancenorm=True):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    result = keras.Sequential()\n    result.add(layers.Conv2D(filters, size, strides=2, padding='same',\n                             kernel_initializer=initializer, use_bias=False))\n\n    if apply_instancenorm:\n        result.add(GroupNormalization(groups=32, axis=-1,\n                                      gamma_initializer=gamma_init))\n\n    result.add(layers.LeakyReLU())\n\n    return result\n\ndef upsample(filters, size, apply_dropout=False):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    result = keras.Sequential()\n    result.add(layers.Conv2DTranspose(filters, size, strides=2,\n                                      padding='same',\n                                      kernel_initializer=initializer,\n                                      use_bias=False))\n\n    result.add(GroupNormalization(groups=32, axis=-1,\n                                  gamma_initializer=gamma_init))\n\n    if apply_dropout:\n        result.add(layers.Dropout(0.5))\n\n    result.add(layers.ReLU())\n\n    return result\n\ndef build_generator():\n    inputs = layers.Input(shape=[256,256,3])\n\n    # bs = batch size\n    down_stack = [\n        downsample(64, 4, apply_instancenorm=False), # (bs, 128, 128, 64)\n        downsample(128, 4), # (bs, 64, 64, 128)\n        downsample(256, 4), # (bs, 32, 32, 256)\n        downsample(512, 4), # (bs, 16, 16, 512)\n        downsample(512, 4), # (bs, 8, 8, 512)\n        downsample(512, 4), # (bs, 4, 4, 512)\n        downsample(512, 4), # (bs, 2, 2, 512)\n        downsample(512, 4), # (bs, 1, 1, 512)\n    ]\n\n    up_stack = [\n        upsample(512, 4, apply_dropout=True), # (bs, 2, 2, 1024)\n        upsample(512, 4, apply_dropout=True), # (bs, 4, 4, 1024)\n        upsample(512, 4, apply_dropout=True), # (bs, 8, 8, 1024)\n        upsample(512, 4), # (bs, 16, 16, 1024)\n        upsample(256, 4), # (bs, 32, 32, 512)\n        upsample(128, 4), # (bs, 64, 64, 256)\n        upsample(64, 4), # (bs, 128, 128, 128)\n    ]\n\n    initializer = tf.random_normal_initializer(0., 0.02)\n    last = layers.Conv2DTranspose(3, 4,\n                                  strides=2,\n                                  padding='same',\n                                  kernel_initializer=initializer,\n                                  activation='tanh') # (bs, 256, 256, 3)\n    x = inputs\n\n    # Downsampling through the model\n    skips = []\n    for down in down_stack:\n        x = down(x)\n        skips.append(x)\n\n    skips = reversed(skips[:-1])\n\n    # Upsampling and establishing the skip connections\n    for up, skip in zip(up_stack, skips):\n        x = up(x)\n        x = layers.Concatenate()([x, skip])\n\n    x = last(x)\n\n    return keras.Model(inputs=inputs, outputs=x)\n\ndef build_discriminator():\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    inp = layers.Input(shape=[256, 256, 3], name='input_image')\n\n    x = inp\n\n    down1 = downsample(64, 4, False)(x) # (bs, 128, 128, 64)\n    down2 = downsample(128, 4)(down1) # (bs, 64, 64, 128)\n    down3 = downsample(256, 4)(down2) # (bs, 32, 32, 256)\n\n    zero_pad1 = layers.ZeroPadding2D()(down3) # (bs, 34, 34, 256)\n    conv = layers.Conv2D(512, 4, strides=1,\n                         kernel_initializer=initializer,\n                         use_bias=False)(zero_pad1) # (bs, 31, 31, 512)\n\n    norm1 = GroupNormalization(groups=32, axis=-1,\n                               gamma_initializer=gamma_init)(conv)\n\n    leaky_relu = layers.LeakyReLU()(norm1)\n\n    zero_pad2 = layers.ZeroPadding2D()(leaky_relu) # (bs, 33, 33, 512)\n\n    last = layers.Conv2D(1, 4, strides=1,\n                         kernel_initializer=initializer)(zero_pad2) # (bs, 30, 30, 1)\n\n    return tf.keras.Model(inputs=inp, outputs=last)","metadata":{"id":"Qq9kU-7CHCFb"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Build new generators, discriminators, optimizers, and model instance.\n\nmonet_generator = build_generator() # convert photos to Monet\nphoto_generator = build_generator() # convert Monet to photos\n\nmonet_discriminator = build_discriminator() # determines real Monet vs generated Monet (real vs fake)\nphoto_discriminator = build_discriminator() # determines real photos vs generated photos (real vs fake)\n\nmonet_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\nphoto_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n\nmonet_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\nphoto_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n\ncycle_gan_model = CycleGAN(\n    monet_generator, photo_generator, monet_discriminator, photo_discriminator\n)\n\ncycle_gan_model.compile(\n    m_gen_optimizer = monet_generator_optimizer,\n    p_gen_optimizer = photo_generator_optimizer,\n    m_disc_optimizer = monet_discriminator_optimizer,\n    p_disc_optimizer = photo_discriminator_optimizer,\n    gen_loss_fn = generator_loss,\n    disc_loss_fn = discriminator_loss,\n    cycle_loss_fn = calc_cycle_loss,\n    identity_loss_fn = identity_loss\n)\n","metadata":{"id":"b7pVNjOTKbhL"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Train/fit the model using the input Dataset objects.\nh = cycle_gan_model.fit(\n    tf.data.Dataset.zip((monet_ds, photo_ds)),\n    epochs=2,\n    verbose=2,\n)","metadata":{"trusted":true,"id":"gq-plNL4MX46"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"fig, ax = plt.subplots(5, 2, figsize=(12, 12))\nfor i, img in enumerate(photo_ds.take(5)):\n    prediction = monet_generator(img, training=False)[0].numpy()\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    img = (img[0] * 127.5 + 127.5).numpy().astype(np.uint8)\n\n    ax[i, 0].imshow(img)\n    ax[i, 1].imshow(prediction)\n    ax[i, 0].set_title(\"Input Photo\")\n    ax[i, 1].set_title(\"Transformed\")\n    ax[i, 0].axis(\"off\")\n    ax[i, 1].axis(\"off\")\nfig.suptitle(\"Results: 2 epochs (new arch)\", fontsize=16)\nplt.show()","metadata":{"id":"Y_6gfJ1sMhjb"},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Canonical - 2 epochs\n# create_submission()","metadata":{"trusted":true,"id":"WQdH5Y19kHsC"},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"With only two training epochs, we immediately see visualy that the new model architecture as presented in most of the CycleGAN tutorials performs quite well.\n\nAfter submission to the competition, we have a score of 87.00184, far surpasing the baseline model with both 1 epoch and 10 epoch runs.","metadata":{"id":"Tm4u5KRWaraR"}},{"cell_type":"markdown","source":"# Results and Analysis\n\nOverall, the Tutorial model architecture performed best visually as well as for the Kaggle score.\n\n### **Summary of Scores**\n\n| Model | Epochs | Score |\n| :--- | :--- | :--- |\n| Baseline model | 1 | 232.49346 |\n| Baseline model | 10 | 492.09268 |\n| **Tutorial model** | **2** | **87.00184** 🥇 |","metadata":{"id":"YV-YhL4VqMKd"}},{"cell_type":"markdown","source":"### Hyperparameter Tuning Summary\n\nDue to the training time required for the CycleGAN, we did not perform an in-depth hyperparameter search. To the extent that it was possible in the given wallclock time and GPU time, we experimented with small vs. larger training epochs for the Baseline CycleGAN model, which used vanilla `PatchGAN` and `U-Net` architectures with very basic normalization.\n\nOnce the canonical tutorial architectures for the `PatchGAN` and `U-Net` models were impleneted, we could immediately see the benefits of more robust normalization throughout the layers. The baseline model hinted that this might be the case when we saw the `generator` model losses failing to improve over 10 epochs. Additionally, skip connections in the `generator`s helped retain details during downsampling for later use during upsampling.\n\n### Troubleshooting\n\nUsing `GroupNormalization` instead of `InstanceNormalization` required some troubleshooting due to the dimensions expected by the `GroupNormalization`. Once it was understood that we want to normalize over each channel, it became clear that the groups needed to match the number of input channels (along the channel axis, i.e. `-1`).","metadata":{"id":"YxQD_4PxqMKd"}},{"cell_type":"markdown","source":"# Conclusion\n\n### Results Summary\n\nAs mentioned above, the canonical model architectures performed best in the overall CycleGAN due to much better regularization at all layers, along with skip connections in the generator (U-Net) models.\n\n### Learning and Takeaways\n\nThe CycleGAN architecture with embedded models was very interesting and openend a new world of larger challenges that can be addressed via model composition. Learning how to implement a training step using autograd and manual application of gradients to the optimizers was especially enlightning.\n\n### Challenges\n\nMost challenges were technical and not related to the assignment/project itself. Kaggle had autosave failures many times during the project, which resulted in lost work - frustrating to say the least. Additionally, submitting results to the competiton failed sporadically with unhelpful errors. Ultimately, I moved to Colab for implementation iterations before re-importing to Kaggle for submissions.\n\n### Future Improvements\n\nGiven time (wallclock and GPU), it would be interesting to take a deeper dive into what parts of the `generator` and `discriminator` model architectures contributed the most to improving the overall `CycleGAN` performance. For example, comparing various combinations of \"Vanilla U-Net\", \"Canonical U-Net\", \"Vanilla PatchGAN\", and \"Canonical PatchGAN\", combined with diving deeper into standard Hyperparameter search for individual layer components.","metadata":{"id":"YxR8uvrgqMKd"}},{"cell_type":"markdown","source":"# References\n\n* https://keras.io/examples/generative/dcgan_overriding_train_step/\n* https://www.geeksforgeeks.org/machine-learning/cycle-generative-adversarial-network-cyclegan-2/\n* https://www.kaggle.com/code/amyjang/monet-cyclegan-tutorial\n* https://keras.io/examples/generative/cyclegan/\n* https://www.tensorflow.org/api_docs/python/tf/data/Dataset\n* https://www.tensorflow.org/guide/autodiff","metadata":{"id":"mOEttY5XqMKd"}}]}