{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.12.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"tpuV5e8","dataSources":[{"sourceType":"competition","sourceId":21755,"databundleVersionId":1475600}],"dockerImageVersionId":31286,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"!export PATH=\"${HOME}/.local/bin:${PATH}\" && uv pip uninstall --system jax","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:48:48.796903Z","iopub.execute_input":"2026-03-07T04:48:48.797080Z","iopub.status.idle":"2026-03-07T04:48:49.628504Z","shell.execute_reply.started":"2026-03-07T04:48:48.797062Z","shell.execute_reply":"2026-03-07T04:48:49.627604Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"!export PATH=\"${HOME}/.local/bin:${PATH}\" && uv pip install --system tensorflow-tpu==\"2.18.0\" --find-links https://storage.googleapis.com/libtpu-tf-releases/index.html","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:48:51.971200Z","iopub.execute_input":"2026-03-07T04:48:51.971426Z","iopub.status.idle":"2026-03-07T04:49:06.170875Z","shell.execute_reply.started":"2026-03-07T04:48:51.971404Z","shell.execute_reply":"2026-03-07T04:49:06.169885Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import tensorflow as tf\nprint('TensorFlow version' + tf.__version__)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:49:06.171500Z","iopub.execute_input":"2026-03-07T04:49:06.171677Z","iopub.status.idle":"2026-03-07T04:49:21.329130Z","shell.execute_reply.started":"2026-03-07T04:49:06.171657Z","shell.execute_reply":"2026-03-07T04:49:21.327902Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import tensorflow as tf\n\n# Detect and initialize TPU\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver(tpu=\"local\")  \n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.TPUStrategy(tpu)\n    print(f\"Running on TPU: {tpu.master()}\")\nexcept Exception as e:\n    print(f\"Could not initialize TPU: {e}\")\n    strategy = tf.distribute.get_strategy()  # Fallback to default CPU/GPU\n    print(\"Running on CPU or single GPU\")\n\n# Print number of available TPU cores\nprint(\"Number of devices:\", strategy.num_replicas_in_sync, \"🚀\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:49:25.447828Z","iopub.execute_input":"2026-03-07T04:49:25.448455Z","iopub.status.idle":"2026-03-07T04:49:40.860169Z","shell.execute_reply.started":"2026-03-07T04:49:25.448434Z","shell.execute_reply":"2026-03-07T04:49:40.858976Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport tensorflow as tf\nimport os\nimport matplotlib.pyplot as plt\nfrom tensorflow.keras import layers, Model\nimport gc","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:49:40.860959Z","iopub.execute_input":"2026-03-07T04:49:40.861181Z","iopub.status.idle":"2026-03-07T04:49:40.865301Z","shell.execute_reply.started":"2026-03-07T04:49:40.861163Z","shell.execute_reply":"2026-03-07T04:49:40.864452Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Problem Statement\nThe goal of this project is to train a CycleGAN model capable of transforming real-world landscape photographs into paintings in the style of Claude Monet. The dataset contains two unpaired domains: real photos (domain X) and Monet paintings (domain Y). Because there is no direct one-to-one correspondence between individual photos and Monet paintings, this task is considered an unsupervised image-to-image translation problem.\n\nCycleGAN addresses this challenge using two key ideas. First, the adversarial component of the model trains generators to produce images that resemble the target domain, while discriminators attempt to distinguish between real and generated images. This adversarial loss encourages the generators to produce outputs that visually match the Monet painting distribution. Second, the cycle-consistency loss ensures that if an image is translated from one domain to the other and back again, the result should resemble the original image. This constraint helps the model learn meaningful mappings between domains despite the lack of paired training data. Additionally, an identity loss is used to encourage the generators to preserve important characteristics such as color composition and structural content when images are already in the target domain.\n\nEarlier experiments showed that increasing model capacity to nine residual blocks and training for forty epochs significantly improved stylistic quality compared to the baseline configuration. Building on these results, the goal of the current phase of the project is to further improve the visual quality and stability of the generated images through additional training refinements. Once the model is trained, it will be used to generate between 7,000 and 10,000 Monet-style images for submission. Before training the model, we first examine and preprocess the dataset used for training.","metadata":{}},{"cell_type":"markdown","source":"## EDA","metadata":{}},{"cell_type":"code","source":"MONET_JPG_DATA_DIR = \"/kaggle/input/competitions/gan-getting-started/monet_jpg\"\nMONET_TF_DATA_DIR = \"/kaggle/input/competitions/gan-getting-started/monet_tfrec\"\nPHOTO_JPG_DATA_DIR = \"/kaggle/input/competitions/gan-getting-started/photo_jpg\"\nPHOTO_TF_DATA_DIR = \"/kaggle/input/competitions/gan-getting-started/photo_tfrec\"\n\nmonet_jpg_filenames = tf.io.gfile.listdir(MONET_JPG_DATA_DIR)\nprint(\"Number of Monet JPG images:\", len(monet_jpg_filenames))\nmonet_tfrec_filenames = tf.io.gfile.listdir(MONET_TF_DATA_DIR)\nprint(\"Number of Monet TFRecord files:\", len(monet_tfrec_filenames))\nphoto_jpg_filenames = tf.io.gfile.listdir(PHOTO_JPG_DATA_DIR)\nprint(\"Number of photo jpg images:\", len(photo_jpg_filenames))\nphoto_tfrec_filenames = tf.io.gfile.listdir(PHOTO_TF_DATA_DIR)\nprint(\"Number of photo TFRecord files:\", len(photo_tfrec_filenames))\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:49:45.051993Z","iopub.execute_input":"2026-03-07T04:49:45.052304Z","iopub.status.idle":"2026-03-07T04:49:45.412913Z","shell.execute_reply.started":"2026-03-07T04:49:45.052284Z","shell.execute_reply":"2026-03-07T04:49:45.411759Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We have a set of 300 Monet paitings and 7,038 real photos in jpg format which are also offered in tfrecord format. We can visually inspect the jpg files and use the tfrecord files to train our tensorflow model. First let's take a look at the size of images.","metadata":{}},{"cell_type":"code","source":"def load_image(path):\n    img = tf.io.read_file(path)\n    img = tf.image.decode_jpeg(img, channels=3)\n    return img\n\ndef summarize_image_sizes(directory):\n        filenames = tf.io.gfile.listdir(directory)\n        heights, widths = [], []\n\ndef summarize_image_sizes(directory):\n        filenames = tf.io.gfile.listdir(directory)\n        heights, widths = [], []","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:49:47.742874Z","iopub.execute_input":"2026-03-07T04:49:47.743220Z","iopub.status.idle":"2026-03-07T04:49:47.747550Z","shell.execute_reply.started":"2026-03-07T04:49:47.743200Z","shell.execute_reply":"2026-03-07T04:49:47.746545Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"for directory in [MONET_JPG_DATA_DIR, PHOTO_JPG_DATA_DIR]:\n    filenames = tf.io.gfile.listdir(directory)\n    heights, widths, colors = [], [], []\n\n    for f in filenames:\n        path = os.path.join(directory, f)\n        img = load_image(path)\n        h, w, c = img.shape\n        heights.append(h)\n        widths.append(w)\n        colors.append(c)\n\n    heights = np.array(heights)\n    widths = np.array(widths)\n    colors = np.array(colors)\n\n    print(f\"\\nDirectory: {directory}\")\n    print(f\"Total images: {len(filenames)}\")\n    print(f\"Min size: {heights.min()} x {widths.min()}\")\n    print(f\"Max size: {heights.max()} x {widths.max()}\")\n    print(f\"Max colors: {colors.max()} x {colors.max()}\")\n    print(f\"Min colors: {colors.min()} x {colors.min()}\")\n    print(f\"Average size: {heights.mean():.1f} x {widths.mean():.1f}\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:49:49.437327Z","iopub.execute_input":"2026-03-07T04:49:49.437522Z","iopub.status.idle":"2026-03-07T04:50:45.114862Z","shell.execute_reply.started":"2026-03-07T04:49:49.437507Z","shell.execute_reply":"2026-03-07T04:50:45.113556Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Our images are all 256x256 pixels in size with 3 color channels (256x256x3). Let's check out some Monet paintings and then some real photos.","metadata":{}},{"cell_type":"code","source":"def load_image(path):\n    img = tf.io.read_file(path)\n    img = tf.image.decode_jpeg(img, channels=3)\n    return img\n\nsample_paths = [os.path.join(MONET_JPG_DATA_DIR, f) for f in monet_jpg_filenames[:6]]\n\nplt.figure(figsize=(12, 8))\nfor i, p in enumerate(sample_paths):\n    img = load_image(p)\n    plt.subplot(2, 3, i+1)\n    plt.imshow(img)\n    plt.axis(\"off\")\nplt.savefig(\"MonetEDA.png\", dpi=300, bbox_inches='tight')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:50:45.115686Z","iopub.execute_input":"2026-03-07T04:50:45.115881Z","iopub.status.idle":"2026-03-07T04:50:45.825564Z","shell.execute_reply.started":"2026-03-07T04:50:45.115863Z","shell.execute_reply":"2026-03-07T04:50:45.824425Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"sample_paths = [os.path.join(PHOTO_JPG_DATA_DIR, f) for f in photo_jpg_filenames[:6]]\n\nplt.figure(figsize=(12, 8))\nfor i, p in enumerate(sample_paths):\n    img = load_image(p)\n    plt.subplot(2, 3, i+1)\n    plt.imshow(img)\n    plt.axis(\"off\")\nplt.savefig(\"PhotoEDA\", dpi=300, bbox_inches='tight')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:50:53.898673Z","iopub.execute_input":"2026-03-07T04:50:53.898990Z","iopub.status.idle":"2026-03-07T04:50:54.488364Z","shell.execute_reply.started":"2026-03-07T04:50:53.898955Z","shell.execute_reply":"2026-03-07T04:50:54.487219Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"It looks like the set of real photos share some similarities with the Monet paintings. They both focus on landscapes or buildings and usually have some view of the sky as well. This will make it easier to learn the mapping from one domain to the other. Next let's look at the distribution of pixel intensities of each color in our two sets of images.","metadata":{}},{"cell_type":"code","source":"def get_channel_values(directory, max_images=200):\n    filenames = tf.io.gfile.listdir(directory)\n    r_vals, g_vals, b_vals = [], [], []\n\n    for f in filenames[:max_images]:\n        path = os.path.join(directory, f)\n        img = load_image(path).numpy()\n\n        r_vals.extend(img[:,:,0].flatten())\n        g_vals.extend(img[:,:,1].flatten())\n        b_vals.extend(img[:,:,2].flatten())\n\n    return np.array(r_vals), np.array(g_vals), np.array(b_vals)\ndef plot_histograms(r, g, b, title):\n    plt.figure(figsize=(12, 4))\n\n    plt.subplot(1, 3, 1)\n    plt.hist(r, bins=50, color='red', alpha=0.7)\n    plt.title(f'{title} – Red (0–255)')\n\n    plt.subplot(1, 3, 2)\n    plt.hist(g, bins=50, color='green', alpha=0.7)\n    plt.title(f'{title} – Green (0–255)')\n\n    plt.subplot(1, 3, 3)\n    plt.hist(b, bins=50, color='blue', alpha=0.7)\n    plt.title(f'{title} – Blue (0–255)')\n\n    plt.tight_layout()\n    plt.show()\n    ","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:51:00.463212Z","iopub.execute_input":"2026-03-07T04:51:00.463633Z","iopub.status.idle":"2026-03-07T04:51:00.469721Z","shell.execute_reply.started":"2026-03-07T04:51:00.463612Z","shell.execute_reply":"2026-03-07T04:51:00.468639Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"r_m, g_m, b_m = get_channel_values(MONET_JPG_DATA_DIR)\nplot_histograms(r_m, g_m, b_m, \"Monet\")\nplt.savefig(\"MonetHistogram\", dpi=200, bbox_inches=\"tight\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:51:02.013108Z","iopub.execute_input":"2026-03-07T04:51:02.013441Z","iopub.status.idle":"2026-03-07T04:51:06.577674Z","shell.execute_reply.started":"2026-03-07T04:51:02.013418Z","shell.execute_reply":"2026-03-07T04:51:06.576369Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"r_p, g_p, b_p = get_channel_values(PHOTO_JPG_DATA_DIR)\nplot_histograms(r_p, g_p, b_p, \"Photo\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:51:07.952047Z","iopub.execute_input":"2026-03-07T04:51:07.952356Z","iopub.status.idle":"2026-03-07T04:51:12.465574Z","shell.execute_reply.started":"2026-03-07T04:51:07.952336Z","shell.execute_reply":"2026-03-07T04:51:12.464351Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The Monet paintings appear to have warmer colors (higher intensity of the red), and sharper peaks across all colors. The wider distribution of colors in the real photos mean they have more dynamic contrast (more extreme highlights and darker shadows). The cycleGAN is going to learn the mapping between photo and Monet pixel distributions. It will be best to scale these pixel values between 0-1 before we train the model.","metadata":{}},{"cell_type":"markdown","source":"## Plan of analysis\n\nWe begin by exploring the provided image domains: 300 Monet paintings and 7,038 real photos, each with a uniform resolution of (256 × 256 × 3). All images are rescaled from the original pixel range of [0, 255] to [0, 1] to stabilize optimization and make them suitable for neural network training. Using these preprocessed datasets, we first implement and train a baseline CycleGAN model on TPU to learn bidirectional mappings between the photo and Monet domains. Initial experiments evaluate the effects of training duration and model capacity, leading to an improved configuration using nine residual blocks and forty training epochs.\n\nBuilding on the results of this improved model, the next stage of analysis focuses on refining training strategies to further enhance image quality and stylistic consistency. These refinements include applying data augmentation techniques such as random horizontal flips, crops, and mild color perturbations to improve generalization, as well as experimenting with training adjustments such as learning rate scheduling and other hyperparameter tuning. Through these iterative improvements, the objective is to generate more visually convincing Monet-style translations while preserving important structural elements from the original photographs.","metadata":{}},{"cell_type":"markdown","source":"# Model Architecture\nI'll be using the TPUv5e-8 accelerator to speed up training time, so I need to define a stragey to use it. Each replica is given a slice of the batch.","metadata":{}},{"cell_type":"code","source":"tf.config.list_logical_devices()\n\nBATCH_SIZE = 1\nGLOBAL_BATCH = BATCH_SIZE * strategy.num_replicas_in_sync\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:51:47.618759Z","iopub.execute_input":"2026-03-07T04:51:47.619194Z","iopub.status.idle":"2026-03-07T04:51:47.623050Z","shell.execute_reply.started":"2026-03-07T04:51:47.619173Z","shell.execute_reply":"2026-03-07T04:51:47.621934Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We'll be using the tfrec files to do the training, let's load those in being sure to normalize pixel values.","metadata":{}},{"cell_type":"code","source":"IMG_SIZE = 256\n\ndef parse_tfrecord(example):\n    feature_description = {\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n    }\n    example = tf.io.parse_single_example(example, feature_description)\n    img = tf.io.decode_jpeg(example[\"image\"], channels=3)\n    img = tf.image.convert_image_dtype(img, tf.float32)  # [0,1]\n    img = tf.image.resize(img, [IMG_SIZE, IMG_SIZE])\n    return img","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:51:49.402652Z","iopub.execute_input":"2026-03-07T04:51:49.402941Z","iopub.status.idle":"2026-03-07T04:51:49.407574Z","shell.execute_reply.started":"2026-03-07T04:51:49.402918Z","shell.execute_reply":"2026-03-07T04:51:49.406422Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def load_tfrecord_dataset(tfrecord_dir, batch_size):\n    files = tf.io.gfile.glob(tfrecord_dir + \"/*.tfrec\")\n    ds = tf.data.TFRecordDataset(files, num_parallel_reads=tf.data.AUTOTUNE)\n    ds = ds.map(parse_tfrecord, num_parallel_calls=tf.data.AUTOTUNE)\n    ds = ds.batch(batch_size, drop_remainder=True)\n    ds = ds.prefetch(tf.data.AUTOTUNE)\n    return ds","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:51:50.313067Z","iopub.execute_input":"2026-03-07T04:51:50.313282Z","iopub.status.idle":"2026-03-07T04:51:50.317085Z","shell.execute_reply.started":"2026-03-07T04:51:50.313263Z","shell.execute_reply":"2026-03-07T04:51:50.316137Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"monet_ds = load_tfrecord_dataset(MONET_TF_DATA_DIR, GLOBAL_BATCH)\nphoto_ds = load_tfrecord_dataset(PHOTO_TF_DATA_DIR, GLOBAL_BATCH)\n\ntrain_ds = tf.data.Dataset.zip((monet_ds, photo_ds))\ntrain_ds = strategy.experimental_distribute_dataset(train_ds)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:51:51.217645Z","iopub.execute_input":"2026-03-07T04:51:51.218068Z","iopub.status.idle":"2026-03-07T04:51:51.329987Z","shell.execute_reply.started":"2026-03-07T04:51:51.218037Z","shell.execute_reply":"2026-03-07T04:51:51.328822Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Next, we'll define our generator architecture. I decided to go with a ResNet style network which are known to perform well in computer vision applications. This network be used to translate from one image domain to another. It has downsampling to extract features, residual blocks to transform the content, and upsampling to reconstruct an image in the opposite domain. From previous work, 9 residual blocks was found to be optimal.","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow.keras import layers\n\nclass InstanceNormalization(layers.Layer):\n    def __init__(self, epsilon=1e-5):\n        super().__init__()\n        self.epsilon = epsilon\n\n    def build(self, input_shape):\n        channels = input_shape[-1]\n        self.gamma = self.add_weight(\n            shape=(channels,),\n            initializer=\"ones\",\n            trainable=True)\n        self.beta = self.add_weight(\n            shape=(channels,),\n            initializer=\"zeros\",\n            trainable=True)\n\n    def call(self, x):\n        mean, var = tf.nn.moments(x, axes=[1, 2], keepdims=True)\n        return self.gamma * (x - mean) / tf.sqrt(var + self.epsilon) + self.beta","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:51:53.078731Z","iopub.execute_input":"2026-03-07T04:51:53.079033Z","iopub.status.idle":"2026-03-07T04:51:53.083919Z","shell.execute_reply.started":"2026-03-07T04:51:53.079012Z","shell.execute_reply":"2026-03-07T04:51:53.082999Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def residual_block(x, filters):\n    init = tf.random_normal_initializer(0., 0.02)\n\n    y = layers.Conv2D(filters, 3, padding='same', kernel_initializer=init, use_bias=False)(x)\n    y = InstanceNormalization()(y)\n    y = layers.ReLU()(y)\n\n    y = layers.Conv2D(filters, 3, padding='same', kernel_initializer=init, use_bias=False)(y)\n    y = InstanceNormalization()(y)\n\n    return layers.Add()([x, y])\n\ndef build_generator(img_size=256, num_res_blocks=9):\n    init = tf.random_normal_initializer(0., 0.02)\n    inputs = layers.Input(shape=(img_size, img_size, 3))\n\n    # Initial conv\n    x = layers.Conv2D(64, 7, padding='same', kernel_initializer=init, use_bias=False)(inputs)\n    x = InstanceNormalization()(x)\n    x = layers.ReLU()(x)\n\n    # Downsample\n    x = layers.Conv2D(128, 3, strides=2, padding='same', kernel_initializer=init, use_bias=False)(x)\n    x = InstanceNormalization()(x)\n    x = layers.ReLU()(x)\n\n    x = layers.Conv2D(256, 3, strides=2, padding='same', kernel_initializer=init, use_bias=False)(x)\n    x = InstanceNormalization()(x)\n    x = layers.ReLU()(x)\n\n    # Residual blocks\n    for _ in range(num_res_blocks):\n        x = residual_block(x, 256)\n\n    # Upsample\n    x = layers.Conv2DTranspose(128, 3, strides=2, padding='same', kernel_initializer=init, use_bias=False)(x)\n    x = InstanceNormalization()(x)\n    x = layers.ReLU()(x)\n\n    x = layers.Conv2DTranspose(64, 3, strides=2, padding='same', kernel_initializer=init, use_bias=False)(x)\n    x = InstanceNormalization()(x)\n    x = layers.ReLU()(x)\n\n    # Output layer (tanh)\n    outputs = layers.Conv2D(3, 7, padding='same', kernel_initializer=init, activation='tanh')(x)\n\n    return Model(inputs, outputs, name=\"generator\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:51:54.845137Z","iopub.execute_input":"2026-03-07T04:51:54.845450Z","iopub.status.idle":"2026-03-07T04:51:54.852186Z","shell.execute_reply.started":"2026-03-07T04:51:54.845429Z","shell.execute_reply":"2026-03-07T04:51:54.851207Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"For the discriminator, I will use a 70x70 PatchGAN. This looks at 70x70 overlapping patches of images coming out of the generator vs real images and classifies as fake or real. Looking at just patches reduces the number of parameters and speeds up our training times.","metadata":{}},{"cell_type":"code","source":"def build_discriminator(img_size=256):\n    init = tf.random_normal_initializer(0., 0.02)\n    inputs = layers.Input(shape=(img_size, img_size, 3))\n\n    x = layers.Conv2D(64, 4, strides=2, padding='same', kernel_initializer=init)(inputs)\n    x = layers.LeakyReLU(0.2)(x)\n\n    x = layers.Conv2D(128, 4, strides=2, padding='same', kernel_initializer=init, use_bias=False)(x)\n    x = InstanceNormalization()(x)\n    x = layers.LeakyReLU(0.2)(x)\n\n    x = layers.Conv2D(256, 4, strides=2, padding='same', kernel_initializer=init, use_bias=False)(x)\n    x = InstanceNormalization()(x)\n    x = layers.LeakyReLU(0.2)(x)\n\n    x = layers.Conv2D(512, 4, strides=1, padding='same', kernel_initializer=init, use_bias=False)(x)\n    x = InstanceNormalization()(x)\n    x = layers.LeakyReLU(0.2)(x)\n\n    outputs = layers.Conv2D(1, 4, strides=1, padding='same', kernel_initializer=init)(x)\n\n    return Model(inputs, outputs, name=\"discriminator\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:51:57.908442Z","iopub.execute_input":"2026-03-07T04:51:57.908759Z","iopub.status.idle":"2026-03-07T04:51:57.913893Z","shell.execute_reply.started":"2026-03-07T04:51:57.908739Z","shell.execute_reply":"2026-03-07T04:51:57.912997Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We'll put this together in the TPU strategy.","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    G_XtoY = build_generator()   # photo → Monet\n    G_YtoX = build_generator()   # Monet → photo\n    D_X = build_discriminator()  # real photos\n    D_Y = build_discriminator()  # real Monet","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:52:00.439253Z","iopub.execute_input":"2026-03-07T04:52:00.439572Z","iopub.status.idle":"2026-03-07T04:52:07.800665Z","shell.execute_reply.started":"2026-03-07T04:52:00.439551Z","shell.execute_reply":"2026-03-07T04:52:07.799344Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We need to define our loss functions as discussed earlier there are 3 different functions:\n1. Adversarial loss encourages each generator to produce outputs that fool the corresponding discriminator.\n2. Cycle‑consistency loss enforces that translating an image to the other domain and back returns something close to the original.\n3. Identity loss stabilizes color mapping by penalizing unnecessary changes when an image is already in the target domain.","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    bce = tf.keras.losses.BinaryCrossentropy(from_logits=True)\n\n    def generator_adversarial_loss(fake_logits):\n        return bce(tf.ones_like(fake_logits), fake_logits)\n\n    def discriminator_loss(real_logits, fake_logits):\n        real_loss = bce(tf.ones_like(real_logits), real_logits)\n        fake_loss = bce(tf.zeros_like(fake_logits), fake_logits)\n        return (real_loss + fake_loss) * 0.5\n\n    def cycle_loss(real, cycled, lambda_cycle=10.0):\n        return lambda_cycle * tf.reduce_mean(tf.abs(real - cycled))\n\n    def identity_loss(real, same, lambda_identity=5.0):\n        return lambda_identity * tf.reduce_mean(tf.abs(real - same))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:52:11.802367Z","iopub.execute_input":"2026-03-07T04:52:11.802675Z","iopub.status.idle":"2026-03-07T04:52:11.808109Z","shell.execute_reply.started":"2026-03-07T04:52:11.802655Z","shell.execute_reply":"2026-03-07T04:52:11.807000Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We'll use an Adam optimizer for each network with some parameters known to be stable with GANs","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    G_XtoY_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    G_YtoX_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    D_X_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    D_Y_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:52:13.705760Z","iopub.execute_input":"2026-03-07T04:52:13.706022Z","iopub.status.idle":"2026-03-07T04:52:13.754129Z","shell.execute_reply.started":"2026-03-07T04:52:13.706002Z","shell.execute_reply":"2026-03-07T04:52:13.752990Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Finally lets setup the training step including forward passes, computing loss, and gradient updates to the weights.","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    @tf.function\n    def train_step(batch):\n        monet, photo = batch\n\n        # scale [0,1] → [-1,1]\n        monet_in = monet * 2.0 - 1.0\n        photo_in = photo * 2.0 - 1.0\n\n        with tf.GradientTape(persistent=True) as tape:\n            fake_monet = G_XtoY(photo_in, training=True)\n            fake_photo = G_YtoX(monet_in, training=True)\n\n            cycled_photo = G_YtoX(fake_monet, training=True)\n            cycled_monet = G_XtoY(fake_photo, training=True)\n\n            same_monet = G_XtoY(monet_in, training=True)\n            same_photo = G_YtoX(photo_in, training=True)\n\n            disc_real_monet = D_Y(monet_in, training=True)\n            disc_real_photo = D_X(photo_in, training=True)\n            disc_fake_monet = D_Y(fake_monet, training=True)\n            disc_fake_photo = D_X(fake_photo, training=True)\n\n            gen_XtoY_loss = generator_adversarial_loss(disc_fake_monet)\n            gen_YtoX_loss = generator_adversarial_loss(disc_fake_photo)\n\n            total_cycle_loss = cycle_loss(photo_in, cycled_photo) + cycle_loss(monet_in, cycled_monet)\n            total_identity_loss = identity_loss(monet_in, same_monet) + identity_loss(photo_in, same_photo)\n\n            total_gen_XtoY = gen_XtoY_loss + total_cycle_loss + total_identity_loss\n            total_gen_YtoX = gen_YtoX_loss + total_cycle_loss + total_identity_loss\n\n            disc_X_loss = discriminator_loss(disc_real_photo, disc_fake_photo)\n            disc_Y_loss = discriminator_loss(disc_real_monet, disc_fake_monet)\n\n        G_XtoY_grads = tape.gradient(total_gen_XtoY, G_XtoY.trainable_variables)\n        G_YtoX_grads = tape.gradient(total_gen_YtoX, G_YtoX.trainable_variables)\n        D_X_grads = tape.gradient(disc_X_loss, D_X.trainable_variables)\n        D_Y_grads = tape.gradient(disc_Y_loss, D_Y.trainable_variables)\n\n        G_XtoY_optimizer.apply_gradients(zip(G_XtoY_grads, G_XtoY.trainable_variables))\n        G_YtoX_optimizer.apply_gradients(zip(G_YtoX_grads, G_YtoX.trainable_variables))\n        D_X_optimizer.apply_gradients(zip(D_X_grads, D_X.trainable_variables))\n        D_Y_optimizer.apply_gradients(zip(D_Y_grads, D_Y.trainable_variables))\n\n        return {\n            \"G_XtoY\": total_gen_XtoY,\n            \"G_YtoX\": total_gen_YtoX,\n            \"D_X\": disc_X_loss,\n            \"D_Y\": disc_Y_loss,\n        }\n\n    @tf.function\n    def distributed_train_step(batch):\n        return strategy.run(train_step, args=(batch,))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:52:18.296060Z","iopub.execute_input":"2026-03-07T04:52:18.296401Z","iopub.status.idle":"2026-03-07T04:52:18.304355Z","shell.execute_reply.started":"2026-03-07T04:52:18.296381Z","shell.execute_reply":"2026-03-07T04:52:18.303263Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Results and Analysis","metadata":{}},{"cell_type":"markdown","source":"## Baseline\nWe're ready to train this baseline model. It has 9 residual blocks and trains for 40 epochs.","metadata":{}},{"cell_type":"code","source":"\n# ---------------------------\n# Experiment name\n# ---------------------------\nEXP_NAME = \"baseline\"\n\nEPOCHS = 40\n\nG_XtoY_history = []\nG_YtoX_history = []\nD_X_history = []\nD_Y_history = []\n\nfor epoch in range(EPOCHS):\n    epoch_G_XtoY = []\n    epoch_G_YtoX = []\n    epoch_D_X = []\n    epoch_D_Y = []\n\n    for step, batch in enumerate(train_ds):\n        per_replica = distributed_train_step(batch)\n\n        # reduce across TPU replicas\n        G_XtoY_loss = strategy.reduce(tf.distribute.ReduceOp.MEAN, per_replica[\"G_XtoY\"], axis=None)\n        G_YtoX_loss = strategy.reduce(tf.distribute.ReduceOp.MEAN, per_replica[\"G_YtoX\"], axis=None)\n        D_X_loss = strategy.reduce(tf.distribute.ReduceOp.MEAN, per_replica[\"D_X\"], axis=None)\n        D_Y_loss = strategy.reduce(tf.distribute.ReduceOp.MEAN, per_replica[\"D_Y\"], axis=None)\n\n        epoch_G_XtoY.append(G_XtoY_loss.numpy())\n        epoch_G_YtoX.append(G_YtoX_loss.numpy())\n        epoch_D_X.append(D_X_loss.numpy())\n        epoch_D_Y.append(D_Y_loss.numpy())\n\n    # store epoch averages\n    G_XtoY_history.append(sum(epoch_G_XtoY) / len(epoch_G_XtoY))\n    G_YtoX_history.append(sum(epoch_G_YtoX) / len(epoch_G_YtoX))\n    D_X_history.append(sum(epoch_D_X) / len(epoch_D_X))\n    D_Y_history.append(sum(epoch_D_Y) / len(epoch_D_Y))\n\n    print(f\"{EXP_NAME} | Epoch {epoch+1}/{EPOCHS} completed\")\n\n\n# ---------------------------\n# Save models\n# ---------------------------\nG_XtoY.export(f\"/kaggle/working/{EXP_NAME}_G_XtoY\")\nG_YtoX.export(f\"/kaggle/working/{EXP_NAME}_G_YtoX\")\nD_X.export(f\"/kaggle/working/{EXP_NAME}_D_X\")\nD_Y.export(f\"/kaggle/working/{EXP_NAME}_D_Y\")\n\nG_XtoY.save_weights(f\"/kaggle/working/{EXP_NAME}_G_XtoY.weights.h5\")\nG_YtoX.save_weights(f\"/kaggle/working/{EXP_NAME}_G_YtoX.weights.h5\")\nD_X.save_weights(f\"/kaggle/working/{EXP_NAME}_D_X.weights.h5\")\nD_Y.save_weights(f\"/kaggle/working/{EXP_NAME}_D_Y.weights.h5\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T02:46:48.075946Z","iopub.execute_input":"2026-03-07T02:46:48.076234Z","iopub.status.idle":"2026-03-07T02:52:50.754876Z","shell.execute_reply.started":"2026-03-07T02:46:48.076213Z","shell.execute_reply":"2026-03-07T02:52:50.753453Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Plotting loss","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(12, 6))\n\nplt.plot(G_XtoY_history, label=\"Generator X→Y (Photo→Monet)\")\nplt.plot(G_YtoX_history, label=\"Generator Y→X (Monet→Photo)\")\nplt.plot(D_X_history, label=\"Discriminator X (Photo)\")\nplt.plot(D_Y_history, label=\"Discriminator Y (Monet)\")\n\nplt.xlabel(\"Epoch\")\nplt.ylabel(\"Loss\")\nplt.title(\"CycleGAN Training Losses\")\nplt.legend()\nplt.grid(True)\nplt.savefig(f\"{EXP_NAME}_LossPlots.png\", dpi=300, bbox_inches='tight')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T03:06:36.995072Z","iopub.execute_input":"2026-03-07T03:06:36.995380Z","iopub.status.idle":"2026-03-07T03:06:37.379241Z","shell.execute_reply.started":"2026-03-07T03:06:36.995361Z","shell.execute_reply":"2026-03-07T03:06:37.377944Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The loss curves indicate stable adversarial training, with generator losses steadily decreasing and discriminator losses remaining low and consistent. The slower rate of improvement in later epochs suggests that the model enters a refinement stage where smaller optimization steps may be beneficial. This is why my next idea was to introduce a learning rate decay to encourage smoother convergence and reduce the likelihood of late-stage artifacts. Let's generate some fake Monet paintings and take a look.","metadata":{}},{"cell_type":"code","source":"@tf.function\ndef generate_step(batch):\n    batch_in = batch * 2.0 - 1.0\n    return G_XtoY(batch_in, training=False)\n\n# Run on TPU and gather results\nper_replica = strategy.run(generate_step, args=(sample_photos,))\nfake_monet = strategy.gather(per_replica, axis=0)\n\n# Convert back to [0,1]\nfake_monet = (fake_monet + 1.0) / 2.0\nfake_monet = tf.clip_by_value(fake_monet, 0.0, 1.0)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T03:08:09.537104Z","iopub.execute_input":"2026-03-07T03:08:09.537393Z","iopub.status.idle":"2026-03-07T03:08:16.209569Z","shell.execute_reply.started":"2026-03-07T03:08:09.537374Z","shell.execute_reply":"2026-03-07T03:08:16.208014Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Take 10 images (unbatched)\nsample_photos = []\nfor img in photo_ds.unbatch().take(10):\n    sample_photos.append(img)\nsample_photos = tf.stack(sample_photos, axis=0)   # shape (10,256,256,3)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T03:08:46.669430Z","iopub.execute_input":"2026-03-07T03:08:46.669754Z","iopub.status.idle":"2026-03-07T03:08:46.721344Z","shell.execute_reply.started":"2026-03-07T03:08:46.669734Z","shell.execute_reply":"2026-03-07T03:08:46.720069Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nplt.figure(figsize=(20, 8))\n\nfor i in range(10):\n    # original photo\n    plt.subplot(2, 10, i + 1)\n    plt.imshow(sample_photos[i])\n    plt.axis(\"off\")\n    plt.title(\"Photo\")\n\n    # generated Monet\n    plt.subplot(2, 10, 10 + i + 1)\n    plt.imshow(fake_monet[i])\n    plt.axis(\"off\")\n    plt.title(\"Monet\")\n    \nplt.tight_layout()\nplt.savefig(f\"{EXP_NAME}_Results.png\", dpi=300, bbox_inches='tight')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T03:08:48.800723Z","iopub.execute_input":"2026-03-07T03:08:48.801007Z","iopub.status.idle":"2026-03-07T03:08:53.570144Z","shell.execute_reply.started":"2026-03-07T03:08:48.800987Z","shell.execute_reply":"2026-03-07T03:08:53.568798Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"I'm pretty satisfied with these results they appear to correctly preserve the content of original photos. The color palete is softer and texture does resemble the distinct brushstroke used by Monet. It is interesting how different runs can produce varying results even with the same hyperparameters because GAN training is stochastic. Certain images are prone to artifacts (the 4th and 8th from the left) and this could be due to the use of Conv2dTranspose in the generator as it is known have this behavoir. Another possibility is that, in later epochs, the weights are being updated too aggresively. Let's see if learning rate decay addresses some of the issues.","metadata":{}},{"cell_type":"markdown","source":"## Learning Rate Decay","metadata":{}},{"cell_type":"markdown","source":"One thing I wanted to try adding was a decay to the learning rate. So far, we have used a constant learning rate in our Adam optimizer of 2e-4. It is possible that the generator is continuing to make large updates even after the model has converged. I am going to add a function that will start reducing the learning rate after epoch 20.","metadata":{}},{"cell_type":"code","source":"INITIAL_LR = 2e-4\nDECAY_START_EPOCH = 20\nTOTAL_EPOCHS = 40\n\ndef get_lr(epoch):\n    if epoch < DECAY_START_EPOCH:\n        return INITIAL_LR\n    lr = INITIAL_LR * (1 - (epoch - DECAY_START_EPOCH) / (TOTAL_EPOCHS - DECAY_START_EPOCH))\n    return max(lr, 1e-6)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T05:18:06.324062Z","iopub.execute_input":"2026-03-07T05:18:06.324420Z","iopub.status.idle":"2026-03-07T05:18:06.328855Z","shell.execute_reply.started":"2026-03-07T05:18:06.324397Z","shell.execute_reply":"2026-03-07T05:18:06.327645Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Now I need to rebuild the generators, discriminators, optimizers, and all @tf.functions. Resetting variables from the previous model. We'll set our learning rate with this new function.","metadata":{}},{"cell_type":"code","source":"tf.keras.backend.clear_session()\ngc.collect()\n\nG_XtoY_history = []\nG_YtoX_history = []\nD_X_history = []\nD_Y_history = []\n\nwith strategy.scope():\n    G_XtoY = build_generator()\n    G_YtoX = build_generator()\n    D_X = build_discriminator()\n    D_Y = build_discriminator()\n\n    G_XtoY_optimizer = tf.keras.optimizers.Adam(INITIAL_LR, beta_1=0.5)\n    G_YtoX_optimizer = tf.keras.optimizers.Adam(INITIAL_LR, beta_1=0.5)\n    D_X_optimizer = tf.keras.optimizers.Adam(INITIAL_LR, beta_1=0.5)\n    D_Y_optimizer = tf.keras.optimizers.Adam(INITIAL_LR, beta_1=0.5)\nwith strategy.scope():\n    @tf.function\n    def train_step(batch):\n        monet, photo = batch\n\n        # scale [0,1] → [-1,1]\n        monet_in = monet * 2.0 - 1.0\n        photo_in = photo * 2.0 - 1.0\n\n        with tf.GradientTape(persistent=True) as tape:\n            fake_monet = G_XtoY(photo_in, training=True)\n            fake_photo = G_YtoX(monet_in, training=True)\n\n            cycled_photo = G_YtoX(fake_monet, training=True)\n            cycled_monet = G_XtoY(fake_photo, training=True)\n\n            same_monet = G_XtoY(monet_in, training=True)\n            same_photo = G_YtoX(photo_in, training=True)\n\n            disc_real_monet = D_Y(monet_in, training=True)\n            disc_real_photo = D_X(photo_in, training=True)\n            disc_fake_monet = D_Y(fake_monet, training=True)\n            disc_fake_photo = D_X(fake_photo, training=True)\n\n            gen_XtoY_loss = generator_adversarial_loss(disc_fake_monet)\n            gen_YtoX_loss = generator_adversarial_loss(disc_fake_photo)\n\n            total_cycle_loss = cycle_loss(photo_in, cycled_photo) + cycle_loss(monet_in, cycled_monet)\n            total_identity_loss = identity_loss(monet_in, same_monet) + identity_loss(photo_in, same_photo)\n\n            total_gen_XtoY = gen_XtoY_loss + total_cycle_loss + total_identity_loss\n            total_gen_YtoX = gen_YtoX_loss + total_cycle_loss + total_identity_loss\n\n            disc_X_loss = discriminator_loss(disc_real_photo, disc_fake_photo)\n            disc_Y_loss = discriminator_loss(disc_real_monet, disc_fake_monet)\n\n        G_XtoY_grads = tape.gradient(total_gen_XtoY, G_XtoY.trainable_variables)\n        G_YtoX_grads = tape.gradient(total_gen_YtoX, G_YtoX.trainable_variables)\n        D_X_grads = tape.gradient(disc_X_loss, D_X.trainable_variables)\n        D_Y_grads = tape.gradient(disc_Y_loss, D_Y.trainable_variables)\n\n        G_XtoY_optimizer.apply_gradients(zip(G_XtoY_grads, G_XtoY.trainable_variables))\n        G_YtoX_optimizer.apply_gradients(zip(G_YtoX_grads, G_YtoX.trainable_variables))\n        D_X_optimizer.apply_gradients(zip(D_X_grads, D_X.trainable_variables))\n        D_Y_optimizer.apply_gradients(zip(D_Y_grads, D_Y.trainable_variables))\n\n        return {\n            \"G_XtoY\": total_gen_XtoY,\n            \"G_YtoX\": total_gen_YtoX,\n            \"D_X\": disc_X_loss,\n            \"D_Y\": disc_Y_loss,\n        }\n\n    @tf.function\n    def distributed_train_step(batch):\n        return strategy.run(train_step, args=(batch,))\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:03:35.624909Z","iopub.execute_input":"2026-03-07T04:03:35.625184Z","iopub.status.idle":"2026-03-07T04:03:48.824796Z","shell.execute_reply.started":"2026-03-07T04:03:35.625164Z","shell.execute_reply":"2026-03-07T04:03:48.823396Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# ---------------------------\n# Experiment name\n# ---------------------------\nEXP_NAME = \"LRDecay\"\n\nEPOCHS = TOTAL_EPOCHS\n\nfor epoch in range(EPOCHS):\n    lr = get_lr(epoch)\n\n    G_XtoY_optimizer.learning_rate.assign(lr)\n    G_YtoX_optimizer.learning_rate.assign(lr)\n    D_X_optimizer.learning_rate.assign(lr)\n    D_Y_optimizer.learning_rate.assign(lr)\n\n    print(f\"Epoch {epoch+1}/{EPOCHS} - learning rate: {lr:.6f}\")\n\n    epoch_G_XtoY = []\n    epoch_G_YtoX = []\n    epoch_D_X = []\n    epoch_D_Y = []\n\n    for step, batch in enumerate(train_ds):\n        per_replica = distributed_train_step(batch)\n\n        G_XtoY_loss = strategy.reduce(tf.distribute.ReduceOp.MEAN, per_replica[\"G_XtoY\"], axis=None)\n        G_YtoX_loss = strategy.reduce(tf.distribute.ReduceOp.MEAN, per_replica[\"G_YtoX\"], axis=None)\n        D_X_loss = strategy.reduce(tf.distribute.ReduceOp.MEAN, per_replica[\"D_X\"], axis=None)\n        D_Y_loss = strategy.reduce(tf.distribute.ReduceOp.MEAN, per_replica[\"D_Y\"], axis=None)\n\n        epoch_G_XtoY.append(float(G_XtoY_loss))\n        epoch_G_YtoX.append(float(G_YtoX_loss))\n        epoch_D_X.append(float(D_X_loss))\n        epoch_D_Y.append(float(D_Y_loss))\n\n    G_XtoY_history.append(sum(epoch_G_XtoY) / len(epoch_G_XtoY))\n    G_YtoX_history.append(sum(epoch_G_YtoX) / len(epoch_G_YtoX))\n    D_X_history.append(sum(epoch_D_X) / len(epoch_D_X))\n    D_Y_history.append(sum(epoch_D_Y) / len(epoch_D_Y))\n\n    print(f\"Epoch {epoch+1} completed\")\n\n\n# ---------------------------\n# Save models\n# ---------------------------\nG_XtoY.export(f\"/kaggle/working/{EXP_NAME}_G_XtoY\")\nG_YtoX.export(f\"/kaggle/working/{EXP_NAME}_G_YtoX\")\nD_X.export(f\"/kaggle/working/{EXP_NAME}_D_X\")\nD_Y.export(f\"/kaggle/working/{EXP_NAME}_D_Y\")\n\nG_XtoY.save_weights(f\"/kaggle/working/{EXP_NAME}_G_XtoY.weights.h5\")\nG_YtoX.save_weights(f\"/kaggle/working/{EXP_NAME}_G_YtoX.weights.h5\")\nD_X.save_weights(f\"/kaggle/working/{EXP_NAME}_D_X.weights.h5\")\nD_Y.save_weights(f\"/kaggle/working/{EXP_NAME}_D_Y.weights.h5\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:04:11.999238Z","iopub.execute_input":"2026-03-07T04:04:11.999492Z","iopub.status.idle":"2026-03-07T04:09:57.595015Z","shell.execute_reply.started":"2026-03-07T04:04:11.999475Z","shell.execute_reply":"2026-03-07T04:09:57.593831Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.figure(figsize=(12, 6))\n\nplt.plot(G_XtoY_history, label=\"Generator X→Y (Photo→Monet)\")\nplt.plot(G_YtoX_history, label=\"Generator Y→X (Monet→Photo)\")\nplt.plot(D_X_history, label=\"Discriminator X (Photo)\")\nplt.plot(D_Y_history, label=\"Discriminator Y (Monet)\")\n\nplt.xlabel(\"Epoch\")\nplt.ylabel(\"Loss\")\nplt.title(\"CycleGAN Training Losses\")\nplt.legend()\nplt.grid(True)\nplt.savefig(f\"{EXP_NAME}_LossPlots.png\", dpi=300, bbox_inches='tight')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:10:35.538138Z","iopub.execute_input":"2026-03-07T04:10:35.538426Z","iopub.status.idle":"2026-03-07T04:10:35.939459Z","shell.execute_reply.started":"2026-03-07T04:10:35.538404Z","shell.execute_reply":"2026-03-07T04:10:35.938312Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"This is a great result, the decay of learning rate appears to doing exactly as intended. With the generator the loss decline is smooth and monotonic, with only small fluctuations. This suggests the optimization process remained stable throughout training, and it reaches a lower value compared to no decay (4 vs 5).","metadata":{}},{"cell_type":"code","source":"@tf.function\ndef generate_step(batch):\n    batch_in = batch * 2.0 - 1.0\n    return G_XtoY(batch_in, training=False)\n\n# Run on TPU and gather results\nper_replica = strategy.run(generate_step, args=(sample_photos,))\nfake_monet = strategy.gather(per_replica, axis=0)\n\n# Convert back to [0,1]\nfake_monet = (fake_monet + 1.0) / 2.0\nfake_monet = tf.clip_by_value(fake_monet, 0.0, 1.0)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Take 10 images (unbatched)\nsample_photos = []\nfor img in photo_ds.unbatch().take(10):\n    sample_photos.append(img)\nsample_photos = tf.stack(sample_photos, axis=0)   # shape (10,256,256,3)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:12:01.434369Z","iopub.execute_input":"2026-03-07T04:12:01.434614Z","iopub.status.idle":"2026-03-07T04:12:01.489027Z","shell.execute_reply.started":"2026-03-07T04:12:01.434596Z","shell.execute_reply":"2026-03-07T04:12:01.487714Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nplt.figure(figsize=(20, 8))\n\nfor i in range(10):\n    # original photo\n    plt.subplot(2, 10, i + 1)\n    plt.imshow(sample_photos[i])\n    plt.axis(\"off\")\n    plt.title(\"Photo\")\n\n    # generated Monet\n    plt.subplot(2, 10, 10 + i + 1)\n    plt.imshow(fake_monet[i])\n    plt.axis(\"off\")\n    plt.title(\"Monet\")\n    \nplt.tight_layout()\nplt.savefig(f\"{EXP_NAME}_Results.png\", dpi=300, bbox_inches='tight')\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T04:12:03.990083Z","iopub.execute_input":"2026-03-07T04:12:03.990453Z","iopub.status.idle":"2026-03-07T04:12:11.788684Z","shell.execute_reply.started":"2026-03-07T04:12:03.990425Z","shell.execute_reply":"2026-03-07T04:12:11.787337Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Compared to the baseline model, the learning rate decay experiment produces slightly smoother textures, more stable color transitions, and marginally reduced noise, indicating that the scheduler helped the model converge more gradually in later epochs. The checkerboard artifacts are still noteiceable in certain images like the cliff, the sun, and the arch (images 4th, 7th, and 8th from the left). While the overall stylistic transformation remains similar, the results suggest that learning rate decay improves training stability and helps refine the visual quality of generated Monet-style images.","metadata":{}},{"cell_type":"markdown","source":"## Simple Image Augmentation\n\nTo improve generalization, light data augmentation will be added to the training pipeline. Random horizontal flipping will be applied to increase variation in scene composition without changing semantic content. If this is successful, I will look at additional augmentations. Let's create a function to perform horizontal flips.","metadata":{}},{"cell_type":"code","source":"def augment_image(img):\n    img = tf.image.random_flip_left_right(img)\n    return img","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T05:04:52.179445Z","iopub.execute_input":"2026-03-07T05:04:52.179771Z","iopub.status.idle":"2026-03-07T05:04:52.183819Z","shell.execute_reply.started":"2026-03-07T05:04:52.179751Z","shell.execute_reply":"2026-03-07T05:04:52.182652Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We'll create a 'training' flag to our tfrec loader so we can do these augmentations only to our training data.","metadata":{}},{"cell_type":"code","source":"IMG_SIZE = 256\n\ndef parse_tfrecord(example, training=False):\n    feature_description = {\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n    }\n\n    example = tf.io.parse_single_example(example, feature_description)\n\n    img = tf.io.decode_jpeg(example[\"image\"], channels=3)\n    img = tf.image.convert_image_dtype(img, tf.float32)\n    img = tf.image.resize(img, [IMG_SIZE, IMG_SIZE])\n\n    if training:\n        img = augment_image(img)\n\n    return img","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T05:09:36.226070Z","iopub.execute_input":"2026-03-07T05:09:36.226381Z","iopub.status.idle":"2026-03-07T05:09:36.231826Z","shell.execute_reply.started":"2026-03-07T05:09:36.226348Z","shell.execute_reply":"2026-03-07T05:09:36.230439Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We'll modify our dataloader to incorporate the flips.","metadata":{}},{"cell_type":"code","source":"def load_tfrecord_dataset(tfrecord_dir, batch_size, drop_remainder=True, training=False):\n    files = tf.io.gfile.glob(tfrecord_dir + \"/*.tfrec\")\n\n    ds = tf.data.TFRecordDataset(files, num_parallel_reads=tf.data.AUTOTUNE)\n\n    ds = ds.map(\n        lambda x: parse_tfrecord(x, training=training),\n        num_parallel_calls=tf.data.AUTOTUNE\n    )\n\n    ds = ds.batch(batch_size, drop_remainder=drop_remainder)\n    ds = ds.prefetch(tf.data.AUTOTUNE)\n\n    return ds","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T05:09:50.290720Z","iopub.execute_input":"2026-03-07T05:09:50.291078Z","iopub.status.idle":"2026-03-07T05:09:50.295758Z","shell.execute_reply.started":"2026-03-07T05:09:50.291054Z","shell.execute_reply":"2026-03-07T05:09:50.294774Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"And create the augmented training sets.","metadata":{}},{"cell_type":"code","source":"monet_ds = load_tfrecord_dataset(\n    MONET_TF_DATA_DIR,\n    GLOBAL_BATCH,\n    drop_remainder=True,\n    training=True\n)\n\nphoto_ds = load_tfrecord_dataset(\n    PHOTO_TF_DATA_DIR,\n    GLOBAL_BATCH,\n    drop_remainder=True,\n    training=True\n)\n\ntrain_ds = tf.data.Dataset.zip((monet_ds, photo_ds))\ntrain_ds = strategy.experimental_distribute_dataset(train_ds)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T05:12:59.908496Z","iopub.execute_input":"2026-03-07T05:12:59.908796Z","iopub.status.idle":"2026-03-07T05:13:00.067293Z","shell.execute_reply.started":"2026-03-07T05:12:59.908775Z","shell.execute_reply":"2026-03-07T05:13:00.066110Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Okay we're ready to train after we reset all variables.","metadata":{}},{"cell_type":"code","source":"tf.keras.backend.clear_session()\ngc.collect()\n\nG_XtoY_history = []\nG_YtoX_history = []\nD_X_history = []\nD_Y_history = []\n\nwith strategy.scope():\n    G_XtoY = build_generator()\n    G_YtoX = build_generator()\n    D_X = build_discriminator()\n    D_Y = build_discriminator()\n\n    G_XtoY_optimizer = tf.keras.optimizers.Adam(INITIAL_LR, beta_1=0.5)\n    G_YtoX_optimizer = tf.keras.optimizers.Adam(INITIAL_LR, beta_1=0.5)\n    D_X_optimizer = tf.keras.optimizers.Adam(INITIAL_LR, beta_1=0.5)\n    D_Y_optimizer = tf.keras.optimizers.Adam(INITIAL_LR, beta_1=0.5)\nwith strategy.scope():\n    @tf.function\n    def train_step(batch):\n        monet, photo = batch\n\n        # scale [0,1] → [-1,1]\n        monet_in = monet * 2.0 - 1.0\n        photo_in = photo * 2.0 - 1.0\n\n        with tf.GradientTape(persistent=True) as tape:\n            fake_monet = G_XtoY(photo_in, training=True)\n            fake_photo = G_YtoX(monet_in, training=True)\n\n            cycled_photo = G_YtoX(fake_monet, training=True)\n            cycled_monet = G_XtoY(fake_photo, training=True)\n\n            same_monet = G_XtoY(monet_in, training=True)\n            same_photo = G_YtoX(photo_in, training=True)\n\n            disc_real_monet = D_Y(monet_in, training=True)\n            disc_real_photo = D_X(photo_in, training=True)\n            disc_fake_monet = D_Y(fake_monet, training=True)\n            disc_fake_photo = D_X(fake_photo, training=True)\n\n            gen_XtoY_loss = generator_adversarial_loss(disc_fake_monet)\n            gen_YtoX_loss = generator_adversarial_loss(disc_fake_photo)\n\n            total_cycle_loss = cycle_loss(photo_in, cycled_photo) + cycle_loss(monet_in, cycled_monet)\n            total_identity_loss = identity_loss(monet_in, same_monet) + identity_loss(photo_in, same_photo)\n\n            total_gen_XtoY = gen_XtoY_loss + total_cycle_loss + total_identity_loss\n            total_gen_YtoX = gen_YtoX_loss + total_cycle_loss + total_identity_loss\n\n            disc_X_loss = discriminator_loss(disc_real_photo, disc_fake_photo)\n            disc_Y_loss = discriminator_loss(disc_real_monet, disc_fake_monet)\n\n        G_XtoY_grads = tape.gradient(total_gen_XtoY, G_XtoY.trainable_variables)\n        G_YtoX_grads = tape.gradient(total_gen_YtoX, G_YtoX.trainable_variables)\n        D_X_grads = tape.gradient(disc_X_loss, D_X.trainable_variables)\n        D_Y_grads = tape.gradient(disc_Y_loss, D_Y.trainable_variables)\n\n        G_XtoY_optimizer.apply_gradients(zip(G_XtoY_grads, G_XtoY.trainable_variables))\n        G_YtoX_optimizer.apply_gradients(zip(G_YtoX_grads, G_YtoX.trainable_variables))\n        D_X_optimizer.apply_gradients(zip(D_X_grads, D_X.trainable_variables))\n        D_Y_optimizer.apply_gradients(zip(D_Y_grads, D_Y.trainable_variables))\n\n        return {\n            \"G_XtoY\": total_gen_XtoY,\n            \"G_YtoX\": total_gen_YtoX,\n            \"D_X\": disc_X_loss,\n            \"D_Y\": disc_Y_loss,\n        }\n\n    @tf.function\n    def distributed_train_step(batch):\n        return strategy.run(train_step, args=(batch,))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T05:18:16.965861Z","iopub.execute_input":"2026-03-07T05:18:16.966193Z","iopub.status.idle":"2026-03-07T05:18:26.586003Z","shell.execute_reply.started":"2026-03-07T05:18:16.966172Z","shell.execute_reply":"2026-03-07T05:18:26.584495Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# ---------------------------\n# Experiment name\n# ---------------------------\nEXP_NAME = \"LRDecay_simple_augment\"\n\nEPOCHS = TOTAL_EPOCHS\n\nfor epoch in range(EPOCHS):\n    lr = get_lr(epoch)\n\n    G_XtoY_optimizer.learning_rate.assign(lr)\n    G_YtoX_optimizer.learning_rate.assign(lr)\n    D_X_optimizer.learning_rate.assign(lr)\n    D_Y_optimizer.learning_rate.assign(lr)\n\n    print(f\"Epoch {epoch+1}/{EPOCHS} - learning rate: {lr:.6f}\")\n\n    epoch_G_XtoY = []\n    epoch_G_YtoX = []\n    epoch_D_X = []\n    epoch_D_Y = []\n\n    for step, batch in enumerate(train_ds):\n        per_replica = distributed_train_step(batch)\n\n        G_XtoY_loss = strategy.reduce(tf.distribute.ReduceOp.MEAN, per_replica[\"G_XtoY\"], axis=None)\n        G_YtoX_loss = strategy.reduce(tf.distribute.ReduceOp.MEAN, per_replica[\"G_YtoX\"], axis=None)\n        D_X_loss = strategy.reduce(tf.distribute.ReduceOp.MEAN, per_replica[\"D_X\"], axis=None)\n        D_Y_loss = strategy.reduce(tf.distribute.ReduceOp.MEAN, per_replica[\"D_Y\"], axis=None)\n\n        epoch_G_XtoY.append(float(G_XtoY_loss))\n        epoch_G_YtoX.append(float(G_YtoX_loss))\n        epoch_D_X.append(float(D_X_loss))\n        epoch_D_Y.append(float(D_Y_loss))\n\n    G_XtoY_history.append(sum(epoch_G_XtoY) / len(epoch_G_XtoY))\n    G_YtoX_history.append(sum(epoch_G_YtoX) / len(epoch_G_YtoX))\n    D_X_history.append(sum(epoch_D_X) / len(epoch_D_X))\n    D_Y_history.append(sum(epoch_D_Y) / len(epoch_D_Y))\n\n    print(f\"Epoch {epoch+1} completed\")\n\n\n# ---------------------------\n# Save models\n# ---------------------------\nG_XtoY.export(f\"/kaggle/working/{EXP_NAME}_G_XtoY\")\nG_YtoX.export(f\"/kaggle/working/{EXP_NAME}_G_YtoX\")\nD_X.export(f\"/kaggle/working/{EXP_NAME}_D_X\")\nD_Y.export(f\"/kaggle/working/{EXP_NAME}_D_Y\")\n\nG_XtoY.save_weights(f\"/kaggle/working/{EXP_NAME}_G_XtoY.weights.h5\")\nG_YtoX.save_weights(f\"/kaggle/working/{EXP_NAME}_G_YtoX.weights.h5\")\nD_X.save_weights(f\"/kaggle/working/{EXP_NAME}_D_X.weights.h5\")\nD_Y.save_weights(f\"/kaggle/working/{EXP_NAME}_D_Y.weights.h5\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T05:18:26.586763Z","iopub.execute_input":"2026-03-07T05:18:26.586959Z","iopub.status.idle":"2026-03-07T05:24:28.719499Z","shell.execute_reply.started":"2026-03-07T05:18:26.586940Z","shell.execute_reply":"2026-03-07T05:24:28.718124Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.figure(figsize=(12, 6))\n\nplt.plot(G_XtoY_history, label=\"Generator X→Y (Photo→Monet)\")\nplt.plot(G_YtoX_history, label=\"Generator Y→X (Monet→Photo)\")\nplt.plot(D_X_history, label=\"Discriminator X (Photo)\")\nplt.plot(D_Y_history, label=\"Discriminator Y (Monet)\")\n\nplt.xlabel(\"Epoch\")\nplt.ylabel(\"Loss\")\nplt.title(\"CycleGAN Training Losses\")\nplt.legend()\nplt.grid(True)\nplt.savefig(f\"{EXP_NAME}_LossPlots.png\", dpi=300, bbox_inches='tight')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T05:28:54.978584Z","iopub.execute_input":"2026-03-07T05:28:54.979002Z","iopub.status.idle":"2026-03-07T05:28:55.388449Z","shell.execute_reply.started":"2026-03-07T05:28:54.978946Z","shell.execute_reply":"2026-03-07T05:28:55.387141Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Take 10 images (unbatched)\nsample_photos = []\nfor img in photo_ds.unbatch().take(10):\n    sample_photos.append(img)\nsample_photos = tf.stack(sample_photos, axis=0)   # shape (10,256,256,3)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T05:30:54.995926Z","iopub.execute_input":"2026-03-07T05:30:54.996235Z","iopub.status.idle":"2026-03-07T05:30:55.058255Z","shell.execute_reply.started":"2026-03-07T05:30:54.996211Z","shell.execute_reply":"2026-03-07T05:30:55.056793Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"@tf.function\ndef generate_step(batch):\n    batch_in = batch * 2.0 - 1.0\n    return G_XtoY(batch_in, training=False)\n\n# Run on TPU and gather results\nper_replica = strategy.run(generate_step, args=(sample_photos,))\nfake_monet = strategy.gather(per_replica, axis=0)\n\n# Convert back to [0,1]\nfake_monet = (fake_monet + 1.0) / 2.0\nfake_monet = tf.clip_by_value(fake_monet, 0.0, 1.0)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T05:30:57.503250Z","iopub.execute_input":"2026-03-07T05:30:57.503480Z","iopub.status.idle":"2026-03-07T05:31:04.009090Z","shell.execute_reply.started":"2026-03-07T05:30:57.503464Z","shell.execute_reply":"2026-03-07T05:31:04.007713Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nplt.figure(figsize=(20, 8))\n\nfor i in range(10):\n    # original photo\n    plt.subplot(2, 10, i + 1)\n    plt.imshow(sample_photos[i])\n    plt.axis(\"off\")\n    plt.title(\"Photo\")\n\n    # generated Monet\n    plt.subplot(2, 10, 10 + i + 1)\n    plt.imshow(fake_monet[i])\n    plt.axis(\"off\")\n    plt.title(\"Monet\")\n    \nplt.tight_layout()\nplt.savefig(f\"{EXP_NAME}_Results.png\", dpi=300, bbox_inches='tight')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T05:31:13.107947Z","iopub.execute_input":"2026-03-07T05:31:13.108341Z","iopub.status.idle":"2026-03-07T05:31:17.847325Z","shell.execute_reply.started":"2026-03-07T05:31:13.108317Z","shell.execute_reply":"2026-03-07T05:31:17.846178Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The addition of horizontal flip augmentation produced slightly more stable training and modest visual improvements in the generated images. The loss curves continue to decrease smoothly and end up slightly lower than without augmentation (3.95 vs 4) suggesting that the model remains stable while benefiting from the additional data variation. Visually, some scenes show improved texture consistency and smoother color blending, particularly in landscape regions such as mountains, water, and skies. The generated images maintain clearer structure while applying Monet-like brush textures more consistently. However, some artifacts and noise patterns remain (although less distinct) in complex scenes, I think we could benefit from some additional augmentation.","metadata":{}},{"cell_type":"markdown","source":"## Robust Image Augmentation\nTo implement more robust image augmentation, I will add resize jitter and random crop in addition to the horizontal flips. In addition to random horizontal flipping, images will be resized to a slightly larger resolution and randomly cropped back to the original size. This resize-jitter augmentation is intended to improve robustness to small spatial variations while preserving the overall content needed for style translation. These augments should provide examples that are still similar enough that they won't take away from the style transfer. I have to update the dataloaders just like we did before.","metadata":{}},{"cell_type":"code","source":"IMG_SIZE = 256\nJITTER_SIZE = 286\n\ndef augment_image(img):\n    # Resize up, then randomly crop back down\n    img = tf.image.resize(img, [JITTER_SIZE, JITTER_SIZE])\n    img = tf.image.random_crop(img, [IMG_SIZE, IMG_SIZE, 3])\n\n    # Random horizontal flip\n    img = tf.image.random_flip_left_right(img)\n\n    return img","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T05:54:49.713204Z","iopub.execute_input":"2026-03-07T05:54:49.713530Z","iopub.status.idle":"2026-03-07T05:54:49.718141Z","shell.execute_reply.started":"2026-03-07T05:54:49.713510Z","shell.execute_reply":"2026-03-07T05:54:49.716878Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def parse_tfrecord(example, training=False):\n    feature_description = {\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n    }\n\n    example = tf.io.parse_single_example(example, feature_description)\n\n    img = tf.io.decode_jpeg(example[\"image\"], channels=3)\n    img = tf.image.convert_image_dtype(img, tf.float32)\n    img = tf.image.resize(img, [IMG_SIZE, IMG_SIZE])\n\n    if training:\n        img = augment_image(img)\n\n    return img","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T05:54:54.010187Z","iopub.execute_input":"2026-03-07T05:54:54.010499Z","iopub.status.idle":"2026-03-07T05:54:54.015311Z","shell.execute_reply.started":"2026-03-07T05:54:54.010476Z","shell.execute_reply":"2026-03-07T05:54:54.014168Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def load_tfrecord_dataset(tfrecord_dir, batch_size, drop_remainder=True, training=False):\n    files = tf.io.gfile.glob(tfrecord_dir + \"/*.tfrec\")\n\n    ds = tf.data.TFRecordDataset(files, num_parallel_reads=tf.data.AUTOTUNE)\n    ds = ds.map(\n        lambda x: parse_tfrecord(x, training=training),\n        num_parallel_calls=tf.data.AUTOTUNE\n    )\n    ds = ds.batch(batch_size, drop_remainder=drop_remainder)\n    ds = ds.prefetch(tf.data.AUTOTUNE)\n\n    return ds","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T05:54:58.274560Z","iopub.execute_input":"2026-03-07T05:54:58.274877Z","iopub.status.idle":"2026-03-07T05:54:58.278987Z","shell.execute_reply.started":"2026-03-07T05:54:58.274854Z","shell.execute_reply":"2026-03-07T05:54:58.278122Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"monet_ds = load_tfrecord_dataset(\n    MONET_TF_DATA_DIR,\n    GLOBAL_BATCH,\n    drop_remainder=True,\n    training=True\n)\n\nphoto_ds = load_tfrecord_dataset(\n    PHOTO_TF_DATA_DIR,\n    GLOBAL_BATCH,\n    drop_remainder=True,\n    training=True\n)\n\ntrain_ds = tf.data.Dataset.zip((monet_ds, photo_ds))\ntrain_ds = strategy.experimental_distribute_dataset(train_ds)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T05:55:02.837587Z","iopub.execute_input":"2026-03-07T05:55:02.837904Z","iopub.status.idle":"2026-03-07T05:55:03.018244Z","shell.execute_reply.started":"2026-03-07T05:55:02.837883Z","shell.execute_reply":"2026-03-07T05:55:03.016846Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Okay let's clear variables and we can train with robust image augmentation.","metadata":{}},{"cell_type":"code","source":"tf.keras.backend.clear_session()\ngc.collect()\n\nG_XtoY_history = []\nG_YtoX_history = []\nD_X_history = []\nD_Y_history = []\n\nwith strategy.scope():\n    G_XtoY = build_generator()\n    G_YtoX = build_generator()\n    D_X = build_discriminator()\n    D_Y = build_discriminator()\n\n    G_XtoY_optimizer = tf.keras.optimizers.Adam(INITIAL_LR, beta_1=0.5)\n    G_YtoX_optimizer = tf.keras.optimizers.Adam(INITIAL_LR, beta_1=0.5)\n    D_X_optimizer = tf.keras.optimizers.Adam(INITIAL_LR, beta_1=0.5)\n    D_Y_optimizer = tf.keras.optimizers.Adam(INITIAL_LR, beta_1=0.5)\nwith strategy.scope():\n    @tf.function\n    def train_step(batch):\n        monet, photo = batch\n\n        # scale [0,1] → [-1,1]\n        monet_in = monet * 2.0 - 1.0\n        photo_in = photo * 2.0 - 1.0\n\n        with tf.GradientTape(persistent=True) as tape:\n            fake_monet = G_XtoY(photo_in, training=True)\n            fake_photo = G_YtoX(monet_in, training=True)\n\n            cycled_photo = G_YtoX(fake_monet, training=True)\n            cycled_monet = G_XtoY(fake_photo, training=True)\n\n            same_monet = G_XtoY(monet_in, training=True)\n            same_photo = G_YtoX(photo_in, training=True)\n\n            disc_real_monet = D_Y(monet_in, training=True)\n            disc_real_photo = D_X(photo_in, training=True)\n            disc_fake_monet = D_Y(fake_monet, training=True)\n            disc_fake_photo = D_X(fake_photo, training=True)\n\n            gen_XtoY_loss = generator_adversarial_loss(disc_fake_monet)\n            gen_YtoX_loss = generator_adversarial_loss(disc_fake_photo)\n\n            total_cycle_loss = cycle_loss(photo_in, cycled_photo) + cycle_loss(monet_in, cycled_monet)\n            total_identity_loss = identity_loss(monet_in, same_monet) + identity_loss(photo_in, same_photo)\n\n            total_gen_XtoY = gen_XtoY_loss + total_cycle_loss + total_identity_loss\n            total_gen_YtoX = gen_YtoX_loss + total_cycle_loss + total_identity_loss\n\n            disc_X_loss = discriminator_loss(disc_real_photo, disc_fake_photo)\n            disc_Y_loss = discriminator_loss(disc_real_monet, disc_fake_monet)\n\n        G_XtoY_grads = tape.gradient(total_gen_XtoY, G_XtoY.trainable_variables)\n        G_YtoX_grads = tape.gradient(total_gen_YtoX, G_YtoX.trainable_variables)\n        D_X_grads = tape.gradient(disc_X_loss, D_X.trainable_variables)\n        D_Y_grads = tape.gradient(disc_Y_loss, D_Y.trainable_variables)\n\n        G_XtoY_optimizer.apply_gradients(zip(G_XtoY_grads, G_XtoY.trainable_variables))\n        G_YtoX_optimizer.apply_gradients(zip(G_YtoX_grads, G_YtoX.trainable_variables))\n        D_X_optimizer.apply_gradients(zip(D_X_grads, D_X.trainable_variables))\n        D_Y_optimizer.apply_gradients(zip(D_Y_grads, D_Y.trainable_variables))\n\n        return {\n            \"G_XtoY\": total_gen_XtoY,\n            \"G_YtoX\": total_gen_YtoX,\n            \"D_X\": disc_X_loss,\n            \"D_Y\": disc_Y_loss,\n        }\n\n    @tf.function\n    def distributed_train_step(batch):\n        return strategy.run(train_step, args=(batch,))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T05:57:04.331560Z","iopub.execute_input":"2026-03-07T05:57:04.331893Z","iopub.status.idle":"2026-03-07T05:57:14.105335Z","shell.execute_reply.started":"2026-03-07T05:57:04.331870Z","shell.execute_reply":"2026-03-07T05:57:14.102424Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# ---------------------------\n# Experiment name\n# ---------------------------\nEXP_NAME = \"LRDecay_robust_augment\"\n\nEPOCHS = TOTAL_EPOCHS\n\nfor epoch in range(EPOCHS):\n    lr = get_lr(epoch)\n\n    G_XtoY_optimizer.learning_rate.assign(lr)\n    G_YtoX_optimizer.learning_rate.assign(lr)\n    D_X_optimizer.learning_rate.assign(lr)\n    D_Y_optimizer.learning_rate.assign(lr)\n\n    print(f\"Epoch {epoch+1}/{EPOCHS} - learning rate: {lr:.6f}\")\n\n    epoch_G_XtoY = []\n    epoch_G_YtoX = []\n    epoch_D_X = []\n    epoch_D_Y = []\n\n    for step, batch in enumerate(train_ds):\n        per_replica = distributed_train_step(batch)\n\n        G_XtoY_loss = strategy.reduce(tf.distribute.ReduceOp.MEAN, per_replica[\"G_XtoY\"], axis=None)\n        G_YtoX_loss = strategy.reduce(tf.distribute.ReduceOp.MEAN, per_replica[\"G_YtoX\"], axis=None)\n        D_X_loss = strategy.reduce(tf.distribute.ReduceOp.MEAN, per_replica[\"D_X\"], axis=None)\n        D_Y_loss = strategy.reduce(tf.distribute.ReduceOp.MEAN, per_replica[\"D_Y\"], axis=None)\n\n        epoch_G_XtoY.append(float(G_XtoY_loss))\n        epoch_G_YtoX.append(float(G_YtoX_loss))\n        epoch_D_X.append(float(D_X_loss))\n        epoch_D_Y.append(float(D_Y_loss))\n\n    G_XtoY_history.append(sum(epoch_G_XtoY) / len(epoch_G_XtoY))\n    G_YtoX_history.append(sum(epoch_G_YtoX) / len(epoch_G_YtoX))\n    D_X_history.append(sum(epoch_D_X) / len(epoch_D_X))\n    D_Y_history.append(sum(epoch_D_Y) / len(epoch_D_Y))\n\n    print(f\"Epoch {epoch+1} completed\")\n\n\n# ---------------------------\n# Save models\n# ---------------------------\nG_XtoY.export(f\"/kaggle/working/{EXP_NAME}_G_XtoY\")\nG_YtoX.export(f\"/kaggle/working/{EXP_NAME}_G_YtoX\")\nD_X.export(f\"/kaggle/working/{EXP_NAME}_D_X\")\nD_Y.export(f\"/kaggle/working/{EXP_NAME}_D_Y\")\n\nG_XtoY.save_weights(f\"/kaggle/working/{EXP_NAME}_G_XtoY.weights.h5\")\nG_YtoX.save_weights(f\"/kaggle/working/{EXP_NAME}_G_YtoX.weights.h5\")\nD_X.save_weights(f\"/kaggle/working/{EXP_NAME}_D_X.weights.h5\")\nD_Y.save_weights(f\"/kaggle/working/{EXP_NAME}_D_Y.weights.h5\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T05:57:59.903419Z","iopub.execute_input":"2026-03-07T05:57:59.903652Z","iopub.status.idle":"2026-03-07T06:03:47.191143Z","shell.execute_reply.started":"2026-03-07T05:57:59.903635Z","shell.execute_reply":"2026-03-07T06:03:47.189750Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"plt.figure(figsize=(12, 6))\n\nplt.plot(G_XtoY_history, label=\"Generator X→Y (Photo→Monet)\")\nplt.plot(G_YtoX_history, label=\"Generator Y→X (Monet→Photo)\")\nplt.plot(D_X_history, label=\"Discriminator X (Photo)\")\nplt.plot(D_Y_history, label=\"Discriminator Y (Monet)\")\n\nplt.xlabel(\"Epoch\")\nplt.ylabel(\"Loss\")\nplt.title(\"CycleGAN Training Losses\")\nplt.legend()\nplt.grid(True)\nplt.savefig(f\"{EXP_NAME}_LossPlots.png\", dpi=300, bbox_inches='tight')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T06:04:26.215591Z","iopub.execute_input":"2026-03-07T06:04:26.215959Z","iopub.status.idle":"2026-03-07T06:04:26.621489Z","shell.execute_reply.started":"2026-03-07T06:04:26.215930Z","shell.execute_reply":"2026-03-07T06:04:26.620536Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Take 10 images (unbatched)\nsample_photos = []\nfor img in photo_ds.unbatch().take(10):\n    sample_photos.append(img)\nsample_photos = tf.stack(sample_photos, axis=0)   # shape (10,256,256,3)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T06:04:34.977686Z","iopub.execute_input":"2026-03-07T06:04:34.977962Z","iopub.status.idle":"2026-03-07T06:04:35.053790Z","shell.execute_reply.started":"2026-03-07T06:04:34.977940Z","shell.execute_reply":"2026-03-07T06:04:35.052532Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"@tf.function\ndef generate_step(batch):\n    batch_in = batch * 2.0 - 1.0\n    return G_XtoY(batch_in, training=False)\n\n# Run on TPU and gather results\nper_replica = strategy.run(generate_step, args=(sample_photos,))\nfake_monet = strategy.gather(per_replica, axis=0)\n\n# Convert back to [0,1]\nfake_monet = (fake_monet + 1.0) / 2.0\nfake_monet = tf.clip_by_value(fake_monet, 0.0, 1.0)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T06:04:42.782936Z","iopub.execute_input":"2026-03-07T06:04:42.783247Z","iopub.status.idle":"2026-03-07T06:04:49.263841Z","shell.execute_reply.started":"2026-03-07T06:04:42.783228Z","shell.execute_reply":"2026-03-07T06:04:49.262588Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nplt.figure(figsize=(20, 8))\n\nfor i in range(10):\n    # original photo\n    plt.subplot(2, 10, i + 1)\n    plt.imshow(sample_photos[i])\n    plt.axis(\"off\")\n    plt.title(\"Photo\")\n\n    # generated Monet\n    plt.subplot(2, 10, 10 + i + 1)\n    plt.imshow(fake_monet[i])\n    plt.axis(\"off\")\n    plt.title(\"Monet\")\n    \nplt.tight_layout()\nplt.savefig(f\"{EXP_NAME}_Results.png\", dpi=300, bbox_inches='tight')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T06:04:59.263619Z","iopub.execute_input":"2026-03-07T06:04:59.263862Z","iopub.status.idle":"2026-03-07T06:05:04.250056Z","shell.execute_reply.started":"2026-03-07T06:04:59.263837Z","shell.execute_reply":"2026-03-07T06:05:04.248744Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"The loss curves continue to decrease smoothly without introducing instability, indicating that the model is able to learn effectively despite the increased variability in the training data. Visually, several scenes show slightly improved texture consistency and more natural color blending in skies, water, and distant landscape regions. The generated images preserve scene structure while applying Monet-style textures more evenly. However, some checkerboard artifacts and noise patterns remain in complex scenes.","metadata":{}},{"cell_type":"markdown","source":"| Model | Description of Results |\n|------|------------------------|\n| Baseline (9 Residual Blocks, 40 Epochs) | The baseline CycleGAN produced recognizable Monet-style translations with reasonable color shifts and painterly textures. However, several images showed noise patterns and checkerboard artifacts, particularly in complex scenes such as cliffs and arches. While structure was preserved well, texture consistency and smooth color blending were limited. |\n| Learning Rate Decay | Introducing linear learning rate decay after mid-training improved stability in later epochs. The generated images showed smoother gradients in skies and landscapes and slightly reduced noise compared to the baseline. Training losses decreased more gradually toward convergence, suggesting more stable generator refinement in the final epochs. |\n| Simple Image Augmentation (Horizontal Flip) | Adding horizontal flipping increased data variation and slightly improved generalization. Some scenes displayed more consistent texture patterns and better preservation of landscape structure. Improvements were modest but noticeable in certain environments such as mountains and water surfaces. |\n| Robust Image Augmentation (Resize Jitter + Random Crop + Flip) | Stronger spatial augmentation further increased dataset diversity while maintaining stable training. The model produced slightly more natural color blending and more consistent Monet-style textures across several scenes. While artifacts still appeared in highly detailed regions, the results showed improved robustness and better stylistic consistency compared to earlier models. |","metadata":{}},{"cell_type":"markdown","source":"# Conclusion\nIn this project, a CycleGAN model was trained to translate real-world landscape photos into images resembling Monet-style paintings. A baseline model using nine residual blocks and forty epochs was first implemented to establish a reference point. This architecture was shown to be successful in my week 5 mini-project. From this baseline, several training improvements were explored, including learning rate decay and progressively stronger data augmentation strategies. Learning rate decay helped stabilize the later stages of training, leading to smoother color transitions and slightly reduced noise in generated images. Introducing horizontal flip augmentation increased dataset variability and produced modest improvements in texture consistency. Finally, more robust spatial augmentation using resize jitter and random cropping further improved generalization while maintaining stable training dynamics.\n\nOverall, these experiments demonstrate that training stability and dataset diversity both contribute to improved image translation quality. While the improvements were incremental rather than dramatic, the combination of learning rate scheduling and spatial augmentation produced the most consistent results. Future work could explore additional architectural improvements, specifically alternative upsampling methods to reduce checkerboard artifacts, or perceptual losses to better capture painterly texture and style.","metadata":{}},{"cell_type":"markdown","source":"Here is the submission generation.","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nimport zipfile\n\nzip_path = \"images.zip\"\nTARGET_IMAGES = 7000\n\n@tf.function\ndef generate_step(batch):\n    batch_in = batch * 2.0 - 1.0\n    return G_XtoY(batch_in, training=False)\n\nwith zipfile.ZipFile(zip_path, 'w', zipfile.ZIP_DEFLATED) as zipf:\n    idx = 0\n\n    for batch in photo_ds:\n        if idx >= TARGET_IMAGES:\n            break\n\n        # Run generator on TPU\n        per_replica = strategy.run(generate_step, args=(batch,))\n        fake_monet = strategy.gather(per_replica, axis=0)\n\n        # Convert back to [0,1]\n        fake_monet = (fake_monet + 1.0) / 2.0\n        fake_monet = tf.clip_by_value(fake_monet, 0.0, 1.0)\n\n        # Encode and write to zip\n        for i in range(fake_monet.shape[0]):\n            if idx >= TARGET_IMAGES:\n                break\n\n            img = fake_monet[i]\n            img_uint8 = tf.image.convert_image_dtype(img, tf.uint8)\n            jpeg_bytes = tf.io.encode_jpeg(img_uint8).numpy()\n\n            filename = f\"{idx:05d}.jpg\"\n            zipf.writestr(filename, jpeg_bytes)\n            idx += 1\n\nprint(\"Finished writing images.zip with\", idx, \"images\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2026-03-07T06:17:37.078577Z","iopub.execute_input":"2026-03-07T06:17:37.078928Z","iopub.status.idle":"2026-03-07T06:18:51.610492Z","shell.execute_reply.started":"2026-03-07T06:17:37.078899Z","shell.execute_reply":"2026-03-07T06:18:51.609279Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# References\n* https://www.kaggle.com/competitions/gan-getting-started/overview\n* https://www.kaggle.com/code/amyjang/monet-cyclegan-tutorial\n* ChatGPT","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}