{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":21755,"databundleVersionId":1475600,"sourceType":"competition"}],"dockerImageVersionId":30918,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# I'm Something of a Painter Myself \nThis competition asks to create a Generational Adversial Network, or GAN, to generate images in the style of Monet. This is utilized through a **discriminator** and **generator**, which the generator attempts to \"trick\" the discriminator with generated images.\n\nThis notebook covers a CycleGAN architecture.","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\n\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom PIL import Image\nimport shutil\nfrom pathlib import Path\nfrom tqdm import tqdm\nimport os","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2025-04-08T04:59:26.864431Z","iopub.execute_input":"2025-04-08T04:59:26.864744Z","iopub.status.idle":"2025-04-08T04:59:26.869236Z","shell.execute_reply.started":"2025-04-08T04:59:26.864720Z","shell.execute_reply":"2025-04-08T04:59:26.868271Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(\"Num GPUs Available:\", len(tf.config.list_physical_devices('GPU')))\n\nAUTOTUNE = tf.data.experimental.AUTOTUNE\nprint(tf.__version__)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T01:52:49.571333Z","iopub.execute_input":"2025-04-08T01:52:49.571804Z","iopub.status.idle":"2025-04-08T01:52:50.230758Z","shell.execute_reply.started":"2025-04-08T01:52:49.571780Z","shell.execute_reply":"2025-04-08T01:52:50.229837Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Data EDA\n\nSince we are using an unsupervised method (unpaired images), we don't need the labels or image IDs. We are also told:\n- Images are sized **256x256**\n- We are using RGB images, so **3 channels** are needed.\n\nWe are also utilizing the `.tfrecord` files given instead of the `jpeg` files for faster loading with tensorflow.","metadata":{}},{"cell_type":"code","source":"monet_tf_path = '/kaggle/input/gan-getting-started/monet_tfrec/*.tfrec'\nphoto_tf_path = '/kaggle/input/gan-getting-started/photo_tfrec/*.tfrec'\n\nmonet_tf = tf.data.Dataset.list_files(monet_tf_path)\nphoto_tf = tf.data.Dataset.list_files(photo_tf_path)\n\nprint('Monet TFRecord Files: ', len(monet_tf))\nprint('Photo TFRecord Files: ', len(photo_tf))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T01:52:50.232779Z","iopub.execute_input":"2025-04-08T01:52:50.233114Z","iopub.status.idle":"2025-04-08T01:52:51.315845Z","shell.execute_reply.started":"2025-04-08T01:52:50.233092Z","shell.execute_reply":"2025-04-08T01:52:51.315077Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"When we load in the images, as described in the [TensorFlow CycleGAN Tutorial](https://www.tensorflow.org/tutorials/generative/cyclegan) and the [CycleGAN Paper](https://arxiv.org/abs/1703.10593), applying random jittering / mirroring will prevent overfitting","metadata":{}},{"cell_type":"code","source":"IMAGE_SIZE = [256, 256]\n\ndef decode(img, training):\n    img = tf.image.decode_jpeg(img, channels=3)\n\n    if training:\n        # apply random jittering\n        img = tf.image.resize(img, [286, 286]) # upsize\n        img = tf.image.random_crop(img, size=[256, 256, 3]) # crop\n    \n        # apply mirroring\n        img = tf.image.random_flip_left_right(img) \n    \n    # final image details then return\n    img = (tf.cast(img, tf.float32) / 127.5) - 1 # normalization :>\n    img = tf.reshape(img, [*IMAGE_SIZE, 3]) # 256x256 + 3 channels for rgb\n    return img\n\ndef read_tfrecord(tfrecord, training):\n    format_tf = {\n        'image_name': tf.io.FixedLenFeature([], tf.string),\n        'image': tf.io.FixedLenFeature([], tf.string),\n        'target': tf.io.FixedLenFeature([], tf.string),\n    }\n    ex = tf.io.parse_single_example(tfrecord, format_tf)\n    image = decode(ex['image'], training)\n    return image\n\ndef load_dataset(tf_files, training=False):\n    dataset = tf.data.TFRecordDataset(tf_files)\n    dataset = dataset.map(lambda x: read_tfrecord(x, training), num_parallel_calls=AUTOTUNE)\n    return dataset","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T01:52:51.317131Z","iopub.execute_input":"2025-04-08T01:52:51.317415Z","iopub.status.idle":"2025-04-08T01:52:51.323403Z","shell.execute_reply.started":"2025-04-08T01:52:51.317383Z","shell.execute_reply":"2025-04-08T01:52:51.322622Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(\"Monet Images: \", sum(1 for _ in tf.data.TFRecordDataset(monet_tf)))\nprint('Photo Images: ', sum(1 for _ in tf.data.TFRecordDataset(photo_tf)))\n\nmonet_ds = load_dataset(monet_tf, training=True).shuffle(512).batch(1).repeat().prefetch(tf.data.AUTOTUNE)\nphoto_ds = load_dataset(photo_tf, training=True).shuffle(512).batch(1).repeat().prefetch(tf.data.AUTOTUNE)\ntest_monet_ds = load_dataset(monet_tf, training=False).batch(1).prefetch(tf.data.AUTOTUNE)\ntest_photo_ds = load_dataset(photo_tf, training=False).batch(1).prefetch(tf.data.AUTOTUNE)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T01:52:51.324094Z","iopub.execute_input":"2025-04-08T01:52:51.324340Z","iopub.status.idle":"2025-04-08T01:52:55.162812Z","shell.execute_reply.started":"2025-04-08T01:52:51.324319Z","shell.execute_reply":"2025-04-08T01:52:55.162032Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Lets take a look at a sample of the images","metadata":{}},{"cell_type":"code","source":"ex_monet, ex_photo = next(iter(monet_ds)), next(iter(photo_ds))\n\n\ndef show_img(ds, title):\n    plt.figure(figsize=(12,6))\n    for i, img in enumerate(ds.take(8)):\n        plt.subplot(2, 4, i+1)\n        plt.imshow(tf.squeeze(img, axis=0) * 0.5 + 0.5)\n        plt.axis('off')\n    plt.suptitle(title)\n    plt.tight_layout()\n    plt.show()\n\nshow_img(monet_ds, 'Monet')\nshow_img(photo_ds, 'Photo')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T01:52:55.163630Z","iopub.execute_input":"2025-04-08T01:52:55.163850Z","iopub.status.idle":"2025-04-08T01:52:58.375669Z","shell.execute_reply.started":"2025-04-08T01:52:55.163831Z","shell.execute_reply":"2025-04-08T01:52:58.374548Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# CycleGAN Model","metadata":{}},{"cell_type":"markdown","source":"## Model Description\n\nFor our baseline model, we are going to be using the CycleGAN model to translate our photos (`X`) to our monet style (`Y`).\n- CycleGAN uses 2 generators and discriminators, for monet->photo, photo->monet\n- It also encompasses cycle consistency and identity loss, so our translations can be reversed / unneccesary changes to existing monet images.\n\nWe will also be using the `UNET` architecture:\n- For each `downsample`, concat feature map with `upsampling layer`\n- This bypasses the vanishing gradient problem since the output of a layer will now be connected to multiple layers instead of one\n\nA lot of inspiration came from [Amy Jang's CycleGAN Tutorial](https://www.kaggle.com/code/amyjang/monet-cyclegan-tutorial/notebook)","metadata":{}},{"cell_type":"markdown","source":"## Generator\n\nTo start with our model, we need our image generator. This will take our image and learn features and styles, so it can create Monet-style paintings.\n\nThe architecture includes the following components:\n- `downsample` then `upsamples` image via convolutional layers to extract image features / details\n- As stated earlier, uses skip connections to bypass vanishing gradient and preserve spatial details.\n- Instance Normalization for model stability + consistent style across images\n- Dropout for improved model generalization\n- Output with the shape of (256, 256, 3), since our image is 256x256 and has 3 channels (RGB)","metadata":{}},{"cell_type":"code","source":"def upsample(filters, size, dropout=False):\n    result = keras.Sequential()\n    \n    result.add(layers.Conv2DTranspose(filters, size, strides=2, padding='same'))\n    result.add(layers.GroupNormalization(1)) # instance normalization \n\n    if dropout:\n        result.add(layers.Dropout(0.3))\n\n    result.add(layers.ReLU())\n    return result\n\ndef downsample(filters, size, inorm=True):\n    result = keras.Sequential()\n\n    result.add(layers.Conv2D(filters, size, strides=2, padding='same'))\n    \n    if inorm:\n        result.add(layers.GroupNormalization(1)) # instance normalization \n\n    result.add(layers.LeakyReLU())\n    return result\n    \ndef generator(input_shape=(256, 256, 3)):\n    inputs = keras.Input(shape=input_shape)\n\n    # add downsampling layers (image feature extractors)\n    downsample_stack = [\n        downsample(64, 4, inorm=False), # batch, 128, 128, 64\n        downsample(128, 4), # batch, 64, 64, 128\n        downsample(256, 4), # batch, 32, 32, 256\n        downsample(512, 4), # batch, 16, 16, 512\n        downsample(512, 4), # batch, 8, 8, 512\n        downsample(512, 4), # batch, 4, 4, 512\n        downsample(512, 4), # batch, 2, 2, 512\n        downsample(512, 4), # batch, 1, 1, 512\n    ]\n    \n    # add upsampling layers (image feature classifiers)\n    upsample_stack = [\n        upsample(512, 4, dropout=True), # batch, 2, 2, 1024\n        upsample(512, 4, dropout=True), # batch, 4, 4, 1024\n        upsample(512, 4, dropout=True), # batch, 8, 8, 1024\n        upsample(512, 4), # batch, 16 ,16, 1024\n        upsample(256, 4), # batch, 32, 32, 512\n        upsample(128, 4), # batch, 64, 64, 256\n        upsample(64, 4) # batch, 128, 128, 128\n    ]\n\n    fc = layers.Conv2DTranspose(3, 4, strides=2, padding='same', activation='tanh') # batch, 256, 256, 3\n\n    # downsampling\n    x = inputs\n    skips = []\n    for down in downsample_stack:\n        x = down(x)\n        skips.append(x)\n    skips = reversed(skips[:-1])\n\n    # upsampling + skip concatenation\n    for up, skip in zip(upsample_stack, skips):\n        x = up(x)\n        x = layers.Concatenate()([x, skip])\n\n    # return model after final layer applied (256x256 RGB image)\n    x = fc(x)\n    return keras.Model(inputs=inputs, outputs=x, name='generator')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T02:20:29.674883Z","iopub.execute_input":"2025-04-08T02:20:29.675182Z","iopub.status.idle":"2025-04-08T02:20:29.684035Z","shell.execute_reply.started":"2025-04-08T02:20:29.675151Z","shell.execute_reply":"2025-04-08T02:20:29.683139Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Discriminator\n\nNow we will create our discriminator, which will classify our images as real or ai-generated images.\n\nOur discriminator is actually a PatchGAN discriminator, which converts our classifications into \"patch maps\", which helps with high-frequency detail accuracy\n\nThe architecture includes the following components:\n- `downsample` the image via convolutional layers to extract image features / details\n- Instance Normalization for model stability\n- Non-linearity techniques, LeakyReLU\n- Convert image to our patch map (is each 32x32 patch real/fake)","metadata":{}},{"cell_type":"code","source":"def discriminator(input_shape=(256, 256, 3)):\n    inputs = layers.Input(shape=input_shape, name='input_img')\n    x = inputs\n\n    ds_stack = [\n        downsample(64, 4, inorm=False), # batch, 128, 128, 64\n        downsample(128, 4), # batch, 64, 64, 128\n        downsample(256, 4), # batch, 32, 32, 256\n    ]\n\n    for ds in ds_stack:\n        x = ds(x)\n    \n    result = keras.Sequential([\n        layers.Conv2D(512, 4, strides=1, padding='same'), # batch, 32, 32, 512\n        layers.GroupNormalization(1), # instance normalization\n        layers.LeakyReLU(),\n        layers.Conv2D(1, 4, strides=1, padding='same'), # batch, 32, 32, 1 <-- our patch map (Real/Fake)\n    ])\n\n    x = result(x)\n\n    return tf.keras.Model(inputs=inputs, outputs=x, name='discriminator')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T02:20:41.985418Z","iopub.execute_input":"2025-04-08T02:20:41.985749Z","iopub.status.idle":"2025-04-08T02:20:41.991283Z","shell.execute_reply.started":"2025-04-08T02:20:41.985715Z","shell.execute_reply":"2025-04-08T02:20:41.990264Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"monet_gen = generator() # photo -> monet\nphoto_gen = generator() # monet -> photo\n\nmonet_discrim = discriminator() # real vs generated monet paintings\nphoto_discrim = discriminator() # real vs generated photos\n\nmonet_gen.summary()\nmonet_discrim.summary()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T02:20:42.381944Z","iopub.execute_input":"2025-04-08T02:20:42.382176Z","iopub.status.idle":"2025-04-08T02:20:42.976447Z","shell.execute_reply.started":"2025-04-08T02:20:42.382156Z","shell.execute_reply":"2025-04-08T02:20:42.975835Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"We shouldn't expect anything out of our untrained generators yet. When we feed an image, we should get nothing back.\n\nHowever, if we take a look at the example images, we might be able to see a couple of features already. Good signs!","metadata":{}},{"cell_type":"code","source":"to_monet = monet_gen(ex_photo)\n\nplt.subplot(121)\nplt.title('Original Photo')\nplt.imshow(ex_photo[0] * 0.5 + 0.5)\n\nplt.subplot(122)\nplt.title('Generated Monet Photo')\nplt.imshow(to_monet[0] * 0.5 + 0.5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T02:20:48.858879Z","iopub.execute_input":"2025-04-08T02:20:48.859168Z","iopub.status.idle":"2025-04-08T02:20:49.316944Z","shell.execute_reply.started":"2025-04-08T02:20:48.859145Z","shell.execute_reply":"2025-04-08T02:20:49.316064Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Model Creation\n\nWith the generator and discriminator completed, we can get started creating our model. Since we are not used paired data, we use 4 loss functions for this problem:\n- **Generator loss**: Can the generator produce images that fool the discriminator?\n- **Discriminator loss**: Can the discriminator distinguish real vs. generated images?\n- **Cycle consistency loss**: Can the generator return the original image after retranslating a translated image?\n- **Identity loss**: Are there alterations to an image that are the target? (Monet -> Monet, Photo -> Photo)\n\nWe also have 4 optimizers for each of our generators / discriminators, that are compiled.","metadata":{}},{"cell_type":"code","source":"class CycleGAN(keras.Model):\n    def __init__(self, monet_gen, photo_gen, monet_dis, photo_dis, lambda_cycle=10):\n        super(CycleGAN, self).__init__()\n        self.monet_gen = monet_gen\n        self.monet_dis = monet_dis\n        self.photo_gen = photo_gen\n        self.photo_dis = photo_dis\n        self.lambda_cycle = lambda_cycle\n    def compile(self, m_g_opt, p_g_opt, m_dis_opt, p_dis_opt, gen_loss_fn, dis_loss_fn, cycle_loss_fn, identity_loss_fn):\n        ''' \n            Compile optimizers and loss functions\n            Args:\n                m_g_opt, p_g_opt: monet / photo generator optimizer\n                m_dis_opt, p_dis_opt: monet / photo discriminator optimizer\n                gen_loss_fn, dis_loss_fn: generator / discriminator loss function\n                cycle_loss_fn, identity_loss_fn: cycle / identity loss function\n        '''\n        super(CycleGAN, self).compile()\n        self.m_g_opt, self.p_g_opt = m_g_opt, p_g_opt # gen optimizers\n        self.m_dis_opt, self.p_dis_opt = m_dis_opt, p_dis_opt # dis optimizers\n        self.gen_loss_fn, self.dis_loss_fn = gen_loss_fn, dis_loss_fn # loss functions\n        self.cycle_loss_fn, self.identity_loss_fn = cycle_loss_fn, identity_loss_fn # loss functions\n\n    def apply_gradients(self, tape, losses, models, optimizers):\n\n        for loss, model, opt in zip(losses, models, optimizers):\n            grad = tape.gradient(loss, model.trainable_variables)\n            opt.apply_gradients(zip(grad, model.trainable_variables))\n            \n    def train_step(self, batch):\n        monet, photo = batch\n\n        with tf.GradientTape(persistent=True) as tape:\n            # Generator\n            fake_monet = self.monet_gen(photo, training=True) # photo -> monet\n            cycled_photo = self.photo_gen(fake_monet, training=True) # photo -> monet -> photo\n\n            fake_photo = self.photo_gen(monet, training=True) # monet -> photo\n            cycled_monet = self.monet_gen(fake_photo, training=True) # monet -> photo -> monet\n\n            same_monet = self.monet_gen(monet, training=True) # monet -> monet\n            same_photo = self.photo_gen(photo, training=True) # photo -> photo\n\n            # Discriminator (are our real / generated photos real?)\n            dis_real_monet = self.monet_dis(monet, training=True) \n            dis_real_photo = self.photo_dis(photo, training=True)\n            \n            dis_fake_monet = self.monet_dis(fake_monet, training=True)\n            dis_fake_photo = self.photo_dis(fake_photo, training=True)\n\n            # Generator loss functions\n            monet_gen_loss = self.gen_loss_fn(dis_fake_monet)\n            photo_gen_loss = self.gen_loss_fn(dis_fake_photo)\n\n            cycle_loss = self.cycle_loss_fn(monet, cycled_monet, self.lambda_cycle) + self.cycle_loss_fn(photo, cycled_photo, self.lambda_cycle)\n            total_monet_gen_loss = monet_gen_loss + cycle_loss + self.identity_loss_fn(monet, same_monet, self.lambda_cycle)\n            total_photo_gen_loss = photo_gen_loss + cycle_loss + self.identity_loss_fn(photo, same_photo, self.lambda_cycle)\n\n            # Discriminator loss functions\n            monet_dis_loss = self.dis_loss_fn(dis_real_monet, dis_fake_monet)\n            photo_dis_loss = self.dis_loss_fn(dis_real_photo, dis_fake_photo)\n\n        # Calc + apply gradients :o\n        self.apply_gradients(tape, losses=[total_monet_gen_loss, total_photo_gen_loss, monet_dis_loss, photo_dis_loss],\n                            models=[self.monet_gen, self.photo_gen, self.monet_dis, self.photo_dis],\n                            optimizers=[self.m_g_opt, self.p_g_opt, self.m_dis_opt, self.p_dis_opt])\n        return {\n            'monet_gen_loss': total_monet_gen_loss,\n            'photo_gen_loss': total_photo_gen_loss,\n            'monet_dis_loss': monet_dis_loss,\n            'photo_dis_loss': photo_dis_loss\n        }","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T02:20:54.598574Z","iopub.execute_input":"2025-04-08T02:20:54.598884Z","iopub.status.idle":"2025-04-08T02:20:54.608881Z","shell.execute_reply.started":"2025-04-08T02:20:54.598860Z","shell.execute_reply":"2025-04-08T02:20:54.607951Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\nNow, we need to create our discriminator, generator, cycle, and identity loss functions.","metadata":{}},{"cell_type":"code","source":"def dis_loss(real, gen):\n    '''\n        real_loss: compares real imgs to matrix of 1s\n        gen_loss: compares fake imgs to matrix of 0s\n\n        returns avg of loss\n    '''\n    real_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(real), real)\n    gen_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.zeros_like(gen), gen)\n    return (real_loss + gen_loss) * 0.5 \n\ndef gen_loss(gen):\n    '''compare gen image to matrix of 1s (gen goal is discrim outputs only real (1) in the matrix)'''\n    return tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(gen), gen)\n\ndef cycle_loss(real, cycled, _lambda):\n    '''average of difference between real image and cycled image'''\n    loss = tf.reduce_mean(tf.abs(real - cycled))\n    return _lambda * loss\n\ndef identity_loss(real, same, _lambda):\n    '''average of difference between real image and same regenerated image'''\n    loss = tf.reduce_mean(tf.abs(real - same))\n    return _lambda * 0.5 * loss","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T02:20:56.929390Z","iopub.execute_input":"2025-04-08T02:20:56.929718Z","iopub.status.idle":"2025-04-08T02:20:56.935558Z","shell.execute_reply.started":"2025-04-08T02:20:56.929692Z","shell.execute_reply":"2025-04-08T02:20:56.934691Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Model Training\n\nOnce we have created our loss functions, we finally create our optimizers (and use a learning rate scheduler!) and train the CycleGAN","metadata":{}},{"cell_type":"code","source":"lr = 2e-4\ndecay = 25000\nepochs= 50\n\nlr_sched = tf.keras.optimizers.schedules.PolynomialDecay(initial_learning_rate=lr,\n                                                            decay_steps=decay,\n                                                            end_learning_rate=0.0,\n                                                            power=1.0)\n\n\nmonet_gen_optimizer = tf.keras.optimizers.Adam(learning_rate=lr_sched, beta_1=0.5)\nphoto_gen_optimizer = tf.keras.optimizers.Adam(learning_rate=lr_sched, beta_1=0.5)\nmonet_dis_optimizer = tf.keras.optimizers.Adam(learning_rate=lr_sched, beta_1=0.5)\nphoto_dis_optimizer = tf.keras.optimizers.Adam(learning_rate=lr_sched, beta_1=0.5)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T02:40:09.950858Z","iopub.execute_input":"2025-04-08T02:40:09.951195Z","iopub.status.idle":"2025-04-08T02:40:09.962291Z","shell.execute_reply.started":"2025-04-08T02:40:09.951165Z","shell.execute_reply":"2025-04-08T02:40:09.961597Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"model = CycleGAN(monet_gen, photo_gen, monet_discrim, photo_discrim)\n\nmodel.compile(monet_gen_optimizer, photo_gen_optimizer, monet_dis_optimizer, photo_dis_optimizer, \n              gen_loss, dis_loss, cycle_loss, identity_loss)\n\nhistory = model.fit(tf.data.Dataset.zip((monet_ds, photo_ds)), epochs=epochs, steps_per_epoch=300)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T02:40:10.204951Z","iopub.execute_input":"2025-04-08T02:40:10.205164Z","iopub.status.idle":"2025-04-08T03:19:11.544623Z","shell.execute_reply.started":"2025-04-08T02:40:10.205146Z","shell.execute_reply":"2025-04-08T03:19:11.543867Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"imgs = list(test_photo_ds.take(8))\n\ndef show_img(title, generated=False):\n    plt.figure(figsize=(12,6))\n    for i, img in enumerate(imgs):\n        plt.subplot(2, 4, i+1)\n        if generated:\n            out = monet_gen(img, training=False)[0].numpy()\n            plt.imshow((out * 127.5 + 127.5).astype(np.uint8))\n        else:\n            plt.imshow(tf.squeeze(img, axis=0) * 0.5 + 0.5)\n        plt.axis('off')\n    plt.suptitle(title)\n    plt.tight_layout()\n    plt.show()\n\nshow_img('Input Photos')\nshow_img('Generated Monet', True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T03:19:11.545989Z","iopub.execute_input":"2025-04-08T03:19:11.546297Z","iopub.status.idle":"2025-04-08T03:19:13.310729Z","shell.execute_reply.started":"2025-04-08T03:19:11.546269Z","shell.execute_reply":"2025-04-08T03:19:13.309840Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## Model Overview\n\n- From the images we can see our model is able to apply the Monet style to the images\n    - The generated image however, retains some of the artifacts let over from the generating process.\n- The loss also stays steady, with the discriminator having the upper hand the majority of the time, signalling that we have not trained our model enough to fool the discriminator.\n- Our regularization techniques are effective, as the generator does find edge cases (loss spikes), or the generator outperforms the discriminator (loss drops)","metadata":{}},{"cell_type":"code","source":"for name, loss in history.history.items():\n    loss_avg = [tf.reduce_mean(l).numpy() for l in loss]\n    plt.plot(range(1, len(loss_avg)+1), loss_avg, label=name)\nplt.xlabel('Epoch')\nplt.ylabel('Loss')\nplt.title('CycleGAN Loss / Epoch')\nplt.legend()\nplt.grid(True)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T03:28:11.392557Z","iopub.execute_input":"2025-04-08T03:28:11.392851Z","iopub.status.idle":"2025-04-08T03:28:11.649753Z","shell.execute_reply.started":"2025-04-08T03:28:11.392828Z","shell.execute_reply":"2025-04-08T03:28:11.648963Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Our patch map shows interesting results from the discriminator, the brighter the color, the more \"real\" it looks to our discriminator.\n\nWe can notice that while the model has learnt to generate Monet-esque images, only certain areas (object edges, backgrounds) are recognized as realistic.\n- This suggests the generator still needs training to improve consistency","metadata":{}},{"cell_type":"code","source":"imgs = list(test_photo_ds.take(3))\ngen_imgs = [monet_gen(img, training=False) for img in imgs]\npatch_outs = [monet_discrim(gen_img, training=False) for gen_img in gen_imgs]\npatch_outs = [tf.squeeze(patch_out, axis=[0,3]) for patch_out in patch_outs]\n\n\nplt.figure(figsize=(12,12))\nsubplots = int(len(imgs)) * 3\n\nfor i in range(len(imgs)):\n    plt.subplot(len(imgs), 3, i * 3 + 1)\n    plt.imshow(tf.squeeze(imgs[i], axis=0) * 0.5 + 0.5)\n    plt.title('Input Photo')\n    plt.axis('off')\n\n    plt.subplot(len(imgs), 3, i * 3 + 2)\n    plt.imshow(tf.squeeze(patch_outs[i]), cmap='viridis')\n    plt.title('Patch Map')\n    plt.axis('off')\n    \n    plt.subplot(len(imgs), 3, i * 3 + 3)\n    plt.imshow(tf.squeeze(gen_imgs[i], axis=0) * 0.5 + 0.5)\n    plt.title('Monet-esque Photo')\n    plt.axis('off')\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T04:50:17.119755Z","iopub.execute_input":"2025-04-08T04:50:17.120053Z","iopub.status.idle":"2025-04-08T04:50:18.488058Z","shell.execute_reply.started":"2025-04-08T04:50:17.120028Z","shell.execute_reply":"2025-04-08T04:50:18.486671Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Conclusion\n\nOverall, our CycleGAN model was able to learn the monet style and capture stylistic elements. However, there still leaves regions that lack realism.\n\nWith more training (more epochs, etc.), dropout adjusting, and adjusting the loss weights the generator could produce more convincing images.","metadata":{}},{"cell_type":"markdown","source":"# Submission","metadata":{}},{"cell_type":"code","source":"savedir = Path('../images')\nsavedir.mkdir(parents=True, exist_ok=True)\n\nfor i, img in enumerate(tqdm(test_photo_ds), 1):\n    pred = monet_gen(img, training=False)[0].numpy()\n    pred = (pred * 127.5 + 127.5).astype(np.uint8)\n    \n    im = Image.fromarray(pred)\n    im.save(savedir / f'{i}.jpg')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T05:05:13.069288Z","iopub.execute_input":"2025-04-08T05:05:13.069718Z","iopub.status.idle":"2025-04-08T05:05:24.604539Z","shell.execute_reply.started":"2025-04-08T05:05:13.069680Z","shell.execute_reply":"2025-04-08T05:05:24.603221Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"shutil.make_archive('/kaggle/working/images', 'zip', '/kaggle/images')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-08T05:04:51.709972Z","iopub.execute_input":"2025-04-08T05:04:51.710333Z","iopub.status.idle":"2025-04-08T05:04:52.170252Z","shell.execute_reply.started":"2025-04-08T05:04:51.710305Z","shell.execute_reply":"2025-04-08T05:04:52.169365Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# Endnotes\n- https://www.kaggle.com/code/amyjang/monet-cyclegan-tutorial/notebook\n- https://www.tensorflow.org/tutorials/generative/cyclegan\n- https://arxiv.org/abs/1703.10593\n- https://arxiv.org/abs/1406.2661","metadata":{}}]}