{"cells":[{"metadata":{},"cell_type":"markdown","source":"# A Monet style transfer challenge:  Exploring DualGAN, data augmentations and receptive field sizes\n\nThis is the final notebook for the *I'm Something of a Painter Myself* competition completed by MLIP 21. This submission is part of the Machine Learning in Practice  course at the Radboud University Nijmegen, The Netherlands.\n\n### What we did\n\nThe purpose of this competitions is to perform style transfer on photos to make them look like Monet's paintings. We used the notebook, [Introduction to CycleGAN - Monet paintings by DimitreOliveira](https://www.kaggle.com/dimitreoliveira/introduction-to-cyclegan-monet-paintings), as a baseline cycleGAN (Zhu et al., 2017) implementation for the task. This notebook is very similar to the tutorial notebook, but already contained some additional structure that could be used to add data augmentation. We sought to improve this baseline by experimenting with three possible adaptations:\n1. Combine the original CycleGAN implementation with the comparable dualGAN structure. The main difference is in how the loss functions can be balanced. For more details see Yi et al. (2017). \n2. Using data augmentation to artificially increase the size of the dataset that the GAN can use to learn from. \n3. Using patchGAN discriminators with different receptive field sizes. PatchGAN discriminators that are known to capture local features (texture and style) efficiently, as orignally proposed in Yi et al. (2017). \n\n### How to read this notebook\n\nWe kept the general structure of the baseline notebook. For every subsection of this notebook, indicated with a subtitle cell, we indicate when no changes have been made to the baseline. If we made changes, we explain these in the subtitle cell directly. \n\n### Authors\n\nThis notebook was created by MLIP group 21, formed by Thijs Luttikholt, Romeo Haak and Thijme de Valk.\n\n### References\nYi, Z., Zhang, H., Tan, P., & Gong, M. (2017). Dualgan: Unsupervised dual learning for image-to-image translation. In Proceedings of the IEEE international conference on computer vision (pp. 2849-2857).\n\nZhu, J. Y., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision (pp. 2223-2232).\n"},{"metadata":{},"cell_type":"markdown","source":"## Dependencies (no changes made)"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import os, random, json, PIL, shutil, re\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom kaggle_datasets import KaggleDatasets\nimport tensorflow as tf\nimport tensorflow.keras.layers as L\nimport tensorflow_addons as tfa\nfrom tensorflow.keras import Model, losses, optimizers","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## TPU configuration (no changes made)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"try:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print(f'Running on TPU {tpu.master()}')\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy()\n\n\nREPLICAS = strategy.num_replicas_in_sync\nprint(f'REPLICAS: {REPLICAS}')\nAUTO = tf.data.experimental.AUTOTUNE","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Model parameters\n\nFirstly, we made the number of epochs lower, in order to prevent the runtime from exceeding three hours. We changed the batch_size parameter from 1 to 4, in order to use batch weight updates. We added a parameter for the receptive field size of the PatchGAN discriminator that should be used. Additionally, we added four parameters for setting the lambda values that balance the different losses. Our experiments indicated that these values allow the model to perform best on this data.  "},{"metadata":{"_kg_hide-input":false,"trusted":true},"cell_type":"code","source":"HEIGHT = 256\nWIDTH = 256\nCHANNELS = 3\nEPOCHS = 31 # default 50\nBATCH_SIZE = 4 # default 1\nRECEPTIVE_FIELD_SIZE = 70 # 16, 46 or default 70\nLAMBDA_MONET_CYCLE = 5 # default 10\nLAMBDA_PHOTO_CYCLE = 7 # default 10\nLAMBDA_MONET_ID = 5 # default 10\nLAMBDA_PHOTO_ID = 5 # default 10","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load data\n\nWe deleted the path to the images, as Kaggle now finds these automatically. "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"GCS_PATH = KaggleDatasets().get_gcs_path()\n\nMONET_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/monet_tfrec/*.tfrec'))\nPHOTO_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/photo_tfrec/*.tfrec'))\n\ndef count_data_items(filenames):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)\n\nn_monet_samples = count_data_items(MONET_FILENAMES)\nn_photo_samples = count_data_items(PHOTO_FILENAMES)\n\nprint(f'Monet TFRecord files: {len(MONET_FILENAMES)}')\nprint(f'Monet image files: {n_monet_samples}')\nprint(f'Photo TFRecord files: {len(PHOTO_FILENAMES)}')\nprint(f'Photo image files: {n_photo_samples}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data augmentation method (new)\n\nWe added a function that gets an image and randomly augments it, possibly using rotation, flipping and/or cropping. "},{"metadata":{"trusted":true},"cell_type":"code","source":"def data_augment(image):\n    p_rotate = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_spatial = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_crop = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    if p_crop > .5:\n        image = tf.image.resize(image, [286, 286])\n        image = tf.image.random_crop(image, size=[256, 256, 3])\n        if p_crop > .9:\n            image = tf.image.resize(image, [300, 300])\n            image = tf.image.random_crop(image, size=[256, 256, 3])\n    \n    if p_rotate > .9:\n        image = tf.image.rot90(image, k=3) # rotate 270º\n    elif p_rotate > .7:\n        image = tf.image.rot90(image, k=2) # rotate 180º\n    elif p_rotate > .5:\n        image = tf.image.rot90(image, k=1) # rotate 90º\n        \n    if p_spatial > .6:\n        image = tf.image.random_flip_left_right(image)\n        image = tf.image.random_flip_up_down(image)\n        if p_spatial > .9:\n            image = tf.image.transpose(image)\n    \n    return image","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Auxiliar functions\n\nWe added the possibility to increase the size of the dataset using data augmentation in the get_gan_dataset function."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def decode_image(image):\n    image = tf.image.decode_jpeg(image, channels=CHANNELS)\n    image = (tf.cast(image, tf.float32) / 127.5) - 1\n    image = tf.reshape(image, [HEIGHT, WIDTH, CHANNELS])\n    return image\n\ndef read_tfrecord(example):\n    tfrecord_format = {\n        'image_name': tf.io.FixedLenFeature([], tf.string),\n        'image':      tf.io.FixedLenFeature([], tf.string),\n        'target':     tf.io.FixedLenFeature([], tf.string)\n    }\n    example = tf.io.parse_single_example(example, tfrecord_format)\n    image = decode_image(example['image'])\n    return image\n\ndef load_dataset(filenames):\n    dataset = tf.data.TFRecordDataset(filenames)\n    dataset = dataset.map(read_tfrecord, num_parallel_calls=AUTO)\n    return dataset\n\ndef get_gan_dataset(monet_files, photo_files, augment=None, repeat=True, shuffle=True, batch_size=1):\n\n    monet_ds = load_dataset(monet_files)\n    photo_ds = load_dataset(photo_files)\n    \n    if augment:\n        monet_ds_aug = monet_ds.map(augment, num_parallel_calls=AUTO)\n        photo_ds_aug = photo_ds.map(augment, num_parallel_calls=AUTO)\n        \n        monet_ds = monet_ds.concatenate(monet_ds_aug)\n        photo_ds = photo_ds.concatenate(photo_ds_aug)\n    if repeat:\n        monet_ds = monet_ds.repeat()\n        photo_ds = photo_ds.repeat()\n    if shuffle:\n        monet_ds = monet_ds.shuffle(2048)\n        photo_ds = photo_ds.shuffle(2048)\n        \n    monet_ds = monet_ds.batch(batch_size, drop_remainder=True)\n    photo_ds = photo_ds.batch(batch_size, drop_remainder=True)\n    monet_ds = monet_ds.cache()\n    photo_ds = photo_ds.cache()\n    monet_ds = monet_ds.prefetch(AUTO)\n    photo_ds = photo_ds.prefetch(AUTO)\n    \n    gan_ds = tf.data.Dataset.zip((monet_ds, photo_ds))\n    \n    return gan_ds\n\ndef display_samples(ds, row, col):\n    ds_iter = iter(ds)\n    plt.figure(figsize=(15, int(15*row/col)))\n    for j in range(row*col):\n        example_sample = next(ds_iter)\n        plt.subplot(row,col,j+1)\n        plt.axis('off')\n        plt.imshow(example_sample[0] * 0.5 + 0.5)\n    plt.show()\n        \ndef display_generated_samples(ds, model, n_samples):\n    ds_iter = iter(ds)\n    for n_sample in range(n_samples):\n        example_sample = next(ds_iter)\n        generated_sample = model.predict(example_sample)\n        \n        plt.subplot(121)\n        plt.title(\"input image\")\n        plt.imshow(example_sample[0] * 0.5 + 0.5)\n        plt.axis('off')\n        \n        plt.subplot(122)\n        plt.title(\"Generated image\")\n        plt.imshow(generated_sample[0] * 0.5 + 0.5)\n        plt.axis('off')\n        plt.show()\n        \ndef predict_and_save(input_ds, generator_model, output_path):\n    i = 1\n    for img in input_ds:\n        prediction = generator_model(img, training=False)[0].numpy() # make predition\n        prediction = (prediction * 127.5 + 127.5).astype(np.uint8)   # re-scale\n        im = PIL.Image.fromarray(prediction)\n        im.save(f'{output_path}{str(i)}.jpg')\n        i += 1\n                \n\n# Model functions\ndef downsample(filters, size, apply_instancenorm=True, strides=2):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = tf.keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    result = tf.keras.Sequential()\n    result.add(L.Conv2D(filters, size, strides=strides, padding='same',\n                        kernel_initializer=initializer, use_bias=False))\n\n    if apply_instancenorm:\n        result.add(tfa.layers.InstanceNormalization(gamma_initializer=gamma_init))\n\n    result.add(L.LeakyReLU())\n\n    return result\n\ndef upsample(filters, size, apply_dropout=False, strides=2):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = tf.keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    result = tf.keras.Sequential()\n    result.add(L.Conv2DTranspose(filters, size, strides=strides, padding='same',\n                                 kernel_initializer=initializer, use_bias=False))\n\n    result.add(tfa.layers.InstanceNormalization(gamma_initializer=gamma_init))\n\n    if apply_dropout:\n        result.add(L.Dropout(0.5))\n\n    result.add(L.ReLU())\n\n    return result","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Looking at a few Monet paintings (no changes made)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"display_samples(load_dataset(MONET_FILENAMES).batch(1), 4, 6)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Looking at a few photo samples (no changes made)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"display_samples(load_dataset(PHOTO_FILENAMES).batch(1), 4, 6)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Look at a few augmented photos (new)\n\nWe added a visualization for the augmented dataset. "},{"metadata":{"trusted":true},"cell_type":"code","source":"display_samples(load_dataset(MONET_FILENAMES).map(data_augment, num_parallel_calls=AUTO).batch(1), 4, 6)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Generator model (no changes made)"},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"OUTPUT_CHANNELS = 3\n\ndef generator_fn():\n    inputs = L.Input(shape=[HEIGHT, WIDTH, CHANNELS])\n\n    down_stack = [\n        downsample(64, 4, apply_instancenorm=False), # (bs, 128, 128, 64)\n        downsample(128, 4),                          # (bs, 64, 64, 128)\n        downsample(256, 4),                          # (bs, 32, 32, 256)\n        downsample(512, 4),                          # (bs, 16, 16, 512)\n        downsample(512, 4),                          # (bs, 8, 8, 512)\n        downsample(512, 4),                          # (bs, 4, 4, 512)\n        downsample(512, 4),                          # (bs, 2, 2, 512)\n        downsample(512, 4),                          # (bs, 1, 1, 512)\n    ]\n\n    up_stack = [\n        upsample(512, 4, apply_dropout=True), # (bs, 2, 2, 1024)\n        upsample(512, 4, apply_dropout=True), # (bs, 4, 4, 1024)\n        upsample(512, 4, apply_dropout=True), # (bs, 8, 8, 1024)\n        upsample(512, 4),                     # (bs, 16, 16, 1024)\n        upsample(256, 4),                     # (bs, 32, 32, 512)\n        upsample(128, 4),                     # (bs, 64, 64, 256)\n        upsample(64, 4),                      # (bs, 128, 128, 128)\n    ]\n\n    initializer = tf.random_normal_initializer(0., 0.02)\n    last = L.Conv2DTranspose(OUTPUT_CHANNELS, 4,\n                             strides=2,\n                             padding='same',\n                             kernel_initializer=initializer,\n                             activation='tanh') # (bs, 256, 256, 3)\n\n    x = inputs\n\n    # Downsampling through the model\n    skips = []\n    for down in down_stack:\n        x = down(x)\n        skips.append(x)\n\n    skips = reversed(skips[:-1])\n\n    # Upsampling and establishing the skip connections\n    for up, skip in zip(up_stack, skips):\n        x = up(x)\n        x = L.Concatenate()([x, skip])\n\n    x = last(x)\n\n    return Model(inputs=inputs, outputs=x)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Discriminator model\n\nWe experimented with three different PatchGAN discriminator architectures. In the end, we found that the original 70x70 receptive field architecture was most effective for capturing style. We added the receptive field sizes for each layer in comments. "},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"def discriminator_fn():\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = tf.keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    inp = L.Input(shape=[HEIGHT, WIDTH, CHANNELS], name='input_image')\n\n    x = inp\n    \n    if RECEPTIVE_FIELD_SIZE == 16:\n        \n        down1 = downsample(64, 4, False)(x) # (bs, 128, 128, 64) \n        # The receptive field size is now 4x4\n        down2 = downsample(128, 4, strides=1)(down1) # (bs, 128, 128, 128)\n        # The receptive field size is now 10x10\n        down3 = downsample(256, 4, strides=1)(down2) # (bs, 128, 128, 256)\n        # The receptive field size is now 16x16\n\n        last = L.Conv2D(1, 1, strides=1,\n                    kernel_initializer=initializer)(down3) # (bs, 128, 128, 1)\n        \n        # The receptive field size is now 16x16\n\n        return Model(inputs=inp, outputs=last)\n    \n    elif RECEPTIVE_FIELD_SIZE == 46:\n\n        down1 = downsample(64, 4, False)(x) # (bs, 128, 128, 64)\n        # The receptive field size is now 4x4\n        down2 = downsample(128, 4)(down1) # (bs, 64, 64, 128)\n        # The receptive field size is now 10x10\n        down3 = downsample(256, 4)(down2) # (bs, 32, 32, 256)\n        # The receptive field size is now 22x22\n        down4 = downsample(512, 4)(down3) # (bs, 16, 16, 512)\n        # The receptive field size is now 46x46\n\n        last = L.Conv2D(1, 1, strides=1,\n                        kernel_initializer=initializer)(down4) # (bs, 16, 16, 1)\n\n        # The receptive field size is now 46x46\n\n        return Model(inputs=inp, outputs=last)\n        \n    else:\n\n        down1 = downsample(64, 4, False)(x) # (bs, 128, 128, 64) \n        # The receptive field size is now 4x4\n        down2 = downsample(128, 4)(down1) # (bs, 64, 64, 128)\n        # The receptive field size is now 10x10\n        down3 = downsample(256, 4)(down2) # (bs, 32, 32, 256)\n        # The receptive field size is now 22x22\n\n        zero_pad1 = L.ZeroPadding2D()(down3) # (bs, 34, 34, 256)\n        conv = L.Conv2D(512, 4, strides=1,\n                        kernel_initializer=initializer,\n                        use_bias=False)(zero_pad1) # (bs, 31, 31, 512)\n\n        # The receptive field size is now 46x46\n\n        norm1 = tfa.layers.InstanceNormalization(gamma_initializer=gamma_init)(conv)\n\n        leaky_relu = L.LeakyReLU()(norm1)\n\n        zero_pad2 = L.ZeroPadding2D()(leaky_relu) # (bs, 33, 33, 512)\n\n        last = L.Conv2D(1, 4, strides=1,\n                        kernel_initializer=initializer)(zero_pad2) # (bs, 30, 30, 1)\n\n        # The receptive field size is now 70x70\n\n        return Model(inputs=inp, outputs=last)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Build model (CycleGAN)\n\nWe adapted the model to work with the four different lambda values provided by the user earlier. These lambda values balance the different losses associated with the model. "},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"with strategy.scope():\n    monet_generator = generator_fn() # transforms photos to Monet-esque paintings\n    photo_generator = generator_fn() # transforms Monet paintings to be more like photos\n\n    monet_discriminator = discriminator_fn() # differentiates real Monet paintings and generated Monet paintings\n    photo_discriminator = discriminator_fn() # differentiates real photos and generated photos\n\n\nclass CycleGan(Model):\n    def __init__(\n        self,\n        monet_generator,\n        photo_generator,\n        monet_discriminator,\n        photo_discriminator,\n        lambda_monet_cycle=LAMBDA_MONET_CYCLE,\n        lambda_photo_cycle=LAMBDA_PHOTO_CYCLE,\n        lambda_monet_id=LAMBDA_MONET_ID,\n        lambda_photo_id=LAMBDA_PHOTO_ID,\n    ):\n        super(CycleGan, self).__init__()\n        self.m_gen = monet_generator\n        self.p_gen = photo_generator\n        self.m_disc = monet_discriminator\n        self.p_disc = photo_discriminator\n        self.lambda_monet_cycle = lambda_monet_cycle\n        self.lambda_photo_cycle = lambda_photo_cycle\n        self.lambda_monet_id = lambda_monet_id\n        self.lambda_photo_id = lambda_photo_id\n        \n    def compile(\n        self,\n        m_gen_optimizer,\n        p_gen_optimizer,\n        m_disc_optimizer,\n        p_disc_optimizer,\n        gen_loss_fn,\n        disc_loss_fn,\n        cycle_loss_fn,\n        identity_loss_fn\n    ):\n        super(CycleGan, self).compile()\n        self.m_gen_optimizer = m_gen_optimizer\n        self.p_gen_optimizer = p_gen_optimizer\n        self.m_disc_optimizer = m_disc_optimizer\n        self.p_disc_optimizer = p_disc_optimizer\n        self.gen_loss_fn = gen_loss_fn\n        self.disc_loss_fn = disc_loss_fn\n        self.cycle_loss_fn = cycle_loss_fn\n        self.identity_loss_fn = identity_loss_fn\n        \n    def train_step(self, batch_data):\n        real_monet, real_photo = batch_data\n        \n        with tf.GradientTape(persistent=True) as tape:\n            # photo to monet back to photo\n            fake_monet = self.m_gen(real_photo, training=True)\n            cycled_photo = self.p_gen(fake_monet, training=True)\n\n            # monet to photo back to monet\n            fake_photo = self.p_gen(real_monet, training=True)\n            cycled_monet = self.m_gen(fake_photo, training=True)\n\n            # generating itself\n            same_monet = self.m_gen(real_monet, training=True)\n            same_photo = self.p_gen(real_photo, training=True)\n\n            # discriminator used to check, inputing real images\n            disc_real_monet = self.m_disc(real_monet, training=True)\n            disc_real_photo = self.p_disc(real_photo, training=True)\n\n            # discriminator used to check, inputing fake images\n            disc_fake_monet = self.m_disc(fake_monet, training=True)\n            disc_fake_photo = self.p_disc(fake_photo, training=True)\n\n            # evaluates generator loss\n            monet_gen_loss = self.gen_loss_fn(disc_fake_monet)\n            photo_gen_loss = self.gen_loss_fn(disc_fake_photo)\n\n            # evaluates total cycle consistency loss\n            total_cycle_loss = self.cycle_loss_fn(real_monet, cycled_monet, self.lambda_monet_cycle) + self.cycle_loss_fn(real_photo, cycled_photo, self.lambda_photo_cycle)\n\n            # evaluates total generator loss\n            total_monet_gen_loss = monet_gen_loss + total_cycle_loss + self.identity_loss_fn(real_monet, same_monet, self.lambda_monet_id)\n            total_photo_gen_loss = photo_gen_loss + total_cycle_loss + self.identity_loss_fn(real_photo, same_photo, self.lambda_photo_id)\n\n            # evaluates discriminator loss\n            monet_disc_loss = self.disc_loss_fn(disc_real_monet, disc_fake_monet)\n            photo_disc_loss = self.disc_loss_fn(disc_real_photo, disc_fake_photo)\n\n        # Calculate the gradients for generator and discriminator\n        monet_generator_gradients = tape.gradient(total_monet_gen_loss,\n                                                  self.m_gen.trainable_variables)\n        photo_generator_gradients = tape.gradient(total_photo_gen_loss,\n                                                  self.p_gen.trainable_variables)\n\n        monet_discriminator_gradients = tape.gradient(monet_disc_loss,\n                                                      self.m_disc.trainable_variables)\n        photo_discriminator_gradients = tape.gradient(photo_disc_loss,\n                                                      self.p_disc.trainable_variables)\n\n        # Apply the gradients to the optimizer\n        self.m_gen_optimizer.apply_gradients(zip(monet_generator_gradients,\n                                                 self.m_gen.trainable_variables))\n\n        self.p_gen_optimizer.apply_gradients(zip(photo_generator_gradients,\n                                                 self.p_gen.trainable_variables))\n\n        self.m_disc_optimizer.apply_gradients(zip(monet_discriminator_gradients,\n                                                  self.m_disc.trainable_variables))\n\n        self.p_disc_optimizer.apply_gradients(zip(photo_discriminator_gradients,\n                                                  self.p_disc.trainable_variables))\n        \n        return {\n            'monet_gen_loss': total_monet_gen_loss,\n            'photo_gen_loss': total_photo_gen_loss,\n            'monet_disc_loss': monet_disc_loss,\n            'photo_disc_loss': photo_disc_loss\n        }","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Loss functions (no changes made)"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"with strategy.scope():\n    # Discriminator loss {0: fake, 1: real} (The discriminator loss outputs the average of the real and generated loss)\n    def discriminator_loss(real, generated):\n        real_loss = losses.BinaryCrossentropy(from_logits=True, reduction=losses.Reduction.NONE)(tf.ones_like(real), real)\n\n        generated_loss = losses.BinaryCrossentropy(from_logits=True, reduction=losses.Reduction.NONE)(tf.zeros_like(generated), generated)\n\n        total_disc_loss = real_loss + generated_loss\n\n        return total_disc_loss * 0.5\n        \n    \n    # Generator loss\n    def generator_loss(generated):\n        return losses.BinaryCrossentropy(from_logits=True, reduction=losses.Reduction.NONE)(tf.ones_like(generated), generated)\n    \n    \n    # Cycle consistency loss (measures if original photo and the twice transformed photo to be similar to one another)\n    with strategy.scope():\n        def calc_cycle_loss(real_image, cycled_image, LAMBDA):\n            loss1 = tf.reduce_mean(tf.abs(real_image - cycled_image))\n\n            return LAMBDA * loss1\n\n    # Identity loss (compares the image with its generator (i.e. photo with photo generator))\n    with strategy.scope():\n        def identity_loss(real_image, same_image, LAMBDA):\n            loss = tf.reduce_mean(tf.abs(real_image - same_image))\n            return LAMBDA * 0.5 * loss","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train\n\nWe adapted the fit call to generate and work with the augmented dataset. Additionally, we changed the steps_per_epoch to take into account the full available data. The smallest dataset is repeated to provide samples to work with when the larger dataset still has samples. "},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"with strategy.scope():\n    # Create generators\n    monet_generator_optimizer = optimizers.Adam(2e-4, beta_1=0.5)\n    photo_generator_optimizer = optimizers.Adam(2e-4, beta_1=0.5)\n\n    # Create discriminators\n    monet_discriminator_optimizer = optimizers.Adam(2e-4, beta_1=0.5)\n    photo_discriminator_optimizer = optimizers.Adam(2e-4, beta_1=0.5)\n    \n    # Create GAN\n    gan_model = CycleGan(monet_generator, photo_generator, \n                         monet_discriminator, photo_discriminator)\n\n    gan_model.compile(m_gen_optimizer=monet_generator_optimizer,\n                      p_gen_optimizer=photo_generator_optimizer,\n                      m_disc_optimizer=monet_discriminator_optimizer,\n                      p_disc_optimizer=photo_discriminator_optimizer,\n                      gen_loss_fn=generator_loss,\n                      disc_loss_fn=discriminator_loss,\n                      cycle_loss_fn=calc_cycle_loss,\n                      identity_loss_fn=identity_loss)\n    \n\nhistory = gan_model.fit(get_gan_dataset(MONET_FILENAMES, PHOTO_FILENAMES, augment=data_augment, repeat=True, shuffle=True, batch_size=BATCH_SIZE), \n                        steps_per_epoch=((max(n_monet_samples, n_photo_samples))//BATCH_SIZE),\n                        epochs=EPOCHS,\n                        verbose=2).history","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Visualize predictions (no changes made)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"display_generated_samples(load_dataset(PHOTO_FILENAMES).batch(1), monet_generator, 8)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Make predictions (no changes made)"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"os.makedirs('../images/') # Create folder to save generated images\n\npredict_and_save(load_dataset(PHOTO_FILENAMES).batch(1), monet_generator, '../images/')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Submission file (no changes made)"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"shutil.make_archive('/kaggle/working/images/', 'zip', '../images')\n\nprint(f\"Generated samples: {len([name for name in os.listdir('../images/') if os.path.isfile(os.path.join('../images/', name))])}\")","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}