{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":21755,"databundleVersionId":1475600,"sourceType":"competition"}],"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Monet CycleGAN \n# Introduction\n\nThe aim of this notebook is to to transfer style of Monet paintings to photos using CycleGAN. The generator learns to transform photos into Monet style while discriminator distinguishes real Monet's and transformed photos, helping each other to improve. \n\nThis notebook is based on the competition baseline work of Amy Jang.\n","metadata":{}},{"cell_type":"markdown","source":"## CycleGAN\nAccording to Keras Implementation example, CycleGAN is a model that aims to solve the image-to-image translation problem. The goal of the image-to-image translation problem is to learn the mapping between an input image and an output image using a training set of aligned image pairs. However, obtaining paired examples isn't always feasible. CycleGAN tries to learn this mapping without requiring paired input-output images, using cycle-consistent adversarial networks. This makes it perfect fit for our task of transforming photos to Monet style paintings as they are not paired at all to start with.\n\nThe paper \"Understanding and Implementing CycleGAN in TensorFlow\" https://hardikbansal.github.io/CycleGANBlog/ gives a great overview. \n\nHere is the network architecture from above mentioned paper:\n\n<img src=\"https://hardikbansal.github.io/CycleGANBlog/images/model.jpg\" alt=\"architecture\" class=\"bg-primary mb-1\">\n\n\nWe are using modified UNET architecture for CycleGAN for this project.\n\n","metadata":{}},{"cell_type":"markdown","source":"### Importing Libraries and TPU Configuration\nHere we are importing the necessary libraries and configure TPU or GPU, if available.","metadata":{}},{"cell_type":"code","source":"\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\n\nfrom kaggle_datasets import KaggleDatasets\nimport matplotlib.pyplot as plt\nimport numpy as np\n","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","scrolled":true,"execution":{"iopub.status.busy":"2024-06-15T13:00:00.266066Z","iopub.execute_input":"2024-06-15T13:00:00.266405Z","iopub.status.idle":"2024-06-15T13:00:12.558228Z","shell.execute_reply.started":"2024-06-15T13:00:00.266379Z","shell.execute_reply":"2024-06-15T13:00:12.557275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Configure to use TPU, GPU or CPU based on availability:","metadata":{}},{"cell_type":"code","source":"try:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Device:', tpu.master())\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nexcept:\n        if tf.config.list_physical_devices('GPU'):\n            strategy = tf.distribute.MirroredStrategy()\n            print(\"Running on GPU.\")\n        else:\n            strategy = tf.distribute.get_strategy()\n            print(\"Running on CPU.\")\n\nprint('Number of replicas:', strategy.num_replicas_in_sync)\n\nAUTOTUNE = tf.data.experimental.AUTOTUNE\n    \nprint(tf.__version__)","metadata":{"execution":{"iopub.status.busy":"2024-06-15T13:00:18.590207Z","iopub.execute_input":"2024-06-15T13:00:18.591348Z","iopub.status.idle":"2024-06-15T13:00:19.222222Z","shell.execute_reply.started":"2024-06-15T13:00:18.591313Z","shell.execute_reply":"2024-06-15T13:00:19.221064Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loading the Data\n\nFirst we will load the filenames of TFRecords for monet and photos. ","metadata":{}},{"cell_type":"code","source":"GCS_PATH = KaggleDatasets().get_gcs_path()","metadata":{"execution":{"iopub.status.busy":"2024-06-15T14:54:44.334397Z","iopub.execute_input":"2024-06-15T14:54:44.335285Z","iopub.status.idle":"2024-06-15T14:54:44.783380Z","shell.execute_reply.started":"2024-06-15T14:54:44.335252Z","shell.execute_reply":"2024-06-15T14:54:44.782604Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"MONET_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/monet_tfrec/*.tfrec'))\nprint('Monet TFRecord Files:', len(MONET_FILENAMES))\n\nPHOTO_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/photo_tfrec/*.tfrec'))\nprint('Photo TFRecord Files:', len(PHOTO_FILENAMES))","metadata":{"execution":{"iopub.status.busy":"2024-06-15T13:00:29.430308Z","iopub.execute_input":"2024-06-15T13:00:29.431451Z","iopub.status.idle":"2024-06-15T13:00:30.245694Z","shell.execute_reply.started":"2024-06-15T13:00:29.431406Z","shell.execute_reply":"2024-06-15T13:00:30.244619Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"According to the data description on the kaggle website the images for the competition are sized  256x256 in jpeg or tfrec format. The channel is set to 3 as the these are RGB images. ","metadata":{}},{"cell_type":"code","source":"IMAGE_SIZE = [256, 256]\n\ndef decode_image(image):\n    image = tf.image.decode_jpeg(image, channels=3)\n    image = (tf.cast(image, tf.float32) / 127.5) - 1\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    return image\n\ndef read_tfrecord(example):\n    tfrecord_format = {\n        \"image_name\": tf.io.FixedLenFeature([], tf.string),\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"target\": tf.io.FixedLenFeature([], tf.string)\n    }\n    example = tf.io.parse_single_example(example, tfrecord_format)\n    image = decode_image(example['image'])\n    return image","metadata":{"execution":{"iopub.status.busy":"2024-06-15T13:00:33.802256Z","iopub.execute_input":"2024-06-15T13:00:33.802622Z","iopub.status.idle":"2024-06-15T13:00:33.809769Z","shell.execute_reply.started":"2024-06-15T13:00:33.802595Z","shell.execute_reply":"2024-06-15T13:00:33.808812Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"load_dataset function extracts the image from the files.","metadata":{}},{"cell_type":"code","source":"def load_dataset(filenames, labeled=True, ordered=False):\n    dataset = tf.data.TFRecordDataset(filenames)\n    dataset = dataset.map(read_tfrecord, num_parallel_calls=AUTOTUNE)\n    return dataset","metadata":{"execution":{"iopub.status.busy":"2024-06-15T13:00:37.891568Z","iopub.execute_input":"2024-06-15T13:00:37.891953Z","iopub.status.idle":"2024-06-15T13:00:37.896792Z","shell.execute_reply.started":"2024-06-15T13:00:37.891923Z","shell.execute_reply":"2024-06-15T13:00:37.895820Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will load monet and photo datasets.","metadata":{}},{"cell_type":"code","source":"monet_ds = load_dataset(MONET_FILENAMES, labeled=True).batch(1)\nphoto_ds = load_dataset(PHOTO_FILENAMES, labeled=True).batch(1)","metadata":{"execution":{"iopub.status.busy":"2024-06-15T13:00:41.371083Z","iopub.execute_input":"2024-06-15T13:00:41.371815Z","iopub.status.idle":"2024-06-15T13:00:41.557737Z","shell.execute_reply.started":"2024-06-15T13:00:41.371785Z","shell.execute_reply":"2024-06-15T13:00:41.556933Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Exploratory Data Analysis\nWe will start by visualizing a photo example and Monet example","metadata":{}},{"cell_type":"code","source":"# pick a monet picture and a photo from the data sets\nexample_monet = next(iter(monet_ds))\nexample_photo = next(iter(photo_ds))","metadata":{"execution":{"iopub.status.busy":"2024-06-15T13:00:44.781302Z","iopub.execute_input":"2024-06-15T13:00:44.781932Z","iopub.status.idle":"2024-06-15T13:00:45.983451Z","shell.execute_reply.started":"2024-06-15T13:00:44.781902Z","shell.execute_reply":"2024-06-15T13:00:45.982553Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here is the photo example and Monet example:","metadata":{}},{"cell_type":"code","source":"plt.subplot(121)\nplt.title('Photo')\nplt.imshow(example_photo[0] * 0.5 + 0.5)\n\nplt.subplot(122)\nplt.title('Monet')\nplt.imshow(example_monet[0] * 0.5 + 0.5)","metadata":{"execution":{"iopub.status.busy":"2024-06-15T13:00:50.084213Z","iopub.execute_input":"2024-06-15T13:00:50.084914Z","iopub.status.idle":"2024-06-15T13:00:50.631790Z","shell.execute_reply.started":"2024-06-15T13:00:50.084885Z","shell.execute_reply":"2024-06-15T13:00:50.630872Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's see how many images we have in monet and photo directories:","metadata":{}},{"cell_type":"code","source":"\nMONET_JPG = tf.io.gfile.glob(str(GCS_PATH + '/monet_jpg/*.jpg'))\nPHOTO_JPG = tf.io.gfile.glob(str(GCS_PATH + '/photo_jpg/*.jpg')) \n\n# Display number of files in Monet and Photo directories\nprint('Number of images in Monet directory: ', len(MONET_JPG))\nprint('Number of images in photo directory: ', len(PHOTO_JPG))","metadata":{"execution":{"iopub.status.busy":"2024-06-15T15:00:37.041469Z","iopub.execute_input":"2024-06-15T15:00:37.042210Z","iopub.status.idle":"2024-06-15T15:00:38.859415Z","shell.execute_reply.started":"2024-06-15T15:00:37.042180Z","shell.execute_reply":"2024-06-15T15:00:38.858483Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Generator\nHigh level structure of the Generator from https://hardikbansal.github.io/CycleGANBlog/ :\n\n<img src=\"https://hardikbansal.github.io/CycleGANBlog/images/Generator.jpg\" alt=\"generator\" class=\"bg-primary mb-1\">\n\nDownsample method is the encoder, upsample method is the decoder. \n\nWe'll be using a UNET architecture for our CycleGAN.\n\nA U-Net consists of an encoder (downsampler) and decoder (upsampler).\n\nEach block in the encoder is: Convolution -> Batch normalization -> Leaky ReLU \n\nEach block in the decoder is: Transposed convolution -> Batch normalization -> Dropout -> ReLU There are skip connections between the encoder and decoder (as in the U-Net).\n\nhttps://www.tensorflow.org/tutorials/generative/pix2pix gives great overview on this.\n\nWe will define our downsample and upsample methods for Generator.","metadata":{}},{"cell_type":"markdown","source":"### Encoder: downsample\ndownsample function reduces width and height of the image by 2, applying the filter to every other pixel. ","metadata":{}},{"cell_type":"code","source":"OUTPUT_CHANNELS = 3\n\ndef downsample(filters, size, apply_instancenorm=True):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    result = keras.Sequential()\n    result.add(layers.Conv2D(filters, size, strides=2, padding='same',\n                             kernel_initializer=initializer, use_bias=False))\n\n    if apply_instancenorm:\n        result.add(tf.keras.layers.GroupNormalization(gamma_initializer=gamma_init, groups = -1))\n\n    result.add(layers.LeakyReLU())\n\n    return result","metadata":{"execution":{"iopub.status.busy":"2024-06-15T13:00:55.345377Z","iopub.execute_input":"2024-06-15T13:00:55.346029Z","iopub.status.idle":"2024-06-15T13:00:55.352921Z","shell.execute_reply.started":"2024-06-15T13:00:55.345999Z","shell.execute_reply":"2024-06-15T13:00:55.351948Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Decoder: upsample\nupsample function increases the dimensions of the of the image.","metadata":{}},{"cell_type":"code","source":"def upsample(filters, size, apply_dropout=False):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    result = keras.Sequential()\n    result.add(layers.Conv2DTranspose(filters, size, strides=2,\n                                      padding='same',\n                                      kernel_initializer=initializer,\n                                      use_bias=False))\n\n    result.add(tf.keras.layers.GroupNormalization(gamma_initializer=gamma_init, groups=-1))\n\n    if apply_dropout:\n        result.add(layers.Dropout(0.5))\n\n    result.add(layers.ReLU())\n\n    return result","metadata":{"execution":{"iopub.status.busy":"2024-06-15T13:00:58.988339Z","iopub.execute_input":"2024-06-15T13:00:58.988726Z","iopub.status.idle":"2024-06-15T13:00:58.996426Z","shell.execute_reply.started":"2024-06-15T13:00:58.988691Z","shell.execute_reply":"2024-06-15T13:00:58.995277Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Generator\n\nThe generator downsamples the input image and then upsample and establishes skip connections. Skip connections preserver image information and help bypass the vanishing gradient problem by concatenating the output of a layer to multiple layers. ","metadata":{}},{"cell_type":"code","source":"def Generator():\n    inputs = layers.Input(shape=[256,256,3])\n\n    # bs = batch size\n    down_stack = [\n        downsample(64, 4, apply_instancenorm=False), # (bs, 128, 128, 64)\n        downsample(128, 4), # (bs, 64, 64, 128)\n        downsample(256, 4), # (bs, 32, 32, 256)\n        downsample(512, 4), # (bs, 16, 16, 512)\n        downsample(512, 4), # (bs, 8, 8, 512)\n        downsample(512, 4), # (bs, 4, 4, 512)\n        downsample(512, 4), # (bs, 2, 2, 512)\n        downsample(512, 4), # (bs, 1, 1, 512)\n    ]\n\n    up_stack = [\n        upsample(512, 4, apply_dropout=True), # (bs, 2, 2, 1024)\n        upsample(512, 4, apply_dropout=True), # (bs, 4, 4, 1024)\n        upsample(512, 4, apply_dropout=True), # (bs, 8, 8, 1024)\n        upsample(512, 4), # (bs, 16, 16, 1024)\n        upsample(256, 4), # (bs, 32, 32, 512)\n        upsample(128, 4), # (bs, 64, 64, 256)\n        upsample(64, 4), # (bs, 128, 128, 128)\n    ]\n\n    initializer = tf.random_normal_initializer(0., 0.02)\n    last = layers.Conv2DTranspose(OUTPUT_CHANNELS, 4,\n                                  strides=2,\n                                  padding='same',\n                                  kernel_initializer=initializer,\n                                  activation='tanh') # (bs, 256, 256, 3)\n\n    x = inputs\n\n    # Downsampling through the model\n    skips = []\n    for down in down_stack:\n        x = down(x)\n        skips.append(x)\n\n    skips = reversed(skips[:-1])\n\n    # Upsampling and establishing the skip connections\n    for up, skip in zip(up_stack, skips):\n        x = up(x)\n        x = layers.Concatenate()([x, skip])\n\n    x = last(x)\n\n    return keras.Model(inputs=inputs, outputs=x)","metadata":{"execution":{"iopub.status.busy":"2024-06-15T13:01:02.640751Z","iopub.execute_input":"2024-06-15T13:01:02.641112Z","iopub.status.idle":"2024-06-15T13:01:02.651738Z","shell.execute_reply.started":"2024-06-15T13:01:02.641083Z","shell.execute_reply":"2024-06-15T13:01:02.650786Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's visualize generator architecture:","metadata":{}},{"cell_type":"code","source":"!pip install pydot","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"generator = Generator()\ntf.keras.utils.plot_model(generator, show_shapes=True, dpi=64)","metadata":{"execution":{"iopub.status.busy":"2024-06-15T13:01:07.704232Z","iopub.execute_input":"2024-06-15T13:01:07.705037Z","iopub.status.idle":"2024-06-15T13:01:08.353556Z","shell.execute_reply.started":"2024-06-15T13:01:07.705008Z","shell.execute_reply":"2024-06-15T13:01:08.352632Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Discriminator\n\nThe discriminator classifies the image as real or generated. .","metadata":{}},{"cell_type":"code","source":"def Discriminator():\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    inp = layers.Input(shape=[256, 256, 3], name='input_image')\n\n    x = inp\n\n    down1 = downsample(64, 4, False)(x) # (bs, 128, 128, 64)\n    down2 = downsample(128, 4)(down1) # (bs, 64, 64, 128)\n    down3 = downsample(256, 4)(down2) # (bs, 32, 32, 256)\n\n    zero_pad1 = layers.ZeroPadding2D()(down3) # (bs, 34, 34, 256)\n    conv = layers.Conv2D(512, 4, strides=1,\n                         kernel_initializer=initializer,\n                         use_bias=False)(zero_pad1) # (bs, 31, 31, 512)\n\n    norm1 = tf.keras.layers.GroupNormalization(gamma_initializer=gamma_init, groups=-1)(conv)\n\n    leaky_relu = layers.LeakyReLU()(norm1)\n\n    zero_pad2 = layers.ZeroPadding2D()(leaky_relu) # (bs, 33, 33, 512)\n\n    last = layers.Conv2D(1, 4, strides=1,\n                         kernel_initializer=initializer)(zero_pad2) # (bs, 30, 30, 1)\n\n    return tf.keras.Model(inputs=inp, outputs=last)","metadata":{"execution":{"iopub.status.busy":"2024-06-15T13:01:12.924286Z","iopub.execute_input":"2024-06-15T13:01:12.924887Z","iopub.status.idle":"2024-06-15T13:01:12.933529Z","shell.execute_reply.started":"2024-06-15T13:01:12.924859Z","shell.execute_reply":"2024-06-15T13:01:12.932459Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's look at Discriminator architecture:","metadata":{}},{"cell_type":"code","source":"discriminator = Discriminator()\ntf.keras.utils.plot_model(discriminator, show_shapes=True, dpi=64)","metadata":{"execution":{"iopub.status.busy":"2024-06-15T13:01:17.226453Z","iopub.execute_input":"2024-06-15T13:01:17.227363Z","iopub.status.idle":"2024-06-15T13:01:17.387329Z","shell.execute_reply.started":"2024-06-15T13:01:17.227319Z","shell.execute_reply":"2024-06-15T13:01:17.386445Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n    monet_generator = Generator() # transforms photos to Monet-esque paintings\n    photo_generator = Generator() # transforms Monet paintings to be more like photos\n\n    monet_discriminator = Discriminator() # differentiates real Monet paintings and generated Monet paintings\n    photo_discriminator = Discriminator() # differentiates real photos and generated photos","metadata":{"execution":{"iopub.status.busy":"2024-06-15T13:01:22.787212Z","iopub.execute_input":"2024-06-15T13:01:22.788078Z","iopub.status.idle":"2024-06-15T13:01:23.429809Z","shell.execute_reply.started":"2024-06-15T13:01:22.788046Z","shell.execute_reply":"2024-06-15T13:01:23.428975Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we are generating one example monet style photo before we train the model. It is not very useful yet. ","metadata":{}},{"cell_type":"code","source":"to_monet = monet_generator(example_photo)\n\nplt.subplot(1, 2, 1)\nplt.title(\"Original Photo\")\nplt.imshow(example_photo[0] * 0.5 + 0.5)\n\nplt.subplot(1, 2, 2)\nplt.title(\"Untrained Monet style Photo\")\nplt.imshow(to_monet[0] * 0.5 + 0.5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-06-15T16:07:01.776258Z","iopub.execute_input":"2024-06-15T16:07:01.776589Z","iopub.status.idle":"2024-06-15T16:07:01.817408Z","shell.execute_reply.started":"2024-06-15T16:07:01.776564Z","shell.execute_reply":"2024-06-15T16:07:01.816160Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CycleGAN model\n\nCycleGan class inherits from`tf.keras.Model` and uses generators and discriminators just defined. During the training step, the model transforms a photo to a Monet style painting and then back to a photo. The original photo and the twice-transformed photo should be as similar to one another as possible.\n","metadata":{}},{"cell_type":"code","source":"class CycleGan(keras.Model):\n    def __init__(\n        self,\n        monet_generator,\n        photo_generator,\n        monet_discriminator,\n        photo_discriminator,\n        lambda_cycle=10,\n    ):\n        super(CycleGan, self).__init__()\n        self.m_gen = monet_generator\n        self.p_gen = photo_generator\n        self.m_disc = monet_discriminator\n        self.p_disc = photo_discriminator\n        self.lambda_cycle = lambda_cycle\n        \n    def compile(\n        self,\n        m_gen_optimizer,\n        p_gen_optimizer,\n        m_disc_optimizer,\n        p_disc_optimizer,\n        gen_loss_fn,\n        disc_loss_fn,\n        cycle_loss_fn,\n        identity_loss_fn\n    ):\n        super(CycleGan, self).compile()\n        self.m_gen_optimizer = m_gen_optimizer\n        self.p_gen_optimizer = p_gen_optimizer\n        self.m_disc_optimizer = m_disc_optimizer\n        self.p_disc_optimizer = p_disc_optimizer\n        self.gen_loss_fn = gen_loss_fn\n        self.disc_loss_fn = disc_loss_fn\n        self.cycle_loss_fn = cycle_loss_fn\n        self.identity_loss_fn = identity_loss_fn\n        \n    def train_step(self, batch_data):\n        real_monet, real_photo = batch_data\n        \n        with tf.GradientTape(persistent=True) as tape:\n            # photo to monet back to photo\n            fake_monet = self.m_gen(real_photo, training=True)\n            cycled_photo = self.p_gen(fake_monet, training=True)\n\n            # monet to photo back to monet\n            fake_photo = self.p_gen(real_monet, training=True)\n            cycled_monet = self.m_gen(fake_photo, training=True)\n\n            # generating itself\n            same_monet = self.m_gen(real_monet, training=True)\n            same_photo = self.p_gen(real_photo, training=True)\n\n            # discriminator used to check, inputing real images\n            disc_real_monet = self.m_disc(real_monet, training=True)\n            disc_real_photo = self.p_disc(real_photo, training=True)\n\n            # discriminator used to check, inputing fake images\n            disc_fake_monet = self.m_disc(fake_monet, training=True)\n            disc_fake_photo = self.p_disc(fake_photo, training=True)\n\n            # evaluates generator loss\n            monet_gen_loss = self.gen_loss_fn(disc_fake_monet)\n            photo_gen_loss = self.gen_loss_fn(disc_fake_photo)\n\n            # evaluates total cycle consistency loss\n            total_cycle_loss = self.cycle_loss_fn(real_monet, cycled_monet, self.lambda_cycle) + self.cycle_loss_fn(real_photo, cycled_photo, self.lambda_cycle)\n\n            # evaluates total generator loss\n            total_monet_gen_loss = monet_gen_loss + total_cycle_loss + self.identity_loss_fn(real_monet, same_monet, self.lambda_cycle)\n            total_photo_gen_loss = photo_gen_loss + total_cycle_loss + self.identity_loss_fn(real_photo, same_photo, self.lambda_cycle)\n\n            # evaluates discriminator loss\n            monet_disc_loss = self.disc_loss_fn(disc_real_monet, disc_fake_monet)\n            photo_disc_loss = self.disc_loss_fn(disc_real_photo, disc_fake_photo)\n\n        # Calculate the gradients for generator and discriminator\n        monet_generator_gradients = tape.gradient(total_monet_gen_loss,\n                                                  self.m_gen.trainable_variables)\n        photo_generator_gradients = tape.gradient(total_photo_gen_loss,\n                                                  self.p_gen.trainable_variables)\n\n        monet_discriminator_gradients = tape.gradient(monet_disc_loss,\n                                                      self.m_disc.trainable_variables)\n        photo_discriminator_gradients = tape.gradient(photo_disc_loss,\n                                                      self.p_disc.trainable_variables)\n\n        # Apply the gradients to the optimizer\n        self.m_gen_optimizer.apply_gradients(zip(monet_generator_gradients,\n                                                 self.m_gen.trainable_variables))\n\n        self.p_gen_optimizer.apply_gradients(zip(photo_generator_gradients,\n                                                 self.p_gen.trainable_variables))\n\n        self.m_disc_optimizer.apply_gradients(zip(monet_discriminator_gradients,\n                                                  self.m_disc.trainable_variables))\n\n        self.p_disc_optimizer.apply_gradients(zip(photo_discriminator_gradients,\n                                                  self.p_disc.trainable_variables))\n        \n        return {\n            \"monet_gen_loss\": total_monet_gen_loss,\n            \"photo_gen_loss\": total_photo_gen_loss,\n            \"monet_disc_loss\": monet_disc_loss,\n            \"photo_disc_loss\": photo_disc_loss\n        }","metadata":{"execution":{"iopub.status.busy":"2024-06-15T13:01:43.550726Z","iopub.execute_input":"2024-06-15T13:01:43.551106Z","iopub.status.idle":"2024-06-15T13:01:43.568750Z","shell.execute_reply.started":"2024-06-15T13:01:43.551075Z","shell.execute_reply":"2024-06-15T13:01:43.567672Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loss functions\n\nBelow are the loss functions for disciminator, generator, cycle consistency and identity loss. ","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    def discriminator_loss(real, generated):\n        real_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(real), real)\n\n        generated_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.zeros_like(generated), generated)\n\n        total_disc_loss = real_loss + generated_loss\n\n        return total_disc_loss * 0.5","metadata":{"execution":{"iopub.status.busy":"2024-06-15T13:01:50.113864Z","iopub.execute_input":"2024-06-15T13:01:50.114585Z","iopub.status.idle":"2024-06-15T13:01:50.120632Z","shell.execute_reply.started":"2024-06-15T13:01:50.114553Z","shell.execute_reply":"2024-06-15T13:01:50.119583Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"generator_loss function compares the generated image to a matrix of 1s to find the loss.","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    def generator_loss(generated):\n        return tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(generated), generated)","metadata":{"execution":{"iopub.status.busy":"2024-06-15T13:01:55.581252Z","iopub.execute_input":"2024-06-15T13:01:55.581621Z","iopub.status.idle":"2024-06-15T13:01:55.587352Z","shell.execute_reply.started":"2024-06-15T13:01:55.581593Z","shell.execute_reply":"2024-06-15T13:01:55.586279Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The cycle consistency loss is calculated by the average of original and twice transformed.","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    def calc_cycle_loss(real_image, cycled_image, LAMBDA):\n        loss1 = tf.reduce_mean(tf.abs(real_image - cycled_image))\n\n        return LAMBDA * loss1","metadata":{"execution":{"iopub.status.busy":"2024-06-15T13:01:59.286519Z","iopub.execute_input":"2024-06-15T13:01:59.286877Z","iopub.status.idle":"2024-06-15T13:01:59.294055Z","shell.execute_reply.started":"2024-06-15T13:01:59.286848Z","shell.execute_reply":"2024-06-15T13:01:59.293272Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"identity_loss function compares the input with the output of the generator.","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    def identity_loss(real_image, same_image, LAMBDA):\n        loss = tf.reduce_mean(tf.abs(real_image - same_image))\n        return LAMBDA * 0.5 * loss","metadata":{"execution":{"iopub.status.busy":"2024-06-15T13:02:02.895414Z","iopub.execute_input":"2024-06-15T13:02:02.895779Z","iopub.status.idle":"2024-06-15T13:02:02.904213Z","shell.execute_reply.started":"2024-06-15T13:02:02.895752Z","shell.execute_reply":"2024-06-15T13:02:02.903289Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CycleGAN Model training\n\nLet's compile our model. Since we used `tf.keras.Model` to build our CycleGAN, we can just ude the `fit` function to train our model.","metadata":{}},{"cell_type":"markdown","source":"#### Optimizers","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    monet_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    photo_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n\n    monet_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    photo_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)","metadata":{"execution":{"iopub.status.busy":"2024-06-15T13:02:06.146382Z","iopub.execute_input":"2024-06-15T13:02:06.146750Z","iopub.status.idle":"2024-06-15T13:02:06.163222Z","shell.execute_reply.started":"2024-06-15T13:02:06.146715Z","shell.execute_reply":"2024-06-15T13:02:06.162311Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## CycleGAN Model Instantiation","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    cycle_gan_model = CycleGan(\n        monet_generator, photo_generator, monet_discriminator, photo_discriminator\n    )\n\n    cycle_gan_model.compile(\n        m_gen_optimizer = monet_generator_optimizer,\n        p_gen_optimizer = photo_generator_optimizer,\n        m_disc_optimizer = monet_discriminator_optimizer,\n        p_disc_optimizer = photo_discriminator_optimizer,\n        gen_loss_fn = generator_loss,\n        disc_loss_fn = discriminator_loss,\n        cycle_loss_fn = calc_cycle_loss,\n        identity_loss_fn = identity_loss\n    )","metadata":{"execution":{"iopub.status.busy":"2024-06-15T13:02:13.557287Z","iopub.execute_input":"2024-06-15T13:02:13.557655Z","iopub.status.idle":"2024-06-15T13:02:13.570581Z","shell.execute_reply.started":"2024-06-15T13:02:13.557628Z","shell.execute_reply":"2024-06-15T13:02:13.569715Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model fit","metadata":{}},{"cell_type":"code","source":"cycle_gan_model.fit(\n    tf.data.Dataset.zip((monet_ds, photo_ds)),\n    epochs=25\n)","metadata":{"execution":{"iopub.status.busy":"2024-06-15T13:02:16.697174Z","iopub.execute_input":"2024-06-15T13:02:16.697595Z","iopub.status.idle":"2024-06-15T13:40:03.618986Z","shell.execute_reply.started":"2024-06-15T13:02:16.697565Z","shell.execute_reply":"2024-06-15T13:40:03.618078Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Monet-like images from our Model fit:","metadata":{}},{"cell_type":"code","source":"_, ax = plt.subplots(5, 2, figsize=(12, 12))\nfor i, img in enumerate(photo_ds.take(5)):\n    prediction = monet_generator(img, training=False)[0].numpy()\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    img = (img[0] * 127.5 + 127.5).numpy().astype(np.uint8)\n\n    ax[i, 0].imshow(img)\n    ax[i, 1].imshow(prediction)\n    ax[i, 0].set_title(\"Input Photo\")\n    ax[i, 1].set_title(\"Monet Style\")\n    ax[i, 0].axis(\"off\")\n    ax[i, 1].axis(\"off\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-06-15T13:40:40.113478Z","iopub.execute_input":"2024-06-15T13:40:40.114214Z","iopub.status.idle":"2024-06-15T13:40:42.144944Z","shell.execute_reply.started":"2024-06-15T13:40:40.114184Z","shell.execute_reply":"2024-06-15T13:40:42.143891Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Generate submission file\nHere we will transform the photos to Monet style and save them to a zip file for submission to kaggle. ","metadata":{}},{"cell_type":"code","source":"import PIL\n! mkdir ../images","metadata":{"execution":{"iopub.status.busy":"2024-06-15T13:40:49.266327Z","iopub.execute_input":"2024-06-15T13:40:49.266699Z","iopub.status.idle":"2024-06-15T13:40:50.275433Z","shell.execute_reply.started":"2024-06-15T13:40:49.266671Z","shell.execute_reply":"2024-06-15T13:40:50.274435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"i = 1\nfor img in photo_ds:\n    prediction = monet_generator(img, training=False)[0].numpy()\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    im = PIL.Image.fromarray(prediction)\n    im.save(\"../images/\" + str(i) + \".jpg\")\n    i += 1","metadata":{"execution":{"iopub.status.busy":"2024-06-15T13:40:58.156227Z","iopub.execute_input":"2024-06-15T13:40:58.156641Z","iopub.status.idle":"2024-06-15T13:48:56.480354Z","shell.execute_reply.started":"2024-06-15T13:40:58.156606Z","shell.execute_reply":"2024-06-15T13:48:56.479304Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import shutil\nshutil.make_archive(\"/kaggle/working/images\", 'zip', \"/kaggle/images\")","metadata":{"execution":{"iopub.status.busy":"2024-06-15T13:50:03.569216Z","iopub.execute_input":"2024-06-15T13:50:03.570092Z","iopub.status.idle":"2024-06-15T13:50:07.744255Z","shell.execute_reply.started":"2024-06-15T13:50:03.570059Z","shell.execute_reply":"2024-06-15T13:50:07.743314Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Conclusion\nIn this project we transformed photos into Monet style paintings using modified UNet Architecture for CycleGAN. It served to bolster understanding of CycleGAN architecture with its' generator encoder, decoder and discriminator components. \n\nThe final score from kaggle was 58.99, which is ok for the first result. The transformed images show that our model captured some of the essence of monet style, although real artistry and genious of Monet is far from its' reach.\n\nFor future improvement data augmentation, such as cropping, enlarging or rotating images can be employed. Different epoch counts can be tried out or different, more sophisticated models can be implemented. Due to time constraints further explorations are left to the future. \n","metadata":{}},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}