{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":21755,"databundleVersionId":1475600,"sourceType":"competition"}],"dockerImageVersionId":30698,"isInternetEnabled":false,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nfrom tensorflow.keras.layers import GroupNormalization\nfrom tensorflow.keras.optimizers import Adam\nfrom kaggle_datasets import KaggleDatasets\nimport matplotlib.pyplot as plt\nimport numpy as np","metadata":{"execution":{"iopub.status.busy":"2024-04-23T20:03:56.531514Z","iopub.execute_input":"2024-04-23T20:03:56.531954Z","iopub.status.idle":"2024-04-23T20:04:12.699123Z","shell.execute_reply.started":"2024-04-23T20:03:56.531918Z","shell.execute_reply":"2024-04-23T20:04:12.697840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load the datasets\n","metadata":{}},{"cell_type":"markdown","source":"The data is loaded into two datasets: the real monet images, and the real photo images. Both sets have image names and targets stored in TFRecord files, which are first loaded and correctly mapped to be able to access image data. Code/functions provided by the public Kaggle starter notebook 'Monet CycleGAN Tutorial' by Amy Jang are used to gather the images.","metadata":{}},{"cell_type":"code","source":"MONET_FILENAMES = tf.io.gfile.glob('/kaggle/input/gan-getting-started/monet_tfrec/*.tfrec')\nprint('Monet TFRecord Files:', len(MONET_FILENAMES))\n\nPHOTO_FILENAMES = tf.io.gfile.glob('/kaggle/input/gan-getting-started/photo_tfrec/*.tfrec')\nprint('Photo TFRecord Files:', len(PHOTO_FILENAMES))","metadata":{"execution":{"iopub.status.busy":"2024-04-23T03:37:02.827716Z","iopub.execute_input":"2024-04-23T03:37:02.827980Z","iopub.status.idle":"2024-04-23T03:37:02.853394Z","shell.execute_reply.started":"2024-04-23T03:37:02.827958Z","shell.execute_reply":"2024-04-23T03:37:02.852576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Functions to load the dataset","metadata":{}},{"cell_type":"markdown","source":"The TFRecord files can be read, and the associated images accessed through the following provided functions. Each TFRecord entry comes with string fields for image_name, image, and target. Additionally, the image files themselves are RGB images already sized 256x256. Each image thus contains 3 channels, and each channel is rescaled to a value in the range [-1,1].","metadata":{}},{"cell_type":"code","source":"IMAGE_SIZE = [256, 256]\n\ndef decode_image(image):\n    image = tf.image.decode_jpeg(image, channels=3) # 3 channels for RGB\n    image = (tf.cast(image, tf.float32) / 127.5) - 1 # rescale each channel to [-1,1]\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    return image\n\n#function to parse a TFRecord file's structure. This is passed into the dataset.map function for decoding\ndef read_tfrecord(example):\n    tfrecord_format = {\n        \"image_name\": tf.io.FixedLenFeature([], tf.string),\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"target\": tf.io.FixedLenFeature([], tf.string)\n    }\n    example = tf.io.parse_single_example(example, tfrecord_format)\n    image = decode_image(example['image'])\n    return image\n\n#function to load the entire dataset from each TFRecord. Maps using the above TFRecord file structure\ndef load_dataset(filenames, labeled=True, ordered=False):\n    dataset = tf.data.TFRecordDataset(filenames)\n    dataset = dataset.map(read_tfrecord, num_parallel_calls=tf.data.AUTOTUNE)\n    return dataset","metadata":{"execution":{"iopub.status.busy":"2024-04-23T03:37:02.854766Z","iopub.execute_input":"2024-04-23T03:37:02.855025Z","iopub.status.idle":"2024-04-23T03:37:02.863063Z","shell.execute_reply.started":"2024-04-23T03:37:02.855002Z","shell.execute_reply":"2024-04-23T03:37:02.862094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Viewing example images","metadata":{}},{"cell_type":"code","source":"monet_ds = load_dataset(MONET_FILENAMES, labeled=True).batch(1)\nphoto_ds = load_dataset(PHOTO_FILENAMES, labeled=True).batch(1)\n\nexample_monet = next(iter(monet_ds))\nexample_photo = next(iter(photo_ds))\n\nplt.subplot(121)\nplt.title('Photo')\nplt.imshow(example_photo[0] * 0.5 + 0.5)\n\nplt.subplot(122)\nplt.title('Monet')\nplt.imshow(example_monet[0] * 0.5 + 0.5)","metadata":{"execution":{"iopub.status.busy":"2024-04-23T03:37:02.864455Z","iopub.execute_input":"2024-04-23T03:37:02.865122Z","iopub.status.idle":"2024-04-23T03:37:03.515321Z","shell.execute_reply.started":"2024-04-23T03:37:02.865090Z","shell.execute_reply":"2024-04-23T03:37:03.514399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Generator","metadata":{}},{"cell_type":"markdown","source":"## Downsampling\nAn encoder is used for downsampling. The downsample function creates a downsampling block consisting of a 2D Convlution layer, an optional instance normalization layer, and a leaky ReLU activation function. The Conv2D layer additionally uses a stride of 2, which will halve the weight and height of the input size by applying the convolution operation to every other pixel. Additionally, we apply instance normalization as opposed to batch normalization. This greatly reduces computation time with similar results, as outlined in this paper (https://arxiv.org/pdf/1607.08022.pdf).","metadata":{}},{"cell_type":"code","source":"#creates a downsampling block\ndef downsample(filters, size, i, apply_instancenorm=True):\n    initializer = tf.random_normal_initializer(0., 0.02) #kernel weight initialization\n\n    result = keras.Sequential(name=f'Downsample_{i}')\n    result.add(layers.Conv2D(filters, size, strides=2, padding='same',\n                             kernel_initializer=initializer, use_bias=False))\n\n    if apply_instancenorm:\n        result.add(tf.keras.layers.GroupNormalization(groups=-1)) #Applies an instance normalization\n\n    result.add(layers.LeakyReLU())\n\n    return result","metadata":{"execution":{"iopub.status.busy":"2024-04-23T20:04:18.073901Z","iopub.execute_input":"2024-04-23T20:04:18.074329Z","iopub.status.idle":"2024-04-23T20:04:18.082918Z","shell.execute_reply.started":"2024-04-23T20:04:18.074295Z","shell.execute_reply":"2024-04-23T20:04:18.081203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Upsampling\nA downsample is followed by an upsample, which increases the dimension of the input. This is accomplished through a Conv2DTranspose layer, and the function to create an upsampling block is provided below.","metadata":{}},{"cell_type":"code","source":"def upsample(filters, size, i, apply_dropout=False):\n    initializer = tf.random_normal_initializer(0., 0.02)\n\n    result = keras.Sequential(name=f'Upsample_{i}')\n    result.add(layers.Conv2DTranspose(filters, size, strides=2,\n                                      padding='same',\n                                      kernel_initializer=initializer,\n                                      use_bias=False))\n\n    result.add(tf.keras.layers.GroupNormalization(groups=-1))\n\n    if apply_dropout:\n        result.add(layers.Dropout(0.5))\n\n    result.add(layers.ReLU())\n\n    return result","metadata":{"execution":{"iopub.status.busy":"2024-04-23T20:04:23.311076Z","iopub.execute_input":"2024-04-23T20:04:23.311522Z","iopub.status.idle":"2024-04-23T20:04:23.319887Z","shell.execute_reply.started":"2024-04-23T20:04:23.311488Z","shell.execute_reply":"2024-04-23T20:04:23.318702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Building the generator\nThis function defines the model used for the generators. It consists of downsampling blocks to get lower level features of the image, and upsampling blocks to rebuild the image. Each upsample block is concatenated with the output of the downsampling block with the same output size. This process is known as a long skip connection, where the output of a downsample block is additionally forwarded to a later layer. Long skip connections help prevent vanishing gradients. In this case, it follows a U-NET architecture. Each downsample and upsample uses a leakyReLU activation, except for the final upsample block, which uses a tanh activation for the output layer as a reconstructed image.\n\nAdditionally, two convolutional layers (stride 1) are added after each concatenation.","metadata":{}},{"cell_type":"code","source":"OUTPUT_CHANNELS = 3\ndef Generator():\n    inputs = layers.Input(shape=[256,256,3])\n    \n    #Set of downsample blocks, each downsample halves the image size due to stride=2\n    # bs = batch size\n    down_stack = [\n        downsample(64, 4, 1, apply_instancenorm=False), # (bs, 128, 128, 64)\n        downsample(128, 4, 2), # (bs, 64, 64, 128)\n        downsample(256, 4, 3), # (bs, 32, 32, 256)\n        downsample(512, 4, 4), # (bs, 16, 16, 512)\n        downsample(512, 4, 5), # (bs, 8, 8, 512)\n        downsample(512, 4, 6), # (bs, 4, 4, 512)\n        downsample(512, 4, 7), # (bs, 2, 2, 512)\n        downsample(512, 4, 8), # (bs, 1, 1, 512)\n    ]\n    \n    #Set of upsample blocks, each upsample doubles the image size due to stride=2.\n    #The filter size should match its complimenting downsample for concatenation,\n    #and should end with a (bs, 128, 128, filters) shape for the final tanh output layer\n    up_stack = [\n        upsample(512, 4, 7, apply_dropout=True), # (bs, 2, 2, 1024)\n        upsample(512, 4, 6, apply_dropout=True), # (bs, 4, 4, 1024)\n        upsample(512, 4, 5, apply_dropout=True), # (bs, 8, 8, 1024)\n        upsample(512, 4, 4), # (bs, 16, 16, 1024)\n        upsample(256, 4, 3), # (bs, 32, 32, 512)\n        upsample(128, 4, 2), # (bs, 64, 64, 256)\n        upsample(64, 4, 1), # (bs, 128, 128, 128)\n    ]\n    \n    #Final upsampling, must be shaped (bs, 256, 256, 3)\n    initializer = tf.random_normal_initializer(0., 0.02)\n    last = layers.Conv2DTranspose(OUTPUT_CHANNELS, 4,\n                                  strides=2,\n                                  padding='same',\n                                  kernel_initializer=initializer,\n                                  activation='tanh') # (bs, 256, 256, 3)\n\n    x = inputs\n\n    # Downsampling through the model\n    skips = []\n    for down in down_stack:\n        x = down(x)\n        skips.append(x)\n\n    skips = reversed(skips[:-1])\n\n    # Upsampling and establishing the skip connections\n    for up, skip in zip(up_stack, skips):\n        x = up(x)\n        x = layers.Concatenate()([x, skip])\n        #print(x.shape[3])\n        x = layers.Conv2D(x.shape[3], (3,3), strides=1, padding='same',\n                             kernel_initializer=initializer, use_bias=False, activation=layers.LeakyReLU())(x)\n        x = layers.Conv2D(x.shape[3], (3,3), strides=1, padding='same',\n                             kernel_initializer=initializer, use_bias=False, activation=layers.LeakyReLU())(x)\n\n    x = last(x)\n\n    return keras.Model(inputs=inputs, outputs=x)","metadata":{"execution":{"iopub.status.busy":"2024-04-23T03:37:03.538868Z","iopub.execute_input":"2024-04-23T03:37:03.539307Z","iopub.status.idle":"2024-04-23T03:37:03.551746Z","shell.execute_reply.started":"2024-04-23T03:37:03.539277Z","shell.execute_reply":"2024-04-23T03:37:03.550954Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_generator = Generator()\ntest_generator.summary()","metadata":{"execution":{"iopub.status.busy":"2024-04-23T03:37:03.552603Z","iopub.execute_input":"2024-04-23T03:37:03.552836Z","iopub.status.idle":"2024-04-23T03:37:03.908761Z","shell.execute_reply.started":"2024-04-23T03:37:03.552815Z","shell.execute_reply":"2024-04-23T03:37:03.907915Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Discriminator","metadata":{}},{"cell_type":"markdown","source":"The role of the discriminator is to classify images as either a real image or a fake image. Fake images are those which are created by the generator. In this case, the discriminator is implemented as a second neural network which accepts an image of shape (256x256x3) and outputs a smaller image of shape (30,30,1). This form is known as a PatchGAN, where each \"patch\" or section of the original 256x256 image is mapped to 30x30 sections and classified as real or fake. Other discriminators can be designed to return a single value to provide classification.","metadata":{}},{"cell_type":"code","source":"def Discriminator():\n    initializer = tf.random_normal_initializer(0., 0.02)\n\n    inp = layers.Input(shape=[256, 256, 3], name='input_image')\n\n    x = inp\n\n    down1 = downsample(64, 4, 1, False)(x) # (bs, 128, 128, 64)\n    down2 = downsample(128, 4, 2)(down1) # (bs, 64, 64, 128)\n    down3 = downsample(256, 4, 3)(down2) # (bs, 32, 32, 256)\n\n    zero_pad1 = layers.ZeroPadding2D()(down3) # (bs, 34, 34, 256)\n    conv = layers.Conv2D(512, 4, strides=1,\n                         kernel_initializer=initializer,\n                         use_bias=False)(zero_pad1) # (bs, 31, 31, 512)\n\n    norm1 = tf.keras.layers.GroupNormalization(groups=-1)(conv)\n\n    leaky_relu = layers.LeakyReLU()(norm1)\n\n    zero_pad2 = layers.ZeroPadding2D()(leaky_relu) # (bs, 33, 33, 512)\n\n    last = layers.Conv2D(1, 4, strides=1,\n                         kernel_initializer=initializer)(zero_pad2) # (bs, 30, 30, 1)\n\n    return tf.keras.Model(inputs=inp, outputs=last)","metadata":{"execution":{"iopub.status.busy":"2024-04-23T20:04:30.884591Z","iopub.execute_input":"2024-04-23T20:04:30.885004Z","iopub.status.idle":"2024-04-23T20:04:30.894572Z","shell.execute_reply.started":"2024-04-23T20:04:30.884973Z","shell.execute_reply":"2024-04-23T20:04:30.893670Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test_discriminator = Discriminator()\ntest_discriminator.summary()","metadata":{"execution":{"iopub.status.busy":"2024-04-23T20:04:33.985140Z","iopub.execute_input":"2024-04-23T20:04:33.985529Z","iopub.status.idle":"2024-04-23T20:04:34.197003Z","shell.execute_reply.started":"2024-04-23T20:04:33.985500Z","shell.execute_reply":"2024-04-23T20:04:34.195197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CycleGAN class","metadata":{}},{"cell_type":"markdown","source":"The model uses the CycleGan class created in Amy Jang's tutorial to contain the necessary components of a cycleGAN. The class inherits from a tf.keras.Model to allow the use of fit, compile, and other functions of a Keras Model. Explained in the subsequent sections, the cycleGAN model contains two generators and two discriminators, and utilizes 4 loss functions to calculate the total loss of the model.","metadata":{}},{"cell_type":"code","source":"class CycleGan(keras.Model):\n    def __init__(\n        self,\n        monet_generator,\n        photo_generator,\n        monet_discriminator,\n        photo_discriminator,\n        lambda_cycle=10,\n    ):\n        super(CycleGan, self).__init__()\n        self.m_gen = monet_generator\n        self.p_gen = photo_generator\n        self.m_disc = monet_discriminator\n        self.p_disc = photo_discriminator\n        self.lambda_cycle = lambda_cycle\n        \n    def compile(\n        self,\n        m_gen_optimizer,\n        p_gen_optimizer,\n        m_disc_optimizer,\n        p_disc_optimizer,\n        gen_loss_fn,\n        disc_loss_fn,\n        cycle_loss_fn,\n        identity_loss_fn\n    ):\n        super(CycleGan, self).compile()\n        self.m_gen_optimizer = m_gen_optimizer\n        self.p_gen_optimizer = p_gen_optimizer\n        self.m_disc_optimizer = m_disc_optimizer\n        self.p_disc_optimizer = p_disc_optimizer\n        self.gen_loss_fn = gen_loss_fn\n        self.disc_loss_fn = disc_loss_fn\n        self.cycle_loss_fn = cycle_loss_fn\n        self.identity_loss_fn = identity_loss_fn\n        \n    def train_step(self, batch_data):\n        real_monet, real_photo = batch_data\n        \n        with tf.GradientTape(persistent=True) as tape:\n            # photo to monet back to photo\n            fake_monet = self.m_gen(real_photo, training=True)\n            cycled_photo = self.p_gen(fake_monet, training=True)\n\n            # monet to photo back to monet\n            fake_photo = self.p_gen(real_monet, training=True)\n            cycled_monet = self.m_gen(fake_photo, training=True)\n\n            # generating itself\n            same_monet = self.m_gen(real_monet, training=True)\n            same_photo = self.p_gen(real_photo, training=True)\n\n            # discriminator used to check, inputing real images\n            disc_real_monet = self.m_disc(real_monet, training=True)\n            disc_real_photo = self.p_disc(real_photo, training=True)\n\n            # discriminator used to check, inputing fake images\n            disc_fake_monet = self.m_disc(fake_monet, training=True)\n            disc_fake_photo = self.p_disc(fake_photo, training=True)\n\n            # evaluates generator loss\n            monet_gen_loss = self.gen_loss_fn(disc_fake_monet)\n            photo_gen_loss = self.gen_loss_fn(disc_fake_photo)\n\n            # evaluates total cycle consistency loss\n            total_cycle_loss = self.cycle_loss_fn(real_monet, cycled_monet, self.lambda_cycle) + self.cycle_loss_fn(real_photo, cycled_photo, self.lambda_cycle)\n\n            # evaluates total generator loss\n            total_monet_gen_loss = monet_gen_loss + total_cycle_loss + self.identity_loss_fn(real_monet, same_monet, self.lambda_cycle)\n            total_photo_gen_loss = photo_gen_loss + total_cycle_loss + self.identity_loss_fn(real_photo, same_photo, self.lambda_cycle)\n\n            # evaluates discriminator loss\n            monet_disc_loss = self.disc_loss_fn(disc_real_monet, disc_fake_monet)\n            photo_disc_loss = self.disc_loss_fn(disc_real_photo, disc_fake_photo)\n\n        # Calculate the gradients for generator and discriminator\n        monet_generator_gradients = tape.gradient(total_monet_gen_loss,\n                                                  self.m_gen.trainable_variables)\n        photo_generator_gradients = tape.gradient(total_photo_gen_loss,\n                                                  self.p_gen.trainable_variables)\n\n        monet_discriminator_gradients = tape.gradient(monet_disc_loss,\n                                                      self.m_disc.trainable_variables)\n        photo_discriminator_gradients = tape.gradient(photo_disc_loss,\n                                                      self.p_disc.trainable_variables)\n\n        # Apply the gradients to the optimizer\n        self.m_gen_optimizer.apply_gradients(zip(monet_generator_gradients,\n                                                 self.m_gen.trainable_variables))\n\n        self.p_gen_optimizer.apply_gradients(zip(photo_generator_gradients,\n                                                 self.p_gen.trainable_variables))\n\n        self.m_disc_optimizer.apply_gradients(zip(monet_discriminator_gradients,\n                                                  self.m_disc.trainable_variables))\n\n        self.p_disc_optimizer.apply_gradients(zip(photo_discriminator_gradients,\n                                                  self.p_disc.trainable_variables))\n        \n        return {\n            \"monet_gen_loss\": total_monet_gen_loss,\n            \"photo_gen_loss\": total_photo_gen_loss,\n            \"monet_disc_loss\": monet_disc_loss,\n            \"photo_disc_loss\": photo_disc_loss\n        }","metadata":{"execution":{"iopub.status.busy":"2024-04-23T03:37:03.919145Z","iopub.execute_input":"2024-04-23T03:37:03.919446Z","iopub.status.idle":"2024-04-23T03:37:03.936771Z","shell.execute_reply.started":"2024-04-23T03:37:03.919401Z","shell.execute_reply":"2024-04-23T03:37:03.936014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Preparing the cycleGAN","metadata":{}},{"cell_type":"markdown","source":"In a cycleGAN model, we use two generators and two discriminators to be able to convert in both directions (i.e. Generate Monet-like paintings from photographs, as well as generate photographs from Monet paintings). To instantiate an object of the cycleGAN class above, we must first create both generators and both discriminators. When we compile the model, we will additionally need 4 optimizers (one for each generator and discriminator) as well as formulate the overall loss of the cycleGAN model.\n\nEach of the 4 optimizers used will use an Adam optimizer with default values, save for learning rate of 0.0001 and beta_1 of 0.5.","metadata":{}},{"cell_type":"code","source":"monet_generator = Generator() # photograph -> Monet\nphoto_generator = Generator() # Monet -> photograph\n\nmonet_discriminator = Discriminator() # differentiates real and generated Monet paintings\nphoto_discriminator = Discriminator() # differentiates real and generated photographs\n\nmonet_generator_optimizer = Adam(1e-4, beta_1=0.5)\nphoto_generator_optimizer = Adam(1e-4, beta_1=0.5)\n\nmonet_discriminator_optimizer = Adam(1e-4, beta_1=0.5)\nphoto_discriminator_optimizer = Adam(1e-4, beta_1=0.5)","metadata":{"execution":{"iopub.status.busy":"2024-04-23T03:37:03.940275Z","iopub.execute_input":"2024-04-23T03:37:03.940549Z","iopub.status.idle":"2024-04-23T03:37:04.617988Z","shell.execute_reply.started":"2024-04-23T03:37:03.940527Z","shell.execute_reply":"2024-04-23T03:37:04.616999Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Implementing the loss functions\nThe overall loss in the cycleGAN model is a combination of 4 loss functions:\n- discriminator loss\n- generator loss\n- cycle loss\n- identity loss","metadata":{}},{"cell_type":"markdown","source":"### Discriminator loss\nThe output of the discriminator is a smaller image (30x30) where each pixel contains a scalar value classifying that section of the input image as real or generated. A real image will thus be defined as a matrix of 1s, and a generated image as a matrix of 0s. That is, a perfect discriminator should output only 1s for real images, and only 0s for generated ones. The overall discriminator loss is the average of the loss from real images and the loss from generated images. \n\nThe discriminator and generator losses are similar to that used in Pix2Pix, and the function is additionally used from the tensorflow tutorial.","metadata":{}},{"cell_type":"code","source":"loss_obj = tf.keras.losses.BinaryCrossentropy(from_logits=True)\n\ndef discriminator_loss(real, generated):\n  real_loss = loss_obj(tf.ones_like(real), real)\n\n  generated_loss = loss_obj(tf.zeros_like(generated), generated)\n\n  total_disc_loss = real_loss + generated_loss\n\n  return total_disc_loss * 0.5","metadata":{"execution":{"iopub.status.busy":"2024-04-23T03:37:04.619028Z","iopub.execute_input":"2024-04-23T03:37:04.619311Z","iopub.status.idle":"2024-04-23T03:37:04.624613Z","shell.execute_reply.started":"2024-04-23T03:37:04.619288Z","shell.execute_reply":"2024-04-23T03:37:04.623623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Generator loss\nThe generator aims to fool the discriminator. While the discriminator's goal is to classify generated images as a matrix of 0s, the generator's goal is for generated images to be classified as a matrix of 1s. The loss of the generator has a similar form to the discriminator loss, however it compares the discriminator's output of generated images to a matrix of 1s.","metadata":{}},{"cell_type":"code","source":"def generator_loss(generated):\n  return loss_obj(tf.ones_like(generated), generated)","metadata":{"execution":{"iopub.status.busy":"2024-04-23T03:37:04.625648Z","iopub.execute_input":"2024-04-23T03:37:04.625895Z","iopub.status.idle":"2024-04-23T03:37:04.635384Z","shell.execute_reply.started":"2024-04-23T03:37:04.625874Z","shell.execute_reply":"2024-04-23T03:37:04.634485Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Cycle loss\nThe cycle loss is unique to the cycleGAN implementation, in contrast to other methods such as pix2pix. Unlike pix2pix, cycleGAN can operate on unpaired data and thus an input/target pair is not necessarily meaningful. Thus cycleGAN additionally includes a cycle consistency loss, or the reconstruction loss incurred by applying a transformation with a generator and returning back to the original image with the other generator. In other words, a photograph converted to a Monet painting through the Monet generator and back through the Photograph generator should be identical to the original photograph.","metadata":{}},{"cell_type":"code","source":"def calc_cycle_loss(real_image, cycled_image, LAMBDA):\n    loss1 = tf.reduce_mean(tf.abs(real_image - cycled_image))\n\n    return LAMBDA * loss1","metadata":{"execution":{"iopub.status.busy":"2024-04-23T03:37:04.636489Z","iopub.execute_input":"2024-04-23T03:37:04.636817Z","iopub.status.idle":"2024-04-23T03:37:04.645848Z","shell.execute_reply.started":"2024-04-23T03:37:04.636786Z","shell.execute_reply":"2024-04-23T03:37:04.645091Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Identity loss\nThe identity loss enforces the fact that a generator should not drastically alter an image that is already of the target class. In our example, passing a Monet painting into the Monet generator should output the original Monet painting, and likewise for a photograph in the photograph generator.","metadata":{}},{"cell_type":"code","source":"def identity_loss(real_image, same_image, LAMBDA):\n    loss = tf.reduce_mean(tf.abs(real_image - same_image))\n    return LAMBDA * 0.5 * loss","metadata":{"execution":{"iopub.status.busy":"2024-04-23T03:37:04.646783Z","iopub.execute_input":"2024-04-23T03:37:04.647011Z","iopub.status.idle":"2024-04-23T03:37:04.655195Z","shell.execute_reply.started":"2024-04-23T03:37:04.646991Z","shell.execute_reply":"2024-04-23T03:37:04.654387Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Building the cycleGAN\nA cycleGAN is instantiated with the previously built generators and discriminators, and compiled with the desired optimizers and loss functions. As the cycleGAN class inherited from a keras.Model, it can be applied like any other model with the compile and fit routines.","metadata":{}},{"cell_type":"code","source":"#instantiate the cycleGAN\ncycle_gan_model = CycleGan(\n    monet_generator, photo_generator, monet_discriminator, photo_discriminator\n)\n\ncycle_gan_model.compile(\n    m_gen_optimizer = monet_generator_optimizer,\n    p_gen_optimizer = photo_generator_optimizer,\n    m_disc_optimizer = monet_discriminator_optimizer,\n    p_disc_optimizer = photo_discriminator_optimizer,\n    gen_loss_fn = generator_loss,\n    disc_loss_fn = discriminator_loss,\n    cycle_loss_fn = calc_cycle_loss,\n    identity_loss_fn = identity_loss\n)\n\n#train the cycleGAN on the dataset\ncycle_gan_model.fit(\n    tf.data.Dataset.zip((monet_ds, photo_ds)),\n    epochs=25,\n)\n\nmonet_generator.save(\"Monet_Generator.keras\")","metadata":{"execution":{"iopub.status.busy":"2024-04-23T03:37:04.656150Z","iopub.execute_input":"2024-04-23T03:37:04.656404Z","iopub.status.idle":"2024-04-23T04:17:38.167668Z","shell.execute_reply.started":"2024-04-23T03:37:04.656383Z","shell.execute_reply":"2024-04-23T04:17:38.166623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Using the Monet generator\nImages can now be passed into the Monet generator to create monet images (the photograph generator could also be used to transform Monet paintings, though we arent interested in it). The preprocessing steps applied earlier must additionally be reversed to create displayable RGB images. Amy Jang's tutorial provides this code.","metadata":{}},{"cell_type":"code","source":"_, ax = plt.subplots(5, 2, figsize=(12, 12))\nfor i, img in enumerate(photo_ds.take(5)):\n    prediction = monet_generator(img, training=False)[0].numpy()\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    img = (img[0] * 127.5 + 127.5).numpy().astype(np.uint8)\n\n    ax[i, 0].imshow(img)\n    ax[i, 1].imshow(prediction)\n    ax[i, 0].set_title(\"Input Photo\")\n    ax[i, 1].set_title(\"Monet-esque\")\n    ax[i, 0].axis(\"off\")\n    ax[i, 1].axis(\"off\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-04-23T04:17:38.169000Z","iopub.execute_input":"2024-04-23T04:17:38.169756Z","iopub.status.idle":"2024-04-23T04:17:41.550862Z","shell.execute_reply.started":"2024-04-23T04:17:38.169719Z","shell.execute_reply":"2024-04-23T04:17:41.549801Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submitting to Kaggle competition\nThe code to create the submission file in Kaggle's requested format is provided by Amy Jang's tutorial.","metadata":{}},{"cell_type":"code","source":"import PIL\n! mkdir ../images\n\ni = 1\nfor img in photo_ds:\n    prediction = monet_generator(img, training=False)[0].numpy()\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    im = PIL.Image.fromarray(prediction)\n    im.save(\"../images/\" + str(i) + \".jpg\")\n    i += 1\n    \nimport shutil\nshutil.make_archive(\"/kaggle/working/images\", 'zip', \"/kaggle/images\")","metadata":{"execution":{"iopub.status.busy":"2024-04-23T04:17:41.552147Z","iopub.execute_input":"2024-04-23T04:17:41.552462Z","iopub.status.idle":"2024-04-23T04:26:54.327616Z","shell.execute_reply.started":"2024-04-23T04:17:41.552424Z","shell.execute_reply":"2024-04-23T04:26:54.326523Z"},"trusted":true},"execution_count":null,"outputs":[]}]}