{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# How to Paint with Cycle Generative Adversarial Networks\n\n   A Cycle GAN is a type of generative adversarial network (GAN) that is used to learn the mapping between two different domains. The goal of a Cycle GAN is to be able to translate images from one domain to another, such as converting a photograph of a horse into a painting of a horse. To create a Cycle GAN model, you would first need to gather a dataset of images from both domains. Then, you would define and train two separate generative models, one for each domain. These models would be trained to translate images from one domain to the other, and back again. Finally, you would combine the two models into a single Cycle GAN model, which can be used to translate images between the two domains.\n\n![](https://godatadriven.com/wp-content/images/how-to-style-transfer/style-transfer-example.jpg)\n\n## Why Cycle GAN\n\n   Using a Cycle GAN can be a good idea for a number of reasons. First, a Cycle GAN can learn to translate images between two different domains without requiring any paired training data. This is useful because it can be difficult or impossible to collect such data for some tasks. Additionally, a Cycle GAN can learn to preserve the key characteristics of an image when translating it to a different domain, such as the overall shape and structure of the objects in the image. This can be useful for tasks such as image-to-image translation, where it is important to maintain the key features of an image. Finally, a Cycle GAN can be used for a variety of tasks, such as photo enhancement, style transfer, and domain adaptation.\n   \n### Delicious Cycle GAN\n![](https://junyanz.github.io/CycleGAN/images/faces_and_ramens.jpg)\n\n## Inspired by DCGAN, but Why?\n\nDCGAN, or Deep Convolutional Generative Adversarial Network, is a type of GAN that uses deep convolutional neural networks to generate images. Some potential benefits of using a DCGAN include:\n\n1. The ability to generate high-resolution images. DCGANs use deep convolutional networks, which are capable of processing large amounts of data and can generate high-resolution images.\n\n2. The ability to generate diverse images. DCGANs can learn to generate a wide range of images, including images that are realistic and those that are more abstract.\n\n3. The ability to control the features of generated images. By carefully controlling the training process, it is possible to use a DCGAN to generate images with specific characteristics, such as images that are centered around a particular object or images with a specific color scheme.\n\n4. The ability to use unsupervised learning. Unlike many other generative models, a DCGAN can be trained using unsupervised learning, which means it can learn to generate images without the need for labeled training data.\n\nOverall, DCGANs offer a powerful and flexible tool for generating images and can be used in a wide range of applications.\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-12-10T21:32:15.044654Z","iopub.execute_input":"2022-12-10T21:32:15.045131Z","iopub.status.idle":"2022-12-10T21:32:15.094022Z","shell.execute_reply.started":"2022-12-10T21:32:15.045014Z","shell.execute_reply":"2022-12-10T21:32:15.092864Z"}}},{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nfrom tensorflow.keras.utils import plot_model\nimport tensorflow_addons as tfa\nimport tensorflow_datasets as tfds\n\nfrom kaggle_datasets import KaggleDatasets\nimport matplotlib.pyplot as plt\nimport numpy as np\n\nfrom functools import partial\nfrom albumentations import (\n    Compose, RandomBrightness, JpegCompression, HueSaturationValue, RandomContrast, HorizontalFlip,\n    Rotate\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## TPU","metadata":{}},{"cell_type":"code","source":"try:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Device:', tpu.master())\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nexcept:\n    strategy = tf.distribute.get_strategy()\nprint('Number of replicas:', strategy.num_replicas_in_sync)\n\nAUTOTUNE = tf.data.experimental.AUTOTUNE\n\nprint(tf.__version__)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Load the Data","metadata":{}},{"cell_type":"code","source":"GCS_PATH = KaggleDatasets().get_gcs_path()\n\nMONET_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/monet_tfrec/*.tfrec'))\nprint('Monet TFRecord Files:', len(MONET_FILENAMES))\n\nPHOTO_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/photo_tfrec/*.tfrec'))\nprint('Photo TFRecord Files:', len(PHOTO_FILENAMES))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"IMAGE_SIZE = [256, 256]\n\ndef normalize(image):\n    return (tf.cast(image, tf.float32) / 127.5) - 1\n\ndef decode_image(image):\n    image = tf.image.decode_jpeg(image, channels=3)\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    return image\n\ndef random_crop(image):\n    cropped_image = tf.image.random_crop(image, size=[256, 256, 3])\n    return cropped_image\n\ndef random_jitter(image):\n    image = tf.image.resize(image, [int(256*1.3), int(256*1.3)],\n                          method=tf.image.ResizeMethod.NEAREST_NEIGHBOR)\n    image = random_crop(image)\n    return image\n\ndef flip(image):\n    return tf.image.flip_left_right(image)\n\ndef preprocess_image_train(image, label=None):\n    image = random_jitter(image)\n    return image\n\ndef read_tfrecord(example):\n    tfrecord_format = {\n        \"image_name\": tf.io.FixedLenFeature([], tf.string),\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"target\": tf.io.FixedLenFeature([], tf.string)\n    }\n    example = tf.io.parse_single_example(example, tfrecord_format)\n    image = decode_image(example['image'])\n    return image\n\ndef load_dataset(filenames, labeled=False, ordered=False, repeats=200):\n    dataset = tf.data.TFRecordDataset(filenames)\n    dataset = dataset.map(read_tfrecord, num_parallel_calls=AUTOTUNE)\n    dataset = dataset.concatenate(dataset.map(flip, num_parallel_calls=AUTOTUNE).shuffle(100000))\n    dataset = dataset.concatenate(dataset.map(random_jitter, num_parallel_calls=AUTOTUNE).shuffle(10000, reshuffle_each_iteration=True).repeat(repeats))\n    dataset = dataset.map(normalize, num_parallel_calls=AUTOTUNE).shuffle(10000)\n    return dataset","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"monet_ds = load_dataset(MONET_FILENAMES, labeled=True, repeats=50).batch(100, drop_remainder=True)\nphoto_ds = load_dataset(PHOTO_FILENAMES, labeled=True, repeats=2  ).batch(100, drop_remainder=True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# DCGAN\n## Network Upsample and Downsample","metadata":{}},{"cell_type":"markdown","source":"Here is the summary of DCGAN attribute we are testing:\n\n- Replace all max pooling with convolutional stride\n- Use transposed convolution for upsampling.\n- Eliminate fully connected layers.\n- Use Batch normalization except the output layer for the generator and the input layer of the discriminator.\n- Use ReLU in the generator except for the output which uses tanh.\n- Use LeakyReLU in the discriminator.","metadata":{}},{"cell_type":"code","source":"OUTPUT_CHANNELS = 3\nLATENT_DIM = 1024\n\ndef downsample(filters, size, activation, dropout, apply_instancenorm=True, \n                                                   apply_dropout=False):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    result = keras.Sequential()\n    result.add(layers.Conv2D(filters, size, strides=2, padding='same',\n                             kernel_initializer=initializer, use_bias=False))\n    #result.add(layers.MaxPool2D()) replaced\n\n    if apply_instancenorm:\n        result.add(tfa.layers.InstanceNormalization(gamma_initializer=gamma_init))\n    if apply_dropout:\n        result.add(layers.Dropout(dropout))\n    \n    if activation == \"leakyrelu\":\n        result.add(layers.LeakyReLU())\n    elif activation == \"relu\":\n        result.add(layers.ReLU())\n\n    return result","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def upsample(filters, size, activation, dropout, apply_dropout=False):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    result = keras.Sequential()\n    result.add(layers.Conv2DTranspose(filters, size, strides=2,\n                                      padding='same',\n                                      kernel_initializer=initializer,\n                                      use_bias=False))\n\n    result.add(tfa.layers.InstanceNormalization(gamma_initializer=gamma_init))\n\n    if apply_dropout:\n        result.add(layers.Dropout(dropout))\n\n    if activation == \"leakyrelu\":\n        result.add(layers.LeakyReLU())\n    elif activation == \"relu\":\n        result.add(layers.ReLU())\n\n    return result","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Build Network\nIn the Adam optimization algorithm, beta_1 and beta_2 are two hyperparameters that control the behavior of the algorithm. Beta_1 determines the decay rate for the first moment estimate, while beta_2 determines the decay rate for the second moment estimate. These parameters are used to compute an exponentially weighted average of the gradient and the squared gradient, which are used to update the weights in the network. In general, beta_1 and beta_2 are set to values close to 1, such as 0.9 and 0.999, respectively. This helps the Adam optimization algorithm to converge quickly and produce good results.","metadata":{}},{"cell_type":"code","source":"EPOCHS = 100\n\ngen_lrate = 0.0002\ndis_lrate = 0.0002\n\nbeta_1=0.9\nbeta_2=0.999\n\nlabel_real = .9","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Encoder-decode","metadata":{}},{"cell_type":"markdown","source":"Bottleneck, upsampling, and downsampling are all techniques that are commonly used in deep learning models, including generative models such as GANs.\n\nBottleneck refers to a layer in a neural network that has a smaller number of units or channels than the layers surrounding it. This can help reduce the number of parameters in the model and prevent overfitting. In generative models, bottleneck layers can be used to compress the input data before it is processed by the rest of the network.\n\nUpsampling and downsampling are techniques that are used to change the size or resolution of an image. Upsampling refers to increasing the size or resolution of an image, while downsampling refers to decreasing the size or resolution of an image. In generative models, upsampling and downsampling are often used in the generator network to manipulate the size of the input images and the size of the generated images.\n\nOverall, these techniques can be useful for improving the performance and efficiency of deep learning models, including generative models such as GANs.","metadata":{}},{"cell_type":"markdown","source":"## Generator","metadata":{}},{"cell_type":"markdown","source":"![1.png](https://i.ibb.co/KV4pTWd/1.png)","metadata":{}},{"cell_type":"code","source":"def CycleGenerator():\n    inputs = layers.Input(shape=[256,256,3])\n\n    down_stack = [\n        downsample(64, 4, \"relu\", 0.8, apply_instancenorm=False, apply_dropout=True),\n        downsample(128, 4, \"relu\", 0.5, apply_dropout=True),\n        downsample(256, 4, \"relu\", 0.5, apply_dropout=True),\n        downsample(512, 4, \"relu\", 0.5),\n        downsample(512, 4, \"relu\", 0.5),\n        downsample(512, 4, \"relu\", 0.5),\n        downsample(512, 4, \"relu\", 0.5),\n        downsample(512, 4, \"relu\", 0.5),\n    ]\n\n    up_stack = [\n        upsample(512, 4, \"relu\", 0.5, apply_dropout=True),\n        upsample(512, 4, \"relu\", 0.5, apply_dropout=True),\n        upsample(512, 4, \"relu\", 0.5, apply_dropout=True),\n        upsample(512, 4, \"relu\", 0.5, apply_dropout=True),\n        upsample(256, 4, \"relu\", 0.5, apply_dropout=False),\n        upsample(128, 4, \"relu\", 0.5, apply_dropout=False), \n        upsample(64, 4, \"relu\", 0.5, apply_dropout=False),\n    ]\n\n    initializer = tf.random_normal_initializer(0., 0.02)\n    last = layers.Conv2DTranspose(OUTPUT_CHANNELS, 4,\n                                  strides=2,\n                                  padding='same',\n                                  kernel_initializer=initializer,\n                                  activation='tanh')\n\n    x = inputs\n\n    skips = []\n    for down in down_stack:\n        x = down(x)\n        skips.append(x)\n\n    skips = reversed(skips[:-1])\n\n    for up, skip in zip(up_stack, skips):\n        x = up(x)\n        x = layers.Concatenate()([x, skip])\n\n    x = last(x)\n\n    return keras.Model(inputs=inputs, outputs=x)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def CycleGenerator_test():\n    inputs = layers.Input(shape=[256,256,3])\n\n    down_stack = [\n        downsample(64, 4, \"leakyrelu\", 0.8, apply_instancenorm=False, apply_dropout=False),\n        downsample(128, 4, \"leakyrelu\", 0.5, apply_dropout=False),\n        downsample(256, 4, \"leakyrelu\", 0.5, apply_dropout=False),\n        downsample(512, 4, \"leakyrelu\", 0.5, apply_dropout=False),\n        downsample(512, 4, \"leakyrelu\", 0.5, apply_dropout=False),\n        downsample(512, 4, \"leakyrelu\", 0.5, apply_dropout=False),\n        downsample(512, 4, \"leakyrelu\", 0.5, apply_dropout=False),\n        downsample(512, 4, \"leakyrelu\", 0.5, apply_dropout=False),\n    ]\n\n    up_stack = [\n        upsample(512, 4, \"relu\", 0.5, apply_dropout=True),\n        upsample(512, 4, \"relu\", 0.5, apply_dropout=True),\n        upsample(512, 4, \"relu\", 0.5, apply_dropout=True),\n        upsample(512, 4, \"relu\", 0.5, apply_dropout=False),\n        upsample(256, 4, \"relu\", 0.5, apply_dropout=False),\n        upsample(128, 4, \"relu\", 0.5, apply_dropout=False), \n        upsample(64, 4, \"relu\", 0.5, apply_dropout=False),\n    ]\n\n    initializer = tf.random_normal_initializer(0., 0.02)\n    last = layers.Conv2DTranspose(OUTPUT_CHANNELS, 4,\n                                  strides=2,\n                                  padding='same',\n                                  kernel_initializer=initializer,\n                                  activation='tanh')\n\n    x = inputs\n\n    skips = []\n    for down in down_stack:\n        x = down(x)\n        skips.append(x)\n\n    skips = reversed(skips[:-1])\n\n    for up, skip in zip(up_stack, skips):\n        x = up(x)\n        x = layers.Concatenate()([x, skip])\n\n    x = last(x)\n\n    return keras.Model(inputs=inputs, outputs=x)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Discriminator","metadata":{}},{"cell_type":"markdown","source":"![2.png](https://i.ibb.co/gjG9dpP/2.png)","metadata":{}},{"cell_type":"code","source":"def CycleDiscriminator():\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    inp = layers.Input(shape=[256, 256, 3], name='input_image')\n\n    x = inp\n\n    down1 = downsample(64, 4, \"leakyrelu\", 0.8, apply_instancenorm = False, apply_dropout=True)(x) # (bs, 128, 128, 64)\n    down2 = downsample(128, 4, \"leakyrelu\", 0.5)(down1)\n    down3 = downsample(256, 4, \"leakyrelu\", 0.5)(down2)\n\n    zero_pad1 = layers.ZeroPadding2D()(down3)\n    conv = layers.Conv2D(512, 4, strides=1,\n                         kernel_initializer=initializer,\n                         use_bias=False)(zero_pad1)\n\n    norm1 = tfa.layers.InstanceNormalization(gamma_initializer=gamma_init)(conv)\n\n    leaky_relu = layers.LeakyReLU()(norm1)\n\n    zero_pad2 = layers.ZeroPadding2D()(leaky_relu)\n\n    last_conv = layers.Conv2D(1, 4, strides=1,\n                         kernel_initializer=initializer)(zero_pad2)\n\n    last_relu = layers.LeakyReLU(alpha=0.2)(last_conv)\n    last_pool = layers.Flatten()(last_relu)\n    last = layers.Dense(1, activation='sigmoid')(last_pool)\n\n    return tf.keras.Model(inputs=inp, outputs=last)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def CycleDiscriminator_test():\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    inp = layers.Input(shape=[256, 256, 3], name='input_image')\n\n    x = inp\n\n    down1 = downsample(64, 4, \"leakyrelu\", 0.8, apply_instancenorm = False, apply_dropout=False)(x) # (bs, 128, 128, 64)\n    down2 = downsample(128, 4, \"leakyrelu\", 0.5, apply_dropout=False)(down1)\n    down3 = downsample(256, 4, \"leakyrelu\", 0.5, apply_dropout=False)(down2)\n\n    zero_pad1 = layers.ZeroPadding2D()(down3)\n    conv = layers.Conv2D(512, 4, strides=1,\n                         kernel_initializer=initializer,\n                         use_bias=False)(zero_pad1)\n\n    norm1 = tfa.layers.InstanceNormalization(gamma_initializer=gamma_init)(conv)\n\n    leaky_relu = layers.LeakyReLU()(norm1)\n\n    zero_pad2 = layers.ZeroPadding2D()(leaky_relu)\n\n    last_conv = layers.Conv2D(1, 4, strides=1,\n                         kernel_initializer=initializer)(zero_pad2)\n\n    last_relu = layers.LeakyReLU(alpha=0.2)(last_conv)\n    last_pool = layers.Flatten()(last_relu)\n    last = layers.Dense(1, activation='sigmoid')(last_pool)\n\n    return tf.keras.Model(inputs=inp, outputs=last)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n    monet_cycle_generator = CycleGenerator_test() # Photos to Monet \n    photo_cycle_generator = CycleGenerator_test() #  Monet to Photos\n\n    monet_cycle_discriminator = CycleDiscriminator_test() \n    photo_cycle_discriminator = CycleDiscriminator_test() ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_model(monet_cycle_generator, show_shapes = True, show_layer_names = True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_model(monet_cycle_discriminator, show_shapes = True, show_layer_names = True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## CycleGAN model\n","metadata":{}},{"cell_type":"code","source":"class CycleGan(keras.Model):\n    def __init__(\n        self,\n        monet_generator,\n        photo_generator,\n        monet_discriminator,\n        photo_discriminator,\n        lambda_cycle=10,\n        label_real=.5\n    ):\n        super(CycleGan, self).__init__()\n        self.m_gen = monet_generator\n        self.p_gen = photo_generator\n        self.m_disc = monet_discriminator\n        self.p_disc = photo_discriminator\n        self.lambda_cycle = lambda_cycle\n        self.label_real = label_real\n        \n    def compile(\n        self,\n        m_gen_optimizer,\n        p_gen_optimizer,\n        m_disc_optimizer,\n        p_disc_optimizer,\n        gen_loss_fn,\n        disc_loss_fn,\n        cycle_loss_fn,\n        identity_loss_fn\n    ):\n        super(CycleGan, self).compile()\n        self.m_gen_optimizer = m_gen_optimizer\n        self.p_gen_optimizer = p_gen_optimizer\n        self.m_disc_optimizer = m_disc_optimizer\n        self.p_disc_optimizer = p_disc_optimizer\n        self.gen_loss_fn = gen_loss_fn\n        self.disc_loss_fn = disc_loss_fn\n        self.cycle_loss_fn = cycle_loss_fn\n        self.identity_loss_fn = identity_loss_fn\n        \n    def train_step(self, batch_data):\n        real_monet, real_photo = batch_data\n        \n        batch_size = tf.shape(real_photo)[0]\n        labels_real = tf.zeros((batch_size, 1)) + self.label_real\n        labels_real += 0.05 * tf.random.uniform(tf.shape(labels_real))        \n        \n        with tf.GradientTape(persistent=True) as tape:\n            # photo to monet back to photo\n            fake_monet = self.m_gen(real_photo, training=True)\n            cycled_photo = self.p_gen(fake_monet, training=True)\n\n            # monet to photo back to monet\n            fake_photo = self.p_gen(real_monet, training=True)\n            cycled_monet = self.m_gen(fake_photo, training=True)\n\n            # generating itself\n            same_monet = self.m_gen(real_monet, training=True)\n            same_photo = self.p_gen(real_photo, training=True)\n\n            # discriminator used to check, inputing real images\n            disc_real_monet = self.m_disc(real_monet, training=True)\n            disc_real_photo = self.p_disc(real_photo, training=True)\n\n            # discriminator used to check, inputing fake images\n            disc_fake_monet = self.m_disc(fake_monet, training=True)\n            disc_fake_photo = self.p_disc(fake_photo, training=True)\n\n            # evaluates generator loss\n            monet_gen_loss = self.gen_loss_fn(disc_real_monet, disc_fake_monet, labels_real)\n            photo_gen_loss = self.gen_loss_fn(disc_real_photo, disc_fake_photo, labels_real)\n\n            # evaluates total cycle consistency loss\n            total_cycle_loss = self.cycle_loss_fn(real_monet, cycled_monet, self.lambda_cycle) + self.cycle_loss_fn(real_photo, cycled_photo, self.lambda_cycle)\n\n            # evaluates total generator loss\n            total_monet_gen_loss = monet_gen_loss + total_cycle_loss + self.identity_loss_fn(real_monet, same_monet, self.lambda_cycle)\n            total_photo_gen_loss = photo_gen_loss + total_cycle_loss + self.identity_loss_fn(real_photo, same_photo, self.lambda_cycle)\n\n            # evaluates discriminator loss\n            monet_disc_loss = self.disc_loss_fn(disc_real_monet, disc_fake_monet, labels_real)\n            photo_disc_loss = self.disc_loss_fn(disc_real_photo, disc_fake_photo, labels_real)\n\n        # Calculate the gradients for generator and discriminator\n        monet_generator_gradients = tape.gradient(total_monet_gen_loss,\n                                                  self.m_gen.trainable_variables)\n        photo_generator_gradients = tape.gradient(total_photo_gen_loss,\n                                                  self.p_gen.trainable_variables)\n\n        monet_discriminator_gradients = tape.gradient(monet_disc_loss,\n                                                      self.m_disc.trainable_variables)\n        photo_discriminator_gradients = tape.gradient(photo_disc_loss,\n                                                      self.p_disc.trainable_variables)\n\n        # Apply the gradients to the optimizer\n        self.m_gen_optimizer.apply_gradients(zip(monet_generator_gradients,\n                                                 self.m_gen.trainable_variables))\n\n        self.p_gen_optimizer.apply_gradients(zip(photo_generator_gradients,\n                                                 self.p_gen.trainable_variables))\n\n        self.m_disc_optimizer.apply_gradients(zip(monet_discriminator_gradients,\n                                                  self.m_disc.trainable_variables))\n\n        self.p_disc_optimizer.apply_gradients(zip(photo_discriminator_gradients,\n                                                  self.p_disc.trainable_variables))\n        \n        return {\n            \"monet_gen_loss\": total_monet_gen_loss,\n            \"photo_gen_loss\": total_photo_gen_loss,\n            \"monet_disc_loss\": monet_disc_loss,\n            \"photo_disc_loss\": photo_disc_loss\n        }","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Loss functions\n","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    def discriminator_loss(predictions_real, predictions_gen, labels_real):\n        return (tf.reduce_mean((predictions_gen  - tf.reduce_mean(predictions_real) + labels_real) ** 2) +\n                tf.reduce_mean((predictions_real - tf.reduce_mean(predictions_gen)  - labels_real) ** 2))/2\n    \n    def generator_loss(predictions_real, predictions_gen, labels_real):\n        return (tf.reduce_mean((predictions_real - tf.reduce_mean(predictions_gen)  + labels_real) ** 2) +\n                tf.reduce_mean((predictions_gen  - tf.reduce_mean(predictions_real) - labels_real) ** 2)) / 2","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n    def calc_cycle_loss(real_image, cycled_image, LAMBDA):\n        loss1 = tf.reduce_mean(tf.abs(real_image - cycled_image))\n\n        return LAMBDA * loss1","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n    def identity_loss(real_image, same_image, LAMBDA):\n        loss = tf.reduce_mean(tf.abs(real_image - same_image))\n        return LAMBDA * 0.5 * loss","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Train","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    monet_generator_optimizer = tf.keras.optimizers.Adam(gen_lrate, beta_1=0.9, beta_2=0.999)\n    photo_generator_optimizer = tf.keras.optimizers.Adam(gen_lrate,  beta_1=0.9, beta_2=0.999)\n\n    monet_discriminator_optimizer = tf.keras.optimizers.Adam(dis_lrate,  beta_1=0.9, beta_2=0.999)\n    photo_discriminator_optimizer = tf.keras.optimizers.Adam(dis_lrate,  beta_1=0.9, beta_2=0.999)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n    cycle_gan_model = CycleGan(\n        monet_cycle_generator, photo_cycle_generator, monet_cycle_discriminator, photo_cycle_discriminator, label_real=0.66\n    )\n\n    cycle_gan_model.compile(\n        m_gen_optimizer = monet_generator_optimizer,\n        p_gen_optimizer = photo_generator_optimizer,\n        m_disc_optimizer = monet_discriminator_optimizer,\n        p_disc_optimizer = photo_discriminator_optimizer,\n        gen_loss_fn = generator_loss,\n        disc_loss_fn = discriminator_loss,\n        cycle_loss_fn = calc_cycle_loss,\n        identity_loss_fn = identity_loss\n    )","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = cycle_gan_model.fit(\n    tf.data.Dataset.zip((monet_ds, photo_ds)),\n    epochs=30\n)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Plot Loss for Gen","metadata":{}},{"cell_type":"code","source":"loss_G_M = history.history['monet_gen_loss']","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"loss_G_M","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(loss_G_M, label=\"Generator Monet\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Visualize","metadata":{}},{"cell_type":"code","source":"_, ax = plt.subplots(2, 3, figsize=(26, 9))\nfor i, img in enumerate(photo_ds.take(3)):\n    prediction = monet_cycle_generator(img, training=False)[0].numpy()\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    img = (img[0] * 127.5 + 127.5).numpy().astype(np.uint8)\n\n    ax[0, i].imshow(img)\n    ax[1, i].imshow(prediction)\n    ax[0, i].set_title(\"Photo\")\n    ax[1, i].set_title(\"Monet\")\n    ax[0, i].axis(\"off\")\n    ax[1, i].axis(\"off\")\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Save Model","metadata":{}},{"cell_type":"code","source":"#monet_cycle_discriminator.save(\"../models/monet_cycle_discriminator_100.h5\")\n#photo_cycle_discriminator.save(\"../models/photo_cycle_discriminator_100.h5\")\n#photo_cycle_generator.save(\"../models/photo_cycle_generator_100.h5\")\n#monet_cycle_generator.save(\"../models/monet_cycle_generator_100.h5\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#import shutil\n#shutil.make_archive(\"/kaggle/working/models\", 'zip', \"/kaggle/models\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission","metadata":{}},{"cell_type":"code","source":"import PIL\n! mkdir ../images","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i, img in enumerate(photo_ds.take(9999)):\n    prediction = monet_cycle_generator(img, training=False)[0].numpy()\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    im = PIL.Image.fromarray(prediction)\n    im.save(\"../images/\" + str(i) + \".jpg\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import shutil\nshutil.make_archive(\"/kaggle/working/images\", 'zip', \"/kaggle/images\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}