{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**In this notebook we will implement CycleGan from scratch.**\n\nIn the first part I am gonna do a quick overview for the paper, to answer the questions bellow:\n* What CycleGan is?\n* How the trainig looks like?\n* How the architecture looks like?\n\nBut the main part of this notebook is gonna be for the implementation using tensorflow and keras.","metadata":{}},{"cell_type":"markdown","source":"# Table of Contents:\n\n* [Overview for the paper](#c1)\n* [Load all dependencies we need](#c2)\n* [Set up TPU](#c3)\n* [Hyperparameter](#c4)\n* [Data Processing](#c5)\n* [Define models and losses](#c6)\n* [Training](#c7)\n* [Results](#c8)\n\n","metadata":{}},{"cell_type":"markdown","source":"<a id=\"c1\"></a>\n\n# Overview for the paper\n**CycleGan** can transform inpaired images to images across domains where the dataset doesn't contains the images in one domain and their corresponding images in another domain (unsupervised learning). for example you can take an image of a horse and convert it to zebra, or take an image of summer and convert it into winter or any season you want.\n\n\nIn this notebbok we will convert an image into the style of Monet image\n\n**How do we train this ?**\n\nTo train CycleGan we're gonna have two generators **G** and **F**, and we're going to have two discriminator **Dx** and **Dy**.\n\n\nTo make things a little bit simpler let's say that **X** is an image with no style, and **Y** is a style of Monet image.\n\n**The generator G     :** Take an image without any style and try to convert it into style of Monet image.\n\n**The generator F     :** Take a style of Monet image and try to convert it into an image with no style.\n\n**The discriminator Dy:** Try to say if an image of style of Monet image is 'Real' or 'Fake' \n\n**The discriminator Dx:** Try to say if an normal image with no style is 'Real' or 'Fake' \n\nThis is sort of a standard GAN setup except we have double the ammount of discriminators and genertos, normally we have one generator and one discriminator.\n\n\nThe authors add an additional loss term, they called a cycle consistency. If we take an image X, and try to generate Y^ the style of Monet for X using the generator G, and then if we take Y^ and try to generate X^ an image with no Monet style using the generator F. We should ideally have X=X^, and we enforced this with an L1-loss.\n\n\n\nThey also add a sort of idetity mapping loss. If we take an image X with no Monet style, and send it to the generator F that is supposed to generate images with no Monet style, then it shouldn't do anything and output the same image X, and again we enforced this with an L1-loss. The reason why the authors add this loss is because they don't want the generators to change the coloring or the tint of the image too much.\n\nFor the final loss for each generator we have a standard GAN loss(The adversial loss), but instead of BinaryCrossentropy they used least squares loss, then we have cycle consistency loss, and finally we have the idetity loss. \n\n> the idetity loss is not acctualy used in all dataset.\n\nFor the discriminator loss it pretty standard, it will be identify if an image 'Real' or 'Fake'. \n\n**How the architecture looks like?**\n\nThe generator has three sections: \n* Encoder that take as input the image and hase three convolution layers all with a stride of two.\n* Transformer wich is a series of six residual blocks\n* Decoder which uses two transpose convolutions to enlarge the representation size, and one output layer to produce the final image in RGB\n\n\n\nThe discriminator it can be pretty simple, it has four convolution layers all with a stride of two, and the output is a patchGAN, because we're not going to output a single scaler between 0 and 1, but what we are going to output is a grid of values, each of those values are going to be between 0 and 1. It called patchGan because each of those values corresponds to seeing a patch in the original image.\n\n\n","metadata":{}},{"cell_type":"markdown","source":"Now let's focus to build cycleGAN from scratch to convert an image into the style of Monet image.","metadata":{}},{"cell_type":"markdown","source":"<a id=\"c2\"></a>\n# Load all dependencies we need\n","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nimport tensorflow_addons as tfa\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport re\nimport os\nfrom PIL import Image\nfrom kaggle_datasets import KaggleDatasets","metadata":{"execution":{"iopub.status.busy":"2022-06-23T18:41:05.827082Z","iopub.execute_input":"2022-06-23T18:41:05.827605Z","iopub.status.idle":"2022-06-23T18:41:12.569888Z","shell.execute_reply.started":"2022-06-23T18:41:05.827512Z","shell.execute_reply":"2022-06-23T18:41:12.568888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"c3\"></a>\n# Set up TPU","metadata":{}},{"cell_type":"code","source":"try:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Device:', tpu.master())\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nexcept:\n    strategy = tf.distribute.get_strategy()\nprint('Number of replicas:', strategy.num_replicas_in_sync)\n\nAUTOTUNE = tf.data.experimental.AUTOTUNE\n    \nprint(tf.__version__)","metadata":{"execution":{"iopub.status.busy":"2022-06-23T18:41:12.571913Z","iopub.execute_input":"2022-06-23T18:41:12.57268Z","iopub.status.idle":"2022-06-23T18:41:18.288989Z","shell.execute_reply.started":"2022-06-23T18:41:12.572625Z","shell.execute_reply":"2022-06-23T18:41:18.288319Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"c4\"></a>\n# Hyperparameter ","metadata":{}},{"cell_type":"code","source":"datafolder = KaggleDatasets().get_gcs_path()\n\nMONET_FILENAMES = tf.io.gfile.glob(str(datafolder + '/monet_tfrec/*.tfrec'))\nPHOTO_FILENAMES = tf.io.gfile.glob(str(datafolder + '/photo_tfrec/*.tfrec'))\nIMAGE_SIZE = [256, 256]\nBATCH_SIZE = 1\nOUTPUT_CHANNELS = 3\n","metadata":{"execution":{"iopub.status.busy":"2022-06-23T18:41:18.292642Z","iopub.execute_input":"2022-06-23T18:41:18.292894Z","iopub.status.idle":"2022-06-23T18:41:18.893877Z","shell.execute_reply.started":"2022-06-23T18:41:18.292864Z","shell.execute_reply":"2022-06-23T18:41:18.892948Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"c5\"></a>\n# Data Processing ","metadata":{}},{"cell_type":"markdown","source":"Let's first see how the images looks like.","metadata":{}},{"cell_type":"code","source":"_, ax = plt.subplots(3,3, figsize=(8,8))\nplt.suptitle('Some images with monet style', fontsize=19, fontweight='bold')\n\nind = 0 \nfor k in range(3):\n    for kk in range(3):\n        ax[k][kk].imshow(Image.open('../input/gan-getting-started/monet_jpg/'+os.listdir('../input/gan-getting-started/monet_jpg')[ind]))\n        ind += 1","metadata":{"execution":{"iopub.status.busy":"2022-06-23T18:41:18.897977Z","iopub.execute_input":"2022-06-23T18:41:18.898277Z","iopub.status.idle":"2022-06-23T18:41:20.547544Z","shell.execute_reply.started":"2022-06-23T18:41:18.898237Z","shell.execute_reply":"2022-06-23T18:41:20.546466Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"_, ax = plt.subplots(3,3, figsize=(8,8))\nplt.suptitle('Some images with no monet style', fontsize=19, fontweight='bold')\n\nind = 0 \nfor k in range(3):\n    for kk in range(3):\n        ax[k][kk].imshow(Image.open('../input/gan-getting-started/photo_jpg/'+os.listdir('../input/gan-getting-started/photo_jpg')[ind]))\n        ind += 1","metadata":{"execution":{"iopub.status.busy":"2022-06-23T18:41:20.549011Z","iopub.execute_input":"2022-06-23T18:41:20.549485Z","iopub.status.idle":"2022-06-23T18:41:22.134657Z","shell.execute_reply.started":"2022-06-23T18:41:20.549446Z","shell.execute_reply":"2022-06-23T18:41:22.133978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Data aumgmentation:** it a technique to increase the diversity of the training set by applying random (but realistic) transformations, such as image rotation, and it can be done very easily using the API **tf.image**. To learn more about it check out the official decantation: https://www.tensorflow.org/tutorials/images/data_augmentation.","metadata":{}},{"cell_type":"code","source":"def data_augment(image):\n    image = tf.image.random_flip_left_right(image)\n    image = tf.image.random_hue(image, 0.01)\n    image = tf.image.random_saturation(image, 0.70, 1.30)\n    image = tf.image.random_contrast(image, 0.80, 1.20)\n    image = tf.image.random_brightness(image, 0.10)\n    return image","metadata":{"execution":{"iopub.status.busy":"2022-06-23T18:41:22.13585Z","iopub.execute_input":"2022-06-23T18:41:22.136205Z","iopub.status.idle":"2022-06-23T18:41:22.141497Z","shell.execute_reply.started":"2022-06-23T18:41:22.136177Z","shell.execute_reply":"2022-06-23T18:41:22.140738Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's check how the images look like after applying data augmentation","metadata":{}},{"cell_type":"code","source":"_, ax = plt.subplots(3,3, figsize=(8,8))\nplt.suptitle('Some augmented images with monet style', fontsize=19, fontweight='bold')\n\nind = 0 \nfor k in range(3):\n    for kk in range(3):\n        ax[k][kk].imshow(data_augment(np.array(Image.open('../input/gan-getting-started/monet_jpg/'+os.listdir('../input/gan-getting-started/monet_jpg')[ind]))))\n        ind += 1","metadata":{"execution":{"iopub.status.busy":"2022-06-23T18:41:22.143268Z","iopub.execute_input":"2022-06-23T18:41:22.14353Z","iopub.status.idle":"2022-06-23T18:41:23.6148Z","shell.execute_reply.started":"2022-06-23T18:41:22.143501Z","shell.execute_reply":"2022-06-23T18:41:23.614153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"_, ax = plt.subplots(3,3, figsize=(8,8))\nplt.suptitle('Some augmented images with no monet style', fontsize=19, fontweight='bold')\n\nind = 0 \nfor k in range(3):\n    for kk in range(3):\n        ax[k][kk].imshow(data_augment(np.array(Image.open('../input/gan-getting-started/photo_jpg/'+os.listdir('../input/gan-getting-started/photo_jpg')[ind]))))\n        ind += 1","metadata":{"execution":{"iopub.status.busy":"2022-06-23T18:41:23.615954Z","iopub.execute_input":"2022-06-23T18:41:23.616318Z","iopub.status.idle":"2022-06-23T18:41:25.069254Z","shell.execute_reply.started":"2022-06-23T18:41:23.616269Z","shell.execute_reply":"2022-06-23T18:41:25.06817Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"You can notice that some images don't change, that because the augmentation function augments the dataset randomly. You can play with the function data_augment to make different augmented data, but try to keep it realistic.","metadata":{}},{"cell_type":"markdown","source":"**Transform dataset into TFRecord format:** is a simple format for storing a sequence of binary records to make processing dataset more easily and efficiently, especialy when the dataset is large and complex.\n\nIn this dataset, the images already stored in TFRecord. But you can create a tfrecord-format for any dataset you want, for more information check out the official decantation: https://www.tensorflow.org/tutorials/load_data/tfrecord.","metadata":{}},{"cell_type":"markdown","source":"Now let's define a function **decode_image** that decode a JPEG-encoded image to a uint8 tensor, casts it to a float32, divide it by 127.5 and subtract it by 1, to make the values in the tensor between -1 and 1, and finally, reshape it to (IMAGE_SIZE,IMAGE_SIZE, NUM_CHANNEL) .","metadata":{}},{"cell_type":"code","source":"def decode_image(image):\n    image = tf.image.decode_jpeg(image, channels=3)\n    image = (tf.cast(image, tf.float32) / 127.5) - 1\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    return image","metadata":{"execution":{"iopub.status.busy":"2022-06-23T18:41:25.07072Z","iopub.execute_input":"2022-06-23T18:41:25.071495Z","iopub.status.idle":"2022-06-23T18:41:25.077956Z","shell.execute_reply.started":"2022-06-23T18:41:25.071448Z","shell.execute_reply":"2022-06-23T18:41:25.077134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's define a function **read_tfrecord** to Parses a single Example proto.","metadata":{}},{"cell_type":"code","source":"def read_tfrecord(example):\n    tfrecord_format = {\n        \"image\": tf.io.FixedLenFeature([], tf.string)\n    }\n    example = tf.io.parse_single_example(example, tfrecord_format)\n    image = decode_image(example['image'])\n    return image","metadata":{"execution":{"iopub.status.busy":"2022-06-23T18:41:25.080853Z","iopub.execute_input":"2022-06-23T18:41:25.081127Z","iopub.status.idle":"2022-06-23T18:41:25.09051Z","shell.execute_reply.started":"2022-06-23T18:41:25.081098Z","shell.execute_reply":"2022-06-23T18:41:25.089584Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's define a function **load_dataset** to load our dataset.","metadata":{}},{"cell_type":"code","source":"def load_dataset(filenames):\n    dataset = tf.data.TFRecordDataset(filenames)\n    dataset = dataset.map(read_tfrecord, num_parallel_calls=AUTOTUNE)\n    return dataset","metadata":{"execution":{"iopub.status.busy":"2022-06-23T18:41:25.092392Z","iopub.execute_input":"2022-06-23T18:41:25.093077Z","iopub.status.idle":"2022-06-23T18:41:25.102536Z","shell.execute_reply.started":"2022-06-23T18:41:25.093019Z","shell.execute_reply":"2022-06-23T18:41:25.101839Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"monet_ds = load_dataset(MONET_FILENAMES).batch(1)\nphoto_ds = load_dataset(PHOTO_FILENAMES).batch(1)","metadata":{"execution":{"iopub.status.busy":"2022-06-23T18:41:25.103987Z","iopub.execute_input":"2022-06-23T18:41:25.104344Z","iopub.status.idle":"2022-06-23T18:41:25.270551Z","shell.execute_reply.started":"2022-06-23T18:41:25.104291Z","shell.execute_reply":"2022-06-23T18:41:25.269458Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's define a function **get_gan_dataset** that load datasets from monet_fils, and photo_files, then augment the data with the function that we defined earlier  **data_augment**, then let's combine consecutive elements of this dataset into batches, then let's use prefetch, to allows later elements to be prepared while the current element is being processed. To improve latency and throughput, at the cost of using additional memory to store prefetched elements. and let's create our final Dataset by zipping together the given datasets (monet_ds and photo_ds).\n\nCheck out the official documentation to learn more about how to work with tf.data.Dataset :https://www.tensorflow.org/api_docs/python/tf/data/Dataset#batch","metadata":{}},{"cell_type":"code","source":"def get_gan_dataset(monet_files, photo_files, batch_size=BATCH_SIZE):\n\n    monet_ds = load_dataset(monet_files)\n    photo_ds = load_dataset(photo_files)\n    \n    monet_ds = monet_ds.map(data_augment, num_parallel_calls=AUTOTUNE)\n    photo_ds = photo_ds.map(data_augment, num_parallel_calls=AUTOTUNE)\n        \n    monet_ds = monet_ds.batch(batch_size)\n    photo_ds = photo_ds.batch(batch_size)\n    \n    monet_ds = monet_ds.prefetch(AUTOTUNE)\n    photo_ds = photo_ds.prefetch(AUTOTUNE)\n    \n    gan_ds = tf.data.Dataset.zip((monet_ds, photo_ds))\n    \n    return gan_ds\n\nfinal_dataset = get_gan_dataset(MONET_FILENAMES, PHOTO_FILENAMES, batch_size=BATCH_SIZE)","metadata":{"execution":{"iopub.status.busy":"2022-06-23T18:41:25.271937Z","iopub.execute_input":"2022-06-23T18:41:25.272202Z","iopub.status.idle":"2022-06-23T18:41:25.464493Z","shell.execute_reply.started":"2022-06-23T18:41:25.272172Z","shell.execute_reply":"2022-06-23T18:41:25.463593Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"c6\"></a>\n\n# Define models and losses","metadata":{}},{"cell_type":"markdown","source":"To create our neural netwoeks easily, we will use powerful API tensorflow.keras.\n\nCheck out the officiel documentation foe more information: https://www.tensorflow.org/tutorials/customization/custom_layers\n\nLet's start by the the generator.\n\nTo make a clear code let's define a layer for down_sample, and onother layer for up_sample.","metadata":{}},{"cell_type":"code","source":"def down_sample(filters, size, apply_instancenorm=True):\n    # In the paper the weights are initialized from a Gaussian distribution N (0, 0.02).\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    layer = keras.Sequential()\n    layer.add(layers.Conv2D(filters, size, strides=2, padding='same', kernel_initializer=initializer, use_bias=False))\n\n    if apply_instancenorm:\n        layer.add(tfa.layers.InstanceNormalization(gamma_initializer=gamma_init))\n\n    layer.add(layers.LeakyReLU())\n\n    return layer","metadata":{"execution":{"iopub.status.busy":"2022-06-23T18:41:25.46562Z","iopub.execute_input":"2022-06-23T18:41:25.465875Z","iopub.status.idle":"2022-06-23T18:41:25.473349Z","shell.execute_reply.started":"2022-06-23T18:41:25.465849Z","shell.execute_reply":"2022-06-23T18:41:25.472425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def up_sample(filters, size, apply_dropout=False):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    layer = keras.Sequential()\n    layer.add(layers.Conv2DTranspose(filters, size, strides=2, padding='same', kernel_initializer=initializer,use_bias=False))\n    layer.add(tfa.layers.InstanceNormalization(gamma_initializer=gamma_init))\n\n    if apply_dropout:\n        layer.add(layers.Dropout(0.5))\n\n    layer.add(layers.ReLU())\n\n    return layer","metadata":{"execution":{"iopub.status.busy":"2022-06-23T18:41:25.475175Z","iopub.execute_input":"2022-06-23T18:41:25.475419Z","iopub.status.idle":"2022-06-23T18:41:25.488144Z","shell.execute_reply.started":"2022-06-23T18:41:25.475391Z","shell.execute_reply":"2022-06-23T18:41:25.486814Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def Generator():\n    inputs = layers.Input(shape=[256,256,3])\n    down_stack = [\n        down_sample(64, 4, apply_instancenorm=False),\n        down_sample(128, 4),                        \n        down_sample(256, 4),                        \n        down_sample(512, 4),                        \n        down_sample(512, 4),                      \n        down_sample(512, 4),                      \n        down_sample(512, 4),                      \n        down_sample(512, 4),                      \n    ]\n\n    up_stack = [\n        up_sample(512, 4, apply_dropout=True),    \n        up_sample(512, 4, apply_dropout=True),    \n        up_sample(512, 4, apply_dropout=True),    \n        up_sample(512, 4),                          \n        up_sample(256, 4),                         \n        up_sample(128, 4),                           \n        up_sample(64, 4),                           \n    ]\n\n    initializer = tf.random_normal_initializer(0., 0.02)\n    # The last activaltion function is tanh because we want to force the model to generate pixels between 1 and -1, to be the same as the input pixels after preprocessing.\n    last = layers.Conv2DTranspose(3, 4, strides=2, padding='same', kernel_initializer=initializer, activation='tanh') \n   \n\n    x = inputs\n\n    # Downsampling through the model\n    skips = []\n    for down in down_stack:\n        x = down(x)\n        skips.append(x)\n\n    skips = reversed(skips[:-1])\n\n    # Upsampling and establishing the skip connections\n    for up, skip in zip(up_stack, skips):\n        x = up(x)\n        x = layers.Concatenate()([x, skip])\n\n    x = last(x)\n\n    return keras.Model(inputs=inputs, outputs=x)","metadata":{"execution":{"iopub.status.busy":"2022-06-23T18:41:25.490121Z","iopub.execute_input":"2022-06-23T18:41:25.490408Z","iopub.status.idle":"2022-06-23T18:41:25.503957Z","shell.execute_reply.started":"2022-06-23T18:41:25.490367Z","shell.execute_reply":"2022-06-23T18:41:25.503217Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's build the discriminator","metadata":{}},{"cell_type":"code","source":"def Discriminator():\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n    \n    inp = layers.Input(shape=[256, 256, 3], name='input_image')\n    x = inp\n    \n    down1 = down_sample(64, 4, False)(x)       \n    down2 = down_sample(128, 4)(down1)        \n    down3 = down_sample(256, 4)(down2)        \n\n    zero_pad1 = layers.ZeroPadding2D()(down3)\n    conv = layers.Conv2D(512, 4, strides=1, kernel_initializer=initializer, use_bias=False)(zero_pad1)\n\n    norm1 = tfa.layers.InstanceNormalization(gamma_initializer=gamma_init)(conv)\n    leaky_relu = layers.LeakyReLU()(norm1)\n    zero_pad2 = layers.ZeroPadding2D()(leaky_relu)\n    last = layers.Conv2D(1, 4, strides=1, kernel_initializer=initializer)(zero_pad2)\n\n    return tf.keras.Model(inputs=inp, outputs=last)","metadata":{"execution":{"iopub.status.busy":"2022-06-23T18:41:25.505239Z","iopub.execute_input":"2022-06-23T18:41:25.505953Z","iopub.status.idle":"2022-06-23T18:41:25.522744Z","shell.execute_reply.started":"2022-06-23T18:41:25.505913Z","shell.execute_reply":"2022-06-23T18:41:25.521534Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's create the models in strategy.scope() for TPU","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    monet_generator = Generator() # transforms photos to Monet style \n    photo_generator = Generator() # transforms Monet style to be more like photos\n\n    monet_discriminator = Discriminator() # differentiates real images with Monet style andi mages with generated Monet style\n    photo_discriminator = Discriminator() # differentiates real photos and generated photos","metadata":{"execution":{"iopub.status.busy":"2022-06-23T18:41:25.524105Z","iopub.execute_input":"2022-06-23T18:41:25.52436Z","iopub.status.idle":"2022-06-23T18:41:35.544979Z","shell.execute_reply.started":"2022-06-23T18:41:25.524332Z","shell.execute_reply":"2022-06-23T18:41:35.544192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's build CycleGan class","metadata":{}},{"cell_type":"code","source":"class CycleGan(keras.Model):\n    def __init__(\n        self,\n        monet_generator,\n        photo_generator,\n        monet_discriminator,\n        photo_discriminator, \n        lambda_cycle=10, \n    ):\n        super(CycleGan, self).__init__()\n        self.m_gen = monet_generator\n        self.p_gen = photo_generator\n        self.m_disc = monet_discriminator\n        self.p_disc = photo_discriminator\n        self.lambda_cycle = lambda_cycle\n        \n    def compile(\n        self,\n        m_gen_optimizer,\n        p_gen_optimizer,\n        m_disc_optimizer,\n        p_disc_optimizer,\n        gen_loss_fn,\n        disc_loss_fn,\n        cycle_loss_fn,\n        identity_loss_fn,\n    ):\n        super(CycleGan, self).compile()\n        self.m_gen_optimizer = m_gen_optimizer\n        self.p_gen_optimizer = p_gen_optimizer\n        self.m_disc_optimizer = m_disc_optimizer\n        self.p_disc_optimizer = p_disc_optimizer\n        self.gen_loss_fn = gen_loss_fn\n        self.disc_loss_fn = disc_loss_fn\n        self.cycle_loss_fn = cycle_loss_fn\n        self.identity_loss_fn = identity_loss_fn\n\n        \n    def train_step(self, batch_data):\n        real_monet, real_photo = batch_data\n        batch_size = tf.shape(real_monet)[0]\n        with tf.GradientTape(persistent=True) as tape:\n        \n            # photo to monet back to photo\n            fake_monet = self.m_gen(real_photo, training=True)\n            cycled_photo = self.p_gen(fake_monet, training=True)\n\n            # monet to photo back to monet\n            fake_photo = self.p_gen(real_monet, training=True)\n            cycled_monet = self.m_gen(fake_photo, training=True)\n\n            # generating itself\n            same_monet = self.m_gen(real_monet, training=True)\n            same_photo = self.p_gen(real_photo, training=True)\n\n            \n            \n            # discriminator used to check, inputing real images\n            disc_real_monet = self.m_disc(real_monet, training=True)\n            disc_real_photo = self.p_disc(real_photo, training=True)\n\n            # discriminator used to check, inputing fake images\n            disc_fake_monet = self.m_disc(fake_monet, training=True)\n            disc_fake_photo = self.p_disc(fake_photo, training=True)\n\n            # evaluates generator loss\n            monet_gen_loss = self.gen_loss_fn(disc_fake_monet)\n            photo_gen_loss = self.gen_loss_fn(disc_fake_photo)\n\n            # evaluates total cycle consistency loss\n            total_cycle_loss = self.cycle_loss_fn(real_monet, cycled_monet, self.lambda_cycle) + self.cycle_loss_fn(real_photo, cycled_photo, self.lambda_cycle)\n\n            # evaluates total generator loss\n            total_monet_gen_loss = monet_gen_loss + total_cycle_loss + self.identity_loss_fn(real_monet, same_monet, self.lambda_cycle)\n            total_photo_gen_loss = photo_gen_loss + total_cycle_loss + self.identity_loss_fn(real_photo, same_photo, self.lambda_cycle)\n\n            # evaluates discriminator loss\n            monet_disc_loss = self.disc_loss_fn(disc_real_monet, disc_fake_monet)\n            photo_disc_loss = self.disc_loss_fn(disc_real_photo, disc_fake_photo)\n\n        # Calculate the gradients for generator and discriminator\n        monet_generator_gradients = tape.gradient(total_monet_gen_loss,\n                                                  self.m_gen.trainable_variables)\n        photo_generator_gradients = tape.gradient(total_photo_gen_loss,\n                                                  self.p_gen.trainable_variables)\n\n        monet_discriminator_gradients = tape.gradient(monet_disc_loss,\n                                                      self.m_disc.trainable_variables)\n        photo_discriminator_gradients = tape.gradient(photo_disc_loss,\n                                                      self.p_disc.trainable_variables)\n\n        # Apply the gradients to the optimizer\n        self.m_gen_optimizer.apply_gradients(zip(monet_generator_gradients,\n                                                 self.m_gen.trainable_variables))\n\n        self.p_gen_optimizer.apply_gradients(zip(photo_generator_gradients,\n                                                 self.p_gen.trainable_variables))\n\n        self.m_disc_optimizer.apply_gradients(zip(monet_discriminator_gradients,\n                                                  self.m_disc.trainable_variables))\n\n        self.p_disc_optimizer.apply_gradients(zip(photo_discriminator_gradients,\n                                                  self.p_disc.trainable_variables))\n        \n        return {\n            \"monet_gen_loss\": total_monet_gen_loss,\n            \"photo_gen_loss\": total_photo_gen_loss,\n            \"monet_disc_loss\": monet_disc_loss,\n            \"photo_disc_loss\": photo_disc_loss\n        }","metadata":{"execution":{"iopub.status.busy":"2022-06-23T18:41:35.546398Z","iopub.execute_input":"2022-06-23T18:41:35.546772Z","iopub.status.idle":"2022-06-23T18:41:35.566655Z","shell.execute_reply.started":"2022-06-23T18:41:35.546742Z","shell.execute_reply":"2022-06-23T18:41:35.565645Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's build the loss function for the discriminator that compares the original images to the grid of 1, and the false ones to the grid of 0. The ideal discriminant will output only the grid of 1 for the true image and the grid of zeros for false.","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    def discriminator_loss(real, generated):\n        real_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(real), real)\n\n        generated_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.zeros_like(generated), generated)\n\n        total_disc_loss = real_loss + generated_loss\n\n        return total_disc_loss * 0.5","metadata":{"execution":{"iopub.status.busy":"2022-06-23T18:41:35.568293Z","iopub.execute_input":"2022-06-23T18:41:35.568637Z","iopub.status.idle":"2022-06-23T18:41:35.585452Z","shell.execute_reply.started":"2022-06-23T18:41:35.568597Z","shell.execute_reply":"2022-06-23T18:41:35.58421Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's build the loss function for the generator that  tries to trick the discriminator into generating an image so that the discriminator considers it original. An ideal generator will cause the discriminator on the output to return a grid filled with 1.","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    def generator_loss(generated):\n        return tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(generated), generated)","metadata":{"execution":{"iopub.status.busy":"2022-06-23T18:41:35.587051Z","iopub.execute_input":"2022-06-23T18:41:35.587478Z","iopub.status.idle":"2022-06-23T18:41:35.600372Z","shell.execute_reply.started":"2022-06-23T18:41:35.587443Z","shell.execute_reply":"2022-06-23T18:41:35.599406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's build the Cycle consistency, wich is the arithmetic mean of the differences between the original photo and the transformed twice.","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    def calc_cycle_loss(real_image, cycled_image, LAMBDA):\n        loss1 = tf.reduce_mean(tf.abs(real_image - cycled_image))\n\n        return LAMBDA * loss1","metadata":{"execution":{"iopub.status.busy":"2022-06-23T18:41:35.604376Z","iopub.execute_input":"2022-06-23T18:41:35.605179Z","iopub.status.idle":"2022-06-23T18:41:35.611699Z","shell.execute_reply.started":"2022-06-23T18:41:35.605138Z","shell.execute_reply":"2022-06-23T18:41:35.610953Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's build the Identity loss, that is used to compare the image x and that image generator F that produces the images. We expect F (x) ~ x, i.e. if the Monet style image generator is a Monet image, the output should get the same image.","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    def identity_loss(real_image, same_image, LAMBDA):\n        loss = tf.reduce_mean(tf.abs(real_image - same_image))\n        return LAMBDA * 0.5 * loss","metadata":{"execution":{"iopub.status.busy":"2022-06-23T18:41:35.612731Z","iopub.execute_input":"2022-06-23T18:41:35.61313Z","iopub.status.idle":"2022-06-23T18:41:35.624056Z","shell.execute_reply.started":"2022-06-23T18:41:35.61309Z","shell.execute_reply":"2022-06-23T18:41:35.623078Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"c7\"></a>\n# Training","metadata":{}},{"cell_type":"markdown","source":"Let's initialize the optimizers for the models in strategy.scope() because we are using TPU","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    monet_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    photo_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n\n    monet_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    photo_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)","metadata":{"execution":{"iopub.status.busy":"2022-06-23T18:41:35.62522Z","iopub.execute_input":"2022-06-23T18:41:35.625455Z","iopub.status.idle":"2022-06-23T18:41:35.638326Z","shell.execute_reply.started":"2022-06-23T18:41:35.625427Z","shell.execute_reply":"2022-06-23T18:41:35.637399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now let's compile and fit  the model","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    cycle_gan_model = CycleGan(\n        monet_generator, photo_generator, monet_discriminator, photo_discriminator\n    )\n\n    cycle_gan_model.compile(\n        m_gen_optimizer = monet_generator_optimizer,\n        p_gen_optimizer = photo_generator_optimizer,\n        m_disc_optimizer = monet_discriminator_optimizer,\n        p_disc_optimizer = photo_discriminator_optimizer,\n        gen_loss_fn = generator_loss,\n        disc_loss_fn = discriminator_loss,\n        cycle_loss_fn = calc_cycle_loss,\n        identity_loss_fn = identity_loss\n    )","metadata":{"execution":{"iopub.status.busy":"2022-06-23T18:41:35.639917Z","iopub.execute_input":"2022-06-23T18:41:35.640449Z","iopub.status.idle":"2022-06-23T18:41:35.707721Z","shell.execute_reply.started":"2022-06-23T18:41:35.640405Z","shell.execute_reply":"2022-06-23T18:41:35.706737Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cycle_gan_model.fit(\n    final_dataset, \n    epochs=100\n)","metadata":{"execution":{"iopub.status.busy":"2022-06-23T18:41:35.70911Z","iopub.execute_input":"2022-06-23T18:41:35.709407Z","iopub.status.idle":"2022-06-23T18:43:59.594414Z","shell.execute_reply.started":"2022-06-23T18:41:35.709377Z","shell.execute_reply":"2022-06-23T18:43:59.593196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"c8\"></a>\n# Results","metadata":{}},{"cell_type":"code","source":"_, ax = plt.subplots(5, 2, figsize=(10, 10))\nfor i, img in enumerate(photo_ds.take(5)):\n    prediction = monet_generator(img, training=False)[0].numpy()\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    img = (img[0] * 127.5 + 127.5).numpy().astype(np.uint8)\n\n    ax[i, 0].imshow(img)\n    ax[i, 1].imshow(prediction)\n    ax[i, 0].set_title(\"Input Photo\")\n    ax[i, 1].set_title(\"Photo with monet style\")\n    ax[i, 0].axis(\"off\")\n    ax[i, 1].axis(\"off\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-23T18:43:59.596265Z","iopub.execute_input":"2022-06-23T18:43:59.596526Z","iopub.status.idle":"2022-06-23T18:44:01.781652Z","shell.execute_reply.started":"2022-06-23T18:43:59.596497Z","shell.execute_reply":"2022-06-23T18:44:01.781001Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import PIL\n! mkdir ../images\n\ni = 1\nfor img in photo_ds:\n    prediction = monet_generator(img, training=False)[0].numpy()\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    im = PIL.Image.fromarray(prediction)\n    im.save(\"../images/\" + str(i) + \".jpg\")\n    i += 1\n    \n\nimport shutil\nshutil.make_archive(\"/kaggle/working/images\", 'zip', \"/kaggle/images\")","metadata":{},"execution_count":null,"outputs":[]}]}