{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **Competetion Objective**\n\n**Using GANs for generating Monet Style Art**\n\nThe Task for generating Monet style Art Falls in the domain of **Image to Image Translation**\n\nImage-to-Image Translation is a framework of conditional generation that transforms images into different styles.\nTaking in an image and transforming it to get a different image of a different style, but maintaining the content of that Image is what we try to achieve when working with Image to Image Translation. \n\nBecause GANs are really good at realistic generation, they are really well-suited for this image-to-image translation task. \n\n","metadata":{}},{"cell_type":"markdown","source":"## **Unpaired Image to Image Translation**\n\nSince we have a limited number of Images, 300 monet style paintings without any corresponding pair with real Image, Unpaired Image to Image Translation is where out task categorises to.\n\n**Unpaired image to image translation** is an Unsupervised method, uses piles of different styled images instead of paired images.\nThe model learns that mapping between those two piles by ***keeping the contents*** that are present in both, while ***changing the style*** which is different or unique to each of those piles. ","metadata":{}},{"cell_type":"markdown","source":"## Dependencies ","metadata":{}},{"cell_type":"code","source":"import os, random, json, PIL, shutil, re\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom kaggle_datasets import KaggleDatasets\nimport tensorflow as tf\nimport tensorflow.keras.layers as L\nimport tensorflow_addons as tfa\nfrom tensorflow import keras\nfrom tensorflow.keras import Model, losses, optimizers","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:54:45.539706Z","iopub.execute_input":"2023-10-04T10:54:45.540095Z","iopub.status.idle":"2023-10-04T10:54:51.897241Z","shell.execute_reply.started":"2023-10-04T10:54:45.540007Z","shell.execute_reply":"2023-10-04T10:54:51.896470Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Configuring TPU","metadata":{}},{"cell_type":"code","source":"try:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print(f'Running on TPU {tpu.master()}')\nexcept ValueError:\n    tpu = None\n\nif tpu:\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nelse:\n    strategy = tf.distribute.get_strategy()\n\n\nREPLICAS = strategy.num_replicas_in_sync\nprint(f'REPLICAS: {REPLICAS}')\nAUTO = tf.data.experimental.AUTOTUNE\n\n\nprint(tf.__version__)","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:54:51.898883Z","iopub.execute_input":"2023-10-04T10:54:51.899296Z","iopub.status.idle":"2023-10-04T10:54:51.913932Z","shell.execute_reply.started":"2023-10-04T10:54:51.899258Z","shell.execute_reply":"2023-10-04T10:54:51.913069Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Model Parameters","metadata":{}},{"cell_type":"code","source":"HEIGHT = 256\nWIDTH = 256\nCHANNELS = 3\nEPOCHS = 50\nBATCH_SIZE = 1","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:54:51.915349Z","iopub.execute_input":"2023-10-04T10:54:51.915621Z","iopub.status.idle":"2023-10-04T10:54:51.925932Z","shell.execute_reply.started":"2023-10-04T10:54:51.915559Z","shell.execute_reply":"2023-10-04T10:54:51.925061Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Load in the Data","metadata":{}},{"cell_type":"code","source":"GCS_PATH = KaggleDatasets().get_gcs_path('gan-getting-started')\n\nMONET_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/monet_tfrec/*.tfrec'))\nPHOTO_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/photo_tfrec/*.tfrec'))\n\ndef count_data_items(filenames):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)\n\nn_monet_samples = count_data_items(MONET_FILENAMES)\nn_photo_samples = count_data_items(PHOTO_FILENAMES)\n\nprint(f'Monet TFRecord files: {len(MONET_FILENAMES)}')\nprint(f'Monet image files: {n_monet_samples}')\nprint(f'Photo TFRecord files: {len(PHOTO_FILENAMES)}')\nprint(f'Photo image files: {n_photo_samples}')","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:54:51.927854Z","iopub.execute_input":"2023-10-04T10:54:51.928377Z","iopub.status.idle":"2023-10-04T10:54:52.723453Z","shell.execute_reply.started":"2023-10-04T10:54:51.928344Z","shell.execute_reply":"2023-10-04T10:54:52.722705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Loading TFRecord Dataset and Visualization functions","metadata":{}},{"cell_type":"code","source":"def decode_image(image):\n    image = tf.image.decode_jpeg(image, channels=CHANNELS)\n    image = (tf.cast(image, tf.float32) / 127.5) - 1\n    image = tf.reshape(image, [HEIGHT, WIDTH, CHANNELS])\n    return image\n\ndef read_tfrecord(example):\n    tfrecord_format = {\n        'image_name': tf.io.FixedLenFeature([], tf.string),\n        'image':      tf.io.FixedLenFeature([], tf.string),\n        'target':     tf.io.FixedLenFeature([], tf.string)\n    }\n    example = tf.io.parse_single_example(example, tfrecord_format)\n    image = decode_image(example['image'])\n    return image\n\ndef load_dataset(filenames):\n    dataset = tf.data.TFRecordDataset(filenames)\n    dataset = dataset.map(read_tfrecord, num_parallel_calls=AUTO)\n    return dataset\n\ndef get_gan_dataset(monet_files, photo_files, augment=None, repeat=True, shuffle=True, batch_size=1):\n\n    monet_ds = load_dataset(monet_files)\n    photo_ds = load_dataset(photo_files)\n\n    if repeat:\n        monet_ds = monet_ds.repeat()\n        photo_ds = photo_ds.repeat()\n    if shuffle:\n        monet_ds = monet_ds.shuffle(2048)\n        photo_ds = photo_ds.shuffle(2048)\n        \n    monet_ds = monet_ds.batch(batch_size, drop_remainder=True)\n    photo_ds = photo_ds.batch(batch_size, drop_remainder=True)\n    monet_ds = monet_ds.cache()\n    photo_ds = photo_ds.cache()\n    monet_ds = monet_ds.prefetch(AUTO)\n    photo_ds = photo_ds.prefetch(AUTO)\n    \n    gan_ds = tf.data.Dataset.zip((monet_ds, photo_ds))\n    \n    return gan_ds\n\ndef display_samples(ds, row, col):\n    ds_iter = iter(ds)\n    plt.figure(figsize=(15, int(15*row/col)))\n    for j in range(row*col):\n        example_sample = next(ds_iter)\n        plt.subplot(row,col,j+1)\n        plt.axis('off')\n        plt.imshow(example_sample[0] * 0.5 + 0.5)\n    plt.show()\n        \ndef display_generated_samples(ds, model, n_samples):\n    ds_iter = iter(ds)\n    for n_sample in range(n_samples):\n        example_sample = next(ds_iter)\n        generated_sample = model.predict(example_sample)\n        \n        plt.subplot(121)\n        plt.title(\"input image\")\n        plt.imshow(example_sample[0] * 0.5 + 0.5)\n        plt.axis('off')\n        \n        plt.subplot(122)\n        plt.title(\"Generated image\")\n        plt.imshow(generated_sample[0] * 0.5 + 0.5)\n        plt.axis('off')\n        plt.show()\n        ","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:54:52.724586Z","iopub.execute_input":"2023-10-04T10:54:52.724997Z","iopub.status.idle":"2023-10-04T10:54:52.739218Z","shell.execute_reply.started":"2023-10-04T10:54:52.724961Z","shell.execute_reply":"2023-10-04T10:54:52.738409Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display_samples(load_dataset(MONET_FILENAMES).batch(1), 4, 6)","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:54:52.740167Z","iopub.execute_input":"2023-10-04T10:54:52.741064Z","iopub.status.idle":"2023-10-04T10:54:58.540841Z","shell.execute_reply.started":"2023-10-04T10:54:52.741030Z","shell.execute_reply":"2023-10-04T10:54:58.539909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display_samples(load_dataset(PHOTO_FILENAMES).batch(1), 4, 6)","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:54:58.542566Z","iopub.execute_input":"2023-10-04T10:54:58.543553Z","iopub.status.idle":"2023-10-04T10:55:00.774378Z","shell.execute_reply.started":"2023-10-04T10:54:58.543502Z","shell.execute_reply":"2023-10-04T10:55:00.773680Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"monet_ds = load_dataset(MONET_FILENAMES).batch(1)\nphoto_ds = load_dataset(PHOTO_FILENAMES).batch(1)\n\n\nfast_photo_ds = load_dataset(PHOTO_FILENAMES).batch(32*strategy.num_replicas_in_sync).prefetch(32)\nfid_photo_ds = load_dataset(PHOTO_FILENAMES).take(1024).batch(32*strategy.num_replicas_in_sync).prefetch(32)\nfid_monet_ds = load_dataset(MONET_FILENAMES).batch(32*strategy.num_replicas_in_sync).prefetch(32)","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:55:00.776071Z","iopub.execute_input":"2023-10-04T10:55:00.781972Z","iopub.status.idle":"2023-10-04T10:55:00.902459Z","shell.execute_reply.started":"2023-10-04T10:55:00.781934Z","shell.execute_reply":"2023-10-04T10:55:00.901882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n\n    inception_model = tf.keras.applications.InceptionV3(input_shape=(256,256,3),pooling=\"avg\",include_top=False)\n\n    mix3  = inception_model.get_layer(\"mixed9\").output\n    f0 = tf.keras.layers.GlobalAveragePooling2D()(mix3)\n\n    inception_model = tf.keras.Model(inputs=inception_model.input, outputs=f0)\n    inception_model.trainable = False\n\n    \n    \n    def calculate_activation_statistics_mod(images,fid_model):\n\n            act=tf.cast(fid_model.predict(images), tf.float32)\n\n            mu = tf.reduce_mean(act, axis=0)\n            mean_x = tf.reduce_mean(act, axis=0, keepdims=True)\n            mx = tf.matmul(tf.transpose(mean_x), mean_x)\n            vx = tf.matmul(tf.transpose(act), act)/tf.cast(tf.shape(act)[0], tf.float32)\n            sigma = vx - mx\n            return mu, sigma\n    myFID_mu2, myFID_sigma2 = calculate_activation_statistics_mod(fid_monet_ds,inception_model)        \n    fids=[]","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:55:00.903651Z","iopub.execute_input":"2023-10-04T10:55:00.904104Z","iopub.status.idle":"2023-10-04T10:55:12.481875Z","shell.execute_reply.started":"2023-10-04T10:55:00.904069Z","shell.execute_reply":"2023-10-04T10:55:12.481098Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n    def calculate_frechet_distance(mu1,sigma1,mu2,sigma2):\n        fid_epsilon = 1e-14\n       \n        covmean = tf.linalg.sqrtm(tf.cast(tf.matmul(sigma1,sigma2),tf.complex64))\n#         isgood=tf.cast(tf.math.is_finite(covmean), tf.int32)\n#         if tf.size(isgood)!=tf.math.reduce_sum(isgood):\n#             return 0\n\n        covmean = tf.cast(tf.math.real(covmean),tf.float32)\n  \n        tr_covmean = tf.linalg.trace(covmean)\n\n\n        return tf.matmul(tf.expand_dims(mu1 - mu2, axis=0),tf.expand_dims(mu1 - mu2, axis=1)) + tf.linalg.trace(sigma1) + tf.linalg.trace(sigma2) - 2 * tr_covmean\n\n\n    \n    \n    def FID(images,gen_model,inception_model=inception_model,myFID_mu2=myFID_mu2, myFID_sigma2=myFID_sigma2):\n                inp = keras.Input(shape=[256, 256, 3], name='input_image')\n                x  = gen_model(inp)\n                x=inception_model(x)\n                fid_model = tf.keras.Model(inputs=inp, outputs=x)\n                \n                mu1, sigma1= calculate_activation_statistics_mod(images,fid_model)\n\n                fid_value = calculate_frechet_distance(mu1, sigma1,myFID_mu2, myFID_sigma2)\n\n\n                return fid_value\n","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:55:12.484813Z","iopub.execute_input":"2023-10-04T10:55:12.485696Z","iopub.status.idle":"2023-10-04T10:55:12.494938Z","shell.execute_reply.started":"2023-10-04T10:55:12.485648Z","shell.execute_reply":"2023-10-04T10:55:12.494225Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"example_monet = next(iter(monet_ds))\nexample_photo = next(iter(photo_ds))\n\n\n\nplt.subplot(121)\nplt.title('Photo')\nplt.imshow(example_photo[0] * 0.5 + 0.5)\n\nplt.subplot(122)\nplt.title('Monet')\nplt.imshow(example_monet[0] * 0.5 + 0.5)","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:55:12.497972Z","iopub.execute_input":"2023-10-04T10:55:12.498781Z","iopub.status.idle":"2023-10-04T10:55:14.095172Z","shell.execute_reply.started":"2023-10-04T10:55:12.498731Z","shell.execute_reply":"2023-10-04T10:55:14.094526Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **GAN Architecture: CycleGAN**\n\n**CycleGAN** is made up of two GANs, then make a cycle and they rely on each other to try to compute all different types of loss terms.\nIn fact, the generators have six loss terms in total:\n* The least squares adversarial (main loss term).\n* The cycle consistency loss, \n* The optional identity loss for each of the generators.\n\nAnd the discriminators are a bit simpler with just least squares adversarial loss\nusing a ***PatchGAN*** that you learn from pix2pix. \n\n**Cycle consistency** is important in transferring of common style elements while maintaining common content across those images, and it is a really, ***really important loss term***.\nThis can be done by adding that pixel distance loss to the or adversarial loss to encourage cycle consistency in both directions.\nLooking at fake zebra to real zebra and fake horse to real horse. The ablation studies show that the cycle consistency loss term in both directions help prevent mode collapse and help with this uwieldy, unpaired image to image translation task. ","metadata":{}},{"cell_type":"markdown","source":"## Model Functions","metadata":{}},{"cell_type":"code","source":"def ContractingBlock(inputs, filters, kernel_size, strides = 2, apply_instancenorm=True):\n    initializer = tf.keras.initializers.RandomNormal(0., 0.02)\n    gamma_init = tf.keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    result = L.Conv2D(filters, kernel_size, strides=strides, padding='same',\n                             kernel_initializer=initializer, use_bias=False)(inputs)\n\n    if apply_instancenorm:\n        result = tfa.layers.InstanceNormalization(gamma_initializer=gamma_init)(result)\n\n    result = L.LeakyReLU(alpha=0.2)(result)\n\n    return result\n\n\n\ndef ResnetBlock(input_layer, filters, kernel_size):\n    \n    out_res_1 = ContractingBlock(input_layer, filters, kernel_size, strides = 1)\n    out_res_2 = ContractingBlock(out_res_1, filters, kernel_size, strides = 1)\n    \n    return out_res_2 + input_layer\n\n\n\ndef ExpandingBlock(inputs, filters, kernel_size, strides = 2, apply_dropout=False):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = tf.keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    result = L.Conv2DTranspose(filters, kernel_size, strides=strides, padding='same',\n                             kernel_initializer=initializer, use_bias=False)(inputs)\n\n    result = tfa.layers.InstanceNormalization(gamma_initializer=gamma_init)(result)\n    \n    if apply_dropout:\n        result = L.Dropout(0.5)(result)\n\n    result = L.ReLU()(result)\n\n    return result","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:55:14.096319Z","iopub.execute_input":"2023-10-04T10:55:14.096693Z","iopub.status.idle":"2023-10-04T10:55:14.107413Z","shell.execute_reply.started":"2023-10-04T10:55:14.096656Z","shell.execute_reply":"2023-10-04T10:55:14.106609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Building the Generator","metadata":{}},{"cell_type":"code","source":"def Generator():\n    \"\"\"\n     Generator Module\n    A series of 3 contracting blocks, 9 residual blocks, and 3 expanding blocks to \n    transform an input image into an translated image from the other class, with an upfeature\n    layer at the start and a downfeature layer at the end.\n    \"\"\"\n    \n    inputs = keras.Input(shape=[HEIGHT, WIDTH, CHANNELS])\n    \n    gen_img = ContractingBlock(inputs, 64, 7, strides=1, apply_instancenorm=False) # (bs, 256, 256, 64)\n    gen_img = ContractingBlock(gen_img, 128, 3) # (bs, 128, 128, 128)\n    gen_img = ContractingBlock(gen_img, 256, 3) # (bs, 64, 64, 256)\n#     gen_img = ContractingBlock(gen_img, 256, 3) # (bs, 32, 32, 256)\n    \n    \n    res_img = ResnetBlock(gen_img, 256, 3)\n    res_img = ResnetBlock(res_img, 256, 3)\n    res_img = ResnetBlock(res_img, 256, 3)\n    res_img = ResnetBlock(res_img, 256, 3)\n    res_img = ResnetBlock(res_img, 256, 3)\n    res_img = ResnetBlock(res_img, 256, 3)\n    res_img = ResnetBlock(res_img, 256, 3)\n    res_img = ResnetBlock(res_img, 256, 3)\n    res_img = ResnetBlock(res_img, 256, 3)\n    \n    \n    exp_img = ExpandingBlock(res_img, 128, 3)\n    exp_img = ExpandingBlock(exp_img, 64, 3)\n#     exp_img = ExpandingBlock(exp_img, 3, 7, strides=1)\n    initializer = tf.random_normal_initializer(0., 0.02)\n    last = L.Conv2DTranspose(3, 7,\n                                  strides=1,\n                                  padding='same',\n                                  kernel_initializer=initializer,\n                                  activation='tanh')\n    out  = last(exp_img)\n#     exp_img = ExpandingBlock(exp_img, 3, 7)\n    \n    return Model(inputs=inputs, outputs=out)","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:55:14.108545Z","iopub.execute_input":"2023-10-04T10:55:14.109191Z","iopub.status.idle":"2023-10-04T10:55:14.123495Z","shell.execute_reply.started":"2023-10-04T10:55:14.109146Z","shell.execute_reply":"2023-10-04T10:55:14.122679Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen = Generator()\ninp = keras.Input(shape=[HEIGHT, WIDTH, CHANNELS])\nx = gen(inp)","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:55:14.124733Z","iopub.execute_input":"2023-10-04T10:55:14.124990Z","iopub.status.idle":"2023-10-04T10:55:15.468139Z","shell.execute_reply.started":"2023-10-04T10:55:14.124936Z","shell.execute_reply":"2023-10-04T10:55:15.467407Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.keras.utils.plot_model(gen, show_shapes=True, dpi=64)","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:55:15.469201Z","iopub.execute_input":"2023-10-04T10:55:15.469438Z","iopub.status.idle":"2023-10-04T10:55:16.798867Z","shell.execute_reply.started":"2023-10-04T10:55:15.469405Z","shell.execute_reply":"2023-10-04T10:55:16.798107Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Build The Discriminator","metadata":{}},{"cell_type":"code","source":"def Discriminator():\n    '''\n    Discriminator Module\n    Structured like the contracting path of the Generator, the discriminator will\n    output a matrix of values classifying corresponding portions of the image as real or fake. \n    Parameters:\n        input_channels: the number of image input channels\n        hidden_channels: the initial number of discriminator convolutional filters\n    '''\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    inp = keras.Input(shape=[256, 256, 3], name='input_image')\n    \n    x = inp\n    \n    des_img = ContractingBlock(x, 64, 7)\n    des_img = ContractingBlock(x, 128, 4)    \n    des_img = ContractingBlock(x, 256, 4)    \n    \n    zero_pad1 = L.ZeroPadding2D()(des_img) \n    conv = L.Conv2D(512, 4, strides=1,\n                         kernel_initializer=initializer,\n                         use_bias=False)(zero_pad1) \n\n    norm1 = tfa.layers.InstanceNormalization(gamma_initializer=gamma_init)(conv)\n\n    leaky_relu = L.LeakyReLU()(norm1)\n\n    zero_pad2 = L.ZeroPadding2D()(leaky_relu) \n\n    last = L.Conv2D(1, 4, strides=1,\n                         kernel_initializer=initializer)(zero_pad2) \n\n    return tf.keras.Model(inputs=inp, outputs=last)\n","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:55:16.800339Z","iopub.execute_input":"2023-10-04T10:55:16.800724Z","iopub.status.idle":"2023-10-04T10:55:16.813562Z","shell.execute_reply.started":"2023-10-04T10:55:16.800689Z","shell.execute_reply":"2023-10-04T10:55:16.812382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dis = Discriminator()\ninp = keras.Input(shape=[HEIGHT, WIDTH, CHANNELS])\nx = dis(inp)","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:55:16.814936Z","iopub.execute_input":"2023-10-04T10:55:16.815902Z","iopub.status.idle":"2023-10-04T10:55:16.995571Z","shell.execute_reply.started":"2023-10-04T10:55:16.815855Z","shell.execute_reply":"2023-10-04T10:55:16.994904Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.keras.utils.plot_model(dis, show_shapes=True, dpi=64)","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:55:16.996694Z","iopub.execute_input":"2023-10-04T10:55:16.997309Z","iopub.status.idle":"2023-10-04T10:55:17.187465Z","shell.execute_reply.started":"2023-10-04T10:55:16.997274Z","shell.execute_reply":"2023-10-04T10:55:17.186713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Building the CycleGAN Model","metadata":{}},{"cell_type":"markdown","source":"**The CycleGAN architecture works as follows**:\n\nThe model architecture is comprised of two generator models: one generator (Generator-A) for generating images for the first domain (Domain-A) and the second generator (Generator-B) for generating images for the second domain (Domain-B).\n\n*   **Generator-A -> Domain-A**\n*   **Generator-B -> Domain-B**\n\nThe generator models perform image translation, meaning that the image generation process is conditional on an input image, specifically an image from the other domain. Generator-A takes an image from Domain-B as input and Generator-B takes an image from Domain-A as input.\n\n *   **Domain-B -> Generator-A -> Domain-A**\n *  **Domain-A -> Generator-B -> Domain-B**\n\nEach generator has a corresponding discriminator model.\n\nThe first discriminator model (Discriminator-A) takes real images from Domain-A and generated images from Generator-A and predicts whether they are real or fake. The second discriminator model (Discriminator-B) takes real images from Domain-B and generated images from Generator-B and predicts whether they are real or fake.\n\n*    **Domain-A -> Discriminator-A -> [Real/Fake]**\n*    **Domain-B -> Generator-A -> Discriminator-A -> [Real/Fake]**\n*    **Domain-B -> Discriminator-B -> [Real/Fake]**\n*   **Domain-A -> Generator-B -> Discriminator-B -> [Real/Fake]**\n\nThe discriminator and generator models are trained in an adversarial zero-sum process, like normal GAN models.\n\nThe generated image must retain the property of original image, so if we generate a fake image using a generator say GeneratorA→B then we must be able to get back to original image using the another generator GeneratorB→A - it must satisfy cyclic-consistency.\n\n[Reference](https://machinelearningmastery.com/how-to-develop-cyclegan-models-from-scratch-with-keras/)","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    monet_generator = Generator() # transforms photos to Monet-esque paintings\n    photo_generator = Generator() # transforms Monet paintings to be more like photos\n\n    monet_discriminator = Discriminator() # differentiates real Monet paintings and generated Monet paintings\n    photo_discriminator = Discriminator() # differentiates real photos and generated photos","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:55:17.188983Z","iopub.execute_input":"2023-10-04T10:55:17.189698Z","iopub.status.idle":"2023-10-04T10:55:18.663389Z","shell.execute_reply.started":"2023-10-04T10:55:17.189647Z","shell.execute_reply":"2023-10-04T10:55:18.662669Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Since our generators are not trained yet, the generated Monet-esque photo does not show what is expected at this point.**","metadata":{}},{"cell_type":"code","source":"to_monet = monet_generator(example_photo)\n\nplt.subplot(1, 2, 1)\nplt.title(\"Original Photo\")\nplt.imshow(example_photo[0] * 0.5 + 0.5)\n\nplt.subplot(1, 2, 2)\nplt.title(\"Monet-esque Photo\")\nplt.imshow(to_monet[0] * 0.5 + 0.5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:55:18.664559Z","iopub.execute_input":"2023-10-04T10:55:18.664977Z","iopub.status.idle":"2023-10-04T10:55:19.552812Z","shell.execute_reply.started":"2023-10-04T10:55:18.664935Z","shell.execute_reply":"2023-10-04T10:55:19.552118Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We will subclass a ***Model*** so that we can run ***fit()*** later to train our model. During the training step, the model transforms a photo to a Monet painting and then back to a photo. The difference between the original photo and the twice-transformed photo is the cycle-consistency loss. We want the original photo and the twice-transformed photo to be similar to one another.\n","metadata":{}},{"cell_type":"code","source":"class CycleGan(Model):\n    def __init__(\n        self,\n        monet_generator,\n        photo_generator,\n        monet_discriminator,\n        photo_discriminator,\n        lambda_cycle=10,\n    ):\n        super(CycleGan, self).__init__()\n        self.m_gen = monet_generator\n        self.p_gen = photo_generator\n        self.m_disc = monet_discriminator\n        self.p_disc = photo_discriminator\n        self.lambda_cycle = lambda_cycle\n        \n    def compile(\n        self,\n        m_gen_optimizer,\n        p_gen_optimizer,\n        m_disc_optimizer,\n        p_disc_optimizer,\n        gen_loss_fn,\n        disc_loss_fn,\n        cycle_loss_fn,\n        identity_loss_fn\n    ):\n        super(CycleGan, self).compile()\n        self.m_gen_optimizer = m_gen_optimizer\n        self.p_gen_optimizer = p_gen_optimizer\n        self.m_disc_optimizer = m_disc_optimizer\n        self.p_disc_optimizer = p_disc_optimizer\n        self.gen_loss_fn = gen_loss_fn\n        self.disc_loss_fn = disc_loss_fn\n        self.cycle_loss_fn = cycle_loss_fn\n        self.identity_loss_fn = identity_loss_fn\n        \n    def train_step(self, batch_data):\n        real_monet, real_photo = batch_data\n        \n        with tf.GradientTape(persistent=True) as tape:\n            # photo to monet back to photo\n            fake_monet = self.m_gen(real_photo, training=True)\n            cycled_photo = self.p_gen(fake_monet, training=True)\n\n            # monet to photo back to monet\n            fake_photo = self.p_gen(real_monet, training=True)\n            cycled_monet = self.m_gen(fake_photo, training=True)\n\n            # generating itself\n            same_monet = self.m_gen(real_monet, training=True)\n            same_photo = self.p_gen(real_photo, training=True)\n\n            # discriminator used to check, inputing real images\n            disc_real_monet = self.m_disc(real_monet, training=True)\n            disc_real_photo = self.p_disc(real_photo, training=True)\n\n            # discriminator used to check, inputing fake images\n            disc_fake_monet = self.m_disc(fake_monet, training=True)\n            disc_fake_photo = self.p_disc(fake_photo, training=True)\n\n            # evaluates generator loss\n            monet_gen_loss = self.gen_loss_fn(disc_fake_monet)\n            photo_gen_loss = self.gen_loss_fn(disc_fake_photo)\n\n            # evaluates total cycle consistency loss\n            total_cycle_loss = self.cycle_loss_fn(real_monet, cycled_monet, self.lambda_cycle) + self.cycle_loss_fn(real_photo, cycled_photo, self.lambda_cycle)\n\n            # evaluates total generator loss\n            total_monet_gen_loss = monet_gen_loss + total_cycle_loss + self.identity_loss_fn(real_monet, same_monet, self.lambda_cycle)\n            total_photo_gen_loss = photo_gen_loss + total_cycle_loss + self.identity_loss_fn(real_photo, same_photo, self.lambda_cycle)\n\n            # evaluates discriminator loss\n            monet_disc_loss = self.disc_loss_fn(disc_real_monet, disc_fake_monet)\n            photo_disc_loss = self.disc_loss_fn(disc_real_photo, disc_fake_photo)\n\n        # Calculate the gradients for generator and discriminator\n        monet_generator_gradients = tape.gradient(total_monet_gen_loss,\n                                                  self.m_gen.trainable_variables)\n        photo_generator_gradients = tape.gradient(total_photo_gen_loss,\n                                                  self.p_gen.trainable_variables)\n\n        monet_discriminator_gradients = tape.gradient(monet_disc_loss,\n                                                      self.m_disc.trainable_variables)\n        photo_discriminator_gradients = tape.gradient(photo_disc_loss,\n                                                      self.p_disc.trainable_variables)\n\n        # Apply the gradients to the optimizer\n        self.m_gen_optimizer.apply_gradients(zip(monet_generator_gradients,\n                                                 self.m_gen.trainable_variables))\n\n        self.p_gen_optimizer.apply_gradients(zip(photo_generator_gradients,\n                                                 self.p_gen.trainable_variables))\n\n        self.m_disc_optimizer.apply_gradients(zip(monet_discriminator_gradients,\n                                                  self.m_disc.trainable_variables))\n\n        self.p_disc_optimizer.apply_gradients(zip(photo_discriminator_gradients,\n                                                  self.p_disc.trainable_variables))\n        \n        return {\n            \"monet_gen_loss\": total_monet_gen_loss,\n            \"photo_gen_loss\": total_photo_gen_loss,\n            \"monet_disc_loss\": monet_disc_loss,\n            \"photo_disc_loss\": photo_disc_loss\n        }\n","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:55:19.554074Z","iopub.execute_input":"2023-10-04T10:55:19.554447Z","iopub.status.idle":"2023-10-04T10:55:19.569419Z","shell.execute_reply.started":"2023-10-04T10:55:19.554411Z","shell.execute_reply":"2023-10-04T10:55:19.568691Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Discriminator loss\n### Adversarial loss Part 1\n\nDiscriminator must be trained such that recommendation for images from category A must be as close to 1, and vice versa for discriminator B. So Discriminator A would like to minimize ***(DiscriminatorA(a)−1)^2*** and same goes for B as well. \n\n\n### Adversarial loss Part 2\n\nSince, discriniator should be able to distinguish between generated and original images, it should also be predicting 0 for images produced by the generator, i.e. Discriminator A wwould like to minimize ***(DiscriminatorA(GeneratorB→A(b)))2***. \n\nThe above two losses can be implemented as follows:\n","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    def discriminator_loss(real, generated):\n        real_loss = tf.keras.losses.MeanSquaredError(reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(real), real)\n\n        generated_loss = tf.keras.losses.MeanSquaredError(reduction=tf.keras.losses.Reduction.NONE)(tf.zeros_like(generated), generated)\n\n        total_disc_loss = real_loss + generated_loss\n\n        return total_disc_loss * 0.5","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:55:19.570659Z","iopub.execute_input":"2023-10-04T10:55:19.571205Z","iopub.status.idle":"2023-10-04T10:55:19.586956Z","shell.execute_reply.started":"2023-10-04T10:55:19.571167Z","shell.execute_reply":"2023-10-04T10:55:19.586301Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Genrator Loss\n\nGenerator should eventually be able to fool the discriminator about the authencity of it's generated images. This can done if the recommendation by discriminator for the generated images is as close to 1 as possible. So generator would like to minimize ***(DiscriminatorB(GeneratorA→B(a))−1)2***\nSo the loss is:","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    def generator_loss(generated):\n        return tf.keras.losses.MeanSquaredError(reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(generated), generated)","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:55:19.588077Z","iopub.execute_input":"2023-10-04T10:55:19.588742Z","iopub.status.idle":"2023-10-04T10:55:19.600051Z","shell.execute_reply.started":"2023-10-04T10:55:19.588707Z","shell.execute_reply":"2023-10-04T10:55:19.599328Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Cyclic loss\n\nOne of the most important one is the cyclic loss that captures that we are able to get the image back using another generator and thus the difference between the original image and the cyclic image should be as small as possible.\n","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    def calc_cycle_loss(real_image, cycled_image, LAMBDA):\n        loss1 = tf.reduce_mean(tf.abs(real_image - cycled_image))\n\n        return LAMBDA * loss1","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:55:19.601251Z","iopub.execute_input":"2023-10-04T10:55:19.601705Z","iopub.status.idle":"2023-10-04T10:55:19.612894Z","shell.execute_reply.started":"2023-10-04T10:55:19.601671Z","shell.execute_reply":"2023-10-04T10:55:19.611985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Identity Loss\n\nWe'll want to measure the change in an image when you pass the generator an example from the target domain instead of the input domain it's expecting. **The output should be the same as the input since it is already of the target domain class.** For example, if you put a horse through a zebra -> horse generator, you'd expect the output to be the same horse because nothing needed to be transformed. It's already a horse! You don't want your generator to be transforming it into any other thing, so you want to encourage this behavior. In encouraging this identity mapping, the authors of CycleGAN found that for some tasks, this helped properly preserve the colors of an image, even when the expected input (here, a zebra) was put in. This is particularly useful for **the photos <-> paintings mapping** and, while an optional aesthetic component, you might find it useful for your applications down the line.","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    def identity_loss(real_image, same_image, LAMBDA):\n        loss = tf.reduce_mean(tf.abs(real_image - same_image))\n        return LAMBDA * 0.5 * loss","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:55:19.614173Z","iopub.execute_input":"2023-10-04T10:55:19.614408Z","iopub.status.idle":"2023-10-04T10:55:19.625077Z","shell.execute_reply.started":"2023-10-04T10:55:19.614376Z","shell.execute_reply":"2023-10-04T10:55:19.624329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Train the CycleGAN\n\nLet's compile our model. Since we used ***tf.keras.Model*** to build our CycleGAN, we can just use the fit function to train our model.\n","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    monet_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    photo_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n\n    monet_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    photo_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:55:19.626224Z","iopub.execute_input":"2023-10-04T10:55:19.626601Z","iopub.status.idle":"2023-10-04T10:55:19.638001Z","shell.execute_reply.started":"2023-10-04T10:55:19.626569Z","shell.execute_reply":"2023-10-04T10:55:19.637262Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n    cycle_gan_model = CycleGan(\n        monet_generator, photo_generator, monet_discriminator, photo_discriminator\n    )\n    cycle_gan_model.compile(\n        m_gen_optimizer = monet_generator_optimizer,\n        p_gen_optimizer = photo_generator_optimizer,\n        m_disc_optimizer = monet_discriminator_optimizer,\n        p_disc_optimizer = photo_discriminator_optimizer,\n        gen_loss_fn = generator_loss,\n        disc_loss_fn = discriminator_loss,\n        cycle_loss_fn = calc_cycle_loss,\n        identity_loss_fn = identity_loss\n    )\n","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:55:19.639199Z","iopub.execute_input":"2023-10-04T10:55:19.639678Z","iopub.status.idle":"2023-10-04T10:55:19.668874Z","shell.execute_reply.started":"2023-10-04T10:55:19.639643Z","shell.execute_reply":"2023-10-04T10:55:19.668153Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cycle_gan_model.load_weights(\"../input/cyclegan-monet/my_checkpoint.ckpt\")    ","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:55:19.672517Z","iopub.execute_input":"2023-10-04T10:55:19.672819Z","iopub.status.idle":"2023-10-04T10:55:23.697179Z","shell.execute_reply.started":"2023-10-04T10:55:19.672788Z","shell.execute_reply":"2023-10-04T10:55:23.696492Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cycle_gan_model.fit(\n    tf.data.Dataset.zip((monet_ds, photo_ds)),\n    epochs=50\n)\n\n","metadata":{"execution":{"iopub.status.busy":"2023-10-04T10:55:23.698435Z","iopub.execute_input":"2023-10-04T10:55:23.698693Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"FID(fid_photo_ds,monet_generator) ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"_, ax = plt.subplots(6, 2, figsize=(12, 12))\nfor i, img in enumerate(photo_ds.take(6)):\n    prediction = monet_generator(img, training=False)[0].numpy()\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    img = (img[0] * 127.5 + 127.5).numpy().astype(np.uint8)\n\n    ax[i, 0].imshow(img)\n    ax[i, 1].imshow(prediction)\n    ax[i, 0].set_title(\"Input Photo\")\n    ax[i, 1].set_title(\"Monet-esque\")\n    ax[i, 0].axis(\"off\")\n    ax[i, 1].axis(\"off\")\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import PIL\n! mkdir ./images","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"i = 1\nfor img in photo_ds:\n    prediction = monet_generator(img, training=False)[0].numpy()\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    im = PIL.Image.fromarray(prediction)\n    im.save(\"./images/\" + str(i) + \".jpg\")\n    i += 1","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import shutil\nshutil.make_archive(\"/kaggle/working/images\", 'zip', \"/kaggle/working/images\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cycle_gan_model.save_weights('./my_checkpoint.ckpt')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(len(os.listdir(\"./images\")))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}