{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## I’m Something of a Painter Myself Project\n\nThis project covers the Kaggle intoduction to GAN models, especially CycleGAN. The CycleGAN is an extension of the GAN architecture that involves the simultaneous training of two generator models and two discriminator models, its mainly used for style transfer in this project.\n\nOne generator takes images from the first domain as input and outputs images for the second domain, and the other generator takes images from the second domain as input and generates images for the first domain. Discriminator models are then used to determine how plausible the generated images are and update the generator models accordingly. In essence CycleGANs are extended architecture over the simple GAN.\n\nIn the course we were taught general GANs only, old assignment also covered simple GANs where we create One Generator and one Discriminator based Deep Convolutional Generative Adversarial Network. However since this assignment covers the topics not covered in the course we have accessed various blogposts, videos, and Kaggle introductry notebooks to create this project.\n\nDataset is based on the Kaggle I’m Something of a Painter Myself competition: https://www.kaggle.com/competitions/gan-getting-started\n\nWe are tasked to create a CycleGAN which is able to apply transfer learning and transfer the monet style to non monet styled images.\n\nWe are given 300 Monet Images and 7038 Other images. In the submission we will submit the monet converted images to the Kaggle competition.\n\nThis project is broken down into following parts:\n1. Packages\n2. EDA\n3. Model Creation\n4. Model Fitting and Hyperparameter Tuning\n5. Model Evaluation\n6. Conclusion","metadata":{}},{"cell_type":"markdown","source":"## 1. Packages","metadata":{}},{"cell_type":"code","source":"import os\nimport cv2\n\nimport tensorflow as tf\nfrom tensorflow import keras\nimport tensorflow.keras.layers as tfl\nimport tensorflow_addons as tfa\n\nAUTOTUNE = tf.data.experimental.AUTOTUNE\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nfrom matplotlib import image as Image\n\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Device:', tpu.master())\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nexcept:\n    strategy = tf.distribute.get_strategy()\nprint('Number of replicas:', strategy.num_replicas_in_sync)\n\nfrom kaggle_datasets import KaggleDatasets\nimport warnings\nwarnings.simplefilter('ignore')","metadata":{"execution":{"iopub.status.busy":"2023-08-22T17:34:05.545739Z","iopub.execute_input":"2023-08-22T17:34:05.546441Z","iopub.status.idle":"2023-08-22T17:34:14.570442Z","shell.execute_reply.started":"2023-08-22T17:34:05.546407Z","shell.execute_reply":"2023-08-22T17:34:14.569470Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2. EDA","metadata":{}},{"cell_type":"markdown","source":"This dataset has been already cleaned and tidied. We have also assumed that all the monet images are indeed monet images and not other type of images.\n\nNow we will load the datasets and perform basic EDA by looking through the images among other things.","metadata":{}},{"cell_type":"code","source":"path = KaggleDatasets().get_gcs_path()","metadata":{"execution":{"iopub.status.busy":"2023-08-22T17:34:14.572679Z","iopub.execute_input":"2023-08-22T17:34:14.573765Z","iopub.status.idle":"2023-08-22T17:34:14.988915Z","shell.execute_reply.started":"2023-08-22T17:34:14.573730Z","shell.execute_reply":"2023-08-22T17:34:14.987695Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#We have been given .tfrec files and jpeg, for the project we will use .tfrec files\n\nmonet_filenames = tf.io.gfile.glob(str(path + '/monet_tfrec/*.tfrec'))\nprint('Monet TFRecord Files:', len(monet_filenames))\n\nphoto_filenames = tf.io.gfile.glob(str(path + '/photo_tfrec/*.tfrec'))\nprint('Photo TFRecord Files:', len(monet_filenames))","metadata":{"execution":{"iopub.status.busy":"2023-08-22T17:34:14.990995Z","iopub.execute_input":"2023-08-22T17:34:14.991615Z","iopub.status.idle":"2023-08-22T17:34:15.111650Z","shell.execute_reply.started":"2023-08-22T17:34:14.991578Z","shell.execute_reply":"2023-08-22T17:34:15.110608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"monet_filenames[0]","metadata":{"execution":{"iopub.status.busy":"2023-08-22T17:34:15.114472Z","iopub.execute_input":"2023-08-22T17:34:15.114819Z","iopub.status.idle":"2023-08-22T17:34:15.122260Z","shell.execute_reply.started":"2023-08-22T17:34:15.114788Z","shell.execute_reply":"2023-08-22T17:34:15.120904Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we will look through a random Monet image from the jpeg folder.","metadata":{}},{"cell_type":"code","source":"image_monet = Image.imread('/kaggle/input/gan-getting-started/monet_jpg/000c1e3bff.jpg')\nprint(image_monet.shape)\nplt.imshow(image_monet)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-08-22T17:34:15.124209Z","iopub.execute_input":"2023-08-22T17:34:15.124670Z","iopub.status.idle":"2023-08-22T17:34:15.515831Z","shell.execute_reply.started":"2023-08-22T17:34:15.124637Z","shell.execute_reply":"2023-08-22T17:34:15.514943Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we will look at a random others images.","metadata":{}},{"cell_type":"code","source":"image_others = Image.imread('/kaggle/input/gan-getting-started/photo_jpg/00068bc07f.jpg')\nprint(image_others.shape)\nplt.imshow(image_others)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-08-22T17:34:15.516820Z","iopub.execute_input":"2023-08-22T17:34:15.517150Z","iopub.status.idle":"2023-08-22T17:34:15.844672Z","shell.execute_reply.started":"2023-08-22T17:34:15.517121Z","shell.execute_reply":"2023-08-22T17:34:15.843777Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see from above Monet Image can be easily be identified as painted image while other images are captured via camera, we will create a CycleGAN which will convert the non monet images into monet images. Many photoshop tools like Luminar AI actually uses more advanced transfer learning model to do similar things.","metadata":{}},{"cell_type":"code","source":"monet_jpg_filepath = \"../input/gan-getting-started/monet_jpg\"\nphoto_jpg_filepath = \"../input/gan-getting-started/photo_jpg\"","metadata":{"execution":{"iopub.status.busy":"2023-08-22T17:34:15.846202Z","iopub.execute_input":"2023-08-22T17:34:15.846949Z","iopub.status.idle":"2023-08-22T17:34:15.851538Z","shell.execute_reply.started":"2023-08-22T17:34:15.846913Z","shell.execute_reply":"2023-08-22T17:34:15.850573Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"monet_image_paths = []\nfor dirname, _, filenames in os.walk(monet_jpg_filepath):\n    for filename in filenames:\n        fullpath = os.path.join(dirname, filename)\n        monet_image_paths.append(fullpath)\nmonet_image_paths[:2]","metadata":{"execution":{"iopub.status.busy":"2023-08-22T17:34:15.853129Z","iopub.execute_input":"2023-08-22T17:34:15.853849Z","iopub.status.idle":"2023-08-22T17:34:15.940375Z","shell.execute_reply.started":"2023-08-22T17:34:15.853813Z","shell.execute_reply":"2023-08-22T17:34:15.939308Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"others_image_paths = []\nfor dirname, _, filenames in os.walk(photo_jpg_filepath):\n    for filename in filenames:\n        fullpath = os.path.join(dirname, filename)\n        others_image_paths.append(fullpath)\nothers_image_paths[:2]","metadata":{"execution":{"iopub.status.busy":"2023-08-22T17:34:15.942051Z","iopub.execute_input":"2023-08-22T17:34:15.942375Z","iopub.status.idle":"2023-08-22T17:34:21.122840Z","shell.execute_reply.started":"2023-08-22T17:34:15.942344Z","shell.execute_reply":"2023-08-22T17:34:21.121845Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Number of Other Images:', len(others_image_paths))","metadata":{"execution":{"iopub.status.busy":"2023-08-22T17:34:21.126240Z","iopub.execute_input":"2023-08-22T17:34:21.126542Z","iopub.status.idle":"2023-08-22T17:34:21.132017Z","shell.execute_reply.started":"2023-08-22T17:34:21.126517Z","shell.execute_reply":"2023-08-22T17:34:21.130853Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Number of Monet Images:', len(others_image_paths))","metadata":{"execution":{"iopub.status.busy":"2023-08-22T17:34:21.133579Z","iopub.execute_input":"2023-08-22T17:34:21.133964Z","iopub.status.idle":"2023-08-22T17:34:21.146501Z","shell.execute_reply.started":"2023-08-22T17:34:21.133932Z","shell.execute_reply":"2023-08-22T17:34:21.145429Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"figure, ax = plt.subplots(nrows = 3,ncols = 2,figsize=(5,10))\nfor i, image_path in enumerate(monet_image_paths):\n    image = cv2.imread(image_path)\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) \n    try:\n        ax.ravel()[i].imshow(image)\n        ax.ravel()[i].set_axis_off()\n    except:\n        continue;\n        \nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-08-22T17:34:21.148026Z","iopub.execute_input":"2023-08-22T17:34:21.148685Z","iopub.status.idle":"2023-08-22T17:34:23.831807Z","shell.execute_reply.started":"2023-08-22T17:34:21.148654Z","shell.execute_reply":"2023-08-22T17:34:23.830927Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we have some more monet images above.","metadata":{}},{"cell_type":"code","source":"figure, ax = plt.subplots(nrows = 3,ncols = 2,figsize=(5,10))\nfor i, image_path in enumerate(others_image_paths):\n    image = cv2.imread(image_path)\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) \n    try:\n        ax.ravel()[i].imshow(image)\n        ax.ravel()[i].set_axis_off()\n    except:\n        continue;\n        \nplt.tight_layout()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-08-22T17:34:23.833147Z","iopub.execute_input":"2023-08-22T17:34:23.836438Z","iopub.status.idle":"2023-08-22T17:35:10.670588Z","shell.execute_reply.started":"2023-08-22T17:34:23.836403Z","shell.execute_reply":"2023-08-22T17:35:10.669769Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And here we have few more other images.\n\nAll these images are of shape of 256,256,3 that is height and width of 256 and RGB Channels. With basic EDA we have confirmed that Monet and Other Images are indeed visually distinguishable. Now we will create the CycleGAN model.","metadata":{}},{"cell_type":"markdown","source":"## 3. Model Creation","metadata":{}},{"cell_type":"markdown","source":"The course didn't provide any reading or other resources for CycleGAN however since it was asked as the assignment we have covered multiple articles, jupyter notebooks like Kaggle Tutorials, and videos to learn how to implement CycleGAN model.\n\nIn this part we will create a U-Net Architecture for the CycleGAN. U-Net is a popular deep-learning architecture for semantic segmentation, while it was used first for medical purposes now its used in vide range of applications and is a popular CNN architecture.\n\nWe will create downsample and upsample layers for the GycleGAN. In upsample we will use Conv2D transpose layers  to increase the height and width to the target shape and in downsample we wil use stride to reduce the height and width.\n\nWe will also be using an instance normalization instead of batch normalizations as many notebooks use this instead of normal BatchNormalization layer. We will also adopt this practice.","metadata":{}},{"cell_type":"code","source":"image_size = [256, 256]","metadata":{"execution":{"iopub.status.busy":"2023-08-22T17:35:10.671855Z","iopub.execute_input":"2023-08-22T17:35:10.672781Z","iopub.status.idle":"2023-08-22T17:35:10.677552Z","shell.execute_reply.started":"2023-08-22T17:35:10.672746Z","shell.execute_reply":"2023-08-22T17:35:10.676357Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#These formulas will allow us to read the tfrec files.\n\ndef decode_image(image):\n    image = tf.image.decode_jpeg(image, channels=3)\n    image = (tf.cast(image, tf.float32) / 127.5) - 1\n    image = tf.reshape(image, [*image_size, 3])\n    return image\n\ndef read_tfrecord(example):\n    tfrecord_format = {\n        \"image_name\": tf.io.FixedLenFeature([], tf.string),\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"target\": tf.io.FixedLenFeature([], tf.string)\n    }\n    example = tf.io.parse_single_example(example, tfrecord_format)\n    image = decode_image(example['image'])\n    return image","metadata":{"execution":{"iopub.status.busy":"2023-08-22T17:35:10.678990Z","iopub.execute_input":"2023-08-22T17:35:10.679998Z","iopub.status.idle":"2023-08-22T17:35:10.688810Z","shell.execute_reply.started":"2023-08-22T17:35:10.679967Z","shell.execute_reply":"2023-08-22T17:35:10.687815Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Here we will load the dataset and use Autotune to fetch the batch dataset more efficiently\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    dataset = tf.data.TFRecordDataset(filenames)\n    dataset = dataset.map(read_tfrecord, num_parallel_calls = AUTOTUNE)\n    return dataset","metadata":{"execution":{"iopub.status.busy":"2023-08-22T17:35:10.692954Z","iopub.execute_input":"2023-08-22T17:35:10.694283Z","iopub.status.idle":"2023-08-22T17:35:10.700138Z","shell.execute_reply.started":"2023-08-22T17:35:10.694245Z","shell.execute_reply":"2023-08-22T17:35:10.698928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"monet_df = load_dataset(monet_filenames, labeled=True).batch(1)\nphoto_df = load_dataset(photo_filenames, labeled=True).batch(1)","metadata":{"execution":{"iopub.status.busy":"2023-08-22T17:35:10.701640Z","iopub.execute_input":"2023-08-22T17:35:10.702645Z","iopub.status.idle":"2023-08-22T17:35:15.089495Z","shell.execute_reply.started":"2023-08-22T17:35:10.702613Z","shell.execute_reply":"2023-08-22T17:35:15.088554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we will plot the image directly from the batch dataset.","metadata":{}},{"cell_type":"code","source":"plt.subplot(121)\nplt.title('Monet')\nplt.imshow(next(iter(monet_df))[0]*0.6 + 0.5)\n\nplt.subplot(122)\nplt.title('Photo')\nplt.imshow(next(iter(photo_df))[0]*0.6 + 0.5)","metadata":{"execution":{"iopub.status.busy":"2023-08-22T17:35:15.090939Z","iopub.execute_input":"2023-08-22T17:35:15.091280Z","iopub.status.idle":"2023-08-22T17:35:16.111397Z","shell.execute_reply.started":"2023-08-22T17:35:15.091246Z","shell.execute_reply":"2023-08-22T17:35:16.107119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Batch is working properly as we can see images loading correctly above. Now we will make the downsample and upsample function. Both of them will output sequential models.","metadata":{}},{"cell_type":"code","source":"output_channels = 3\n\n#In downsample we will use Conv2D to reduce the size of the image\n\ndef downsample(filters, size, instancenorm = True):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    model = keras.Sequential()\n    model.add(tfl.Conv2D(filters = filters, kernel_size = size, strides = 2, padding='same',\n                             kernel_initializer = initializer, use_bias=False))\n\n    if instancenorm:\n        model.add(tfa.layers.InstanceNormalization(gamma_initializer = gamma_init))\n\n    model.add(tfl.LeakyReLU())\n\n    return model","metadata":{"execution":{"iopub.status.busy":"2023-08-22T17:35:16.112902Z","iopub.execute_input":"2023-08-22T17:35:16.113614Z","iopub.status.idle":"2023-08-22T17:35:16.121604Z","shell.execute_reply.started":"2023-08-22T17:35:16.113574Z","shell.execute_reply":"2023-08-22T17:35:16.120661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# In upsample we will use Conv2DTranspose to increase the size of the image, stride = 2 will \n# increase the dimension by twice\n\ndef upsample(filters, size, dropout = False):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    model = keras.Sequential()\n    model.add(tfl.Conv2DTranspose(filters = filters, kernel_size = size, strides = 2,\n                                      padding = 'same',\n                                      kernel_initializer = initializer,\n                                      use_bias = False))\n\n    model.add(tfa.layers.InstanceNormalization(gamma_initializer = gamma_init))\n\n    if dropout:\n        model.add(tfl.Dropout(0.5)) #We tried to change dropout to 0.2 but it was throwing an error\n\n    model.add(tfl.ReLU())\n\n    return model","metadata":{"execution":{"iopub.status.busy":"2023-08-22T17:35:16.123209Z","iopub.execute_input":"2023-08-22T17:35:16.123537Z","iopub.status.idle":"2023-08-22T17:35:16.133031Z","shell.execute_reply.started":"2023-08-22T17:35:16.123504Z","shell.execute_reply":"2023-08-22T17:35:16.132123Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we will create Generator model.","metadata":{}},{"cell_type":"code","source":"def Generator():\n    inputs = tfl.Input(shape=[256,256,3])\n\n    down_stack = [\n        downsample(filters = 64, size = 4, instancenorm = False), # Every Downstack layer has stride = 2\n        downsample(filters = 128, size = 4),\n        downsample(filters = 256, size = 4),\n        downsample(filters = 512, size = 4),\n        downsample(filters = 512, size = 4),\n        downsample(filters = 512, size = 4),\n        downsample(filters = 512, size = 4),\n        downsample(filters = 512, size = 4),\n    ]\n\n    up_stack = [\n        upsample(filters = 512, size = 4, dropout = True), # (bs, 2, 2, 1024)\n        upsample(filters = 512, size = 4, dropout = True), # (bs, 4, 4, 1024)\n        upsample(filters = 512, size = 4, dropout = True), # (bs, 8, 8, 1024)\n        upsample(filters = 512, size = 4), # (bs, 16, 16, 1024)\n        upsample(256, 4),\n        upsample(128, 4),\n        upsample(64, 4),\n    ]\n\n    initializer = tf.random_normal_initializer(0., 0.02)\n    last_layer = tfl.Conv2DTranspose(output_channels, kernel_size = 4, strides = 2, padding='same',\n                                  kernel_initializer = initializer,\n                                  activation = 'tanh')\n\n    x = inputs\n\n    # Downsampling through the model\n    skips = []\n    for down_layer in down_stack:\n        x = down_layer(x)\n        skips.append(x)\n\n    skips = reversed(skips[:-1])\n\n    # Upsampling and establishing the skip connections\n    for up, skip in zip(up_stack, skips):\n        x = up(x)\n        x = tfl.Concatenate()([x, skip])\n\n    x = last_layer(x)\n\n    return keras.Model(inputs = inputs, outputs = x)","metadata":{"execution":{"iopub.status.busy":"2023-08-22T17:35:16.134674Z","iopub.execute_input":"2023-08-22T17:35:16.135053Z","iopub.status.idle":"2023-08-22T17:35:16.148826Z","shell.execute_reply.started":"2023-08-22T17:35:16.135022Z","shell.execute_reply":"2023-08-22T17:35:16.147735Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we will create the Discrimintor.","metadata":{}},{"cell_type":"code","source":"def Discriminator():\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    x = tfl.Input(shape=[256, 256, 3])\n    \n    \n    #Down layers are Conv2D layers with their own initializers and leakyReLU activation\n    \n    down_layer_1 = downsample(filters = 64, size = 4,instancenorm = False)(x)\n    down_layer_2 = downsample(filters = 128, size = 4)(down_layer_1)\n    down_layer_3 = downsample(filters = 256, size = 4)(down_layer_2) \n\n    zero_pad1 = tfl.ZeroPadding2D(padding = 2)(down_layer_3)\n    conv_layer_1 = tfl.Conv2D(filters = 512, kernel_size = 4, strides = 2,\n                         kernel_initializer = initializer, use_bias=False)(zero_pad1)\n\n    norm_1 = tfa.layers.InstanceNormalization(gamma_initializer = gamma_init)(conv_layer_1)\n\n    leaky_relu_1 = tfl.LeakyReLU()(norm_1)\n\n    zero_pad2 = tfl.ZeroPadding2D()(leaky_relu_1)\n    \n    last_layer = tfl.Conv2D(filters = 1, kernel_size = 4, strides=1,\n                         kernel_initializer=initializer)(zero_pad2) \n\n    return keras.Model(inputs = x, outputs = last_layer)","metadata":{"execution":{"iopub.status.busy":"2023-08-22T17:35:16.150369Z","iopub.execute_input":"2023-08-22T17:35:16.150698Z","iopub.status.idle":"2023-08-22T17:35:16.164428Z","shell.execute_reply.started":"2023-08-22T17:35:16.150668Z","shell.execute_reply":"2023-08-22T17:35:16.163532Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n    monet_generator = Generator() # transforms photos to Monet-esque paintings\n    photo_generator = Generator() # transforms Monet paintings to be more like photos\n\n    monet_discriminator = Discriminator() # differentiates real Monet paintings and generated Monet paintings\n    photo_discriminator = Discriminator() # differentiates real photos and generated photos","metadata":{"execution":{"iopub.status.busy":"2023-08-22T17:35:16.165830Z","iopub.execute_input":"2023-08-22T17:35:16.166522Z","iopub.status.idle":"2023-08-22T17:35:19.298368Z","shell.execute_reply.started":"2023-08-22T17:35:16.166472Z","shell.execute_reply":"2023-08-22T17:35:19.297368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we will import the CycleGAN class from the Kaggle Tutorial Notebook, this will apply and train the U-Net Architecture with CycleGAN. ","metadata":{}},{"cell_type":"code","source":"# The CycleGAN class has been imported from the Kaggle Tutorial notebook\n# link: https://www.kaggle.com/code/amyjang/monet-cyclegan-tutorial/notebook#Build-the-generator\n\nclass CycleGan(keras.Model):\n    def __init__(\n        self,\n        monet_generator,\n        photo_generator,\n        monet_discriminator,\n        photo_discriminator,\n        lambda_cycle=10,\n    ):\n        super(CycleGan, self).__init__()\n        self.m_gen = monet_generator\n        self.p_gen = photo_generator\n        self.m_disc = monet_discriminator\n        self.p_disc = photo_discriminator\n        self.lambda_cycle = lambda_cycle\n        \n    def compile(\n        self,\n        m_gen_optimizer,\n        p_gen_optimizer,\n        m_disc_optimizer,\n        p_disc_optimizer,\n        gen_loss_fn,\n        disc_loss_fn,\n        cycle_loss_fn,\n        identity_loss_fn\n    ):\n        super(CycleGan, self).compile()\n        self.m_gen_optimizer = m_gen_optimizer\n        self.p_gen_optimizer = p_gen_optimizer\n        self.m_disc_optimizer = m_disc_optimizer\n        self.p_disc_optimizer = p_disc_optimizer\n        self.gen_loss_fn = gen_loss_fn\n        self.disc_loss_fn = disc_loss_fn\n        self.cycle_loss_fn = cycle_loss_fn\n        self.identity_loss_fn = identity_loss_fn\n        \n    def train_step(self, batch_data):\n        real_monet, real_photo = batch_data\n        \n        with tf.GradientTape(persistent=True) as tape:\n            # photo to monet back to photo\n            fake_monet = self.m_gen(real_photo, training=True)\n            cycled_photo = self.p_gen(fake_monet, training=True)\n\n            # monet to photo back to monet\n            fake_photo = self.p_gen(real_monet, training=True)\n            cycled_monet = self.m_gen(fake_photo, training=True)\n\n            # generating itself\n            same_monet = self.m_gen(real_monet, training=True)\n            same_photo = self.p_gen(real_photo, training=True)\n\n            # discriminator used to check, inputing real images\n            disc_real_monet = self.m_disc(real_monet, training=True)\n            disc_real_photo = self.p_disc(real_photo, training=True)\n\n            # discriminator used to check, inputing fake images\n            disc_fake_monet = self.m_disc(fake_monet, training=True)\n            disc_fake_photo = self.p_disc(fake_photo, training=True)\n\n            # evaluates generator loss\n            monet_gen_loss = self.gen_loss_fn(disc_fake_monet)\n            photo_gen_loss = self.gen_loss_fn(disc_fake_photo)\n\n            # evaluates total cycle consistency loss\n            total_cycle_loss = self.cycle_loss_fn(real_monet, cycled_monet, self.lambda_cycle) + self.cycle_loss_fn(real_photo, cycled_photo, self.lambda_cycle)\n\n            # evaluates total generator loss\n            total_monet_gen_loss = monet_gen_loss + total_cycle_loss + self.identity_loss_fn(real_monet, same_monet, self.lambda_cycle)\n            total_photo_gen_loss = photo_gen_loss + total_cycle_loss + self.identity_loss_fn(real_photo, same_photo, self.lambda_cycle)\n\n            # evaluates discriminator loss\n            monet_disc_loss = self.disc_loss_fn(disc_real_monet, disc_fake_monet)\n            photo_disc_loss = self.disc_loss_fn(disc_real_photo, disc_fake_photo)\n\n        # Calculate the gradients for generator and discriminator\n        monet_generator_gradients = tape.gradient(total_monet_gen_loss,\n                                                  self.m_gen.trainable_variables)\n        photo_generator_gradients = tape.gradient(total_photo_gen_loss,\n                                                  self.p_gen.trainable_variables)\n\n        monet_discriminator_gradients = tape.gradient(monet_disc_loss,\n                                                      self.m_disc.trainable_variables)\n        photo_discriminator_gradients = tape.gradient(photo_disc_loss,\n                                                      self.p_disc.trainable_variables)\n\n        # Apply the gradients to the optimizer\n        self.m_gen_optimizer.apply_gradients(zip(monet_generator_gradients,\n                                                 self.m_gen.trainable_variables))\n\n        self.p_gen_optimizer.apply_gradients(zip(photo_generator_gradients,\n                                                 self.p_gen.trainable_variables))\n\n        self.m_disc_optimizer.apply_gradients(zip(monet_discriminator_gradients,\n                                                  self.m_disc.trainable_variables))\n\n        self.p_disc_optimizer.apply_gradients(zip(photo_discriminator_gradients,\n                                                  self.p_disc.trainable_variables))\n        \n        return {\n            \"monet_gen_loss\": total_monet_gen_loss,\n            \"photo_gen_loss\": total_photo_gen_loss,\n            \"monet_disc_loss\": monet_disc_loss,\n            \"photo_disc_loss\": photo_disc_loss\n        }","metadata":{"execution":{"iopub.status.busy":"2023-08-22T14:27:26.162085Z","iopub.execute_input":"2023-08-22T14:27:26.162487Z","iopub.status.idle":"2023-08-22T14:27:26.181517Z","shell.execute_reply.started":"2023-08-22T14:27:26.162455Z","shell.execute_reply":"2023-08-22T14:27:26.180488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we will define our loss functions.\n\nThe discriminator loss function below compares real images to a matrix of 1s and fake images to a matrix of 0s. The perfect discriminator will output all 1s for real images and all 0s for fake images. The discriminator loss outputs the average of the real and generated loss.","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    def discriminator_loss(real, generated):\n        \n        cross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits = True, \n                                                           reduction = tf.keras.losses.Reduction.NONE)\n        \n        # Loss for discriminator includes two parts, first where it tries to flag real image as real\n        # Second part where it try to minimize flagging generated images as real\n        \n        real_loss = cross_entropy(tf.ones_like(real), real)\n        fake_loss = cross_entropy(tf.zeros_like(generated), generated)\n\n        total_loss = real_loss + fake_loss\n\n        return total_loss*0.5","metadata":{"execution":{"iopub.status.busy":"2023-08-22T14:27:27.418201Z","iopub.execute_input":"2023-08-22T14:27:27.418585Z","iopub.status.idle":"2023-08-22T14:27:27.426033Z","shell.execute_reply.started":"2023-08-22T14:27:27.418553Z","shell.execute_reply":"2023-08-22T14:27:27.424843Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The generator aims to fool the discriminator into thinking that the fake generated image is real. The perfect generator will have the discriminator output only 1s. Thus, it compares the generated image to a matrix of 1s to find the loss.","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    def generator_loss(generated):\n        \n        cross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits = True, \n                                                           reduction = tf.keras.losses.Reduction.NONE)\n        #For generator we have only one term for loss\n        \n        return cross_entropy(tf.ones_like(generated), generated)","metadata":{"execution":{"iopub.status.busy":"2023-08-22T14:27:28.364396Z","iopub.execute_input":"2023-08-22T14:27:28.364804Z","iopub.status.idle":"2023-08-22T14:27:28.371792Z","shell.execute_reply.started":"2023-08-22T14:27:28.364753Z","shell.execute_reply":"2023-08-22T14:27:28.370465Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since we are using CycleGAN we have to add specific losses for it too. \n\nWe want our original photo and the twice-transformed photo to be similar to one another. Cycle Loss here will be the average of the differences between these images.","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    def calc_cycle_loss(real_image, cycled_image, LAMBDA):\n\n        loss1 = tf.reduce_mean(tf.abs(real_image - cycled_image))\n        \n        return LAMBDA*loss1","metadata":{"execution":{"iopub.status.busy":"2023-08-22T14:27:29.264264Z","iopub.execute_input":"2023-08-22T14:27:29.264656Z","iopub.status.idle":"2023-08-22T14:27:29.271295Z","shell.execute_reply.started":"2023-08-22T14:27:29.264622Z","shell.execute_reply":"2023-08-22T14:27:29.270015Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The identity loss below compares the input with the output of the generator.","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    def identity_loss(real_image, same_image, LAMBDA):\n        \n        loss = tf.reduce_mean(tf.abs(real_image - same_image))\n        \n        return LAMBDA*0.5*loss","metadata":{"execution":{"iopub.status.busy":"2023-08-22T14:27:30.146871Z","iopub.execute_input":"2023-08-22T14:27:30.147266Z","iopub.status.idle":"2023-08-22T14:27:30.152789Z","shell.execute_reply.started":"2023-08-22T14:27:30.147233Z","shell.execute_reply":"2023-08-22T14:27:30.151608Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4. Model Training and Hyperparameter Tuning","metadata":{}},{"cell_type":"markdown","source":"Now we have train the model and perform Hyperparamter Tuning.\n\nWe have performed Hyperparameter Tuning Manually for this project. We have tried two different learning rate and beta_1, mainly (0.0001, 0.9) and (0.0002, 0.5). We have chosen (0.0002, 0.5) as optimum parameter as its more stable and a little faster.\n\nWe have also tuned the Number of epochs at 5, 10, and 25. We have chosen 25 as the final parameter for our model, epoch for 5 result has been reflected here, please uncomment the code and try for 10 epochs for yourself if you want to validate the results, but it is to be noted that with Kaggle GPU it takes around 50 minutes to train 10 epochs.","metadata":{}},{"cell_type":"code","source":"#Hyperparameter Tuning\n#Here we manually checked the Learning rate of 0.0001 and beta if 0.9. While it was working okay,\n#Learning rate of 0.0002 and beta_1 of 0.5 provided a big faster learning. Please uncomment to try yourself\n#It may take hours to try these hyperparamters so please note time left in your environemnt\n\n# with strategy.scope():\n    \n#     monet_generator_optimizer = tf.keras.optimizers.Adam(learning_rate = 0.0001, beta_1 = 0.9)\n#     photo_generator_optimizer = tf.keras.optimizers.Adam(learning_rate = 0.0001, beta_1 = 0.9)\n\n#     monet_discriminator_optimizer = tf.keras.optimizers.Adam(learning_rate = 0.0001, beta_1 = 0.9)\n#     photo_discriminator_optimizer = tf.keras.optimizers.Adam(learning_rate = 0.0001, beta_1 = 0.)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n    \n    monet_generator_optimizer = tf.keras.optimizers.Adam(learning_rate = 0.0002, beta_1 = 0.5)\n    photo_generator_optimizer = tf.keras.optimizers.Adam(learning_rate = 0.0002, beta_1 = 0.5)\n\n    monet_discriminator_optimizer = tf.keras.optimizers.Adam(learning_rate = 0.0002, beta_1 = 0.5)\n    photo_discriminator_optimizer = tf.keras.optimizers.Adam(learning_rate = 0.0002, beta_1 = 0.5)","metadata":{"execution":{"iopub.status.busy":"2023-08-22T14:27:33.442808Z","iopub.execute_input":"2023-08-22T14:27:33.443946Z","iopub.status.idle":"2023-08-22T14:27:33.456061Z","shell.execute_reply.started":"2023-08-22T14:27:33.443902Z","shell.execute_reply":"2023-08-22T14:27:33.455053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n    cycle_gan_model = CycleGan(\n        monet_generator, photo_generator, monet_discriminator, photo_discriminator\n    )\n\n    cycle_gan_model.compile(\n        m_gen_optimizer = monet_generator_optimizer,\n        p_gen_optimizer = photo_generator_optimizer,\n        m_disc_optimizer = monet_discriminator_optimizer,\n        p_disc_optimizer = photo_discriminator_optimizer,\n        gen_loss_fn = generator_loss,\n        disc_loss_fn = discriminator_loss,\n        cycle_loss_fn = calc_cycle_loss,\n        identity_loss_fn = identity_loss\n    )","metadata":{"execution":{"iopub.status.busy":"2023-08-22T14:44:07.252146Z","iopub.execute_input":"2023-08-22T14:44:07.252530Z","iopub.status.idle":"2023-08-22T14:44:07.273003Z","shell.execute_reply.started":"2023-08-22T14:44:07.252497Z","shell.execute_reply":"2023-08-22T14:44:07.271941Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# cycle_gan_model.fit(\n#     tf.data.Dataset.zip((monet_df, photo_df)),\n#     epochs = 5\n# )","metadata":{"execution":{"iopub.status.busy":"2023-08-22T14:27:37.023309Z","iopub.execute_input":"2023-08-22T14:27:37.023696Z","iopub.status.idle":"2023-08-22T14:41:10.642420Z","shell.execute_reply.started":"2023-08-22T14:27:37.023665Z","shell.execute_reply":"2023-08-22T14:41:10.641360Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#This warning is unnecessary, please ignore as model will train properly.\n\nhistory_gan = cycle_gan_model.fit(\n    tf.data.Dataset.zip((monet_df, photo_df)),\n    epochs = 25\n)\nwarnings.simplefilter('ignore')","metadata":{"execution":{"iopub.status.busy":"2023-08-22T14:44:49.474105Z","iopub.execute_input":"2023-08-22T14:44:49.474506Z","iopub.status.idle":"2023-08-22T15:42:38.773825Z","shell.execute_reply.started":"2023-08-22T14:44:49.474473Z","shell.execute_reply":"2023-08-22T15:42:38.772935Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5. Model Evaluation","metadata":{}},{"cell_type":"markdown","source":"Below are the best results obtained by training different epochs of CycleGAN model:\n\nmonet_gen_loss: 2.6568, epoch = 25\nphoto_gen_loss: 2.7744, epoch = 24\nmonet_disc_loss: 0.6126, epoch = 1\nphoto_disc_loss: 0.5985, epoch = 5\n\nFrom this observation we can state that the Generator is converging at 24/25th Epoch and its performing best at those epochs. However Discriminator has converged much earlier, it has converged around 5th Epoch and is not changing much after this. \n\nWe have chosen 25 Epochs as Parameter to let Generator learn properly, we can do early stopping at 5th or 10th Epoch but it will not train to its optimum level. After 25th Epoch we may get slightly lower generator loss but we also also see an increase in discriminator loss. Reader can try to run it and check themselves in hyperparameter tuning part.","metadata":{}},{"cell_type":"markdown","source":"Now we will apply transfer learning to some of the images and observe how CycleGAN is performing.","metadata":{}},{"cell_type":"code","source":"_, ax = plt.subplots(5, 2, figsize=(6, 20))\nfor i, img in enumerate(photo_df.take(5)):\n    prediction = monet_generator(img, training=False)[0].numpy()\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    img = (img[0] * 127.5 + 127.5).numpy().astype(np.uint8)\n\n    ax[i, 0].imshow(img)\n    ax[i, 1].imshow(prediction)\n    ax[i, 0].set_title(\"Input Photo\")\n    ax[i, 1].set_title(\"Monet-esque\")\n    ax[i, 0].axis(\"off\")\n    ax[i, 1].axis(\"off\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-08-22T15:56:13.432616Z","iopub.execute_input":"2023-08-22T15:56:13.433010Z","iopub.status.idle":"2023-08-22T15:56:15.164556Z","shell.execute_reply.started":"2023-08-22T15:56:13.432974Z","shell.execute_reply":"2023-08-22T15:56:15.163668Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Above we have converted the first 5 images in the batch dataset into Monet versions. And we can observe some issues too. While our model is able to transfer Monet style almost perfectly to the blue skies and even do a good job to green palate forests, it was not able to a good job at all in higher contrast images or images in low contrast as we can see from 4th and 5th image here.\n\n4th Image has a smaller colour gamut overall, and 5th one has wider colour gamut, and it failed in transfering Monet styles to both of them. We can","metadata":{}},{"cell_type":"markdown","source":"Before wrapping up the model evaluation we will submit the result and get the MiFID metric, for this project smaller MiFID will indicate better generated images.","metadata":{}},{"cell_type":"code","source":"import PIL\n! mkdir ../images","metadata":{"execution":{"iopub.status.busy":"2023-08-22T15:57:33.314983Z","iopub.execute_input":"2023-08-22T15:57:33.315399Z","iopub.status.idle":"2023-08-22T15:57:34.496866Z","shell.execute_reply.started":"2023-08-22T15:57:33.315366Z","shell.execute_reply":"2023-08-22T15:57:34.495171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"i = 1\nfor img in photo_df:\n    prediction = monet_generator(img, training=False)[0].numpy()\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    im = PIL.Image.fromarray(prediction)\n    im.save(\"../images/\" + str(i) + \".jpg\")\n    i += 1","metadata":{"execution":{"iopub.status.busy":"2023-08-22T15:57:35.353100Z","iopub.execute_input":"2023-08-22T15:57:35.353582Z","iopub.status.idle":"2023-08-22T16:07:37.050984Z","shell.execute_reply.started":"2023-08-22T15:57:35.353537Z","shell.execute_reply":"2023-08-22T16:07:37.049720Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import shutil\nshutil.make_archive(\"/kaggle/working/images\", 'zip', \"/kaggle/images\")","metadata":{"execution":{"iopub.status.busy":"2023-08-22T16:07:41.745597Z","iopub.execute_input":"2023-08-22T16:07:41.746311Z","iopub.status.idle":"2023-08-22T16:07:46.325519Z","shell.execute_reply.started":"2023-08-22T16:07:41.746270Z","shell.execute_reply":"2023-08-22T16:07:46.324515Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We got the score of 59.23486 and are ranked 40 on the leaderboard as of the writing. \n\nWe can try to improve this score by using more complex CNN Architectures and even fine tuning the CycleGAN even more. However as the first attempt to train a CycleGAN and use it for style transfer and creating Monet images out of not monet ones is a surrel experience on its own. This shows the power of AI at even a beginner level and the power of the community as a whole which helped us to learn and practice CycleGAN from the scratch.","metadata":{}}],"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}}