{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.10","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Table of Contents\n\n1. Data\n2. Model\n3. Loss\n4. Training Loop\n5. Hyperparameter Tuning\n6. Training and validation of CycleGAN\n7. Conclusion\n8. References","metadata":{}},{"cell_type":"markdown","source":"# Project Topic\n\nThis is a notebook for [**I’m Something of a Painter Myself**](https://www.kaggle.com/competitions/gan-getting-started). The task of this project is to convert landscape images into monet style images. It is unpaired image to image translation task which requires CycleGAN training. \n\n**target**\n\nTarget of this project is to achieve **FID < 100** in Kaggle Score. In addition, I would like to visually check how my cycle gan works. Therefore, following objectives should be achieved too. \n\n* Generated images look like monet painting.\n* Gycled images look similar to original images (small cycle loss)\n* Landscape images do not collapse when they go through Generator(Monet --> Landscape), and vise versa. (small identity loss)\n","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd \nimport cv2\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport os\nimport time\nimport gc\n\nimport tensorflow as tf\nfrom keras import backend as K\n\nfrom tensorflow.keras import layers\nfrom tensorflow.keras import Model\nfrom tensorflow.keras.models import Sequential\nimport tensorflow_addons as tfa","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-08-30T06:03:42.892951Z","iopub.execute_input":"2023-08-30T06:03:42.893402Z","iopub.status.idle":"2023-08-30T06:03:53.227082Z","shell.execute_reply.started":"2023-08-30T06:03:42.893360Z","shell.execute_reply":"2023-08-30T06:03:53.224876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Data","metadata":{}},{"cell_type":"markdown","source":"There are 300 Monet images and 7038 Landscape images. Size of those images are 256x256x3.\n\n**data source**\n\nThe original data is available in kaggle competition website.\n\nhttps://www.kaggle.com/competitions/gan-getting-started/data\n\n","metadata":{}},{"cell_type":"code","source":"dir_monet = \"/kaggle/input/gan-getting-started/monet_jpg/\"\ndir_photo = \"/kaggle/input/gan-getting-started/photo_jpg/\"\nfiles_monet = os.listdir(dir_monet)\nfiles_photo = os.listdir(dir_photo)\n\nn1 = len(files_monet)\nn2 = len(files_photo)","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:03:53.229032Z","iopub.execute_input":"2023-08-30T06:03:53.230420Z","iopub.status.idle":"2023-08-30T06:03:54.072222Z","shell.execute_reply.started":"2023-08-30T06:03:53.230380Z","shell.execute_reply":"2023-08-30T06:03:54.071164Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Number of Monet images = \", n1)\nprint(\"Number of Landscape images = \", n2)","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:03:54.074504Z","iopub.execute_input":"2023-08-30T06:03:54.075220Z","iopub.status.idle":"2023-08-30T06:03:54.080674Z","shell.execute_reply.started":"2023-08-30T06:03:54.075186Z","shell.execute_reply":"2023-08-30T06:03:54.079710Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Functions \n\n`convert_img` is the function to convert normalized image to original scale. `plot_photo` visualize 36 images. `plot_OneLine` plots 6 images in one line. ","metadata":{}},{"cell_type":"code","source":"# Function to convert normalized image to original scale\ndef convert_img(image):\n    \n    img2 = ((np.array(image) + 1)*127.5).astype(int)\n    img2[img2 > 255] = 255\n    img2[img2 < 0] = 0\n    \n    img2 = img2.astype(np.uint8)\n    \n    return img2","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:03:54.083589Z","iopub.execute_input":"2023-08-30T06:03:54.084242Z","iopub.status.idle":"2023-08-30T06:03:54.090612Z","shell.execute_reply.started":"2023-08-30T06:03:54.084206Z","shell.execute_reply":"2023-08-30T06:03:54.089475Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_photo(image):\n    \n    img2 = ((np.array(image) + 1)*127.5).astype(int)\n    img2[img2 > 255] = 255\n    img2[img2 < 0] = 0\n\n    \n    fig, ax = plt.subplots(6,6, figsize = (15,15))\n    \n    N = 36\n    for k in range(N):\n        i =  int(k/6)\n        j = k % 6\n    \n        ax[i,j].imshow(img2[k])\n        ax[i,j].tick_params(left = False, right = False , labelleft = False , labelbottom = False, bottom = False)\n    \n\ndef plot_OneLine(image, title, rescale = False):\n    \n    img2 = ((np.array(image) + 1)*127.5).astype(int)\n    img2[img2 > 255] = 255\n    img2[img2 < 0] = 0\n\n    \n    fig, ax = plt.subplots(1,6, figsize = (15,2.5))\n    \n    N = 6\n    for k in range(N):\n    \n        ax[k].imshow(img2[k])\n        ax[k].tick_params(left = False, right = False , labelleft = False , labelbottom = False, bottom = False)\n        \n    \n    ax[0].set_title(title)","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:03:54.093277Z","iopub.execute_input":"2023-08-30T06:03:54.093631Z","iopub.status.idle":"2023-08-30T06:03:54.106372Z","shell.execute_reply.started":"2023-08-30T06:03:54.093562Z","shell.execute_reply":"2023-08-30T06:03:54.105342Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Reading images\n\nI usually use OpenCV for reading images. Since OpenCV reads images in `BGR` scale, it shall be transformed into `RGB`. Thus, `cv2.imread` function is followed by `[:,:,::-1]`. Only 100 Landscape images are read at the beginning to reduce RAM usage. They are used for only validation during cycle gan training.\n\n","metadata":{}},{"cell_type":"code","source":"n2 = 100","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:03:54.108039Z","iopub.execute_input":"2023-08-30T06:03:54.108392Z","iopub.status.idle":"2023-08-30T06:03:54.115975Z","shell.execute_reply.started":"2023-08-30T06:03:54.108359Z","shell.execute_reply":"2023-08-30T06:03:54.114997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"M_data = np.zeros((n1,256,256,3)).astype(np.uint8)\nF_data = np.zeros((n2,256,256,3)).astype(np.uint8)","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:03:54.117604Z","iopub.execute_input":"2023-08-30T06:03:54.117947Z","iopub.status.idle":"2023-08-30T06:03:54.376486Z","shell.execute_reply.started":"2023-08-30T06:03:54.117917Z","shell.execute_reply":"2023-08-30T06:03:54.375473Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for i in range(n1):\n    M_data[i] = cv2.imread(dir_monet + files_monet[i])[:,:,::-1]\n    \nfor i in range(n2):\n    F_data[i] = cv2.imread(dir_photo + files_photo[i])[:,:,::-1]","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:03:54.377839Z","iopub.execute_input":"2023-08-30T06:03:54.380467Z","iopub.status.idle":"2023-08-30T06:03:57.630506Z","shell.execute_reply.started":"2023-08-30T06:03:54.380428Z","shell.execute_reply":"2023-08-30T06:03:57.629563Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Images are normalized by following code.","metadata":{"execution":{"iopub.status.busy":"2023-07-26T13:31:28.250894Z","iopub.execute_input":"2023-07-26T13:31:28.259436Z","iopub.status.idle":"2023-07-26T13:31:28.264508Z","shell.execute_reply.started":"2023-07-26T13:31:28.259399Z","shell.execute_reply":"2023-07-26T13:31:28.263438Z"}}},{"cell_type":"code","source":"M_data = M_data/(255/2) - 1\nF_data = F_data/(255/2) - 1","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:03:57.632184Z","iopub.execute_input":"2023-08-30T06:03:57.632543Z","iopub.status.idle":"2023-08-30T06:03:57.900710Z","shell.execute_reply.started":"2023-08-30T06:03:57.632488Z","shell.execute_reply":"2023-08-30T06:03:57.899687Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"real_A = M_data[0:36]\nreal_B = F_data[0:36]","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:03:57.905425Z","iopub.execute_input":"2023-08-30T06:03:57.905755Z","iopub.status.idle":"2023-08-30T06:03:57.910341Z","shell.execute_reply.started":"2023-08-30T06:03:57.905727Z","shell.execute_reply":"2023-08-30T06:03:57.909408Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Monet images**","metadata":{}},{"cell_type":"code","source":"plot_photo(M_data)","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:03:57.911726Z","iopub.execute_input":"2023-08-30T06:03:57.912769Z","iopub.status.idle":"2023-08-30T06:04:05.662425Z","shell.execute_reply.started":"2023-08-30T06:03:57.912736Z","shell.execute_reply":"2023-08-30T06:04:05.661132Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Landscape images**","metadata":{}},{"cell_type":"code","source":"plot_photo(F_data)","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:04:05.663803Z","iopub.execute_input":"2023-08-30T06:04:05.664131Z","iopub.status.idle":"2023-08-30T06:04:12.906548Z","shell.execute_reply.started":"2023-08-30T06:04:05.664100Z","shell.execute_reply":"2023-08-30T06:04:12.904900Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gc.collect()","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:04:12.915080Z","iopub.execute_input":"2023-08-30T06:04:12.915395Z","iopub.status.idle":"2023-08-30T06:04:13.210677Z","shell.execute_reply.started":"2023-08-30T06:04:12.915368Z","shell.execute_reply":"2023-08-30T06:04:13.209714Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Model\n## 2.1 GAN Components\n\nIn cycle gan, there are some common components in generators and discriminators. It is convenient to define those components as functions.\n\n`FeatureMapBlock` is feature extractor which does not change image size. It is also used at the end of generator to improve quality of  generated images. `ContractingBlock` consists of **Conv2D**, **InstanceNormalization** and **activation fuction** which reduces image size just like typical CNN. `ResidualBlock` has two layers of **Conv2D** and **InstanceNormalization**. Original Input is skipped to the end of the block which minitage dead neuron problem. `ExandingBlock` consists of **Conv2DTranspose** which increases image size. It is followed by InstanceNormalization and activation function. For CycleGan model, InstanceNormalization is selected instead of BatchNormalization, because batch size = 1 in CycleGan training.\n","metadata":{}},{"cell_type":"code","source":"def FeatureMapBlock(channel, X, final = False):\n    \n    if final:\n        X = layers.Conv2D(channel, kernel_size = 7, padding = \"same\", activation = \"tanh\")(X)\n    \n    else:\n        X = layers.Conv2D(channel, kernel_size = 7, padding = \"same\")(X)\n        X = tfa.layers.InstanceNormalization()(X)\n        X = layers.LeakyReLU(alpha=0.2)(X)\n        \n    return X","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:04:13.212088Z","iopub.execute_input":"2023-08-30T06:04:13.212667Z","iopub.status.idle":"2023-08-30T06:04:13.226453Z","shell.execute_reply.started":"2023-08-30T06:04:13.212635Z","shell.execute_reply":"2023-08-30T06:04:13.225575Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def ContractingBlock(channel, X, relu = True, ksize = 3, use_bn = True):\n    \n    X =  layers.Conv2D(channel, kernel_size = ksize, strides = 2, padding = \"same\")(X)\n    \n    if use_bn:\n        X = tfa.layers.InstanceNormalization()(X)\n    \n    if relu:\n        X = layers.ReLU()(X)\n    else:\n        X = layers.LeakyReLU(alpha=0.2)(X)\n        \n    return X","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:04:13.229948Z","iopub.execute_input":"2023-08-30T06:04:13.230249Z","iopub.status.idle":"2023-08-30T06:04:13.238496Z","shell.execute_reply.started":"2023-08-30T06:04:13.230217Z","shell.execute_reply":"2023-08-30T06:04:13.237576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def ResidualBlock(channel, X, relu = True):\n    \n    X_original = X # skip connection\n    \n    X = layers.Conv2D(channel, kernel_size = 3, padding = \"same\")(X)\n    X = tfa.layers.InstanceNormalization()(X)\n    \n    if relu:\n        X = layers.ReLU()(X)\n    else:\n        X = layers.LeakyReLU(alpha=0.2)(X)\n        \n    X = layers.Conv2D(channel, kernel_size = 3, padding = \"same\")(X)\n    X = tfa.layers.InstanceNormalization()(X)\n    \n    \n    return X + X_original\n    ","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:04:13.241457Z","iopub.execute_input":"2023-08-30T06:04:13.241761Z","iopub.status.idle":"2023-08-30T06:04:13.249325Z","shell.execute_reply.started":"2023-08-30T06:04:13.241738Z","shell.execute_reply":"2023-08-30T06:04:13.248423Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def ExpandingBlock(channel, X, ksize):\n    \n    X = layers.Conv2DTranspose(channel, kernel_size=ksize, strides=2, padding=\"SAME\")(X)\n    X = tfa.layers.InstanceNormalization()(X)\n    X = layers.LeakyReLU(alpha=0.2)(X)\n    \n    return X","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:04:13.250931Z","iopub.execute_input":"2023-08-30T06:04:13.251246Z","iopub.status.idle":"2023-08-30T06:04:13.259288Z","shell.execute_reply.started":"2023-08-30T06:04:13.251218Z","shell.execute_reply":"2023-08-30T06:04:13.258424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.2. Generator\n\nGenerator consists of following elements:\n\n* FeatureMapBlock\n* two or three ContractingBlocks\n* several ResidualBlocks\n* ExpandingBlocks\n* Final Layer (FeatureMapBlock with `tanh`)\n\nThis structure is common for the both of generators AB and BA.","metadata":{}},{"cell_type":"code","source":"def Generator():\n    Input =  layers.Input(shape=(256, 256, 3))\n    \n    channel = 64\n\n    X = FeatureMapBlock(channel, Input, final = False) #\n    KS = 5\n    X = ContractingBlock(channel*2, X, False, KS)#128\n    X = ContractingBlock(channel*4, X, False, KS)#64\n    X = ContractingBlock(channel*8, X, False, KS)#32\n    X = ContractingBlock(channel*8, X, False, KS)#16\n    #X = ContractingBlock(channel*8, X, False, 3)#8\n    #X = ContractingBlock(channel*8, X, False, 3)#4\n    \n    X = ResidualBlock(channel*8, X, False)\n    X = ResidualBlock(channel*8, X, False)\n    X = ResidualBlock(channel*8, X, False)\n    \n    KS = 5\n    #X = ExpandingBlock(channel*8, X, 5)#8\n    #X = ExpandingBlock(channel*8, X, 5)#16\n    X = ExpandingBlock(channel*8, X, KS)#32\n    X = ExpandingBlock(channel*4, X, KS)#64\n    X = ExpandingBlock(channel*2, X, KS)#128\n    X = ExpandingBlock(channel, X, KS)#256\n    \n    X =  FeatureMapBlock(3, X, final = True)\n    \n    model = Model(inputs = Input, outputs = X)\n    \n    return model\n    ","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:04:13.260693Z","iopub.execute_input":"2023-08-30T06:04:13.261171Z","iopub.status.idle":"2023-08-30T06:04:13.271754Z","shell.execute_reply.started":"2023-08-30T06:04:13.261136Z","shell.execute_reply":"2023-08-30T06:04:13.270781Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen_AB = Generator()\ngen_BA = Generator()\ngen_AB.summary()","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:04:13.273218Z","iopub.execute_input":"2023-08-30T06:04:13.273585Z","iopub.status.idle":"2023-08-30T06:04:17.964576Z","shell.execute_reply.started":"2023-08-30T06:04:13.273554Z","shell.execute_reply":"2023-08-30T06:04:17.963768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.keras.utils.plot_model(gen_AB)","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:04:17.965712Z","iopub.execute_input":"2023-08-30T06:04:17.966066Z","iopub.status.idle":"2023-08-30T06:04:18.362411Z","shell.execute_reply.started":"2023-08-30T06:04:17.966032Z","shell.execute_reply":"2023-08-30T06:04:18.361501Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.3 Discriminator\n\nDiscriminator consists of following elements:\n\n* FeatureMapBlock\n* Three ContractingBlocks\n* Final Layer: Conv2D of size 1 without activation\n\nThis structure is common for both discriminator A and B.","metadata":{}},{"cell_type":"code","source":"def Discriminator():\n\n    Input =  layers.Input(shape=(256, 256, 3))\n    \n    channel = 64\n    \n    X = FeatureMapBlock(channel, Input, False) #\n    X = ContractingBlock(channel*2, X, False, 4) #128\n    X = ContractingBlock(channel*4, X, False, 4) #64\n    X = ContractingBlock(channel*4, X, False, 4) #32\n    \n    X = layers.Conv2D(1, kernel_size = 1, padding = \"same\")(X) #32\n    \n    model = Model(inputs = Input, outputs = X)\n    \n    return model","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:04:18.363944Z","iopub.execute_input":"2023-08-30T06:04:18.364558Z","iopub.status.idle":"2023-08-30T06:04:18.375252Z","shell.execute_reply.started":"2023-08-30T06:04:18.364510Z","shell.execute_reply":"2023-08-30T06:04:18.372098Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"disc_A = Discriminator()\ndisc_B = Discriminator()\ndisc_A.summary()","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:04:18.376529Z","iopub.execute_input":"2023-08-30T06:04:18.376994Z","iopub.status.idle":"2023-08-30T06:04:18.763610Z","shell.execute_reply.started":"2023-08-30T06:04:18.376947Z","shell.execute_reply":"2023-08-30T06:04:18.762802Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"tf.keras.utils.plot_model(disc_A)\n","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:04:18.764739Z","iopub.execute_input":"2023-08-30T06:04:18.765195Z","iopub.status.idle":"2023-08-30T06:04:18.874456Z","shell.execute_reply.started":"2023-08-30T06:04:18.765160Z","shell.execute_reply":"2023-08-30T06:04:18.873544Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Loss\n## 3.1 Discriminator Loss\n\n`get_disc_loss` calculate adversarial loss of a discriminator. It takes prediction (by discriminator) of real and fake images. It calculate loss based on adv_criterion. ","metadata":{}},{"cell_type":"code","source":"def get_disc_loss(real_pred, fake_pred, adv_criterion):\n    \n    ones_label = tf.ones_like(real_pred)\n    zeros_label = tf.zeros_like(fake_pred)\n    \n    disc_loss = (adv_criterion(ones_label, real_pred) + adv_criterion(zeros_label, fake_pred))/2\n    \n    return disc_loss","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:04:18.875771Z","iopub.execute_input":"2023-08-30T06:04:18.876126Z","iopub.status.idle":"2023-08-30T06:04:18.882533Z","shell.execute_reply.started":"2023-08-30T06:04:18.876092Z","shell.execute_reply":"2023-08-30T06:04:18.881378Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.2 Generator Loss\n\nThere are three types of generator losses. First, `get_gen_adversarial_loss` calculate adv loss of a generator. It takes prediction (by discriminator) of fake images. True label is always 1 in this case. Then loss is calculated based on **adv_criterion**\n\n`get_cycle_consistency_loss` compare cycled image with the original one. Typically MAE is selected as cycle_criterion. `get_identity_loss` has similar criterion. It calculates gap between original image with generated image. For example, if original is Monet, then we need to make sure that Monet image does not change after going through generator_Landscape_Monet (gen_BA). ","metadata":{}},{"cell_type":"code","source":"def get_gen_adversarial_loss(fake_pred, adv_criterion):\n    \n    #label is always one\n    ones_label = tf.ones_like(fake_pred)\n    \n    loss = adv_criterion(ones_label, fake_pred)\n    \n    return loss\n    ","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:04:18.883726Z","iopub.execute_input":"2023-08-30T06:04:18.884725Z","iopub.status.idle":"2023-08-30T06:04:18.894467Z","shell.execute_reply.started":"2023-08-30T06:04:18.884689Z","shell.execute_reply":"2023-08-30T06:04:18.893571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_cycle_consistency_loss(real_X, cycle_X, cycle_criterion):\n    \n    cycle_loss = cycle_criterion(real_X, cycle_X)\n    \n    return cycle_loss ","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:04:18.895768Z","iopub.execute_input":"2023-08-30T06:04:18.896251Z","iopub.status.idle":"2023-08-30T06:04:18.907181Z","shell.execute_reply.started":"2023-08-30T06:04:18.896219Z","shell.execute_reply":"2023-08-30T06:04:18.906350Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_identity_loss(real_X, identity_X, identity_criterion):\n    \n    identity_loss = identity_criterion(real_X, identity_X)\n\n    return identity_loss #, identity_X\n","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:04:18.908555Z","iopub.execute_input":"2023-08-30T06:04:18.909267Z","iopub.status.idle":"2023-08-30T06:04:18.916901Z","shell.execute_reply.started":"2023-08-30T06:04:18.909236Z","shell.execute_reply":"2023-08-30T06:04:18.916054Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Training Loop\n\n","metadata":{}},{"cell_type":"markdown","source":"**Optimizers**\n\nTraining cycle gan means training four models: generator AB and BA, discriminator A and B. Threfore, it needs four optimizers. \n","metadata":{}},{"cell_type":"code","source":"optimizer_disc_A = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\noptimizer_disc_B = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\noptimizer_gen_AB = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\noptimizer_gen_BA = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:04:18.923931Z","iopub.execute_input":"2023-08-30T06:04:18.924425Z","iopub.status.idle":"2023-08-30T06:04:18.938251Z","shell.execute_reply.started":"2023-08-30T06:04:18.924401Z","shell.execute_reply":"2023-08-30T06:04:18.937379Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"To monitor cycle gan training progress, losses will be stored in the following lists.","metadata":{}},{"cell_type":"code","source":"loss_discA = []\nloss_discB = []\n\nloss_gen_AB = []\nloss_gen_BA = []\n\n\nloss_gen_adv_AB = []\nloss_gen_adv_BA = []\n\nloss_gen_cycle_ABA = []\nloss_gen_cycle_BAB = []\n\nloss_gen_iden_AB = []\nloss_gen_iden_BA = []","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:04:18.939569Z","iopub.execute_input":"2023-08-30T06:04:18.940132Z","iopub.status.idle":"2023-08-30T06:04:18.945821Z","shell.execute_reply.started":"2023-08-30T06:04:18.940102Z","shell.execute_reply":"2023-08-30T06:04:18.944709Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Loss functions**\n\nFor adversarial loss, BCE or MSE are selected. For other criterion, MAE is the choice. ","metadata":{}},{"cell_type":"code","source":"BCE_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True)\nMSE_loss = tf.keras.losses.MeanSquaredError()\nMAE_loss =  tf.keras.losses.MeanAbsoluteError()","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:04:18.947263Z","iopub.execute_input":"2023-08-30T06:04:18.947910Z","iopub.status.idle":"2023-08-30T06:04:18.955745Z","shell.execute_reply.started":"2023-08-30T06:04:18.947879Z","shell.execute_reply":"2023-08-30T06:04:18.954785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"adv_loss_function = BCE_loss","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:04:18.957167Z","iopub.execute_input":"2023-08-30T06:04:18.957837Z","iopub.status.idle":"2023-08-30T06:04:18.965161Z","shell.execute_reply.started":"2023-08-30T06:04:18.957806Z","shell.execute_reply":"2023-08-30T06:04:18.964474Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`train_CycleGan` trains discriminator and generator separately. To keep balance between generator and discriminator, it does not train disciminator when loss is less than threshold. After training discriminators, it traing generators.\n\n`Train_OneEpoch` function contains `train_CycleGan` in `for loop`. It reads Monet image and Landscape image, then input those images into training. It also has horizontal flip of Monet image as data augmentation, because there are only 300 of Monet images. Everytime of training, it randomly select Monet and Landscape. Thus, combination of two images changes by every training routine. ","metadata":{}},{"cell_type":"code","source":"@tf.function\ndef train_CycleGan(real_A, real_B):\n    \n    lambda_cycle = 10\n    lambda_identity = 2\n    lambda_adv = 1\n    \n    # Threshold of training discriminator. It skips training discriminator when loss <  threshold\n    disc_train_threshold = 0.35\n    \n    \n    with tf.GradientTape() as tape:\n    \n        #fake \n        fake_A =  gen_BA(real_B)\n        \n        #pred by disc\n        real_A_pred = disc_A(real_A)\n        fake_A_pred = disc_A(fake_A)\n        \n        #disc loss    \n        disc_loss_A = get_disc_loss(real_A_pred, fake_A_pred, adv_loss_function)### MSE or BCE\n        #loss_discA.append(disc_loss_A)\n        \n        \n    #if disc_loss_A > disc_train_threshold:\n\n    gradients = tape.gradient(disc_loss_A, disc_A.trainable_weights)\n    optimizer_disc_A.apply_gradients(zip(gradients, disc_A.trainable_weights))\n    \n    \n    with tf.GradientTape() as tape:\n    \n        #fake \n        fake_B = gen_AB(real_A)\n        \n        #pred by disc\n        real_B_pred = disc_B(real_B)\n        fake_B_pred = disc_B(fake_B)\n        \n        #disc loss    \n        disc_loss_B = get_disc_loss(real_B_pred, fake_B_pred, adv_loss_function)### MSE or BCE\n        #loss_discB.append(disc_loss_B)\n        \n    #if disc_loss_B > disc_train_threshold:\n\n        \n    gradients = tape.gradient(disc_loss_B, disc_B.trainable_weights)    \n    optimizer_disc_B.apply_gradients(zip(gradients, disc_B.trainable_weights))\n        \n\n    with tf.GradientTape(persistent=True) as tape:\n        \n        #fake \n        fake_A =  gen_BA(real_B)\n        fake_B = gen_AB(real_A)\n        \n        #cycle\n        cycle_A = gen_BA(fake_B)\n        cycle_B = gen_AB(fake_A)\n\n        #identity \n        iden_B = gen_AB(real_B)\n        iden_A = gen_BA(real_A)\n\n        #pred by disc\n        fake_B_pred = disc_B(fake_B)\n        fake_A_pred = disc_A(fake_A)\n\n        \n        #gen loss\n        gen_adversarial_loss_AB = get_gen_adversarial_loss(fake_B_pred, adv_loss_function)### MSE or BCE\n        gen_adversarial_loss_BA = get_gen_adversarial_loss(fake_A_pred, adv_loss_function)### MSE or BCE\n        \n        #loss_gen_adv_AB.append(gen_adversarial_loss_AB)\n        #loss_gen_adv_BA.append(gen_adversarial_loss_BA)\n\n        #cycle loss\n        cycle_loss_A = get_cycle_consistency_loss(real_A, cycle_A, MAE_loss)\n        cycle_loss_B = get_cycle_consistency_loss(real_B, cycle_B, MAE_loss) \n        total_cycle_loss = cycle_loss_A + cycle_loss_B\n        \n        #loss_gen_cycle_ABA.append(cycle_loss_A)\n        #loss_gen_cycle_BAB.append(cycle_loss_B)\n        \n        #identity loss\n        identity_loss_AB = get_identity_loss(real_B, iden_B, MAE_loss)\n        identity_loss_BA = get_identity_loss(real_A, iden_A, MAE_loss)\n        \n        #loss_gen_iden_AB.append(identity_loss_AB)\n        #loss_gen_iden_BA.append(identity_loss_BA)\n\n        #gen loss\n        #gen_loss = gen_adversarial_loss*lambda_adv + cycle_loss*lambda_cycle + identity_loss*lambda_identity\n        gen_loss_AB = gen_adversarial_loss_AB*lambda_adv + total_cycle_loss*lambda_cycle + identity_loss_AB*lambda_identity\n        gen_loss_BA = gen_adversarial_loss_BA*lambda_adv + total_cycle_loss*lambda_cycle + identity_loss_BA*lambda_identity\n        #loss_gen_AB.append(gen_loss_AB)\n        #loss_gen_BA.append(gen_loss_BA)\n    \n    gradients = tape.gradient(gen_loss_AB, gen_AB.trainable_weights)\n    optimizer_gen_AB.apply_gradients(zip(gradients, gen_AB.trainable_weights))\n    \n    gradients = tape.gradient(gen_loss_BA, gen_BA.trainable_weights)\n    optimizer_gen_BA.apply_gradients(zip(gradients, gen_BA.trainable_weights))    \n\n    del tape\n    gc.collect()\n    \n    #return disc_loss_A, disc_loss_B, total_cycle_loss, identity_loss_AB, identity_loss_BA, gen_loss_AB, gen_loss_BA","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:04:18.966564Z","iopub.execute_input":"2023-08-30T06:04:18.967131Z","iopub.status.idle":"2023-08-30T06:04:18.983549Z","shell.execute_reply.started":"2023-08-30T06:04:18.967099Z","shell.execute_reply":"2023-08-30T06:04:18.982590Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def read_img(dir_name, file_name, convert = False):\n    \n    img = cv2.imread(dir_name + file_name)[:,:,::-1]\n    \n    if convert:\n        img = img/(255/2) - 1\n    \n    return img","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:04:18.985054Z","iopub.execute_input":"2023-08-30T06:04:18.985363Z","iopub.status.idle":"2023-08-30T06:04:18.997293Z","shell.execute_reply.started":"2023-08-30T06:04:18.985333Z","shell.execute_reply":"2023-08-30T06:04:18.996400Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def Train_OneEpoch(EPOCH):\n    \n    time1 = time.time()\n    print(\"EPOCH \", EPOCH)\n    count = 0\n\n    np.random.seed(EPOCH)\n\n    m_idx = np.arange(n1)\n    np.random.shuffle(m_idx)\n\n    f_idx = np.arange(len(files_photo))\n    np.random.shuffle(f_idx)\n\n\n    for i in range(n1):\n        F_data = read_img(dir_photo, files_photo[f_idx[i]], convert = True)\n\n        M_data2 = M_data[m_idx[i]]\n\n        #Horizontal Flip of Monet image\n        if np.random.rand(1)[0] > 0.5:\n            train_CycleGan(M_data2[:,::-1,:].reshape(1,256,256,3), F_data.reshape(1,256,256,3))\n        else:\n            train_CycleGan(M_data2.reshape(1,256,256,3), F_data.reshape(1,256,256,3))\n\n\n        if (i+1) % 100 == 0:\n            print(i+1)\n\n\n\n    time2 = time.time()\n\n    time3 = np.round(time2 - time1)\n\n    print(\"time \", time3, \"sec\")\n    ","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:04:18.998586Z","iopub.execute_input":"2023-08-30T06:04:18.999118Z","iopub.status.idle":"2023-08-30T06:04:19.009708Z","shell.execute_reply.started":"2023-08-30T06:04:18.999086Z","shell.execute_reply":"2023-08-30T06:04:19.008894Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`check_output` visualize training progress after each epoch. It shows original Landscape image, converted image, cycled image, and identity image.","metadata":{}},{"cell_type":"code","source":"def check_output(EPOCH, plot_line, plot_all = False):\n    \n    real_A = M_data[0:36]\n    real_B = F_data[0:36]\n\n    fake_B = gen_AB(real_A)\n    fake_A = gen_BA(real_B)\n\n    cycle_A = gen_BA(fake_B)\n    identity_A = gen_BA(real_A)\n\n    cycle_B = gen_AB(fake_A)\n    identity_B = gen_AB(real_B)\n\n\n    tag = \"_\" + str(EPOCH)\n\n\n    #np.save(\"real_A\", convert_img(real_A))\n    np.save(\"real_B\", convert_img(real_B))\n\n    #np.save(\"fake_B\" +tag, convert_img(fake_B))\n    #np.save(\"cycle_A\" +tag, convert_img(cycle_A))\n    #np.save(\"identity_A\" +tag, convert_img(identity_A))\n    \n    np.save(\"fake_A\" +tag, convert_img(fake_A))\n    np.save(\"cycle_B\" +tag, convert_img(cycle_B))\n    np.save(\"identity_B\" +tag, convert_img(identity_B))\n\n    tag = \", epoch \" + str(EPOCH)\n\n    if plot_all:\n        #plot_photo(fake_B)\n        plot_photo(fake_A)\n        \n\n    if plot_line:\n\n        #plot_OneLine(real_A, \"real_A\")\n        #plot_OneLine(fake_B, \"fake_B \" + tag)\n        #plot_OneLine(cycle_A, \"cycle_A \" + tag)\n        #plot_OneLine(identity_A, \"identity_A \" + tag)\n\n        plot_OneLine(real_B, \"real_B\")\n        plot_OneLine(fake_A, \"fake_A \" + tag)\n        plot_OneLine(cycle_B, \"cycle_B \" + tag)\n        plot_OneLine(identity_B, \"identity_B \" + tag)          \n            \n    ","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:04:19.010940Z","iopub.execute_input":"2023-08-30T06:04:19.011550Z","iopub.status.idle":"2023-08-30T06:04:19.023245Z","shell.execute_reply.started":"2023-08-30T06:04:19.011501Z","shell.execute_reply":"2023-08-30T06:04:19.022508Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def Train_Loop(EPOCH, EPOCH_prev):\n    \n    plot_line = True\n    \n    for i in range(EPOCH):\n    \n        Train_OneEpoch(i + EPOCH_prev)\n        \n        plot_line = False\n        if (i + EPOCH_prev) % 5 == 0:   \n            plot_line = True\n            \n            check_output(i + EPOCH_prev, plot_line, False)\n","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:04:19.024604Z","iopub.execute_input":"2023-08-30T06:04:19.025239Z","iopub.status.idle":"2023-08-30T06:04:19.036540Z","shell.execute_reply.started":"2023-08-30T06:04:19.025202Z","shell.execute_reply":"2023-08-30T06:04:19.035661Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Hypterparameter Tuning\n\nFollowing hyperparameters are tuned in previous versions of this notebook. They are also based on Coursera's GAN Specialization [1] and CycleGan paper [2].\n\n#### **Activation Function**\n\nI selected `LeakyRelu` for all activation function in generator and discriminator. GAN Specialization[1] uses `Relu` for Generator's ContractingBlock and ResidualBlock, but `Relu` made dark parts of generated images into completely black (**blackout**), because it cut off small values. In contrast, `LeakyRelu` preserved information in dark zones, and generated image did not have blackouts. \n\n#### **Number of ConractingBlocks in generator**\n\nWhen number of ContractingBlock is 2, training was stable and fast. Increasing ContractingBlocks did not improve result. It ended up with blurred image with larger training epochs.\n\n#### **Number of ResidualBlocks in generator**\n\nNumber of ResidualBlock is 3 in final model. Even when there is more than 3 ResidualBlocks, it got the similar result (appearance) of generated image. \n\n#### **kernel size of discrimninator**\n\nKernel size = 5 was more stable than 4 which I have seen in GAN Specialization. \n\n\n#### **learning rate**\n\nLearning rate is set at 0.0002 as per CycleGAN paper[2]. Increasing learning rate did not improve the result. \n\n#### **Adversarial Loss Function**\n\n**GAN Specialization [1] recommends MSE** for adversarial loss to avoid vanishing gradient. However, BCE works slightly better than MSE for my GAN architecture of this project. Hence, **BCE is selected for adv loss**. Furthermore, BCE is easier to monitor occourence of overfit.\n\n","metadata":{}},{"cell_type":"markdown","source":"# 6. Training and validation of Cycle GAN\n\nDuring training, CycleGAN is validated visually by plotting fake image, cycled image, and identity image from perspective of image structure, color, texture, etc. Furthermore, train loss is plotted to make sure the mode collapse (over fitting) does not occour.","metadata":{}},{"cell_type":"code","source":"N_EPOCH = 10","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:04:19.038827Z","iopub.execute_input":"2023-08-30T06:04:19.039121Z","iopub.status.idle":"2023-08-30T06:04:19.047327Z","shell.execute_reply.started":"2023-08-30T06:04:19.039098Z","shell.execute_reply":"2023-08-30T06:04:19.046321Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Train_Loop(N_EPOCH, 0)","metadata":{"execution":{"iopub.status.busy":"2023-08-30T06:04:19.048672Z","iopub.execute_input":"2023-08-30T06:04:19.049483Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Train_Loop(N_EPOCH, N_EPOCH*1)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Train_Loop(N_EPOCH, N_EPOCH*2)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Train_Loop(N_EPOCH, N_EPOCH*3)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Train_Loop(N_EPOCH, N_EPOCH*4)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Train_Loop(N_EPOCH, N_EPOCH*5)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Train_Loop(N_EPOCH, N_EPOCH*6)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Train_Loop(N_EPOCH, N_EPOCH*7)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Train_Loop(N_EPOCH, N_EPOCH*8)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gen_AB.save_weights(\"gen_AB/ckpt1\")\ngen_BA.save_weights(\"gen_BA/ckpt2\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"disc_A.save_weights(\"disc_A/ckpt3\")\ndisc_B.save_weights(\"disc_B/ckpt4\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Training Loss","metadata":{}},{"cell_type":"code","source":"def plot_loss():\n    \n    fig, ax = plt.subplots(3, figsize = (10, 12))\n\n    #cut off the first 40 train loss\n    start = 0\n    ax[0].plot(loss_discA[start:], label  =\"disc_A\", alpha = 0.7, lw = 0.6)\n    ax[0].plot(loss_discB[start:], label = \"disc_B\", alpha = 0.7, lw = 0.6)\n    ax[0].plot(loss_gen_adv_AB[start:], label = \"gen_AB\", alpha = 0.7, lw = 0.6)\n    ax[0].plot(loss_gen_adv_BA[start:], label = \"gen_BA\", alpha = 0.7, lw = 0.6)\n    ax[0].set_title(\"Adversarial Loss\")\n    ax[0].set_ylim(0,2)\n\n\n    #ax[1].plot(np.array(loss_gen_adv), label = \"adv\", alpha = 0.8)\n    ax[1].plot(loss_gen_cycle_ABA[start:], label  =\"cycle_ABA\", alpha = 0.7, lw = 0.6)\n    ax[1].plot(loss_gen_cycle_BAB[start:], label  =\"cycle_BAB\", alpha = 0.7, lw = 0.6)\n    ax[1].plot(loss_gen_iden_AB[start:], label = \"identity_AB\", alpha = 0.7, lw = 0.6)\n    ax[1].plot(loss_gen_iden_BA[start:], label = \"identity_BA\", alpha = 0.7, lw = 0.6)\n    ax[1].set_title(\"Generator Loss\")\n\n    ax[2].plot(loss_gen_AB[start:], label = \"gen_AB\", alpha = 0.7, lw = 0.6)\n    ax[2].plot(loss_gen_BA[start:], label = \"gen_BA\", alpha = 0.7, lw = 0.6)\n    ax[2].set_title(\"Generator Loss (total)\")\n\n    for i in range(3):\n        ax[i].grid()\n        ax[i].legend()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Final Output \n\n36 samples of final output image are shown below. It has different color from the original image. ","metadata":{}},{"cell_type":"code","source":"check_output(N_EPOCH, False, True)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Write File","metadata":{}},{"cell_type":"code","source":"import PIL\n! mkdir ../images\n\ndef submit_files():\n\n    n3 = len(files_photo)\n    \n    \n    for i in range(n3):\n\n        real_F = cv2.imread(dir_photo + files_photo[i])[:,:,::-1].reshape(1,256,256,3)\n\n        real_F = real_F/(255/2) - 1\n\n        fake_A = np.array(gen_BA(real_F))[0]\n\n        prediction = convert_img(fake_A).astype(np.uint8)\n\n        im = PIL.Image.fromarray(prediction, mode=\"RGB\")\n        im.save(\"../images/\" + str(i) + \".jpg\")\n    ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submit_files()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import shutil\nshutil.make_archive(\"/kaggle/working/images\", 'zip', \"/kaggle/images\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Conclusion\n\nThis model got **FID 74.5 in version 24 of this notebook**. It achieved the target of this project FID < 100. The final output has faded color which is different from original images. Texture looks like slightly rougher than original, yet does not look like Monet. However, I could check that CycleGAN worked as it should be:\n\n* Color of cycled images are closer to original image compared with fake images.\n* Landscape images did not collapse after going through generator Monet - Landsape, just some color difference is observed. \n* Adversarial loss are balanced between generator and discriminator.\n* Total generator loss kept getting smaller\n\n**other takeaways**\n\nTo implement CycleGAN, I had to study in GAN Specialization[1]. Its problem was that all assignemnts are in Pytorch although I prefer coding in tensorflow. Therefore, I had to re-code everything in tensorflow which was good experience to verify my understanding. ","metadata":{}},{"cell_type":"markdown","source":"# References\n\n[1] Apply Generative Adversarial Networks (GANs). Deeplearning.ai \n\nhttps://www.coursera.org/learn/apply-generative-adversarial-networks-gans?specialization=generative-adversarial-networks-gans\n\n[2] Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks (Zhu, Park, Isola, and Efros, 2020): \n\nhttps://arxiv.org/abs/1703.10593","metadata":{}}],"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}}