{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# AIM\n
WARNING : This notebook assumes that you are familiar with the basic concepts of GANs and only focuses on the implementation of CycleGAN. The authors of the CycleGAN paper claimed that their model can \"paint\" photos in Monet style. They have also shown great results backing their claim. We will implement the CycleGAN architecture as mentioned in the paper itself (that is, use ResNet as generator in place of U-net).
\n\n# Paper can be found [here](https://arxiv.org/pdf/1703.10593.pdf)\n# Dataset has been taken from [Kaggle Competition](https://www.kaggle.com/c/gan-getting-started)","metadata":{}},{"cell_type":"code","source":"import cv2\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport tensorflow_addons as tfa\nfrom tensorflow import keras\nimport tensorflow as tf","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:48:33.241394Z","iopub.execute_input":"2021-06-05T07:48:33.241782Z","iopub.status.idle":"2021-06-05T07:48:38.382392Z","shell.execute_reply.started":"2021-06-05T07:48:33.241702Z","shell.execute_reply":"2021-06-05T07:48:38.381581Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Performance","metadata":{}},{"cell_type":"code","source":"img = cv2.imread('../input/another-image/Capture.PNG')\nimg = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\nplt.figure(figsize = (20,20))\nplt.imshow(img)\nplt.axis('off')","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:48:38.383847Z","iopub.execute_input":"2021-06-05T07:48:38.384193Z","iopub.status.idle":"2021-06-05T07:48:39.044606Z","shell.execute_reply.started":"2021-06-05T07:48:38.384155Z","shell.execute_reply":"2021-06-05T07:48:39.043642Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model\n## Definition\n
CycleGAN was first introduced in August, 2020. This was model could perform image-to-image translation without paired data. Paired data means having a source and corresponding target image. First we need to know what is Neural style transfer. Neural style transfer is an optimization technique that takes in two sets of images - source image and style reference image (usually a painting), and draws the source image in the fashion of style reference image. And that is what we aim to do as well.
\n\n## Methodology\n
CycleGAN uses the same adverasrial loss (minimax between Discriminator and Generator), used in vanilla GANs. However, as you might have guessed this is not enough. Strictly speaking in terms of Neural style transfer, what we want is to retain the basic structure of source image, but style it in terms of Monet paintings. However, what the model will try to do is completely convert the image to Monet painting.
\n\n#### *Confused !! Look at the example below,*","metadata":{}},{"cell_type":"code","source":"eg = ['../input/examples/poppies.jpg', '../input/examples/hith-eiffel-tower-istock_000016468972large-2.jpg']\n\nimg1 = cv2.cvtColor(cv2.resize(cv2.imread(eg[0]), (256,256)), cv2.COLOR_BGR2RGB)\nimg2 = cv2.cvtColor(cv2.resize(cv2.imread(eg[1]), (256,256)), cv2.COLOR_BGR2RGB)\nplt.figure(figsize = (20,30))\nplt.subplot(1,2,1)\nplt.imshow(img2)\nplt.axis('off')\nplt.title('Eiffel tower Photo')\n\nplt.subplot(1,2,2)\nplt.imshow(img1)\nplt.axis('off')\nplt.title('Poppy field Painting by Claude Monet')","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:48:39.046413Z","iopub.execute_input":"2021-06-05T07:48:39.046813Z","iopub.status.idle":"2021-06-05T07:48:39.632702Z","shell.execute_reply.started":"2021-06-05T07:48:39.046778Z","shell.execute_reply":"2021-06-05T07:48:39.631726Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
Let's say, your image in X domain is the image of Eiffel tower and the corresponding image in Y domain is Poppy Field painting. If you only consider the adversarial loss to minimize, then the Generator will aim to make the two images indistinguishable, and you will end up with the painting of poppy field. However, what we want is a painting of Eiffel tower painted by Monet. I hope you understand the difference.
\n\n## Cyclic Consistency\n
To handle the above issue, we add another loss function called cyclic consistency. What we do is generate an image using the Generator G and then feed the generated image to another Generator F whose aim is to regenerate the original image. In continuation with the above example, G takes in the photo of Eiffel tower and outputs the Poppy Field painting. Then this painting is fed back to F which tries to output original photo of Eiffel tower. What is the loss in all this ? Now, if we only focus on G, there can be an additional L1 loss between the regenerated photo of Eiffel tower and the original photograph (which will surely not be the same before training.) How does that help ? This will prevent G from blindly \"transforming the image completely\" and with enough training only the style of the target image will incorporated to the source image. (Painting of Eiffel tower by Monet)
","metadata":{}},{"cell_type":"code","source":"img = cv2.cvtColor(cv2.imread('../input/loss-diagram/Capture.PNG'), cv2.COLOR_BGR2RGB)\nplt.figure(figsize = (30,30))\nplt.imshow(img)\nplt.axis('off')","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:48:39.634342Z","iopub.execute_input":"2021-06-05T07:48:39.634733Z","iopub.status.idle":"2021-06-05T07:48:39.883868Z","shell.execute_reply.started":"2021-06-05T07:48:39.634697Z","shell.execute_reply":"2021-06-05T07:48:39.883096Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
The generators G and F are same as explained above. The discriminator Dy aims to distinguish between actual painting y and generated painting G(x). The discriminator Dx aims to distinguish between actual photo x and regenerated photo F(y). As you can see in the 2nd and 3rd pictures, there is cycle-consistency loss that calcultes L1 distance between original image and regenerated image.
","metadata":{}},{"cell_type":"markdown","source":"# Create Dataset","metadata":{}},{"cell_type":"code","source":"import os\nfrom PIL import Image","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:48:39.886352Z","iopub.execute_input":"2021-06-05T07:48:39.886678Z","iopub.status.idle":"2021-06-05T07:48:39.892263Z","shell.execute_reply.started":"2021-06-05T07:48:39.886643Z","shell.execute_reply":"2021-06-05T07:48:39.891289Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"paths = ['../input/gan-getting-started/photo_jpg/',\n '../input/gan-getting-started/monet_jpg/']","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:48:39.893867Z","iopub.execute_input":"2021-06-05T07:48:39.894286Z","iopub.status.idle":"2021-06-05T07:48:39.899785Z","shell.execute_reply.started":"2021-06-05T07:48:39.894248Z","shell.execute_reply":"2021-06-05T07:48:39.898706Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from keras.preprocessing.image import img_to_array","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:48:39.901487Z","iopub.execute_input":"2021-06-05T07:48:39.902128Z","iopub.status.idle":"2021-06-05T07:48:39.953939Z","shell.execute_reply.started":"2021-06-05T07:48:39.901947Z","shell.execute_reply":"2021-06-05T07:48:39.953241Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_img (path) :\n \n x = []\n for img_path in os.listdir(path) :\n x.append(img_to_array(Image.fromarray(cv2.cvtColor(cv2.imread(\n os.path.join(path,img_path)), cv2.COLOR_BGR2RGB))))\n \n x = np.array(x)\n x/= 255.0\n return x","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:48:39.957240Z","iopub.execute_input":"2021-06-05T07:48:39.957564Z","iopub.status.idle":"2021-06-05T07:48:39.962689Z","shell.execute_reply.started":"2021-06-05T07:48:39.957536Z","shell.execute_reply":"2021-06-05T07:48:39.961667Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X = get_img(paths[0])\ny = get_img(paths[1])","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:48:39.964967Z","iopub.execute_input":"2021-06-05T07:48:39.965435Z","iopub.status.idle":"2021-06-05T07:49:39.567016Z","shell.execute_reply.started":"2021-06-05T07:48:39.965397Z","shell.execute_reply":"2021-06-05T07:49:39.566113Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Dataset overview and Visualization","metadata":{}},{"cell_type":"code","source":"print(X.shape[1:])\nprint(y.shape[1:])","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:49:39.568397Z","iopub.execute_input":"2021-06-05T07:49:39.568945Z","iopub.status.idle":"2021-06-05T07:49:39.574916Z","shell.execute_reply.started":"2021-06-05T07:49:39.568908Z","shell.execute_reply":"2021-06-05T07:49:39.573932Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(X.min())\nprint(X.max())","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:49:39.576615Z","iopub.execute_input":"2021-06-05T07:49:39.577263Z","iopub.status.idle":"2021-06-05T07:49:40.615763Z","shell.execute_reply.started":"2021-06-05T07:49:39.577210Z","shell.execute_reply":"2021-06-05T07:49:40.608866Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(y.min())\nprint(y.max())","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:49:40.617219Z","iopub.execute_input":"2021-06-05T07:49:40.617953Z","iopub.status.idle":"2021-06-05T07:49:40.672385Z","shell.execute_reply.started":"2021-06-05T07:49:40.617911Z","shell.execute_reply":"2021-06-05T07:49:40.671344Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (10,50))\n\ni = 0\n\nwhile i < 16 :\n \n plt.subplot(8,2,i+1)\n plt.imshow(X[i])\n plt.axis('off')\n plt.title('Photo images')\n \n plt.subplot(8,2,i+2)\n plt.imshow(y[i])\n plt.axis('off')\n plt.title('Monet images')\n \n i += 2","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:49:40.673701Z","iopub.execute_input":"2021-06-05T07:49:40.674229Z","iopub.status.idle":"2021-06-05T07:49:42.464926Z","shell.execute_reply.started":"2021-06-05T07:49:40.674193Z","shell.execute_reply":"2021-06-05T07:49:42.464010Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train validation split","metadata":{}},{"cell_type":"code","source":"photo_images = X[:300]\nmonet_images = y","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:49:42.466138Z","iopub.execute_input":"2021-06-05T07:49:42.466616Z","iopub.status.idle":"2021-06-05T07:49:42.469992Z","shell.execute_reply.started":"2021-06-05T07:49:42.466579Z","shell.execute_reply":"2021-06-05T07:49:42.469283Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(photo_images.shape)\nprint(monet_images.shape)","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:49:42.471086Z","iopub.execute_input":"2021-06-05T07:49:42.471588Z","iopub.status.idle":"2021-06-05T07:49:42.481601Z","shell.execute_reply.started":"2021-06-05T07:49:42.471551Z","shell.execute_reply":"2021-06-05T07:49:42.480676Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
What matters in only the training data, because all of the photo images will be converted to Monet style paintings during testing phase. Now we must convert the train set to tf.dataset, with a buffer size of 1000 and batch size of 1 (Stochastic gradient descent applied)
","metadata":{}},{"cell_type":"code","source":"train_dataset = tf.data.Dataset.from_tensor_slices((photo_images, monet_images))\ntrain_dataset = train_dataset.shuffle(1000).batch(1)","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:49:42.482939Z","iopub.execute_input":"2021-06-05T07:49:42.483442Z","iopub.status.idle":"2021-06-05T07:49:44.438269Z","shell.execute_reply.started":"2021-06-05T07:49:42.483406Z","shell.execute_reply":"2021-06-05T07:49:44.437448Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f'Number of training samples : {len(train_dataset)}.')","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:49:44.439562Z","iopub.execute_input":"2021-06-05T07:49:44.439895Z","iopub.status.idle":"2021-06-05T07:49:44.445077Z","shell.execute_reply.started":"2021-06-05T07:49:44.439860Z","shell.execute_reply":"2021-06-05T07:49:44.444161Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize = (10,50))\n\nfor n, (photo, monet) in train_dataset.enumerate() :\n plt.subplot(1,2,1)\n plt.imshow(photo[0])\n plt.title('Photo Image')\n plt.axis('off')\n \n plt.subplot(1,2,2)\n plt.imshow(monet[0])\n plt.title('Monet Image')\n plt.axis('off')\n \n if n == 10 :\n break\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:49:44.446586Z","iopub.execute_input":"2021-06-05T07:49:44.447329Z","iopub.status.idle":"2021-06-05T07:49:46.105288Z","shell.execute_reply.started":"2021-06-05T07:49:44.447290Z","shell.execute_reply":"2021-06-05T07:49:46.104362Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Architecture\n## Discriminator (PatchGAN)\n
\n\n![img](https://miro.medium.com/max/1050/1*46CddTc5JwkFW_pQb4nGZQ.png)\n\n
The discriminator is a PatchGAN model, where each output in the arrays refers to a 70x70 overlapping patch in the input image. The model uses the principle of effectice receptive field, where a certain number of pixels in the input image or a patch can be mapped to a single output in the array. The output values are between 0 and 1, and this tells the probability that a given patch in the input image is real or fake. The output for all cells can be averaged to get the probability for the entire image. Which patch is represented by which output value ? This can be learnt by backtracking from the output image to the input image, tracing back the receptive fields.
","metadata":{}},{"cell_type":"code","source":"from keras.models import Model\nfrom keras.layers import Input\nfrom keras.layers import Conv2D\nfrom keras.layers import Dropout\nfrom keras.layers import LeakyReLU\nfrom keras.layers import Activation\nfrom keras.layers import Concatenate\nfrom keras.layers import ZeroPadding2D\nfrom keras.layers import Conv2DTranspose\nfrom keras.initializers import RandomNormal","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:49:46.106803Z","iopub.execute_input":"2021-06-05T07:49:46.107192Z","iopub.status.idle":"2021-06-05T07:49:46.112522Z","shell.execute_reply.started":"2021-06-05T07:49:46.107143Z","shell.execute_reply":"2021-06-05T07:49:46.111621Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"init = RandomNormal(0., 0.02)","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:49:46.113965Z","iopub.execute_input":"2021-06-05T07:49:46.114628Z","iopub.status.idle":"2021-06-05T07:49:46.122994Z","shell.execute_reply.started":"2021-06-05T07:49:46.114586Z","shell.execute_reply":"2021-06-05T07:49:46.121932Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def dis () :\n \n src = Input((256, 256, 3,))\n \n conv = Conv2D(2**6, (4,4), strides = 2, padding = 'same', kernel_initializer = init, use_bias = False) (src)\n leak = LeakyReLU(0.2)(conv)\n \n conv = Conv2D(2**7, (4,4), strides = 2, padding = 'same', kernel_initializer = init, use_bias = False)(leak)\n norm = tfa.layers.InstanceNormalization(gamma_initializer = init)(conv)\n leak = LeakyReLU(0.2)(norm)\n \n conv = Conv2D(2**8, (4,4), strides = 2, padding = 'same', kernel_initializer = init, use_bias = False)(leak)\n norm = tfa.layers.InstanceNormalization(gamma_initializer = init)(conv)\n leak = LeakyReLU(0.2)(norm)\n \n zero = ZeroPadding2D()(leak)\n \n conv = Conv2D(2**9, (4,4), strides = 1, padding = 'valid',kernel_initializer = init, use_bias = False)(zero)\n norm = tfa.layers.InstanceNormalization(gamma_initializer = init)(conv)\n leak = LeakyReLU(0.2)(norm)\n \n zero = ZeroPadding2D()(leak)\n \n conv = Conv2D(2**0, (4,4), strides = 1, padding = 'valid',kernel_initializer = init, use_bias = False)(zero)\n \n return Model(inputs = src, outputs = conv)","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:49:46.124472Z","iopub.execute_input":"2021-06-05T07:49:46.124885Z","iopub.status.idle":"2021-06-05T07:49:46.137614Z","shell.execute_reply.started":"2021-06-05T07:49:46.124842Z","shell.execute_reply":"2021-06-05T07:49:46.136566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Dx = dis()\nDy = dis()","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:49:46.139049Z","iopub.execute_input":"2021-06-05T07:49:46.139476Z","iopub.status.idle":"2021-06-05T07:49:46.838724Z","shell.execute_reply.started":"2021-06-05T07:49:46.139436Z","shell.execute_reply":"2021-06-05T07:49:46.837858Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"keras.utils.plot_model(Dx, './dis.png', show_shapes = True, dpi = 64)","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:49:46.842416Z","iopub.execute_input":"2021-06-05T07:49:46.842702Z","iopub.status.idle":"2021-06-05T07:49:47.312259Z","shell.execute_reply.started":"2021-06-05T07:49:46.842676Z","shell.execute_reply":"2021-06-05T07:49:47.311446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Generator (U-net)\n\n
\n\n![img](https://miro.medium.com/max/1050/1*lvXoKMHoPJMKpKK7keZMEA.png)\n\n
We will be using U-net architecture for the Generator. In the paper, the authors used a residual network, but it did not work for me. The U-net architecture consists of two parts - Encoder and Decoder. In the Encoder, the image is downsampled till the bottleneck layer to a size of 1x1 and then, in the Decoder part, it is upsampled from bottleneck layer to the output layer. The main idea behind this model, is that in the encoder path, the model learns what features are in the image and in the decoder part, it learns where these features are in the image.
","metadata":{}},{"cell_type":"code","source":"def gen () :\n \n src = Input((256, 256, 3,))\n \n conv_064_0 = Conv2D(2**6, (4,4), strides = 2, padding = 'same', kernel_initializer = init, use_bias = False)(src)\n leak_064_0 = LeakyReLU(0.2)(conv_064_0)\n \n conv_128_0 = Conv2D(2**7, (4,4), strides = 2, padding = 'same', kernel_initializer = init, use_bias = False)(leak_064_0)\n norm_128_0 = tfa.layers.InstanceNormalization(gamma_initializer = init)(conv_128_0)\n leak_128_0 = LeakyReLU(0.2)(norm_128_0)\n \n conv_256_0 = Conv2D(2**8, (4,4), strides = 2, padding = 'same', kernel_initializer = init, use_bias = False)(leak_128_0)\n norm_256_0 = tfa.layers.InstanceNormalization(gamma_initializer = init)(conv_256_0)\n leak_256_0 = LeakyReLU(0.2)(norm_256_0)\n \n conv_512_0 = Conv2D(2**9, (4,4), strides = 2, padding = 'same', kernel_initializer = init, use_bias = False)(leak_256_0)\n norm_512_0 = tfa.layers.InstanceNormalization(gamma_initializer = init)(conv_512_0)\n leak_512_0 = LeakyReLU(0.2)(norm_512_0)\n \n conv_512_1 = Conv2D(2**9, (4,4), strides = 2, padding = 'same', kernel_initializer = init, use_bias = False)(leak_512_0)\n norm_512_1 = tfa.layers.InstanceNormalization(gamma_initializer = init)(conv_512_1)\n leak_512_1 = LeakyReLU(0.2)(norm_512_1)\n \n conv_512_2 = Conv2D(2**9, (4,4), strides = 2, padding = 'same', kernel_initializer = init, use_bias = False)(leak_512_1)\n norm_512_2 = tfa.layers.InstanceNormalization(gamma_initializer = init)(conv_512_2)\n leak_512_2 = LeakyReLU(0.2)(norm_512_2)\n \n conv_512_3 = Conv2D(2**9, (4,4), strides = 2, padding = 'same', kernel_initializer = init, use_bias = False)(leak_512_2)\n norm_512_3 = tfa.layers.InstanceNormalization(gamma_initializer = init)(conv_512_3)\n leak_512_3 = LeakyReLU(0.2)(norm_512_3)\n \n conv_512_4 = Conv2D(2**9, (4,4), strides = 2, padding = 'same', kernel_initializer = init, use_bias = False)(leak_512_3)\n norm_512_4 = tfa.layers.InstanceNormalization(gamma_initializer = init)(conv_512_4)\n leak_512_4 = LeakyReLU(0.2)(norm_512_4)\n \n \n \n tran_512_3 = Conv2DTranspose(2**9, (4,4), strides = 2, padding = 'same', kernel_initializer = init, use_bias = False)(leak_512_4)\n norm_512_3 = tfa.layers.InstanceNormalization(gamma_initializer = init)(tran_512_3)\n drop_512_3 = Dropout(0.5)(norm_512_3)\n relu_512_3 = Activation('relu')(drop_512_3)\n conc_512_3 = Concatenate()([relu_512_3, leak_512_3])\n \n tran_512_2 = Conv2DTranspose(2**9, (4,4), strides = 2, padding = 'same', kernel_initializer = init, use_bias = False)(conc_512_3)\n norm_512_2 = tfa.layers.InstanceNormalization(gamma_initializer = init)(tran_512_2)\n drop_512_2 = Dropout(0.5)(norm_512_2)\n relu_512_2 = Activation('relu')(drop_512_2)\n conc_512_2 = Concatenate()([relu_512_2, leak_512_2])\n \n tran_512_1 = Conv2DTranspose(2**9, (4,4), strides = 2, padding = 'same', kernel_initializer = init, use_bias = False)(conc_512_2)\n norm_512_1 = tfa.layers.InstanceNormalization(gamma_initializer = init)(tran_512_1)\n drop_512_1 = Dropout(0.5)(norm_512_1)\n relu_512_1 = Activation('relu')(drop_512_1)\n conc_512_1 = Concatenate()([relu_512_1, leak_512_1])\n \n tran_512_0 = Conv2DTranspose(2**9, (4,4), strides = 2, padding = 'same', kernel_initializer = init, use_bias = False)(conc_512_1)\n norm_512_0 = tfa.layers.InstanceNormalization(gamma_initializer = init)(tran_512_0)\n relu_512_0 = Activation('relu')(norm_512_0)\n conc_512_0 = Concatenate()([relu_512_0, leak_512_0])\n \n tran_256_0 = Conv2DTranspose(2**8, (4,4), strides = 2, padding = 'same', kernel_initializer = init, use_bias = False)(conc_512_0)\n norm_256_0 = tfa.layers.InstanceNormalization(gamma_initializer = init)(tran_256_0)\n relu_256_0 = Activation('relu')(norm_256_0)\n conc_256_0 = Concatenate()([relu_256_0, leak_256_0])\n \n tran_128_0 = Conv2DTranspose(2**7, (4,4), strides = 2, padding = 'same', kernel_initializer = init, use_bias = False)(conc_256_0)\n norm_128_0 = tfa.layers.InstanceNormalization(gamma_initializer = init)(tran_128_0)\n relu_128_0 = Activation('relu')(norm_128_0)\n conc_128_0 = Concatenate()([relu_128_0, leak_128_0])\n \n tran_064_0 = Conv2DTranspose(2**6, (4,4), strides = 2, padding = 'same', kernel_initializer = init, use_bias = False)(conc_128_0)\n norm_064_0 = tfa.layers.InstanceNormalization(gamma_initializer = init)(tran_064_0)\n relu_064_0 = Activation('relu')(norm_064_0)\n conc_064_0 = Concatenate()([relu_064_0, leak_064_0])\n \n conv_003_0 = Conv2DTranspose(3**1, (4,4), strides = 2, padding = 'same', kernel_initializer = init, use_bias = False)(conc_064_0)\n \n return Model(inputs = src, outputs = conv_003_0)","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:49:47.314612Z","iopub.execute_input":"2021-06-05T07:49:47.314992Z","iopub.status.idle":"2021-06-05T07:49:47.340148Z","shell.execute_reply.started":"2021-06-05T07:49:47.314950Z","shell.execute_reply":"2021-06-05T07:49:47.339213Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Gg = gen()\nGf = gen()","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:49:47.341955Z","iopub.execute_input":"2021-06-05T07:49:47.342407Z","iopub.status.idle":"2021-06-05T07:49:48.280044Z","shell.execute_reply.started":"2021-06-05T07:49:47.342354Z","shell.execute_reply":"2021-06-05T07:49:48.279201Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"keras.utils.plot_model(Gg, './gen.png', show_shapes = True, dpi = 64)","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:49:48.281461Z","iopub.execute_input":"2021-06-05T07:49:48.281807Z","iopub.status.idle":"2021-06-05T07:49:48.751511Z","shell.execute_reply.started":"2021-06-05T07:49:48.281770Z","shell.execute_reply":"2021-06-05T07:49:48.750385Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loss functions","metadata":{}},{"cell_type":"code","source":"bin_entropy = keras.losses.BinaryCrossentropy(from_logits = True)","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:49:48.753210Z","iopub.execute_input":"2021-06-05T07:49:48.753564Z","iopub.status.idle":"2021-06-05T07:49:48.759318Z","shell.execute_reply.started":"2021-06-05T07:49:48.753526Z","shell.execute_reply":"2021-06-05T07:49:48.757899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"LAMBDA = 10\n\n'''\nGenerator loss\n'''\ndef g_loss (dis_output) :\n return bin_entropy(np.ones(dis_output.shape), dis_output)\n\n'''\nDiscriminator loss\n'''\ndef d_loss (dis_output_real, dis_output_fake) :\n \n real_loss = bin_entropy(np.ones(dis_output_real.shape), dis_output_real)\n fake_loss = bin_entropy(np.zeros(dis_output_fake.shape),dis_output_fake)\n total_loss = real_loss + fake_loss\n \n return total_loss * 0.5\n\n'''\nCycle Consistency loss\n'''\ndef c_loss (original_image,regenerated_image) :\n return keras.losses.mean_absolute_error(original_image,regenerated_image) * LAMBDA\n\n'''\nIdentity loss\n'''\ndef i_loss (target_image,output_target_image) :\n return keras.losses.mean_absolute_error(target_image,output_target_image) * LAMBDA * 0.5","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:49:48.761013Z","iopub.execute_input":"2021-06-05T07:49:48.761473Z","iopub.status.idle":"2021-06-05T07:49:48.772058Z","shell.execute_reply.started":"2021-06-05T07:49:48.761433Z","shell.execute_reply":"2021-06-05T07:49:48.771147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model Optimizers","metadata":{}},{"cell_type":"code","source":"gen_g_optimizer = keras.optimizers.Adam(learning_rate = 0.0002, beta_1 = 0.5)\ngen_f_optimizer = keras.optimizers.Adam(learning_rate = 0.0002, beta_1 = 0.5)\ndis_y_optimizer = keras.optimizers.Adam(learning_rate = 0.0002, beta_1 = 0.5)\ndis_x_optimizer = keras.optimizers.Adam(learning_rate = 0.0002, beta_1 = 0.5)","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:49:48.773538Z","iopub.execute_input":"2021-06-05T07:49:48.773912Z","iopub.status.idle":"2021-06-05T07:49:48.784637Z","shell.execute_reply.started":"2021-06-05T07:49:48.773875Z","shell.execute_reply":"2021-06-05T07:49:48.783778Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training function\n#### *For the sake of understanding, we named the images as 'horse' and 'zebras'*","metadata":{}},{"cell_type":"code","source":"@tf.function\ndef train_batch (src_horse_image, src_zebra_image) :\n \n with tf.GradientTape(persistent = True) as tape :\n \n # horse->zebra->horse (Gg and Dy)\n gen_zebra_image = Gg(src_horse_image, training = True)\n gen_horse_image = Gf(gen_zebra_image, training = True)\n \n sam_zebra_image = Gg(src_zebra_image, training = True)\n \n real_output_Dy = Dy(src_zebra_image, training = True)\n fake_output_Dy = Dy(gen_zebra_image, training = True)\n \n \n gen_g_loss = g_loss(fake_output_Dy)\n cyc_g_loss = c_loss(src_horse_image, gen_horse_image)\n idn_g_loss = i_loss(src_zebra_image, sam_zebra_image)\n dis_y_loss = d_loss(real_output_Dy , fake_output_Dy )\n \n # zebra->horse->zebra (Gf and Dx)\n gen_horse_image = Gf(src_zebra_image, training = True)\n gen_zebra_image = Gg(gen_horse_image, training = True)\n \n sam_horse_image = Gf(src_horse_image, training = True)\n \n real_output_Dx = Dx(src_horse_image, training = True)\n fake_output_Dx = Dx(gen_horse_image, training = True)\n \n \n gen_f_loss = g_loss(fake_output_Dx)\n cyc_f_loss = c_loss(src_zebra_image, gen_zebra_image)\n idn_f_loss = i_loss(src_horse_image, sam_horse_image)\n dis_x_loss = d_loss(real_output_Dx , fake_output_Dx )\n \n total_gen_g_loss = gen_g_loss + idn_g_loss + (cyc_g_loss + cyc_f_loss)\n total_gen_f_loss = gen_f_loss + idn_f_loss + (cyc_g_loss + cyc_f_loss)\n \n gen_g_grad = tape.gradient(total_gen_g_loss, Gg.trainable_variables)\n gen_f_grad = tape.gradient(total_gen_f_loss, Gf.trainable_variables)\n dis_y_grad = tape.gradient(dis_y_loss, Dy.trainable_variables)\n dis_x_grad = tape.gradient(dis_x_loss, Dx.trainable_variables)\n \n gen_g_optimizer.apply_gradients(zip(gen_g_grad, Gg.trainable_variables))\n gen_f_optimizer.apply_gradients(zip(gen_f_grad, Gf.trainable_variables))\n dis_y_optimizer.apply_gradients(zip(dis_y_grad, Dy.trainable_variables))\n dis_x_optimizer.apply_gradients(zip(dis_x_grad, Dx.trainable_variables))","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:49:48.786207Z","iopub.execute_input":"2021-06-05T07:49:48.786754Z","iopub.status.idle":"2021-06-05T07:49:48.799735Z","shell.execute_reply.started":"2021-06-05T07:49:48.786713Z","shell.execute_reply":"2021-06-05T07:49:48.798805Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Custom fit() function","metadata":{}},{"cell_type":"code","source":"def fig_plot (sam_photo, gen_image, sam_monet) :\n \n plt.figure(figsize = (20,50))\n \n plt.subplot(1,3,1)\n plt.imshow(sam_photo[0])\n plt.title('Photo Image')\n plt.axis('off')\n \n plt.subplot(1,3,2)\n plt.imshow(gen_image[0])\n plt.title('GenerateImg')\n plt.axis('off')\n \n plt.subplot(1,3,3)\n plt.imshow(sam_monet[0])\n plt.title('Monet Image')\n plt.axis('off')\n plt.show()\n \ndef fit (EPOCHS) :\n \n for epoch in range(EPOCHS) :\n \n print('[',end='')\n for n, (photo, monet) in train_dataset.enumerate() :\n if (n+1)%10== 0 :\n print('#',end='')\n if (n+1) == 300 :\n print(']',end='')\n train_batch(photo, monet)\n print()\n \n for sam_photo , sam_monet in train_dataset.take(1) :\n gen_image = Gg(sam_photo, training = True)\n fig_plot(sam_photo, gen_image , sam_monet)","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:49:48.801141Z","iopub.execute_input":"2021-06-05T07:49:48.801584Z","iopub.status.idle":"2021-06-05T07:49:48.814176Z","shell.execute_reply.started":"2021-06-05T07:49:48.801543Z","shell.execute_reply":"2021-06-05T07:49:48.813308Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fit(40)","metadata":{"execution":{"iopub.status.busy":"2021-06-05T07:49:48.815472Z","iopub.execute_input":"2021-06-05T07:49:48.815979Z","iopub.status.idle":"2021-06-05T08:38:38.345257Z","shell.execute_reply.started":"2021-06-05T07:49:48.815939Z","shell.execute_reply":"2021-06-05T08:38:38.344478Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Validation","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize = (10,50))\n\ni = 0\nwhile i < 16 :\n \n x = np.random.randint(0,7038)\n \n plt.subplot(8,2,i+1)\n plt.imshow(X[x])\n plt.axis('off')\n plt.title('Photo')\n \n monet = Gg(np.reshape(X[x], (1, 256, 256, 3)))\n plt.subplot(8,2,i+2)\n plt.imshow(monet[0])\n plt.axis('off')\n plt.title('Monet')\n \n i += 2","metadata":{"execution":{"iopub.status.busy":"2021-06-05T08:38:38.346961Z","iopub.execute_input":"2021-06-05T08:38:38.347489Z","iopub.status.idle":"2021-06-05T08:38:40.054677Z","shell.execute_reply.started":"2021-06-05T08:38:38.347451Z","shell.execute_reply":"2021-06-05T08:38:40.053446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"keras.models.save_model(Gg, './Gen_g.h5')\nkeras.models.save_model(Gf, './Gen_f.h5')\nkeras.models.save_model(Dy, './Dis_y.h5')\nkeras.models.save_model(Dx, './Dis_x.h5')","metadata":{"execution":{"iopub.status.busy":"2021-06-05T08:38:40.056323Z","iopub.execute_input":"2021-06-05T08:38:40.056718Z","iopub.status.idle":"2021-06-05T08:38:41.012150Z","shell.execute_reply.started":"2021-06-05T08:38:40.056677Z","shell.execute_reply":"2021-06-05T08:38:41.011224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission","metadata":{}},{"cell_type":"code","source":"! mkdir ../images","metadata":{"execution":{"iopub.status.busy":"2021-06-05T08:38:41.013557Z","iopub.execute_input":"2021-06-05T08:38:41.013902Z","iopub.status.idle":"2021-06-05T08:38:41.850978Z","shell.execute_reply.started":"2021-06-05T08:38:41.013867Z","shell.execute_reply":"2021-06-05T08:38:41.849927Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"i = 1\nfor img in X:\n \n prediction = Gg(np.reshape(img, (1,256,256,3)), training=False)[0].numpy()\n prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n \n im = Image.fromarray(prediction)\n im.save(\"../images/\" + str(i) + \".jpg\")\n i += 1","metadata":{"execution":{"iopub.status.busy":"2021-06-05T08:38:41.854235Z","iopub.execute_input":"2021-06-05T08:38:41.854521Z","iopub.status.idle":"2021-06-05T08:42:30.302009Z","shell.execute_reply.started":"2021-06-05T08:38:41.854492Z","shell.execute_reply":"2021-06-05T08:42:30.301124Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import shutil\nshutil.make_archive(\"/kaggle/working/images\", 'zip', \"/kaggle/images\")","metadata":{"execution":{"iopub.status.busy":"2021-06-05T08:42:30.303391Z","iopub.execute_input":"2021-06-05T08:42:30.303723Z","iopub.status.idle":"2021-06-05T08:42:33.076260Z","shell.execute_reply.started":"2021-06-05T08:42:30.303688Z","shell.execute_reply":"2021-06-05T08:42:33.075468Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### *You can use further augmentation to improve the results, but this was not the aim of my notebook. It was meant only for learning purposes.*","metadata":{}}]}