{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.10","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":21755,"databundleVersionId":1475600,"sourceType":"competition"},{"sourceId":2896364,"sourceType":"datasetVersion","datasetId":1737890}],"dockerImageVersionId":30146,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# CycleGAN, Monet-ization of Photohgraphs\n\nThis notebook is going to describe how one can use a CycleGAN to transform images from a domain $X$ to another domain $Y$. In the case of this notebook, we are going to be working on transforming real life photos into paintings in the style of [Claude Monet][1].\n\nA Generative Adverserial Network (GAN) made up of two subnetworks, a generative ($G$) and a discriminative network ($D_x$). The task of the generative network is to transform a photo ($X$) into a painting ($Y$). The generative process can be described simply as $G(X) -> Y$, where $G$ is a generative network that has learnt the mapping from domain $X$ to domain $Y$. While training, the two networks are constantly playing a game of cat and mouse. The generative network is trying to generate photos that can pass as paintings while the discriminative network is trying to detect which paintings are real and which painting have been generated by the generative network. \n\n\nWhat stands out from CycleGANs compared to regular GANs is that a CycleGAN introduces two additional networks, an additional generative network ($F$) and an additional discriminator ($D_y$ that can identify real photographs from generated photographs. The task of generator $F$ is to find the mapping $Y -> X$. The adverserial loss for generator $F$ is trained in the same way as that of generator $G$. The two additional networks create a cyclic connection where the mapping $F(G(X)) = X$ and $G(F(Y)) = Y$. The reasoning for the cyclic mapping is to make sure that no two inputs $X$ can map to the same $Y$, which is a problem called mode collapse. To train the cyclic mapping of the networks, another type of loss called cyclic loss has to be introduced.\n\nI would recommend you to read the [original paper][2] for a more correct and in-depth explanation of the model\n\n[1]: https://en.wikipedia.org/wiki/Claude_Monet\n[2]: https://arxiv.org/pdf/1703.10593.pdf\n","metadata":{}},{"cell_type":"markdown","source":"## Load Kaggle Data\n\nLocating and Loading data from the Kaggle dataset 'gan-getting-started'. We need images of real photos and images of monet paintings for the CycleGAN. \n\nThe datasets are saved as BatchData with batch size 1 and image size 256x256.","metadata":{}},{"cell_type":"code","source":"import os\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom functools import partial\n\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nfrom kaggle_datasets import KaggleDatasets\n\nimport tensorflow_datasets as tfds\n\nGCS_PATH = KaggleDatasets().get_gcs_path('gan-getting-started')\n\nFILENAMES_MONET = tf.io.gfile.glob(GCS_PATH + \"/monet_tfrec/\" + \"*.tfrec\")\nFILENAMES_REAL = tf.io.gfile.glob(GCS_PATH + \"/photo_tfrec/\" + \"*.tfrec\")\n#FILENAMES_MONET_TEST = tf.io.gfile.glob('../input/gan-getting-started/monet_tfrec/' + \"*.tfrec\")\n#FILENAMES_REAL_TEST = tf.io.gfile.glob('../input/gan-getting-started/photo_tfrec/' + \"*.tfrec\")\n\nprint(\"Train TFRecord Files:\", len(FILENAMES_MONET))\nprint(\"Train TFRecord Files:\", len(FILENAMES_REAL))","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-04-22T17:30:07.454666Z","iopub.execute_input":"2024-04-22T17:30:07.455416Z","iopub.status.idle":"2024-04-22T17:30:15.411702Z","shell.execute_reply.started":"2024-04-22T17:30:07.455321Z","shell.execute_reply":"2024-04-22T17:30:15.410912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"AUTOTUNE = tf.data.AUTOTUNE\nBATCH_SIZE = 1\n\nIMAGE_SIZE = [256,256]\n\ndef decode_image(image,IMAGE_SIZE=256):\n    image = tf.image.decode_jpeg(image, channels=3)\n    image = tf.cast(image, tf.float32)\n    #image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    return image\n\ndef normalize_img(img):\n    img = tf.cast(img, dtype=tf.float32)\n    return (img / 127.5) - 1.0\n\n\n\ndef read_tfrecord(example, labeled=False):\n    tfrecord_format = (\n        {\n            \"image\": tf.io.FixedLenFeature([], tf.string)\n        }\n    )\n    example = tf.io.parse_single_example(example, tfrecord_format)\n    image = decode_image(example[\"image\"])\n    image = normalize_img(image)\n\n    return image\n\n\ndef load_dataset(filenames, labeled=False):\n    ignore_order = tf.data.Options()\n    ignore_order.experimental_deterministic = False  # disable order, increase speed\n    dataset = tf.data.TFRecordDataset(\n        filenames\n    )  \n    dataset = dataset.with_options(\n        ignore_order\n    )  \n    dataset = dataset.map(\n        partial(read_tfrecord, labeled=labeled), num_parallel_calls=AUTOTUNE\n    )\n    return dataset\n\ndef get_dataset(filenames, labeled=True):\n    dataset = load_dataset(filenames, labeled=labeled)\n    #dataset = dataset.shuffle(2048)\n    dataset = dataset.prefetch(buffer_size=AUTOTUNE)\n    dataset = dataset.batch(BATCH_SIZE)\n    return dataset\n\ndef data():\n    return get_dataset(FILENAMES_REAL),get_dataset(FILENAMES_MONET)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-04-22T17:30:15.413384Z","iopub.execute_input":"2024-04-22T17:30:15.413606Z","iopub.status.idle":"2024-04-22T17:30:15.425525Z","shell.execute_reply.started":"2024-04-22T17:30:15.413579Z","shell.execute_reply":"2024-04-22T17:30:15.424671Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Augmenting Data\n\nData augmentation has been proven to improve performance of neural networks for images in order to prevent overfitting. The augmentation methods used for this problem is random mirroring along with resizing and random cropping.\n\nSource for data Augmentation:\nhttps://colab.research.google.com/github/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/CycleGAN.ipynb","metadata":{}},{"cell_type":"code","source":"from keras.preprocessing.image import ImageDataGenerator\nimport imgaug.augmenters as iaa\n\nclass DataAugmenter:\n    def __init__(self):\n        data_generator = ImageDataGenerator(brightness_range=[0.8,1.2])\n    \n    def normalize_img(self,img):\n        img = tf.cast(img, dtype=tf.float32)\n        return (img / 127.5) - 1.0\n\n    def random_crop(self,image):\n        cropped_image = tf.image.random_crop(\n        image, size=[1,IMAGE_SIZE[0],IMAGE_SIZE[1], 3])\n\n        return cropped_image\n    \n    \"\"\"def augment_color(self):\n        aug = iaa.Sequential([\n            iaa.MultiplyHue((0.1,9.9)),\n            iaa.MultiplyBrightness(mul=(0.1,9.9)),\n            iaa.LogContrast(gain=(0.1,9.9))\n        ])\n        return aug\"\"\"\n    \n    def aug_color(self,image):\n        image = tf.image.random_brightness(image, 0.2)\n        image = tf.image.random_contrast(image, 0.8,1.2)\n        return tf.image.random_saturation(image, 0.8,1.2)\n\n    def augment(self,image):\n        # resizing to 300 x 300 x 3\n        image = tf.image.resize(image, [300, 300],\n                              method=tf.image.ResizeMethod.NEAREST_NEIGHBOR)\n        image = self.random_crop(image)\n\n        image = tf.image.random_flip_left_right(image)\n        #image = self.aug_color(image)\n        return image","metadata":{"execution":{"iopub.status.busy":"2024-04-22T17:30:15.426784Z","iopub.execute_input":"2024-04-22T17:30:15.427077Z","iopub.status.idle":"2024-04-22T17:30:17.303055Z","shell.execute_reply.started":"2024-04-22T17:30:15.427038Z","shell.execute_reply":"2024-04-22T17:30:17.302360Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Sample some photos\nThe code snippet below samples random images of real photographs and monet paintings respectively.","metadata":{}},{"cell_type":"code","source":"train_real, train_monet = data()\n_, ax = plt.subplots(2, 2, figsize=(15, 15))\nfor i, samples in enumerate(zip(train_real.take(2), train_monet.take(2))):\n    real = (((samples[0][0] * 127.5) + 127.5).numpy()).astype(np.uint8)\n    monet = (((samples[1][0] * 127.5) + 127.5).numpy()).astype(np.uint8)\n    ax[i, 0].imshow(monet)\n    ax[i, 0].set_title(\"Monet Sample\")\n    ax[i, 1].imshow(real)\n    ax[i, 1].set_title(\"Real Photo Sample\")\n\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2024-04-22T17:30:17.304206Z","iopub.execute_input":"2024-04-22T17:30:17.304436Z","iopub.status.idle":"2024-04-22T17:30:22.841440Z","shell.execute_reply.started":"2024-04-22T17:30:17.304406Z","shell.execute_reply":"2024-04-22T17:30:22.840489Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Network Architecture\n\nWe follow the architectures used in the original paper, [Unpaired Image-to-Image Translation\nusing Cycle-Consistent Adversarial Networks][1] \n\n[1]: https://arxiv.org/pdf/1703.10593.pdf.\n\nCycleGANs require two network architectures, one for the generator and one for the discriminator. ","metadata":{}},{"cell_type":"markdown","source":"### Discriminative Network Architecture\nDiscriminator: C64->C128->C256->C512 as described in the paper. Ck is a convolutional layer uses LeakyReLU as activation along with an InstanceNormalization layer where k is the amount of filters in the convolution.\n\n\n","metadata":{}},{"cell_type":"code","source":"from tensorflow import keras\nfrom tensorflow.keras.layers import Input, LeakyReLU, Conv2D\nfrom tensorflow_addons.layers import InstanceNormalization\nfrom tensorflow.keras.initializers import RandomNormal\n\nfrom tensorflow.keras.models import Model\n\n#  C64->C128->C256->C512\nclass Discriminator:\n    def __init__(self,padding ='valid',strides=(2,2),kernel=(4,4),initializer = RandomNormal(mean=0.,stddev=0.02),alpha=0.2):\n        img_inp = Input(shape = (256, 256, 3))\n        conv_1 = Conv2D(64,kernel,strides=2,use_bias=False,kernel_initializer=initializer,padding=padding)(img_inp)\n        act_1 = LeakyReLU(alpha)(conv_1)\n    \n        conv_2 = Conv2D(128,kernel,strides=strides,use_bias=False,kernel_initializer=initializer,padding=padding)(act_1)\n        \n        batch_norm_2 = InstanceNormalization(gamma_initializer=keras.initializers.RandomNormal(mean=0.0, stddev=0.02))(conv_2)\n        act_2 = LeakyReLU(alpha)(batch_norm_2)\n    \n        conv_3 = Conv2D(256,kernel,strides=strides,use_bias=False,kernel_initializer=initializer,padding=padding)(act_2)\n        batch_norm_3 = InstanceNormalization(gamma_initializer=keras.initializers.RandomNormal(mean=0.0, stddev=0.02))(conv_3)\n        act_3 = LeakyReLU(alpha)(batch_norm_3)\n    \n        conv_4 = Conv2D(512,kernel,strides=(1,1),use_bias=False,kernel_initializer=initializer)(act_3)\n        batch_norm_4 = InstanceNormalization(gamma_initializer=keras.initializers.RandomNormal(mean=0.0, stddev=0.02))(conv_4)\n        act_4 = LeakyReLU(alpha)(batch_norm_4)\n    \n        #zero_pad_1 = ZeroPadding2D()(act_4)\n        outputs = Conv2D(1,kernel,strides=1,use_bias=False,kernel_initializer=initializer)(act_4)\n    \n        self.model = Model(img_inp, outputs)\n","metadata":{"execution":{"iopub.status.busy":"2024-04-22T17:30:22.843403Z","iopub.execute_input":"2024-04-22T17:30:22.843634Z","iopub.status.idle":"2024-04-22T17:30:22.986812Z","shell.execute_reply.started":"2024-04-22T17:30:22.843605Z","shell.execute_reply":"2024-04-22T17:30:22.986185Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Generative Network Architecture\nThe generative network follows a different structure to the disicriminative network. The network has a few downsampling layers, followed by residual skip blocks ending with upsampling layers.\n\nNote: `c7s1-k` means a convolutional layer where `kernel_size=7`, `strides=1` and `filters=k`\n\n`dk` is a downsampling convolutional layer with `strides=2`, `kernel_size=3` and `filters=k` \n\n`uk` is an upsampling layer with `strides=1/2`, `kernel_size=2` and `filters=k`. (`Conv2dTranspose` with `strides=(2,2)`)\n\n`Rk` is a residual block with `filters=k`, `strides=1` and `kernel_size=3`.\n\n**Downsampling layers:** `c7s1-64->d128->d256`\n\n**Residual blocks:** \n`R256->R256->R256->R256->R256->R256->R256->R256->R256`\n\n**Upsampling layers:** `u128->u64->c7s1-3`\n\n**Residual skip connections** are used useful for several reasons, one of those being to avoid the problem of vanishing gradients. Another problem that is solved by skip connections is the degradation problem in deep neural networks. Instead of learning the mapping between input and output, the network learns the residual/difference between input and the output function. [Here is a link][1] to a nice video explaining the concept and advantages of residual block.\n\nPixels on the border of an image are convolved less frequently than pixels more to the center of an image and will therefore not be preserved very well by the network. To combat this, we introduce **reflection padding** where the images get an additional layer added on top of the borders.\n\nSources for some of the code:\nhttps://keras.io/examples/generative/cyclegan/\nhttps://theailearner.com/tag/patchgan/\nhttps://machinelearningmastery.com/how-to-develop-cyclegan-models-from-scratch-with-keras/\n\n\n[1]:https://www.youtube.com/watch?v=rya-1nX8ktc&t=521s&ab_channel=TheCodingLib","metadata":{}},{"cell_type":"code","source":"### https://stackoverflow.com/questions/50677544/reflection-padding-conv2d\nfrom tensorflow.keras.layers import Layer, InputSpec\n\nclass ReflectionPadding2D(Layer):\n    def __init__(self, padding=(1, 1), **kwargs):\n        self.padding = tuple(padding)\n        self.input_spec = [InputSpec(ndim=3)]\n        super(ReflectionPadding2D, self).__init__(**kwargs)\n\n    def get_output_shape_for(self, s):\n        \"\"\" If you are using \"channels_last\" configuration\"\"\"\n        return (s[0], s[1] + 2 * self.padding[0], s[2] + 2 * self.padding[1], s[3])\n\n    def call(self, x, mask=None):\n        w_pad,h_pad = self.padding\n        return tf.pad(x, [[0,0], [h_pad,h_pad], [w_pad,w_pad], [0,0] ], 'REFLECT')\n\n","metadata":{"execution":{"iopub.status.busy":"2024-04-22T17:30:22.988016Z","iopub.execute_input":"2024-04-22T17:30:22.988244Z","iopub.status.idle":"2024-04-22T17:30:22.996734Z","shell.execute_reply.started":"2024-04-22T17:30:22.988216Z","shell.execute_reply":"2024-04-22T17:30:22.995935Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from keras.layers import Reshape, Dense, Input, ReLU, Conv2D, Conv2DTranspose, Concatenate, ReLU, ZeroPadding2D\nfrom tensorflow_addons.layers import InstanceNormalization\nclass Generator:\n    def __init__(self,k=64,n_res=8):\n        img_inp = Input(shape = (256, 256, 3))\n        c7s164 = ReflectionPadding2D(padding = (3,3))(img_inp)\n        c7s164 = Conv2D(64,(7,7),(1,1),kernel_initializer=tf.random_normal_initializer(0., 0.02))(c7s164)\n        #c7s164 = InstanceNormalization(gamma_initializer=keras.initializers.RandomNormal(mean=0.0, stddev=0.02))(c7s164)\n        c7s164 = ReLU()(c7s164)\n\n        d128 = Conv2D(128,(3,3),(2,2),kernel_initializer=tf.random_normal_initializer(0., 0.02),padding=\"same\")(c7s164)\n        d128 = InstanceNormalization(gamma_initializer=keras.initializers.RandomNormal(mean=0.0, stddev=0.02))(d128)\n        d128 = ReLU()(d128)\n\n        d256 = Conv2D(256,(3,3),(2,2),kernel_initializer=tf.random_normal_initializer(0., 0.02),padding=\"same\")(d128)\n        d256 = InstanceNormalization(gamma_initializer=keras.initializers.RandomNormal(mean=0.0, stddev=0.02))(d256)\n        d256 = ReLU()(d256)\n\n\n        # RESIDUAL BLCOKS\n\n        curr = d256\n        res = d256\n        k=256\n        for _ in range(n_res):\n            res = ReflectionPadding2D()(res)\n            res = Conv2D(k,(3,3),kernel_initializer=tf.random_normal_initializer(0., 0.02))(res)\n            res = InstanceNormalization(gamma_initializer=keras.initializers.RandomNormal(mean=0.0, stddev=0.02))(res)\n            res = ReLU()(res)\n\n            res = ReflectionPadding2D()(res)\n            res = Conv2D(k,(3,3),kernel_initializer=tf.random_normal_initializer(0., 0.02))(res)\n            res = InstanceNormalization(gamma_initializer=keras.initializers.RandomNormal(mean=0.0, stddev=0.02))(res)\n            res = Concatenate()([res,curr])\n            curr = res\n            \n\n        u128 = Conv2DTranspose(128,(3,3),(2,2),kernel_initializer=tf.random_normal_initializer(0., 0.02),padding=\"same\")(res)\n        u128 = InstanceNormalization(gamma_initializer=keras.initializers.RandomNormal(mean=0.0, stddev=0.02))(u128)\n        u128 = ReLU()(u128)\n\n        u64 = Conv2DTranspose(64,(3,3),(2,2),kernel_initializer=tf.random_normal_initializer(0., 0.02),padding=\"same\")(u128)\n        u64 = InstanceNormalization(gamma_initializer=keras.initializers.RandomNormal(mean=0.0, stddev=0.02))(u64)\n        u64 = ReLU()(u64)\n\n        c7s13 = ReflectionPadding2D(padding=(3,3))(u64)\n        c7s13 = Conv2D(3,(7,7),activation='tanh')(c7s13)\n\n        self.model = Model(img_inp, c7s13)\n","metadata":{"execution":{"iopub.status.busy":"2024-04-22T17:30:22.997973Z","iopub.execute_input":"2024-04-22T17:30:22.998207Z","iopub.status.idle":"2024-04-22T17:30:23.018061Z","shell.execute_reply.started":"2024-04-22T17:30:22.998179Z","shell.execute_reply":"2024-04-22T17:30:23.017306Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training and Loss Function\n\nTraining can take some time, use a GPU accelerator.","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\n\ndevice_name = tf.test.gpu_device_name()\nif \"GPU\" not in device_name:\n    print(\"GPU device not found\")\nprint('Found GPU at: {}'.format(device_name))","metadata":{"execution":{"iopub.status.busy":"2024-04-22T17:30:23.019345Z","iopub.execute_input":"2024-04-22T17:30:23.019609Z","iopub.status.idle":"2024-04-22T17:30:23.035428Z","shell.execute_reply.started":"2024-04-22T17:30:23.019581Z","shell.execute_reply":"2024-04-22T17:30:23.034655Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Loss Functions\nThere are three types of losses for the generator, **adversarial**-, **identity** and **cyclic consistensy loss**.\n\n**Adverserial loss** forces generated images to be as indistinguishable from Monet paintings as possible. The adversarial loss can be described as a least squares loss which should be minimized according to $\\mathbb{E}_{x \\sim p_{\\text {data }}(x)}\\left[(D(G(x))-1)^{2}\\right]$\n\nTo prevent **Mode Collapse** from domain $X -> Y$, we add a cyclic transformation. Both transformations $F : X->Y$ and $G: Y->X$ must be satisfied. The transformation loss, known as **cycle consistency loss** is added onto the adverserial loss in training so that $G(F(x))≈x$ and $F(G(x))≈x$. Cycle consistency loss is minimized through $\\mathbb{E}_{x \\sim p_{\\text {data }}(x)}\\left[\\|F(G(x))-x\\|\\right]$ for $G$ and $\\mathbb{E}_{y \\sim p_{\\text {data }}(y)}\\left[\\|G(F(y))-y\\|\\right]$ for $F$.\n\nLastly is the **identity loss**. Identity loss is useful for color composition preservation when mapping between input and output. Identity loss for G should be minimized through $\\left.\\mathbb{E}_{y \\sim p_{\\text {data }}(y)}[\\| G(y)-y \\|\\right]$.\n\nThere is only one relevant loss function for the discriminators and that is an **adversarial loss**. For the mapping $G: X->Y$, the discriminator $D_y$ minimizes $\\mathbb{E}_{y \\sim p_{\\text {data }}(y)}\\left[\\log D_{Y}(y)\\right]$ $+$ $\\mathbb{E}_{x \\sim p_{\\text {data }}(x)}\\left[\\log \\left(1-D_{Y}(G(x))\\right]\\right.$\n\n\n","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.layers import Input\nfrom tensorflow.keras.models import Model\nfrom tensorflow.keras.losses import MeanAbsoluteError, MeanSquaredError,BinaryCrossentropy\nimport tensorflow as tf\nfrom tensorflow.keras import backend as K\n\nclass CycleGAN(Model):\n\n    def __init__(self,shape=((256, 256, 3))):#,batch):\n        super(CycleGAN,self).__init__()\n        x = Input(shape=shape)\n        y = Input(shape=shape)\n        self.cycle_weight = 10\n        self.identity_weight = 0.5\n        self.augmenter = DataAugmenter()\n        \n        super(CycleGAN,self).compile()\n        self.genG = Generator().model\n        self.genF = Generator().model\n        self.discX = Discriminator().model\n        self.discY = Discriminator().model\n\n        self.genG_optimizer = tf.keras.optimizers.Adam(learning_rate=2e-4,beta_1=0.5)\n        self.genF_optimizer = tf.keras.optimizers.Adam(learning_rate=2e-4,beta_1=0.5)\n        self.discX_optimizer = tf.keras.optimizers.Adam(learning_rate=2e-4,beta_1=0.5)\n        self.discY_optimizer = tf.keras.optimizers.Adam(learning_rate=2e-4,beta_1=0.5)\n\n        self.cycle_loss = MeanAbsoluteError()\n        self.identity_loss = MeanAbsoluteError()\n        self.gen_loss = BinaryCrossentropy(from_logits=True)\n        self.disc_loss = BinaryCrossentropy(from_logits=True)\n\n    @tf.function\n    def train_step(self,data_batch):\n        x,y = data_batch\n        x = self.augmenter.augment(x)\n        y = self.augmenter.augment(y)\n        with tf.GradientTape(persistent=True) as tape:\n            gen_y = self.genG(x, training=True)\n            gen_x = self.genF(y, training=True)\n            recon_x = self.genF(gen_y, training=True)\n            recon_y = self.genG(gen_x, training=True)\n\n            # Identity\n            identity_x = self.genF(x, training=True)\n            identity_y = self.genG(y, training=True)\n\n            # disc\n            predict_x = self.discX(x, training=True)\n            predict_gen_x = self.discX(gen_x, training=True)\n\n            predict_y = self.discY(y, training=True)\n            predict_gen_y = self.discY(gen_y, training=True)\n\n            G_identity_loss =  self.identity_loss(y,identity_y)* self.identity_weight * self.cycle_weight\n            F_identity_loss = self.identity_loss(x, identity_x)* self.identity_weight * self.cycle_weight\n\n            G_cycle_loss = self.cycle_loss(x, recon_x)* self.cycle_weight\n            F_cycle_loss = self.cycle_loss(y, recon_y)* self.cycle_weight\n\n            G_gen_loss = self.gen_loss(tf.ones_like(predict_gen_y),predict_gen_y)\n            F_gen_loss = self.gen_loss(tf.ones_like(predict_gen_x),predict_gen_x,)\n\n            Y_disc_loss = self.disc_loss(tf.ones_like(predict_y),predict_y)/2 + self.disc_loss(tf.zeros_like(predict_gen_y),predict_gen_y)/2\n            X_disc_loss = self.disc_loss(tf.ones_like(predict_x),predict_x)/2 + self.disc_loss(tf.zeros_like(predict_gen_x),predict_gen_x)/2\n            G_total_loss = G_cycle_loss+G_identity_loss+G_gen_loss\n            F_total_loss = F_cycle_loss+F_identity_loss+F_gen_loss\n    \n        gradsG = tape.gradient(G_total_loss, self.genG.trainable_variables)\n        gradsF = tape.gradient(F_total_loss, self.genF.trainable_variables)\n\n        discX_grads = tape.gradient(X_disc_loss, self.discX.trainable_variables)\n        discY_grads = tape.gradient(Y_disc_loss, self.discY.trainable_variables)\n\n        self.genG_optimizer.apply_gradients(\n            zip(gradsG, self.genG.trainable_variables)\n        )\n        self.genF_optimizer.apply_gradients(\n            zip(gradsF, self.genF.trainable_variables)\n        )\n\n        # Update the weights of the discriminators\n        self.discX_optimizer.apply_gradients(\n            zip(discX_grads, self.discX.trainable_variables)\n        )\n        self.discY_optimizer.apply_gradients(\n            zip(discY_grads, self.discY.trainable_variables)\n        )\n        \n\n        return {\n            \"G_loss\": G_cycle_loss,\n            \"F_loss\": F_total_loss,\n            \"D_X_loss\": X_disc_loss,\n            \"D_Y_loss\": Y_disc_loss,\n        }","metadata":{"execution":{"iopub.status.busy":"2024-04-22T17:30:23.036603Z","iopub.execute_input":"2024-04-22T17:30:23.036841Z","iopub.status.idle":"2024-04-22T17:30:23.062216Z","shell.execute_reply.started":"2024-04-22T17:30:23.036814Z","shell.execute_reply":"2024-04-22T17:30:23.061462Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Keeping track of the training using Callbacks\n\nWe can monitor the evolution of the networks generative prowess by implementing our own keras Callback object. The ```on_epoch_begin(self,epoch)``` function is called before every single epoch of training.","metadata":{}},{"cell_type":"code","source":"class ShowProgressCallback(keras.callbacks.Callback):\n    def __init__(self):\n        super(keras.callbacks.Callback,self).__init__()\n        self.photo = zip(train_real.take(1))\n        for i, image in enumerate(self.photo):\n            self.image=image\n    \n    def generated_monet(self,epoch):\n        f, axarr = plt.subplots(1,2,figsize=(10,10))\n        generated_image = cgan.genG(self.image)\n        real = (((self.image[0][0] * 127.5) + 127.5).numpy()).astype(np.uint8)\n        gen = (((generated_image[0] * 127.5) + 127.5).numpy()).astype(np.uint8)\n        axarr[0].imshow(gen)\n        axarr[1].imshow(real)\n        axarr[1].set_title(\"epoch: \" + str(epoch) + ', Real photo' )\n        axarr[0].set_title(\"epoch: \" + str(epoch) + ', Generated Painting' )\n        plt.show()    \n        \n    def generated_real(self,epoch,images_to_show = 2):\n        f, axarr = plt.subplots(images_to_show,2,figsize=(10,10))\n        for i, image in enumerate(zip(train_monet.take(images_to_show))):\n            generated_image = cgan.genF(image)\n            real = (((image[0][0] * 127.5) + 127.5).numpy()).astype(np.uint8)\n            monet = (((generated_image[0] * 127.5) + 127.5).numpy()).astype(np.uint8)\n            axarr[i,1].imshow(monet)\n            axarr[i,0].imshow(real)\n        plt.show()\n    \n    def on_epoch_begin(self, epoch, logs=None):\n        if epoch%5 ==0:\n            print(\"Epoch:\", epoch)\n            #self.generated_real(epoch)\n            self.generated_monet(epoch)\n            ","metadata":{"execution":{"iopub.status.busy":"2024-04-22T17:30:23.063453Z","iopub.execute_input":"2024-04-22T17:30:23.064019Z","iopub.status.idle":"2024-04-22T17:30:23.078682Z","shell.execute_reply.started":"2024-04-22T17:30:23.063978Z","shell.execute_reply":"2024-04-22T17:30:23.077998Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I'm loading an already trained network from a previous session, set ```new_model = True``` if you want to train a new model from scratch.","metadata":{}},{"cell_type":"code","source":"# TRAINING\nnew_model = True\ncgan = CycleGAN()\nepochs=25\n\n#cgan.built=True\n#cgan.load_weights('../input/cgan-model/cycleGAN_BCE_225.h5')\n\nif new_model:\n    cgan.fit(tf.data.Dataset.zip((train_real, train_monet)),epochs=epochs, callbacks=[ShowProgressCallback()],verbose=1)\n    cgan.save_weights('cycleGAN.h5')\nelse:\n    cgan.built=True\n    cgan.load_weights('../input/cgan-model/cycleGAN_BCE_250.h5')\n","metadata":{"execution":{"iopub.status.busy":"2024-04-22T17:30:23.079811Z","iopub.execute_input":"2024-04-22T17:30:23.080343Z","iopub.status.idle":"2024-04-22T18:41:34.796729Z","shell.execute_reply.started":"2024-04-22T17:30:23.080306Z","shell.execute_reply":"2024-04-22T18:41:34.796104Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Some generated samples**","metadata":{}},{"cell_type":"code","source":"images_to_show = 10\nfor i, image in enumerate(zip(train_real.take(images_to_show))):\n    f, axarr = plt.subplots(1,2,figsize=(15,15))\n    gen = cgan.genG(image)\n    real = (((image[0][0] * 127.5) + 127.5).numpy()).astype(np.uint8)\n    gen = (((gen[0] * 127.5) + 127.5).numpy()).astype(np.uint8)\n\n    axarr[1].imshow(real)\n    axarr[1].set_title('Real Photohraph')\n    axarr[0].imshow(gen)\n    axarr[0].set_title('Generated Painting')\n    plt.show()\n","metadata":{"execution":{"iopub.status.busy":"2024-04-22T18:41:34.797957Z","iopub.execute_input":"2024-04-22T18:41:34.798198Z","iopub.status.idle":"2024-04-22T18:41:41.975077Z","shell.execute_reply.started":"2024-04-22T18:41:34.798169Z","shell.execute_reply":"2024-04-22T18:41:41.974224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Predicting and Saving","metadata":{}},{"cell_type":"code","source":"from PIL import Image\nos.makedirs('./images')\nfor i,element, in enumerate(train_real.as_numpy_iterator()):\n    if i % 200 == 0:\n        print(i)\n    generated_image = cgan.genG(element)\n    gen = (((generated_image[0] * 127.5) + 127.5).numpy()).astype(np.uint8)\n    im = Image.fromarray(gen)\n    im.save( './images/' + str(i) +\".jpg\")\n","metadata":{"execution":{"iopub.status.busy":"2024-04-22T18:41:41.976523Z","iopub.execute_input":"2024-04-22T18:41:41.976766Z","iopub.status.idle":"2024-04-22T18:48:32.385854Z","shell.execute_reply.started":"2024-04-22T18:41:41.976735Z","shell.execute_reply":"2024-04-22T18:48:32.385161Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import shutil\nshutil.make_archive(\"/kaggle/working/images\", 'zip', \"./images\")","metadata":{"execution":{"iopub.status.busy":"2024-04-22T18:48:32.388614Z","iopub.execute_input":"2024-04-22T18:48:32.388861Z","iopub.status.idle":"2024-04-22T18:48:36.364569Z","shell.execute_reply.started":"2024-04-22T18:48:32.388831Z","shell.execute_reply":"2024-04-22T18:48:36.363904Z"},"trusted":true},"execution_count":null,"outputs":[]}]}