{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### The problem and the data\nArtists like Claude Monet can  be imitated with algorithms thanks to generative adversarial networks (GANs). In this mini project, we will bring that style to the photos taken from camera.\n\nThe dataset contains four directories: *monet_tfrec*, *photo_tfrec*, *monet_jpg*, and *photo_jpg*. The monet_tfrec and monet_jpg directories contain the same painting images, and the photo_tfrec and photo_jpg directories contain the same photos.\n\nWe will use TFRecords to deserialize the Monet images as well as the camera photos from tfrec files where Monet images will be used to train the model. As the outcome of the project, we will add Monet-style to these images so that the camera taken photos will have a flavor of Monet. A zip file containing 10,000 immages will eventually produced for final submission.\n\n","metadata":{"papermill":{"duration":0.013215,"end_time":"2020-08-29T08:27:01.892915","exception":false,"start_time":"2020-08-29T08:27:01.879700","status":"completed"},"tags":[],"id":"DnK5oNQrAeLm"}},{"cell_type":"code","source":"# !pip install tensorflow_addons","metadata":{"id":"5LbL5T93A_wL","outputId":"470cde61-b8c0-4830-f9bc-3eedf0fc3c23"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nimport tensorflow_addons as tfa\nimport matplotlib.pyplot as plt\nimport numpy as np\nfrom kaggle_datasets import KaggleDatasets\nimport PIL\n\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Device:', tpu.master())\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nexcept:\n    strategy = tf.distribute.get_strategy()\nprint('Number of replicas:', strategy.num_replicas_in_sync)\n\nAUTOTUNE = tf.data.experimental.AUTOTUNE\n    \nprint(tf.__version__)","metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","papermill":{"duration":10.623182,"end_time":"2020-08-29T08:27:12.529697","exception":false,"start_time":"2020-08-29T08:27:01.906515","status":"completed"},"tags":[],"id":"RRcz5dcRAeLn","outputId":"4d3fceb3-97a0-462d-c7f7-d5ee2aa14d1c"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Exploratory data analysis\n\nImages are  sized to 256x256. Since these images are RGB images, the channel will be 3. Additionally, we need to scale the images to a [-1, 1] scale. Because we are building a generative model, we don't need the labels or the image id so we'll only return the image from the TFRecord.","metadata":{"papermill":{"duration":0.010743,"end_time":"2020-08-29T08:27:12.552004","exception":false,"start_time":"2020-08-29T08:27:12.541261","status":"completed"},"tags":[],"id":"y_wCGStHAeLp"}},{"cell_type":"code","source":"# # GCS_PATH = KaggleDatasets().get_gcs_path()\n# from google.colab import drive\n# drive.mount('/content/gdrive/', force_remount=True)\n","metadata":{"execution":{"iopub.execute_input":"2020-08-29T08:27:12.594856Z","iopub.status.busy":"2020-08-29T08:27:12.593948Z","iopub.status.idle":"2020-08-29T08:27:13.591083Z","shell.execute_reply":"2020-08-29T08:27:13.590238Z"},"papermill":{"duration":1.027954,"end_time":"2020-08-29T08:27:13.591219","exception":false,"start_time":"2020-08-29T08:27:12.563265","status":"completed"},"tags":[],"id":"t3aTxAyzAeLp","outputId":"0ccc7540-ae33-4b8b-8759-609167b3f519"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##GCS_PATH = '../input/gan-getting-started'\nGCS_PATH = KaggleDatasets().get_gcs_path()","metadata":{"id":"LEF5aFXCBoK5"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"MONET_FILENAMES = tf.io.gfile.glob(str(GCS_PATH) + '/monet_tfrec/*.tfrec')\nprint('Monet TFRecord Files:', len(MONET_FILENAMES))\n\nPHOTO_FILENAMES = tf.io.gfile.glob(str(GCS_PATH) + '/photo_tfrec/*.tfrec')\nprint('Photo TFRecord Files:', len(PHOTO_FILENAMES))","metadata":{"execution":{"iopub.execute_input":"2020-08-29T08:27:13.928234Z","iopub.status.busy":"2020-08-29T08:27:13.927282Z","iopub.status.idle":"2020-08-29T08:27:13.932144Z","shell.execute_reply":"2020-08-29T08:27:13.931490Z"},"papermill":{"duration":0.329473,"end_time":"2020-08-29T08:27:13.932267","exception":false,"start_time":"2020-08-29T08:27:13.602794","status":"completed"},"tags":[],"id":"85DvaZ1-AeLp","outputId":"47a46d07-cfb1-45c3-be1b-2251a2bb3065"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"All the images for the competition are already sized to 256x256. As these images are RGB images, we set the channel to 3. Additionally, we will scale the images to a [-1, 1]. Since we need to compare the input image $𝐼$\n to the output image $\\hat{𝐼}$\n, it should be readily possible to enforce the pixel values of $\\hat{I}$\n into a simple, known, hard range. Using sigmoid produces values in [0,1]\n, while using tanh does so in [−1,1]\n. However, it is often thought that that tanh is better than sigmoid.","metadata":{"papermill":{"duration":0.011089,"end_time":"2020-08-29T08:27:13.954949","exception":false,"start_time":"2020-08-29T08:27:13.943860","status":"completed"},"tags":[],"id":"TYO-GtxZAeLp"}},{"cell_type":"code","source":"IMAGE_SIZE = [256, 256]\n\ndef decodeNormalizeImage(image):\n    image = tf.image.decode_jpeg(image, channels=3)\n    image = (tf.cast(image, tf.float32) / 127.5) - 1\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    return image\n\ndef readTfrecord(example):\n    tfrecord_format = {\n        \"image_name\": tf.io.FixedLenFeature([], tf.string),\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"target\": tf.io.FixedLenFeature([], tf.string)\n    }\n    example = tf.io.parse_single_example(example, tfrecord_format)\n    image = decodeNormalizeImage(example['image'])\n    return image","metadata":{"execution":{"iopub.execute_input":"2020-08-29T08:27:13.990842Z","iopub.status.busy":"2020-08-29T08:27:13.989787Z","iopub.status.idle":"2020-08-29T08:27:13.993120Z","shell.execute_reply":"2020-08-29T08:27:13.992453Z"},"papermill":{"duration":0.026749,"end_time":"2020-08-29T08:27:13.993244","exception":false,"start_time":"2020-08-29T08:27:13.966495","status":"completed"},"tags":[],"id":"D7d43dbFAeLp"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Define the function to extract the image from the files.","metadata":{"papermill":{"duration":0.011183,"end_time":"2020-08-29T08:27:14.016170","exception":false,"start_time":"2020-08-29T08:27:14.004987","status":"completed"},"tags":[],"id":"lUM53ZOzAeLq"}},{"cell_type":"code","source":"def loadData(filenames):\n    dataset = tf.data.TFRecordDataset(filenames)\n    dataset = dataset.map(readTfrecord, num_parallel_calls=AUTOTUNE)\n    return dataset","metadata":{"execution":{"iopub.execute_input":"2020-08-29T08:27:14.046298Z","iopub.status.busy":"2020-08-29T08:27:14.045511Z","iopub.status.idle":"2020-08-29T08:27:14.048424Z","shell.execute_reply":"2020-08-29T08:27:14.048985Z"},"papermill":{"duration":0.021327,"end_time":"2020-08-29T08:27:14.049161","exception":false,"start_time":"2020-08-29T08:27:14.027834","status":"completed"},"tags":[],"id":"IVmd-lFZAeLq"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's load in our datasets.","metadata":{"papermill":{"duration":0.011381,"end_time":"2020-08-29T08:27:14.072330","exception":false,"start_time":"2020-08-29T08:27:14.060949","status":"completed"},"tags":[],"id":"2GSejXuLAeLq"}},{"cell_type":"code","source":"monetDs = loadData(MONET_FILENAMES).batch(1)\nphotoDs = loadData(PHOTO_FILENAMES).batch(1)","metadata":{"execution":{"iopub.execute_input":"2020-08-29T08:27:14.105277Z","iopub.status.busy":"2020-08-29T08:27:14.103723Z","iopub.status.idle":"2020-08-29T08:27:14.417232Z","shell.execute_reply":"2020-08-29T08:27:14.416589Z"},"papermill":{"duration":0.333224,"end_time":"2020-08-29T08:27:14.417363","exception":false,"start_time":"2020-08-29T08:27:14.084139","status":"completed"},"tags":[],"id":"_WsWFHcSAeLq"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Visualizating the Monet image and photo","metadata":{"papermill":{"duration":0.011344,"end_time":"2020-08-29T08:27:15.780434","exception":false,"start_time":"2020-08-29T08:27:15.769090","status":"completed"},"tags":[],"id":"d9hOGlLMAeLq"}},{"cell_type":"code","source":"exampleMonet = next(iter(monetDs))\nexamplePhoto = next(iter(photoDs))\n\nfig, (ax1, ax2) = plt.subplots(1, 2)\nfig.suptitle('A photo example and a Monet example')\nax1.imshow(examplePhoto[0] * 0.5 + 0.5)\nax1.set_title('Photo')\nax2.imshow(exampleMonet[0] * 0.5 + 0.5)\nax2.set_title('Monet')","metadata":{"execution":{"iopub.execute_input":"2020-08-29T08:27:15.811259Z","iopub.status.busy":"2020-08-29T08:27:15.810410Z","iopub.status.idle":"2020-08-29T08:27:16.258921Z","shell.execute_reply":"2020-08-29T08:27:16.259502Z"},"papermill":{"duration":0.467428,"end_time":"2020-08-29T08:27:16.259679","exception":false,"start_time":"2020-08-29T08:27:15.792251","status":"completed"},"tags":[],"id":"waqDMZQtAeLq","outputId":"50e296ff-7265-4ef2-ea06-6519539e84a0"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model Architecture\n\nThe GAN architecture is an approach to training a model for image synthesis that is comprised of two models: a generator model and a discriminator model. The generator takes a point from a latent space as input and generates new plausible images from the domain, and the discriminator takes an image as input and predicts whether it is real (from a dataset) or fake (generated). Both models are trained, such that the generator is updated to better fool the discriminator and the discriminator is updated to better detect generated images.\n\nTo solve the problem in this project, we will build CycleGAN.The CycleGAN is an extension of the GAN architecture that involves the simultaneous training of **two generator models and two discriminator models**.\n\nOne generator takes images from the first domain as input and outputs images for the second domain, and the other generator takes images from the second domain as input and generates images for the first domain. Discriminator models are then used to determine how plausible the generated images are and update the generator models accordingly.\n\nThis extension alone might be enough to generate plausible images in each domain, but not sufficient to generate translations of the input images.\n\nWe'll be using a UNET architecture for our CycleGAN. To build our generator, let's first define our `downsample` and `upsample` methods. As the name suggests, `downsample` will decrease the number of dimensions of the image, whereas `upsample` will increase the number of dimensions of the image. With downsampling, the stride is the length of the step the filter takes. Since we will set the stride as 2, the filter is applied to every other pixel, hence reducing the weight and height by 2.\n\nWe'll be using an instance normalization instead of batch normalization, as it computes the mean/variance across each channel of each training image. It is used in style transfer applications and has also been suggested as a replacement to batch normalization in GANs.","metadata":{"papermill":{"duration":0.012983,"end_time":"2020-08-29T08:27:16.285923","exception":false,"start_time":"2020-08-29T08:27:16.272940","status":"completed"},"tags":[],"id":"K9CclYqpAeLr"}},{"cell_type":"code","source":"OUTPUT_CHANNELS = 3\n\ndef downsample(filters, size, apply_instancenorm=True):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    result = keras.Sequential()\n    result.add(layers.Conv2D(filters, size, strides=2, padding='same',\n                             kernel_initializer=initializer, use_bias=False))\n\n    if apply_instancenorm:\n        result.add(tfa.layers.InstanceNormalization(gamma_initializer=gamma_init))\n\n    result.add(layers.LeakyReLU())\n\n    return result","metadata":{"execution":{"iopub.execute_input":"2020-08-29T08:27:16.323098Z","iopub.status.busy":"2020-08-29T08:27:16.322049Z","iopub.status.idle":"2020-08-29T08:27:16.325579Z","shell.execute_reply":"2020-08-29T08:27:16.324929Z"},"papermill":{"duration":0.026742,"end_time":"2020-08-29T08:27:16.325757","exception":false,"start_time":"2020-08-29T08:27:16.299015","status":"completed"},"tags":[],"id":"8lkBp3VzAeLr"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def upsample(filters, size, apply_dropout=False):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    result = keras.Sequential()\n    result.add(layers.Conv2DTranspose(filters, size, strides=2,\n                                      padding='same',\n                                      kernel_initializer=initializer,\n                                      use_bias=False))\n\n    result.add(tfa.layers.InstanceNormalization(gamma_initializer=gamma_init))\n\n    if apply_dropout:\n        result.add(layers.Dropout(0.5))\n\n    result.add(layers.ReLU())\n\n    return result","metadata":{"execution":{"iopub.execute_input":"2020-08-29T08:27:16.388875Z","iopub.status.busy":"2020-08-29T08:27:16.388093Z","iopub.status.idle":"2020-08-29T08:27:16.391770Z","shell.execute_reply":"2020-08-29T08:27:16.390974Z"},"papermill":{"duration":0.026678,"end_time":"2020-08-29T08:27:16.391892","exception":false,"start_time":"2020-08-29T08:27:16.365214","status":"completed"},"tags":[],"id":"U1JNPRzdAeLr"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Build the generator\n\nThe generator first downsamples the input image and then upsample while establishing long skip connections. Skip connections are a way to help bypass the vanishing gradient problem by concatenating the output of a layer to multiple layers instead of only one. Here we concatenate the output of the downsample layer to the upsample layer in a symmetrical fashion.","metadata":{"papermill":{"duration":0.012895,"end_time":"2020-08-29T08:27:16.418061","exception":false,"start_time":"2020-08-29T08:27:16.405166","status":"completed"},"tags":[],"id":"CUTxsbMlAeLr"}},{"cell_type":"code","source":"def Generator():\n    inputs = layers.Input(shape=[256,256,3])\n\n    # bs = batch size\n    down_stack = [\n        downsample(64, 4, apply_instancenorm=False), # (bs, 128, 128, 64)\n        downsample(128, 4), # (bs, 64, 64, 128)\n        downsample(256, 4), # (bs, 32, 32, 256)\n        downsample(512, 4), # (bs, 16, 16, 512)\n        downsample(512, 4), # (bs, 8, 8, 512)\n        downsample(512, 4), # (bs, 4, 4, 512)\n        downsample(512, 4), # (bs, 2, 2, 512)\n        downsample(512, 4), # (bs, 1, 1, 512)\n    ]\n\n    up_stack = [\n        upsample(512, 4, apply_dropout=True), # (bs, 2, 2, 1024)\n        upsample(512, 4, apply_dropout=True), # (bs, 4, 4, 1024)\n        upsample(512, 4, apply_dropout=True), # (bs, 8, 8, 1024)\n        upsample(512, 4), # (bs, 16, 16, 1024)\n        upsample(256, 4), # (bs, 32, 32, 512)\n        upsample(128, 4), # (bs, 64, 64, 256)\n        upsample(64, 4), # (bs, 128, 128, 128)\n    ]\n\n    initializer = tf.random_normal_initializer(0., 0.02)\n    last = layers.Conv2DTranspose(OUTPUT_CHANNELS, 4,\n                                  strides=2,\n                                  padding='same',\n                                  kernel_initializer=initializer,\n                                  activation='tanh') # (bs, 256, 256, 3)\n\n    x = inputs\n\n    # Downsampling through the model\n    skips = []\n    for down in down_stack:\n        x = down(x)\n        skips.append(x)\n\n    skips = reversed(skips[:-1])\n\n    # Upsampling and establishing the skip connections\n    for up, skip in zip(up_stack, skips):\n        x = up(x)\n        x = layers.Concatenate()([x, skip])\n\n    x = last(x)\n\n    return keras.Model(inputs=inputs, outputs=x)","metadata":{"execution":{"iopub.execute_input":"2020-08-29T08:27:16.462513Z","iopub.status.busy":"2020-08-29T08:27:16.461714Z","iopub.status.idle":"2020-08-29T08:27:16.465151Z","shell.execute_reply":"2020-08-29T08:27:16.464516Z"},"papermill":{"duration":0.033907,"end_time":"2020-08-29T08:27:16.465273","exception":false,"start_time":"2020-08-29T08:27:16.431366","status":"completed"},"tags":[],"id":"97o9SqDxAeLr"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Build the discriminator\n\nThe discriminator takes in the input image and classifies it as real or fake (generated). Instead of outputing a single node, the discriminator outputs a smaller 2D image with higher pixel values indicating a real classification and lower values indicating a fake classification.","metadata":{"papermill":{"duration":0.012989,"end_time":"2020-08-29T08:27:16.491503","exception":false,"start_time":"2020-08-29T08:27:16.478514","status":"completed"},"tags":[],"id":"5_lTI_J7AeLs"}},{"cell_type":"markdown","source":"Leaky ReLU is used below, as it is very popular because they help the gradients flow easier through the architecture.\n\nA regular ReLU function works by truncating negative values to 0. This has the effect of blocking the gradients to flow through the network. Instead of the function being zero, leaky RELUs allow a small negative value to pass through. That is, the function computes the greatest value between the features and a small factor.\n\nLeaky RELUs represent an attempt to solve the dying ReLU problem. This situation occurs when the neurons get stuck in a state in which RELU units always output 0s for all inputs. For these cases, the gradients are completely shut to flow back through the network. This is especially important for GANs since the only way the generator has to learn is by receiving the gradients from the discriminator.","metadata":{"id":"vrMGwHhket2k"}},{"cell_type":"code","source":"def Discriminator():\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    inp = layers.Input(shape=[256, 256, 3], name='input_image')\n\n    x = inp\n\n    down1 = downsample(64, 4, False)(x) # (bs, 128, 128, 64)\n    down2 = downsample(128, 4)(down1) # (bs, 64, 64, 128)\n    down3 = downsample(256, 4)(down2) # (bs, 32, 32, 256)\n\n    zero_pad1 = layers.ZeroPadding2D()(down3) # (bs, 34, 34, 256)\n    conv = layers.Conv2D(512, 4, strides=1,\n                         kernel_initializer=initializer,\n                         use_bias=False)(zero_pad1) # (bs, 31, 31, 512)\n\n    norm1 = tfa.layers.InstanceNormalization(gamma_initializer=gamma_init)(conv)\n\n    leaky_relu = layers.LeakyReLU()(norm1)\n\n    zero_pad2 = layers.ZeroPadding2D()(leaky_relu) # (bs, 33, 33, 512)\n\n    last = layers.Conv2D(1, 4, strides=1,\n                         kernel_initializer=initializer)(zero_pad2) # (bs, 30, 30, 1)\n\n    return tf.keras.Model(inputs=inp, outputs=last)","metadata":{"execution":{"iopub.execute_input":"2020-08-29T08:27:16.532441Z","iopub.status.busy":"2020-08-29T08:27:16.531535Z","iopub.status.idle":"2020-08-29T08:27:16.534854Z","shell.execute_reply":"2020-08-29T08:27:16.534266Z"},"papermill":{"duration":0.030335,"end_time":"2020-08-29T08:27:16.534977","exception":false,"start_time":"2020-08-29T08:27:16.504642","status":"completed"},"tags":[],"id":"HlSheNLEAeLs"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n    monetGenerator = Generator() # transforms photos to Monet styled paintings\n    photoGenerator = Generator() # transforms Monet paintings to be like photos\n\n    monetDiscriminator = Discriminator() # differentiates real Monet paintings and generated Monet paintings\n    photoDiscriminator = Discriminator() # differentiates real photos and generated photos","metadata":{"execution":{"iopub.execute_input":"2020-08-29T08:27:16.568360Z","iopub.status.busy":"2020-08-29T08:27:16.567608Z","iopub.status.idle":"2020-08-29T08:27:26.527227Z","shell.execute_reply":"2020-08-29T08:27:26.526444Z"},"papermill":{"duration":9.978841,"end_time":"2020-08-29T08:27:26.527361","exception":false,"start_time":"2020-08-29T08:27:16.548520","status":"completed"},"tags":[],"id":"zvfmiXwGAeLs"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Since our generators are not trained yet, the generated Monet-esque photo does not show what is expected at this point.","metadata":{"papermill":{"duration":0.013578,"end_time":"2020-08-29T08:27:26.554960","exception":false,"start_time":"2020-08-29T08:27:26.541382","status":"completed"},"tags":[],"id":"RwC56SWuAeLs"}},{"cell_type":"markdown","source":"### Build the CycleGAN model\n\nCycleGan will extend `tf.keras.Model` so that `fit()` can be called later to train the model. During the training step, the model transforms a photo to a Monet painting and then back to a photo. The difference between the original photo and the twice-transformed photo is the cycle-consistency loss. We want the original photo and the twice-transformed photo to be similar to one another.","metadata":{"papermill":{"duration":0.013939,"end_time":"2020-08-29T08:27:27.319522","exception":false,"start_time":"2020-08-29T08:27:27.305583","status":"completed"},"tags":[],"id":"1XPZFYrqAeLs"}},{"cell_type":"code","source":"class CycleGan(keras.Model):\n    def __init__(\n        self,\n        monetGenerator,\n        photoGenerator,\n        monetDiscriminator,\n        photoDiscriminator,\n        lambdaCycle=10,\n    ):\n        super(CycleGan, self).__init__()\n        self.m_gen = monetGenerator\n        self.p_gen = photoGenerator\n        self.m_disc = monetDiscriminator\n        self.p_disc = photoDiscriminator\n        self.lambdaCycle = lambdaCycle\n        \n    def compile(\n        self,\n        m_gen_optimizer,\n        p_gen_optimizer,\n        m_disc_optimizer,\n        p_disc_optimizer,\n        gen_loss_fn,\n        disc_loss_fn,\n        cycle_loss_fn,\n        identity_loss_fn\n    ):\n        super(CycleGan, self).compile()\n        self.m_gen_optimizer = m_gen_optimizer\n        self.p_gen_optimizer = p_gen_optimizer\n        self.m_disc_optimizer = m_disc_optimizer\n        self.p_disc_optimizer = p_disc_optimizer\n        self.gen_loss_fn = gen_loss_fn\n        self.disc_loss_fn = disc_loss_fn\n        self.cycle_loss_fn = cycle_loss_fn\n        self.identity_loss_fn = identity_loss_fn\n        \n    def train_step(self, batch_data):\n        real_monet, real_photo = batch_data\n        \n        with tf.GradientTape(persistent=True) as tape:\n            # photo to monet back to photo\n            fake_monet = self.m_gen(real_photo, training=True)\n            cycled_photo = self.p_gen(fake_monet, training=True)\n\n            # monet to photo back to monet\n            fake_photo = self.p_gen(real_monet, training=True)\n            cycled_monet = self.m_gen(fake_photo, training=True)\n\n            # generating itself\n            same_monet = self.m_gen(real_monet, training=True)\n            same_photo = self.p_gen(real_photo, training=True)\n\n            # discriminator used to check, inputing real images\n            disc_real_monet = self.m_disc(real_monet, training=True)\n            disc_real_photo = self.p_disc(real_photo, training=True)\n\n            # discriminator used to check, inputing fake images\n            disc_fake_monet = self.m_disc(fake_monet, training=True)\n            disc_fake_photo = self.p_disc(fake_photo, training=True)\n\n            # evaluates generator loss\n            monet_gen_loss = self.gen_loss_fn(disc_fake_monet)\n            photo_gen_loss = self.gen_loss_fn(disc_fake_photo)\n\n            # evaluates total cycle consistency loss\n            total_cycle_loss = self.cycle_loss_fn(real_monet, cycled_monet, self.lambdaCycle) + self.cycle_loss_fn(real_photo, cycled_photo, self.lambdaCycle)\n\n            # evaluates total generator loss\n            total_monet_gen_loss = monet_gen_loss + total_cycle_loss + self.identity_loss_fn(real_monet, same_monet, self.lambdaCycle)\n            total_photo_gen_loss = photo_gen_loss + total_cycle_loss + self.identity_loss_fn(real_photo, same_photo, self.lambdaCycle)\n\n            # evaluates discriminator loss\n            monet_disc_loss = self.disc_loss_fn(disc_real_monet, disc_fake_monet)\n            photo_disc_loss = self.disc_loss_fn(disc_real_photo, disc_fake_photo)\n\n        # Calculate the gradients for generator and discriminator\n        monet_generator_gradients = tape.gradient(total_monet_gen_loss,\n                                                  self.m_gen.trainable_variables)\n        photo_generator_gradients = tape.gradient(total_photo_gen_loss,\n                                                  self.p_gen.trainable_variables)\n\n        monet_discriminator_gradients = tape.gradient(monet_disc_loss,\n                                                      self.m_disc.trainable_variables)\n        photo_discriminator_gradients = tape.gradient(photo_disc_loss,\n                                                      self.p_disc.trainable_variables)\n\n        # Apply the gradients to the optimizer\n        self.m_gen_optimizer.apply_gradients(zip(monet_generator_gradients,\n                                                 self.m_gen.trainable_variables))\n\n        self.p_gen_optimizer.apply_gradients(zip(photo_generator_gradients,\n                                                 self.p_gen.trainable_variables))\n\n        self.m_disc_optimizer.apply_gradients(zip(monet_discriminator_gradients,\n                                                  self.m_disc.trainable_variables))\n\n        self.p_disc_optimizer.apply_gradients(zip(photo_discriminator_gradients,\n                                                  self.p_disc.trainable_variables))\n        \n        return {\n            \"monet_gen_loss\": total_monet_gen_loss,\n            \"photo_gen_loss\": total_photo_gen_loss,\n            \"monet_disc_loss\": monet_disc_loss,\n            \"photo_disc_loss\": photo_disc_loss\n        }","metadata":{"execution":{"iopub.execute_input":"2020-08-29T08:27:27.381412Z","iopub.status.busy":"2020-08-29T08:27:27.380367Z","iopub.status.idle":"2020-08-29T08:27:27.382858Z","shell.execute_reply":"2020-08-29T08:27:27.383391Z"},"papermill":{"duration":0.049519,"end_time":"2020-08-29T08:27:27.383553","exception":false,"start_time":"2020-08-29T08:27:27.334034","status":"completed"},"tags":[],"id":"fM7JG-QwAeLs"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Loss functions\n\nThe discriminator loss function below compares real images to a matrix of 1s and fake images to a matrix of 0s. The perfect discriminator will output all 1s for real images and all 0s for fake images. The discriminator loss outputs the average of the real and generated loss.","metadata":{"papermill":{"duration":0.01417,"end_time":"2020-08-29T08:27:27.412256","exception":false,"start_time":"2020-08-29T08:27:27.398086","status":"completed"},"tags":[],"id":"uGQ-sBO6AeLt"}},{"cell_type":"code","source":"with strategy.scope():\n    def discriminatorLoss(real, generated):\n        real_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(real), real)\n\n        generated_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.zeros_like(generated), generated)\n\n        total_disc_loss = real_loss + generated_loss\n\n        return total_disc_loss * 0.5","metadata":{"execution":{"iopub.execute_input":"2020-08-29T08:27:27.450634Z","iopub.status.busy":"2020-08-29T08:27:27.449541Z","iopub.status.idle":"2020-08-29T08:27:27.452601Z","shell.execute_reply":"2020-08-29T08:27:27.453213Z"},"papermill":{"duration":0.02693,"end_time":"2020-08-29T08:27:27.453371","exception":false,"start_time":"2020-08-29T08:27:27.426441","status":"completed"},"tags":[],"id":"bBw2jDMEAeLt"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The generator wants to fool the discriminator into thinking the generated image is real. The perfect generator will have the discriminator output only 1s. Thus, it compares the generated image to a matrix of 1s to find the loss.","metadata":{"papermill":{"duration":0.014183,"end_time":"2020-08-29T08:27:27.482436","exception":false,"start_time":"2020-08-29T08:27:27.468253","status":"completed"},"tags":[],"id":"uE0-hzDnAeLt"}},{"cell_type":"code","source":"with strategy.scope():\n    def generatorLoss(generated):\n        return tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(generated), generated)","metadata":{"execution":{"iopub.execute_input":"2020-08-29T08:27:27.519607Z","iopub.status.busy":"2020-08-29T08:27:27.518561Z","iopub.status.idle":"2020-08-29T08:27:27.521897Z","shell.execute_reply":"2020-08-29T08:27:27.521139Z"},"papermill":{"duration":0.025103,"end_time":"2020-08-29T08:27:27.522023","exception":false,"start_time":"2020-08-29T08:27:27.496920","status":"completed"},"tags":[],"id":"gzDeqfo4AeLt"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We want our original photo and the twice transformed photo to be similar to one another. Thus, we can calculate the cycle consistency loss be finding the average of their difference.","metadata":{"papermill":{"duration":0.014145,"end_time":"2020-08-29T08:27:27.550631","exception":false,"start_time":"2020-08-29T08:27:27.536486","status":"completed"},"tags":[],"id":"B8VPM3X3AeLt"}},{"cell_type":"code","source":"with strategy.scope():\n    def calcCycleLoss(real_image, cycled_image, LAMBDA):\n        loss1 = tf.reduce_mean(tf.abs(real_image - cycled_image))\n\n        return LAMBDA * loss1","metadata":{"execution":{"iopub.execute_input":"2020-08-29T08:27:27.586626Z","iopub.status.busy":"2020-08-29T08:27:27.585608Z","iopub.status.idle":"2020-08-29T08:27:27.588341Z","shell.execute_reply":"2020-08-29T08:27:27.589010Z"},"papermill":{"duration":0.023987,"end_time":"2020-08-29T08:27:27.589167","exception":false,"start_time":"2020-08-29T08:27:27.565180","status":"completed"},"tags":[],"id":"KHv3Mu1TAeLt"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The identity loss compares the image with its generator (i.e. photo with photo generator). If given a photo as input, we want it to generate the same image as the image was originally a photo. The identity loss compares the input with the output of the generator.","metadata":{"papermill":{"duration":0.014192,"end_time":"2020-08-29T08:27:27.617994","exception":false,"start_time":"2020-08-29T08:27:27.603802","status":"completed"},"tags":[],"id":"Jgrjl-2uAeLt"}},{"cell_type":"code","source":"with strategy.scope():\n    def identityLoss(real_image, same_image, LAMBDA):\n        loss = tf.reduce_mean(tf.abs(real_image - same_image))\n        return LAMBDA * 0.5 * loss","metadata":{"execution":{"iopub.execute_input":"2020-08-29T08:27:27.654107Z","iopub.status.busy":"2020-08-29T08:27:27.653191Z","iopub.status.idle":"2020-08-29T08:27:27.655732Z","shell.execute_reply":"2020-08-29T08:27:27.656297Z"},"papermill":{"duration":0.024098,"end_time":"2020-08-29T08:27:27.656443","exception":false,"start_time":"2020-08-29T08:27:27.632345","status":"completed"},"tags":[],"id":"VrR9TlEpAeLt"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Train the CycleGAN\n\nLet's compile our model. Since we used `tf.keras.Model` to build our CycleGAN, we can just ude the `fit` function to train our model.","metadata":{"papermill":{"duration":0.014013,"end_time":"2020-08-29T08:27:27.685063","exception":false,"start_time":"2020-08-29T08:27:27.671050","status":"completed"},"tags":[],"id":"A5caHfK7AeLu"}},{"cell_type":"code","source":"with strategy.scope():\n    monetGeneratorOptimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    photoGeneratorOptimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n\n    monetDiscriminatorOptimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    photoDiscriminatorOptimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)","metadata":{"execution":{"iopub.execute_input":"2020-08-29T08:27:27.722632Z","iopub.status.busy":"2020-08-29T08:27:27.721700Z","iopub.status.idle":"2020-08-29T08:27:27.724727Z","shell.execute_reply":"2020-08-29T08:27:27.724130Z"},"papermill":{"duration":0.025521,"end_time":"2020-08-29T08:27:27.724855","exception":false,"start_time":"2020-08-29T08:27:27.699334","status":"completed"},"tags":[],"id":"1UFlZbNDAeLu"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n    cycleganModel = CycleGan(\n        monetGenerator, photoGenerator, monetDiscriminator, photoDiscriminator\n    )\n\n    cycleganModel.compile(\n        m_gen_optimizer = monetGeneratorOptimizer,\n        p_gen_optimizer = photoGeneratorOptimizer,\n        m_disc_optimizer = monetDiscriminatorOptimizer,\n        p_disc_optimizer = photoDiscriminatorOptimizer,\n        gen_loss_fn = generatorLoss,\n        disc_loss_fn = discriminatorLoss,\n        cycle_loss_fn = calcCycleLoss,\n        identity_loss_fn = identityLoss\n    )","metadata":{"execution":{"iopub.execute_input":"2020-08-29T08:27:27.769820Z","iopub.status.busy":"2020-08-29T08:27:27.769042Z","iopub.status.idle":"2020-08-29T08:27:27.797882Z","shell.execute_reply":"2020-08-29T08:27:27.797025Z"},"papermill":{"duration":0.058411,"end_time":"2020-08-29T08:27:27.798037","exception":false,"start_time":"2020-08-29T08:27:27.739626","status":"completed"},"tags":[],"id":"8g33dGODAeLu"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cycleganModel.fit(\n    tf.data.Dataset.zip((monetDs, photoDs)),\n    epochs=25\n)","metadata":{"execution":{"iopub.execute_input":"2020-08-29T08:27:27.842380Z","iopub.status.busy":"2020-08-29T08:27:27.836985Z","iopub.status.idle":"2020-08-29T08:45:16.338257Z","shell.execute_reply":"2020-08-29T08:45:16.337486Z"},"papermill":{"duration":1068.52287,"end_time":"2020-08-29T08:45:16.338383","exception":false,"start_time":"2020-08-29T08:27:27.815513","status":"completed"},"tags":[],"id":"Wkn1NnOPAeLu","outputId":"150e1467-846c-423a-ceee-04d7889644e4"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Visualize our Monet styled photos","metadata":{"papermill":{"duration":0.520583,"end_time":"2020-08-29T08:45:17.377538","exception":false,"start_time":"2020-08-29T08:45:16.856955","status":"completed"},"tags":[],"id":"IKupbLZ5AeLu"}},{"cell_type":"code","source":"_, ax = plt.subplots(5, 2, figsize=(24, 24))\nfor i, img in enumerate(photoDs.take(5)):\n    prediction = monetGenerator(img, training=False)[0].numpy()\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    img = (img[0] * 127.5 + 127.5).numpy().astype(np.uint8)\n\n    ax[i, 0].imshow(img)\n    ax[i, 1].imshow(prediction)\n    ax[i, 0].set_title(\"Input Photo\")\n    ax[i, 1].set_title(\"Monet styled\")\n    ax[i, 0].axis(\"off\")\n    ax[i, 1].axis(\"off\")\nplt.show()","metadata":{"execution":{"iopub.execute_input":"2020-08-29T08:45:18.486444Z","iopub.status.busy":"2020-08-29T08:45:18.485675Z","iopub.status.idle":"2020-08-29T08:45:20.822875Z","shell.execute_reply":"2020-08-29T08:45:20.823454Z"},"papermill":{"duration":2.873747,"end_time":"2020-08-29T08:45:20.823619","exception":false,"start_time":"2020-08-29T08:45:17.949872","status":"completed"},"tags":[],"id":"sJ8tMMIQAeLu","outputId":"a76227f0-2450-43b7-a565-505faeb7b3a1"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Create submission file","metadata":{"papermill":{"duration":0.589149,"end_time":"2020-08-29T08:45:21.943852","exception":false,"start_time":"2020-08-29T08:45:21.354703","status":"completed"},"tags":[],"id":"OJJXQ4WvAeLu"}},{"cell_type":"code","source":"! mkdir ../images\ni = 1\nfor img in photoDs:\n    prediction = monetGenerator(img, training=False)[0].numpy()\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    im = PIL.Image.fromarray(prediction)\n    im.save(\"../images/\" + str(i) + \".jpg\")\n    i += 1","metadata":{"execution":{"iopub.execute_input":"2020-08-29T08:45:24.898093Z","iopub.status.busy":"2020-08-29T08:45:24.897216Z","iopub.status.idle":"2020-08-29T09:21:45.221956Z","shell.execute_reply":"2020-08-29T09:21:45.221049Z"},"papermill":{"duration":2180.855246,"end_time":"2020-08-29T09:21:45.222099","exception":false,"start_time":"2020-08-29T08:45:24.366853","status":"completed"},"tags":[],"id":"cK7NIXd3AeLu"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import shutil\nshutil.make_archive(\"/kaggle/working/images\", 'zip', \"/kaggle/images\")","metadata":{"execution":{"iopub.execute_input":"2020-08-29T09:21:46.345701Z","iopub.status.busy":"2020-08-29T09:21:46.344786Z","iopub.status.idle":"2020-08-29T09:21:50.874034Z","shell.execute_reply":"2020-08-29T09:21:50.874615Z"},"papermill":{"duration":5.118423,"end_time":"2020-08-29T09:21:50.874795","exception":false,"start_time":"2020-08-29T09:21:45.756372","status":"completed"},"tags":[],"id":"D5xG0svZAeLu","outputId":"f88bf8a2-65f5-4119-eb29-c3ed6c2f67e3"},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Conclusion\nIn this project, we discovered the CycleGAN technique for unpaired image-to-image translation where we learned CycleGAN is a technique for training unsupervised image translation models via the GAN architecture using unpaired collections of images from two different domains,i.e.,Monet styled and camera taken. If I had more time, I would lile to try using DCGan to genearte Monet styled images from scratch.","metadata":{"id":"7awYMMPMvQUf"}}]}