{"cells":[{"metadata":{"papermill":{"duration":0.015088,"end_time":"2020-09-08T17:15:37.173668","exception":false,"start_time":"2020-09-08T17:15:37.15858","status":"completed"},"tags":[]},"cell_type":"markdown","source":"\n# Introduction and Setup\n\nThis notebook utilizes a CycleGAN architecture to add Monet-style to photos. For this tutorial, we will be using the TFRecord dataset. Import the following packages and change the accelerator to TPU."},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:15:37.213041Z","iopub.status.busy":"2020-09-08T17:15:37.21222Z","iopub.status.idle":"2020-09-08T17:15:47.232794Z","shell.execute_reply":"2020-09-08T17:15:47.232015Z"},"papermill":{"duration":10.044525,"end_time":"2020-09-08T17:15:47.232924","exception":false,"start_time":"2020-09-08T17:15:37.188399","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nimport tensorflow_addons as tfa\n\nfrom kaggle_datasets import KaggleDatasets\nimport matplotlib.pyplot as plt\nimport numpy as np\n\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Device:', tpu.master())\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nexcept:\n    strategy = tf.distribute.get_strategy()\nprint('Number of replicas:', strategy.num_replicas_in_sync)\n\nAUTOTUNE = tf.data.experimental.AUTOTUNE\n    \nprint(tf.__version__)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.011729,"end_time":"2020-09-08T17:15:47.256753","exception":false,"start_time":"2020-09-08T17:15:47.245024","status":"completed"},"tags":[]},"cell_type":"markdown","source":"\n# Load in the data\n\nWe want to keep our photo dataset and our Monet dataset separate. First, load in the filenames of the TFRecords"},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:15:47.299668Z","iopub.status.busy":"2020-09-08T17:15:47.286353Z","iopub.status.idle":"2020-09-08T17:15:48.470075Z","shell.execute_reply":"2020-09-08T17:15:48.469425Z"},"papermill":{"duration":1.20146,"end_time":"2020-09-08T17:15:48.470226","exception":false,"start_time":"2020-09-08T17:15:47.268766","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"GCS_PATH = KaggleDatasets().get_gcs_path()","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:15:48.780718Z","iopub.status.busy":"2020-09-08T17:15:48.77976Z","iopub.status.idle":"2020-09-08T17:15:48.78502Z","shell.execute_reply":"2020-09-08T17:15:48.784458Z"},"papermill":{"duration":0.302529,"end_time":"2020-09-08T17:15:48.785169","exception":false,"start_time":"2020-09-08T17:15:48.48264","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"MONET_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/monet_tfrec/*.tfrec'))\nprint('Monet TFRecord Files:', len(MONET_FILENAMES))\n\nPHOTO_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/photo_tfrec/*.tfrec'))\nprint('Photo TFRecord Files:', len(PHOTO_FILENAMES))","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.011914,"end_time":"2020-09-08T17:15:48.809524","exception":false,"start_time":"2020-09-08T17:15:48.79761","status":"completed"},"tags":[]},"cell_type":"markdown","source":"\n\nAll the images for the competition are already sized to 256x256. As these images are RGB images, set the channel to 3. Additionally, we need to scale the images to a [-1, 1] scale. Because we are building a generative model, we don't need the labels or the image id so we'll only return the image from the TFRecord.\n"},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:15:48.845787Z","iopub.status.busy":"2020-09-08T17:15:48.845036Z","iopub.status.idle":"2020-09-08T17:15:48.847776Z","shell.execute_reply":"2020-09-08T17:15:48.848321Z"},"papermill":{"duration":0.026818,"end_time":"2020-09-08T17:15:48.848511","exception":false,"start_time":"2020-09-08T17:15:48.821693","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"IMAGE_SIZE = [256, 256]\n\ndef decode_image(image):\n    image = tf.image.decode_jpeg(image, channels=3)\n    image = (tf.cast(image, tf.float32) / 127.5) - 1\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    return image\n\ndef read_tfrecord(example):\n    tfrecord_format = {\n        \"image_name\": tf.io.FixedLenFeature([], tf.string),\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"target\": tf.io.FixedLenFeature([], tf.string)\n    }\n    example = tf.io.parse_single_example(example, tfrecord_format)\n    image = decode_image(example['image'])\n    return image\n\n","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.011934,"end_time":"2020-09-08T17:15:48.872658","exception":false,"start_time":"2020-09-08T17:15:48.860724","status":"completed"},"tags":[]},"cell_type":"markdown","source":"\n\nDefine the function to extract the image from the files."},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:15:48.9036Z","iopub.status.busy":"2020-09-08T17:15:48.902852Z","iopub.status.idle":"2020-09-08T17:15:48.906011Z","shell.execute_reply":"2020-09-08T17:15:48.906585Z"},"papermill":{"duration":0.022042,"end_time":"2020-09-08T17:15:48.906747","exception":false,"start_time":"2020-09-08T17:15:48.884705","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def load_dataset(filenames, labeled=True, ordered=False):\n    dataset = tf.data.TFRecordDataset(filenames)\n    dataset = dataset.map(read_tfrecord, num_parallel_calls=AUTOTUNE)\n    return dataset","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.012236,"end_time":"2020-09-08T17:15:48.931485","exception":false,"start_time":"2020-09-08T17:15:48.919249","status":"completed"},"tags":[]},"cell_type":"markdown","source":"Let's load in our datasets."},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:15:48.966284Z","iopub.status.busy":"2020-09-08T17:15:48.964686Z","iopub.status.idle":"2020-09-08T17:15:49.280693Z","shell.execute_reply":"2020-09-08T17:15:49.279961Z"},"papermill":{"duration":0.336655,"end_time":"2020-09-08T17:15:49.280839","exception":false,"start_time":"2020-09-08T17:15:48.944184","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"monet_ds = load_dataset(MONET_FILENAMES, labeled=True).batch(1)\nphoto_ds = load_dataset(PHOTO_FILENAMES, labeled=True).batch(1)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:15:49.312542Z","iopub.status.busy":"2020-09-08T17:15:49.311684Z","iopub.status.idle":"2020-09-08T17:15:50.346796Z","shell.execute_reply":"2020-09-08T17:15:50.346125Z"},"papermill":{"duration":1.053467,"end_time":"2020-09-08T17:15:50.346926","exception":false,"start_time":"2020-09-08T17:15:49.293459","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"example_monet = next(iter(monet_ds))\nexample_photo = next(iter(photo_ds))","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.012299,"end_time":"2020-09-08T17:15:50.371858","exception":false,"start_time":"2020-09-08T17:15:50.359559","status":"completed"},"tags":[]},"cell_type":"markdown","source":"\n\nLet's visualize a photo example and a Monet example."},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:15:50.404969Z","iopub.status.busy":"2020-09-08T17:15:50.404068Z","iopub.status.idle":"2020-09-08T17:15:50.833026Z","shell.execute_reply":"2020-09-08T17:15:50.833625Z"},"papermill":{"duration":0.44894,"end_time":"2020-09-08T17:15:50.83378","exception":false,"start_time":"2020-09-08T17:15:50.38484","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"plt.subplot(121)\nplt.title('Photo')\nplt.imshow(example_photo[0] * 0.5 + 0.5)\n\nplt.subplot(122)\nplt.title('Monet')\nplt.imshow(example_monet[0] * 0.5 + 0.5)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.013628,"end_time":"2020-09-08T17:15:50.86142","exception":false,"start_time":"2020-09-08T17:15:50.847792","status":"completed"},"tags":[]},"cell_type":"markdown","source":"\n# Build the generator\n\nWe'll be using a UNET architecture for our CycleGAN. To build our generator, let's first define our downsample and upsample methods.\n\nThe downsample, as the name suggests, reduces the 2D dimensions, the width and height, of the image by the stride. The stride is the length of the step the filter takes. Since the stride is 2, the filter is applied to every other pixel, hence reducing the weight and height by 2.\n\nWe'll be using an instance normalization instead of batch normalization. As the instance normalization is not standard in the TensorFlow API, we'll use the layer from TensorFlow Add-ons.\n"},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:15:50.899356Z","iopub.status.busy":"2020-09-08T17:15:50.898342Z","iopub.status.idle":"2020-09-08T17:15:50.901949Z","shell.execute_reply":"2020-09-08T17:15:50.901245Z"},"papermill":{"duration":0.026937,"end_time":"2020-09-08T17:15:50.902097","exception":false,"start_time":"2020-09-08T17:15:50.87516","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"OUTPUT_CHANNELS = 3\n\ndef downsample(filters, size, apply_instancenorm=True):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    result = keras.Sequential()\n    result.add(layers.Conv2D(filters, size, strides=2, padding='same',\n                             kernel_initializer=initializer, use_bias=False))\n\n    if apply_instancenorm:\n        result.add(tfa.layers.InstanceNormalization(gamma_initializer=gamma_init))\n\n    result.add(layers.LeakyReLU())\n\n    return result","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.01361,"end_time":"2020-09-08T17:15:50.929868","exception":false,"start_time":"2020-09-08T17:15:50.916258","status":"completed"},"tags":[]},"cell_type":"markdown","source":"Upsample does the opposite of downsample and increases the dimensions of the of the image. Conv2DTranspose does basically the opposite of a Conv2D layer."},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:15:50.969104Z","iopub.status.busy":"2020-09-08T17:15:50.968073Z","iopub.status.idle":"2020-09-08T17:15:50.971388Z","shell.execute_reply":"2020-09-08T17:15:50.970633Z"},"papermill":{"duration":0.027627,"end_time":"2020-09-08T17:15:50.971513","exception":false,"start_time":"2020-09-08T17:15:50.943886","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def upsample(filters, size, apply_dropout=False):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    result = keras.Sequential()\n    result.add(layers.Conv2DTranspose(filters, size, strides=2,\n                                      padding='same',\n                                      kernel_initializer=initializer,\n                                      use_bias=False))\n\n    result.add(tfa.layers.InstanceNormalization(gamma_initializer=gamma_init))\n\n    if apply_dropout:\n        result.add(layers.Dropout(0.5))\n\n    result.add(layers.ReLU())\n\n    return result","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.013599,"end_time":"2020-09-08T17:15:50.999072","exception":false,"start_time":"2020-09-08T17:15:50.985473","status":"completed"},"tags":[]},"cell_type":"markdown","source":"\n\nLet's build our generator!\n\nThe generator first downsamples the input image and then upsample while establishing long skip connections. Skip connections are a way to help bypass the vanishing gradient problem by concatenating the output of a layer to multiple layers instead of only one. Here we concatenate the output of the downsample layer to the upsample layer in a symmetrical fashion."},{"metadata":{"papermill":{"duration":0.013504,"end_time":"2020-09-08T17:15:51.026713","exception":false,"start_time":"2020-09-08T17:15:51.013209","status":"completed"},"tags":[]},"cell_type":"markdown","source":"To build the Generator\n\n    The architecture of generator is a modified U-Net.\n    Each block in the encoder is (Conv -> Batchnorm -> Leaky ReLU)\n    Each block in the decoder is (Transposed Conv -> Batchnorm -> Dropout(applied to the first 3 blocks) -> ReLU)\n    There are skip connections between the encoder and decoder (as in U-Net).\n"},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:15:51.071727Z","iopub.status.busy":"2020-09-08T17:15:51.070958Z","iopub.status.idle":"2020-09-08T17:15:51.074492Z","shell.execute_reply":"2020-09-08T17:15:51.073785Z"},"papermill":{"duration":0.034093,"end_time":"2020-09-08T17:15:51.074629","exception":false,"start_time":"2020-09-08T17:15:51.040536","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def Generator():\n    inputs = layers.Input(shape=[256,256,3])\n\n    # bs = batch size\n    down_stack = [\n        downsample(64, 4, apply_instancenorm=False), # (bs, 128, 128, 64)\n        downsample(128, 4), # (bs, 64, 64, 128)\n        downsample(256, 4), # (bs, 32, 32, 256)\n        downsample(512, 4), # (bs, 16, 16, 512)\n        downsample(512, 4), # (bs, 8, 8, 512)\n        downsample(512, 4), # (bs, 4, 4, 512)\n        downsample(512, 4), # (bs, 2, 2, 512)\n        downsample(512, 4), # (bs, 1, 1, 512)\n    ]\n    \n    up_stack = [\n        upsample(512, 4, apply_dropout=True), # (bs, 2, 2, 1024)\n        upsample(512, 4, apply_dropout=True), # (bs, 4, 4, 1024)\n        upsample(512, 4, apply_dropout=True), # (bs, 8, 8, 1024)\n        upsample(512, 4), # (bs, 16, 16, 1024)\n        upsample(256, 4), # (bs, 32, 32, 512)\n        upsample(128, 4), # (bs, 64, 64, 256)\n        upsample(64, 4), # (bs, 128, 128, 128)\n    ]\n        \n    initializer = tf.random_normal_initializer(0., 0.02)\n    last = layers.Conv2DTranspose(OUTPUT_CHANNELS, 4,\n                                  strides=2,\n                                  padding='same',\n                                  kernel_initializer=initializer,\n                                  activation='tanh') # (bs, 256, 256, 3)\n\n    x = inputs\n\n    # Downsampling through the model\n    skips = []\n    for down in down_stack:\n        x = down(x)\n        skips.append(x)  \n\n    skips = reversed(skips[:-1])\n\n    # Upsampling and establishing the skip connections\n    for up, skip in zip(up_stack, skips):\n        x = up(x)\n        x = layers.Concatenate()([x, skip])\n\n    x = last(x)\n\n    return keras.Model(inputs=inputs, outputs=x)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.013617,"end_time":"2020-09-08T17:15:51.102293","exception":false,"start_time":"2020-09-08T17:15:51.088676","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Build the discriminator\n\nThe discriminator takes in the input image and classifies it as real or fake (generated). Instead of outputing a single node, the discriminator outputs a smaller 2D image with higher pixel values indicating a real classification and lower values indicating a fake classification."},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:15:51.143455Z","iopub.status.busy":"2020-09-08T17:15:51.142689Z","iopub.status.idle":"2020-09-08T17:15:51.14565Z","shell.execute_reply":"2020-09-08T17:15:51.144882Z"},"papermill":{"duration":0.029632,"end_time":"2020-09-08T17:15:51.145781","exception":false,"start_time":"2020-09-08T17:15:51.116149","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"def Discriminator():\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    inp = layers.Input(shape=[256, 256, 3], name='input_image')\n\n    x = inp\n\n    down1 = downsample(64, 4, False)(x) # (bs, 128, 128, 64)\n    down2 = downsample(128, 4)(down1) # (bs, 64, 64, 128)\n    down3 = downsample(256, 4)(down2) # (bs, 32, 32, 256)\n\n    zero_pad1 = layers.ZeroPadding2D()(down3) # (bs, 34, 34, 256)\n    conv = layers.Conv2D(512, 4, strides=1,\n                         kernel_initializer=initializer,\n                         use_bias=False)(zero_pad1) # (bs, 31, 31, 512)\n\n    norm1 = tfa.layers.InstanceNormalization(gamma_initializer=gamma_init)(conv)\n\n    leaky_relu = layers.LeakyReLU()(norm1)\n\n    zero_pad2 = layers.ZeroPadding2D()(leaky_relu) # (bs, 33, 33, 512)\n\n    last = layers.Conv2D(1, 4, strides=1,\n                         kernel_initializer=initializer)(zero_pad2) # (bs, 30, 30, 1)\n\n    return tf.keras.Model(inputs=inp, outputs=last)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:15:51.17997Z","iopub.status.busy":"2020-09-08T17:15:51.179271Z","iopub.status.idle":"2020-09-08T17:16:00.776786Z","shell.execute_reply":"2020-09-08T17:16:00.776064Z"},"papermill":{"duration":9.617183,"end_time":"2020-09-08T17:16:00.77691","exception":false,"start_time":"2020-09-08T17:15:51.159727","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    monet_generator = Generator() # transforms photos to Monet-esque paintings\n    photo_generator = Generator() # transforms Monet paintings to be more like photos\n\n    monet_discriminator = Discriminator() # differentiates real Monet paintings and generated Monet paintings\n    photo_discriminator = Discriminator() # differentiates real photos and generated photos","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:16:00.811289Z","iopub.status.busy":"2020-09-08T17:16:00.81045Z","iopub.status.idle":"2020-09-08T17:16:01.449928Z","shell.execute_reply":"2020-09-08T17:16:01.450543Z"},"papermill":{"duration":0.659665,"end_time":"2020-09-08T17:16:01.4507","exception":false,"start_time":"2020-09-08T17:16:00.791035","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# plotting generator architecture\ntf.keras.utils.plot_model(monet_generator, show_shapes=True, dpi=64)","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:16:01.489012Z","iopub.status.busy":"2020-09-08T17:16:01.488062Z","iopub.status.idle":"2020-09-08T17:16:01.610619Z","shell.execute_reply":"2020-09-08T17:16:01.611229Z"},"papermill":{"duration":0.14485,"end_time":"2020-09-08T17:16:01.611392","exception":false,"start_time":"2020-09-08T17:16:01.466542","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"# plotting discriminator architecture\ntf.keras.utils.plot_model(monet_discriminator, show_shapes=True, dpi=64)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.016205,"end_time":"2020-09-08T17:16:01.644421","exception":false,"start_time":"2020-09-08T17:16:01.628216","status":"completed"},"tags":[]},"cell_type":"markdown","source":"\n\nSince our generators are not trained yet, the generated Monet-esque photo does not show what is expected at this point.\n"},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:16:01.689942Z","iopub.status.busy":"2020-09-08T17:16:01.688996Z","iopub.status.idle":"2020-09-08T17:16:02.365103Z","shell.execute_reply":"2020-09-08T17:16:02.365698Z"},"papermill":{"duration":0.705172,"end_time":"2020-09-08T17:16:02.365855","exception":false,"start_time":"2020-09-08T17:16:01.660683","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"to_monet = monet_generator(example_photo)\n\nplt.subplot(1, 2, 1)\nplt.title(\"Original Photo\")\nplt.imshow(example_photo[0] * 0.5 + 0.5)\n\nplt.subplot(1, 2, 2)\nplt.title(\"Monet-esque Photo\")\nplt.imshow(to_monet[0] * 0.5 + 0.5)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.01653,"end_time":"2020-09-08T17:16:02.399668","exception":false,"start_time":"2020-09-08T17:16:02.383138","status":"completed"},"tags":[]},"cell_type":"markdown","source":"\n# Build the CycleGAN model\n\nWe will subclass a tf.keras.Model so that we can run fit() later to train our model. During the training step, the model transforms a photo to a Monet painting and then back to a photo. The difference between the original photo and the twice-transformed photo is the cycle-consistency loss. We want the original photo and the twice-transformed photo to be similar to one another.\n\nOverriding the training step function of the Model class. This is the function that is called by fit() for every batch of data.For this,\nWe create a new class that subclasses keras.Model.\nWe just override the method train_step(self, data).\n\nAnd also overriding compile() to use its own signature.\n\nIf you pass a tf.data.Dataset, by calling fit(dataset, ...), then data will be what gets yielded by dataset at each batch.\n\nThe losses are defined in the next section."},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:16:02.444344Z","iopub.status.busy":"2020-09-08T17:16:02.443234Z","iopub.status.idle":"2020-09-08T17:16:02.469015Z","shell.execute_reply":"2020-09-08T17:16:02.46841Z"},"papermill":{"duration":0.052324,"end_time":"2020-09-08T17:16:02.469166","exception":false,"start_time":"2020-09-08T17:16:02.416842","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"class CycleGan(keras.Model):\n    def __init__(\n        self,\n        monet_generator,\n        photo_generator,\n        monet_discriminator,\n        photo_discriminator,\n        lambda_cycle=10,\n    ):\n        super(CycleGan, self).__init__()\n        self.m_gen = monet_generator\n        self.p_gen = photo_generator\n        self.m_disc = monet_discriminator\n        self.p_disc = photo_discriminator\n        self.lambda_cycle = lambda_cycle\n        \n    def compile(\n        self,\n        m_gen_optimizer,\n        p_gen_optimizer,\n        m_disc_optimizer,\n        p_disc_optimizer,\n        gen_loss_fn,\n        disc_loss_fn,\n        cycle_loss_fn,\n        identity_loss_fn\n    ):\n        super(CycleGan, self).compile()\n        self.m_gen_optimizer = m_gen_optimizer\n        self.p_gen_optimizer = p_gen_optimizer\n        self.m_disc_optimizer = m_disc_optimizer\n        self.p_disc_optimizer = p_disc_optimizer\n        self.gen_loss_fn = gen_loss_fn\n        self.disc_loss_fn = disc_loss_fn\n        self.cycle_loss_fn = cycle_loss_fn\n        self.identity_loss_fn = identity_loss_fn\n        \n    def train_step(self, batch_data): # it is called for model.fit function\n        real_monet, real_photo = batch_data\n       \n    \n        # Run the forward pass of the layer.\n        # The operations that the layer applies to its inputs are going to be recorded on the GradientTape.\n\n        with tf.GradientTape(persistent=True) as tape:\n            # photo to monet back to photo\n            fake_monet = self.m_gen(real_photo, training=True)  # Forward pass\n            cycled_photo = self.p_gen(fake_monet, training=True)\n\n            # monet to photo back to monet\n            fake_photo = self.p_gen(real_monet, training=True)\n            cycled_monet = self.m_gen(fake_photo, training=True)\n\n            # generating itself\n            same_monet = self.m_gen(real_monet, training=True)\n            same_photo = self.p_gen(real_photo, training=True)\n\n            # discriminator used to check, inputing real images\n            disc_real_monet = self.m_disc(real_monet, training=True)\n            disc_real_photo = self.p_disc(real_photo, training=True)\n\n            # discriminator used to check, inputing fake images\n            disc_fake_monet = self.m_disc(fake_monet, training=True)\n            disc_fake_photo = self.p_disc(fake_photo, training=True)\n\n            # evaluates generator loss\n            monet_gen_loss = self.gen_loss_fn(disc_fake_monet)\n            photo_gen_loss = self.gen_loss_fn(disc_fake_photo)\n\n            # evaluates total cycle consistency loss\n            total_cycle_loss = self.cycle_loss_fn(real_monet, cycled_monet, self.lambda_cycle) + self.cycle_loss_fn(real_photo, cycled_photo, self.lambda_cycle)\n\n            # evaluates total generator loss\n            total_monet_gen_loss = monet_gen_loss + total_cycle_loss + self.identity_loss_fn(real_monet, same_monet, self.lambda_cycle)\n            total_photo_gen_loss = photo_gen_loss + total_cycle_loss + self.identity_loss_fn(real_photo, same_photo, self.lambda_cycle)\n\n            # evaluates discriminator loss\n            monet_disc_loss = self.disc_loss_fn(disc_real_monet, disc_fake_monet)\n            photo_disc_loss = self.disc_loss_fn(disc_real_photo, disc_fake_photo)\n\n        # Calculate the gradients for generator and discriminator\n        monet_generator_gradients = tape.gradient(total_monet_gen_loss,\n                                                  self.m_gen.trainable_variables)\n        photo_generator_gradients = tape.gradient(total_photo_gen_loss,\n                                                  self.p_gen.trainable_variables)\n\n        monet_discriminator_gradients = tape.gradient(monet_disc_loss,\n                                                      self.m_disc.trainable_variables)\n        photo_discriminator_gradients = tape.gradient(photo_disc_loss,\n                                                      self.p_disc.trainable_variables)\n\n        # Apply the gradients to the optimizer;Update weights\n        # Run one step of gradient descent by updating the value of the variables to minimize the loss.\n        self.m_gen_optimizer.apply_gradients(zip(monet_generator_gradients,\n                                                 self.m_gen.trainable_variables))\n\n        self.p_gen_optimizer.apply_gradients(zip(photo_generator_gradients,\n                                                 self.p_gen.trainable_variables))\n\n        self.m_disc_optimizer.apply_gradients(zip(monet_discriminator_gradients,\n                                                  self.m_disc.trainable_variables))\n\n        self.p_disc_optimizer.apply_gradients(zip(photo_discriminator_gradients,\n                                                  self.p_disc.trainable_variables))\n        \n        return {\n            \"monet_gen_loss\": total_monet_gen_loss,\n            \"photo_gen_loss\": total_photo_gen_loss,\n            \"monet_disc_loss\": monet_disc_loss,\n            \"photo_disc_loss\": photo_disc_loss\n        }","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.017062,"end_time":"2020-09-08T17:16:02.503701","exception":false,"start_time":"2020-09-08T17:16:02.486639","status":"completed"},"tags":[]},"cell_type":"markdown","source":"\n# Define loss functions\n\nThe discriminator loss function below compares real images to a matrix of 1s and fake images to a matrix of 0s. The perfect discriminator will output all 1s for real images and all 0s for fake images. The discriminator loss outputs the average of the real and generated loss.\n"},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:16:02.547741Z","iopub.status.busy":"2020-09-08T17:16:02.546724Z","iopub.status.idle":"2020-09-08T17:16:02.549994Z","shell.execute_reply":"2020-09-08T17:16:02.549295Z"},"papermill":{"duration":0.028944,"end_time":"2020-09-08T17:16:02.550121","exception":false,"start_time":"2020-09-08T17:16:02.521177","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    def discriminator_loss(real, generated):\n        real_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(real), real)\n\n        generated_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.zeros_like(generated), generated)\n\n        total_disc_loss = real_loss + generated_loss\n\n        return total_disc_loss * 0.5","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.016698,"end_time":"2020-09-08T17:16:02.584007","exception":false,"start_time":"2020-09-08T17:16:02.567309","status":"completed"},"tags":[]},"cell_type":"markdown","source":"Generator loss is a sigmoid cross entropy loss of the generated images and an array of ones.\n\nThe generator wants to fool the discriminator into thinking the generated image is real. The perfect generator will have the discriminator output only 1s. Thus, it compares the generated image to a matrix of 1s to find the loss."},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:16:02.6251Z","iopub.status.busy":"2020-09-08T17:16:02.624398Z","iopub.status.idle":"2020-09-08T17:16:02.627479Z","shell.execute_reply":"2020-09-08T17:16:02.6269Z"},"papermill":{"duration":0.026354,"end_time":"2020-09-08T17:16:02.627598","exception":false,"start_time":"2020-09-08T17:16:02.601244","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    def generator_loss(generated):\n        return tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(generated), generated)\n","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.016913,"end_time":"2020-09-08T17:16:02.661631","exception":false,"start_time":"2020-09-08T17:16:02.644718","status":"completed"},"tags":[]},"cell_type":"markdown","source":"Cycle consistency means the result should be close to the original input.\n\nWe want our original photo and the twice transformed photo to be similar to one another. Thus, we can calculate the cycle consistency loss be finding the average of their difference."},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:16:02.702749Z","iopub.status.busy":"2020-09-08T17:16:02.702037Z","iopub.status.idle":"2020-09-08T17:16:02.704921Z","shell.execute_reply":"2020-09-08T17:16:02.705516Z"},"papermill":{"duration":0.026606,"end_time":"2020-09-08T17:16:02.705658","exception":false,"start_time":"2020-09-08T17:16:02.679052","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    def calc_cycle_loss(real_image, cycled_image, LAMBDA):\n        loss1 = tf.reduce_mean(tf.abs(real_image - cycled_image))\n\n        return LAMBDA * loss1","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.016772,"end_time":"2020-09-08T17:16:02.739948","exception":false,"start_time":"2020-09-08T17:16:02.723176","status":"completed"},"tags":[]},"cell_type":"markdown","source":"The identity loss compares the image with its generator (i.e. photo with photo generator). If given a photo as input, we want it to generate the same image as the image was originally a photo. The identity loss compares the input with the output of the generator.\n\nIf Generator G is responsible for translating image X to image Y. Identity loss says that, if you fed image Y to generator G, it should yield the real image Y or something close to image Y."},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:16:02.780857Z","iopub.status.busy":"2020-09-08T17:16:02.780144Z","iopub.status.idle":"2020-09-08T17:16:02.783417Z","shell.execute_reply":"2020-09-08T17:16:02.782712Z"},"papermill":{"duration":0.026262,"end_time":"2020-09-08T17:16:02.783531","exception":false,"start_time":"2020-09-08T17:16:02.757269","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    def identity_loss(real_image, same_image, LAMBDA):\n        loss = tf.reduce_mean(tf.abs(real_image - same_image))\n        return LAMBDA * 0.5 * loss","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.017075,"end_time":"2020-09-08T17:16:02.817782","exception":false,"start_time":"2020-09-08T17:16:02.800707","status":"completed"},"tags":[]},"cell_type":"markdown","source":"Initialize the optimizers for all the generators and the discriminators.\n\nLet's compile our model. Since we used tf.keras.Model to build our CycleGAN, we can just use the fit function to train our model."},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:16:02.860582Z","iopub.status.busy":"2020-09-08T17:16:02.859813Z","iopub.status.idle":"2020-09-08T17:16:02.863176Z","shell.execute_reply":"2020-09-08T17:16:02.862587Z"},"papermill":{"duration":0.028066,"end_time":"2020-09-08T17:16:02.863299","exception":false,"start_time":"2020-09-08T17:16:02.835233","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    monet_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    photo_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n\n    monet_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    photo_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.017126,"end_time":"2020-09-08T17:16:02.89811","exception":false,"start_time":"2020-09-08T17:16:02.880984","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Training\n\n- For each example input generate an output.\n- The discriminator receives the generated image as the first input. The second input is the target_image.\n- Next, we calculate the generator and the discriminator loss.\n- Then, we calculate the gradients of loss with respect to both the generator and the discriminator variables(inputs) and apply those to the optimizer.\n\n\n   Steps for training:\n   \n    Forward pass the image input through keras.model whenever .fit is called\n    Calculate the loss.\n    Calculate the gradients using backpropagation.\n    Apply the gradients to the optimizer.\n"},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:16:02.952486Z","iopub.status.busy":"2020-09-08T17:16:02.948423Z","iopub.status.idle":"2020-09-08T17:16:02.978024Z","shell.execute_reply":"2020-09-08T17:16:02.978679Z"},"papermill":{"duration":0.063109,"end_time":"2020-09-08T17:16:02.978847","exception":false,"start_time":"2020-09-08T17:16:02.915738","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    cycle_gan_model = CycleGan(\n        monet_generator, photo_generator, monet_discriminator, photo_discriminator\n    )\n\n    cycle_gan_model.compile(\n        m_gen_optimizer = monet_generator_optimizer,\n        p_gen_optimizer = photo_generator_optimizer,\n        m_disc_optimizer = monet_discriminator_optimizer,\n        p_disc_optimizer = photo_discriminator_optimizer,\n        gen_loss_fn = generator_loss,\n        disc_loss_fn = discriminator_loss,\n        cycle_loss_fn = calc_cycle_loss,\n        identity_loss_fn = identity_loss\n    )","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:16:03.022931Z","iopub.status.busy":"2020-09-08T17:16:03.022205Z","iopub.status.idle":"2020-09-08T17:16:03.028052Z","shell.execute_reply":"2020-09-08T17:16:03.027371Z"},"papermill":{"duration":0.029195,"end_time":"2020-09-08T17:16:03.028193","exception":false,"start_time":"2020-09-08T17:16:02.998998","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":" tf.data.Dataset.zip((monet_ds, photo_ds))","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:16:03.070193Z","iopub.status.busy":"2020-09-08T17:16:03.069137Z","iopub.status.idle":"2020-09-08T17:33:41.251565Z","shell.execute_reply":"2020-09-08T17:33:41.250428Z"},"papermill":{"duration":1058.205658,"end_time":"2020-09-08T17:33:41.2517","exception":false,"start_time":"2020-09-08T17:16:03.046042","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"cycle_gan_model.fit(\n    tf.data.Dataset.zip((monet_ds, photo_ds)),\n    epochs=25\n)","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.533754,"end_time":"2020-09-08T17:33:42.315522","exception":false,"start_time":"2020-09-08T17:33:41.781768","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Visualize our Monet-esque photos"},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:33:43.458021Z","iopub.status.busy":"2020-09-08T17:33:43.456819Z","iopub.status.idle":"2020-09-08T17:33:45.759796Z","shell.execute_reply":"2020-09-08T17:33:45.760466Z"},"papermill":{"duration":2.854605,"end_time":"2020-09-08T17:33:45.760653","exception":false,"start_time":"2020-09-08T17:33:42.906048","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"_, ax = plt.subplots(5, 2, figsize=(15, 15))\nfor i, img in enumerate(photo_ds.take(5)):\n    prediction = monet_generator(img, training=False)[0].numpy()\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    img = (img[0] * 127.5 + 127.5).numpy().astype(np.uint8)\n\n    ax[i, 0].imshow(img)\n    ax[i, 1].imshow(prediction)\n    ax[i, 0].set_title(\"Input Photo\")\n    ax[i, 1].set_title(\"Monet-esque\")\n    ax[i, 0].axis(\"off\")\n    ax[i, 1].axis(\"off\")\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{"papermill":{"duration":0.576315,"end_time":"2020-09-08T17:33:46.867255","exception":false,"start_time":"2020-09-08T17:33:46.29094","status":"completed"},"tags":[]},"cell_type":"markdown","source":"# Create submission file"},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:33:47.960062Z","iopub.status.busy":"2020-09-08T17:33:47.95915Z","iopub.status.idle":"2020-09-08T17:33:48.744827Z","shell.execute_reply":"2020-09-08T17:33:48.744107Z"},"papermill":{"duration":1.347043,"end_time":"2020-09-08T17:33:48.744948","exception":false,"start_time":"2020-09-08T17:33:47.397905","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"import PIL\n! mkdir ../images","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T17:33:49.876756Z","iopub.status.busy":"2020-09-08T17:33:49.875951Z","iopub.status.idle":"2020-09-08T18:08:57.214804Z","shell.execute_reply":"2020-09-08T18:08:57.213953Z"},"papermill":{"duration":2107.887255,"end_time":"2020-09-08T18:08:57.214984","exception":false,"start_time":"2020-09-08T17:33:49.327729","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"from tqdm import tqdm\ni = 1\nfor img in tqdm(photo_ds):\n    prediction = monet_generator(img, training=False)[0].numpy()\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    im = PIL.Image.fromarray(prediction)\n    im.save(\"../images/\" + str(i) + \".jpg\")\n    i += 1","execution_count":null,"outputs":[]},{"metadata":{"execution":{"iopub.execute_input":"2020-09-08T18:08:59.259214Z","iopub.status.busy":"2020-09-08T18:08:59.258351Z","iopub.status.idle":"2020-09-08T18:09:03.845527Z","shell.execute_reply":"2020-09-08T18:09:03.844795Z"},"papermill":{"duration":5.55879,"end_time":"2020-09-08T18:09:03.84565","exception":false,"start_time":"2020-09-08T18:08:58.28686","status":"completed"},"tags":[],"trusted":true},"cell_type":"code","source":"import shutil\nshutil.make_archive(\"/kaggle/working/images\", 'zip', \"/kaggle/images\")","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}