{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# My CycleGAN is somewhat of a painter itself"},{"metadata":{},"cell_type":"markdown","source":"This notebook is part of the Machine Learning in Practice Course 2021 at Radboud University. With it, we are participating in the \"I am somewhat of a painter myself\" challenge.\n\n**Objective of the challenge**: Build a GAN that generates 7,000 to 10,000 Monet-style images\n\n## Outline of this notebook\n\n0. <a href='#imports'>Imports</a>\n1. <a href='#eda'>Exploratory Data Analysis (EDA) (short version)</a>\n2. <a href='#data_loading'>Data Loading</a>\n3. <a href='#data_augmentation'>Data Augmentation</a>\n4. <a href='#model'>The model</a>\n    1. <a href='#generator'>Generator</a>\n    2. <a href='#discriminator'>Discriminator</a>\n    3. <a href='#cycle'>CycleGan</a>\n    4. <a href='#losses'>Losses</a>\n5. <a href='#training'>Model Training</a>\n6. <a href='#results'>Results</a>\n7. <a href='#eda2'>Exhaustive EDA</a>"},{"metadata":{},"cell_type":"markdown","source":"<a id='imports'></a>\n## Imports"},{"metadata":{"trusted":true},"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nimport tensorflow_addons as tfa\n\nfrom tensorflow.keras.layers import LeakyReLU, BatchNormalization, Conv2D, Add, Layer, Conv2DTranspose, Activation, ZeroPadding2D, Input\nfrom tensorflow_addons.layers import InstanceNormalization\nfrom keras.models import Model\nfrom keras.activations import *\nfrom keras.layers import *\n\nfrom kaggle_datasets import KaggleDatasets\nimport matplotlib.pyplot as plt\nimport numpy as np\n\nimport os, random, json, PIL, shutil, re, imageio, glob\n\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Device:', tpu.master())\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nexcept:\n    strategy = tf.distribute.get_strategy()\nprint('Number of replicas:', strategy.num_replicas_in_sync)\n\nAUTOTUNE = tf.data.experimental.AUTOTUNE\n    \nprint(tf.__version__)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"<a id='eda'></a>\n## Exploratory Data Analysis\n"},{"metadata":{},"cell_type":"markdown","source":"## Visualization of a few images\n\nLet's have a first look at the data we got. All the images for the challenge are of the size 256x256 and sorted by type already"},{"metadata":{"trusted":true},"cell_type":"code","source":"BASE_PATH = '../input/gan-getting-started'\nMONET_PATH = os.path.join(BASE_PATH, 'monet_jpg')\nPHOTO_PATH = os.path.join(BASE_PATH, 'photo_jpg')\n\nimport cv2\nimport math\nimport random\n\n\ndef load_images(paths):\n    images = []\n    for img in paths:\n        try:\n            img = cv2.imread(img)\n            img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        except:\n            print(\"Could not load {}\".format(img))\n        images.append(img)\n    return images\n        \n        \ndef show_folder_info(path):\n    d_image_sizes = {}\n    for image_name in os.listdir(path):\n        image = cv2.imread(os.path.join(path, image_name))\n        d_image_sizes[image.shape] = d_image_sizes.get(image.shape, 0) + 1\n        \n    for size, count in d_image_sizes.items():\n        print(f'shape: {size}\\tcount: {count}')\n        \ndef visualize_images(images, title=None):\n    plt.figure(figsize=(16,16))\n    \n    w = int(len(images) ** .5)\n    h = math.ceil(len(images) / w)\n    \n    if title:\n        plt.suptitle(title)\n        \n    for idx, image in enumerate(images):\n        \n        plt.subplot(h, w, idx + 1)\n        plt.imshow(image)\n        plt.axis('off')\n    \n    \n    \n    plt.show()\n        ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"MONET_IMAGES = [os.path.join(MONET_PATH, file) for file in os.listdir(MONET_PATH)]\nmonet_images = load_images(MONET_IMAGES)\n\nPHOTO_IMAGES = [os.path.join(PHOTO_PATH, file) for file in os.listdir(PHOTO_PATH)]\nphoto_images = load_images(PHOTO_IMAGES)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Monet Style Images"},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_images(random.sample(monet_images,15), \"Samples of the Monet Dataset\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Photos"},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_images(random.sample(photo_images,15), \"Samples of the Photo Dataset\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"*Visual Inspection*\n\nComparing the Monet images and the photos we can obviously see which are paintings and which are photos. The photos have way smoother transitions, the monet images \"jump\" in colours. Monet was a impressionist painter, an art style which is characterized by a small but still visible brush strokes. \nSo our final images should also have this characteristic but showing the content of the content images.  \n\nFurther, the colours of photos are more natural than the colours of the mone style images. This is not too surprising, but we want to keep this in mind when judging the final output of our GAN"},{"metadata":{},"cell_type":"markdown","source":"### Data Stats:"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Monet Picture Overview\")\nshow_folder_info(MONET_PATH)\n\nprint(\"Content Picture Overview\")\nshow_folder_info(PHOTO_PATH)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id='data_loading'></a>\n## Data Loading"},{"metadata":{"trusted":true},"cell_type":"code","source":"GCS_PATH_FULL_DATA = KaggleDatasets().get_gcs_path(\"gan-getting-started\")\nGCS_PATH_CLEAN_DATA = KaggleDatasets().get_gcs_path(\"cleaned-monet-data\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"excluded =  ['05144e306f.jpg',\n             '9d9a4fccfb.jpg',\n             '3283442e33.jpg',\n             'b5c2fe7c4c.jpg',\n             '8ee2933868.jpg',\n             'b1ea5d5a7d.jpg',\n             'cdddf326e3.jpg',\n             'cb50326950.jpg',\n             '23d6aeb485.jpg',\n             '47a0548067.jpg',\n             '9963d64ebf.jpg',\n             '16dabe418c.jpg',\n             'c78b4fa3a9.jpg',\n             '23b07c3769.jpg',\n             '2e0d0e6e19.jpg',\n             '6a03aea8be.jpg']\n\nexcluded_images = load_images([f\"{MONET_PATH}/{img}\" for img in excluded])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We load a cleaned version of the monet dataset. The original had two round images and some really dark and not quite \"Monet-like\" images. The following images were excluded:\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"visualize_images(excluded_images, \"Images excluded from dataset\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"MONET_FILENAMES = tf.io.gfile.glob(str(GCS_PATH_CLEAN_DATA + '/*.tfrec'))\nPHOTO_FILENAMES = tf.io.gfile.glob(str(GCS_PATH_FULL_DATA + '/photo_tfrec/*.tfrec'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The images for this challenges are already cleaned. All are of size 256x256 and have three channels (RGB)"},{"metadata":{"trusted":true},"cell_type":"code","source":"IMAGE_SIZE = [256, 256]\nHEIGHT_IMG = 256\nWIDTH_IMG = 256\nN_CHANNELS = 3 ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Below we define two functions. `read_tfrecord(example)` retrieves the images data from the tf record, `decode_image` transforms it into a `tf.image`"},{"metadata":{"trusted":true},"cell_type":"code","source":"def decode_image(image):\n    image = tf.image.decode_jpeg(image, channels=3)\n    image = (tf.cast(image, tf.float32) / 127.5) - 1\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    return image\n\ndef read_tfrecord(example):\n    tfrecord_format = {\n        \"image_name\": tf.io.FixedLenFeature([], tf.string),\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"target\": tf.io.FixedLenFeature([], tf.string)\n    }\n    example = tf.io.parse_single_example(example, tfrecord_format)\n    image = decode_image(example['image'])\n    return image","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lastly, we create a function that loads the dataset and combines the upper functions"},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_dataset(filenames, labeled=True, ordered=False):\n    dataset = tf.data.TFRecordDataset(filenames)\n    dataset = dataset.map(read_tfrecord, num_parallel_calls=AUTOTUNE)\n    return dataset","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id='data_augmentation'></a>\n## Data Augmentation"},{"metadata":{},"cell_type":"markdown","source":"Data augmentation can help to improve our model. However, we need to make sure to not change the style too much, as otherwise we might decrease the performance of our model. We therefore decided to only apply augmentation strategies which will not change the image style too much. This includes:\n\n- Mirroring\n- Rotation\n- Changes in Saturation\n- Cropping\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"def data_augment_crop(image):\n    p_crop = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    \n    if p_crop > .5:\n        height_crop = 275\n        width_crop = 275\n        image = tf.image.resize(image, [height_crop, width_crop])\n        image = tf.image.random_crop(image, size=[HEIGHT_IMG, WIDTH_IMG, N_CHANNELS])\n        if p_crop > .9:\n            height_crop = 300\n            width_crop = 300\n            image = tf.image.resize(image, [height_crop, width_crop])\n            image = tf.image.random_crop(image, size=[HEIGHT_IMG, WIDTH_IMG, N_CHANNELS])\n            \n    return image\n\ndef data_augment_mirror(image):\n    p_spatial = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n\n    if p_spatial > .5:\n        image = tf.image.random_flip_left_right(image)\n        image = tf.image.random_flip_up_down(image)\n        if p_spatial > .9:\n            image = tf.image.transpose(image)\n            \n    return image\n\ndef data_augment_rotate(image):\n    p_rotate = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n\n    if p_rotate > .9:\n        image = tf.image.rot90(image, k=3) # rotate 270º\n    elif p_rotate > .7:\n        image = tf.image.rot90(image, k=2) # rotate 180º\n    elif p_rotate > .5:\n        image = tf.image.rot90(image, k=1) # rotate 90º\n            \n    return image\n\ndef data_augment_color(image):\n    p_color = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    x = tf.image.random_saturation(image, 0.5, 1.5)\n    \n    if p_color > .5:\n        x = tf.image.random_brightness(x, 0.05)\n    else:\n        x = tf.image.random_contrast(x, 0.5, 1.5)\n\n    return image  ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lastly, we need a function that combines all our data loading functions"},{"metadata":{"trusted":true},"cell_type":"code","source":"def process_dataset(filenames, augment = False, repeat = True, shuffle = False, label = True, batch_size = 1):\n    ds = load_dataset(filenames, labeled = label)\n    \n    if augment:\n        ds = ds.map(data_augment_crop, num_parallel_calls=AUTOTUNE)\n        ds = ds.concatenate(load_dataset(filenames))\n        ds = ds.map(data_augment_mirror, num_parallel_calls=AUTOTUNE)\n        ds = ds.concatenate(load_dataset(filenames))\n        ds = ds.map(data_augment_color, num_parallel_calls=AUTOTUNE)\n        ds = ds.concatenate(load_dataset(filenames))\n        ds = ds.map(data_augment_rotate, num_parallel_calls=AUTOTUNE)\n        ds = ds.concatenate(load_dataset(filenames))\n    if repeat:\n        ds = ds.repeat(count=1)\n    \n    if shuffle:\n        ds = ds.shuffle(2021)\n    \n    ds = ds.batch(batch_size,drop_remainder=True)\n    ds = ds.cache()\n    ds = ds.prefetch(AUTOTUNE)\n    \n    return ds","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_s = 32\nmonet_ds = process_dataset(MONET_FILENAMES, augment = True, shuffle = True, batch_size = batch_s)\nphoto_ds = process_dataset(PHOTO_FILENAMES, augment = False, shuffle = True, batch_size = batch_s)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id='model'></a>\n## The Model"},{"metadata":{},"cell_type":"markdown","source":"<a id='generator'></a>\n### Generator"},{"metadata":{},"cell_type":"markdown","source":"<img src= \"https://imgur.com/AhIJfHj.png\" alt =\"Titanic\" style='width: 500px;'>"},{"metadata":{"trusted":true},"cell_type":"code","source":"def residual_block(x):\n    layer_block = x\n    x = Conv2D(128, (3, 3), strides=1, padding='same', use_bias=False)(x)\n    x = BatchNormalization()(x)\n    x = Activation('relu')(x)\n    x = Conv2D(128, (3, 3), strides=1, padding='same', use_bias=False)(x)\n    x = BatchNormalization()(x)\n    m = layers.add([x, layer_block])\n    return m","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def effnet_block(x_in, ch_in, ch_out):\n    x = Conv2D(ch_in, kernel_size=(1, 1), padding='same', use_bias=False)(x_in) #x_in\n    x = get_post(x)\n\n    x = DepthwiseConv2D(kernel_size=(1, 3), padding='same', use_bias=False)(x)\n    x = get_post(x)\n    x = MaxPool2D(pool_size=(2, 1), strides=(2, 1))(x)\n    \n    x = DepthwiseConv2D(kernel_size=(3, 1), padding='same', use_bias=False)(x)\n    x = get_post(x)\n\n    x = Conv2D(ch_out, kernel_size=(2, 1), strides=(1, 2), padding='same', use_bias=False)(x)\n    x = get_post(x)\n\n    return x\n\ndef get_post(x_in):\n    x = LeakyReLU()(x_in)\n    x = BatchNormalization()(x)\n    return x","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_generator():\n    input_layer = layers.Input(shape=(256, 256, 3), dtype='float32')\n    \n    ef1 = effnet_block(input_layer, 32, 64)\n    ef2 = effnet_block(ef1, 64, 128)\n    \n    r1 = residual_block(ef2)\n    r2 = residual_block(r1)\n    r3 = residual_block(r2)\n    r4 = residual_block(r3)\n    r5 = residual_block(r4)\n\n    d1 = Conv2DTranspose(64, (3, 3), strides=2, padding='same', use_bias=False)(r5)\n    d1 = BatchNormalization()(d1)\n    d1 = LeakyReLU(alpha=0.05)(d1)\n\n    d2 = Conv2DTranspose(32, (3, 3), strides=2, padding='same', use_bias=False)(d1)\n    d2 = BatchNormalization()(d2)\n    d2 = LeakyReLU(alpha=0.05)(d2)\n\n    c1 = Conv2D(3, (9, 9), strides=1, padding='same', use_bias=False)(d2)\n    c1 = BatchNormalization()(c1)\n    \n    output_layer = Activation('tanh')(c1)\n\n    model = Model([input_layer], output_layer)\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"![](http://)<a id='discriminator'></a>\n<div style=\"display: flex; flex-direction: column; align-items: center\">\n    <div>\n    <h3> Discriminator </h3>\n        <p>Our discriminator follows the following architecture:</p></div>\n<img src= \"https://imgur.com/eJliJxm.png\" alt =\"Our Discriminator\" style='width: 300px;'></div>"},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_discriminator():\n    input_layer = layers.Input(shape=[256, 256, 3])\n    input_noise = layers.GaussianNoise(0.2)(input_layer) #0.1 #input_layer\n\n    d1 = layers.Conv2D(64, 4, strides=2, padding='same', use_bias=False)(input_noise)\n    d1 = layers.LeakyReLU(alpha=0.05)(d1)\n    d2 = layers.Conv2D(128, 4, strides=2, padding='same', use_bias=False)(d1)\n    d2 = layers.LeakyReLU(alpha=0.05)(d2)\n    d3 = layers.Conv2D(256, 4, strides=2, padding='same', use_bias=False)(d2)\n    d3 = layers.LeakyReLU(alpha=0.05)(d3)\n    z1 = layers.ZeroPadding2D()(d3)\n    \n    c1 = layers.Conv2D(512, 4, strides=1, use_bias=False)(z1)\n    c1 = layers.BatchNormalization()(c1)\n    c1 = layers.LeakyReLU()(c1)\n    z2 = layers.ZeroPadding2D()(c1)\n\n    out_layer = layers.Conv2D(1, 4, strides=1, use_bias=False)(z2)\n\n    return tf.keras.Model(inputs=input_layer, outputs=out_layer)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Inception Model\n\nTo encourage our model to adapt the style of monet images, we extract features for both generated and original Monet images from middle layers of the Inception V3 model. \nAt a later point we add an additional term to our losses that requires the model to minimize the distance between the layer output of the real monet and the current training sample. \n\nThis way, we hope to capture the pure style features (what may be recognized as brush strokes, colours etc. for humans) and add an additional constraint to the generation process."},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_activation_model():\n    \n    inception_model = tf.keras.applications.InceptionV3(input_shape=(256,256,3),pooling=\"avg\",include_top=False, weights = \"imagenet\", )\n    activations = []\n    \n    # choose a middle layer of the activation layers. Choose on with 160 channels for performance reasons\n    for layer in inception_model.layers:\n        if \"activation\" in layer.name and layer.output.shape[3] == 160:\n            activations.append(layer)\n            \n    # choose the middle of those activations\n    layer = inception_model.get_layer(activations[int(len(activations)/2)].name)\n    \n    print(f'Selected layer {layer} for the activation model')\n    \n    return tf.keras.models.Model(inputs=inception_model.input, outputs=[layer.output])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id='cycle'></a>\n### CycleGan Architecture"},{"metadata":{},"cell_type":"markdown","source":"First, we create the instances of our generator and discriminator"},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    monet_generator = create_generator() # transforms photos to Monet-esque paintings\n    photo_generator = create_generator() # transforms Monet paintings to be more like photos\n\n    monet_discriminator = create_discriminator() # differentiates real Monet paintings and generated Monet paintings\n    photo_discriminator = create_discriminator() # differentiates real photos and generated photos\n    \n    activation_model = create_activation_model()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And then combine everything in the cycleGAN architecture"},{"metadata":{"trusted":true},"cell_type":"code","source":"class CycleGan(keras.Model):\n    def __init__(\n        self,\n        monet_generator,\n        photo_generator,\n        monet_discriminator,\n        photo_discriminator,\n        activation_model,\n        mu,\n        lambda_cycle=10, #25\n    ):\n        super(CycleGan, self).__init__()\n        self.m_gen = monet_generator\n        self.p_gen = photo_generator\n        self.m_disc = monet_discriminator\n        self.p_disc = photo_discriminator\n        \n        # add the inception model to the mix\n        self.inception = activation_model\n        self.inception.trainable = False\n        # the average  monet activation map\n        self.mu_monet = mu\n        \n        self.lambda_cycle = lambda_cycle\n        \n    def compile(\n        self,\n        m_gen_optimizer,\n        p_gen_optimizer,\n        m_disc_optimizer,\n        p_disc_optimizer,\n        gen_loss_fn,\n        gen_loss_photo,\n        disc_loss_fn,\n        cycle_loss_fn,\n        identity_loss_fn\n    ):\n        super(CycleGan, self).compile()\n        self.m_gen_optimizer = m_gen_optimizer\n        self.p_gen_optimizer = p_gen_optimizer\n        self.m_disc_optimizer = m_disc_optimizer\n        self.p_disc_optimizer = p_disc_optimizer\n        self.gen_loss_fn = gen_loss_fn\n        self.gen_loss_photo = gen_loss_photo\n        self.disc_loss_fn = disc_loss_fn\n        self.cycle_loss_fn = cycle_loss_fn\n        self.identity_loss_fn = identity_loss_fn\n        \n    def train_step(self, batch_data):\n        real_monet, real_photo = batch_data\n        \n        with tf.GradientTape(persistent=True) as tape:\n            # photo to monet back to photo\n            fake_monet = self.m_gen(real_photo, training=True)\n            cycled_photo = self.p_gen(fake_monet, training=True)\n\n            # monet to photo back to monet\n            fake_photo = self.p_gen(real_monet, training=True)\n            cycled_monet = self.m_gen(fake_photo, training=True)\n\n            # generating itself\n            same_monet = self.m_gen(real_monet, training=True)\n            same_photo = self.p_gen(real_photo, training=True)\n\n            # discriminator used to check, inputing real images\n            disc_real_monet = self.m_disc(real_monet, training=True)\n            disc_real_photo = self.p_disc(real_photo, training=True)\n\n            # discriminator used to check, inputing fake images\n            disc_fake_monet = self.m_disc(fake_monet, training=True)\n            disc_fake_photo = self.p_disc(fake_photo, training=True)\n\n            #inception model output\n            inception_fake_monet = self.inception(fake_monet)\n            \n            # evaluates generator loss\n            monet_gen_loss = self.gen_loss_fn(disc_fake_monet, inception_fake_monet, self.mu_monet, self.lambda_cycle)\n            photo_gen_loss = self.gen_loss_photo(disc_fake_photo)\n            \n            # evaluates total cycle consistency loss\n            total_cycle_loss = self.cycle_loss_fn(real_monet, cycled_monet, self.lambda_cycle) + self.cycle_loss_fn(real_photo, cycled_photo, self.lambda_cycle)\n\n            # evaluates total generator loss\n            total_monet_gen_loss = monet_gen_loss + total_cycle_loss + self.identity_loss_fn(real_monet, same_monet, self.lambda_cycle)\n            total_photo_gen_loss = photo_gen_loss + total_cycle_loss + self.identity_loss_fn(real_photo, same_photo, self.lambda_cycle)\n\n            # evaluates discriminator loss\n            monet_disc_loss = self.disc_loss_fn(disc_real_monet, disc_fake_monet)\n            photo_disc_loss = self.disc_loss_fn(disc_real_photo, disc_fake_photo)\n\n        # Calculate the gradients for generator and discriminator\n        monet_generator_gradients = tape.gradient(total_monet_gen_loss,\n                                                  self.m_gen.trainable_variables)\n        photo_generator_gradients = tape.gradient(total_photo_gen_loss,\n                                                  self.p_gen.trainable_variables)\n\n        monet_discriminator_gradients = tape.gradient(monet_disc_loss,\n                                                      self.m_disc.trainable_variables)\n        photo_discriminator_gradients = tape.gradient(photo_disc_loss,\n                                                      self.p_disc.trainable_variables)\n\n        # Apply the gradients to the optimizer\n        self.m_gen_optimizer.apply_gradients(zip(monet_generator_gradients,\n                                                 self.m_gen.trainable_variables))\n\n        self.p_gen_optimizer.apply_gradients(zip(photo_generator_gradients,\n                                                 self.p_gen.trainable_variables))\n\n        self.m_disc_optimizer.apply_gradients(zip(monet_discriminator_gradients,\n                                                  self.m_disc.trainable_variables))\n\n        self.p_disc_optimizer.apply_gradients(zip(photo_discriminator_gradients,\n                                                  self.p_disc.trainable_variables))\n        \n        return {\n            \"monet_gen_loss\": total_monet_gen_loss,\n            \"photo_gen_loss\": total_photo_gen_loss,\n            \"monet_disc_loss\": monet_disc_loss,\n            \"photo_disc_loss\": photo_disc_loss\n        }","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id='losses'></a>\n### Losses"},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    def discriminator_loss(real, generated):\n        real_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(real), real)\n\n        generated_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.zeros_like(generated), generated)\n\n        total_disc_loss = real_loss + generated_loss\n\n        return total_disc_loss * 0.5","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    \n    # for the generated monet, combine BCE with our custom loss based on the activation layers\n    def generator_loss(generated, inception_fake, mu, LAMBDA):\n        loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(generated), generated)\n        \n        return loss + LAMBDA * tf.reduce_mean(tf.abs(mu - inception_fake))\n    \n    # fpr the transformation back keep the BCE\n    def generator_loss_photo(generated):\n        return tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(generated), generated)\n        ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    def calc_cycle_loss(real_image, cycled_image, LAMBDA):\n        loss1 = tf.reduce_mean(tf.abs(real_image - cycled_image))\n\n        return LAMBDA * loss1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    def identity_loss(real_image, same_image, LAMBDA):\n        loss = tf.reduce_mean(tf.abs(real_image - same_image))\n        return LAMBDA * 0.5 * loss","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id='training'></a>\n## Training"},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    monet_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    photo_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n\n    monet_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    photo_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Compute the average activation for the monet images. The inception model is just used for feature extraction which means we have to compute the activation map for the original model only once"},{"metadata":{"trusted":true},"cell_type":"code","source":"monet_ds_single_batch = process_dataset(MONET_FILENAMES, augment = False, shuffle = True, batch_size = 1)\n\nactivations = []\n\nfor batch, images in enumerate(monet_ds_single_batch):\n    pred = activation_model(images)\n    activations.append(pred)\n\n# average over all images\nmu_monet = np.mean(activations, axis=0)\n       ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    cycle_gan_model = CycleGan(\n        monet_generator, photo_generator, monet_discriminator, photo_discriminator, activation_model, mu_monet\n    )\n\n    cycle_gan_model.compile(\n        m_gen_optimizer = monet_generator_optimizer,\n        p_gen_optimizer = photo_generator_optimizer,\n        m_disc_optimizer = monet_discriminator_optimizer,\n        p_disc_optimizer = photo_discriminator_optimizer,\n        gen_loss_fn = generator_loss,\n        gen_loss_photo = generator_loss_photo,\n        disc_loss_fn = discriminator_loss,\n        cycle_loss_fn = calc_cycle_loss,\n        identity_loss_fn = identity_loss\n    )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"callback = tf.keras.callbacks.EarlyStopping(monitor='loss', patience=3)\ncycle_gan_model.fit(\n    tf.data.Dataset.zip((monet_ds, photo_ds)),\n    epochs=75,\n    callbacks=[callback]\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id='results'></a>\n## Results"},{"metadata":{},"cell_type":"markdown","source":"Visualization of the results"},{"metadata":{"trusted":true},"cell_type":"code","source":"_, ax = plt.subplots(5, 2, figsize=(50,50))\nds_predict = load_dataset(PHOTO_FILENAMES).batch(1)\nfor i, img in enumerate(ds_predict.take(5)):\n    prediction = monet_generator(img, training=False)[0].numpy()\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    img = (img[0] * 127.5 + 127.5).numpy().astype(np.uint8)\n\n    ax[i, 0].imshow(img)\n    ax[i, 1].imshow(prediction)\n    ax[i, 0].set_title(\"Input Photo\")\n    ax[i, 1].set_title(\"Monet-esque (generated)\")\n    ax[i, 0].axis(\"off\")\n    ax[i, 1].axis(\"off\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import PIL\n! mkdir ../images","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fast_photo_ds = load_dataset(PHOTO_FILENAMES).batch(32*strategy.num_replicas_in_sync).prefetch(32)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\ni = 1\nfor img in fast_photo_ds:\n    prediction = monet_generator(img, training=False).numpy()\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    for pred in prediction:\n        im = PIL.Image.fromarray(pred)\n        im.save(\"../images/\" + str(i) + \".jpg\")\n        i += 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"Generated {i} images for submission\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import shutil\nshutil.make_archive(\"/kaggle/working/images\", 'zip', \"/kaggle/images\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id='eda2'></a>\n## Exhaustive EDA"},{"metadata":{},"cell_type":"markdown","source":"### Histogram analysis\n\nOpenCV uses the BGR channel order, thus the first histogram of an image is its blue value histogram, the second gree and the third red"},{"metadata":{"trusted":true},"cell_type":"code","source":"monet_histograms = [[cv2.calcHist([image],[i],None,[256],[0,256]) for i in range(3)] for image in monet_images]\nphoto_histograms = [[cv2.calcHist([image],[i],None,[256],[0,256]) for i in range(3)] for image in photo_images]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_histograms(img, title):\n    plt.figure(figsize=(16,16))\n    \n    plt.title(title)\n    color = ('b','g','r')\n    \n    \n    w = 4\n    h = int(len(img)/2)\n    \n    idxs = [i for i in range(len(img)*2)[::2]]\n    \n    for idx, image in zip(idxs,img):\n        \n        plt.subplot(h, w, idx + 1)\n        plt.imshow(image)\n        plt.axis('off')\n        \n        plt.subplot(h, w, idx + 2)\n        for i,col in enumerate(color):   \n            histr = cv2.calcHist([image],[i],None,[256],[0,256])\n            plt.plot(histr,color = col)\n            plt.xlim([0,256])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_histograms(random.sample(monet_images,8), \"Histogram for Monet Image\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_histograms(random.sample(photo_images,8), \"Histogram for Photo\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The histograms indicate that \n\n(1) The monet images seem to use on average more colours for every channel. We come to this conclusion, since the mean pixel intensity for the Monet images is higher than for the photos. However, given the amount of photo files, there are several outliers in both directions. \n\n(2) The range of intensity values used for the Monet images is smaller than for the photos. I.e. the Monet images have more \"unused\" intensity values. This may be explained by the fact that photo transitions are way more smooth than the transitions of colours in expressionistic paintings. \n\nHowever, these observations are not too surprising considering that we have paintings and images."},{"metadata":{},"cell_type":"markdown","source":"### Analysis of the image intensity value distributions"},{"metadata":{"trusted":true},"cell_type":"code","source":"color = ('b','g','r')\nmonet_means = {c: [np.mean(x[:,:,i]) for x in monet_images] for i,c in enumerate(color)}\nphoto_means = {c: [np.mean(x[:,:,i]) for x in photo_images] for i,c in enumerate(color)}\n\nmonet_std = {c: [np.std(x[:,:,i]) for x in monet_images] for i,c in enumerate(color)}\nphoto_std = {c: [np.std(x[:,:,i]) for x in photo_images] for i,c in enumerate(color)}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def boxplots_for_comparison(monet, photo, title):\n    for i,c in enumerate(color):\n        fig, ax = plt.subplots()\n        ax.set_title('{} Channel: {}'.format(c.upper(), title))\n        ax.boxplot([monet[c], photo[c]])\n        ax.set_xticklabels([\"Monet\",\"Photo\"])\n        fig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"boxplots_for_comparison(monet_means, photo_means, \"Pixel Value Mean comparison - Monet and Photo\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"boxplots_for_comparison(monet_std, photo_std, \"Pixel Value standard deviation comparison - Monet and Photo\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### \"Frequency\" of intensity values\n\nThe boxplots only tell us the mean and the standard deviation of the intensity values. Now it would be interesting to see if both Monet and the photos make use of the full range of intensity values. That is, are there any empty bins in images? Our assumption is that Monet images might not make use of the full range of intensity values (based on the histograms)"},{"metadata":{"trusted":true},"cell_type":"code","source":"zero_values_monet = {c: [np.count_nonzero(hist[i].ravel()==0) for hist in monet_histograms] for i,c in enumerate(color)}\nzero_values_photo = {c: [np.count_nonzero(hist[i].ravel()==0) for hist in photo_histograms] for i,c in enumerate(color)}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"boxplots_for_comparison(zero_values_monet,zero_values_photo, \"Number of not used intensity values Monet and Photo\" )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}