{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"---------------------------------------\n# Introduction","metadata":{}},{"cell_type":"markdown","source":"**Neural Style Transfer**\n\n> Neural Style Transfer (NST) refers to a class of software algorithms that manipulate digital images, or videos, in order to adopt the appearance or visual style of another image.\n>\n> ![](https://upload.wikimedia.org/wikipedia/commons/a/a2/Mona_lisa_the_starry_night_o_lbfgs_i_content_h_720_m_vgg19_cw_100000.0_sw_30000.0_tv_1.0.jpg)\n>\n> Mona Lisa in the style of \"The Starry Night\" using neural style transfer.\n>\n> **NST**\n>\n> NST was first published in the paper \"A Neural Algorithm of Artistic Style\" by Leon Gatys et al.\n>\n> NST is based on histogram-based texture synthesis algorithms, notably the method of Portilla and Simoncelli. NST can be summarized as histogram-based texture synthesis with convolutional neural network (CNN) features for the image analogies problem. The original paper used a VGG-19 architecture that has been pre-trained to perform object recognition using the ImageNet dataset.\n\nRef: https://en.wikipedia.org/wiki/Neural_Style_Transfer","metadata":{}},{"cell_type":"markdown","source":"## A brief overview of the approaches and resources\n\nThere are two main approaches to NST:\n1. The first one is based on the original [paper by Gatys et al.](https://arxiv.org/abs/1508.06576) and its subsequent improvements and developments. The main idea here is to take the content image and change it in a way to make its low-level features similar to the style image.\n\n    Roughly, the development of these methods can be summarized like this:\n    * 2015 original Gatys' approach optimizing an image directly ([paper](https://arxiv.org/abs/1508.06576), [keras tutorial](https://keras.io/examples/generative/neural_style_transfer/), [pytorch notebook](https://www.kaggle.com/code/ohseokkim/transfering-style)) ->\n    * 2016 Johnson et al. ([paper](https://link.springer.com/chapter/10.1007/978-3-319-46475-6_43)); Ulyanov et al. ([paper](https://arxiv.org/abs/1603.03417)) started training a network that transforms initial image into new one (added speed but wasn't flexible). It could be a good fit for this particular competition, since we are asked to transfer only one style. ->\n    * 2017 ConditionalIN ([paper](https://arxiv.org/abs/1610.07629)) allowed transferring different styles with one network by learning $\\mu$ and $\\sigma$ ->\n    * 2017 AdaIN ([paper](https://arxiv.org/pdf/1703.06868.pdf), [tutorial](https://keras.io/examples/generative/adain/)) allowed transferring any style by simply useing $\\mu$ and $\\sigma$ of the style image. Howeer, it still needs a decoder trained using style images, so the performance on unseen styles could be worse ->\n    * 2017 Universal Style Transfer via Feature Transforms ([blog](https://towardsdatascience.com/universal-style-transfer-b26ba6760040), [paper](https://arxiv.org/abs/1705.08086)) allows transferring any style while the whole process of training does not use any style images.\n    \n    [A nice video describing this.](https://www.youtube.com/watch?v=8pp0Oa3t52s&t=536s&ab_channel=TheAIEpiphany)\n\n2. The second approach uses Generative Adversarial Networks (GANs) to create an image that is similar to the images of chosen style but preserves the content of the original image. Probably, the most influential work here is CycleGANs  and its developments ([paper](https://arxiv.org/abs/1703.10593) and authors' awesome [project page](https://junyanz.github.io/CycleGAN/) with lots of resources, [keras](https://keras.io/examples/generative/cyclegan/) and [tf](https://www.tensorflow.org/tutorials/generative/cyclegan) tutorials).\n\n[Collection of Keras tutorials, some of which are devoted to NST.](https://keras.io/examples/generative/)\n\nIn this work, we will explore original **Gatys'** approach, **AdaIN** method, and **CycleGANs**.\n","metadata":{}},{"cell_type":"markdown","source":"# 0. Setup and some image exploration\n\nThis part is heavily copy-pasted from [this notedook](https://www.kaggle.com/code/ihelon/monet-visualization-and-augmentation), all credits should go to the author.","metadata":{}},{"cell_type":"code","source":"import os\nimport math\nimport random\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport cv2","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:08:57.035626Z","iopub.execute_input":"2022-05-16T15:08:57.036008Z","iopub.status.idle":"2022-05-16T15:08:57.229064Z","shell.execute_reply.started":"2022-05-16T15:08:57.035916Z","shell.execute_reply":"2022-05-16T15:08:57.228167Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def set_seed(seed):\n    random.seed(seed)\n    os.environ[\"PYTHONHASHSEED\"] = str(seed)\n    np.random.seed(seed)\n\nSEED = 42\nset_seed(SEED)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:08:57.23097Z","iopub.execute_input":"2022-05-16T15:08:57.231294Z","iopub.status.idle":"2022-05-16T15:08:57.237359Z","shell.execute_reply.started":"2022-05-16T15:08:57.231257Z","shell.execute_reply":"2022-05-16T15:08:57.236553Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"BASE_PATH = '/kaggle/input'\n#BASE_PATH = '.'\nfor dirname, _, filenames in os.walk(BASE_PATH):\n    print(dirname)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-05-16T15:08:57.239105Z","iopub.execute_input":"2022-05-16T15:08:57.239728Z","iopub.status.idle":"2022-05-16T15:09:20.491877Z","shell.execute_reply.started":"2022-05-16T15:08:57.23969Z","shell.execute_reply":"2022-05-16T15:09:20.490989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"MONET_PATH_JPG = os.path.join(BASE_PATH, \"gan-getting-started\", \"monet_jpg\")\nPHOTO_PATH_JPG = os.path.join(BASE_PATH, \"gan-getting-started\",\"photo_jpg\")\nSTYLES_FOR_NST = os.path.join(BASE_PATH, \"neural-style-transfer\", \"Style Images\")\nARTS_FOR_NST = os.path.join(BASE_PATH, \"images-for-style-transfer\", \"Data\", \"Artworks\")\nBEST_ARTS_FOR_NST = os.path.join(BASE_PATH, \"best-artworks-of-all-time\", \"resized\", \"resized\")","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:09:20.494245Z","iopub.execute_input":"2022-05-16T15:09:20.494766Z","iopub.status.idle":"2022-05-16T15:09:20.501107Z","shell.execute_reply.started":"2022-05-16T15:09:20.49472Z","shell.execute_reply":"2022-05-16T15:09:20.500363Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def print_folder_statistics(path):\n    d_image_sizes = {}\n    max_to_print = 20\n    try:\n        for image_name in os.listdir(path):\n            image = cv2.imread(os.path.join(path, image_name))\n            d_image_sizes[image.shape] = d_image_sizes.get(image.shape, 0) + 1\n            if len(d_image_sizes) >= max_to_print:\n                break\n        for i, (size, count) in enumerate(d_image_sizes.items()):\n            print(f\"shape: {size}\\tcount: {count}\")\n            if i >= max_to_print:\n                break\n    except:\n        pass\n\nprint(f\"Monet images:\")\nprint_folder_statistics(MONET_PATH_JPG)\nprint(\"-\" * 10)\nprint(f\"Photo images:\")\nprint_folder_statistics(PHOTO_PATH_JPG)\nprint(\"-\" * 10)\nprint(f\"NST style images:\")\nprint_folder_statistics(STYLES_FOR_NST)\nprint(\"-\" * 10)\nprint(f\"Artwork images:\")\nprint_folder_statistics(ARTS_FOR_NST)\nprint(\"-\" * 10)\nprint(f\"Best Artworks:\")\nprint_folder_statistics(BEST_ARTS_FOR_NST)\nprint(\"-\" * 10)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-05-16T15:09:20.503787Z","iopub.execute_input":"2022-05-16T15:09:20.504671Z","iopub.status.idle":"2022-05-16T15:12:30.049587Z","shell.execute_reply.started":"2022-05-16T15:09:20.504635Z","shell.execute_reply":"2022-05-16T15:12:30.048684Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# let's save required size\nIMAGE_SIZE = 256","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:12:30.05281Z","iopub.execute_input":"2022-05-16T15:12:30.05306Z","iopub.status.idle":"2022-05-16T15:12:30.057012Z","shell.execute_reply.started":"2022-05-16T15:12:30.053032Z","shell.execute_reply":"2022-05-16T15:12:30.056196Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def batch_visualization(path, n_images, is_random=True, figsize=(16, 16)):\n    plt.figure(figsize=figsize)    \n    w = int(n_images ** .5)\n    h = math.ceil(n_images / w)\n    all_names = os.listdir(path)\n    image_names = all_names[:n_images]\n    if is_random:\n        image_names = random.sample(all_names, n_images)\n    for ind, image_name in enumerate(image_names):\n        img = cv2.imread(os.path.join(path, image_name))\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) \n        plt.subplot(h, w, ind + 1)\n        plt.imshow(img)\n        plt.axis(\"off\")\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:12:30.059012Z","iopub.execute_input":"2022-05-16T15:12:30.059463Z","iopub.status.idle":"2022-05-16T15:12:30.070067Z","shell.execute_reply.started":"2022-05-16T15:12:30.059417Z","shell.execute_reply":"2022-05-16T15:12:30.06909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Some Monet images:","metadata":{}},{"cell_type":"code","source":"batch_visualization(MONET_PATH_JPG, 9, is_random=True)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:12:30.071521Z","iopub.execute_input":"2022-05-16T15:12:30.071907Z","iopub.status.idle":"2022-05-16T15:12:31.196861Z","shell.execute_reply.started":"2022-05-16T15:12:30.071863Z","shell.execute_reply":"2022-05-16T15:12:31.1954Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"batch_visualization(PHOTO_PATH_JPG, 16, is_random=True)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:12:31.198238Z","iopub.execute_input":"2022-05-16T15:12:31.198542Z","iopub.status.idle":"2022-05-16T15:12:32.534153Z","shell.execute_reply.started":"2022-05-16T15:12:31.198506Z","shell.execute_reply":"2022-05-16T15:12:32.533256Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"batch_visualization(STYLES_FOR_NST, 2, is_random=True)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:12:32.53752Z","iopub.execute_input":"2022-05-16T15:12:32.537962Z","iopub.status.idle":"2022-05-16T15:12:33.458856Z","shell.execute_reply.started":"2022-05-16T15:12:32.537924Z","shell.execute_reply":"2022-05-16T15:12:33.455384Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"batch_visualization(ARTS_FOR_NST, 16, is_random=True)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:12:33.460597Z","iopub.execute_input":"2022-05-16T15:12:33.461341Z","iopub.status.idle":"2022-05-16T15:12:41.403746Z","shell.execute_reply.started":"2022-05-16T15:12:33.461281Z","shell.execute_reply":"2022-05-16T15:12:41.402955Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"batch_visualization(BEST_ARTS_FOR_NST, 16, is_random=True)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:12:41.404959Z","iopub.execute_input":"2022-05-16T15:12:41.405375Z","iopub.status.idle":"2022-05-16T15:12:43.410444Z","shell.execute_reply.started":"2022-05-16T15:12:41.405336Z","shell.execute_reply":"2022-05-16T15:12:43.40939Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"------------------------------------------------\n# 1. Original Gatys' approach\nUsing Keras tutorial: [Neural style transfer](https://keras.io/examples/generative/neural_style_transfer/)\n\nStyle transfer consists in generating an image ($NST$)\nwith the same \"content\" ($C$) as a base image, but with the\n\"style\" ($S$) of a different picture (typically artistic).\nThis is achieved through the optimization of a loss function ($L_t$)\nthat has 3 components: \"style loss\" ($L_s$), \"content loss\" ($L_c$),\nand \"total variation loss\" ($L_v$):\n\n$L_t = w_s \\cdot L_s + w_c \\cdot L_c + w_v \\cdot L_v$\n\n- The style loss is where the deep learning keeps in - that one is defined\nusing a deep convolutional neural network. Precisely, it consists in a sum of\n$L_2$ distances between the [Gram matrices](https://en.wikipedia.org/wiki/Gram_matrix) of the representations of\nthe base image and the style reference image, extracted from\ndifferent layers of a convnet (trained on `ImageNet`). The general idea\nis to capture color/texture information at different spatial\nscales:\n\n    $L_s = \\sum_{i=1}^{l}{||GramMatrix(\\phi_i(NST)) - GramMatrix(\\phi_i(S))||_2}$\n\n    where $\\phi_i$ denotes the layers of convnet used to compute the loss (VGG-19 in this case).\n\n    > If the vectors are centered random variables, the Gram matrix is approximately proportional to the covariance matrix, with the scaling determined by the number of elements in the vector.\n\n    https://en.wikipedia.org/wiki/Gram_matrix\n\n- The content loss is a L2 distance between the features of the base\nimage (extracted from a deep layer $d$) and the features of the combination image,\nkeeping the generated image close enough to the original one:\n\n    $L_c = ||\\phi_d(NST) - \\phi_d(C)||_2$\n\n- The total variation loss imposes local spatial continuity between\nthe pixels of the combination image, giving it visual coherence.\n\n![](https://miro.medium.com/max/1400/1*VAQs1KSfbysnloPah_fHGQ.gif)\n\nPicture Credit: https://miro.medium.com\n\n**Reference:** [A Neural Algorithm of Artistic Style](\n  http://arxiv.org/abs/1508.06576)","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras.applications import vgg19\n!mkdir ../gatys_generated","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:12:43.412139Z","iopub.execute_input":"2022-05-16T15:12:43.412439Z","iopub.status.idle":"2022-05-16T15:12:50.307623Z","shell.execute_reply.started":"2022-05-16T15:12:43.412403Z","shell.execute_reply":"2022-05-16T15:12:50.306372Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Image preprocessing / deprocessing utilities","metadata":{}},{"cell_type":"code","source":"def preprocess_image(image_path):\n    # Util function to open, resize and format pictures into appropriate tensors\n    img = keras.preprocessing.image.load_img(\n        image_path, target_size=(IMAGE_SIZE, IMAGE_SIZE)\n    )\n    img = keras.preprocessing.image.img_to_array(img)\n    img = np.expand_dims(img, axis=0)\n    # vgg16.preprocess_input will convert the input images from RGB to BGR, then will zero-center each color channel with respect to the ImageNet dataset, without scaling\n    img = vgg19.preprocess_input(img)\n    return tf.convert_to_tensor(img)\n\n\ndef deprocess_image(x):\n    x = x.numpy()\n    # Util function to convert a tensor into a valid image\n    x = x.reshape((IMAGE_SIZE, IMAGE_SIZE, 3))\n    # Remove zero-center by mean pixel\n    x[:, :, 0] += 103.939\n    x[:, :, 1] += 116.779\n    x[:, :, 2] += 123.68\n    # 'BGR'->'RGB'\n    x = x[:, :, ::-1]\n    x = np.clip(x, 0, 255).astype(\"uint8\")\n    return x","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:12:50.311362Z","iopub.execute_input":"2022-05-16T15:12:50.311723Z","iopub.status.idle":"2022-05-16T15:12:50.324933Z","shell.execute_reply.started":"2022-05-16T15:12:50.311685Z","shell.execute_reply":"2022-05-16T15:12:50.323201Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Compute the loss functions\n\nThe authors propose to use a pretrained VGG-19 to compute the loss\nfunction of the network.\nThe total loss is a weighted combination of:\n\n- The `style_loss` function, which keeps the generated image close to the local textures\nof the style reference image\n- The `content_loss` function, which keeps the high-level representation of the\ngenerated image close to that of the base image\n- The `total_variation_loss` function, a regularization loss which keeps the generated image locally-coherent\n\nSo, we define 4 utility functions: `gram_matrix`, `style_loss`, `content_loss`, and `total_variation_loss`.","metadata":{}},{"cell_type":"code","source":"# The gram matrix of an image tensor (feature-wise outer product)\ndef gram_matrix(x):\n    # x.shape = (256, 256, n_chanels)\n    x = tf.transpose(x, (2, 0, 1))\n    # x.shape = (n_chanels, 256, 256)\n    features = tf.reshape(x, (tf.shape(x)[0], -1))\n    # features.shape = (n_chanels, 256*256)\n    gram = tf.matmul(features, tf.transpose(features))\n    # gram.shape = (n_chanels, n_chanels)\n    return gram\n\n\n# The \"style loss\" is designed to maintain\n# the style of the reference image in the generated image.\n# It is based on the gram matrices (which capture style) of\n# feature maps from the style reference image\n# and from the generated image\ndef style_loss(style, generated):\n    S = gram_matrix(style)\n    C = gram_matrix(generated)\n    channels = 3\n    size = IMAGE_SIZE * IMAGE_SIZE * channels\n    return tf.reduce_mean(tf.square(S - C)) /  (size ** 2)\n\n\n# An auxiliary loss function\n# designed to maintain the \"content\" of the\n# base image in the generated image\ndef content_loss(base, generated):\n    return tf.reduce_mean(tf.square(generated - base))\n\n\n# The 3rd loss function, total variation loss,\n# designed to keep the generated image locally coherent\ndef total_variation_loss(generated):\n    img_nrows = generated.shape[1]\n    img_ncols = generated.shape[2]\n    horizontal_shift_diff = tf.square(\n        generated[:, : img_nrows - 1, : img_ncols - 1, :] - generated[:, 1:, : img_ncols - 1, :]\n    )\n    vertical_shift_diff = tf.square(\n        generated[:, : img_nrows - 1, : img_ncols - 1, :] - generated[:, : img_nrows - 1, 1:, :]\n    )\n    return tf.reduce_mean(tf.pow(horizontal_shift_diff + vertical_shift_diff, 1.25))","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:12:50.328335Z","iopub.execute_input":"2022-05-16T15:12:50.328851Z","iopub.status.idle":"2022-05-16T15:12:50.343809Z","shell.execute_reply.started":"2022-05-16T15:12:50.328801Z","shell.execute_reply":"2022-05-16T15:12:50.342797Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next, we create a feature extraction model that retrieves the intermediate activations\nof VGG19 (as a dict, by name).","metadata":{}},{"cell_type":"code","source":"# Build a VGG19 model loaded with pre-trained ImageNet weights\nvgg19_model = vgg19.VGG19(weights=\"imagenet\", include_top=False)\n\n# Get the symbolic outputs of each \"key\" layer (we gave them unique names).\noutputs_dict = dict([(layer.name, layer.output) for layer in vgg19_model.layers])\n\n# Set up a model that returns the activation values for every layer in\n# VGG19 (as a dict).\nfeature_extractor = keras.Model(inputs=vgg19_model.inputs, outputs=outputs_dict)\n","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-05-16T15:12:50.345507Z","iopub.execute_input":"2022-05-16T15:12:50.3462Z","iopub.status.idle":"2022-05-16T15:12:54.036268Z","shell.execute_reply.started":"2022-05-16T15:12:50.346152Z","shell.execute_reply":"2022-05-16T15:12:54.035377Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, the code that computes the style transfer loss:","metadata":{}},{"cell_type":"code","source":"def compute_loss(combination_image, base_image, style_reference_image):\n    # List of layers to use for the style loss.\n    style_layer_names = {\n        \"block1_conv1\": 1.,\n        \"block2_conv1\": 0.75,\n        \"block3_conv1\": 0.3,\n        \"block4_conv1\": 0.2,\n        \"block5_conv1\": 0.2,\n    }\n    # The layer to use for the content loss.\n    content_layer_name = \"block5_conv2\"\n    # Weights of the different loss components\n    total_variation_weight = 1e-1\n    style_weight = 1e-2\n    content_weight = 2.5e-2\n    input_tensor = tf.concat(\n        [base_image, style_reference_image, combination_image], axis=0\n    )\n    features = feature_extractor(input_tensor)\n    # Initialize the loss\n    loss = tf.zeros(shape=())\n    # Add content loss\n    layer_features = features[content_layer_name]\n    base_image_features = layer_features[0, :, :, :]\n    combination_features = layer_features[2, :, :, :]\n    loss = loss + content_weight * content_loss(\n        base_image_features, combination_features\n    )\n    # Add style loss\n    for layer_name, weight in style_layer_names.items():\n        layer_features = features[layer_name]\n        style_reference_features = layer_features[1, :, :, :]\n        combination_features = layer_features[2, :, :, :]\n        sl = weight * style_loss(style_reference_features, combination_features)\n        loss += (style_weight / len(style_layer_names)) * sl\n    # Add total variation loss\n    loss += total_variation_weight * total_variation_loss(combination_image)\n    return loss\n\n\n# Add a tf.function decorator to loss & gradient computation\n# to compile it, and thus make it fast.\n@tf.function\ndef compute_loss_and_grads(generated_image, base_image, style_reference_image):\n    with tf.GradientTape() as tape:\n        loss = compute_loss(generated_image, base_image, style_reference_image)\n    grads = tape.gradient(loss, generated_image)\n    return loss, grads","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:12:54.037672Z","iopub.execute_input":"2022-05-16T15:12:54.037953Z","iopub.status.idle":"2022-05-16T15:12:54.051273Z","shell.execute_reply.started":"2022-05-16T15:12:54.037916Z","shell.execute_reply":"2022-05-16T15:12:54.050208Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## The training loop\n\nRepeatedly run vanilla gradient descent steps to minimize the loss, and save the\nresulting image every 100 iterations.\n\nWe decay the learning rate by 0.96 every 100 steps.\n","metadata":{}},{"cell_type":"code","source":"def generate_image(base_image_path, style_reference_image_path, result_prefix, iterations):\n    base_image = preprocess_image(base_image_path)\n    style_reference_image = preprocess_image(style_reference_image_path)\n    generated_image = tf.Variable(preprocess_image(base_image_path))\n    optimizer = keras.optimizers.SGD(\n        keras.optimizers.schedules.ExponentialDecay(\n            initial_learning_rate=100.0, decay_steps=100, decay_rate=0.96\n        )\n    )\n    for i in range(1, iterations + 1):\n        loss, grads = compute_loss_and_grads(\n            generated_image, base_image, style_reference_image\n        )\n        optimizer.apply_gradients([(grads, generated_image)])\n        if i % 50 == 0:\n            print(\"Iteration %d: loss=%.2f\" % (i, loss))\n            img = deprocess_image(generated_image)\n            fname = result_prefix + \"_at_iteration_%d.jpg\" % i\n            keras.preprocessing.image.save_img(fname, img)\n    return deprocess_image(generated_image)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:12:54.05295Z","iopub.execute_input":"2022-05-16T15:12:54.053438Z","iopub.status.idle":"2022-05-16T15:12:54.066052Z","shell.execute_reply.started":"2022-05-16T15:12:54.053394Z","shell.execute_reply":"2022-05-16T15:12:54.065008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def batch_generation(path_content, path_style, n_images, iterations, is_random_base=True, is_random_style=True, figsize=(16, 16)):\n    plt.figure(figsize=figsize)    \n    w = 3 * int(n_images ** .5)\n    h = math.ceil(3 * n_images / w)\n    all_content_names = os.listdir(path_content)\n    all_style_names = os.listdir(path_style)\n    if is_random_style:\n        style_image_names = random.choices(all_style_names, k=n_images)\n    else:\n        style_image_names = all_style_names[:n_images]\n    if is_random_base:\n        content_image_names = random.choices(all_content_names, k=n_images)\n    else:\n        content_image_names = all_content_names[:n_images]\n    for ind, (content_image_name, style_image_name) in enumerate(zip(content_image_names, style_image_names)):\n        base_image_path = os.path.join(path_content, content_image_name)\n        style_reference_image_path = os.path.join(path_style, style_image_name)\n        result_prefix = os.path.join(\"../gatys_generated\",content_image_name)\n        img = generate_image(base_image_path, style_reference_image_path, result_prefix, iterations)\n        plt.subplot(h, w, 3*ind + 1)\n        base_img = keras.preprocessing.image.load_img(\n            base_image_path, target_size=(IMAGE_SIZE, IMAGE_SIZE)\n        )\n        plt.imshow(base_img)\n        plt.axis(\"off\")\n        plt.subplot(h, w, 3*ind + 2)\n        style_img = keras.preprocessing.image.load_img(\n            style_reference_image_path, target_size=(IMAGE_SIZE, IMAGE_SIZE)\n        )\n        plt.imshow(style_img)\n        plt.axis(\"off\")\n        plt.subplot(h, w, 3*ind + 3)\n        plt.imshow(img)\n        plt.axis(\"off\")\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:12:54.067909Z","iopub.execute_input":"2022-05-16T15:12:54.068244Z","iopub.status.idle":"2022-05-16T15:12:54.08266Z","shell.execute_reply.started":"2022-05-16T15:12:54.068201Z","shell.execute_reply":"2022-05-16T15:12:54.081733Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nrandom.seed(11)\nbatch_generation(PHOTO_PATH_JPG, STYLES_FOR_NST, 3, 300, is_random_base=True, is_random_style=True)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:12:54.084412Z","iopub.execute_input":"2022-05-16T15:12:54.084927Z","iopub.status.idle":"2022-05-16T15:13:34.458798Z","shell.execute_reply.started":"2022-05-16T15:12:54.084883Z","shell.execute_reply":"2022-05-16T15:13:34.457948Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nrandom.seed(11)\nbatch_generation(PHOTO_PATH_JPG, MONET_PATH_JPG, 3, 300, is_random_base=True, is_random_style=False)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:13:34.460321Z","iopub.execute_input":"2022-05-16T15:13:34.460789Z","iopub.status.idle":"2022-05-16T15:14:06.629535Z","shell.execute_reply.started":"2022-05-16T15:13:34.460746Z","shell.execute_reply":"2022-05-16T15:14:06.628689Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Conclusion\n\nThis is quite powerful approach that can provide nice looking results, but it requires some fine-tuning of the weights.\n\nThe main disadvantage for this competition however is that it is very slow (~10 sec. per image only for 300 iterations). It would require hours to create 7k images for submission. So, we shall explore other methods.","metadata":{}},{"cell_type":"markdown","source":"------------------------------------------------\n# 2. Adaptive Instance Normalization\nAdopted from Keras tutorial: [Neural Style Transfer with AdaIN](https://keras.io/examples/generative/adain/)\n\nFollow-up papers that introduced\n[Batch Normalization](https://arxiv.org/abs/1502.03167),\n[Instance Normalization](https://arxiv.org/abs/1701.02096) and\n[Conditional Instance Normalization](https://arxiv.org/abs/1610.07629)\nallowed Style Transfer to be performed in new ways, no longer\nrequiring a slow iterative process.\nFollowing these papers, the authors Xun Huang and Serge Belongie proposed\n[Adaptive Instance Normalization](https://arxiv.org/abs/1703.06868) (AdaIN),\nwhich allows arbitrary style transfer in real time.\n\nYou can also try out AdaIN model with your own images with this\n[Hugging Face demo](https://huggingface.co/spaces/keras-io/AdaIN).","metadata":{}},{"cell_type":"markdown","source":"## Architecture\n\nThe style transfer network takes a content image and a style image as\ninputs and outputs the style transferred image. The authors of AdaIN\npropose a simple encoder-decoder structure for achieving this.\n\n![AdaIN architecture](https://i.imgur.com/JbIfoyE.png)\n\nThe content image ($C$) and the style image ($S$) are both fed to the\nencoder networks. The output from these encoder networks (feature maps)\nare then fed to the $AdaIN$ layer. The $AdaIN$ layer computes a combined\nfeature map. This feature map is then fed into a randomly initialized\ndecoder network that serves as the generator for the neural style\ntransferred image.\n\n$E_s = f(S), E_c = f(C)$\n\n$t = AdaIN(E_c, E_s)$\n\n$NST = Generator(t)$\n\nThe style feature map ($E_s$) and the content feature map ($E_c$) are\nfed to the $AdaIN$ layer. This layer produced the combined feature map ($t$).","metadata":{}},{"cell_type":"markdown","source":"### Encoder\n\nThe encoder is a part of the pretrained (on [imagenet](https://www.image-net.org/)) VGG19 model.\nWe slice the model from the `block4-conv1` layer. The output layer is as suggested\nby the authors in their paper.","metadata":{}},{"cell_type":"code","source":"def get_encoder():\n    vgg19 = keras.applications.VGG19(\n        include_top=False,\n        weights=\"imagenet\",\n        input_shape=(IMAGE_SIZE, IMAGE_SIZE, 3),\n    )\n    vgg19.trainable = False\n    mini_vgg19 = keras.Model(vgg19.input, vgg19.get_layer(\"block4_conv1\").output)\n    inputs = keras.layers.Input([IMAGE_SIZE, IMAGE_SIZE, 3])\n    mini_vgg19_out = mini_vgg19(inputs)\n    return keras.Model(inputs, mini_vgg19_out, name=\"mini_vgg19\")","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:14:06.630753Z","iopub.execute_input":"2022-05-16T15:14:06.631059Z","iopub.status.idle":"2022-05-16T15:14:06.647059Z","shell.execute_reply.started":"2022-05-16T15:14:06.63102Z","shell.execute_reply":"2022-05-16T15:14:06.646192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Adaptive Instance Normalization\n\nThe $AdaIN$ layer takes in the features of the content and style image.\nThe layer can be defined via the following equation:\n\n$AdaIN(x, y) = \\sigma (y) \\left( \\frac{x - \\mu (x)}{\\sigma (x)} \\right) + \\mu (y)$\n\nwhere $\\sigma$ is the standard deviation and $\\mu$ is the mean for the\nconcerned variable. In the above equation the mean and variance of the\ncontent feature map $E_c$ is aligned with the mean and variance of the\nstyle feature maps $E_s$.\n\nIt is important to note that the $AdaIN$ layer proposed by the authors\nuses no other parameters apart from mean and variance. The layer also\ndoes not have any trainable parameters. This is why we use a\n*Python function* instead of using a *Keras layer*. The function takes\nstyle and content feature maps, computes the mean and standard deviation\nof the images and returns the adaptive instance normalized feature map.","metadata":{}},{"cell_type":"code","source":"def get_mean_std(x, epsilon=1e-5):\n    axes = [1, 2]\n    # Compute the mean and standard deviation of a tensor.\n    mean, variance = tf.nn.moments(x, axes=axes, keepdims=True)\n    standard_deviation = tf.sqrt(variance + epsilon)\n    return mean, standard_deviation\n\n\ndef ada_in(style, content):\n    content_mean, content_std = get_mean_std(content)\n    style_mean, style_std = get_mean_std(style)\n    t = style_std * (content - content_mean) / content_std + style_mean\n    return t","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:14:06.648504Z","iopub.execute_input":"2022-05-16T15:14:06.649567Z","iopub.status.idle":"2022-05-16T15:14:06.662922Z","shell.execute_reply.started":"2022-05-16T15:14:06.64952Z","shell.execute_reply":"2022-05-16T15:14:06.66195Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Decoder\n\nThe authors specify that the decoder network must mirror the encoder\nnetwork.  We have symmetrically inverted the encoder to build our\ndecoder. We have used `UpSampling2D` layers to increase the spatial\nresolution of the feature maps.\n\nNote that the authors warn against using any normalization layer\nin the decoder network, and do indeed go on to show that including\nbatch normalization or instance normalization hurts the performance\nof the overall network.\n\nThis is the only portion of the entire architecture that is trainable.","metadata":{}},{"cell_type":"code","source":"def get_decoder():\n    config = {\"kernel_size\": 3, \"strides\": 1, \"padding\": \"same\", \"activation\": \"relu\"}\n    decoder = keras.Sequential(\n        [\n            keras.layers.InputLayer((None, None, 512)),\n            keras.layers.Conv2D(filters=512, **config),\n            # Default: size=(2, 2), data_format=\"channels_last\", interpolation=\"nearest\"\n            keras.layers.UpSampling2D(),\n            keras.layers.Conv2D(filters=256, **config),\n            keras.layers.Conv2D(filters=256, **config),\n            keras.layers.Conv2D(filters=256, **config),\n            keras.layers.Conv2D(filters=256, **config),\n            keras.layers.UpSampling2D(),\n            keras.layers.Conv2D(filters=128, **config),\n            keras.layers.Conv2D(filters=128, **config),\n            keras.layers.UpSampling2D(),\n            keras.layers.Conv2D(filters=64, **config),\n            keras.layers.Conv2D(\n                filters=3,\n                kernel_size=3,\n                strides=1,\n                padding=\"same\",\n                activation=\"linear\"#\"sigmoid\",\n            ),\n        ]\n    )\n    return decoder","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:14:06.66554Z","iopub.execute_input":"2022-05-16T15:14:06.666933Z","iopub.status.idle":"2022-05-16T15:14:06.692916Z","shell.execute_reply.started":"2022-05-16T15:14:06.666883Z","shell.execute_reply":"2022-05-16T15:14:06.69192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Loss functions\n\nHere we build the loss functions for the neural style transfer model.\nThe authors propose to use a pretrained VGG-19 to compute the loss\nfunction of the network. It is important to keep in mind that this\nwill be used for training only the decoder network. The total\nloss ($L_t$) is a weighted combination of content loss ($L_c$) and style\nloss ($L_s$). The $w_s$ term is used to vary the amount of style\ntransferred.\n\n$L_t = L_c + w_s L_s$\n\n#### Content Loss\n\nThis is the Euclidean distance between the content image features\nand the features of the neural style transferred image.\n\n$L_c = ||f(NST) - t||_2$\n\nHere the authors propose to use the output from the $AdaIn$ layer $t$ as\nthe content target rather than using features of the original image as\ntarget. This is done to speed up convergence.\n\n#### Style Loss\n\nRather than using the more commonly used `Gram Matrix`,\nthe authors propose to compute the difference between the statistical features\n(mean and variance) which makes it conceptually cleaner. This can be\nexpressed via the following equation:\n\n$L_s = \\sum_{i=1}^{l}{\\left(||\\mu(\\phi_i(NST)) - \\mu(\\phi_i(S))||_2 + ||\\sigma(\\phi_i(NST)) - \\sigma(\\phi_i(S))||_2 \\right)}$\n\nwhere $\\phi_i$ denotes the layers in VGG-19 used to compute the loss.\nIn this case this corresponds to:\n\n- `block1_conv1`\n- `block1_conv2`\n- `block1_conv3`\n- `block1_conv4`","metadata":{}},{"cell_type":"code","source":"def get_loss_net():\n    vgg19 = keras.applications.VGG19(\n        include_top=False, weights=\"imagenet\", input_shape=(IMAGE_SIZE, IMAGE_SIZE, 3)\n    )\n    vgg19.trainable = False\n    layer_names = [\"block1_conv1\", \"block2_conv1\", \"block3_conv1\", \"block4_conv1\"]\n    outputs = [vgg19.get_layer(name).output for name in layer_names]\n    mini_vgg19 = keras.Model(vgg19.input, outputs)\n    inputs = keras.layers.Input([IMAGE_SIZE, IMAGE_SIZE, 3])\n    mini_vgg19_out = mini_vgg19(inputs)\n    return keras.Model(inputs, mini_vgg19_out, name=\"loss_net\")","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:14:06.694914Z","iopub.execute_input":"2022-05-16T15:14:06.69552Z","iopub.status.idle":"2022-05-16T15:14:06.717327Z","shell.execute_reply.started":"2022-05-16T15:14:06.695473Z","shell.execute_reply":"2022-05-16T15:14:06.716331Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## NST Model\n\nWe wrap the encoder and decoder inside of a `tf.keras.Model` subclass.\nThis allows us to customize what happens in the `model.fit()` loop.","metadata":{}},{"cell_type":"code","source":"class AdaInNSTModel(tf.keras.Model):\n    def __init__(self, encoder, decoder, loss_net, style_weight, **kwargs):\n        super().__init__(**kwargs)\n        self.encoder = encoder\n        self.decoder = decoder\n        self.loss_net = loss_net\n        self.style_weight = style_weight\n\n    def compile(self, optimizer, loss_fn):\n        super().compile()\n        self.optimizer = optimizer\n        self.loss_fn = loss_fn\n        self.style_loss_tracker = keras.metrics.Mean(name=\"style_loss\")\n        self.content_loss_tracker = keras.metrics.Mean(name=\"content_loss\")\n        self.total_loss_tracker = keras.metrics.Mean(name=\"total_loss\")\n\n    def train_step(self, inputs):\n        style, content = inputs\n        # Initialize the content and style loss.\n        loss_content = 0.0\n        loss_style = 0.0\n        with tf.GradientTape() as tape:\n            # Encode the style and content image.\n            style_encoded = self.encoder(style)\n            content_encoded = self.encoder(content)\n            # Compute the AdaIN target feature maps.\n            t = ada_in(style=style_encoded, content=content_encoded)\n            # Generate the neural style transferred image.\n            reconstructed_image = self.decoder(t)\n            # Compute the losses.\n            reconstructed_vgg_features = self.loss_net(reconstructed_image)\n            style_vgg_features = self.loss_net(style)\n            loss_content = self.loss_fn(t, reconstructed_vgg_features[-1])\n            for inp, out in zip(style_vgg_features, reconstructed_vgg_features):\n                mean_inp, std_inp = get_mean_std(inp)\n                mean_out, std_out = get_mean_std(out)\n                loss_style += self.loss_fn(mean_inp, mean_out) + self.loss_fn(std_inp, std_out)\n            loss_style = self.style_weight * loss_style\n            total_loss = loss_content + loss_style\n        # Compute gradients and optimize the decoder.\n        trainable_vars = self.decoder.trainable_variables\n        gradients = tape.gradient(total_loss, trainable_vars)\n        self.optimizer.apply_gradients(zip(gradients, trainable_vars))\n        # Update the trackers.\n        self.style_loss_tracker.update_state(loss_style)\n        self.content_loss_tracker.update_state(loss_content)\n        self.total_loss_tracker.update_state(total_loss)\n        return {\n            \"style_loss\": self.style_loss_tracker.result(),\n            \"content_loss\": self.content_loss_tracker.result(),\n            \"total_loss\": self.total_loss_tracker.result(),\n        }\n\n    def test_step(self, inputs):\n        style, content = inputs\n        # Initialize the content and style loss.\n        loss_content = 0.0\n        loss_style = 0.0\n        # Encode the style and content image.\n        style_encoded = self.encoder(style)\n        content_encoded = self.encoder(content)\n        # Compute the AdaIN target feature maps.\n        t = ada_in(style=style_encoded, content=content_encoded)\n        # Generate the neural style transferred image.\n        reconstructed_image = self.decoder(t)\n        # Compute the losses.\n        recons_vgg_features = self.loss_net(reconstructed_image)\n        style_vgg_features = self.loss_net(style)\n        loss_content = self.loss_fn(t, recons_vgg_features[-1])\n        for inp, out in zip(style_vgg_features, recons_vgg_features):\n            mean_inp, std_inp = get_mean_std(inp)\n            mean_out, std_out = get_mean_std(out)\n            loss_style += self.loss_fn(mean_inp, mean_out) + self.loss_fn(std_inp, std_out)\n        loss_style = self.style_weight * loss_style\n        total_loss = loss_content + loss_style\n        # Update the trackers.\n        self.style_loss_tracker.update_state(loss_style)\n        self.content_loss_tracker.update_state(loss_content)\n        self.total_loss_tracker.update_state(total_loss)\n        return {\n            \"style_loss\": self.style_loss_tracker.result(),\n            \"content_loss\": self.content_loss_tracker.result(),\n            \"total_loss\": self.total_loss_tracker.result(),\n        }\n\n    @property\n    def metrics(self):\n        return [\n            self.style_loss_tracker,\n            self.content_loss_tracker,\n            self.total_loss_tracker,\n        ]","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:14:06.719352Z","iopub.execute_input":"2022-05-16T15:14:06.720017Z","iopub.status.idle":"2022-05-16T15:14:06.785534Z","shell.execute_reply.started":"2022-05-16T15:14:06.719971Z","shell.execute_reply":"2022-05-16T15:14:06.784579Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data and tf.data pipeline\nTo show the full power of the approach, we train the model not only on the Monet paintings, but on different examples of the artwork. To do this, we use a great kaggle dataset [best-artworks-of-all-time](https://www.kaggle.com/datasets/ikarus777/best-artworks-of-all-time).\nIn this section, we decode, convert and resize the images from the folder.\nAfter we have our style and content data pipeline ready, we zip the two together to obtain the data pipeline that our model will consume.","metadata":{}},{"cell_type":"code","source":"def decode_and_resize(image_path):\n    image = tf.io.read_file(image_path)\n    image = tf.image.decode_jpeg(image, channels=3)\n    #image = tf.image.convert_image_dtype(image, dtype=\"float32\")\n    image = tf.image.resize(image, (IMAGE_SIZE, IMAGE_SIZE))\n    # vgg16.preprocess_input will convert the input images from RGB to BGR,\n    # then will zero-center each color channel with respect to the ImageNet dataset, without scaling\n    image = vgg19.preprocess_input(image)\n    return image\n\n\n# Get the image file paths\nmonet_style_images = [os.path.join(MONET_PATH_JPG, file_name) for file_name in os.listdir(MONET_PATH_JPG)]\nstyle_images = monet_style_images + [\n    os.path.join(BEST_ARTS_FOR_NST, file_name) for file_name in os.listdir(BEST_ARTS_FOR_NST)]\n# Build the style and content tf.data datasets.\nstyle_ds = (tf.data.Dataset.from_tensor_slices(style_images)\n            .map(decode_and_resize, num_parallel_calls=tf.data.AUTOTUNE).repeat())\ncontent_ds = (keras.utils.image_dataset_from_directory(PHOTO_PATH_JPG, labels=None).unbatch()\n              .map(vgg19.preprocess_input, num_parallel_calls=tf.data.AUTOTUNE).repeat())\n# Zipping the style and content datasets.\nBATCH_SIZE = 32\ntrain_ds = (\n    tf.data.Dataset.zip((style_ds, content_ds))\n    .shuffle(BATCH_SIZE * 2)\n    .batch(BATCH_SIZE)\n    .prefetch(tf.data.AUTOTUNE)\n)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:14:06.788051Z","iopub.execute_input":"2022-05-16T15:14:06.789472Z","iopub.status.idle":"2022-05-16T15:14:09.085801Z","shell.execute_reply.started":"2022-05-16T15:14:06.789421Z","shell.execute_reply":"2022-05-16T15:14:09.084918Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Visualizing the data\nIt is always better to visualize the data before training. To ensure the correctness of our preprocessing pipeline, we visualize 5 samples from our dataset.","metadata":{}},{"cell_type":"code","source":"def visualize_ds(ds, deprocess=True):\n    style, content = next(iter(ds))\n    num_to_vis = 5\n    fig, axes = plt.subplots(nrows=2, ncols=num_to_vis, figsize=(15, 5))\n    [ax.axis(\"off\") for ax in np.ravel(axes)]\n    for (axis, style_image, content_image) in zip(axes.T, style[0:num_to_vis], content[0:num_to_vis]):\n        (ax_style, ax_content) = axis\n        if deprocess:\n            ax_style.imshow(deprocess_image(style_image))\n        else:\n            ax_style.imshow(style_image.numpy().reshape((IMAGE_SIZE, IMAGE_SIZE, 3)) / 2. + 0.5)\n        ax_style.set_title(\"Style Image\")\n        if deprocess:\n            ax_content.imshow(deprocess_image(content_image))\n        else:\n            ax_content.imshow(content_image.numpy().reshape((IMAGE_SIZE, IMAGE_SIZE, 3)) / 2. + 0.5)\n        ax_content.set_title(\"Content Image\")\n        \nvisualize_ds(train_ds)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:14:09.091066Z","iopub.execute_input":"2022-05-16T15:14:09.091815Z","iopub.status.idle":"2022-05-16T15:14:11.07594Z","shell.execute_reply.started":"2022-05-16T15:14:09.091761Z","shell.execute_reply":"2022-05-16T15:14:11.075203Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training Monitor callback\n\nThis callback is used to visualize the style transfer output of\nthe model at the end of each epoch. The objective of style transfer cannot be\nquantified properly, and is to be subjectively evaluated by an audience.\nFor this reason, visualization is a key aspect of evaluating the model.","metadata":{}},{"cell_type":"code","source":"class TrainMonitor(keras.callbacks.Callback):\n    def __init__(self, **kwargs):\n        super().__init__(**kwargs)\n        self.test_style, self.test_content = next(iter(train_ds))\n\n    def on_epoch_end(self, epoch, logs=None):\n        epoch += 1\n        if epoch == 1 or epoch % 10 == 0:\n            # Encode the style and content image.\n            test_style_encoded = self.model.encoder(self.test_style)\n            test_content_encoded = self.model.encoder(self.test_content)\n            # Compute the AdaIN features.\n            test_t = ada_in(style=test_style_encoded, content=test_content_encoded)\n            test_reconstructed_image = self.model.decoder(test_t)\n            # Plot the Style, Content and the NST image.\n            fig, axs = plt.subplots(nrows=1, ncols=3, figsize=(20, 5))\n            [ax.axis(\"off\") for ax in np.ravel(axs)]\n            axs[0].imshow(deprocess_image(self.test_style[0]))\n            axs[0].set_title(f\"Style: {epoch:03d}\")\n            axs[1].imshow(deprocess_image(self.test_content[0]))\n            axs[1].set_title(f\"Content: {epoch:03d}\")\n            axs[2].imshow(deprocess_image(test_reconstructed_image[0]))\n            axs[2].set_title(f\"NST: {epoch:03d}\")\n            plt.show()\n            plt.close()\n            keras.models.save_model(self.model.decoder, \"adain_decoder_checkpoint.h5\")\n","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:14:11.077593Z","iopub.execute_input":"2022-05-16T15:14:11.07813Z","iopub.status.idle":"2022-05-16T15:14:11.093028Z","shell.execute_reply.started":"2022-05-16T15:14:11.078088Z","shell.execute_reply":"2022-05-16T15:14:11.091996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training the model\n\nIn this section, we define the optimizer, the loss funtion, and the\ntrainer module. We compile the trainer module with the optimizer and\nthe loss function and then train it.","metadata":{}},{"cell_type":"code","source":"optimizer = keras.optimizers.Adam(learning_rate=1e-5)\nloss_fn = keras.losses.MeanSquaredError()\nencoder = get_encoder()\nloss_net = get_loss_net()\ndecoder = get_decoder()\nadain_model = AdaInNSTModel(encoder=encoder, decoder=decoder, loss_net=loss_net, style_weight=4.0)\nadain_model.compile(optimizer=optimizer, loss_fn=loss_fn)\ntrain_monitor_cb = TrainMonitor()\nEPOCHS = 1\nhistory = adain_model.fit(train_ds, epochs=EPOCHS, steps_per_epoch=50, callbacks=[train_monitor_cb])","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:14:11.094845Z","iopub.execute_input":"2022-05-16T15:14:11.095418Z","iopub.status.idle":"2022-05-16T15:15:07.271088Z","shell.execute_reply.started":"2022-05-16T15:14:11.095374Z","shell.execute_reply":"2022-05-16T15:15:07.270169Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"keras.models.save_model(decoder, \"adain_decoder_final.h5\")","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:15:07.272733Z","iopub.execute_input":"2022-05-16T15:15:07.273037Z","iopub.status.idle":"2022-05-16T15:15:07.336098Z","shell.execute_reply.started":"2022-05-16T15:15:07.272985Z","shell.execute_reply":"2022-05-16T15:15:07.335253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"decoder = keras.models.load_model(\"/kaggle/input/different-approaches-to-style-transfer-v0/adain_decoder_final.h5\")\nadain_model = AdaInNSTModel(encoder=encoder, decoder=decoder, loss_net=loss_net, style_weight=4.0)\nadain_model.compile(optimizer=optimizer, loss_fn=loss_fn)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:15:07.337454Z","iopub.execute_input":"2022-05-16T15:15:07.33814Z","iopub.status.idle":"2022-05-16T15:15:07.807533Z","shell.execute_reply.started":"2022-05-16T15:15:07.338088Z","shell.execute_reply":"2022-05-16T15:15:07.806662Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Inference\n\nNow we can run inference with the trained model.\nWe will pass arbitrary content and style images from the dataset and take a look at the output images.","metadata":{}},{"cell_type":"code","source":"for style, content in train_ds.take(1):\n    style_encoded = adain_model.encoder(style)\n    content_encoded = adain_model.encoder(content)\n    t = ada_in(style=style_encoded, content=content_encoded)\n    reconstructed_image = adain_model.decoder(t)\n    fig, axes = plt.subplots(nrows=10, ncols=3, figsize=(10, 30))\n    [ax.axis(\"off\") for ax in np.ravel(axes)]\n    for axis, style_image, content_image, reconstructed_image in zip(\n        axes, style[0:10], content[0:10], reconstructed_image[0:10]\n    ):\n        (ax_style, ax_content, ax_reconstructed) = axis\n        ax_style.imshow(deprocess_image(style_image))\n        ax_style.set_title(\"Style Image\")\n        ax_content.imshow(deprocess_image(content_image))\n        ax_content.set_title(\"Content Image\")\n        ax_reconstructed.imshow(deprocess_image(reconstructed_image))\n        ax_reconstructed.set_title(\"NST Image\")","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:15:07.809262Z","iopub.execute_input":"2022-05-16T15:15:07.809747Z","iopub.status.idle":"2022-05-16T15:15:11.411785Z","shell.execute_reply.started":"2022-05-16T15:15:07.809702Z","shell.execute_reply":"2022-05-16T15:15:11.409402Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Create submission file","metadata":{}},{"cell_type":"code","source":"import shutil\n! mkdir ../adain_images","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:15:11.413204Z","iopub.execute_input":"2022-05-16T15:15:11.414052Z","iopub.status.idle":"2022-05-16T15:15:12.243999Z","shell.execute_reply.started":"2022-05-16T15:15:11.414007Z","shell.execute_reply":"2022-05-16T15:15:12.239458Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def save_adain_submission():\n    monet_ds = (tf.data.Dataset.from_tensor_slices(monet_style_images)\n                .map(decode_and_resize, num_parallel_calls=tf.data.AUTOTUNE).repeat())\n    photo_ds = (keras.utils.image_dataset_from_directory(PHOTO_PATH_JPG, labels=None).unbatch()\n                  .map(vgg19.preprocess_input, num_parallel_calls=tf.data.AUTOTUNE))\n    test_ds = (\n        tf.data.Dataset.zip((monet_ds, photo_ds))\n        .shuffle(BATCH_SIZE * 2)\n        .batch(BATCH_SIZE)\n        .prefetch(tf.data.AUTOTUNE)\n    )\n    i = 1\n    for style, content in test_ds:\n        style_encoded = adain_model.encoder(style)\n        content_encoded = adain_model.encoder(content)\n        t = ada_in(style=style_encoded, content=content_encoded)\n        reconstructed_images = adain_model.decoder(t)\n        for reconstructed_image in reconstructed_images:\n            prediction = deprocess_image(reconstructed_image)\n            cv2.imwrite(\"../adain_images/\" + str(i) + \".jpg\", prediction.astype(np.uint8))\n            i += 1\n            if i%500 == 0:\n                print(i)\n    shutil.make_archive(\"/kaggle/working/adain_images\", 'zip', \"/kaggle/adain_images\")\n    \nsave_adain_submission()","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:15:12.246514Z","iopub.execute_input":"2022-05-16T15:15:12.247074Z","iopub.status.idle":"2022-05-16T15:17:22.506571Z","shell.execute_reply.started":"2022-05-16T15:15:12.247022Z","shell.execute_reply":"2022-05-16T15:17:22.505682Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Conclusion\n\nAdaptive Instance Normalization allows arbitrary style transfer in\nreal time, and it take just a few minutes to generate 7k needed images.\nIt is also important to note that the novel proposition of\nthe authors is to achieve this only by aligning the statistical\nfeatures (mean and standard deviation) of the style and the content\nimages.\n\nPossible improvements could include finding a pretrained model and fine-tuning it only for Monet images.","metadata":{}},{"cell_type":"markdown","source":"---------------------------------------\n# 3. CycleGANs\n\nThis section is based on [two](https://towardsdatascience.com/cyclegan-learning-to-translate-images-without-paired-training-data-5b4e93862c8d) nice [blogs about CycleGANs](https://hardikbansal.github.io/CycleGANBlog/) and a [notebook](https://www.kaggle.com/code/dimitreoliveira/improving-cyclegan-monet-paintings/notebook) with a great collection of links for possible improvements, where most of the figures, explanations and code were adopted from.\n\nApart from those, there are couple more useful resources:\n* [Nice and brief explanation of CycleGANs](https://www.cs.toronto.edu/~rgrosse/courses/csc321_2018/assignments/a4-handout.pdf)\n* [CycleGAN project website](https://junyanz.github.io/CycleGAN/)\n* [Paper](https://arxiv.org/pdf/1703.10593.pdf)\n* [Original implementation](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix)","metadata":{}},{"cell_type":"markdown","source":"CycleGAN is a model that aims to solve the image-to-image translation problem. The goal of the image-to-image translation problem is to learn the mapping between an input image and an output image using a training set of aligned image pairs. However, obtaining paired examples isn't always feasible. CycleGAN tries to learn this mapping without requiring paired input-output images, using cycle-consistency: if we transform from source distribution to target and then back again to source distribution, we should get samples from our source distribution.","metadata":{}},{"cell_type":"markdown","source":"## Architecture\n\nCycleGAN is a Generative Adversarial Network (GAN) that uses two generators and two discriminators.\nWe call one generator $G_{A\\rightarrow B}$. It converts images from the $A$ domain to the $B$ domain. The other generator is called $G_{B\\rightarrow A}$, and converts images from $B$ to $A$.\n\nEach generator has a corresponding discriminator, which attempts to tell apart its synthesized images from real ones.\n\n![CycleGAN architecture illustration forward](https://hardikbansal.github.io/CycleGANBlog/images/model.jpg)\n\n------------------------------------------------\n\n![CycleGAN architecture illustration reverse](https://hardikbansal.github.io/CycleGANBlog/images/model1.jpg)","metadata":{}},{"cell_type":"markdown","source":"### Generator\nThe generator in the CycleGAN has layers that implement three stages of computation: 1) the first\nstage encodes the input via a series of convolutional layers that extract the image features; 2) the\nsecond stage then transforms the features by passing them through one or more residual blocks;\nand 3) the third stage decodes the transformed features using a series of transpose convolutional\nlayers, to build an output image of the same size as the input.\n\n![Genarator architecture](https://hardikbansal.github.io/CycleGANBlog/images/Generator.jpg)","metadata":{}},{"cell_type":"markdown","source":"#### Encoder\nEach block applies convolutional filter while also reducing data resolution and increasing the number of features.","metadata":{}},{"cell_type":"code","source":"import tensorflow_addons as tfa\n\nconv_initializer = tf.random_normal_initializer(mean=0.0, stddev=0.02)\ngamma_initializer = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n    \ndef encode(input_layer, filters, size=3, strides=2, apply_instancenorm=True, activation=keras.layers.ReLU(), name='x'):\n    block = keras.layers.Conv2D(filters, size, strides=strides, padding='same', \n                     use_bias=False, kernel_initializer=conv_initializer, \n                     name=f'encoder_{name}')(input_layer)\n    if apply_instancenorm:\n        block = tfa.layers.InstanceNormalization(gamma_initializer=gamma_initializer)(block)\n    block = activation(block)\n    return block","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:17:22.508243Z","iopub.execute_input":"2022-05-16T15:17:22.508544Z","iopub.status.idle":"2022-05-16T15:17:22.769693Z","shell.execute_reply.started":"2022-05-16T15:17:22.508502Z","shell.execute_reply":"2022-05-16T15:17:22.76884Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Transformation\nEach block applies two convolutional filters and adds a residual connection to find relevant data patterns while keeping constant dimension.","metadata":{}},{"cell_type":"code","source":"def transform(input_layer, size=3, strides=1, name='x'):\n    filters = input_layer.shape[-1]\n    block = keras.layers.Conv2D(filters, size, strides=strides, padding='same', use_bias=False, \n                     kernel_initializer=conv_initializer, name=f'transformer_{name}_1')(input_layer)\n    block = keras.layers.ReLU()(block)\n    block = keras.layers.Conv2D(filters, size, strides=strides, padding='same', use_bias=False, \n                     kernel_initializer=conv_initializer, name=f'transformer_{name}_2')(block)\n    block = keras.layers.Add()([block, input_layer])\n    return block","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:17:22.771366Z","iopub.execute_input":"2022-05-16T15:17:22.771659Z","iopub.status.idle":"2022-05-16T15:17:22.781135Z","shell.execute_reply.started":"2022-05-16T15:17:22.771618Z","shell.execute_reply":"2022-05-16T15:17:22.780031Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Decoder\nEach block applies deconvolutional filter which increase data resolution and decrease the number of features.","metadata":{}},{"cell_type":"code","source":"def decode(input_layer, filters, size=3, strides=2, apply_instancenorm=True, name='x'):\n    block = keras.layers.Conv2DTranspose(filters, size,  strides=strides, padding='same', \n                              use_bias=False, kernel_initializer=conv_initializer, \n                              name=f'decoder_{name}')(input_layer)\n    if apply_instancenorm:\n        block = tfa.layers.InstanceNormalization(gamma_initializer=gamma_initializer)(block)\n    block = keras.layers.ReLU()(block)\n    return block","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:17:22.782831Z","iopub.execute_input":"2022-05-16T15:17:22.7845Z","iopub.status.idle":"2022-05-16T15:17:22.794138Z","shell.execute_reply.started":"2022-05-16T15:17:22.784442Z","shell.execute_reply":"2022-05-16T15:17:22.793131Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Generator\nThe architecture was proposed by the authors of CycleGAN paper, and [it was shown](https://www.kaggle.com/code/dimitreoliveira/improving-cyclegan-monet-paintings/notebook) that skipping normalization of the first layer and adding residual connection between encoder and decoder improve performance.","metadata":{}},{"cell_type":"code","source":"def get_generator(height=IMAGE_SIZE, width=IMAGE_SIZE, channels=3, transformer_blocks=6):\n    inputs = keras.layers.Input(shape=[height, width, channels], name='input_image')\n    # Encoder\n    enc_1 = encode(inputs, 64,  7, 1, apply_instancenorm=False, name='block_1') # (bs, 256, 256, 64)\n    enc_2 = encode(enc_1, 128, 3, 2, apply_instancenorm=True, name='block_2')   # (bs, 128, 128, 128)\n    enc_3 = encode(enc_2, 256, 3, 2, apply_instancenorm=True, name='block_3')   # (bs, 64, 64, 256)\n    # Transformer\n    x = enc_3\n    for n in range(transformer_blocks):\n        x = transform(x, 3, 1, name=f'block_{n+1}') # (bs, 64, 64, 256)\n    # Decoder\n    x_skip = keras.layers.Concatenate(name='enc_dec_skip_1')([x, enc_3]) # encoder - decoder skip connection\n    dec_1 = decode(x_skip, 128, 3, 2, apply_instancenorm=True, name='block_1') # (bs, 128, 128, 128)\n    x_skip = keras.layers.Concatenate(name='enc_dec_skip_2')([dec_1, enc_2]) # encoder - decoder skip connection\n    dec_2 = decode(x_skip, 64,  3, 2, apply_instancenorm=True, name='block_2') # (bs, 256, 256, 64)\n    x_skip = keras.layers.Concatenate(name='enc_dec_skip_3')([dec_2, enc_1]) # encoder - decoder skip connection\n    outputs = last = keras.layers.Conv2D(channels, 7, strides=1, padding='same', \n                              kernel_initializer=conv_initializer, use_bias=False, \n                              activation='tanh', name='decoder_output_block')(x_skip) # (bs, 256, 256, 3)\n    return keras.Model(inputs, outputs)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:17:22.796726Z","iopub.execute_input":"2022-05-16T15:17:22.797299Z","iopub.status.idle":"2022-05-16T15:17:22.811169Z","shell.execute_reply.started":"2022-05-16T15:17:22.797206Z","shell.execute_reply":"2022-05-16T15:17:22.810152Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Discriminator\nThe discriminators are fully convolutional neural networks that look at a “patch” of the input image, and output the probability of the patch being “real”. This is both more computationally efficient than trying to look at the entire input image, and is also more effective — it allows the discriminator to focus on more surface-level features, like texture, which is often the sort of thing being changed in an image translation task.\n\n![Discriminator architecture](https://miro.medium.com/max/1400/1*46CddTc5JwkFW_pQb4nGZQ.png)","metadata":{}},{"cell_type":"code","source":"def get_discriminator(height=IMAGE_SIZE, width=IMAGE_SIZE, channels=3):\n    inputs = keras.layers.Input(shape=[height, width, channels], name='input_image')\n    x = encode(inputs, 64,  4, 2, apply_instancenorm=False, activation=keras.layers.LeakyReLU(0.2), name='block_1') # (bs, 128, 128, 64)\n    x = encode(x, 128, 4, 2, apply_instancenorm=True, activation=keras.layers.LeakyReLU(0.2), name='block_2')       # (bs, 64, 64, 128)\n    x = encode(x, 256, 4, 2, apply_instancenorm=True, activation=keras.layers.LeakyReLU(0.2), name='block_3')       # (bs, 32, 32, 256)\n    x = encode(x, 512, 4, 1, apply_instancenorm=True, activation=keras.layers.LeakyReLU(0.2), name='block_4')       # (bs, 32, 32, 512)\n    outputs = keras.layers.Conv2D(1, 4, strides=1, padding='valid', kernel_initializer=conv_initializer)(x)                # (bs, 29, 29, 1)\n    discriminator = keras.Model(inputs, outputs)\n    return discriminator","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:17:22.813691Z","iopub.execute_input":"2022-05-16T15:17:22.814332Z","iopub.status.idle":"2022-05-16T15:17:22.82572Z","shell.execute_reply.started":"2022-05-16T15:17:22.814258Z","shell.execute_reply":"2022-05-16T15:17:22.824562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Objective Function\nThere are two components to the CycleGAN objective function, an _adversarial loss_ and a _cycle consistency loss_. Both are essential to getting good results.\nWe use the least squares loss here (found by [Mao et al.](https://arxiv.org/abs/1611.04076) to be more effective than the typical log likelihood loss).\n\n__Adversarial loss__:\n\nDiscriminator:\n\n$L^D_{real} = \\frac{1}{2} \\left(\\frac{1}{n}\\sum_{i=1}^{n}{(D_A(a_i) - 1)^2} + \\frac{1}{n}\\sum_{j=1}^{n}{(D_B(b_i) - 1)^2}\\right)$\n\n$L^D_{fake} = \\frac{1}{2} \\left(\\frac{1}{n}\\sum_{i=1}^{n}{D_B(G_{A\\rightarrow B}(a_i))^2} + \\frac{1}{n}\\sum_{i=1}^{n}{D_A(G_{B\\rightarrow A}(b_i))^2}\\right)$\n\nGenerator:\n\n$L^{G_{A\\rightarrow B}}_{adv} = \\frac{1}{n}\\sum_{i=1}^{n}{(D_B(G_{A\\rightarrow B}(a_i))-1)^2}$\n\n$L^{G_{B\\rightarrow A}}_{adv} = \\frac{1}{n}\\sum_{i=1}^{n}{(D_A(G_{B\\rightarrow A}(b_i))-1)^2}$\n\n__Cycle consistency loss__:\n\n$L^{A\\rightarrow B\\rightarrow A}_{cycle} = \\frac{1}{n}\\sum_{j=1}^{n}{|a_i - G_{B\\rightarrow A}(G_{A\\rightarrow B}(a_i))|}$\n\n$L^{B\\rightarrow A\\rightarrow B}_{cycle} = \\frac{1}{n}\\sum_{j=1}^{n}{|b_i - G_{A\\rightarrow B}(G_{B\\rightarrow A}(b_i))|}$\n\n$L^{A\\rightarrow B}_{identity} = \\frac{1}{n}\\sum_{j=1}^{n}{|b_i - G_{A\\rightarrow B}(b_i)|}$\n\n$L^{B\\rightarrow A}_{identity} = \\frac{1}{n}\\sum_{j=1}^{n}{|a_i - G_{B\\rightarrow A}(a_i)|}$\n\n__Total generator loss__:\n\n$L^{G_{A\\rightarrow B}}_{total} = L^{G_{A\\rightarrow B}}_{adv} + \\lambda \\left(L^{A\\rightarrow B\\rightarrow A}_{cycle} + L^{B\\rightarrow A\\rightarrow B}_{cycle}\\right) + \\frac{1}{2} \\lambda L^{A\\rightarrow B}_{identity}$\n\n$L^{G_{B\\rightarrow A}}_{total} = L^{G_{B\\rightarrow A}}_{adv} + \\lambda \\left(L^{B\\rightarrow A\\rightarrow B}_{cycle} + L^{A\\rightarrow B\\rightarrow A}_{cycle}\\right) + \\frac{1}{2} \\lambda L^{B\\rightarrow A}_{identity}$","metadata":{}},{"cell_type":"code","source":"adv_loss_fn = keras.losses.MeanSquaredError()\n# Discriminator loss {0: fake, 1: real} (The discriminator loss outputs the average of the real and generated loss)\ndef discriminator_loss(real, generated):\n    real_loss = adv_loss_fn(tf.ones_like(real), real)\n    generated_loss = adv_loss_fn(tf.zeros_like(generated), generated)\n    total_disc_loss = real_loss + generated_loss\n    return total_disc_loss * 0.5\n\n# Generator loss\ndef generator_loss(generated):\n    return adv_loss_fn(tf.ones_like(generated), generated)\n\n\n# Cycle consistency loss (measures if original photo and the twice transformed photo to be similar to one another)\ndef calc_cycle_loss(real_image, cycled_image, Lambda):\n    loss1 = tf.reduce_mean(tf.abs(real_image - cycled_image))\n    return Lambda * loss1\n\n# Identity loss (compares the image with its generator (i.e. photo with photo generator))\ndef identity_loss(real_image, same_image, Lambda):\n    loss = tf.reduce_mean(tf.abs(real_image - same_image))\n    return Lambda * 0.5 * loss","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:17:22.827498Z","iopub.execute_input":"2022-05-16T15:17:22.827817Z","iopub.status.idle":"2022-05-16T15:17:22.840151Z","shell.execute_reply.started":"2022-05-16T15:17:22.827772Z","shell.execute_reply":"2022-05-16T15:17:22.839145Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## CycleGAN Model\nWe will override the `train_step()` method of the Model class for training via `fit()`.","metadata":{}},{"cell_type":"code","source":"class CycleGan(keras.Model):\n    def __init__(self, monet_generator, photo_generator, \n                 monet_discriminator, photo_discriminator, lambda_cycle=10):\n        super(CycleGan, self).__init__()\n        self.monet_generator = monet_generator\n        self.photo_generator = photo_generator\n        self.monet_discriminator = monet_discriminator\n        self.photo_discriminator = photo_discriminator\n        self.lambda_cycle = lambda_cycle\n        \n    def compile(self, monet_generator_optimizer, photo_generator_optimizer,\n                monet_discriminator_optimizer, photo_discriminator_optimizer,\n                gen_loss_fn, disc_loss_fn, cycle_loss_fn, identity_loss_fn):\n        super(CycleGan, self).compile()\n        self.monet_generator_optimizer = monet_generator_optimizer\n        self.photo_generator_optimizer = photo_generator_optimizer\n        self.monet_discriminator_optimizer = monet_discriminator_optimizer\n        self.photo_discriminator_optimizer = photo_discriminator_optimizer\n        self.gen_loss_fn = gen_loss_fn\n        self.disc_loss_fn = disc_loss_fn\n        self.cycle_loss_fn = cycle_loss_fn\n        self.identity_loss_fn = identity_loss_fn\n        \n    def train_step(self, batch_data):\n        real_monet, real_photo = batch_data\n        with tf.GradientTape(persistent=True) as tape:\n            # photo to monet back to photo\n            fake_monet = self.monet_generator(real_photo, training=True)\n            cycled_photo = self.photo_generator(fake_monet, training=True)\n            # monet to photo back to monet\n            fake_photo = self.photo_generator(real_monet, training=True)\n            cycled_monet = self.monet_generator(fake_photo, training=True)\n            # generating itself\n            same_monet = self.monet_generator(real_monet, training=True)\n            same_photo = self.photo_generator(real_photo, training=True)\n            # discriminator used to check, inputing real images\n            disc_real_monet = self.monet_discriminator(real_monet, training=True)\n            disc_real_photo = self.photo_discriminator(real_photo, training=True)\n            # discriminator used to check, inputing fake images\n            disc_fake_monet = self.monet_discriminator(fake_monet, training=True)\n            disc_fake_photo = self.photo_discriminator(fake_photo, training=True)\n            # evaluate generator loss\n            monet_gen_loss = self.gen_loss_fn(disc_fake_monet)\n            photo_gen_loss = self.gen_loss_fn(disc_fake_photo)\n            # evaluate total cycle consistency loss\n            total_cycle_loss = self.cycle_loss_fn(real_monet, cycled_monet, self.lambda_cycle) + \\\n                self.cycle_loss_fn(real_photo, cycled_photo, self.lambda_cycle)\n            # evaluate total generator loss\n            total_monet_gen_loss = monet_gen_loss + total_cycle_loss + self.identity_loss_fn(real_monet, same_monet, self.lambda_cycle)\n            total_photo_gen_loss = photo_gen_loss + total_cycle_loss + self.identity_loss_fn(real_photo, same_photo, self.lambda_cycle)\n            # evaluate discriminator loss\n            monet_disc_loss = self.disc_loss_fn(disc_real_monet, disc_fake_monet)\n            photo_disc_loss = self.disc_loss_fn(disc_real_photo, disc_fake_photo)\n        # Calculate the gradients for generator and discriminator\n        monet_generator_gradients = tape.gradient(total_monet_gen_loss, \n                                                self.monet_generator.trainable_variables)\n        photo_generator_gradients = tape.gradient(total_photo_gen_loss,\n                                                  self.photo_generator.trainable_variables)\n        monet_discriminator_gradients = tape.gradient(monet_disc_loss,\n                                                      self.monet_discriminator.trainable_variables)\n        photo_discriminator_gradients = tape.gradient(photo_disc_loss,\n                                                      self.photo_discriminator.trainable_variables)\n        # Apply the gradients to the optimizer\n        self.monet_generator_optimizer.apply_gradients(zip(monet_generator_gradients,\n                                                 self.monet_generator.trainable_variables))\n        self.photo_generator_optimizer.apply_gradients(zip(photo_generator_gradients,\n                                                 self.photo_generator.trainable_variables))\n        self.monet_discriminator_optimizer.apply_gradients(zip(monet_discriminator_gradients,\n                                                  self.monet_discriminator.trainable_variables))\n        self.photo_discriminator_optimizer.apply_gradients(zip(photo_discriminator_gradients,\n                                                  self.photo_discriminator.trainable_variables))\n        return {'monet_gen_loss': total_monet_gen_loss,\n                'photo_gen_loss': total_photo_gen_loss,\n                'monet_disc_loss': monet_disc_loss,\n                'photo_disc_loss': photo_disc_loss\n               }","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:17:22.842106Z","iopub.execute_input":"2022-05-16T15:17:22.8434Z","iopub.status.idle":"2022-05-16T15:17:22.865294Z","shell.execute_reply.started":"2022-05-16T15:17:22.84335Z","shell.execute_reply":"2022-05-16T15:17:22.864367Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data and tf.data pipeline","metadata":{}},{"cell_type":"code","source":"def normalize_img(img):\n    img = tf.cast(img, dtype=tf.float32)\n    return (img / 127.5) - 1.0\n\nBATCH_SIZE = 1\nmonet_ds = (keras.utils.image_dataset_from_directory(MONET_PATH_JPG, labels=None).unbatch()\n            .map(normalize_img, num_parallel_calls=tf.data.AUTOTUNE).repeat())\nphoto_ds = (keras.utils.image_dataset_from_directory(PHOTO_PATH_JPG, labels=None).unbatch()\n            .map(normalize_img, num_parallel_calls=tf.data.AUTOTUNE))\ngan_ds = tf.data.Dataset.zip((monet_ds, photo_ds)).shuffle(BATCH_SIZE * 2).batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)\n\nvisualize_ds(gan_ds, deprocess=False)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:17:22.869274Z","iopub.execute_input":"2022-05-16T15:17:22.869839Z","iopub.status.idle":"2022-05-16T15:17:25.459107Z","shell.execute_reply.started":"2022-05-16T15:17:22.869801Z","shell.execute_reply":"2022-05-16T15:17:25.458208Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training Monitor callback","metadata":{}},{"cell_type":"code","source":"def plot_cycle(example_sample, generator_a, generator_b, n_samples=1):\n    fig, axes = plt.subplots(n_samples, 3, figsize=(22, (n_samples*6)))\n    axes = axes.flatten()\n    for n_sample in range(n_samples):\n        idx = n_sample*3\n        generated_a_sample = generator_a.predict(example_sample)\n        generated_b_sample = generator_b.predict(generated_a_sample)\n        axes[idx].set_title('Input image', fontsize=18)\n        axes[idx].imshow(example_sample[0] * 0.5 + 0.5)\n        axes[idx].axis('off')\n        axes[idx+1].set_title('Generated image', fontsize=18)\n        axes[idx+1].imshow(generated_a_sample[0] * 0.5 + 0.5)\n        axes[idx+1].axis('off')\n        axes[idx+2].set_title('Cycled image', fontsize=18)\n        axes[idx+2].imshow(generated_b_sample[0] * 0.5 + 0.5)\n        axes[idx+2].axis('off')\n    plt.show()\n    plt.close()\n\n    \nclass GANMonitor(keras.callbacks.Callback):\n    \"\"\"A callback to generate and save images after each epoch\"\"\"\n    def __init__(self, monet_path='monet', photo_path='photo'):\n        self.test_monet, self.test_photo = next(iter(gan_ds))\n        self.monet_path = monet_path\n        self.photo_path = photo_path\n        # Create directories to save the generate images\n        if not os.path.exists(self.monet_path):\n            os.makedirs(self.monet_path)\n        if not os.path.exists(self.photo_path):\n            os.makedirs(self.photo_path)\n\n    def on_epoch_end(self, epoch, logs=None):\n        # Monet generated images\n        prediction = self.model.monet_generator(self.test_photo, training=False)[0].numpy()\n        prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n        cv2.imwrite(f'{self.monet_path}/generated_{epoch+1}.jpg', prediction)        \n        # Photo generated images\n        prediction = self.model.photo_generator(self.test_monet, training=False)[0].numpy()\n        prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n        cv2.imwrite(f'{self.photo_path}/generated_{epoch+1}.jpg', prediction)\n        # Plot the Style, Content and the NST image\n        plot_cycle(self.test_photo, self.model.monet_generator, self.model.photo_generator)\n        plot_cycle(self.test_monet, self.model.photo_generator, self.model.monet_generator)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:17:25.462571Z","iopub.execute_input":"2022-05-16T15:17:25.462824Z","iopub.status.idle":"2022-05-16T15:17:25.481084Z","shell.execute_reply.started":"2022-05-16T15:17:25.462794Z","shell.execute_reply":"2022-05-16T15:17:25.480031Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training the model","metadata":{}},{"cell_type":"code","source":"TRANSFORMER_BLOCKS = 6\n# Networks\nmonet_generator = get_generator(height=None, width=None, transformer_blocks=TRANSFORMER_BLOCKS) # transforms photos to Monet-esque paintings\nphoto_generator = get_generator(height=None, width=None, transformer_blocks=TRANSFORMER_BLOCKS) # transforms Monet paintings to be more like photos\nmonet_discriminator = get_discriminator(height=None, width=None) # differentiates real Monet paintings and generated Monet paintings\nphoto_discriminator = get_discriminator(height=None, width=None) # differentiates real photos and generated photos\n# Optimizers\nlr = 2e-7\nmonet_generator_optimizer = keras.optimizers.Adam(learning_rate=lr, beta_1=0.5)\nphoto_generator_optimizer = keras.optimizers.Adam(learning_rate=lr, beta_1=0.5)\nmonet_discriminator_optimizer = keras.optimizers.Adam(learning_rate=lr, beta_1=0.5)\nphoto_discriminator_optimizer = keras.optimizers.Adam(learning_rate=lr, beta_1=0.5)\n# Create GAN\ngan_model = CycleGan(monet_generator, photo_generator, \n                     monet_discriminator, photo_discriminator)\ngan_model.compile(monet_generator_optimizer, photo_generator_optimizer,\n                  monet_discriminator_optimizer, photo_discriminator_optimizer,\n                  generator_loss, discriminator_loss, calc_cycle_loss, identity_loss)\ngan_monitor_cb = GANMonitor()\nEPOCHS = 2\nhistory = gan_model.fit(gan_ds, epochs=EPOCHS, callbacks=[gan_monitor_cb], steps_per_epoch=(300//2//BATCH_SIZE))","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:44:51.805587Z","iopub.execute_input":"2022-05-16T15:44:51.806687Z","iopub.status.idle":"2022-05-16T15:49:47.217841Z","shell.execute_reply.started":"2022-05-16T15:44:51.806615Z","shell.execute_reply":"2022-05-16T15:49:47.215179Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"monet_generator.save('monet_generator.h5')\nphoto_generator.save('photo_generator.h5')\nmonet_discriminator.save('monet_discriminator.h5')\nphoto_discriminator.save('photo_discriminator.h5')","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:19:14.608437Z","iopub.execute_input":"2022-05-16T15:19:14.611602Z","iopub.status.idle":"2022-05-16T15:19:15.02278Z","shell.execute_reply.started":"2022-05-16T15:19:14.61154Z","shell.execute_reply":"2022-05-16T15:19:15.021899Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Load pretrained model","metadata":{}},{"cell_type":"code","source":"monet_generator = keras.models.load_model(\"/kaggle/input/different-approaches-to-style-transfer-v3-cyclegan/monet_generator.h5\") # transforms photos to Monet-esque paintings\nphoto_generator = keras.models.load_model(\"/kaggle/input/different-approaches-to-style-transfer-v3-cyclegan/photo_generator.h5\") # transforms Monet paintings to be more like photos\nmonet_discriminator = keras.models.load_model(\"/kaggle/input/different-approaches-to-style-transfer-v3-cyclegan/monet_discriminator.h5\") # differentiates real Monet paintings and generated Monet paintings\nphoto_discriminator = keras.models.load_model(\"/kaggle/input/different-approaches-to-style-transfer-v3-cyclegan/photo_discriminator.h5\") # differentiates real photos and generated photos\n# Create GAN\ngan_model = CycleGan(monet_generator, photo_generator, \n                     monet_discriminator, photo_discriminator)\ngan_model.compile(monet_generator_optimizer, photo_generator_optimizer,\n                  monet_discriminator_optimizer, photo_discriminator_optimizer,\n                  generator_loss, discriminator_loss, calc_cycle_loss, identity_loss)\n#EPOCHS = 1\n#history = gan_model.fit(gan_ds, epochs=EPOCHS, callbacks=[gan_monitor_cb], steps_per_epoch=(300//BATCH_SIZE))","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:26:37.941457Z","iopub.execute_input":"2022-05-16T15:26:37.941983Z","iopub.status.idle":"2022-05-16T15:26:40.350138Z","shell.execute_reply.started":"2022-05-16T15:26:37.941942Z","shell.execute_reply":"2022-05-16T15:26:40.348856Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Inference","metadata":{}},{"cell_type":"code","source":"for _ in range(3):\n    plot_cycle(next(iter(photo_ds.batch(1))), monet_generator, photo_generator, n_samples=1)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:26:44.763015Z","iopub.execute_input":"2022-05-16T15:26:44.763353Z","iopub.status.idle":"2022-05-16T15:26:49.029555Z","shell.execute_reply.started":"2022-05-16T15:26:44.763297Z","shell.execute_reply":"2022-05-16T15:26:49.028373Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def display_generated_samples(ds, model, n_samples):\n    ds_iter = iter(ds)\n    for n_sample in range(n_samples):\n        example_sample = next(ds_iter)\n        generated_sample = model.predict(example_sample)\n        f = plt.figure(figsize=(12, 12))\n        plt.subplot(121)\n        plt.title('Input image')\n        plt.imshow(example_sample[0] * 0.5 + 0.5)\n        plt.axis('off')\n        plt.subplot(122)\n        plt.title('Generated image')\n        plt.imshow(generated_sample[0] * 0.5 + 0.5)\n        plt.axis('off')\n        plt.show()\n        \ndisplay_generated_samples(photo_ds.batch(1).take(8), monet_generator, 8)","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:26:56.098866Z","iopub.execute_input":"2022-05-16T15:26:56.099205Z","iopub.status.idle":"2022-05-16T15:26:59.881952Z","shell.execute_reply.started":"2022-05-16T15:26:56.099167Z","shell.execute_reply":"2022-05-16T15:26:59.88088Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Create submission file","metadata":{}},{"cell_type":"code","source":"! mkdir ../gan_images","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:19:15.407462Z","iopub.status.idle":"2022-05-16T15:19:15.408162Z","shell.execute_reply.started":"2022-05-16T15:19:15.407878Z","shell.execute_reply":"2022-05-16T15:19:15.407908Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import shutil\n\ndef save_cycleGAN_submission():\n    i = 1\n    for img in photo_ds.batch(1):\n        prediction = monet_generator.predict(img)[0]\n        prediction = (prediction * 127.5 + 127.5).astype(np.uint8)   # re-scale\n        cv2.imwrite(\"../gan_images/\" + str(i) + \".jpg\", prediction)\n        i += 1\n        if i%500 == 0:\n            print(i)\n    shutil.make_archive(\"/kaggle/working/images\", 'zip', \"/kaggle/gan_images\")\n    \nsave_cycleGAN_submission()","metadata":{"execution":{"iopub.status.busy":"2022-05-16T15:19:15.409746Z","iopub.status.idle":"2022-05-16T15:19:15.410674Z","shell.execute_reply.started":"2022-05-16T15:19:15.41033Z","shell.execute_reply":"2022-05-16T15:19:15.410361Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---------------------------------------\n# Final conclusions and future work\n\nJudging from the other notebooks in the competition, it looks like properly trained CycleGAN performs much better than traditional NST methods.\nHowever, it is quite difficult to train GANs properly.\nSo, it is worth exploring the ways to improve our initial model presented here.\n\nOn the other hand, the objective function of this competition does not take into account any content images, i.e. it only measures how similar the style of the generated image is to the Monet style.\nThat could be the reason why images from AdaIN look much better to the eye, but get much lower score.\n\nPlans for the future work:\n* Learn about further developments of CycleGANs and other more recent methods\n* Learn how to use TPUs","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}