{"metadata":{"colab":{"provenance":[],"authorship_tag":"ABX9TyMLUS8lDiiapx5hFaSyft78"},"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":21755,"databundleVersionId":1475600,"sourceType":"competition"}],"dockerImageVersionId":30588,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Development of an Adaptable Deep Learning Model for Artistic Style Transfer\n\nOur aim is to create an adaptive deep learning model that is capable of adapting an existing work to resemble the aesthetic of any art. The model analyzes the artistic style of the selected art and applies similar stylistic features to a new, original artwork, creating a piece that seems as though it could have been created by the artist themselves. This technique is called **Neural Style Transfer**, and the technique is outlined in *A Neural Algorithm of Artistic Style* (Gatys et al.).\n\n\n","metadata":{"id":"QeXPLcoMY8ws"}},{"cell_type":"markdown","source":"## Table of Contents\n\n- [Introduction to Neural Style Transfer and Problem Statement](#1)\n- [Setup](#2)\n- [Visualizing Input Images](#3)\n- [Defining Content and Style Representations](#4)","metadata":{"id":"xvlXqf9oscOn"}},{"cell_type":"markdown","source":"\n## Introduction to Neural Style Transfer and Problem Statement\n\nNeural style transfer is an optimization technique used to take two images—a *content* image and a *style reference* image (such as an artwork by a famous painter)—and blend them together so the output image looks like the content image, but “painted” in the style of the style reference image.\n\nGiven a **content image (C)** and a **style image (S)**, our aim is to output a **generated image (G)**, such that G combines the content of C with the style of S. This is implemented by optimizing the output image to match the content features of the content image and the style features of the style reference image. These features are extracted from the images using a convolutional network.\n\nFor example, we will be using Neural Style Transfer to generate the image of a dog (content C) painted in the style of Wassily Kandinsky's Composition 7 (style S). The content, style and generated images are given below:\n\n\n\n[Picture of a Yellow Labrador](https://commons.wikimedia.org/wiki/File:YellowLabradorLooking_new.jpg)\n\n\n\n[Wassily Kandinsky's Composition 7](https://commons.wikimedia.org/wiki/File:Vassily_Kandinsky,_1913_-_Composition_7.jpg)\n\n\n\nImage Generated Using Neural Style Transfer","metadata":{"id":"h_P9oED3tOBR"}},{"cell_type":"markdown","source":"\n## Setup\n\nWe will take a look at how to implement Neural Style Transfer using Tensorflow. First we setup a few things.","metadata":{"id":"PYWyjECI4r9m"}},{"cell_type":"markdown","source":"### Import and Configure Modules","metadata":{"id":"HiZbQrj_5Ogv"}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nimport tensorflow as tf\nfrom tensorflow.keras.utils import get_file, plot_model\nfrom tensorflow.keras.applications import vgg19\nfrom tensorflow.keras.models import load_model, Model\nimport time\nimport requests\nimport base64\nfrom pathlib import Path\nfrom io import BytesIO\nimport os, sys, warnings\nwarnings.filterwarnings('ignore')","metadata":{"id":"GKPBHa_-ykkT","executionInfo":{"status":"ok","timestamp":1700219709985,"user_tz":-330,"elapsed":4190,"user":{"displayName":"Abhranil Das","userId":"07322618999020616506"}},"execution":{"iopub.status.busy":"2023-11-17T21:38:33.588695Z","iopub.execute_input":"2023-11-17T21:38:33.58896Z","iopub.status.idle":"2023-11-17T21:38:45.506754Z","shell.execute_reply.started":"2023-11-17T21:38:33.588935Z","shell.execute_reply":"2023-11-17T21:38:45.505889Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from PIL import Image\nimport matplotlib as mpl\nimport matplotlib.pyplot as plt\nfrom matplotlib.pyplot import imshow\n\nmpl.rcParams['figure.figsize'] = (12, 12)\nmpl.rcParams['axes.grid'] = False","metadata":{"id":"QZ5slZTo5sPw","executionInfo":{"status":"ok","timestamp":1700219709986,"user_tz":-330,"elapsed":24,"user":{"displayName":"Abhranil Das","userId":"07322618999020616506"}},"execution":{"iopub.status.busy":"2023-11-17T21:38:45.508313Z","iopub.execute_input":"2023-11-17T21:38:45.508858Z","iopub.status.idle":"2023-11-17T21:38:45.513689Z","shell.execute_reply.started":"2023-11-17T21:38:45.508829Z","shell.execute_reply":"2023-11-17T21:38:45.512848Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load compressed models from tensorflow_hub\nos.environ['TFHUB_MODEL_LOAD_FORMAT'] = 'COMPRESSED'","metadata":{"id":"lUWjSWC85id2","executionInfo":{"status":"ok","timestamp":1700219709986,"user_tz":-330,"elapsed":23,"user":{"displayName":"Abhranil Das","userId":"07322618999020616506"}},"execution":{"iopub.status.busy":"2023-11-17T21:38:45.514811Z","iopub.execute_input":"2023-11-17T21:38:45.515079Z","iopub.status.idle":"2023-11-17T21:38:45.545212Z","shell.execute_reply.started":"2023-11-17T21:38:45.515056Z","shell.execute_reply":"2023-11-17T21:38:45.5443Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The following function converts a tensor to image","metadata":{"id":"8XbDGVOu6_ZY"}},{"cell_type":"code","source":"def tensor_to_image(tensor):\n tensor = tensor * 255\n tensor = np.array(tensor, dtype = np.uint8)\n if np.ndim(tensor) > 3:\n assert tensor.shape[0] == 1\n tensor = tensor[0]\n return Image.fromarray(tensor)","metadata":{"id":"2keM7Yg75jQv","executionInfo":{"status":"ok","timestamp":1700219709986,"user_tz":-330,"elapsed":22,"user":{"displayName":"Abhranil Das","userId":"07322618999020616506"}},"execution":{"iopub.status.busy":"2023-11-17T21:38:45.547441Z","iopub.execute_input":"2023-11-17T21:38:45.547748Z","iopub.status.idle":"2023-11-17T21:38:45.554697Z","shell.execute_reply.started":"2023-11-17T21:38:45.547723Z","shell.execute_reply":"2023-11-17T21:38:45.553846Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Downloading the content and style images for demonstration. We can try with our own images, the process is shown later down.","metadata":{"id":"6uKaC7bW8DUt"}},{"cell_type":"code","source":"content_img_link = 'https://storage.googleapis.com/download.tensorflow.org/example_images/YellowLabradorLooking_new.jpg'\nstyle_img_link = 'https://storage.googleapis.com/download.tensorflow.org/example_images/Vassily_Kandinsky%2C_1913_-_Composition_7.jpg'","metadata":{"id":"yd8W6wWu7UMu","executionInfo":{"status":"ok","timestamp":1700219709986,"user_tz":-330,"elapsed":22,"user":{"displayName":"Abhranil Das","userId":"07322618999020616506"}},"execution":{"iopub.status.busy":"2023-11-17T21:38:45.555605Z","iopub.execute_input":"2023-11-17T21:38:45.555838Z","iopub.status.idle":"2023-11-17T21:38:45.564145Z","shell.execute_reply.started":"2023-11-17T21:38:45.555818Z","shell.execute_reply":"2023-11-17T21:38:45.563299Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"content_img_path = get_file('YellowLabradorLooking_new.jpg', content_img_link)\nstyle_img_path = get_file('kandinsky5.jpg', style_img_link)","metadata":{"id":"B_iNeTZI7IPY","executionInfo":{"status":"ok","timestamp":1700219709986,"user_tz":-330,"elapsed":21,"user":{"displayName":"Abhranil Das","userId":"07322618999020616506"}},"outputId":"470ca88d-a50f-4dd2-a8c1-1d97baef10dd","execution":{"iopub.status.busy":"2023-11-17T21:38:45.565238Z","iopub.execute_input":"2023-11-17T21:38:45.565586Z","iopub.status.idle":"2023-11-17T21:38:45.85072Z","shell.execute_reply.started":"2023-11-17T21:38:45.565552Z","shell.execute_reply":"2023-11-17T21:38:45.849724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n## Visualizing Input Images","metadata":{"id":"KqQk9PMY7q_t"}},{"cell_type":"markdown","source":"The following function loads an image and limits its maximum dimension to 256 pixels.","metadata":{"id":"tcmsL-4G7z4_"}},{"cell_type":"code","source":"def load_image(image_path, dim = None, resize = False):\n img = Image.open(image_path)\n if dim:\n if resize:\n img = img.resize(dim)\n else:\n img.thumbnail(dim)\n img = img.convert(\"RGB\")\n return np.array(img)","metadata":{"id":"QFmHw9en7jYX","executionInfo":{"status":"ok","timestamp":1700185228371,"user_tz":-330,"elapsed":430,"user":{"displayName":"Abhranil Das","userId":"07322618999020616506"}},"execution":{"iopub.status.busy":"2023-11-17T21:45:41.875694Z","iopub.execute_input":"2023-11-17T21:45:41.87607Z","iopub.status.idle":"2023-11-17T21:45:41.881672Z","shell.execute_reply.started":"2023-11-17T21:45:41.876039Z","shell.execute_reply":"2023-11-17T21:45:41.880731Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The following function is used to display an image","metadata":{"id":"IevLXuu98Wcg"}},{"cell_type":"code","source":"def show_image(image, title = None):\n if len(image.shape) > 3:\n image = tf.squeeze(image, axis = 0)\n\n plt.imshow(image)\n plt.axis('off')\n if title:\n plt.title(title)","metadata":{"id":"RvnLQmnj776y","executionInfo":{"status":"ok","timestamp":1700185405372,"user_tz":-330,"elapsed":4,"user":{"displayName":"Abhranil Das","userId":"07322618999020616506"}},"execution":{"iopub.status.busy":"2023-11-17T21:46:46.095405Z","iopub.execute_input":"2023-11-17T21:46:46.096091Z","iopub.status.idle":"2023-11-17T21:46:46.101137Z","shell.execute_reply.started":"2023-11-17T21:46:46.096055Z","shell.execute_reply":"2023-11-17T21:46:46.10012Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The following utility function prints images in a grid","metadata":{}},{"cell_type":"code","source":"def plot_images_grid(images, num_rows = 1):\n n = len(images)\n if n > 1:\n num_cols = np.ceil(n / num_rows)\n fig,axes = plt.subplots(ncols = int(num_cols), nrows = int(num_rows))\n axes = axes.flatten()\n fig.set_size_inches((20, 20))\n for i,image in enumerate(images):\n axes[i].axis('off')\n axes[i].imshow(image)\n else:\n plt.figure(figsize = (10, 10))\n plt.imshow(images[0])","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:38:45.871354Z","iopub.execute_input":"2023-11-17T21:38:45.871628Z","iopub.status.idle":"2023-11-17T21:38:45.879755Z","shell.execute_reply.started":"2023-11-17T21:38:45.871598Z","shell.execute_reply":"2023-11-17T21:38:45.878931Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"content_image = load_image(content_img_path)\nstyle_image = load_image(style_img_path)\n\nplt.subplot(1, 2, 1)\nshow_image(content_image, 'Content Image')\n\nplt.subplot(1, 2, 2)\nshow_image(style_image, 'Style Image')","metadata":{"id":"Evs5LzPd8dOa","executionInfo":{"status":"ok","timestamp":1700185407384,"user_tz":-330,"elapsed":2015,"user":{"displayName":"Abhranil Das","userId":"07322618999020616506"}},"outputId":"789bc74e-c7ca-4e3c-b7a9-295c2fb48b56","execution":{"iopub.status.busy":"2023-11-17T21:47:04.613721Z","iopub.execute_input":"2023-11-17T21:47:04.614081Z","iopub.status.idle":"2023-11-17T21:47:05.212181Z","shell.execute_reply.started":"2023-11-17T21:47:04.614052Z","shell.execute_reply":"2023-11-17T21:47:05.211321Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n## Defining Content and Style Representations\n\nIn a convolutional neural network with multiple layers, starting from the network's input layer, the first few layer activations represent low-level features like edges and textures. Upon moving through the network, the features extracted by the layers become more and more complex. Thus the final few layers represent higher-level features — object parts like *wheels* or *eyes*.\n\nIn this case, we will be using a vanilla CNN architecture, which has been trained to classify artistic images into various categories. The architecture has been trained on the [Art Images: Drawing/Painting/Sculptures/Engravings](https://www.kaggle.com/datasets/thedownhill/art-images-drawings-painting-sculpture-engraving/code) dataset, in which art images are categorized into different classes on the basis of the style of art.\n\nWe use the intermediate layers of the model to get the *content* and *style* representations of the image. These intermediate layers are necessary to define the representation of content and style from the images.\n\nFirst we load the dataset, build the CNN architecture and train it on the dataset.\n","metadata":{"id":"GEwZ7FlwrjoZ"}},{"cell_type":"markdown","source":"**So why do these intermediate outputs within our pretrained image classification network allow us to define style and content representations?**\n\nAt a high level, in order for a network to perform image classification (which this network has been trained to do), it must understand the image. This requires taking the raw image as input pixels and building an internal representation that converts the raw image pixels into a complex understanding of the features present within the image.\n\nThis is also a reason why convolutional neural networks are able to generalize well: they’re able to capture the invariances and defining features within classes (e.g. cats vs. dogs) that are agnostic to background noise and other nuisances. Thus, somewhere between where the raw image is fed into the model and the output classification label, the model serves as a complex feature extractor. By accessing intermediate layers of the model, you're able to describe the content and style of input images.","metadata":{"id":"2aVZhn0CBgC3"}},{"cell_type":"markdown","source":"## Loading the Dataset\n\nWe will be using the [Art Images: Drawing/Painting/Sculptures/Engravings](https://www.kaggle.com/datasets/thedownhill/art-images-drawings-painting-sculpture-engraving/code) dataset, which consists of around 8600 images of various art images. The images are classified into 5 categories: *drawings*, *paintings*, *iconography*, *engravings* and\n*sculptures*. The dataset is further split into two datasets which are contained in different directories - the *train set* with around 7.8k images and the *validation set* with around 850 images.","metadata":{"id":"5awQLLk4C74m"}},{"cell_type":"markdown","source":"Defining the directories containing content images and style images","metadata":{}},{"cell_type":"code","source":"content_path = \"../input/gan-getting-started/photo_jpg\"\nstyle_path = \"../input/gan-getting-started/monet_jpg\"","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:38:49.237263Z","iopub.execute_input":"2023-11-17T21:38:49.237602Z","iopub.status.idle":"2023-11-17T21:38:49.242114Z","shell.execute_reply.started":"2023-11-17T21:38:49.237572Z","shell.execute_reply":"2023-11-17T21:38:49.241111Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The following cell creates a Dataloader for the dataset","metadata":{}},{"cell_type":"code","source":"class DatasetLoader:\n def __init__(self, \n dataset_path, \n batch_size = 4, \n image_size = (256, 256), \n num_images = None):\n \n images_paths = [str(path) for path in Path(dataset_path).glob(\"*.jpg\")]\n self.length = len(images_paths)\n if num_images is not None:\n images_paths = images_paths[0 : num_images]\n dataset = tf.data.Dataset.from_tensor_slices(images_paths).map(\n lambda path: self.load_tf_image(path, dim = image_size),\n num_parallel_calls = tf.data.experimental.AUTOTUNE,\n )\n dataset = dataset.batch(batch_size, drop_remainder = True)\n dataset = dataset.repeat()\n dataset = dataset.prefetch(buffer_size = tf.data.experimental.AUTOTUNE)\n self.dataset = dataset\n \n def __len__(self):\n return self.length\n \n def load_tf_image(self, image_path, dim):\n image = tf.io.read_file(image_path)\n image = tf.image.decode_jpeg(image, channels = 3)\n image= tf.image.resize(image, dim)\n image= image / 255.0\n image = tf.image.convert_image_dtype(image, tf.float32)\n return image","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:38:49.243248Z","iopub.execute_input":"2023-11-17T21:38:49.243632Z","iopub.status.idle":"2023-11-17T21:38:49.255222Z","shell.execute_reply.started":"2023-11-17T21:38:49.24358Z","shell.execute_reply":"2023-11-17T21:38:49.254333Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Defining input shape and batch size, and creating a dataloader for the dataset","metadata":{}},{"cell_type":"code","source":"input_shape = (256,256,3)\nbatch_size = 16","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:38:49.256441Z","iopub.execute_input":"2023-11-17T21:38:49.257232Z","iopub.status.idle":"2023-11-17T21:38:49.264778Z","shell.execute_reply.started":"2023-11-17T21:38:49.257207Z","shell.execute_reply":"2023-11-17T21:38:49.263926Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"loader = DatasetLoader(content_path, batch_size = batch_size)","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:38:49.266065Z","iopub.execute_input":"2023-11-17T21:38:49.266451Z","iopub.status.idle":"2023-11-17T21:38:49.564271Z","shell.execute_reply.started":"2023-11-17T21:38:49.266419Z","shell.execute_reply":"2023-11-17T21:38:49.56334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Visualizing the specifications of dataset images","metadata":{}},{"cell_type":"code","source":"loader.dataset.element_spec","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:38:49.565389Z","iopub.execute_input":"2023-11-17T21:38:49.565685Z","iopub.status.idle":"2023-11-17T21:38:49.571326Z","shell.execute_reply.started":"2023-11-17T21:38:49.56566Z","shell.execute_reply":"2023-11-17T21:38:49.570535Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_images_grid(next(iter(loader.dataset.take(1))), num_rows = 4)","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:38:49.572554Z","iopub.execute_input":"2023-11-17T21:38:49.572867Z","iopub.status.idle":"2023-11-17T21:38:51.841752Z","shell.execute_reply.started":"2023-11-17T21:38:49.572837Z","shell.execute_reply":"2023-11-17T21:38:51.840348Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Building the model\n","metadata":{"id":"Jt3i3RRrJiOX"}},{"cell_type":"markdown","source":"### Loss Model","metadata":{}},{"cell_type":"code","source":"vgg = vgg19.VGG19(weights = 'imagenet', include_top = False)\nvgg.summary()","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:38:51.843113Z","iopub.execute_input":"2023-11-17T21:38:51.843437Z","iopub.status.idle":"2023-11-17T21:38:52.911363Z","shell.execute_reply.started":"2023-11-17T21:38:51.843407Z","shell.execute_reply":"2023-11-17T21:38:52.910484Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"content_layers = ['block4_conv2']\nstyle_layers = ['block1_conv1',\n 'block2_conv1',\n 'block3_conv1',\n 'block4_conv1',\n 'block5_conv1']\n\ncontent_layers_weights = [1]\nstyle_layers_weights = [1] * 5","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:38:52.912594Z","iopub.execute_input":"2023-11-17T21:38:52.912885Z","iopub.status.idle":"2023-11-17T21:38:52.917401Z","shell.execute_reply.started":"2023-11-17T21:38:52.912859Z","shell.execute_reply":"2023-11-17T21:38:52.916518Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Creating Loss model which is used to calcuate perceptual loss by combining style and content loss**","metadata":{}},{"cell_type":"code","source":"class LossModel:\n def __init__(self, pretrained_model, content_layers, style_layers):\n self.model = pretrained_model\n self.content_layers = content_layers\n self.style_layers = style_layers\n self.loss_model = self.get_model()\n\n def get_model(self):\n self.model.trainable = False\n layer_names = self.style_layers + self.content_layers\n outputs = [self.model.get_layer(name).output for name in layer_names]\n new_model = Model(inputs = self.model.input, outputs = outputs)\n return new_model\n \n def get_activations(self,inputs):\n inputs = inputs * 255.0\n style_length = len(self.style_layers)\n outputs = self.loss_model(vgg19.preprocess_input(inputs))\n style_output, content_output = outputs[ : style_length], outputs[style_length : ]\n content_dict = {name: value for name, value in zip(self.content_layers, content_output)}\n style_dict = {name: value for name, value in zip(self.style_layers, style_output)}\n return {'content': content_dict, 'style': style_dict}","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:38:52.91854Z","iopub.execute_input":"2023-11-17T21:38:52.918811Z","iopub.status.idle":"2023-11-17T21:38:52.932271Z","shell.execute_reply.started":"2023-11-17T21:38:52.918788Z","shell.execute_reply":"2023-11-17T21:38:52.931334Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"loss_model = LossModel(vgg, content_layers, style_layers)","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:38:52.933252Z","iopub.execute_input":"2023-11-17T21:38:52.93356Z","iopub.status.idle":"2023-11-17T21:38:52.946677Z","shell.execute_reply.started":"2023-11-17T21:38:52.933529Z","shell.execute_reply":"2023-11-17T21:38:52.945741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Defining the loss function","metadata":{}},{"cell_type":"code","source":"def content_loss(placeholder, content,weight):\n assert placeholder.shape == content.shape\n return weight * tf.reduce_mean(tf.square(placeholder - content))","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:38:52.947909Z","iopub.execute_input":"2023-11-17T21:38:52.94815Z","iopub.status.idle":"2023-11-17T21:38:52.954666Z","shell.execute_reply.started":"2023-11-17T21:38:52.948129Z","shell.execute_reply":"2023-11-17T21:38:52.953745Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def gram_matrix(x):\n gram = tf.linalg.einsum('bijc,bijd->bcd', x, x)\n return gram / tf.cast(x.shape[1] * x.shape[2] * x.shape[3], tf.float32)","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:38:52.955731Z","iopub.execute_input":"2023-11-17T21:38:52.956049Z","iopub.status.idle":"2023-11-17T21:38:52.963947Z","shell.execute_reply.started":"2023-11-17T21:38:52.956019Z","shell.execute_reply":"2023-11-17T21:38:52.96308Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def style_loss(placeholder, style,weight):\n assert placeholder.shape == style.shape\n s = gram_matrix(style)\n p = gram_matrix(placeholder)\n return weight * tf.reduce_mean(tf.square(s - p))","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:38:52.965156Z","iopub.execute_input":"2023-11-17T21:38:52.965482Z","iopub.status.idle":"2023-11-17T21:38:52.97323Z","shell.execute_reply.started":"2023-11-17T21:38:52.96545Z","shell.execute_reply":"2023-11-17T21:38:52.972401Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def perceptual_loss(predicted_activations,\n content_activations,\n style_activations,\n content_weight, \n style_weight,\n content_layers_weights,\n style_layer_weights):\n \n pred_content = predicted_activations[\"content\"]\n pred_style = predicted_activations[\"style\"]\n c_loss = tf.add_n([content_loss(pred_content[name],\n content_activations[name],\n content_layers_weights[i]) for i, name in enumerate(pred_content.keys())])\n c_loss = c_loss * content_weight\n s_loss = tf.add_n([style_loss(pred_style[name],\n style_activations[name],\n style_layer_weights[i]) for i, name in enumerate(pred_style.keys())])\n s_loss = s_loss * style_weight\n return c_loss + s_loss","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:38:52.974321Z","iopub.execute_input":"2023-11-17T21:38:52.974667Z","iopub.status.idle":"2023-11-17T21:38:52.98758Z","shell.execute_reply.started":"2023-11-17T21:38:52.974636Z","shell.execute_reply":"2023-11-17T21:38:52.986782Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Fast Transfer Model","metadata":{}},{"cell_type":"code","source":"class ReflectionPadding2D(tf.keras.layers.Layer):\n def __init__(self, padding = (1, 1), **kwargs):\n super(ReflectionPadding2D, self).__init__(**kwargs)\n self.padding = tuple(padding)\n def call(self, input_tensor):\n padding_width, padding_height = self.padding\n return tf.pad(input_tensor, [[0,0], [padding_height, padding_height], [padding_width, padding_width], [0,0]], 'REFLECT')","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:38:52.988657Z","iopub.execute_input":"2023-11-17T21:38:52.988906Z","iopub.status.idle":"2023-11-17T21:38:52.999481Z","shell.execute_reply.started":"2023-11-17T21:38:52.988885Z","shell.execute_reply":"2023-11-17T21:38:52.998774Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class InstanceNormalization(tf.keras.layers.Layer):\n def __init__(self, **kwargs):\n super(InstanceNormalization, self).__init__(**kwargs)\n def call(self, inputs):\n batch, rows, cols, channels = [i for i in inputs.get_shape()]\n mu, var = tf.nn.moments(inputs, [1, 2], keepdims = True)\n shift = tf.Variable(tf.zeros([channels]))\n scale = tf.Variable(tf.ones([channels]))\n epsilon = 1e-3\n normalized = (inputs - mu) / tf.sqrt(var + epsilon)\n return scale * normalized + shift","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:38:53.000595Z","iopub.execute_input":"2023-11-17T21:38:53.001762Z","iopub.status.idle":"2023-11-17T21:38:53.010026Z","shell.execute_reply.started":"2023-11-17T21:38:53.00173Z","shell.execute_reply":"2023-11-17T21:38:53.009234Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class ConvLayer(tf.keras.layers.Layer):\n def __init__(self, filters, kernel_size, strides = 1, **kwargs):\n super(ConvLayer,self).__init__(**kwargs)\n self.padding = ReflectionPadding2D([k // 2 for k in kernel_size])\n self.conv2d = tf.keras.layers.Conv2D(filters, kernel_size, strides)\n self.bn = InstanceNormalization()\n def call(self, inputs):\n x = self.padding(inputs)\n x = self.conv2d(x)\n x = self.bn(x)\n return x","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:38:53.010887Z","iopub.execute_input":"2023-11-17T21:38:53.01113Z","iopub.status.idle":"2023-11-17T21:38:53.022144Z","shell.execute_reply.started":"2023-11-17T21:38:53.011099Z","shell.execute_reply":"2023-11-17T21:38:53.021454Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class ResidualLayer(tf.keras.layers.Layer):\n def __init__(self, filters, kernel_size, **kwargs):\n super(ResidualLayer, self).__init__(**kwargs)\n self.conv2d_1 = ConvLayer(filters, kernel_size)\n self.conv2d_2 = ConvLayer(filters, kernel_size)\n self.relu = tf.keras.layers.ReLU()\n self.add = tf.keras.layers.Add()\n def call(self, inputs):\n residual = inputs\n x = self.conv2d_1(inputs)\n x = self.relu(x)\n x = self.conv2d_2(x)\n x = self.add([x, residual])\n return x","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:38:53.026565Z","iopub.execute_input":"2023-11-17T21:38:53.026832Z","iopub.status.idle":"2023-11-17T21:38:53.033716Z","shell.execute_reply.started":"2023-11-17T21:38:53.02681Z","shell.execute_reply":"2023-11-17T21:38:53.032801Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class UpsampleLayer(tf.keras.layers.Layer):\n def __init__(self, filters, kernel_size, strides = 1, upsample = 2, **kwargs):\n super(UpsampleLayer, self).__init__(**kwargs)\n self.upsample = tf.keras.layers.UpSampling2D(size = upsample)\n self.padding = ReflectionPadding2D([k//2 for k in kernel_size])\n self.conv2d = tf.keras.layers.Conv2D(filters,kernel_size,strides)\n self.bn = InstanceNormalization()\n def call(self,inputs):\n x = self.upsample(inputs)\n x = self.padding(x)\n x = self.conv2d(x)\n return self.bn(x)","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:38:53.034962Z","iopub.execute_input":"2023-11-17T21:38:53.035294Z","iopub.status.idle":"2023-11-17T21:38:53.043318Z","shell.execute_reply.started":"2023-11-17T21:38:53.035262Z","shell.execute_reply":"2023-11-17T21:38:53.042558Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class StyleTransferModel(tf.keras.Model):\n def __init__(self, **kwargs):\n super(StyleTransferModel, self).__init__(name = 'StyleTransferModel', **kwargs)\n self.conv2d_1 = ConvLayer(filters = 32, kernel_size = (9, 9), strides = 1, name = \"conv2d_1_32\")\n self.conv2d_2 = ConvLayer(filters = 64, kernel_size = (3, 3), strides = 2, name = \"conv2d_2_64\")\n self.conv2d_3 = ConvLayer(filters = 128, kernel_size = (3, 3), strides = 2, name = \"conv2d_3_128\")\n self.res_1 = ResidualLayer(filters = 128, kernel_size = (3, 3), name = \"res_1_128\")\n self.res_2 = ResidualLayer(filters = 128, kernel_size = (3, 3), name = \"res_2_128\")\n self.res_3 = ResidualLayer(filters = 128, kernel_size = (3, 3), name = \"res_3_128\")\n self.res_4 = ResidualLayer(filters = 128, kernel_size = (3, 3), name = \"res_4_128\")\n self.res_5 = ResidualLayer(filters = 128, kernel_size = (3, 3), name = \"res_5_128\")\n self.deconv2d_1 = UpsampleLayer(filters = 64, kernel_size = (3, 3),name = \"deconv2d_1_64\")\n self.deconv2d_2 = UpsampleLayer(filters = 32, kernel_size = (3, 3), name = \"deconv2d_2_32\")\n self.deconv2d_3 = ConvLayer(filters = 3, kernel_size = (9, 9), strides = 1, name = \"deconv2d_3_3\")\n self.relu = tf.keras.layers.ReLU()\n \n def call(self, inputs):\n x = self.conv2d_1(inputs)\n x = self.relu(x)\n x = self.conv2d_2(x)\n x = self.relu(x)\n x = self.conv2d_3(x)\n x = self.relu(x)\n x = self.res_1(x)\n x = self.res_2(x)\n x = self.res_3(x)\n x = self.res_4(x)\n x = self.res_5(x)\n x = self.deconv2d_1(x)\n x = self.relu(x)\n x = self.deconv2d_2(x)\n x = self.relu(x)\n x = self.deconv2d_3(x)\n x = (tf.nn.tanh(x) + 1) * (255.0 / 2)\n return x","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:38:53.044259Z","iopub.execute_input":"2023-11-17T21:38:53.044558Z","iopub.status.idle":"2023-11-17T21:38:53.05825Z","shell.execute_reply.started":"2023-11-17T21:38:53.044521Z","shell.execute_reply":"2023-11-17T21:38:53.057553Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"style_model = StyleTransferModel()","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:38:53.059258Z","iopub.execute_input":"2023-11-17T21:38:53.059601Z","iopub.status.idle":"2023-11-17T21:38:53.134453Z","shell.execute_reply.started":"2023-11-17T21:38:53.059569Z","shell.execute_reply":"2023-11-17T21:38:53.133801Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Training the Model","metadata":{}},{"cell_type":"markdown","source":"### Utilities","metadata":{}},{"cell_type":"code","source":"optimizer = tf.keras.optimizers.Adam(learning_rate = 1e-3)","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:38:53.135451Z","iopub.execute_input":"2023-11-17T21:38:53.135789Z","iopub.status.idle":"2023-11-17T21:38:53.144259Z","shell.execute_reply.started":"2023-11-17T21:38:53.135758Z","shell.execute_reply":"2023-11-17T21:38:53.1434Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def train_step(dataset,\n style_activations,\n steps_per_epoch,\n style_model,\n loss_model,\n optimizer,\n checkpoint_path = \"./\",\n content_weight = 1e4,\n style_weight = 1e-2,\n total_variation_weight = 0.004,\n content_layers_weights = [1],\n style_layers_weights = [1] * 5):\n\n batch_losses = []\n steps = 1\n save_path = os.path.join(checkpoint_path, f\"model_checkpoint.ckpt\")\n print(\"Model Checkpoint Path: \", save_path)\n for input_image_batch in dataset:\n if steps - 1 >= steps_per_epoch:\n break\n with tf.GradientTape() as tape:\n outputs = style_model(input_image_batch)\n outputs = tf.clip_by_value(outputs, 0, 255)\n pred_activations = loss_model.get_activations(outputs / 255.0)\n content_activations = loss_model.get_activations(input_image_batch)[\"content\"] \n curr_loss = preceptual_loss(pred_activations,\n content_activations,\n style_activations,\n content_weight,\n style_weight,\n content_layers_weights,\n style_layers_weights)\n curr_loss += total_variation_weight * tf.image.total_variation(outputs)\n batch_losses.append(curr_loss)\n grad = tape.gradient(curr_loss, style_model.trainable_variables)\n optimizer.apply_gradients(zip(grad, style_model.trainable_variables))\n if steps % 100 == 0:\n print(\"Checkpoint Saved\", end = \" \")\n style_model.save_weights(save_path)\n print(f\"Loss: {tf.reduce_mean(batch_losses).numpy()}\")\n steps += 1\n return tf.reduce_mean(batch_losses)","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:38:53.145448Z","iopub.execute_input":"2023-11-17T21:38:53.14579Z","iopub.status.idle":"2023-11-17T21:38:53.15595Z","shell.execute_reply.started":"2023-11-17T21:38:53.145759Z","shell.execute_reply":"2023-11-17T21:38:53.155089Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Training the style model","metadata":{}},{"cell_type":"markdown","source":"Setting up style image for training","metadata":{}},{"cell_type":"code","source":"style_image_path = os.path.join(style_path, \"3417ace946.jpg\")\nstyle_image = load_image(style_image_path, dim = (input_shape[0], input_shape[1]), resize = True)\nstyle_image = style_image / 255.0","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:48:21.54616Z","iopub.execute_input":"2023-11-17T21:48:21.546576Z","iopub.status.idle":"2023-11-17T21:48:21.555548Z","shell.execute_reply.started":"2023-11-17T21:48:21.546546Z","shell.execute_reply":"2023-11-17T21:48:21.554554Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_image(style_image)","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:47:23.933317Z","iopub.execute_input":"2023-11-17T21:47:23.933706Z","iopub.status.idle":"2023-11-17T21:47:24.501886Z","shell.execute_reply.started":"2023-11-17T21:47:23.933675Z","shell.execute_reply":"2023-11-17T21:47:24.500779Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"style_image = style_image.astype(np.float32)\nstyle_image_batch=np.repeat([style_image],batch_size,axis=0)\nstyle_activations=loss_model.get_activations(style_image_batch)[\"style\"]","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"epochs = 10\ncontent_weight = 2*1e1\nstyle_weight = 1e2\ntotal_variation_weight = 0.004","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:38:53.157013Z","iopub.execute_input":"2023-11-17T21:38:53.157288Z","iopub.status.idle":"2023-11-17T21:38:53.16932Z","shell.execute_reply.started":"2023-11-17T21:38:53.157245Z","shell.execute_reply":"2023-11-17T21:38:53.168494Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_images = len(loader)\nsteps_per_epochs = num_images // batch_size\nprint(steps_per_epochs)","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:38:53.170295Z","iopub.execute_input":"2023-11-17T21:38:53.170586Z","iopub.status.idle":"2023-11-17T21:38:53.179204Z","shell.execute_reply.started":"2023-11-17T21:38:53.170563Z","shell.execute_reply":"2023-11-17T21:38:53.178336Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model_save_path = \"model_checkpoint\"","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:39:40.468427Z","iopub.execute_input":"2023-11-17T21:39:40.468812Z","iopub.status.idle":"2023-11-17T21:39:40.472973Z","shell.execute_reply.started":"2023-11-17T21:39:40.468785Z","shell.execute_reply":"2023-11-17T21:39:40.47212Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"os.makedirs(model_save_path, exist_ok = True)","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:39:49.968007Z","iopub.execute_input":"2023-11-17T21:39:49.968363Z","iopub.status.idle":"2023-11-17T21:39:49.973083Z","shell.execute_reply.started":"2023-11-17T21:39:49.968336Z","shell.execute_reply":"2023-11-17T21:39:49.972183Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"try:\n policy = tf.keras.mixed_precision.experimental.Policy('mixed_float16')\n tf.keras.mixed_precision.experimental.set_policy(policy) \nexcept:\n pass","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:39:59.953416Z","iopub.execute_input":"2023-11-17T21:39:59.953762Z","iopub.status.idle":"2023-11-17T21:39:59.958693Z","shell.execute_reply.started":"2023-11-17T21:39:59.953735Z","shell.execute_reply":"2023-11-17T21:39:59.957689Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"try:\n tf.config.optimizer.set_jit(True)\nexcept:\n pass","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:40:07.744001Z","iopub.execute_input":"2023-11-17T21:40:07.744863Z","iopub.status.idle":"2023-11-17T21:40:07.749156Z","shell.execute_reply.started":"2023-11-17T21:40:07.74483Z","shell.execute_reply":"2023-11-17T21:40:07.748187Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"if os.path.isfile(os.path.join(model_save_path, \"model_checkpoint.ckpt.index\")):\n style_model.load_weights(os.path.join(model_save_path, \"model_checkpoint.ckpt\"))\n print(\"Resuming training ...\")\nelse:\n print(\"Training from scratch ...\")","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:40:32.834476Z","iopub.execute_input":"2023-11-17T21:40:32.834898Z","iopub.status.idle":"2023-11-17T21:40:32.840905Z","shell.execute_reply.started":"2023-11-17T21:40:32.834869Z","shell.execute_reply":"2023-11-17T21:40:32.839949Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"epoch_losses = []\nfor epoch in range(1, epochs + 1):\n print(f\"epoch: {epoch}\")\n batch_loss = train_step(loader.dataset,\n style_activations,\n steps_per_epochs,\n style_model,\n loss_model,\n optimizer,\n model_save_path,\n content_weight,\n style_weight,\n total_variation_weight,\n content_layers_weights,\n style_layers_weights)\n style_model.save_weights(os.path.join(model_save_path, \"model_checkpoint.ckpt\"))\n print(\"Model Checkpointed at: \",os.path.join(model_save_path, \"model_checkpoint.ckpt\"))\n print(f\"loss: {batch_loss.numpy()}\")\n epoch_losses.append(batch_loss)","metadata":{"execution":{"iopub.status.busy":"2023-11-17T21:41:36.477459Z","iopub.execute_input":"2023-11-17T21:41:36.478187Z","iopub.status.idle":"2023-11-17T21:41:36.519441Z","shell.execute_reply.started":"2023-11-17T21:41:36.478156Z","shell.execute_reply":"2023-11-17T21:41:36.518247Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}