{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2021-12-03T09:29:20.999646Z","iopub.execute_input":"2021-12-03T09:29:21.000358Z","iopub.status.idle":"2021-12-03T09:29:34.151296Z","shell.execute_reply.started":"2021-12-03T09:29:21.000246Z","shell.execute_reply":"2021-12-03T09:29:34.150329Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---------------------------------------\n# Introduction","metadata":{}},{"cell_type":"markdown","source":"![](https://miro.medium.com/max/1200/1*XI3beonBnOwp-y5BwNOqCw.gif)\n\nPicture Credit: https://miro.medium.com","metadata":{}},{"cell_type":"markdown","source":"**Neural Style Transfer**\n\n> Neural Style Transfer (NST) refers to a class of software algorithms that manipulate digital images, or videos, in order to adopt the appearance or visual style of another image. NST algorithms are characterized by their use of deep neural networks for the sake of image transformation. Common uses for NST are the creation of artificial artwork from photographs, for example by transferring the appearance of famous paintings to user-supplied photographs. Several notable mobile apps use NST techniques for this purpose, including DeepArt and Prisma. This method has been used by artists and designers around the globe to develop new artwork based on existent style(s).\n\nRef: https://en.wikipedia.org/wiki/Neural_Style_Transfer","metadata":{}},{"cell_type":"markdown","source":"Style transfer means that when a content image and a style image are given, the outline and shape of the image are similar to the content image, and the color or texture is changed to be similar to the style image.\n\nBy separating content and style, you can mix content and style of different images.\n","metadata":{}},{"cell_type":"markdown","source":"A pre-trained VGG19 Net is used as a model to extract content and style. It then uses the losses of the content and style to iteratively update the target image until the desired result is achieved.","metadata":{}},{"cell_type":"code","source":"# import resources\n%matplotlib inline\n\nfrom PIL import Image\nfrom io import BytesIO\nimport matplotlib.pyplot as plt\nimport numpy as np\n\nimport torch\nimport torch.optim as optim\nimport requests\nfrom torchvision import transforms, models","metadata":{"execution":{"iopub.status.busy":"2021-12-03T09:29:34.152927Z","iopub.execute_input":"2021-12-03T09:29:34.153172Z","iopub.status.idle":"2021-12-03T09:29:35.412864Z","shell.execute_reply.started":"2021-12-03T09:29:34.153146Z","shell.execute_reply":"2021-12-03T09:29:35.411989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"------------------------------------------------\n# 1. Load in model\n\nVGG19 is divided into two parts.\n\n* vgg19.features: All convolutional layers and pooling layers\n* vgg19.classifier: The last three linear layers are the classifier layer.\n\nWe only need the features part. And \"freeze\" so that the weight is not updated.","metadata":{}},{"cell_type":"code","source":"# get the \"features\" portion of VGG19 (we will not need the \"classifier\" portion)\nvgg = models.vgg19(pretrained=True).features\n\n# freeze all VGG parameters since we're only optimizing the target image\nfor param in vgg.parameters():\n    param.requires_grad_(False)","metadata":{"execution":{"iopub.status.busy":"2021-12-03T09:29:35.414226Z","iopub.execute_input":"2021-12-03T09:29:35.414447Z","iopub.status.idle":"2021-12-03T09:29:40.18844Z","shell.execute_reply.started":"2021-12-03T09:29:35.414421Z","shell.execute_reply":"2021-12-03T09:29:40.187541Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# move the model to GPU, if available\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n\nvgg.to(device)","metadata":{"execution":{"iopub.status.busy":"2021-12-03T09:29:40.189849Z","iopub.execute_input":"2021-12-03T09:29:40.190108Z","iopub.status.idle":"2021-12-03T09:29:40.202727Z","shell.execute_reply.started":"2021-12-03T09:29:40.19008Z","shell.execute_reply":"2021-12-03T09:29:40.202089Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"-----------------------------------------------\n# 2. Load in Content and Style Images\n\nLoad the content image and style image to be used for style transfer. The load_image function transforms the image and loads it in the form of normalized Tensors.\n\n","metadata":{}},{"cell_type":"code","source":"def load_image(img_path, max_size=128, shape=None):\n    ''' Load in and transform an image, making sure the image\n       is <= 400 pixels in the x-y dims.'''\n    if \"http\" in img_path:\n        response = requests.get(img_path)\n        image = Image.open(BytesIO(response.content)).convert('RGB')\n    else:\n        image = Image.open(img_path).convert('RGB')\n    \n    # large images will slow down processing\n    if max(image.size) > max_size:\n        size = max_size\n    else:\n        size = max(image.size)\n    \n    if shape is not None:\n        size = shape\n        \n    in_transform = transforms.Compose([\n                        transforms.Resize(size),\n                        transforms.ToTensor(),\n                        transforms.Normalize((0.485, 0.456, 0.406), \n                                             (0.229, 0.224, 0.225))])\n\n    # discard the transparent, alpha channel (that's the :3) and add the batch dimension\n    image = in_transform(image)[:3,:,:].unsqueeze(0)\n    \n    return image","metadata":{"execution":{"iopub.status.busy":"2021-12-03T09:29:40.204517Z","iopub.execute_input":"2021-12-03T09:29:40.204876Z","iopub.status.idle":"2021-12-03T09:29:40.449294Z","shell.execute_reply.started":"2021-12-03T09:29:40.204847Z","shell.execute_reply":"2021-12-03T09:29:40.448535Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Load the content image and style image.","metadata":{}},{"cell_type":"code","source":"# load in content and style image\ncontent = load_image('/kaggle/input/images-for-style-transfer/Data/Artworks/856047.jpg').to(device)\n# Resize style to match content, makes code easier\nstyle = load_image('/kaggle/input/neural-style-transfer/Style Images/starry_night.jpg').to(device)","metadata":{"execution":{"iopub.status.busy":"2021-12-03T09:29:40.451282Z","iopub.execute_input":"2021-12-03T09:29:40.451513Z","iopub.status.idle":"2021-12-03T09:29:40.716232Z","shell.execute_reply.started":"2021-12-03T09:29:40.451488Z","shell.execute_reply":"2021-12-03T09:29:40.715341Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# helper function for un-normalizing an image \n# and converting it from a Tensor image to a NumPy image for display\ndef im_convert(tensor):\n    \"\"\" Display a tensor as an image. \"\"\"\n    \n    image = tensor.to(\"cpu\").clone().detach()\n    image = image.numpy().squeeze()\n    image = image.transpose(1,2,0)\n    image = image * np.array((0.229, 0.224, 0.225)) + np.array((0.485, 0.456, 0.406))\n    image = image.clip(0, 1)\n\n    return image","metadata":{"execution":{"iopub.status.busy":"2021-12-03T09:29:40.717305Z","iopub.execute_input":"2021-12-03T09:29:40.717528Z","iopub.status.idle":"2021-12-03T09:29:40.724688Z","shell.execute_reply.started":"2021-12-03T09:29:40.717501Z","shell.execute_reply":"2021-12-03T09:29:40.723804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# display the images\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(20, 10))\n# content and style ims side-by-side\nax1.imshow(im_convert(content))\nax2.imshow(im_convert(style))","metadata":{"execution":{"iopub.status.busy":"2021-12-03T09:29:40.725884Z","iopub.execute_input":"2021-12-03T09:29:40.726203Z","iopub.status.idle":"2021-12-03T09:29:41.34582Z","shell.execute_reply.started":"2021-12-03T09:29:40.726173Z","shell.execute_reply":"2021-12-03T09:29:41.345005Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_features(image, model, layers=None):\n    \"\"\" Run an image forward through a model and get the features for \n        a set of layers. Default layers are for VGGNet matching Gatys et al (2016)\n    \"\"\"\n    \n    if layers is None:\n        layers = {'0': 'conv1_1',\n                  '5': 'conv2_1', \n                  '10': 'conv3_1', \n                  '19': 'conv4_1',\n                  '21': 'conv4_2',  ## content representation\n                  '28': 'conv5_1'}\n        \n    features = {}\n    x = image\n    # model._modules is a dictionary holding each module in the model\n    for name, layer in model._modules.items():\n        x = layer(x)\n        if name in layers:\n            features[layers[name]] = x\n            \n    return features","metadata":{"execution":{"iopub.status.busy":"2021-12-03T09:29:41.346932Z","iopub.execute_input":"2021-12-03T09:29:41.347162Z","iopub.status.idle":"2021-12-03T09:29:41.35433Z","shell.execute_reply.started":"2021-12-03T09:29:41.347137Z","shell.execute_reply":"2021-12-03T09:29:41.353367Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"--------------------------------------------------\n# 3. Gram Matrix\n\n![](https://miro.medium.com/max/1400/1*VAQs1KSfbysnloPah_fHGQ.gif)\n\nPicture Credit: https://miro.medium.com\n\n\n> The matrix expressing the correlation of this Channel is called Gram Matrix. Loss is minimized by defining the difference between this Gram Matrix and the Gram Matrix of the newly created image as a Loss Function. Next, in order to reflect the content, the loss function is calculated in units of pixels from the feature map spit out from each pre-trained CNN. In this way, a new image is created that minimizes the Loss calculated from Style and Loss calculated from Content. \n\nhttps://en.wikipedia.org/wiki/Gram_matrix\n","metadata":{}},{"cell_type":"code","source":"def gram_matrix(tensor):\n    \"\"\" Calculate the Gram Matrix of a given tensor         \n    \"\"\"\n    \n    # get the batch_size, depth, height, and width of the Tensor\n    _, d, h, w = tensor.size()\n    \n    # reshape so we're multiplying the features for each channel\n    tensor = tensor.view(d, h * w)\n    \n    # calculate the gram matrix\n    gram = torch.mm(tensor, tensor.t())\n    \n    return gram ","metadata":{"execution":{"iopub.status.busy":"2021-12-03T09:29:41.355463Z","iopub.execute_input":"2021-12-03T09:29:41.355778Z","iopub.status.idle":"2021-12-03T09:29:41.372561Z","shell.execute_reply.started":"2021-12-03T09:29:41.355742Z","shell.execute_reply":"2021-12-03T09:29:41.371467Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The function that extracts the features of a given convolutional layer and computes the Gram Matrix is made. Putting it all together, we extract the features from the image and compute the Gram Matrix for each layer from the style representation.","metadata":{}},{"cell_type":"code","source":"# get content and style features only once before training\ncontent_features = get_features(content, vgg)\nstyle_features = get_features(style, vgg)\n\n# calculate the gram matrices for each layer of our style representation\nstyle_grams = {layer: gram_matrix(style_features[layer]) for layer in style_features}\n\n# create a third \"target\" image and prep it for change\n# it is a good idea to start off with the target as a copy of our *content* image\n# then iteratively change its style\ntarget = content.clone().requires_grad_(True).to(device)","metadata":{"execution":{"iopub.status.busy":"2021-12-03T09:29:41.374246Z","iopub.execute_input":"2021-12-03T09:29:41.37456Z","iopub.status.idle":"2021-12-03T09:29:41.777339Z","shell.execute_reply.started":"2021-12-03T09:29:41.37451Z","shell.execute_reply":"2021-12-03T09:29:41.776573Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---------------------------------------------\n# 4. Define Losses and Weights\n\n**Individual Layer Style Weights**\n\nYou can give the option to weight the style expression in each relevant layer. It is recommended that the layer weight range from 0 to 1. By giving more weight to conv1_1 and conv2_1, more style artifacts can be reflected in the final target image.\n\n**Content and Style Weight**\n\nDefine alpha (content_weight) and beta (style_weight). This ratio affects the style of the final image. It is recommended to leave content_weight = 1 and set style_weight to achieve the desired ratio.","metadata":{}},{"cell_type":"code","source":"# weights for each style layer \n# weighting earlier layers more will result in *larger* style artifacts\n# notice we are excluding `conv4_2` our content representation\nstyle_weights = {'conv1_1': 1,\n                 'conv2_1': 0.75,\n                 'conv3_1': 0.2,\n                 'conv4_1': 0.2,\n                 'conv5_1': 0.2}\n\ncontent_weight = 1  # alpha\nstyle_weight = 1e3  # beta","metadata":{"execution":{"iopub.status.busy":"2021-12-03T09:29:41.77892Z","iopub.execute_input":"2021-12-03T09:29:41.779258Z","iopub.status.idle":"2021-12-03T09:29:41.786522Z","shell.execute_reply.started":"2021-12-03T09:29:41.779218Z","shell.execute_reply":"2021-12-03T09:29:41.785716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Update Target and Calculate Losses\n\n**Content Loss**\n\nThe content loss is calculated as the MSE between the target and the content feature in the 'conv4_2' layer.\n\n**Style Loss**\n\nThe style loss is the loss between the target image and the style image. That is, it refers to the difference between the gram matrix of the style image and the gram matrix of the target image. Loss is calculated using MSE\n\n**Total Loss**\n\nFinally, the total loss is calculated by summing the style and content losses and weighting them with the specified alpha and beta.","metadata":{}},{"cell_type":"code","source":"# for displaying the target image, intermittently\nshow_every = 500\n\n# iteration hyperparameters\noptimizer = optim.Adam([target], lr=0.003)\nsteps = 5001  # decide how many iterations to update your image (5000)\n\nfor ii in range(1, steps+1):\n    \n    # get the features from your target image\n    target_features = get_features(target, vgg)\n    \n    # the content loss\n    content_loss = torch.mean((target_features['conv4_2'] - content_features['conv4_2'])**2)\n    \n    # the style loss\n    # initialize the style loss to 0\n    style_loss = 0\n    # then add to it for each layer's gram matrix loss\n    for layer in style_weights:\n        # get the \"target\" style representation for the layer\n        target_feature = target_features[layer]\n        target_gram = gram_matrix(target_feature)\n        _, d, h, w = target_feature.shape\n        # get the \"style\" style representation\n        style_gram = style_grams[layer]\n        # the style loss for one layer, weighted appropriately\n        layer_style_loss = style_weights[layer] * torch.mean((target_gram - style_gram)**2)\n        # add to the style loss\n        style_loss += layer_style_loss / (d * h * w)\n        \n    # calculate the *total* loss\n    total_loss = content_weight * content_loss + style_weight * style_loss\n    \n    # update your target image\n    optimizer.zero_grad()\n    total_loss.backward()\n    optimizer.step()\n    \n    # display intermediate images and print the loss\n    if  ii % show_every == 0:\n        print('Total loss: ', total_loss.item())\n        plt.imshow(im_convert(target))\n        plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-12-03T09:29:41.788053Z","iopub.execute_input":"2021-12-03T09:29:41.788576Z","iopub.status.idle":"2021-12-03T09:33:08.264381Z","shell.execute_reply.started":"2021-12-03T09:29:41.788533Z","shell.execute_reply":"2021-12-03T09:33:08.263058Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"-----------------------------------------------------------------\n# 6. Check the last result","metadata":{}},{"cell_type":"code","source":"# display content and final, target image\nfig, (ax1, ax2) = plt.subplots(1, 2, figsize=(20, 10))\nax1.imshow(im_convert(content))\nax2.imshow(im_convert(target))","metadata":{"execution":{"iopub.status.busy":"2021-12-03T09:33:08.265428Z","iopub.status.idle":"2021-12-03T09:33:08.265828Z","shell.execute_reply.started":"2021-12-03T09:33:08.265651Z","shell.execute_reply":"2021-12-03T09:33:08.265669Z"},"trusted":true},"execution_count":null,"outputs":[]}]}