{"cells":[{"metadata":{"id":"6msVLevwcRhm"},"cell_type":"markdown","source":"# Neural style transfer\n\nThis notebook is based on the first portion of the 2018 TensorFlow Authors [Neural Style Transfer](https://www.tensorflow.org/tutorials/generative/style_transfer) tutorial, the notebook from which was released under the [Apache License, version 2.0](https://www.apache.org/licenses/LICENSE-2.0). I have made a few modifications, mostly related to data loading from a local filepath vs. a web url.  You do not need a GPU or TPU to get good results fast."},{"metadata":{"id":"eqxUicSPUOP6"},"cell_type":"markdown","source":"### Import and configure modules"},{"metadata":{"id":"sc1OLbOWhPCO","trusted":true},"cell_type":"code","source":"import tensorflow as tf\nfrom IPython.display import Image\nfrom IPython.core.display import HTML\n\nimport matplotlib.pyplot as plt\nimport matplotlib as mpl\nmpl.rcParams['figure.figsize'] = (12,12)\nmpl.rcParams['axes.grid'] = False\n\nimport numpy as np\nimport PIL.Image","execution_count":null,"outputs":[]},{"metadata":{"id":"GM6VEGrGLh62","trusted":true},"cell_type":"code","source":"def tensor_to_image(tensor):\n\n    tensor = tensor*255\n    tensor = np.array(tensor, dtype=np.uint8)\n    if np.ndim(tensor)>3:\n        assert tensor.shape[0] == 1\n        tensor = tensor[0]\n    return PIL.Image.fromarray(tensor)","execution_count":null,"outputs":[]},{"metadata":{"id":"oeXebYusyHwC"},"cell_type":"markdown","source":"### Choose a content image and a style image (1st Photo and 1st Monet):"},{"metadata":{"id":"wqc0OJHwyFAk","trusted":true},"cell_type":"code","source":"content_path_orig = '../input/gan-getting-started/photo_jpg/00068bc07f.jpg'\nstyle_path_orig = '../input/gan-getting-started/monet_jpg/000c1e3bff.jpg'","execution_count":null,"outputs":[]},{"metadata":{"id":"xE4Yt8nArTeR"},"cell_type":"markdown","source":"## Visualize the input"},{"metadata":{"id":"klh6ObK2t_vH"},"cell_type":"markdown","source":"### Define a function to load an image and limit its maximum dimension to 256 pixels."},{"metadata":{"id":"3TLljcwv5qZs","trusted":true},"cell_type":"code","source":"def load_img(path_to_img):\n    max_dim = 256\n    img = tf.io.read_file(path_to_img)\n    img = tf.image.decode_image(img, channels=3)\n    img = tf.image.convert_image_dtype(img, tf.float32)\n\n    shape = tf.cast(tf.shape(img)[:-1], tf.float32)\n    long_dim = max(shape)\n    scale = max_dim / long_dim\n\n    new_shape = tf.cast(shape * scale, tf.int32)\n\n    img = tf.image.resize(img, new_shape)\n    img = img[tf.newaxis, :]\n    return img","execution_count":null,"outputs":[]},{"metadata":{"id":"2yAlRzJZrWM3"},"cell_type":"markdown","source":"### Create a simple function to display an image:"},{"metadata":{"id":"cBX-eNT8PAK_","trusted":true},"cell_type":"code","source":"def imshow(image, title=None):\n    # convert tensor elements to 0 - 255 values\n    image = image * 255\n    \n    # Remove batch dimension of tensor\n    if len(image.shape) > 3:\n        out = np.squeeze(image, axis=0)\n     \n    out = out.astype('uint8')\n    plt.imshow(out)\n    if title:\n        plt.title(title)\n    plt.imshow(out)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,10))\n\ncontent_image_orig = load_img(content_path_orig)\nstyle_image_orig = load_img(style_path_orig)\n\nplt.subplot(1, 2, 1)\nimshow(content_image_orig, 'Content Image')\n\nplt.subplot(1, 2, 2)\nimshow(style_image_orig, 'Style Image')","execution_count":null,"outputs":[]},{"metadata":{"id":"YMzChXSlKTA2"},"cell_type":"markdown","source":"## Fast Style Transfer using TF-Hub\n\n### Let's download the [TensorFlow Hub style transfer model](https://tfhub.dev/google/magenta/arbitrary-image-stylization-v1-256/2) -- just 81.66 MB.\n### There are versions for mobile devices too!"},{"metadata":{"id":"iYSLexgRKSh-","trusted":true},"cell_type":"code","source":"import tensorflow_hub as hub\n# This is version is updated from the original notebook (/2 vs. /1)\n# Results are greatly improved\nhub_module = hub.load('https://tfhub.dev/google/magenta/arbitrary-image-stylization-v1-256/2')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Run the model and display result"},{"metadata":{"trusted":true},"cell_type":"code","source":"stylized_image_orig = hub_module(tf.constant(content_image_orig), tf.constant(style_image_orig))[0]\nplt.figure(figsize=(15,15))\n\nplt.subplot(1, 3, 1)\nimshow(content_image_orig, 'Content Image')\n\nplt.subplot(1, 3, 2)\nimshow(stylized_image_orig, 'Result')\n\nplt.subplot(1, 3, 3)\nimshow(style_image_orig, 'Style Image')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Is it a Monet?\n\nIf Claude Monet had set up his easel in the field of the content image, would he have painted something like the result image?  Although the color palette looks pretty close to the style image, the style itself seems off.  In Monet's painting (which might be *At Val-Saint-Nicholas near Dieppe (Morning)*, 1897 from the Phillips Collection in Washington, DC), the sea, sky, and cliffs are indistinct. Yet, in the result image, the mountains, clouds, and field are fairly detailed. Have we gotten it wrong?\n\n<p><img src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/3/38/Monet_-_Valle_Buona%2C_Near_Bordighera%2C_1884.jpg/640px-Monet_-_Valle_Buona%2C_Near_Bordighera%2C_1884.jpg\" style=\"width:512px\" /></p>\n\nThis is Monet's *Valle Buona Near Bordighera*, 1884 (photo credit: Jerry Ward / Dallas Museum of Art, via [Wikimedia Commons](https://commons.m.wikimedia.org/wiki/File:Monet_-_Valle_Buona,_Near_Bordighera,_1884.jpg), license [Creative Commons Attribution-ShareAlike 4.0 International](https://creativecommons.org/licenses/by-sa/4.0/deed.en)). Actually, the TensorFlow Hub style transfer model predicted very well how Monet might have painted the scene in the content image, but how? Perhaps it's because, the model was trained on arbitrary images which probably included Impressionist paintings, and the Impressionists painted plenty of detailed fields.  So, are we to conclude that content informs style?  Let's flip the script..."},{"metadata":{"trusted":true},"cell_type":"code","source":"style_path_flip = '../input/gan-getting-started/photo_jpg/00068bc07f.jpg'\ncontent_path_flip = '../input/gan-getting-started/monet_jpg/000c1e3bff.jpg'\n\ncontent_image_flip = load_img(content_path_flip)\nstyle_image_flip = load_img(style_path_flip)\n\nstylized_image_flip = hub_module(tf.constant(content_image_flip), tf.constant(style_image_flip))[0]\nplt.figure(figsize=(15,15))\n\nplt.subplot(1, 3, 1)\nimshow(content_image_flip, 'Content Image')\n\nplt.subplot(1, 3, 2)\nimshow(stylized_image_flip, 'Result')\n\nplt.subplot(1, 3, 3)\nimshow(style_image_flip, 'Style Image')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Wow! Now that we have a modern photographer on the cliffs instead of Monet, we get to see the breaking waves and the clouds in the sky. Those weren't in Monet's painting, or were they?"},{"metadata":{},"cell_type":"markdown","source":"# Explore Further\n\nThe code below selects a random photo and a random Monet and displays results for when Monet is the style and for when his work is the content. To learn more on video -- the BBC television series \"Fake or Fortune\" Season 1, Episode 1 investigates a painting that might be by Monet (available on at least one major streaming service in the U.S. and possibly available in other countries too)."},{"metadata":{},"cell_type":"markdown","source":"### Load image filenames"},{"metadata":{"trusted":true},"cell_type":"code","source":"from os import listdir\nfrom os.path import isfile, join\n\n\nphoto_path = '../input/gan-getting-started/photo_jpg/'\nphoto_files = [f for f in listdir(photo_path) if isfile(join(photo_path, f))]\n\nmonet_path = '../input/gan-getting-started/monet_jpg/'\nmonet_files = [f for f in listdir(monet_path) if isfile(join(monet_path, f))]\nmonet_count = len(monet_files)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Select random photo and random Monet, then stylize and display"},{"metadata":{"trusted":true},"cell_type":"code","source":"from random import randrange\n\n\n# Get random photo\ncontent_path1 = photo_path + photo_files[randrange(len(photo_files))]\nstyle_path1 = monet_path + monet_files[randrange(len(monet_files))]\n\ncontent_image1 = load_img(content_path1)\nstyle_image1 = load_img(style_path1)\n\ncontent_image2 = style_image1\nstyle_image2 = content_image1\n\nstylized_image1 = hub_module(tf.constant(content_image1), tf.constant(style_image1))[0]\nstylized_image2 = hub_module(tf.constant(content_image2), tf.constant(style_image2))[0]\n\nplt.figure(figsize=(20,20))\nplt.subplots_adjust(bottom=0.3, top=0.7, hspace=0.25, right=0.8, wspace=0.3)\n\nplt.subplot(2, 3, 1)\nimshow(content_image1, 'Content Image')\n\nplt.subplot(2, 3, 2)\nimshow(stylized_image1, 'Result')\n\nplt.subplot(2, 3, 3)\nimshow(style_image1, 'Style Image')\n\nplt.subplot(2, 3, 4)\nimshow(content_image2, 'Content Image')\n\nplt.subplot(2, 3, 5)\nimshow(stylized_image2, 'Result')\n\nplt.subplot(2, 3, 6)\nimshow(style_image2, 'Style Image')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}