{"cells":[{"metadata":{},"cell_type":"markdown","source":" # Generating Dog Images with a Deep Convolutional Generative Adversarial Network"},{"metadata":{},"cell_type":"markdown","source":"This kernel attempts be an all in one guide with some of my own writing and also links to great references. Contribution to this kernel is iterative as the competition progresses.\n\nLet us start with a non-technical example of what a GAN is\ninspired by this [article](https://skymind.ai/wiki/generative-adversarial-network-gan). \n\nSuppose there is a currency counterfeiter named George and a detective named Derek. George,\nfor some reason has never seen real currency notes but wishes to make counterfeits in \nhopes of becoming rich. He makes what he thinks is a dollar bill and transports it to detective\nDerek through a network of people. Derek looks at real currency that he has in his pocket\nand at the currency he received from George and easily knows this is fake and looks like \nmoney from the Monopoly boardgame. \n\nFor some odd reason Derek also associates these numerical values\nto the real currency and the fake currency. Initially the real currency has high values associated with them and fake one has low. He also chooses to share the value for fake currency with a colleague. This colleague however is working with George and shares this information so George can improve. So George makes some changes and again has the bills transported to Derek. Some value is shared with Derek's colleague which is carried over to George. \n\nThus this cycle continues with a primary goal of George trying to trick Derek and Derek tyring to determine between real and fake. George is the generator, Derek is the discriminator and they are adversaries. \n\nCheck out this [kernel](https://www.kaggle.com/jesucristo/gan-introduction) for a more detailed introduction. "},{"metadata":{},"cell_type":"markdown","source":"# Import Necessary Libraries"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# imports ...\nfrom __future__ import absolute_import, division, print_function, unicode_literals\nimport tensorflow as tf # ml framework\ntf.enable_eager_execution() # operations are executed immediately as they are called from Python\nimport numpy as np # linear algebra\nimport warnings\nfrom PIL import Image # image processing library\nimport IPython.display as display # for inspecting the images by displaying them in the kernel\nimport random # for the random seed used as input for the generator\nfrom numpy.random import choice\nimport matplotlib.pyplot as plt, zipfile # plotting library\nimport xml.etree.ElementTree as ET # xml parser used during pre-processing stage\nimport time # time the execution of codeblocks\nimport xml.dom.minidom # for printing the annotation xml nicely\nimport pydot\nimport graphviz\nfrom IPython.display import FileLink, FileLinks\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os, glob # for managing files/directories\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"tf.__version__","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Variable Declarations"},{"metadata":{"trusted":true},"cell_type":"code","source":"# So it is easier to navigate throughout the notebook whenever I need to change variables\n\n# Filepaths for image loading code\nROOT = '../input/'\n# list of all image file names in all-dogs\nIMAGES = os.listdir(ROOT + 'all-dogs/all-dogs')\n# list of all the annotation directories, each directory is a dog breed\nbreeds = os.listdir(ROOT + 'annotation/Annotation/') \n\n# Variables that determine how tensorflow will create batches after data load\nBUFFER_SIZE = 20000\nBATCH_SIZE = 32\n\n# Weight initializers for the Generator network\nWEIGHT_INIT = tf.keras.initializers.RandomNormal(mean=0.0, stddev=0.2)\n\n# Image dimensions, generator outputs 64x64 size image while the discriminator expects a 64x64\ndim = 64\n\n# Variables needed for the training part\nEPOCHS = 280\nnoise_dim = 100\nnum_examples_to_generate = 16\n\n# We will reuse this seed overtime (so it's easier)\n# to visualize progress in the animated GIF)\nseed = tf.random.normal([num_examples_to_generate, noise_dim])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Get images and pre-process"},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"# Demo: Show how the Annotation files XML is structured\ndom = xml.dom.minidom.parse('../input/annotation/Annotation/n02097658-silky_terrier/n02097658_98') \npretty_xml_as_string = dom.toprettyxml()\nprint(pretty_xml_as_string)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Thanks to Chris Deotte for the following snippet and Paulo Pinto for posting code on retrieving bounding boxes."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Code slightly modified from user: cdeotte | https://www.kaggle.com/cdeotte/supervised-generative-dog-net\n\nidxIn = 0; namesIn = []\nimagesIn = np.zeros((25000,64,64,3))\n\n# CROP WITH BOUNDING BOXES TO GET DOGS ONLY\n# iterate through each directory in annotation\nfor breed in breeds:\n    # iterate through each file in the directory\n    for dog in os.listdir(ROOT+'annotation/Annotation/'+breed):\n        try: img = Image.open(ROOT+'all-dogs/all-dogs/'+dog+'.jpg') \n        except: continue           \n        # Element Tree library allows for parsing xml and getting specific tag values    \n        tree = ET.parse(ROOT+'annotation/Annotation/'+breed+'/'+dog)\n        # take a look at the print out of an xml previously to get what is going on\n        root = tree.getroot() # <annotation>\n        objects = root.findall('object') # <object>\n        for o in objects:\n            bndbox = o.find('bndbox') # <bndbox>\n            xmin = int(bndbox.find('xmin').text) # <xmin>\n            ymin = int(bndbox.find('ymin').text) # <ymin>\n            xmax = int(bndbox.find('xmax').text) # <xmax>\n            ymax = int(bndbox.find('ymax').text) # <ymax>\n            w = np.min((xmax - xmin, ymax - ymin))\n            img2 = img.crop((xmin, ymin, xmin+w, ymin+w))\n            img2 = img2.resize((64,64), Image.ANTIALIAS)\n            imagesIn[idxIn,:,:,:] = np.asarray(img2)\n            namesIn.append(breed)\n            idxIn += 1                ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Inspect what the previous code created\nprint(\"imagesIn is a {} with {} {} by {} rgb({}) images. Shape: {}\".format(type(imagesIn), imagesIn.shape[0], imagesIn.shape[1], imagesIn.shape[2], imagesIn.shape[3], imagesIn.shape))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# DISPLAY CROPPED IMAGES\n\n# list of 25 random numbers in range 0, idxIn\n# this allows for displaying random images in the for loop using x[k*5+j]\nx = np.random.randint(0,idxIn,25)\n\nfor k in range(5):\n    plt.figure(figsize=(15,3))\n    for j in range(5):\n        plt.subplot(1,5,j+1)\n        img = Image.fromarray( imagesIn[x[k*5+j],:,:,:].astype('uint8') )\n        plt.axis('off')\n        plt.title(namesIn[x[k*5+j]].split('-')[1],fontsize=11)\n        plt.imshow(img)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# normalize the pixel values\nimagesIn = (imagesIn[:idxIn,:,:,:]-127.5)/127.5","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# view some images in the after normalization \nplt.figure(figsize=(8,8))\nfor image in range(4):\n    plt.subplot(2,2, image+1)\n    plt.imshow((imagesIn[image]))\n    plt.xlabel('shape: {}'.format(imagesIn[image].shape))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Create Dataset**"},{"metadata":{"trusted":true},"cell_type":"code","source":"# this is needed because the gradient functions from TF require float32 instead of float64\nimagesIn = tf.cast(imagesIn, 'float32')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The batch sizes will be a \"hyperparameter\" to experiment with for better performance."},{"metadata":{"trusted":true},"cell_type":"code","source":"ds = tf.data.Dataset.from_tensor_slices(imagesIn).shuffle(BUFFER_SIZE).batch(BATCH_SIZE)\nprint(ds)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Generator Model"},{"metadata":{},"cell_type":"markdown","source":"Similar to this image is what the Generator network is doing\n![](https://cdn-images-1.medium.com/max/1043/1*KvMnRfb76DponICrHIbSdg.png)\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# function will return a generator model\ndef make_generator():\n    model = tf.keras.Sequential(\n        [\n            # first layer with 32,768 nodes expecting an input of vector size 100 (random noise)\n            tf.keras.layers.Dense(8*8*512, use_bias=False, input_shape=(100,)),\n            # Normalize the activations of the previous layer at each batch\n            tf.keras.layers.BatchNormalization(),\n            # apply leaky relu activation: f(x) = {x if x > 0 : 0.01*x}\n            tf.keras.layers.LeakyReLU(),\n            # reshape input to (8,8,512)\n            tf.keras.layers.Reshape((8, 8, 512)),\n            \n            # second layer Conv2DTranspose so it is doing the opposite of a convolutional layer\n            tf.keras.layers.Conv2DTranspose(256, (5,5), strides=(2,2), padding='same', use_bias=False,\\\n                kernel_initializer=WEIGHT_INIT),\n            tf.keras.layers.BatchNormalization(),\n            tf.keras.layers.LeakyReLU(),\n            tf.keras.layers.Dropout(0.3),\n            \n            tf.keras.layers.Conv2DTranspose(128, (5,5), strides=(2,2), padding='same', use_bias=False,\\\n                kernel_initializer=WEIGHT_INIT),\n            tf.keras.layers.BatchNormalization(),\n            tf.keras.layers.LeakyReLU(),\n            tf.keras.layers.Dropout(0.3),\n            \n            tf.keras.layers.Conv2DTranspose(64, (5,5), strides=(2,2), padding='same', use_bias=False,\\\n                kernel_initializer=WEIGHT_INIT),\n            tf.keras.layers.BatchNormalization(),\n            tf.keras.layers.LeakyReLU(),\n            \n            tf.keras.layers.Dense(3,activation='tanh', use_bias=False,\\\n                kernel_initializer=WEIGHT_INIT)\n        ]\n    )\n    return model\n# create an instance of the generator model defined\ngenerator = make_generator()\nprint(generator.summary())\ntf.keras.utils.plot_model(\n    generator,\n    to_file='/tmp/gen_mdl.png',\n    show_shapes=True,\n    show_layer_names=True,\n    rankdir='TB',\n)\n# random noise vector\nnoise = tf.random.normal([1,100])\n# run the generator model with the noise vector as input\ngenerated_image = generator(noise, training=False)\n# display output\nplt.imshow(generated_image[0, :, :, :])\nprint(generated_image.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Want to see nice image of the generator network architecture\ndisplay.Image(filename='/tmp/gen_mdl.png')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"FileLinks('.')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Discriminator model"},{"metadata":{},"cell_type":"markdown","source":"The discriminator is a regular CNN which will be used for real dog pictures from training data and fake ones generated by the generator. I am initializing weights in the first layer with standard deviation = WEIGHT_INIT_STDDEV which will be changing while optimizing for better performance. "},{"metadata":{"trusted":true},"cell_type":"code","source":"def make_discriminator():\n    model = tf.keras.Sequential(\n        [\n            tf.keras.layers.Conv2D(64, (4,4), strides=(2,2), padding='same', input_shape=[dim,dim,3],\\\n                kernel_initializer=WEIGHT_INIT),\n            tf.keras.layers.BatchNormalization(),\n            tf.keras.layers.LeakyReLU(),\n#             tf.keras.layers.Dropout(0.25),\n            \n            tf.keras.layers.Conv2D(128, (4,4), strides=(2,2), padding='same',\\\n                kernel_initializer=WEIGHT_INIT),\n            tf.keras.layers.BatchNormalization(),\n            tf.keras.layers.LeakyReLU(),\n#             tf.keras.layers.Dropout(0.25),\n            \n            tf.keras.layers.Conv2D(256, (4,4), strides=(2,2), padding='same',\\\n                kernel_initializer=WEIGHT_INIT),\n            tf.keras.layers.BatchNormalization(),\n            tf.keras.layers.LeakyReLU(),\n#             tf.keras.layers.Dropout(0.25),\n            \n            # flatten input into 1-D and output a single a number from the last layer using sigmoid activation\n            tf.keras.layers.Flatten(),\n            tf.keras.layers.Dense(1, activation='sigmoid')\n        ]\n    )\n    return model\n\ndiscriminator = make_discriminator()\ntf.keras.utils.plot_model(\n    generator,\n    to_file='/tmp/dis_mdl.png',\n    show_shapes=True,\n    show_layer_names=True,\n    rankdir='TB',\n)\nprint(discriminator.summary())\ndecision = discriminator(generated_image)\nprint (decision)\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# discriminator model architecture picture\ndisplay.Image(filename='/tmp/dis_mdl.png')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Loss and optimizers"},{"metadata":{},"cell_type":"markdown","source":"Information on [AdamOptimizer](https://machinelearningmastery.com/adam-optimization-algorithm-for-deep-learning/) and for [Binary Cross Entropy](https://towardsdatascience.com/understanding-binary-cross-entropy-log-loss-a-visual-explanation-a3ac6025181a). \n\nBCE:\n![](https://cdn-images-1.medium.com/max/960/1*rdBw0E-My8Gu3f_BOB6GMA.png)\nWhere y is the label (1 for yes and 0 for no). Great article explaining visually and the math behind it, check it out."},{"metadata":{},"cell_type":"markdown","source":"The discriminator loss function needs to compute loss for both real images and fake images then output the sum of both as the total loss. Generator loss is only the cross entropy of fake images. \n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Label smoothing -- technique from GAN hacks, instead of assigning 1/0 as class labels, we assign a random integer in range [0.7, 1.0] for positive class\n# and [0.0, 0.3] for negative class\n\ndef smooth_positive_labels(y):\n    return y - 0.3 + (np.random.random(y.shape) * 0.3)\n\ndef smooth_negative_labels(y):\n\treturn y + np.random.random(y.shape) * 0.5\n\n# Recomended to introduce some noise to the labels, so out of 1000 real labels, approximately 50 should be flipped to 0 (5%)\n# randomly flip some labels\ndef noisy_labels(y, p_flip):\n\t# determine the number of labels to flip\n\tn_select = int(p_flip * y.shape[0].value)\n\t# choose labels to flip\n\tflip_ix = choice([i for i in range(y.shape[0].value)], size=n_select)\n\t# invert the labels in place\n\ty[flip_ix] = 1 - y[flip_ix]\n\treturn y","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# This method returns a helper function to compute cross entropy loss\n# code from tf dcgan tutorial\ncross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits=False)\n\n# The Discriminator loss function\n\ndef discriminator_loss(real_output, fake_output):\n    # tf.ones_like changes all values in the tensor to 1s\n    # similarly tf.zeros_like changes all values in the tensor to 0\n    # then apply label smoothing\n    real_output_smooth = smooth_positive_labels(tf.ones_like(real_output))\n#     real_output_noisy = noisy_labels(real_output_smooth, 0.05)\n    fake_output_smooth = smooth_negative_labels(tf.zeros_like(fake_output))\n#     fake_output_noisy = noisy_labels(fake_output_smooth, 0.05)\n    real_loss = cross_entropy(real_output_smooth, real_output)\n    fake_loss = cross_entropy(fake_output_smooth, fake_output)\n    total_loss = real_loss + fake_loss\n    return total_loss\n\n# The Generator loss function\n\ndef generator_loss(fake_output):\n    fake_output_smooth = smooth_negative_labels(tf.ones_like(fake_output))\n#     fake_output_noisy = noisy_labels(fake_output_smooth, 0.05)\n    return cross_entropy(fake_output_smooth, fake_output)\n\n# optimizers -- Adam\ngenerator_optimizer = tf.train.AdamOptimizer(learning_rate=0.0002, beta1=0.5)\ndiscriminator_optimizer = tf.train.AdamOptimizer(learning_rate=0.0002, beta1=0.5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training loop"},{"metadata":{},"cell_type":"markdown","source":"The train step will be called iteratively in the train function. It creates noise vectors, the number of vectors is BATCH_SIZE and length is noise_dim which I will keep at 100. \n\nNext there is GradientTape() which is a tensorflow [function](https://www.tensorflow.org/tutorials/eager/automatic_differentiation) for automatic differentiation (AD). \n\nWhat is AD?\n- computer has primitive operations available (e.g. addition, multiplication, logarithm)\n- so every complicated function can be written as a composition of these primitive functions \n- each primitive function has a simple derivative\n- AD are a set of techniques using this logic of simple derivatives of composed functions \nRead this [article](http://www.columbia.edu/~ahd2125/post/2015/12/5/)\n\nTensorflow \"records\" all the operations executed inside the context of a tf.GradientTape onto to a \"tape\". Using that \"tape\" and the gradients associated with recorded operation, tensorflow computes the gradients of a \"recorded\" computation using reverse mode differentiaion. "},{"metadata":{"trusted":true},"cell_type":"code","source":"# code from tf dcgan tutorial\ndef train_step(images, G_loss_list, D_loss_list):\n    noise = tf.random.normal([BATCH_SIZE, noise_dim])\n    \n    with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:\n        # the following are the operations recorded onto the \"tape\"\n        generated_images = generator(noise, training=True)\n        \n        real_output = discriminator(images, training=True)\n        fake_output = discriminator(generated_images, training=True)\n        gen_loss = generator_loss(fake_output)\n        disc_loss = discriminator_loss(real_output, fake_output)\n        \n    G_loss_list.append(gen_loss.numpy())\n    D_loss_list.append(disc_loss.numpy())\n    # the following lines are taking the derivatives and applying gradients using Adam optimizer\n    gradients_of_generator = gen_tape.gradient(gen_loss, generator.trainable_variables)\n    gradients_of_discriminator = disc_tape.gradient(disc_loss, discriminator.trainable_variables)\n    \n    generator_optimizer.apply_gradients(zip(gradients_of_generator, generator.trainable_variables))\n    discriminator_optimizer.apply_gradients(zip(gradients_of_discriminator, discriminator.trainable_variables))\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def train(dataset, epochs):\n    G_loss = []\n    D_loss = []\n    for epoch in range(epochs):\n        start = time.time()\n        for image_batch in dataset:\n            train_step(image_batch, G_loss, D_loss)\n            \n        plot_loss(G_loss, D_loss, epoch)\n        G_loss = []\n        D_loss = []\n        if (epoch % 10 == 0):\n            display.clear_output(wait=True)\n            generate_and_save_images(generator,\n                                 epoch + 1,\n                                 seed)\n        print ('Time for epoch {} is {} sec'.format(epoch + 1, time.time()-start))\n        \n            \n    # Generate after the final epoch\n    print(\"Final Epoch\")\n    generate_and_save_images(generator,\n                           epochs,\n                           seed)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import time\ndef generate_and_save_images(model, epoch, test_input):\n    # Notice `training` is set to False.\n    # This is so all layers run in inference mode (batchnorm).\n    predictions = model(test_input, training=False)    \n    fig = plt.figure(figsize=(8,8))\n    for i in range(predictions.shape[0]):\n        plt.subplot(4, 4, i+1)\n        plt.imshow((predictions[i, :, :, :]+1.)/2.)\n        plt.axis('off')\n#     plt.savefig('image_at_epoch_{}.png'.format(epoch+1))\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# function by Nanashi\n\ndef plot_loss (G_losses, D_losses, epoch):\n    plt.figure(figsize=(10,5))\n    plt.title(\"Generator and Discriminator Loss - EPOCH {}\".format(epoch+1))\n    plt.plot(G_losses,label=\"G\")\n    plt.plot(D_losses,label=\"D\")\n    plt.xlabel(\"iterations\")\n    plt.ylabel(\"Loss\")\n    plt.legend()\n    plt.show()\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nprint('Starting training')\ntrain(ds, EPOCHS)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Save 10000 generated dog images"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# SAVE TO ZIP FILE NAMED IMAGES.ZIP\nz = zipfile.PyZipFile('images.zip', mode='w')\n\nfilename = 'gen_model.h5'\ntf.keras.models.save_model(\n    generator,\n    filename,\n    overwrite=True,\n    include_optimizer=True,\n    save_format=None\n)\n\nfor k in range(10000):\n    generated_image = generator(tf.random.normal([1, noise_dim]), training=False)\n    f = str(k)+'.png'\n    img = ((generated_image[0,:,:,:]+1.)/2.).numpy()\n    tf.keras.preprocessing.image.save_img(\n        f,\n        img,\n        scale=True\n    )\n    z.write(f); os.remove(f)\nz.close()\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"FileLinks('.')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}