{"cells":[{"metadata":{},"cell_type":"markdown","source":"Started on 8 July 2019\n\n**References:**\n1. https://machinelearningmastery.com/how-to-develop-a-generative-adversarial-network-for-a-cifar-10-small-object-photographs-from-scratch/\n2. https://www.kaggle.com/whizzkid/crop-images-using-bounding-box\n3. https://www.kaggle.com/guillaumedesforges/loading-the-cropped-dogs-seamlessly-with-pytorch\n4. https://www.kaggle.com/guillaumedesforges/usable-complete-data-loading-utility\n5. https://www.kaggle.com/cdeotte/supervised-generative-dog-net\n\n*Thank you, authors.*"},{"metadata":{},"cell_type":"markdown","source":"# Introduction"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":false,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":false},"cell_type":"code","source":"import numpy as np\nfrom numpy import random\nfrom numpy import vstack\nimport pandas as pd\nimport os\nprint(os.listdir(\"../input\"))\nimport matplotlib.pyplot as plt\nfrom PIL import Image\nimport xml.etree.ElementTree as ET\nimport zipfile","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"from keras.models import Sequential\nfrom keras.layers import Dense\nfrom keras.layers import Reshape\nfrom keras.layers import Flatten\nfrom keras.layers import Conv2D\nfrom keras.layers import Conv2DTranspose\nfrom keras.layers import LeakyReLU\nfrom keras.layers import Dropout\nfrom keras.layers import BatchNormalization\nfrom keras.initializers import RandomNormal\nfrom keras.optimizers import Adam\nfrom keras.optimizers import RMSprop\nfrom keras.utils.vis_utils import plot_model\nfrom keras.models import load_model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load & prepare dataset"},{"metadata":{"trusted":false},"cell_type":"code","source":"import glob\nimage = glob.glob('../input/all-dogs/all-dogs/*')\nbreed = glob.glob('../input/annotation/Annotation/*')\nannot = glob.glob('../input/annotation/Annotation/*/*')\nprint(len(image), len(breed), len(annot))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def get_bbox(annot):\n    \"\"\"\n    This extracts and returns values of bounding boxes\n    \"\"\"\n    xml = annot\n    tree = ET.parse(xml)\n    root = tree.getroot()\n    objects = root.findall('object')\n    bbox = []\n    for o in objects:\n        bndbox = o.find('bndbox')\n        xmin = int(bndbox.find('xmin').text)\n        ymin = int(bndbox.find('ymin').text)\n        xmax = int(bndbox.find('xmax').text)\n        ymax = int(bndbox.find('ymax').text)\n        bbox.append((xmin,ymin,xmax,ymax))\n    return bbox","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def get_image(annot):\n    \"\"\"\n    Retrieve the corresponding image given annotation file\n    \"\"\"\n    img_path = '../input/all-dogs/all-dogs/'\n    file = annot.split('/')\n    img_filename = img_path+file[-1]+'.jpg'\n    return img_filename","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# initialize tensor for dog images\nn_x = 64\nn_c = 3\ndogs = np.zeros((len(image), n_x, n_x, n_c))\nprint(dogs.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"for a in range(len(image)):\n    bbox = get_bbox(annot[a])\n    dog = get_image(annot[a])\n    if dog == '../input/all-dogs/all-dogs/n02105855_2933.jpg':   # this jpg is not in the dataset\n        continue\n    im = Image.open(dog)\n    im = im.crop(bbox[0])\n    im = im.resize((64,64), Image.ANTIALIAS)\n    dogs[a,:,:,:] = np.asarray(im) / 255.","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# pick some images randomly from dogs and look at these\nplt.figure(figsize=(15,8))\nn_images = 60\nselect = random.randint(low=0,high=dogs.shape[0],size=n_images)\nfor i, index in enumerate(select):  \n    plt.subplot(6, 10, i+1)\n    plt.imshow(dogs[index])\n    plt.axis('off')\nplt.subplots_adjust(wspace=0.3, hspace=-0.1)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Best Practice:** To scale the pixel values from the range of [0,1] to the normalized range of [-1,1]. The generator model will generate images with pixel values in the range [-1,1] as it will use the tanh activation function, a best practice. Hence, the real images are to be scaled to the same range."},{"metadata":{"trusted":false},"cell_type":"code","source":"# scale from [0,1] to [-1,1]\ndogs = (dogs*2)-1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# The Discriminator"},{"metadata":{},"cell_type":"markdown","source":"The discriminator takes a sample \"image\" from our dataset and says whether it is real or fake.\n\n**Inputs:** \"image\" 64x64 pixels in size; three channels (RGB).\n\n**Outputs:** binary classification, likelihood the sample is real."},{"metadata":{},"cell_type":"markdown","source":"### Define the Discriminator"},{"metadata":{"trusted":false},"cell_type":"code","source":"def define_discriminator(in_shape=(n_x,n_x,n_c)):\n    \"\"\"\n    Define the conv net for the discriminator\n    \"\"\"\n    init = RandomNormal(stddev=0.02)\n    model = Sequential()\n    model.add(Conv2D(64,kernel_size=3,padding='same',input_shape=in_shape,kernel_initializer=init))\n    model.add(LeakyReLU(alpha=0.2))\n    model.add(Conv2D(128,kernel_size=4,strides=2,padding='same',kernel_initializer=init))\n    model.add(LeakyReLU(alpha=0.2))\n    model.add(Conv2D(128,kernel_size=4,strides=2,padding='same',kernel_initializer=init))\n    model.add(LeakyReLU(alpha=0.2))\n    model.add(Conv2D(256,kernel_size=4,strides=2,padding='same',kernel_initializer=init))\n    model.add(LeakyReLU(alpha=0.2))\n    model.add(Flatten())\n    model.add(Dropout(rate=0.4))\n    model.add(Dense(1,activation='sigmoid'))\n    opt = Adam(lr=0.0002,beta_1=0.5) # define optimizer\n    model.compile(loss='binary_crossentropy',optimizer=opt,metrics=['accuracy'])\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"discriminator = define_discriminator()\ndiscriminator.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Training of the Discriminator"},{"metadata":{},"cell_type":"markdown","source":"All image samples will be labelled '1' (real). Need to create fake samples labelled as '0'. The fake samples will be created by the Generator. The real and fake samples will be fed into the Discriminator by batches."},{"metadata":{"trusted":false},"cell_type":"code","source":"def generate_real_samples(data, n_samples):\n    \"\"\"\n    Pick 'n_samples' randomly from 'data'\n    \"\"\"\n    idx = random.randint(low=0,high=data.shape[0],size=n_samples)\n    X = data[idx]\n    Y = np.ones((n_samples,1))\n    return X, Y","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# The Generator"},{"metadata":{},"cell_type":"markdown","source":"The Generator creates new, fake but plausible images. It works by taking a point from a latent space as input and output an image.\n\n**Inputs:** Point in latent space, e.g. a 100-element vector of Gaussian random numbers.\n\n**Outputs:** 2D color image (3 channels; RGB) of 64x64 pixels with pixel values in [-1,1]."},{"metadata":{"trusted":false},"cell_type":"code","source":"def define_generator(latent_dim):\n    \"\"\"\n    Define the conv net for the generator\n    \"\"\"\n    init = RandomNormal(stddev=0.02)\n    n_nodes = 128*8*8\n    model = Sequential()\n    model.add(Dense(n_nodes,input_dim=latent_dim,kernel_initializer=init)) # foundation for 8*8 image\n    model.add(LeakyReLU(alpha=0.2))\n    model.add(Reshape((8,8,128)))\n    model.add(Conv2DTranspose(128,kernel_size=4,strides=2,padding='same',kernel_initializer=init)) # up-sample to 16*16 image\n    model.add(LeakyReLU(alpha=0.2))\n    model.add(Conv2DTranspose(128,kernel_size=4,strides=2,padding='same',kernel_initializer=init)) # up-sample to 32*32 image\n    model.add(LeakyReLU(alpha=0.2))\n    model.add(Conv2DTranspose(128,kernel_size=4,strides=2,padding='same',kernel_initializer=init)) # up-sample to 64*64 image\n    model.add(LeakyReLU(alpha=0.2))\n    model.add(Conv2D(3,kernel_size=3,activation='tanh',padding='same',kernel_initializer=init))\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"latent_dim = 200 # define size of latent space\ngenerator = define_generator(latent_dim)\ngenerator.summary()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def generate_latent_points(latent_dim, n_samples):\n    \"\"\"\n    This generates points in the latent space as input for the generator\n    \"\"\"\n    x_input = random.randn(latent_dim * n_samples)    # sample from normal/gaussian distribution (best practice)\n    x_input = x_input.reshape(n_samples, latent_dim)     # reshape into a batch of inputs for the network\n    return x_input","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def generate_fake_samples(g_model, latent_dim, n_samples):\n    \"\"\"\n    Generate 'n_samples' of fake samples from the generator\n    \"\"\"\n    X_input = generate_latent_points(latent_dim, n_samples)\n    X = g_model.predict(X_input)    # generator predicts output\n    Y = np.zeros((n_samples,1))     # create class labels '0' for fake sample\n    return X, Y","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Combining the Discriminator & Generator as a GAN"},{"metadata":{"trusted":false},"cell_type":"code","source":"def define_gan(g_model, d_model):\n    \"\"\"\n    This takes as arguments the generator and discriminator and creates the GAN subsuming these two models. \n    The weights in the discriminator are marked as not trainable, \n    which only affects the weights as seen by the GAN and not the standalone discriminator model.\n    \"\"\"\n    d_model.trainable = False     # make weights in the discriminator not trainable\n    model = Sequential()\n    model.add(g_model)\n    model.add(d_model)\n    opt = Adam(lr=0.0002,beta_1=0.5)\n    model.compile(loss='binary_crossentropy', optimizer=opt)\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"gan = define_gan(generator, discriminator)\ngan.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Functions to evaluate performance of GAN"},{"metadata":{"trusted":false},"cell_type":"code","source":"def save_plot(examples, epoch, n=10):\n    \"\"\"\n    This creates and save a plot of generated images\n    \"\"\"\n    examples = (examples+1)/2.0\n    for i in range(n * n):\n        plt.subplot(n, n, 1 + i)\n        plt.axis('off')\n        plt.imshow(examples[i])\n    plt.title('Epoch %03d' % (epoch+1))\n    filename = 'generated_plot_e%03d.png' % (epoch+1)\n    plt.savefig(filename)\n    plt.close()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# evaluate the discriminator, plot generated images, save generator model\ndef summarize_performance(epoch, g_model, d_model, data, latent_dim, n_samples=100):\n    \"\"\"\n    This evaluates the discriminator, plot generated images, save generator model\n    \"\"\"\n    X_real, Y_real = generate_real_samples(data, n_samples)\n    _, acc_real = d_model.evaluate(X_real, Y_real, verbose=0)   # evaluate discriminator on real samples\n    X_fake, Y_fake = generate_fake_samples(g_model, latent_dim, n_samples)\n    _, acc_fake = d_model.evaluate(X_fake, Y_fake, verbose=0)   # evaluate discriminator on fake samples\n    print('>Accuracy real: %.0f%%, fake: %.0f%%' % (acc_real*100, acc_fake*100))    # summarize discriminator performance\n    save_plot(X_fake, epoch)\n    filename = 'generator_model_%03d.h5' % (epoch + 1)\n    g_model.save(filename)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train the GAN"},{"metadata":{"_kg_hide-output":false,"trusted":false},"cell_type":"code","source":"def train(g_model, d_model, gan_model, data, latent_dim, n_epochs=300, batch_size=128):\n    \"\"\"\n    This trains the combined generator and discriminator models in the GAN\n    \"\"\"\n    batch_per_epoch = data.shape[0] // batch_size\n    half_batch = batch_size // 2\n    for i in range(n_epochs):\n        for j in range(batch_per_epoch):\n            X_real, Y_real = generate_real_samples(data, half_batch)   # randomly select real samples\n            d_loss1, _ = d_model.train_on_batch(X_real, Y_real)   # update discriminator model weights\n            X_fake, Y_fake = generate_fake_samples(g_model, latent_dim, half_batch)   # generate fake samples\n            d_loss2, _ = d_model.train_on_batch(X_fake, Y_fake)   # update discriminator model weights\n            X_gan = generate_latent_points(latent_dim, batch_size)   # as input for generator\n            Y_gan = np.ones((batch_size, 1))\n            g_loss = gan_model.train_on_batch(X_gan, Y_gan)   # update generator via the discriminator's error\n            #print('>%d, %d/%d, d1=%.3f, d2=%.3f, g=%.3f' % (i+1, j+1, batch_per_epoch, d_loss1, d_loss2, g_loss)) # summarize loss for batch\n        # evaluate the model performance, sometimes\n        if (i+1) % 25 == 0: \n            summarize_performance(i, g_model, d_model, data, latent_dim)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":false},"cell_type":"code","source":"latent_dim = 200\nd_model = define_discriminator()\ng_model = define_generator(latent_dim)\ngan_model = define_gan(g_model, d_model)\ndata = dogs\ntrain(g_model, d_model, gan_model, data, latent_dim)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"image = Image.open('generated_plot_e300.png')\nplt.figure(figsize=(15,15))\nplt.axis('off')\nplt.imshow(image)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Using the Generator model to generate new dog images"},{"metadata":{},"cell_type":"markdown","source":"The generation of each image requires a point in the latent space as input."},{"metadata":{"trusted":false},"cell_type":"code","source":"def show_plot(examples, n):\n    \"\"\"\n    This shows the plots from the GAN\n    \"\"\"\n    plt.figure(figsize=(10,10))\n    for i in range(n * n):\n        plt.subplot(n, n, 1 + i)\n        plt.axis('off')\n        plt.imshow(examples[i,:,:])\n    plt.subplots_adjust(wspace=0.3, hspace=-0.1)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"latent_points = generate_latent_points(200, 10000)\nnew_images = g_model.predict(latent_points)\nnew_images = (new_images+1)/2.0\nshow_plot(new_images, 8)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"new_images.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"test = Image.fromarray((255*new_images[8383]).astype('uint8').reshape((64,64,3)))\nplt.imshow(test)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Create zip file to store the images"},{"metadata":{"trusted":false},"cell_type":"code","source":"z = zipfile.PyZipFile('images.zip', mode='w')\nfor d in range(10000):\n    dog_image = Image.fromarray((255*new_images[d]).astype('uint8').reshape((64,64,3)))\n    f = str(d)+'.png'\n    dog_image.save(f,'PNG')\n    z.write(f)\n    os.remove(f)\nz.close()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"}},"nbformat":4,"nbformat_minor":1}