{"cells":[{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5"},"cell_type":"markdown","source":"Imagine if we had access to the true data distribution $P_{data}(x)$ we could sample from that distribution in order to generate new samples, however there is no direct way to do this as typically this distribution is complex and high-dimensional. What if we could instead sample from a random noise (e.g. Normal distribution) and then learn to transform that to $P_{data}(x)$. Neural networks are a prime candidate to capture functions with high complexity and we can use to to capture this transformation. This is exactly what the do. They train the transformer network or Generator along with another network, called the Discriminator, in a game theoretic way. Going back to our image generation example:\n\nThe Generator network ($G$), tries to fool the discriminator in thinking that the generated images are real,meaning that they are taken from $P_{data}$, and\nThe Discriminator network ($D$), tries to differentiate between real ($x\\sim P_{data}$) and fake images.\n\nRandom noise is fed into the Generator that transforms it into a \"fake image\". The Discriminator is fed both from the training set images ($p_{data}(x)$) and the fake images coming from the Generator and it has to tell them apart. The idea behind GAN, is to train both of these networks alternatively to do the best they can in generating and discriminating images. The intuition is that by improving one of these networks, in this game theoretic manner, the other network has to do a better job to win the game, and that in turn improves its performance and this loop continues."},{"metadata":{},"cell_type":"markdown","source":"<img src=\"https://4.bp.blogspot.com/-YTGLQjhch-Q/Wz475NU2TSI/AAAAAAAAsu4/zaC_wfZBX80dePLflgQUaAaxE72od3VCgCEwYBhgL/s1600/Figura_2.png\" height=\"500\" width=\"500\">"},{"metadata":{},"cell_type":"markdown","source":"<pre><b>Credits to Chris Deotte's Kernel</b>\nhttps://www.kaggle.com/cdeotte/dog-memorizer-gan</pre>\n\n<pre><b>Credits to Robin Smits Kernel</b>\nhttps://www.kaggle.com/rsmits/keras-dcgan-with-weight-normalization</pre>"},{"metadata":{},"cell_type":"markdown","source":"# Import Libraries"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import numpy as np\nimport os\nimport shutil\nimport matplotlib.pyplot as plt\nimport zipfile\nimport tensorflow as tf\nimport xml.etree.ElementTree as ET\nfrom tqdm import tqdm\nfrom keras.models import *\nfrom keras.layers.core import *\nfrom keras.layers import *\nfrom keras.layers.advanced_activations import LeakyReLU\nfrom keras.initializers import RandomNormal\nfrom keras.optimizers import Adam\nfrom keras import backend as K\nfrom PIL import Image","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Constants and Directories"},{"metadata":{"trusted":true},"cell_type":"code","source":"NOISE_SIZE = 64\nLR_D = 0.0001\nLR_G = 0.0002\n#BATCH_SIZE = 128 # 128 is for final\nBATCH_SIZE = 64 # pre-final test only\nEPOCHS = 650 # For better results increase this value \nBETA1 = 0.5\nSEED = 4250\nrandom_dim = 128\nimg_rows = 64\nimg_cols = 64\nchannels = 3\nSEED = 4250\nnp.random.seed(SEED)\nrandom_dim = 128","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Constants and Directories\n\nROOT_DIR = '../input/'\nIMAGES_DIR = ROOT_DIR + 'all-dogs/all-dogs/'\nBREEDS_DIR = ROOT_DIR + 'annotation/Annotation/'\n\n# File Lists\nIMAGES = os.listdir(IMAGES_DIR)\nBREEDS = os.listdir(BREEDS_DIR) \n\n# Summary\nprint('Total Images: {}'.format(len(IMAGES)))\nprint('Total Annotations: {}'.format(len(BREEDS)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load and Process Images"},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_images():\n    # Place holder for output \n    all_images = np.zeros((22250, 64, 64, 3))\n    \n    # Index\n    index = 0\n    \n    for breed in BREEDS:\n        for dog in os.listdir(BREEDS_DIR + breed):\n            try: img = Image.open(IMAGES_DIR + dog + '.jpg') \n            except: continue  \n                \n            tree = ET.parse(BREEDS_DIR + breed + '/' + dog)\n            root = tree.getroot()\n            objects = root.findall('object')\n            for o in objects:\n                bndbox = o.find('bndbox') \n                xmin = int(bndbox.find('xmin').text)\n                ymin = int(bndbox.find('ymin').text)\n                xmax = int(bndbox.find('xmax').text)\n                ymax = int(bndbox.find('ymax').text)\n                \n                # Determine each side\n                xdelta = xmax - xmin\n                ydelta = ymax - ymin\n                \n                # Take the mean of the sides\n                #w = int((xdelta + ydelta) / 2)\n                \n                # Filter out images where bounding box is below 64 pixels.\n                # This filters out a couple of 100 images but prevents using low resolution images.\n                if xdelta >= 64 and ydelta >= 64:\n                    img2 = img.crop((xmin, ymin, xmax, ymax))\n                    img2 = img2.resize((64, 64), Image.ANTIALIAS)\n                    image = np.asarray(img2)\n                    \n                    #    # Normalize to range[-1, 1]\n                    all_images[index,:] = (image.astype(np.float32) - 127.5)/127.5\n                    \n                    index += 1\n        \n                # Plot Status\n                if index % 1000 == 0:\n                    print('Processed Images: {}'.format(index))\n\n    print('Total Processed Images: {}'.format(index))\n\n    return all_images","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Weight Normalization\n\nUsing the class AdamWithWeightnorm from [github repository](https://github.com/krasserm/weightnorm/tree/master/keras_2)."},{"metadata":{"trusted":true},"cell_type":"code","source":"# adapted from keras.optimizers.Adam\nclass AdamWithWeightnorm(Adam):\n    def get_updates(self, loss, params):\n        grads = self.get_gradients(loss, params)\n        self.updates = [K.update_add(self.iterations, 1)]\n\n        lr = self.lr\n        if self.initial_decay > 0:\n            lr *= (1. / (1. + self.decay * K.cast(self.iterations, K.floatx())))\n\n        t = K.cast(self.iterations + 1, K.floatx())\n        lr_t = lr * K.sqrt(1. - K.pow(self.beta_2, t)) / (1. - K.pow(self.beta_1, t))\n\n        shapes = [K.get_variable_shape(p) for p in params]\n        ms = [K.zeros(shape) for shape in shapes]\n        vs = [K.zeros(shape) for shape in shapes]\n        self.weights = [self.iterations] + ms + vs\n\n        for p, g, m, v in zip(params, grads, ms, vs):\n\n            # if a weight tensor (len > 1) use weight normalized parameterization\n            # this is the only part changed w.r.t. keras.optimizers.Adam\n            ps = K.get_variable_shape(p)\n            if len(ps)>1:\n\n                # get weight normalization parameters\n                V, V_norm, V_scaler, g_param, grad_g, grad_V = get_weightnorm_params_and_grads(p, g)\n\n                # Adam containers for the 'g' parameter\n                V_scaler_shape = K.get_variable_shape(V_scaler)\n                m_g = K.zeros(V_scaler_shape)\n                v_g = K.zeros(V_scaler_shape)\n\n                # update g parameters\n                m_g_t = (self.beta_1 * m_g) + (1. - self.beta_1) * grad_g\n                v_g_t = (self.beta_2 * v_g) + (1. - self.beta_2) * K.square(grad_g)\n                new_g_param = g_param - lr_t * m_g_t / (K.sqrt(v_g_t) + self.epsilon)\n                self.updates.append(K.update(m_g, m_g_t))\n                self.updates.append(K.update(v_g, v_g_t))\n\n                # update V parameters\n                m_t = (self.beta_1 * m) + (1. - self.beta_1) * grad_V\n                v_t = (self.beta_2 * v) + (1. - self.beta_2) * K.square(grad_V)\n                new_V_param = V - lr_t * m_t / (K.sqrt(v_t) + self.epsilon)\n                self.updates.append(K.update(m, m_t))\n                self.updates.append(K.update(v, v_t))\n\n                # if there are constraints we apply them to V, not W\n                if getattr(p, 'constraint', None) is not None:\n                    new_V_param = p.constraint(new_V_param)\n\n                # wn param updates --> W updates\n                add_weightnorm_param_updates(self.updates, new_V_param, new_g_param, p, V_scaler)\n\n            else: # do optimization normally\n                m_t = (self.beta_1 * m) + (1. - self.beta_1) * g\n                v_t = (self.beta_2 * v) + (1. - self.beta_2) * K.square(g)\n                p_t = p - lr_t * m_t / (K.sqrt(v_t) + self.epsilon)\n\n                self.updates.append(K.update(m, m_t))\n                self.updates.append(K.update(v, v_t))\n\n                new_p = p_t\n                # apply constraints\n                if getattr(p, 'constraint', None) is not None:\n                    new_p = p.constraint(new_p)\n                self.updates.append(K.update(p, new_p))\n        return self.updates\n\ndef get_weightnorm_params_and_grads(p, g):\n    ps = K.get_variable_shape(p)\n\n    # construct weight scaler: V_scaler = g/||V||\n    V_scaler_shape = (ps[-1],)  # assumes we're using tensorflow!\n    V_scaler = K.ones(V_scaler_shape)  # init to ones, so effective parameters don't change\n\n    # get V parameters = ||V||/g * W\n    norm_axes = [i for i in range(len(ps) - 1)]\n    V = p / tf.reshape(V_scaler, [1] * len(norm_axes) + [-1])\n\n    # split V_scaler into ||V|| and g parameters\n    V_norm = tf.sqrt(tf.reduce_sum(tf.square(V), norm_axes))\n    g_param = V_scaler * V_norm\n\n    # get grad in V,g parameters\n    grad_g = tf.reduce_sum(g * V, norm_axes) / V_norm\n    grad_V = tf.reshape(V_scaler, [1] * len(norm_axes) + [-1]) * \\\n             (g - tf.reshape(grad_g / V_norm, [1] * len(norm_axes) + [-1]) * V)\n\n    return V, V_norm, V_scaler, g_param, grad_g, grad_V\n\ndef add_weightnorm_param_updates(updates, new_V_param, new_g_param, W, V_scaler):\n    ps = K.get_variable_shape(new_V_param)\n    norm_axes = [i for i in range(len(ps) - 1)]\n\n    # update W and V_scaler\n    new_V_norm = tf.sqrt(tf.reduce_sum(tf.square(new_V_param), norm_axes))\n    new_V_scaler = new_g_param / new_V_norm\n    new_W = tf.reshape(new_V_scaler, [1] * len(norm_axes) + [-1]) * new_V_param\n    updates.append(K.update(W, new_W))\n    updates.append(K.update(V_scaler, new_V_scaler))\n\n# data based initialization for a given Keras model\ndef data_based_init(model, input):\n    # input can be dict, numpy array, or list of numpy arrays\n    if type(input) is dict:\n        feed_dict = input\n    elif type(input) is list:\n        feed_dict = {tf_inp: np_inp for tf_inp,np_inp in zip(model.inputs,input)}\n    else:\n        feed_dict = {model.inputs[0]: input}\n\n    # add learning phase if required\n    if model.uses_learning_phase and K.learning_phase() not in feed_dict:\n        feed_dict.update({K.learning_phase(): 1})\n\n    # get all layer name, output, weight, bias tuples\n    layer_output_weight_bias = []\n    for l in model.layers:\n        trainable_weights = l.trainable_weights\n        if len(trainable_weights) == 2:\n            W,b = trainable_weights\n            assert(l.built)\n            layer_output_weight_bias.append((l.name,l.get_output_at(0),W,b)) # if more than one node, only use the first\n\n    # iterate over our list and do data dependent init\n    sess = K.get_session()\n    for l,o,W,b in layer_output_weight_bias:\n        print('Performing data dependent initialization for layer ' + l)\n        m,v = tf.nn.moments(o, [i for i in range(len(o.get_shape())-1)])\n        s = tf.sqrt(v + 1e-10)\n        updates = tf.group(W.assign(W/tf.reshape(s,[1]*(len(W.get_shape())-1)+[-1])), b.assign((b-m)/s))\n        sess.run(updates, feed_dict)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Generator"},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_generator_model():\n    # Random Normal Weight Initialization\n    init = RandomNormal(mean = 0.0, stddev = 0.02)\n\n    # Model\n    model = Sequential()\n\n    # Start at 4 * 4\n    start_shape = NOISE_SIZE * 4 * 4\n    model.add(Dense(start_shape, kernel_initializer = init, input_dim = random_dim))\n    model.add(Reshape((4, 4, 64)))\n    \n    # Upsample => 8 * 8 \n    model.add(UpSampling2D())\n    model.add(Conv2D(128, kernel_size = 3, padding = \"same\", kernel_initializer = init))\n    model.add(ReLU())\n    \n    # Upsample => 16 * 16 \n    model.add(UpSampling2D())\n    model.add(Conv2D(128, kernel_size = 3, padding = \"same\", kernel_initializer = init))\n    model.add(ReLU())\n    \n    # Upsample => 32 * 32\n    model.add(UpSampling2D())\n    model.add(Conv2D(128, kernel_size = 3, padding = \"same\", kernel_initializer = init))\n    model.add(ReLU())\n    \n    # Upsample => 64 * 64\n    model.add(UpSampling2D())\n    model.add(Conv2D(128, kernel_size = 3, padding = \"same\", kernel_initializer = init))\n    model.add(ReLU())\n    \n    # output\n    model.add(Conv2D(3, kernel_size = 3, activation = 'tanh', padding = 'same', kernel_initializer=init))\n    model.compile(loss = 'binary_crossentropy', optimizer = AdamWithWeightnorm(lr = LR_G, beta_1 = BETA1))\n    print(model.summary())\n\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Discriminator"},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_discriminator_model():\n    input_shape = (img_rows, img_cols, channels)\n\n    # Random Normal Weight Initialization\n    init = RandomNormal(mean = 0.0, stddev = 0.02)\n\n    # Define Model\n    model = Sequential()\n\n    # Downsample ==> 32 * 32\n    model.add(Conv2D(128, kernel_size = 3, strides = 2, padding = 'same', kernel_initializer = init, input_shape = input_shape))\n    model.add(LeakyReLU(0.2))\n    model.add(Dropout(0.25))\n\n    # Downsample ==> 16 * 16\n    model.add(Conv2D(128, kernel_size = 3, strides = 2, padding = 'same', kernel_initializer = init))\n    model.add(LeakyReLU(0.2))\n    model.add(Dropout(0.25))\n    \n    # Downsample => 8 * 8\n    model.add(Conv2D(128, kernel_size = 3, strides = 2, padding = 'same', kernel_initializer = init))\n    model.add(LeakyReLU(0.2))\n    model.add(Dropout(0.25))\n    \n    # Downsample => 4 * 4\n    model.add(Conv2D(128, kernel_size = 3, strides = 2, padding = 'same', kernel_initializer = init))\n    model.add(LeakyReLU(0.2))\n    model.add(Dropout(0.25))\n    \n    # Final Layers\n    model.add(Flatten())\n    model.add(Dense(1, activation = 'sigmoid', kernel_initializer = init))\n\n    # Compile model\n    model.compile(loss = 'binary_crossentropy', optimizer = AdamWithWeightnorm(lr = LR_D , beta_1 = BETA1))\n    \n    print(model.summary())\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# GAN Model\nNext the code to create the GAN model."},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_gan_model(discriminator, random_dim, generator):\n    # Set trainable to False initially\n    discriminator.trainable = False\n    \n    # Gan Input\n    gan_input = Input(shape = (random_dim,))\n    \n    # Generator Output...an image\n    generator_output = generator(gan_input)\n    \n    # Output of the discriminator is the probability of an image being real or fake\n    gan_output = discriminator(generator_output)\n    gan_model = Model(inputs = gan_input, outputs = gan_output)\n    gan_model.compile(loss = 'binary_crossentropy', optimizer = AdamWithWeightnorm(lr = LR_G, beta_1 = BETA1))\n    print(gan_model.summary())\n    \n    return gan_model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Noise function"},{"metadata":{"trusted":true},"cell_type":"code","source":"def generator_input(latent_dim, n_samples):\n    # Generate points in latent space\n    input = np.random.randn(latent_dim * n_samples)\n\n    # Reshape to input batch for the network\n    input = input.reshape((n_samples, latent_dim))\n\n    return input","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Plot functions\nNext we define some functions to plot the images to give an impression of how the training progressed and one to plot the loss during training."},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_generated_images(epoch, generator, examples = 25, dim = (5, 5)):\n    generated_images = generator.predict(np.random.normal(0, 1, size = [examples, random_dim]))\n    generated_images = ((generated_images + 1) * 127.5).astype('uint8')\n        \n    plt.figure(figsize = (12, 8))\n    for i in range(generated_images.shape[0]):\n        plt.subplot(dim[0], dim[1], i + 1)\n        plt.imshow(generated_images[i], interpolation = 'nearest')\n        plt.axis('off')\n    plt.suptitle('Epoch %d' % epoch, x = 0.5, y = 1.0)\n    plt.tight_layout()\n    plt.savefig('dog_at_epoch_%d.png' % epoch)\n    \ndef plot_loss(d_f, d_r, g):\n    plt.figure(figsize = (18, 12))\n    plt.plot(d_f, label = 'Discriminator Fake Loss')\n    plt.plot(d_r, label = 'Discriminator Real Loss')\n    plt.plot(g, label = 'Generator Loss')\n    plt.legend()\n    plt.savefig('loss_plot.png')\n    plt.close()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_model(epochs = 1, batch_size = 128):\n    # Get the Dog images\n    x_train = load_images()\n    \n    # Calculate amount of batches\n    batch_count = x_train.shape[0] / batch_size\n\n    # Create Generator and Discriminator Models\n    generator = create_generator_model()\n    discriminator = create_discriminator_model()\n    \n    # Create GAN Model\n    gan_model = create_gan_model(discriminator, random_dim, generator)\n    \n    # Lists for Loss History\n    discriminator_fake_hist, discriminator_real_hist, generator_hist = [], [], []\n    \n    for e in range(epochs):\n        \n        # Script Stop Counter\n        script_stopper_counter = 0\n        \n        print('======================== Epoch {} ============================='.format(e))\n        for _ in tqdm(range(int(batch_count))):\n            \n            # Discriminator Loss\n            discriminator_fake_loss, discriminator_real_loss = [], []\n            \n            # Train the Discriminator more than the Generator\n            for _ in range(2):\n                # Train discriminator on Fake Images\n                X_fake = generator.predict(generator_input(random_dim, batch_size))\n                y_fake = np.zeros(batch_size)\n                y_fake[:] = 0\n                discriminator.trainable = True\n                d_fake_loss = discriminator.train_on_batch(X_fake, y_fake)\n                \n                # Train discriminator on Real Images\n                X_real = x_train[np.random.randint(0, x_train.shape[0], size = batch_size)]\n                y_real = np.zeros(batch_size)\n                y_real[:] = 0.9  # label smoothing\n                discriminator.trainable = True\n                d_real_loss = discriminator.train_on_batch(X_real, y_real)\n\n                # Store Loss each iteration\n                discriminator_fake_loss.append(d_fake_loss)\n                discriminator_real_loss.append(d_real_loss)\n\n            # Train generator\n            noise = generator_input(random_dim, batch_size)\n            y_gen = np.ones(batch_size)\n            discriminator.trainable = False\n            generator_loss = gan_model.train_on_batch(noise, y_gen)\n\n            # Summarize Batch Loss\n            # Uncomment Lines below if you want per batch update Loss statistics\n            #print('\\nd_fake_loss = %.4f, d_real_loss = %.4f g_loss = %.4f' % \\\n            #      (np.mean(discriminator_fake_loss), np.mean(discriminator_real_loss), generator_loss))\n\n            # Store Loss in Loss History lists\n            discriminator_fake_hist.append(np.mean(discriminator_fake_loss))\n            discriminator_real_hist.append(np.mean(discriminator_real_loss)) \n            generator_hist.append(generator_loss)\n            \n            # Stop script preliminary Counter\n            # Occasionally the Discriminator Fake Loss explodes and remains high...in that case we stop the script\n            if np.mean(discriminator_fake_loss) > 10:\n                script_stopper_counter += 1\n        \n        # Summarize Image Quality for epochs during training\n        if e % 100 == 0:\n            plot_generated_images(e, generator)\n            \n        # Stop Script? If almost 1 epoch with exploded Loss...then Yes.\n        if script_stopper_counter > 160:\n            plot_generated_images(e, generator)\n            break\n            \n    # Plot Loss during Training\n    plot_loss(discriminator_fake_hist, discriminator_real_hist, generator_hist)\n\n    # Create Images.zip\n    z = zipfile.PyZipFile('images.zip', mode = 'w')\n    for k in range(10000):\n        # Generate new dogs\n        generated_images = generator.predict(np.random.normal(0, 1, size = [1, random_dim]))\n        image = Image.fromarray(((generated_images + 1) * 127.5).astype('uint8').reshape(64, 64, 3))\n\n        # Save to zip file  \n        f = str(k)+'.png'\n        image.save(f, 'PNG')\n        z.write(f)\n        os.remove(f)\n        \n        # Plot Status Counter\n        if k % 1000 == 0: \n            print(k)\n    z.close()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_model(EPOCHS, BATCH_SIZE)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"}},"nbformat":4,"nbformat_minor":1}