{"cells":[{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5"},"cell_type":"markdown","source":"In this notebook I will create and run a Keras DCGAN. With some experimenting I've created a model with a simple architecture that still provides in a nice LB score. I thought it would be a nice one to share with you and allow you to further experiment with it.\n\nI won't be doing a whole lecture on how a GAN is working and why it is working. There are a whole bunch of kernels available in this competition which give a very nice explanation about GAN's.\n\nLet start with importing all the needed libraries."},{"metadata":{},"cell_type":"markdown","source":"# Import Libraries"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"trusted":false},"cell_type":"code","source":"import numpy as np\nimport os\nimport shutil\nimport matplotlib.pyplot as plt\nimport zipfile\nimport tensorflow as tf\nimport xml.etree.ElementTree as ET\nfrom tqdm import tqdm\nfrom keras.models import Model \nfrom keras.models import Sequential\nfrom keras.layers.core import Dense\nfrom keras.layers.core import Dropout\nfrom keras.layers import Input\nfrom keras.layers import BatchNormalization\nfrom keras.models import Sequential\nfrom keras.layers import Dense\nfrom keras.layers import Reshape\nfrom keras.layers import Flatten\nfrom keras.layers import Conv2D\nfrom keras.layers import Conv2DTranspose\nfrom keras.layers import UpSampling2D\nfrom keras.layers import ReLU\nfrom keras.layers.advanced_activations import LeakyReLU\nfrom keras.initializers import RandomNormal\nfrom keras.optimizers import Adam\nfrom keras import backend as K\nfrom PIL import Image","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Constants and Directories\nNext lets define some constants and directories."},{"metadata":{"trusted":false},"cell_type":"code","source":"# Constants and Directories\nSEED = 4250\nnp.random.seed(SEED)\nrandom_dim = 128\nROOT_DIR = '../input/'\nIMAGES_DIR = ROOT_DIR + 'all-dogs/all-dogs/'\nBREEDS_DIR = ROOT_DIR + 'annotation/Annotation/'\n\n# File Lists\nIMAGES = os.listdir(IMAGES_DIR)\nBREEDS = os.listdir(BREEDS_DIR) \n\n# Summary\nprint('Total Images: {}'.format(len(IMAGES)))\nprint('Total Annotations: {}'.format(len(BREEDS)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load and Process Images\nNext we define a method to load and process all images. From this kernel version and onwards I'am using the dog annotations for the images as provided. The bounding box code is based on earlier kernels from [cdeotte](https://www.kaggle.com/cdeotte/dog-memorizer-gan) and [paulorzp](https://www.kaggle.com/paulorzp/show-annotations-and-breeds).\n\nI made some changes however that improved the score even further by about 10 points (at least it did on a private forked kernel of this notebook ;-) ).\n\nI basically filter out images where one of the bounding box sides is smaller than 64 pixels. \n(UPDATED in this version) I use the entire bounding box and just resize that to a square image. This does give some image distortion..however after a visual inspection of multiple images it does not look that bad. So lets try to use the whole bounding box as input and ignore the distortion. In multiple local runs this gave a slightly higher score. So lets see and wait what a run in a public kernel does. Each image is also normalized to have its values between -1 and 1."},{"metadata":{"trusted":false},"cell_type":"code","source":"def load_images():\n    # Place holder for output \n    all_images = np.zeros((22250, 64, 64, 3))\n    \n    # Index\n    index = 0\n    \n    for breed in BREEDS:\n        for dog in os.listdir(BREEDS_DIR + breed):\n            try: img = Image.open(IMAGES_DIR + dog + '.jpg') \n            except: continue  \n                \n            tree = ET.parse(BREEDS_DIR + breed + '/' + dog)\n            root = tree.getroot()\n            objects = root.findall('object')\n            for o in objects:\n                bndbox = o.find('bndbox') \n                xmin = int(bndbox.find('xmin').text)\n                ymin = int(bndbox.find('ymin').text)\n                xmax = int(bndbox.find('xmax').text)\n                ymax = int(bndbox.find('ymax').text)\n                \n                # Determine each side\n                xdelta = xmax - xmin\n                ydelta = ymax - ymin\n                \n                # Take the mean of the sides\n                #w = int((xdelta + ydelta) / 2)\n                \n                # Filter out images where bounding box is below 64 pixels.\n                # This filters out a couple of 100 images but prevents using low resolution images.\n                if xdelta >= 64 and ydelta >= 64:\n                    img2 = img.crop((xmin, ymin, xmax, ymax))\n                    img2 = img2.resize((64, 64), Image.ANTIALIAS)\n                    image = np.asarray(img2)\n                    \n                    #    # Normalize to range[-1, 1]\n                    all_images[index,:] = (image.astype(np.float32) - 127.5)/127.5\n                    \n                    index += 1\n        \n                # Plot Status\n                if index % 1000 == 0:\n                    print('Processed Images: {}'.format(index))\n\n    print('Total Processed Images: {}'.format(index))\n\n    return all_images","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Weight Normalization\nWhen I started with this competition I followed a lot of the tips and tricks on this [site](https://github.com/soumith/ganhacks). One of the tips is to use BatchNormalization. I however noticed this very nice [paper](https://arxiv.org/pdf/1704.03971.pdf). It discusses the usage and effects of Batch and Weights Normalization in GAN's. After reading it I thought I could give my GAN a try with Weights Normalization.\n\nI haven't been able to find an official implementation of a Keras Weight Normalization layer. I was however able to find the code at this [github repository](https://github.com/krasserm/weightnorm/tree/master/keras_2). From that I will use the class AdamWithWeightnorm."},{"metadata":{"trusted":false},"cell_type":"code","source":"# adapted from keras.optimizers.Adam\nclass AdamWithWeightnorm(Adam):\n    def get_updates(self, loss, params):\n        grads = self.get_gradients(loss, params)\n        self.updates = [K.update_add(self.iterations, 1)]\n\n        lr = self.lr\n        if self.initial_decay > 0:\n            lr *= (1. / (1. + self.decay * K.cast(self.iterations, K.floatx())))\n\n        t = K.cast(self.iterations + 1, K.floatx())\n        lr_t = lr * K.sqrt(1. - K.pow(self.beta_2, t)) / (1. - K.pow(self.beta_1, t))\n\n        shapes = [K.get_variable_shape(p) for p in params]\n        ms = [K.zeros(shape) for shape in shapes]\n        vs = [K.zeros(shape) for shape in shapes]\n        self.weights = [self.iterations] + ms + vs\n\n        for p, g, m, v in zip(params, grads, ms, vs):\n\n            # if a weight tensor (len > 1) use weight normalized parameterization\n            # this is the only part changed w.r.t. keras.optimizers.Adam\n            ps = K.get_variable_shape(p)\n            if len(ps)>1:\n\n                # get weight normalization parameters\n                V, V_norm, V_scaler, g_param, grad_g, grad_V = get_weightnorm_params_and_grads(p, g)\n\n                # Adam containers for the 'g' parameter\n                V_scaler_shape = K.get_variable_shape(V_scaler)\n                m_g = K.zeros(V_scaler_shape)\n                v_g = K.zeros(V_scaler_shape)\n\n                # update g parameters\n                m_g_t = (self.beta_1 * m_g) + (1. - self.beta_1) * grad_g\n                v_g_t = (self.beta_2 * v_g) + (1. - self.beta_2) * K.square(grad_g)\n                new_g_param = g_param - lr_t * m_g_t / (K.sqrt(v_g_t) + self.epsilon)\n                self.updates.append(K.update(m_g, m_g_t))\n                self.updates.append(K.update(v_g, v_g_t))\n\n                # update V parameters\n                m_t = (self.beta_1 * m) + (1. - self.beta_1) * grad_V\n                v_t = (self.beta_2 * v) + (1. - self.beta_2) * K.square(grad_V)\n                new_V_param = V - lr_t * m_t / (K.sqrt(v_t) + self.epsilon)\n                self.updates.append(K.update(m, m_t))\n                self.updates.append(K.update(v, v_t))\n\n                # if there are constraints we apply them to V, not W\n                if getattr(p, 'constraint', None) is not None:\n                    new_V_param = p.constraint(new_V_param)\n\n                # wn param updates --> W updates\n                add_weightnorm_param_updates(self.updates, new_V_param, new_g_param, p, V_scaler)\n\n            else: # do optimization normally\n                m_t = (self.beta_1 * m) + (1. - self.beta_1) * g\n                v_t = (self.beta_2 * v) + (1. - self.beta_2) * K.square(g)\n                p_t = p - lr_t * m_t / (K.sqrt(v_t) + self.epsilon)\n\n                self.updates.append(K.update(m, m_t))\n                self.updates.append(K.update(v, v_t))\n\n                new_p = p_t\n                # apply constraints\n                if getattr(p, 'constraint', None) is not None:\n                    new_p = p.constraint(new_p)\n                self.updates.append(K.update(p, new_p))\n        return self.updates\n\ndef get_weightnorm_params_and_grads(p, g):\n    ps = K.get_variable_shape(p)\n\n    # construct weight scaler: V_scaler = g/||V||\n    V_scaler_shape = (ps[-1],)  # assumes we're using tensorflow!\n    V_scaler = K.ones(V_scaler_shape)  # init to ones, so effective parameters don't change\n\n    # get V parameters = ||V||/g * W\n    norm_axes = [i for i in range(len(ps) - 1)]\n    V = p / tf.reshape(V_scaler, [1] * len(norm_axes) + [-1])\n\n    # split V_scaler into ||V|| and g parameters\n    V_norm = tf.sqrt(tf.reduce_sum(tf.square(V), norm_axes))\n    g_param = V_scaler * V_norm\n\n    # get grad in V,g parameters\n    grad_g = tf.reduce_sum(g * V, norm_axes) / V_norm\n    grad_V = tf.reshape(V_scaler, [1] * len(norm_axes) + [-1]) * \\\n             (g - tf.reshape(grad_g / V_norm, [1] * len(norm_axes) + [-1]) * V)\n\n    return V, V_norm, V_scaler, g_param, grad_g, grad_V\n\ndef add_weightnorm_param_updates(updates, new_V_param, new_g_param, W, V_scaler):\n    ps = K.get_variable_shape(new_V_param)\n    norm_axes = [i for i in range(len(ps) - 1)]\n\n    # update W and V_scaler\n    new_V_norm = tf.sqrt(tf.reduce_sum(tf.square(new_V_param), norm_axes))\n    new_V_scaler = new_g_param / new_V_norm\n    new_W = tf.reshape(new_V_scaler, [1] * len(norm_axes) + [-1]) * new_V_param\n    updates.append(K.update(W, new_W))\n    updates.append(K.update(V_scaler, new_V_scaler))\n\n# data based initialization for a given Keras model\ndef data_based_init(model, input):\n    # input can be dict, numpy array, or list of numpy arrays\n    if type(input) is dict:\n        feed_dict = input\n    elif type(input) is list:\n        feed_dict = {tf_inp: np_inp for tf_inp,np_inp in zip(model.inputs,input)}\n    else:\n        feed_dict = {model.inputs[0]: input}\n\n    # add learning phase if required\n    if model.uses_learning_phase and K.learning_phase() not in feed_dict:\n        feed_dict.update({K.learning_phase(): 1})\n\n    # get all layer name, output, weight, bias tuples\n    layer_output_weight_bias = []\n    for l in model.layers:\n        trainable_weights = l.trainable_weights\n        if len(trainable_weights) == 2:\n            W,b = trainable_weights\n            assert(l.built)\n            layer_output_weight_bias.append((l.name,l.get_output_at(0),W,b)) # if more than one node, only use the first\n\n    # iterate over our list and do data dependent init\n    sess = K.get_session()\n    for l,o,W,b in layer_output_weight_bias:\n        print('Performing data dependent initialization for layer ' + l)\n        m,v = tf.nn.moments(o, [i for i in range(len(o.get_shape())-1)])\n        s = tf.sqrt(v + 1e-10)\n        updates = tf.group(W.assign(W/tf.reshape(s,[1]*(len(W.get_shape())-1)+[-1])), b.assign((b-m)/s))\n        sess.run(updates, feed_dict)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Generator\nNext define the code to create the Generator model. Note the usage of the AdamWithWeightnorm optimizer. I don't use any Batch Normalization layers. I also experienced that using Dropout layers hurts the performance and doesn't improve the creation of nice dog images. "},{"metadata":{"trusted":false},"cell_type":"code","source":"def create_generator_model():\n    # Random Normal Weight Initialization\n    init = RandomNormal(mean = 0.0, stddev = 0.02)\n\n    # Model\n    model = Sequential()\n\n    # Start at 4 * 4\n    start_shape = 64 * 4 * 4\n    model.add(Dense(start_shape, kernel_initializer = init, input_dim = random_dim))\n    model.add(Reshape((4, 4, 64)))\n    \n    # Upsample => 8 * 8 \n    model.add(UpSampling2D())\n    model.add(Conv2D(128, kernel_size = 3, padding = \"same\", kernel_initializer = init))\n    model.add(ReLU())\n    \n    # Upsample => 16 * 16 \n    model.add(UpSampling2D())\n    model.add(Conv2D(128, kernel_size = 3, padding = \"same\", kernel_initializer = init))\n    model.add(ReLU())\n    \n    # Upsample => 32 * 32\n    model.add(UpSampling2D())\n    model.add(Conv2D(128, kernel_size = 3, padding = \"same\", kernel_initializer = init))\n    model.add(ReLU())\n    \n    # Upsample => 64 * 64\n    model.add(UpSampling2D())\n    model.add(Conv2D(128, kernel_size = 3, padding = \"same\", kernel_initializer = init))\n    model.add(ReLU())\n    \n    # output\n    model.add(Conv2D(3, kernel_size = 3, activation = 'tanh', padding = 'same', kernel_initializer=init))\n    model.compile(loss = 'binary_crossentropy', optimizer = AdamWithWeightnorm(lr = 0.0002, beta_1 = 0.5))\n    print(model.summary())\n\n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Discriminator\nNext the code to define the Discriminator model. Again note that I don't use BatchNormalization layers as we use the Weight Normalization.\n\nI do use Dropout Layers here as it improves the performance...but only a Dropout of about 25%."},{"metadata":{"trusted":false},"cell_type":"code","source":"def create_discriminator_model():\n    input_shape = (64, 64, 3)\n\n    # Random Normal Weight Initialization\n    init = RandomNormal(mean = 0.0, stddev = 0.02)\n\n    # Define Model\n    model = Sequential()\n\n    # Downsample ==> 32 * 32\n    model.add(Conv2D(128, kernel_size = 3, strides = 2, padding = 'same', kernel_initializer = init, input_shape = input_shape))\n    model.add(LeakyReLU(0.2))\n    model.add(Dropout(0.25))\n\n    # Downsample ==> 16 * 16\n    model.add(Conv2D(128, kernel_size = 3, strides = 2, padding = 'same', kernel_initializer = init))\n    model.add(LeakyReLU(0.2))\n    model.add(Dropout(0.25))\n    \n    # Downsample => 8 * 8\n    model.add(Conv2D(128, kernel_size = 3, strides = 2, padding = 'same', kernel_initializer = init))\n    model.add(LeakyReLU(0.2))\n    model.add(Dropout(0.25))\n    \n    # Downsample => 4 * 4\n    model.add(Conv2D(128, kernel_size = 3, strides = 2, padding = 'same', kernel_initializer = init))\n    model.add(LeakyReLU(0.2))\n    model.add(Dropout(0.25))\n    \n    # Final Layers\n    model.add(Flatten())\n    model.add(Dense(1, activation = 'sigmoid', kernel_initializer = init))\n\n    # Compile model\n    model.compile(loss = 'binary_crossentropy', optimizer = AdamWithWeightnorm(lr = 0.0002, beta_1 = 0.5))\n    \n    print(model.summary())\n    \n    return model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# GAN Model\nNext the code to create the GAN model."},{"metadata":{"trusted":false},"cell_type":"code","source":"def create_gan_model(discriminator, random_dim, generator):\n    # Set trainable to False initially\n    discriminator.trainable = False\n    \n    # Gan Input\n    gan_input = Input(shape = (random_dim,))\n    \n    # Generator Output...an image\n    generator_output = generator(gan_input)\n    \n    # Output of the discriminator is the probability of an image being real or fake\n    gan_output = discriminator(generator_output)\n    gan_model = Model(inputs = gan_input, outputs = gan_output)\n    gan_model.compile(loss = 'binary_crossentropy', optimizer = AdamWithWeightnorm(lr = 0.0002, beta_1 = 0.5))\n    print(gan_model.summary())\n    \n    return gan_model","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Noise function\nNext we define a method to generate the input for the generator."},{"metadata":{"trusted":false},"cell_type":"code","source":"def generator_input(latent_dim, n_samples):\n    # Generate points in latent space\n    input = np.random.randn(latent_dim * n_samples)\n\n    # Reshape to input batch for the network\n    input = input.reshape((n_samples, latent_dim))\n\n    return input","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Plot functions\nNext we define some functions to plot the images to give an impression of how the training progressed and one to plot the loss during training."},{"metadata":{"trusted":false},"cell_type":"code","source":"def plot_generated_images(epoch, generator, examples = 25, dim = (5, 5)):\n    generated_images = generator.predict(np.random.normal(0, 1, size = [examples, random_dim]))\n    generated_images = ((generated_images + 1) * 127.5).astype('uint8')\n        \n    plt.figure(figsize = (12, 8))\n    for i in range(generated_images.shape[0]):\n        plt.subplot(dim[0], dim[1], i + 1)\n        plt.imshow(generated_images[i], interpolation = 'nearest')\n        plt.axis('off')\n    plt.suptitle('Epoch %d' % epoch, x = 0.5, y = 1.0)\n    plt.tight_layout()\n    plt.savefig('dog_at_epoch_%d.png' % epoch)\n    \ndef plot_loss(d_f, d_r, g):\n    plt.figure(figsize = (18, 12))\n    plt.plot(d_f, label = 'Discriminator Fake Loss')\n    plt.plot(d_r, label = 'Discriminator Real Loss')\n    plt.plot(g, label = 'Generator Loss')\n    plt.legend()\n    plt.savefig('loss_plot.png')\n    plt.close()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train Model\nNext we define the method to train the model and generate the submission 'images.zip'. Note that for each batch update we train the generator once on a batch and the discriminator 4 times on a batch (2 * fake and 2 * real images.)"},{"metadata":{"trusted":false},"cell_type":"code","source":"def train_model(epochs = 1, batch_size = 128):\n    # Get the Dog images\n    x_train = load_images()\n    \n    # Calculate amount of batches\n    batch_count = x_train.shape[0] / batch_size\n\n    # Create Generator and Discriminator Models\n    generator = create_generator_model()\n    discriminator = create_discriminator_model()\n    \n    # Create GAN Model\n    gan_model = create_gan_model(discriminator, random_dim, generator)\n    \n    # Lists for Loss History\n    discriminator_fake_hist, discriminator_real_hist, generator_hist = [], [], []\n    \n    for e in range(epochs):\n        \n        # Script Stop Counter\n        script_stopper_counter = 0\n        \n        print('======================== Epoch {} ============================='.format(e))\n        for _ in tqdm(range(int(batch_count))):\n            \n            # Discriminator Loss\n            discriminator_fake_loss, discriminator_real_loss = [], []\n            \n            # Train the Discriminator more than the Generator\n            for _ in range(2):\n                # Train discriminator on Fake Images\n                X_fake = generator.predict(generator_input(random_dim, batch_size))\n                y_fake = np.zeros(batch_size)\n                y_fake[:] = 0\n                discriminator.trainable = True\n                d_fake_loss = discriminator.train_on_batch(X_fake, y_fake)\n                \n                # Train discriminator on Real Images\n                X_real = x_train[np.random.randint(0, x_train.shape[0], size = batch_size)]\n                y_real = np.zeros(batch_size)\n                y_real[:] = 0.9  # label smoothing\n                discriminator.trainable = True\n                d_real_loss = discriminator.train_on_batch(X_real, y_real)\n\n                # Store Loss each iteration\n                discriminator_fake_loss.append(d_fake_loss)\n                discriminator_real_loss.append(d_real_loss)\n\n            # Train generator\n            noise = generator_input(random_dim, batch_size)\n            y_gen = np.ones(batch_size)\n            discriminator.trainable = False\n            generator_loss = gan_model.train_on_batch(noise, y_gen)\n\n            # Summarize Batch Loss\n            # Uncomment Lines below if you want per batch update Loss statistics\n            #print('\\nd_fake_loss = %.4f, d_real_loss = %.4f g_loss = %.4f' % \\\n            #      (np.mean(discriminator_fake_loss), np.mean(discriminator_real_loss), generator_loss))\n\n            # Store Loss in Loss History lists\n            discriminator_fake_hist.append(np.mean(discriminator_fake_loss))\n            discriminator_real_hist.append(np.mean(discriminator_real_loss)) \n            generator_hist.append(generator_loss)\n            \n            # Stop script preliminary Counter\n            # Occasionally the Discriminator Fake Loss explodes and remains high...in that case we stop the script\n            if np.mean(discriminator_fake_loss) > 10:\n                script_stopper_counter += 1\n        \n        # Summarize Image Quality for epochs during training\n        if e % 100 == 0:\n            plot_generated_images(e, generator)\n            \n        # Stop Script? If almost 1 epoch with exploded Loss...then Yes.\n        if script_stopper_counter > 160:\n            plot_generated_images(e, generator)\n            break\n            \n    # Plot Loss during Training\n    plot_loss(discriminator_fake_hist, discriminator_real_hist, generator_hist)\n\n    # Create Images.zip\n    z = zipfile.PyZipFile('images.zip', mode = 'w')\n    for k in range(10000):\n        # Generate new dogs\n        generated_images = generator.predict(np.random.normal(0, 1, size = [1, random_dim]))\n        image = Image.fromarray(((generated_images + 1) * 127.5).astype('uint8').reshape(64, 64, 3))\n\n        # Save to zip file  \n        f = str(k)+'.png'\n        image.save(f, 'PNG')\n        z.write(f)\n        os.remove(f)\n        \n        # Plot Status Counter\n        if k % 1000 == 0: \n            print(k)\n    z.close()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And finally we trigger the function to start the training for a number of epochs."},{"metadata":{"trusted":false},"cell_type":"code","source":"train_model(676, 128)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I hope you enjoyed this notebook and that it provides some more insights in how to create a DCGAN with Keras.\n\nLet me know if you have questions or feedback."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"}},"nbformat":4,"nbformat_minor":1}