{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Table of Contenes\n* [Introduction](#Introduction)\n* [An Introduction to GANs](#An-Introduction-to-GANs)\n* [The Neural Networks in GANs](#The-Neural-Networks-in-GANs)\n* [The Discriminator in GAN](#The-Discriminator-in-GAN)\n * [Training the Discriminator](#Training-the-Discriminator)\n* [The Generator in GAN](#The-Generator-in-GAN)\n * [Training the Generator](#Training-the-Generator)\n* [The Loss Function for Training of GANs](#The-Loss-Function-for-Training-of-GANs)\n * [Loss Function in GANs](#Loss-Function-in-GANs)\n* [Generating MNIST Images using Vanilla GAN After Training for 200 Epochs](#Generating-MNIST-Images-using-Vanilla-GAN-After-Training-for-200-Epochs)\n* [CIFAR10 Images Generated by Deep Convolutional GAN After 25 Epochs](#CIFAR10-Images-Generated-by-Deep-Convolutional-GAN-After-25-Epochs)","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"## Introduction\n* This notebook is specifically aimed for those who are totally new to the concept of GANs. It notebook will not contain much code perhaps. \n* Or it may contain some code but not fully related to the dataset in this competition. \n* Again, it is mainly aimed to help beginner in GANs to get some familiarity, ideas, and intuituins behind GANs.\n\n### I hope that this notebook helps you and you like it.\n\n\n[Figure 1 Image Source](https://9gag.com/gag/aerM7wO)\n\n### So, let's get started","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"## An Introduction to GANs\nGANs belong to the family of generative models in deep learning. And under the hood, they are an unsupervised learning technique. This is because, while training GANs, we do not provided the labels or targets to the neural network model. As GANs are generative models, they try to create new data instances that resemble training data.\n\nThe most common application of GANs is with image data and nowadays they can create very reaslistic images. Just take a look at the image above. All those faces are create by a GAN.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## The Neural Networks in GANs\nGANs contail two neural network in their architecture. One is the **generator** and the other one is the **discriminator**.\n\nWhile training a GAN, the generator tries to generate new fake data from a given noisy sample space. And with each iteration, it tries to generate more and more realistic data.\n\nAt the same time, the discriminator tries to differentiate the real data from the generated fake data of the generator. While training the discriminator, we provide it with both, the real data (positive examples) and the generated data (negative examples). A time will come when the discriminator won’t be able to tell whether the data is real or fake (generated by the generator). This is the time we stop the training and can successfully use the generator to generate realistic data that resembles the input data.\n\n**The following image shows the overall architecture of a Generative Adversarial Network.**\n\n**Figure 2**\n\n## The Discriminator in GAN\nThe discriminator network in a GAN tries to tell apart the fake data that is generated by the generator from the real data. This acts a binary classifier that tries to classify between positive examples and negative examples. For a simple case, we can say that the positive examples are labeled as 1 and the negative examples are labeled as 0.\n\n**The following image shows the dicriminator network in a GAN.**\n\n**Figure 3**\n\nThe above image also shows the flow of data and backpropagation step when we train a GAN.\n### Training the Discriminator\nWhen we think of image data for GANs, then the images generated by the generator are the fake data for the discriminator and the input images are the real data for the discriminator.\n\n**One thing to note here: *When we train the discriminator, then the generator does not train. At that time, the generator weghts are frozen.*** As you can see in the above figure, while training the discriminator, we backpropagate using only the discriminator loss values. Also, the ***updated gradients from the discriminator are not passed down to the generator.***\n\n**Steps to train a discriminator**:\n* First, we give the real data and fake generator data as input to the discriminator.\n* The discriminator classifies between fake data and real data. From this classification, we get the discriminator loss as we see in figure 3.\n* If the discriminator misclassifies the data, then it is penalized. Misclassification can happen in two ways. Either when the discriminator classifies the real data as fake or when it classifies the fake data as real.\n* In the final step, the weights get updated through backpropagation. Now, here is an important point to remember. The backpropagation in discriminator happens through the discriminator network. We will get to the importance of this point in the next section.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## The Generator in GAN\nThe generator neural network tries to generate new data from the random noise (latent vector). With each iteration, it tries to produce more and more realistic data. These generated data also act as negative examples for the discriminator training. Also, the fake data generation process is linked to the feedback that the generator gets from the discriminator. As training progresses, the generator will eventually be able to fool the discriminator into classifying the fake data as real data.\n\n\n**Figure 4**\n\nFigure 4 shows the training of the generator in a Generative Adversarial Network.\n\n### Training the Generator\nLet's go over the steps to train a generator.\n* First, we have the random input or the latent vector or the noise vector.\n* This random input is given to the generator and it generates the fake data.\n* Then the discriminator network comes into play. At this step, the discriminator tries to classify the generated data as fake.\n* We have the outputs from the discriminator which acts as a feedback to the generator. The discriminator feedback is used to penalize the generator for producing bad data instances.\n* Also, we have the generator loss from the generator outputs. This generator loss penalizes the generator itself if it is not able to fool the discriminator.\n\nIf you take a look at **figure 4** you can see that after we get the generator loss, the backpropagation happens both through the discriminator and the generator. We know that while training the generator and updating the weights we need to backpropagate through the generator. Here, the important question is **why do we need to backpropagate the discriminator also?**\n\nAlthough the discriminator parameters are frozen, **still we need feedback about the performance of the generator. This we get by backpropagating through the discriminator also. The discriminator parameters do not get updated in this step but we get feedback about the generator.**\n\nThis is one of the more important steps in GAN training. This is also the reason why GAN training is difficult. This means that it is difficult for a GAN to converge to an optimal solution. We will get into those details as well later in the article.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## The Loss Function for Training of GANs\n\n**Important Note:** *Do not confuse this loss function for GANs as the evaluation criteria for the competition. They are very different from each other*.\n\n*Before moving further remember that Wwile the discriminator trains, the generator does not. Conversely, when the generator trains, the discriminator does not.*\n\n### Loss Function in GANs\nThe minimax loss that is described in the [original paper](https://arxiv.org/abs/1406.2661) on GANs by Goodfellow et al.\n\nThe following is the complete loss function.\n\n$$\n\\mathcal{L}(\\theta^{(G)}, \\theta^{(D)}) = \\mathbb{E}_{x}logD(x) + \\mathbb{E}_{z}(1 – D(G(z)))\n$$\n\nThe above loss function is a variation of the very common **binary cross entropy loss**. The following are the explnation of the various terms in the equation.\n* $Ex$ is the expected value over all the data instances. You may recognize this as taking a sample from all the real images.\n* $Ez$ is the expected value over the latent vector or the random noise vector that we give as input to the generator.\n* $D(x)$ corresponds to the probability by the discriminator that the given data x is real.\n* $G(z)$ is the data generated by the generator when we give it z as the input.\n* $D(G(z))$ is the probability by the discriminator that the data generated by the generator $(G(z))$ is real.\n\nIn GAN training, optimizing the above loss function is a two-sum game for the generator and the discriminator. It is called minimax because the discriminator tries to maximize the $logD(x)$ term. While at the same time, the generator tries to minimize the $log(1−D(G(z)))$ term.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### For the final part, let's take a look at some results to GANs.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Generating MNIST Images using Vanilla GAN After Training for 200 Epochs\n\n**Figure 5**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## CIFAR10 Images Generated by Deep Convolutional GAN After 25 Epochs\n\n**Figure 6**","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### I hope that now you understand the basic concepts of GANs. Although this notebook did not cover the things related to this compeition, still I hope that this provides you with adequate information to search and move forward in the competition.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}