{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction to Generative Adversarial Networks (GANs)"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"import numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\ndef show_img(path, title, figsize=(12,6)):\n    plt.figure(figsize=figsize)\n    img = mpimg.imread(path)\n    imgplot = plt.imshow(img)\n    plt.axis('off')\n    plt.title(title)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In this notebook, we will be familiar with an advanced deep learning technique named Generative Adversarial Networks (GANs). This notebook contains background, application, concept, algorithm, architecture, evaluation, and a code demo of typical GANs. We only focus on two typical GANs: original <a href='http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf'>GANs</a> and Deep Convolutional GAN (<a href='https://arxiv.org/pdf/1511.06434.pdf%C3'>DCGAN</a>)."},{"metadata":{},"cell_type":"markdown","source":"Generative Adversarial Networks (GANs) is a cutting-edge technique of deep neural networks, which was first come up by Ian Goodfellow in 2014. In 2016, Yann LeCun (Facebook AI research director) described GAN as\n> “the most interesting idea in the last 10 years in Machine Learning.”\n\nGAN is a very new stuff and has a promising future. Especially in the recent two years, GAN was developed with an exponential increment (Fig.1). Although it is an infant technique, there are bunch of models proposed with the suffix “GAN”, such as ACGAN, DCGAN, WGAN, BEGAN, CycleGAN, StackGAN. There is a website called [“The GAN Zoo”](https://github.com/hindupuravinash/the-gan-zoo), where includes hundreds variants of GAN."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"show_img('../input/introductiongan/introduction-gan/cumulative_gans.jpg', 'Fig.1 The GAN papers counts by The GAN Zoo')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Application\n### Image Generation\nMost of deep learning techniques we learned previously are used for recognition. GAN technique is going to let the machine learn to generate new stuff. To note, GAN does not simply memorize the given dataset. Generation is always harder than recognition. For example, we first learned how to recognize digits like 0-9, then we tried to mimic the shape of digits and created digits in our styles, which is called generation.\nAs discussed in Chapter 20 of *Deep Learning*, there are many other generative models, such as Restricted Boltzmann Machine (RBM), Generative Stochastic Networks (GSNs), and Variational auto-encoder (VAE). Comparing with these generative models, GAN has the state-of-the-art performance in the filed of image generation. In other words, GAN is the most powerful image generative model.\n<center></center>"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"show_img('../input/introductiongan/introduction-gan/began_face.jpg', 'Fig.2 Generated facial images by BEGAN', (8,4))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Image-to-Image Translation\nGAN can learn the features of two image collections and translate one images from one domain to another. CycleGAN is one representative unsupervised GAN with such functionality. To note, CycleGAN does not require these two datasets to be paired. For example, in Summer<->Winter translation, we need to collect photos of the same scene in different seasons. However, paired datasets are very expensive and often not available. Without paired datasets, CycleGAN can translate landscape photos into a particular painting style, such as Monet, Van Gogh. Besides, it can also translate zebras into horses."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"show_img('../input/introductiongan/introduction-gan/cyclegan.jpg', 'Image-to-Image translation by CycleGAN')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Text-to-Image Translation \nIf we embed text information as the label of corresponding images, GAN will learn the mapping between sentiment vectors and image features. StackGAN is a conditional GAN which can generate images based on text description."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"show_img('../input/introductiongan/introduction-gan/stackgan.jpg', 'Text-to-Image translation by StackGAN')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Image Manipulation\nThere are many other applications in image manipulation. e.g. Super resolution: recovering the photo-realistic texture for the low resolution image. Photo inpainting: filling the missing area in a given image."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"show_img('../input/introductiongan/introduction-gan/srgan.png', 'Super resolution by SRGAN')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"show_img('../input/introductiongan/introduction-gan/inpainting.png', 'Photo inpainting with GAN')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Section Summary\nThe underlying functionality of GAN is transforming data distribution from domain X to domain Y . If we define domain X as a normal (noise) distribution, GAN is going to generate images. If domain X is another image distribution, e.g. different style images, lower resolution images or masked images, GAN is going to transform input images X -> target images Y . If domain X is a text data distribution, GAN is going to generate images based on the input text query. After we learn the GAN technique, you are highly encouraged to propose some creative ideas on further applications."},{"metadata":{},"cell_type":"markdown","source":"# Concept\n### What is a GAN?\nIt is like a zero-sum game in Game Theory (Example 1). i.e. One becomes better means the opponent must be worse. In a typical GAN, the “Criminal” is named as Generator G while the “Investigator” is Discriminator D. In this game, G is trying to generate real-like images to fool D while D is trying to figure all fake images out. These two models compete with each other and eventually reach to a Nash equilibrium, where both G and D cannot get better or worse any more.\n\n---\n**Example 1**. How to be a master of producing fake dollars?  \n*Criminal*: Produced 1st version fake dollars and the *investigator* cannot figure them out.  \n*Investigator*: Found the new fake dollars and successfully figured all 1st version fake dollars out.  \n*Criminal*: Produced 2nd version fake dollars and the *investigator* cannot figure them out.  \n*Investigator*: Found the new fake dollars and successfully figured all 2nd version fake dollars out.  \n...  \nn-th version fake dollars  \n...  \n⇒ *Criminal*: An expert in making fake dollars.  \n⇒ *Investigator*: An expert in figuring fake dollars.  \n\n---"},{"metadata":{},"cell_type":"markdown","source":"### Why GAN works?\nIf a model only learns the data points rather than the whole distribution, this model can only memorize the dataset. A model can generate new data samples when it learns the whole distribution. GAN applies a zero-sum game to help G to simulate the real distribution. Then, we can sample any random points from this distribution as the generated data."},{"metadata":{},"cell_type":"markdown","source":"### Objective function\n- **Discriminator**  \n\nDiscriminator $D$ is going to distinguish all fake images. Intuitively, we can scratch an objective function of $D$:\n$$\\max~J^{(D)} = \\mathbb{E}_{x_r \\sim X_r}[\\textrm{score}(x_r)] - \\mathbb{E}_{x_f \\sim X_f}[\\textrm{score}(x_f)] \\tag{1}$$\nwhere $x_r $ is sampled from the real distribution $X_r $, $x_f $ is sampled from the fake distribution $X_f$, $\\textrm{score}()$ denotes an evaluation function which gives high scores on real images and low scores on fake images. Actually, Discriminator $D$ is like a binary classifier to distinguish real or fake.  \nLet's replace the $\\textrm{score}()$ function by $D$. If the output $D(x)$ is a real number, we hope $D(x_r) \\rightarrow \\infty$ and $D(x_f) \\rightarrow -\\infty$ to achieve the maximum $J^{(D)}$. However, the objective $J^{(D)} \\rightarrow \\infty$ is not applicable in optimization. Thus, we prefer to map the range to probability space, $(-\\infty, \\infty) \\mapsto (0, 1)$. In the objective function, we add a sigmoid transfer function to the output layer.\n$$sigmoid(x) \\equiv \\dfrac{1}{1+e^{-x}}$$  "},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"show_img('../input/introductiongan/introduction-gan/d_simple.png', 'Discriminator with softmax/sigmoid output')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Discriminator with softmax/sigmoid output. In binary classification, these two methods have the same effect. For the following discussion, we use sigmoid output in $D$ by default.  \nWith the sigmoid function, the output denotes the probability of real, $D(x) = p(real)$. We hope the output probability of real images close to 1, $D(x_r) \\rightarrow 1$, while the probability of fake images close to 0, $D(x_f) \\rightarrow 0$. The later one can also be reformulated as $(1 - D(x_f)) \\rightarrow 1$. Then, we modify the objective function (Eq.1) scratched at the beginning. Besides, maximizing the objective function (Eq.2a) is equivalent to minimizing the negative objective function, which is called loss function (Eq.2b).  \n$$\\max~J^{(D)} = \\mathbb{E}_{x_r \\sim X_r}[D(x_r)] + \\mathbb{E}_{x_f \\sim X_f}[1 - D(x_f)] \\tag{2.a}$$\n$$\\Leftrightarrow \\min~L^{(D)} =- \\mathbb{E}_{x_r \\sim X_r}[D(x_r)] - \\mathbb{E}_{x_f \\sim X_f}[1 - D(x_f)] \\tag{2.b}$$  \nThe loss function (Eq.2b) is still based on mean absolute error. As for the loss functions in classification, we often prefer to use cross-entropy (CE) loss rather than mean absolute error (MAE, L1 loss) and mean squared error (MSE, L2 loss). As shown in Fig.3, CE loss can mitigate the gradient vanishing problem when we apply the sigmoid to the output layer."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport numpy as np\n\n# a = sigmoid(wp+b)\n# error = t - a\nerror = np.linspace(0, 1, 100)\n# loss_mse = e^2\n# dL/dw = 2 * e * de/dw = 2 * e * -1 * da/dw = -2 * e * (1 - a) * a * dn/dw\n#  = -2 * e * (1 - a) * a * p = -2 * e * e * (1 - e) * p  ... when t = 1\ngradient_mse = - error ** 2 * (1 - error)\n\n# loss_ce = - t * log a\n# dL/dw = - t * 1/a * da/dw - log a = - t * 1/a * a * (1 - a) dn/dw\n#  = - t * (1 - a) * p = - e * p   ... when t = 1\ngradient_ce = - error\n\n# loss_mae = t - a = e   ... when t = 1\n# dL/dw = - 1 * da/dw = -1 * a * (1 - a) * dn/dw\n#  = -1 * a * (1 - a) * p = - (1 - e) * e * p\ngradient_mae = - error * (1 - error)\n\nplt.plot(error, -gradient_ce, label='CE')\nplt.plot(error, -gradient_mse, label='MSE')\nplt.plot(error, -gradient_mae, label='MAE')\nplt.xlabel(\"absolute error\")\nplt.ylabel(\"norm of gradient\")\nplt.title(\"Fig3. Compare Cross-entropy with MAE, MSE\")\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"MAE/MSE loss: gradient vanishing when the error is large. CE loss: the norm of gradient $\\|\\nabla\\|$ is positively related to the error, a large error leads to a fast learning speed."},{"metadata":{},"cell_type":"markdown","source":"> **Definition 1.** The cross-entropy of the distribution $q$ relative to a distribution $p$ over a given set is defined as follows:\n> $$CE(p, q) = -\\mathbb{E}_p[\\log q]$$\n\nLet's apply the CE loss into the loss function for $D$. For real images, $p$ is  $[1~0]^T$, $q$ is $[D(x_r)~1-D(x_r)]^T$. For fake images, $p$ is $[0~1]^T$, $q$ is $[D(x_f)~1-D(x_f)]^T$. The following equation is a typical binary cross-entropy (BCE) loss."},{"metadata":{},"cell_type":"markdown","source":"$$\n\\begin{align}\nBCE & =\n\t- \\mathbb{E}_{x_r \\sim X_r}\n\t\\left[\n\t\t\\begin{bmatrix}\n           1 & 0\n    \t\t\\end{bmatrix}\n   \t\t ~\n    \t\t\\begin{bmatrix}\n          \\log D(x_r) \\\\\n           \\log (1 - D(x_r))\n    \t\t\\end{bmatrix}\n\t\\right]\n\t-\n\t\\mathbb{E}_{x_f \\sim X_f}\n\t\\left[\n\t\t\\begin{bmatrix}\n           0 & 1\n    \t\t\\end{bmatrix}\n   \t\t ~\n    \t\t\\begin{bmatrix}\n           \\log D(x_f) \\\\\n           \\log (1 - D(x_f))\n    \t\t\\end{bmatrix}\n\t\\right]\n\\end{align}\n$$"},{"metadata":{},"cell_type":"markdown","source":"With algebraic operations, the loss function for $D$ can be derived as below. For notation consistency, we replace $x_f, X_f$ by $x_g, X_g$ because generated images $x_g$ are exactly identified as fake images $x_f$ by $D$. $\\theta_D$ denotes the parameters in $D$ model, which are the weights in neural networks. Our goal is to find the optimal weights to minimize the following BCE loss function.\n$$\\Rightarrow \\underset{\\theta_D} \\min~L^{(D)} =- \\mathbb{E}_{x_r \\sim X_r}[\\log D(x_r)] - \\mathbb{E}_{x_g \\sim X_g}[\\log (1 - D(x_g))] \\tag{3}$$"},{"metadata":{},"cell_type":"markdown","source":"> Any loss consisting of a negative log-likelihood is a cross-entropy between the empirical distribution defined by the training set and the probability distribution defined by model. For example, mean squared error is the cross-entropy between the empirical distribution and a Gaussian model.\n-- Page 132, *Deep Learning*, 2016\n\nThe nature of Eq.3: Cross-entropy is a divergence to measure the difference between two distributions. $- \\mathbb{E}_{x_r \\sim X_r}[\\log D(x_r)]$ is minimizing the divergence between real distribution $X_r$ in the training set and the probability distribution defined by $D$ model; and $- \\mathbb{E}_{x_f \\sim X_f}[\\log (1 - D(x_f))]$ is maximizing the divergence between generated distribution $X_f$ and the probability distribution defined by $D$ model."},{"metadata":{},"cell_type":"markdown","source":"- **Generator**\n\nGenerator $G$ is going to generate real-like images to fool $D$.\nSimilarly, we can scratch an objective function for $G$:\n$$\\max~J^{(G)} = \\mathbb{E}_{x_g \\sim X_g}[\\textrm{score}(x_g)] \\tag{4}$$\nwhere $\\textrm{score}()$ still denotes the $D$ function, $x_g$ is sampled from the generated distribution $X_g$. Here, we use $G$  to map a normal distribution with the generated distribution, $N(0,1) \\overset{G} \\mapsto X_g$. In other words, we feed a noise vector $z \\sim N(0,1)$ into the $G$ function and get a generated image, $x_g = G(z)$."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"show_img('../input/introductiongan/introduction-gan/g_simple.png', 'The training process for Generator', (8,4))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we have trained $D$ to classify real/fake images. The objective of $G$ is to make $D$ classify the generated images into real type."},{"metadata":{},"cell_type":"markdown","source":"Corresponding to $L^{(D)}$ (Eq.3), we also use CE loss on the generated images. There is a trick that $L^{(G)}$ can simply borrow the second term from $L^{(D)}$. i.e. $D$ is maximizing  $- \\mathbb{E}_{x_g \\sim X_g}[\\log (1 - D(x_g))]$ and $G$ can minimizing this term to compete with $D$. This method is exactly a minimax game and is proposed in the original GAN paper. We refer it (Eq.5a) as Saturating loss function. However, Saturating loss function cannot provide sufficient gradient for $G$ to learn. For example, in the early learning step, $D$ can easily distinguish generated images and real images. Thus, $D(x_g)$ is close to 0, $\\log(1-D(x_g))$ will saturate to 0. (Fig.4)\n$$ \\textrm{Saturating: } \\underset{\\theta_G} \\min~L^{(G)} = \\mathbb{E}_{x_g \\sim X_g}[\\log (1 - D(x_g))] \\tag{5.a}$$\nLater, Ian Goodfellow proposed a more stable and efficient loss function for $G$. We can calculate the CE of generated images independently, $p$ is $[1~0]^T$, $q$ is $[D(x_g)~1-D(x_g)]^T$. We refer it (Eq.5b) as Non-Saturating loss function. With this loss function, $-\\log D(x_g)$ is very large and non-saturating at the early learning period. (Fig.4)\n$$ \\textrm{Non-Saturating: } \\underset{\\theta_G} \\min~L^{(G)} = - \\mathbb{E}_{x_g \\sim X_g}[\\log D(x_g)] \\tag{5.b}$$"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"x = np.linspace(0,1,50)[1:-1]\ny_ns = -np.log(x)\ny_s = np.log(1-x)\nplt.plot(x, y_ns, label='Non-saturating')\nplt.plot(x, y_s, label='Saturating')\nplt.xlabel('$D(G(z))$')\nplt.title('Fig.4 Saturating vs Non-saturating')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In practice, Non-Saturating $L^{(G)}$ is better than Saturating $L^{(G)}$. Intuitively, it would be better to set a large learning rate at the early stage of gradient descent while a small learning rate when closing to the optima. "},{"metadata":{},"cell_type":"markdown","source":"# Algorithm\nIf we combine $L^{(G)}$ of Eq.5a and $L^{(D)}$ of Eq.3 together, the aggregate objective function for GAN, $L^{(GAN)}$, can be derived as:\n$$ \\textrm{minimax: } \\underset{\\theta_G} \\min~\\underset{\\theta_D}\\max~L^{(GAN)} = \\mathbb{E}_{x_r \\sim X_r}[\\log D(x_r)] + \\mathbb{E}_{x_g \\sim X_g}[\\log (1 - D(x_g))] \\tag{6.a}$$\nIt is the objective function of original GAN with a minimax game. However, we cannot optimize this combined loss function by changing $G$ and $D$ simultaneously. In practice, we can only train these two models alternatively. In each training iteration, we train $D$ and $G$ sequentially. (Algorithm~1)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"show_img('../input/introductiongan/algorithm.png', 'Pseudo code of GAN')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Training Discriminator.** First, we feed noise vectors $z$ into the $G$ from previous iteration to generate images $x_g = G(z)$. Then, we feed half batch of generated images $x_g$ and half batch of real images $x_r$ into the $D$ from previous iteration. With the loss function (Eq.3), back-propagating gradients to update the parameters in Discriminator $\\theta_D$. To note, the parameters in Generator $\\theta_G$ is frozen. Up to now, we only update $D$ in current iteration."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"show_img('../input/introductiongan/introduction-gan/d_train.png', 'Training Discriminator', (8,4))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Training Generator.** Next, we freeze $\\theta_D$ and going to update $\\theta_G$. Similarly, we feed noise vectors $z$ into the $G$ from previous iteration to generate images $x_g = G(z)$. Then, we feed a batch of generated images $x_g$ into the updated $D$. With the loss function (Eq.5b), back-propagating gradients to update the parameters in Generator $\\theta_G$. After that, we complete one training iteration for GAN."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"show_img('../input/introductiongan/introduction-gan/g_train.png', 'Training Generator', (8,4))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Architecture\n- **Discriminator**\n\nDownsampling networks (Input: images $\\mapsto$ Output: probability)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"show_img('../input/introductiongan/introduction-gan/d_arch.png', 'An example architecture of the Discriminator in DCGAN')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Dense / Fully connected layer: i) Decreasing number of neurons for layers. ii) It cannot be deep and is not good at extracting features.\n- Maxpooling2D. i) The output is just selecting the maximum input value within the  `[height, width]` window of input values. ii) There is no weight and no trainable parameters introduced by this operation."},{"metadata":{},"cell_type":"markdown","source":"$$\n\\begin{align}\n\\texttt{Maxpooling2D([2,2])} & :\n\t\t\\begin{bmatrix}\n           1 & 2 & 3 & 4 \\\\\n           5 & 6 & 7 & 8 \\\\\n           8 & 7 & 4 & 3 \\\\\n           6 & 5 & 2 & 1 \\\\\n    \t\t\\end{bmatrix}\n\t\\mapsto\n    \t\t\\begin{bmatrix}\n           6 & 8 \\\\\n           8 & 4 \n    \t\t\\end{bmatrix}\n\\end{align}\n$$"},{"metadata":{},"cell_type":"markdown","source":"- Convolution2D (`stride=2`). i)The output is a linear combination of the input values times a weight for each cell in the `[height, width]` kernel/filter. ii) These weights become trainable parameters in your model."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"show_img('../input/introductiongan/introduction-gan/conv.png', 'Convolution operation. (Modified from indoml.com)')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- **Generator**\n\nUpsampling networks. (Input: noise vectors $\\mapsto$ Output: images)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"show_img('../input/introductiongan/introduction-gan/g_arch.png', 'An example architecture of the Generator in DCGAN')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Dense / Fully connected layer: i) Increasing number of neurons for layers. ii) It cannot be deep and is not good at generating features.\n- Upsampling2D. i) The `[height, width]` window of output values is just repeating the corresponding input value.\nii) There is no weight and no trainable parameters introduced by this operation."},{"metadata":{},"cell_type":"markdown","source":"$$\n\\begin{align}\n\\texttt{Upsampling2D([2,2])} & :\n\t\t\\begin{bmatrix}\n           1 & 2 \\\\\n           3 & 4 \n    \t\t\\end{bmatrix}\n\t\\mapsto\n\t\t\\begin{bmatrix}\n           1 & 1 & 2 & 2 \\\\\n           1 & 1 & 2 & 2 \\\\\n           3 & 3 & 4 & 4 \\\\\n           3 & 3 & 4 & 4 \\\\\n    \t\t\\end{bmatrix}\n\\end{align}\n$$"},{"metadata":{},"cell_type":"markdown","source":"- ConvTranspose2D / Deconvolution2D (`stride=2`). \ni) It is also named as fractional-strided convolution. Stride here is the reciprocal of the moving step. e.g. `stride=2` $\\Rightarrow$  moving step $=\\frac{1}{2}$. How to move $\\frac{1}{2}$ step? Inserting zero columns and rows to the original input.\nii) Similar to convolution operation, these weights become trainable parameters in your model."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"show_img('../input/introductiongan/introduction-gan/deconv.png', 'Transposed convolution operation. \\n (View more visualizations at \\n https://github.com/vdumoulin/conv_arithmetic \\n by Vincent Dumoulin, Francesco Visin.)')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Evaluation\nGAN is a very new topic, it is still an open problem to find a perfect evaluation of GAN or generated images. Generally, we measure the generated images in two dimensions: quality of images and diversity of images. \n\nApart from comparing the generated images with real images by our eyes, there are two common mathematical metrics to evaluate the quality of the generated images: Inception Score (IS) and Frèchet Inception Distance (FID). Both of these two measurements are based on the Inception V3 network, which is pretrained on ImageNet dataset. IS is derived from the classification output while FID is derived from the feature layer. ****"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"show_img('../input/introductiongan/introduction-gan/fid_is.png', 'FID vs IS')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- **Inception Score**\n\nIS measures the KL divergence (similar to cross-entropy, see Def.2) between the generated sample distribution and the ImageNet distribution, whereas FID calculates the feature-level distance between the generated sample distribution and the real sample distribution."},{"metadata":{},"cell_type":"markdown","source":"$$IS(X_g) = \\exp\\left(\\mathbb{E}_{x_g \\sim X_g} \\left[D_\\textrm{KL}\\left(~p\\left(y|x_g\\right) \\parallel p\\left(y\\right)~\\right)\\right]\\right)$$\n\nwhere $x_g \\sim X_g$ indicates that $x_g$ is an image sampled from $X_g$, $D_\\textrm{KL}(P \\parallel Q)$ is the KL divergence between the distributions P and Q, $p(y|x_g)$ is the conditional class distribution, and $p(y) = \\mathbb{E}_{x_g \\sim X_g} \\left[p(y|x_g)\\right]$ is the marginal class distribution.\n\n$D_\\textrm{KL}$ calculation will output a small number. The $\\exp$ in the expression is there to make the values easier to compare, so it will be ignored if we use $\\ln(IS)$ without loss of generality."},{"metadata":{},"cell_type":"markdown","source":"**Remark.** InceptionV3 network is trained on ImageNet and used to predict the class of given images. If the softmax output is sharp ($p(\\textrm{one class}) \\rightarrow 1, p(\\textrm{other classes}) \\rightarrow 0$), the IS will be small and the quality of this image is high. Besides, the diversity of softmax output indicates the diversity of images."},{"metadata":{},"cell_type":"markdown","source":"> **Definition 2.** In mathematical statistics, the Kullback–Leibler (KL) divergence (also called relative entropy) is a measure of how one probability distribution is different from a second, reference probability distribution. "},{"metadata":{},"cell_type":"markdown","source":"$$\n\\begin{align*}\nD_\\textrm{KL}(P \\parallel Q) &= \\sum_{x\\in\\mathcal{X}} P(x) \\log\\left(\\frac{P(x)}{Q(x)}\\right)\\\\\n&= \\underbrace{\\left(-\\sum_{x\\in\\mathcal{X}} P(x) \\log Q(x)\\right)}_{\\textrm{Cross-entropy}(P,Q)} - \\underbrace{\\left(-\\sum_{x\\in\\mathcal{X}} P(x) \\log P(x)\\right)}_{\\textrm{Cross-entropy}(P,P)}\n\\end{align*}\n$$"},{"metadata":{},"cell_type":"markdown","source":"This equation is a discrete case of KL divergence, where $P$ and $Q$ are two probability distributions, $\\mathcal{X}$ is the probability space. For the continuous case, $P(x), Q(x)$ are probability density functions."},{"metadata":{},"cell_type":"markdown","source":"- **Frèchet Inception Distance**\n\nIS totally depends on the InceptionV3 knowledge on ImageNet. If we input a high quality image which cannot be classified into ImageNet classes to InceptionV3 network, then the softmax output will not be a sharp value. In other words, IS can only evaluate the generated images which should belong to ImageNet dataset. In comparison, FID is a more general evaluation and applicable to new datasets. Besides, feature-level information far surpasses classification-level information.\n\n$$FID={\\Vert}\\mu_r-\\mu_g{\\Vert}^{2}+Tr\\left(\\Sigma_r+\\Sigma_g-2\\left(\\Sigma_r\\Sigma_g\\right)^{1/2}\\right)$$\n\nwhere $\\mu_r $ is the mean of the real features, $\\mu_g $ is the mean of the generated features, $\\Sigma_r $ is the covariance matrix of the real features, $\\Sigma_g $ is the covariance matrix of the generated features. "},{"metadata":{},"cell_type":"markdown","source":"# DCGAN Code Practice"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# %% --------------------------------------- Load Packages -------------------------------------------------------------\nimport os\nimport random\nimport tensorflow as tf  # tf.__version__ >= 2.2.0\nimport numpy as np\nimport matplotlib.pyplot as plt\nfrom tensorflow.keras import Model, Sequential\nfrom tensorflow.keras.layers import Input, Reshape, Dense, Dropout, \\\n    LeakyReLU, Conv2D, Conv2DTranspose, Embedding, \\\n    Concatenate, multiply, Flatten, BatchNormalization\nfrom tensorflow.keras.initializers import glorot_normal\nfrom tensorflow.keras.optimizers import Adam","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# %% --------------------------------------- Fix Seeds -----------------------------------------------------------------\nSEED = 42\nos.environ['PYTHONHASHSEED'] = str(SEED)\nrandom.seed(SEED)\nnp.random.seed(SEED)\ntf.random.set_seed(SEED)\nweight_init = glorot_normal(seed=SEED)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # Load MNIST Fashion\nfrom tensorflow.keras.datasets.fashion_mnist import load_data","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# %% ---------------------------------- Data Preparation ---------------------------------------------------------------\n# change as channel last (n, dim, dim, channel)\ndef change_image_shape(images):\n    shape_tuple = images.shape\n    if len(shape_tuple) == 3:\n        images = images.reshape(-1, shape_tuple[-1], shape_tuple[-1], 1)\n    elif shape_tuple == 4 and shape_tuple[-1] > 3:\n        images = images.reshape(-1, shape_tuple[-1], shape_tuple[-1], shape_tuple[1])\n    return images\n\n# # Load training set\n(x_train, y_train), (x_test, y_test) = load_data()\nx_train, x_test = change_image_shape(x_train), change_image_shape(x_test)\ny_train, y_test = y_train.reshape(-1), y_test.reshape(-1)\n\n######################## Preprocessing ##########################\n# Set channel\nchannel = x_train.shape[-1]\n\n# It is suggested to use [-1, 1] input for GAN training\nx_train = (x_train.astype('float32') - 127.5) / 127.5\nx_test = (x_test.astype('float32') - 127.5) / 127.5\n\n# Get image size\nimg_size = x_train[0].shape\n\n# Get number of classes\nn_classes = len(np.unique(y_train))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# %% ---------------------------------- Hyperparameters ----------------------------------------------------------------\n\n# optimizer = Adam(lr=0.0002, beta_1=0.5, beta_2=0.9)\nlatent_dim = 32\n## trainRatio === times(Train D) / times(Train G)\n# trainRatio = 5","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# %% ---------------------------------- Models Setup -------------------------------------------------------------------\n# Build Generator with convolution layer\ndef generator_conv():\n    noise = Input(shape=(latent_dim,))\n    x = Dense(3 * 3 * 128)(noise)\n    x = LeakyReLU(alpha=0.2)(x)\n\n    ## Out size: 3 x 3 x 128\n    x = Reshape((3, 3, 128))(x)\n\n    ## Size: 7 x 7 x 128\n    # remove padding='same' to scale 6x6 up to 7x7\n    x = Conv2DTranspose(filters=128,\n                        kernel_size=(3, 3),\n                        strides=(2, 2),\n                        # padding='same',\n                        kernel_initializer=weight_init)(x)\n    x = BatchNormalization()(x)\n    x = LeakyReLU(0.2)(x)\n\n    ## Size: 14 x 14 x 64\n    x = Conv2DTranspose(64, (3, 3), strides=(2, 2), padding='same', kernel_initializer=weight_init)(x)\n    x = BatchNormalization()(x)\n    x = LeakyReLU(0.2)(x)\n\n    ## Size: 28 x 28 x channel\n    out = Conv2DTranspose(channel, (3, 3), activation='tanh', strides=(2, 2), padding='same',\n                          kernel_initializer=weight_init)(x)\n\n    model = Model(inputs=noise, outputs=out)\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Build Discriminator with convolution layer\ndef discriminator_conv():\n    # 28 x 28 x channel\n    img = Input(img_size)\n\n    # 14 x 14 x 32\n    x = Conv2D(32, kernel_size=(3, 3), strides=(2, 2), padding='same', kernel_initializer=weight_init)(img)\n    x = LeakyReLU(0.2)(x)\n\n    # 7 x 7 x 64\n    x = Conv2D(64, (3, 3), strides=(2, 2), padding='same', kernel_initializer=weight_init)(x)\n    x = LeakyReLU(0.2)(x)\n\n    # 3 x 3 x 128\n    x = Conv2D(128, (3, 3), strides=(2, 2), kernel_initializer=weight_init)(x)\n    x = LeakyReLU(0.2)(x)\n\n    x = Flatten()(x)\n    x = Dropout(0.4)(x)\n    out = Dense(1)(x)\n\n    model = Model(inputs=img, outputs=out)\n    return model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# %% ----------------------------------- GAN Part ----------------------------------------------------------------------\n# Build our GAN\nclass DCGAN(Model):\n    def __init__(\n        self,\n        discriminator,\n        generator,\n        latent_dim,\n        train_ratio=1,\n    ):\n        super(DCGAN, self).__init__()\n        self.discriminator = discriminator\n        self.generator = generator\n        self.latent_dim = latent_dim\n        self.train_ratio = train_ratio\n\n    def compile(self, d_optimizer, g_optimizer, d_loss_fn, g_loss_fn):\n        super(DCGAN, self).compile()\n        self.d_optimizer = d_optimizer\n        self.g_optimizer = g_optimizer\n        self.d_loss_fn = d_loss_fn\n        self.g_loss_fn = g_loss_fn\n\n    def train_step(self, data):\n        if isinstance(data, tuple):\n            real_images = data[0]\n        else:\n            real_images = data\n        # Get the batch size\n        batch_size = tf.shape(real_images)[0]\n\n        ########################### Train the Discriminator ###########################\n        # training train_ratio times on D while training once on G\n        for i in range(self.train_ratio):\n            # Get the latent vector\n            random_latent_vectors = tf.random.normal(\n                shape=(batch_size, self.latent_dim)\n            )\n            with tf.GradientTape() as tape:\n                # Generate fake images from the latent vector\n                fake_images = self.generator(random_latent_vectors, training=True)\n                # Get the logits for the fake images\n                fake_logits = self.discriminator(fake_images, training=True)\n                # Get the logits for real images\n                real_logits = self.discriminator(real_images, training=True)\n\n                # Calculate loss of D\n                d_loss = self.d_loss_fn(real_logits, fake_logits)\n\n\n            # Get the gradients w.r.t the discriminator loss\n            d_gradient = tape.gradient(d_loss, self.discriminator.trainable_variables)\n            # Update the weights of the discriminator using the discriminator optimizer\n            self.d_optimizer.apply_gradients(\n                zip(d_gradient, self.discriminator.trainable_variables)\n            )\n\n        ########################### Train the Generator ###########################\n        # Get the latent vector\n        random_latent_vectors = tf.random.normal(shape=(batch_size, self.latent_dim))\n        with tf.GradientTape() as tape:\n            # Generate fake images using the generator\n            generated_images = self.generator(random_latent_vectors, training=True)\n            # Get the discriminator logits for fake images\n            gen_img_logits = self.discriminator(generated_images, training=True)\n            # Calculate the generator loss\n            g_loss = self.g_loss_fn(gen_img_logits)\n\n        # Get the gradients w.r.t the generator loss\n        gen_gradient = tape.gradient(g_loss, self.generator.trainable_variables)\n        # Update the weights of the generator using the generator optimizer\n        self.g_optimizer.apply_gradients(\n            zip(gen_gradient, self.generator.trainable_variables)\n        )\n        return {\"d_loss\": d_loss, \"g_loss\": g_loss}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# %% ----------------------------------- Compile Models ----------------------------------------------------------------\n# Optimizer for both the networks\n# learning_rate=0.0002, beta_1=0.5, beta_2=0.9 are recommended\ngenerator_optimizer = Adam(\n    learning_rate=0.0002, beta_1=0.5, beta_2=0.9\n)\ndiscriminator_optimizer = Adam(\n    learning_rate=0.0002, beta_1=0.5, beta_2=0.9\n)\n\n# Define the loss functions to be used for discrimiator\ndef discriminator_loss(real_logits, fake_logits):\n    real_loss = tf.reduce_mean(\n        tf.nn.sigmoid_cross_entropy_with_logits(logits=real_logits, labels=tf.ones_like(real_logits)))\n    fake_loss = tf.reduce_mean(\n        tf.nn.sigmoid_cross_entropy_with_logits(logits=fake_logits, labels=tf.zeros_like(fake_logits)))\n\n    return fake_loss + real_loss\n\n# Define the loss functions to be used for generator\ndef generator_loss(fake_logits):\n    fake_loss = tf.reduce_mean(\n        tf.nn.sigmoid_cross_entropy_with_logits(logits=fake_logits, labels=tf.ones_like(fake_logits)))\n    return fake_loss\n\nd_model = discriminator_conv()\ng_model = generator_conv()\n\ndcgan = DCGAN(generator=g_model,\n              discriminator=d_model,\n              latent_dim=latent_dim,\n              train_ratio=1)\n\ndcgan.compile(\n    d_optimizer=discriminator_optimizer,\n    g_optimizer=generator_optimizer,\n    g_loss_fn=generator_loss,\n    d_loss_fn=discriminator_loss,\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# %% ----------------------------------- Start Training ----------------------------------------------------------------\n# Plot/save generated images through training\ndef plt_img(generator):\n    np.random.seed(42)\n    n = n_classes\n\n    noise = np.random.normal(size=(n * n, latent_dim))\n    decoded_imgs = generator.predict(noise)\n\n    decoded_imgs = decoded_imgs * 0.5 + 0.5\n    x_real = x_test * 0.5 + 0.5\n\n    plt.figure(figsize=(n, n + 1))\n    for i in range(n):\n        # display original\n        ax = plt.subplot(n + 1, n, i + 1)\n        if channel == 3:\n            plt.imshow(x_real[y_test == i][0].reshape(img_size[0], img_size[1], img_size[2]))\n        else:\n            plt.imshow(x_real[y_test == i][0].reshape(img_size[0], img_size[1]))\n            plt.gray()\n        ax.get_xaxis().set_visible(False)\n        ax.get_yaxis().set_visible(False)\n\n        for j in range(n):\n            # display generation\n            ax = plt.subplot(n + 1, n, (i + 1) * n + j + 1)\n            if channel == 3:\n                plt.imshow(decoded_imgs[i * n + j].reshape(img_size[0], img_size[1], img_size[2]))\n            else:\n                plt.imshow(decoded_imgs[i * n + j].reshape(img_size[0], img_size[1]))\n                plt.gray()\n            ax.get_xaxis().set_visible(False)\n            ax.get_yaxis().set_visible(False)\n    plt.show()\n    return\n\n############################# Start training #############################\nLEARNING_STEPS = 6\nfor learning_step in range(LEARNING_STEPS):\n    print('LEARNING STEP # ', learning_step + 1, '-' * 50)\n    dcgan.fit(x_train, epochs=1, batch_size=128)\n    if (learning_step+1)%2 == 0:\n        plt_img(dcgan.generator)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}