{"cells":[{"metadata":{"_uuid":"1cdc07040664b63a1b37beff45bec54df15e54cc","_cell_guid":"3466f9b9-1e50-4228-90c7-a8b3454221a0"},"cell_type":"markdown","source":"# Conditional GAN\nIn a CGAN, you can specify a condition that the generated image has to adhere to. If you were to do this with MNIST, you can choose which label you would like to use to generate an image. This is extremely powerful when you are trying to create images of a certain type. As long as you have labels, you can choose what type of image to create. For example, you could use the CelebA dataset which supplies a picture of a celebrity plus sum attributes which you could use as a set of conditions. Then you could say you want an image of a male actor with sunglasses, etc. We will make a simple CGAN for the MNIST:"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"trusted":true},"cell_type":"code","source":"#Get some packages\nimport numpy as np\nimport pandas as pd\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.autograd import Variable\nimport matplotlib.pyplot as plt\nfrom skimage.io import imshow\nimport time\nimport random","execution_count":1,"outputs":[]},{"metadata":{"_uuid":"92b47155c745c47871f48cff8419907a82af5878","_cell_guid":"add7c98f-a338-45e0-a7d1-5ab82605ed59"},"cell_type":"markdown","source":"## Now let's get an overview of the CGAN architecture\n![](https://i.pinimg.com/736x/05/75/ca/0575cab5214b55e99a59f0b64c35e1c5--arches.jpg)\nAs you can see, it's very simple. When we generate an image, we input the noize $z$ concatenated with the conditions $c$. When we use the discriminator, we concatenate the generated image with the $c$ we used to generate it. If we are showing a real example, we add the condition, $c$ with it.\n<br>\nWe are going to use Pytorch for this demo. This way, I can get into more detail than I could have using Keras. As always, let's download and view the data first:"},{"metadata":{"_uuid":"3bbe4bb934de63740a247ae1e374ad4546bc93d0","_cell_guid":"e5c58ad6-065a-41d9-aed5-fcfdebb37c6c","collapsed":true,"trusted":true},"cell_type":"code","source":"#Read CSV\ncsv = pd.read_csv('../input/train.csv')\n#Separate into matricies\nX_train = csv.iloc[:,1:786].as_matrix()\nY_train = csv.iloc[:,0].as_matrix()","execution_count":2,"outputs":[]},{"metadata":{"_uuid":"cc87b4cf8adc96b2c072c696819654c462d3363f","_cell_guid":"d8980389-c742-492a-a247-75e428695404","collapsed":true,"trusted":true},"cell_type":"code","source":"X_train_imgs = np.zeros([42000,1,28,28])\nfor i in range(X_train.shape[0]):\n    img = X_train[i,:].reshape([1,28,28])/255.\n    X_train_imgs[i] = img","execution_count":3,"outputs":[]},{"metadata":{"_uuid":"e5adea6da5734dc1cf765eac80216e295184edfe","_cell_guid":"919e6d81-1789-45f9-89ec-a10d2bc7d39f","collapsed":true,"trusted":true},"cell_type":"code","source":"Y_train_oh = np.zeros([42000,10])\nfor i in range(Y_train.shape[0]):\n    oh = np.zeros([10])\n    oh[int(Y_train[i])] = 1.\n    Y_train_oh[i] = oh","execution_count":4,"outputs":[]},{"metadata":{"_uuid":"4823c555c5b337ab563aa988eda02a9dbf3f6e3e","_cell_guid":"f99374d2-f230-4a7f-accf-37b6cfbe6ecf","trusted":true},"cell_type":"code","source":"ix = 599 #0-42000\nimshow(np.squeeze(X_train_imgs[ix]))\nplt.show()\nprint ('This is:',Y_train[ix])","execution_count":5,"outputs":[]},{"metadata":{"_uuid":"57135bc7b98c1bb05408e2abf6434fa750bde10e","_cell_guid":"7ac42851-f9ad-4d1f-9640-152496d17857"},"cell_type":"markdown","source":"## Tricks on Training GANs\nGANs are notoriously hard to train. Since we are using 2 neural networks, we need to make sure they are balanced. That is ONE of the problems. Another is mode collapse, where the generator doesn't produce images with lots of variety. This can get problematic, so we try to employ some tricks to keep the GANs balanced:\n1. Sample from a normal distrubution, not a uniform one\n2. Normalize images between -1 and 1, not 1 and 0\n3. Use $\\max\\log D$ instead of $\\min (\\log 1-D)$ as a loss to train the Generator\n4. Construct whole mini-batches of real and generated images, not a mix\n5. Use LeakyReLU not ReLU\n6. Use ConvTranspose2D instead of Upsampling\n7. Use Label Smoothing\n\nWe will also add a more complex set of variables that will be returned when training. Not only will we return the $G$ and $D$ loss, but also the $D$'s variance. We want the variance to stay low in the discriminator loss, so we will keep track of it.\n\n**Note:** We will be using a Deep Convolutional GAN for it's superior performance!\n![](https://i2.wp.com/kawahara.ca/wp-content/uploads/unsupervised_representation_learning_with_DCGAN.png)"},{"metadata":{"_uuid":"aedecfabe2dcfa923955976bceab911185902d28","_cell_guid":"58195cfe-28b7-4f53-b328-c5a506e750ac"},"cell_type":"markdown","source":"### Generator Network"},{"metadata":{"_uuid":"1b5fd2d73c64f420960c43dcf842e688a66de87d","_cell_guid":"cf31d3ca-2a04-4011-8c92-4179b69fe354","collapsed":true,"trusted":true},"cell_type":"code","source":"class _G(nn.Module):\n    def __init__(self, z_size, c_size):\n        super(_G, self).__init__()\n        \n        self.conv2dtranspose_z = nn.ConvTranspose2d(in_channels=z_size, out_channels=256, kernel_size=4, stride=1)\n        self.bn2d_z = nn.BatchNorm2d(256, momentum=0.9)\n        self.conv2dtranspose_c = nn.ConvTranspose2d(in_channels=c_size, out_channels=256, kernel_size=4, stride=1)\n        self.bn2d_c = nn.BatchNorm2d(256, momentum=0.9)\n        self.backbone = nn.Sequential(\n            nn.ConvTranspose2d(in_channels=512, out_channels=256, kernel_size=4, stride=2, padding=1),\n            nn.LeakyReLU(negative_slope=0.2),\n            nn.BatchNorm2d(256, momentum=0.9),\n            nn.ConvTranspose2d(in_channels=256, out_channels=128, kernel_size=4, stride=2, padding=1),\n            nn.LeakyReLU(negative_slope=0.2),\n            nn.BatchNorm2d(128, momentum=0.9),\n            nn.ConvTranspose2d(in_channels=128, out_channels=1, kernel_size=2, stride=2, padding=2),\n            nn.Tanh()\n        )\n    \n    # weight_init\n    def weight_init(self, mean, std):\n        for m in self._modules:\n            normal_init(self._modules[m], mean, std)\n    \n    def forward(self, z, c):\n        z = F.leaky_relu(self.bn2d_z(self.conv2dtranspose_z(z.view(-1,100,1,1))))\n        c = F.leaky_relu(self.bn2d_c(self.conv2dtranspose_c(c.view(-1,10,1,1))))\n        zc = torch.cat([z,c],dim=1)\n        output = self.backbone(zc)\n        return output","execution_count":44,"outputs":[]},{"metadata":{"_uuid":"c37a8fd7b4fed76b47ee398390d62c0b5a4fb167","_cell_guid":"ee7ed6e7-5ce9-462c-83b3-657fb4ff07c8"},"cell_type":"markdown","source":"### Discriminator Network"},{"metadata":{"_uuid":"76241c654ad5c117cc4c4fe68668b44a7e60b905","_cell_guid":"8bfe9f95-2998-4030-8203-a2ad5a2c4501","collapsed":true,"trusted":true},"cell_type":"code","source":"class _D(nn.Module):\n    def __init__(self,c_size):\n        super(_D, self).__init__()\n        \n        self.conv2d_x = nn.Conv2d(in_channels=1, out_channels=64, kernel_size=4, stride=2, padding=1)\n        self.conv2d_c = nn.Conv2d(in_channels=10, out_channels=64, kernel_size=4, stride=2, padding=1)\n        self.backbone = nn.Sequential(\n            nn.Conv2d(in_channels=128, out_channels=256, kernel_size=4, stride=2, padding=1),\n            nn.LeakyReLU(negative_slope=0.2),\n            nn.BatchNorm2d(256, momentum=0.9),\n            nn.Conv2d(in_channels=256, out_channels=512, kernel_size=4, stride=2, padding=1),\n            nn.LeakyReLU(negative_slope=0.2),\n            nn.BatchNorm2d(512, momentum=0.9),\n            nn.Conv2d(in_channels=512, out_channels=1, kernel_size=3, stride=2),\n            nn.Sigmoid()\n        )\n    \n    # weight_init\n    def weight_init(self, mean, std):\n        for m in self._modules:\n            normal_init(self._modules[m], mean, std)\n            \n    \n    def forward(self, x, c):\n        x = self.conv2d_x(x)\n        c = c.view(-1,10,1,1)\n        c = c.expand(-1,10,28,28)\n        c = self.conv2d_c(c)\n        xc = torch.cat([x,c],dim=1)\n        output = self.backbone(xc)\n        output = output.view(-1,1)\n        return output","execution_count":45,"outputs":[]},{"metadata":{"_uuid":"c5c9af21a82201faa6b676de1b22808a2db304a3","_cell_guid":"c262c866-f865-49c2-a833-dabe2a653812"},"cell_type":"markdown","source":"### Now lets instantiate the Networks"},{"metadata":{"_uuid":"19fb11b989497be3ceb8f178ac9b5a4af0fb5c98","_cell_guid":"117374e3-af9f-4173-9c5a-cf7af6d58e40","collapsed":true,"trusted":true},"cell_type":"code","source":"G = _G(100, 10) #Noise vector will have size 100, and we will have a condition vector of 10(1 for each type of item)\nD = _D(10) #The Discriminator will also use the condition, so we say it has size 10\ndef normal_init(m, mean, std):\n    if isinstance(m, nn.ConvTranspose2d) or isinstance(m, nn.Conv2d):\n        m.weight.data.normal_(mean, std)\n        m.bias.data.zero_()\nG.weight_init(mean=0, std=0.2) #GAN works better with these weight initializations\nD.weight_init(mean=0, std=0.2)","execution_count":46,"outputs":[]},{"metadata":{"_uuid":"ec45c498dfe48d031be493cc25838f35466e005f","_cell_guid":"d353b991-22bf-4c2e-a4d4-00c5d2ebaa02"},"cell_type":"markdown","source":"### Moving the networks to the GPU\nUnlike Keras, we have to move the network to the GPU manually. Pytorch doesn't do this automatically. This is because it allows you to construct a complex multithreaded data feeder.....dadada.... The gist is that Pytorch gives you a lot more flexibility than Keras. It is also nearly twice as fast and extremely memory efficient."},{"metadata":{"_uuid":"a894e7f9ac4b7c544c731f57d677a27ec91ef73d","_cell_guid":"90accb03-f723-40a8-ad39-51708bfd6096","trusted":true},"cell_type":"code","source":"G.cuda()\nD.cuda()\n#That's how you move it to the GPU","execution_count":47,"outputs":[]},{"metadata":{"_uuid":"8176c7b3e32b80b9b46a5301ffdb18ca358f0ac7","_cell_guid":"7475f018-0b4e-49d5-8b9b-c5cb7d870667"},"cell_type":"markdown","source":"### Creating the Loss Function and Optimizers"},{"metadata":{"_uuid":"8a494954e19aa985bd265c4088443189c52cded3","_cell_guid":"24cdd7c1-e8d7-4301-b156-d6e5c25ee107","collapsed":true,"trusted":true},"cell_type":"code","source":"criterion = nn.BCELoss()#Binary Cross-Entropy Loss\noptim_G = optim.Adam(G.parameters(), lr=0.0002)\noptim_D = optim.Adam(D.parameters(), lr=0.0002)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"24b0709ed4b42ee8b367cebfe8394239fd9321ba","_cell_guid":"0ec8733d-3f8b-4ad5-8843-2b5c5f69a2c6"},"cell_type":"markdown","source":"## Now let's create two functions for optimizing each network\nI'm going this to simplify the process. Let's start with the optimization function:"},{"metadata":{"_uuid":"a1c3793ffb9d7da155846be49be1046f34e9dfb6","_cell_guid":"2e15c017-3aee-4fa3-8329-0e871d495c43","collapsed":true,"trusted":true},"cell_type":"code","source":"def optimize_G(G, D, z, c, optimizer, criterion):\n    \n    \"\"\"\n    When we train the generator we want it to trick the discriminator. This means that we want the output of D to be close to 1,\n    meaning it thinks its real. Keep that in mind. When we train G, we make the fake labels equal 1 so the optimizer tried to\n    make the generator make an image that tricks D.\n    \"\"\"\n    \n    #Even though the images are fake, we want the discriminator to think they are real\n    trick_labels = Variable(torch.ones([z.shape[0],1])-torch.rand([z.shape[0],1])/3).cuda()\n    #Zero gradient buffers\n    G.zero_grad()\n    #Generate Images\n    fake_x = G.forward(z, c)\n    D_preds = D(fake_x, c)\n    loss = criterion(D_preds, trick_labels)\n    loss.backward()\n    optimizer.step()\n    \n    return fake_x, loss","execution_count":41,"outputs":[]},{"metadata":{"_uuid":"68db90dbd482d7ed6cae45cc2ee8c9ac288c94cd","_cell_guid":"2448b36a-5ce9-4248-830e-2b00632fa230","collapsed":true,"trusted":true},"cell_type":"code","source":"def optimize_D(net, fake_x, fake_c, real_x, real_c, optimizer, criterion):\n    #We cannot feed a numpy variable. We have to use a torch.autograd.Variable\n    fake_labels = Variable(torch.zeros([fake_x.shape[0],1])+torch.rand([z.shape[0],1])/3).cuda()\n    real_labels = Variable(torch.ones([real_x.shape[0],1])-torch.rand([z.shape[0],1])/3).cuda()\n    \n    #We need to empty the gradient buffers\n    net.zero_grad()\n    \n    #Let's get the discriminator predictions for the fake images\n    fake_preds = net.forward(fake_x.detach(), fake_c)\n    #Do the optimization\n    fake_loss = criterion(fake_preds, fake_labels)\n    #Let's get the discriminator predictions for the real images\n    real_preds = net.forward(real_x, real_c)\n    #Do the optimization\n    real_loss = criterion(real_preds, real_labels)\n    \n    loss = fake_loss + real_loss\n    loss.backward()\n    optimizer.step()\n    \n    return fake_loss + real_loss","execution_count":42,"outputs":[]},{"metadata":{"_uuid":"fdbefd0aea33e909ddc665ac717ce452dc3eea75","_cell_guid":"bd980e08-4970-45fe-831e-a1b23767d348"},"cell_type":"markdown","source":"## Now let's start the training\nLets lay out the plan to the code we will use for training:\n1. Sample $z$ from a Gaussian Distribution\n2. Create a batch of condition $c$\n3. Optimize $G$\n4. Optimize $D$\n5. Repeat"},{"metadata":{"_uuid":"d002dc6426ab9a64562d1132762af34addc7fc8a","_cell_guid":"38a91630-865d-4770-8dcf-d62c5f8c30a0","_kg_hide-output":false,"scrolled":true,"trusted":true},"cell_type":"code","source":"D_history = []\nG_history = []\nEPOCHS = 10\nBATCH_SIZE = 128\n\nfor epoch in range(EPOCHS):\n    train_loss = 0\n    speed = 0\n    for batch_number in range(int(Y_train.shape[0]/BATCH_SIZE)):\n        G.train()\n        time_start = time.time()\n        real_x = Variable(torch.FloatTensor(X_train_imgs[batch_number*BATCH_SIZE:(1+batch_number)*BATCH_SIZE])).cuda()\n        real_x = (real_x-real_x.mean())/real_x.std()\n        real_c = Variable(torch.FloatTensor(Y_train_oh[batch_number*BATCH_SIZE:(1+batch_number)*BATCH_SIZE])).cuda()\n        \n        z = Variable(torch.FloatTensor(np.random.randn(BATCH_SIZE, 100))).cuda()\n        fake_x, loss = optimize_G(G, D, z, real_c, optim_G, criterion)\n        G_history.append(loss.data.cpu().numpy()[0])\n        \n        loss = optimize_D(D, fake_x, real_c, real_x, real_c, optim_D, criterion)\n        D_history.append(loss.data.cpu().numpy()[0])\n        \n        \n        if batch_number % 25 == 0:\n            bigfig = []\n            for i in range(0,10):\n                z = np.random.randn(1,100)\n                z = torch.FloatTensor(z)\n                z = Variable(z).cuda()\n                G.eval()\n                fig = []\n                for i in range(0, 10):\n                    c = np.zeros([1,10])\n                    c[0,i] = 1.\n                    c = torch.FloatTensor(c)\n                    c = Variable(c).cuda()\n                    gens = G.forward(z, c)\n                    gens = gens.data.cpu().numpy()\n                    gens = gens.reshape([28,28])\n                    fig.append(gens/2+0.5)\n                fig = np.hstack(fig)\n                bigfig.append(fig)\n            bigfig = np.vstack(bigfig)\n            print (bigfig.shape)\n            imshow(bigfig)\n            plt.show()\n            \n            print ('G loss: ',G_history[-1])\n            print ('D loss: ',D_history[-1])\n            print ('D loss variance: ',np.stack(D_history,axis=0).std())\n    print ('Finished Epoch',epoch+1)","execution_count":43,"outputs":[]},{"metadata":{"_uuid":"7c1af560530e602069533425df6abb51359c3643","_cell_guid":"130f817e-b4ab-4dba-9afa-34416f614c78","collapsed":true,"trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e8dee1bdd388697cbab4c14050f9d637bfa5cf2d","_cell_guid":"67a4bf5e-fc37-42f2-a31f-68bc183f9935","collapsed":true,"trusted":false},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":1}