{"cells":[{"metadata":{},"cell_type":"markdown","source":"Generative Adversarial Networks (GANs)\n---\n\n
"},{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\n- Generative Adversarial Networks (GANs)\n- How GANs Work\n- GANs Process\n- Examples\n\n### Generative Adversarial Networks (GANs)\n\nGenerative Adversarial Networks are used to generate images that never existed before. They learn about the world (objects, animals and so forth) and create new versions of those images that never existed.\n\nThey have two components:\n\n- A **Generator** - this creates the images.\n- A **Discriminator** - this assesses the images and tells the generator if they are similar to what it has been trained on. These are based off real world examples.\n\nWhen training the network, both the generator and discriminator start from scratch and learn together.\n\n### How GANs Work\n\n\n**G** for **Generative** - this is a model that takes an input as a random noise singal and then outputs an image.\n\n\n\n**A** for **Adversarial** - this is the discriminator, the opponent of the generator. This is capable of learning about objects, animals or other features specified. For example: if you supply it with pictures of dogs and non-dogs, it would be able to identify the difference between the two.\n\n\n\nUsing this example, once the discriminator has been trained, showing the discriminator a picture that isn't a dog it will return a 0. Whereas, if you show it a dog it will return a 1.\n\n\n\n**N** for **Network** - meaning the generator and discriminator are both neural networks.\n\n\n### GANs Process\n\n**Step 1** - we input a random noise signal into the generator. The generator creates some images which is used for training the discriminator. We provide the discriminator with some features/images we want it to learn and the discriminator outputs probabilities. These probabilities can be rather high as the discriminator has only just started being trained. The values are then assessed and identified. The error is calculated and these are backpropagated through the discriminator, where the weights are updated.\n\n\n\nNext we train the generator. We take the batch of images that it created and put them through the discriminator again. We do not include the feature images. The generator learns by tricking the discriminator into it outputting false positives.\n\nThe discriminator will provide an output of probabilities. The values are then assessed and compared to what they should have been. The error is calculated and backpropagated through the generator and the weights are updated.\n\n\n\n**Step 2** - This is the same as step 1 but the generator and discriminator are trained a little more. Through backpropagation the generator understands its mistakes and starts to make them more like the feature.\n\nThis is created through a *Deconvolutional Neural Network*.\n\n### Examples\n\n**GANs** can be used for the following:\n\n- Generating Images\n- Image Modification\n- Super Resolution\n- Assisting Artists\n- Photo-Realistic Images\n- Speech Generation\n- Face Ageing\n\n
\n**[It’s Training Cats and Dogs: NVIDIA Research Uses AI to Turn Cats Into Dogs, Lions and Tigers, Too](https://blogs.nvidia.com/blog/2018/04/15/nvidia-research-image-translation/)**\n![](\"https://blogs.nvidia.com/wp-content/uploads/2018/04/cats-dogs-nvresearch1.png\")
\n
\n![](\"https://cdn-images-1.medium.com/max/800/1*HaExieykcOT5oI2_xKisrQ.png\")
\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Importing the libraries**"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from __future__ import print_function\nimport time\nimport torch\nimport torch.nn as nn\nimport torch.nn.parallel\nimport torch.optim as optim\nimport torch.utils.data\nimport torchvision.datasets as dset\nimport torchvision.transforms as transforms\nimport torchvision.utils as vutils\nfrom torch.autograd import Variable\nimport matplotlib.pyplot as plt\nimport numpy as np\nfrom torch import nn, optim\nimport torch.nn.functional as F\nfrom torchvision import datasets, transforms\nfrom torchvision.utils import save_image\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nfrom tqdm import tqdm_notebook as tqdm","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Some dogs\n\n> The Stanford Dogs dataset contains images of 120 breeds of dogs from around the world."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"PATH = '../input/all-dogs/all-dogs/'\nimages = os.listdir(PATH)\nprint(f'There are {len(os.listdir(PATH))} pictures of dogs.')\n\nfig, axes = plt.subplots(nrows=3, ncols=3, figsize=(12,10))\n\nfor indx, axis in enumerate(axes.flatten()):\n rnd_indx = np.random.randint(0, len(os.listdir(PATH)))\n # https://matplotlib.org/users/image_tutorial.html\n img = plt.imread(PATH + images[rnd_indx])\n imgplot = axis.imshow(img)\n axis.set_title(images[rnd_indx])\n axis.set_axis_off()\nplt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Insights\n\nCheck this posts:\n\n>[Quick data explanation and EDA](https://www.kaggle.com/witold1/quick-data-explanation-and-eda) by @witold1\n\n> [New Insights](https://www.kaggle.com/c/generative-dog-images/discussion/97863#latest-564673)\n\n- There are pictures with more than one dog (even with 3 dogs);\n- There are pictures with the dog (-s) and person (people);\n- There are pictures with more than one person (even with 4 people);\n- There are pictures where dogs occupy less than 1/5 of the picture;\n- There are pictures with text (magazine covers, from dog shows, memes and pictures with text);\n- Even wild predators included, e.g. African wild dog or Dingo, but not wolves.\n\n**Examples**\n\n![\"Captura-de-pantalla-de-2019-06-29-12-31-41\"](\"https://i.ibb.co/cxZ3nwd/Captura-de-pantalla-de-2019-06-29-12-31-41.png\")
\n![\"Captura-de-pantalla-de-2019-06-29-12-32-48\"](\"https://i.ibb.co/TRzg1JG/Captura-de-pantalla-de-2019-06-29-12-32-48.png\")
\n![\"Captura-de-pantalla-de-2019-06-29-12-32-28\"](\"https://i.ibb.co/LNgrSTj/Captura-de-pantalla-de-2019-06-29-12-32-28.png\")
\n![\"Captura-de-pantalla-de-2019-06-29-12-31-59\"](\"https://i.ibb.co/p1LBrz3/Captura-de-pantalla-de-2019-06-29-12-31-59.png\")
\n\n"},{"metadata":{},"cell_type":"markdown","source":"# Image Preprocessing\n### Check: [GAN dogs starter](https://www.kaggle.com/wendykan/gan-dogs-starter)"},{"metadata":{},"cell_type":"markdown","source":"**Initial code ...**\n\n```\nbatchSize = 64\nimageSize = 64\n\n# 64x64 images!\ntransform = transforms.Compose([transforms.Resize(64),\n transforms.CenterCrop(64),\n transforms.ToTensor(),\n transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])\n\ntrain_data = datasets.ImageFolder('../input/all-dogs/', transform=transform)\n\ndataloader = torch.utils.data.DataLoader(train_data, shuffle=True,\n batch_size=batch_size)\n\nimgs, label = next(iter(dataloader))\nimgs = imgs.numpy().transpose(0, 2, 3, 1)\n\n```"},{"metadata":{},"cell_type":"markdown","source":"**New data Data loader and Augmentations from [RaLSGAN dogs](https://www.kaggle.com/speedwagon/ralsgan-dogs)**"},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 32\nimage_size = 64\n\nrandom_transforms = [transforms.ColorJitter(), transforms.RandomRotation(degrees=20)]\ntransform = transforms.Compose([transforms.Resize(64),\n transforms.CenterCrop(64),\n transforms.RandomHorizontalFlip(p=0.5),\n transforms.RandomApply(random_transforms, p=0.2),\n transforms.ToTensor(),\n transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])\n\ntrain_data = datasets.ImageFolder('../input/all-dogs/', transform=transform)\ntrain_loader = torch.utils.data.DataLoader(train_data, shuffle=True,\n batch_size=batch_size)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"imgs, label = next(iter(train_loader))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"type(imgs), type(label)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"a = imgs.numpy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"type(a)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"imgs = imgs.numpy().transpose(0, 2, 3, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"type(imgs), imgs.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 32\nimage_size = 64\n\nrandom_transforms = [transforms.ColorJitter(), transforms.RandomRotation(degrees=20)]\ntransform = transforms.Compose([transforms.Resize(64),\n transforms.CenterCrop(64),\n transforms.RandomHorizontalFlip(p=0.5),\n transforms.RandomApply(random_transforms, p=0.2),\n transforms.ToTensor(),\n transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])\n\ntrain_data = datasets.ImageFolder('../input/all-dogs/', transform=transform)\ntrain_loader = torch.utils.data.DataLoader(train_data, shuffle=True,\n batch_size=batch_size)\n \nimgs, label = next(iter(train_loader))\nimgs = imgs.numpy().transpose(0, 2, 3, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"type(imgs), imgs.shape, type(label)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(5):\n plt.imshow(imgs[i])\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Weights\n### Defining the weights_init function"},{"metadata":{"trusted":true},"cell_type":"code","source":"def weights_init(m):\n \"\"\"\n Takes as input a neural network m that will initialize all its weights.\n \"\"\"\n classname = m.__class__.__name__\n if classname.find('Conv') != -1:\n m.weight.data.normal_(0.0, 0.02)\n elif classname.find('BatchNorm') != -1:\n m.weight.data.normal_(1.0, 0.02)\n m.bias.data.fill_(0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Generator"},{"metadata":{"trusted":true},"cell_type":"code","source":"class G(nn.Module):\n def __init__(self):\n # Used to inherit the torch.nn Module\n super(G, self).__init__()\n # Meta Module - consists of different layers of Modules\n self.main = nn.Sequential(\n nn.ConvTranspose2d(100, 512, 4, stride=1, padding=0, bias=False),\n nn.BatchNorm2d(512),\n nn.ReLU(True),\n nn.ConvTranspose2d(512, 256, 4, stride=2, padding=1, bias=False),\n nn.BatchNorm2d(256),\n nn.ReLU(True),\n nn.ConvTranspose2d(256, 128, 4, stride=2, padding=1, bias=False),\n nn.BatchNorm2d(128),\n nn.ReLU(True),\n nn.ConvTranspose2d(128, 64, 4, stride=2, padding=1, bias=False),\n nn.BatchNorm2d(64),\n nn.ReLU(True),\n nn.ConvTranspose2d(64, 3, 4, stride=2, padding=1, bias=False),\n nn.Tanh()\n )\n \n def forward(self, input):\n output = self.main(input)\n return output\n\n# Creating the generator\nnetG = G()\nnetG.apply(weights_init)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Discriminator"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Defining the discriminator\nclass D(nn.Module):\n def __init__(self):\n super(D, self).__init__()\n self.main = nn.Sequential(\n nn.Conv2d(3, 64, 4, stride=2, padding=1, bias=False),\n nn.LeakyReLU(negative_slope=0.2, inplace=True),\n nn.Conv2d(64, 128, 4, stride=2, padding=1, bias=False),\n nn.BatchNorm2d(128),\n nn.LeakyReLU(negative_slope=0.2, inplace=True),\n nn.Conv2d(128, 256, 4, stride=2, padding=1, bias=False),\n nn.BatchNorm2d(256),\n nn.LeakyReLU(negative_slope=0.2, inplace=True),\n nn.Conv2d(256, 512, 4, stride=2, padding=1, bias=False),\n nn.BatchNorm2d(512),\n nn.LeakyReLU(negative_slope=0.2, inplace=True),\n nn.Conv2d(512, 1, 4, stride=1, padding=0, bias=False),\n nn.Sigmoid()\n )\n \n def forward(self, input):\n output = self.main(input)\n # .view(-1) = Flattens the output into 1D instead of 2D\n return output.view(-1)\n \n \n# Creating the discriminator\nnetD = D()\nnetD.apply(weights_init)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Another setup"},{"metadata":{"trusted":true},"cell_type":"code","source":"class Generator(nn.Module):\n def __init__(self, nz=128, channels=3):\n super(Generator, self).__init__()\n \n self.nz = nz\n self.channels = channels\n \n def convlayer(n_input, n_output, k_size=4, stride=2, padding=0):\n block = [\n nn.ConvTranspose2d(n_input, n_output, kernel_size=k_size, stride=stride, padding=padding, bias=False),\n nn.BatchNorm2d(n_output),\n nn.ReLU(inplace=True),\n ]\n return block\n\n self.model = nn.Sequential(\n *convlayer(self.nz, 1024, 4, 1, 0), # Fully connected layer via convolution.\n *convlayer(1024, 512, 4, 2, 1),\n *convlayer(512, 256, 4, 2, 1),\n *convlayer(256, 128, 4, 2, 1),\n *convlayer(128, 64, 4, 2, 1),\n nn.ConvTranspose2d(64, self.channels, 3, 1, 1),\n nn.Tanh()\n )\n\n def forward(self, z):\n z = z.view(-1, self.nz, 1, 1)\n img = self.model(z)\n return img\n\n \nclass Discriminator(nn.Module):\n def __init__(self, channels=3):\n super(Discriminator, self).__init__()\n \n self.channels = channels\n\n def convlayer(n_input, n_output, k_size=4, stride=2, padding=0, bn=False):\n block = [nn.Conv2d(n_input, n_output, kernel_size=k_size, stride=stride, padding=padding, bias=False)]\n if bn:\n block.append(nn.BatchNorm2d(n_output))\n block.append(nn.LeakyReLU(0.2, inplace=True))\n return block\n\n self.model = nn.Sequential(\n *convlayer(self.channels, 32, 4, 2, 1),\n *convlayer(32, 64, 4, 2, 1),\n *convlayer(64, 128, 4, 2, 1, bn=True),\n *convlayer(128, 256, 4, 2, 1, bn=True),\n nn.Conv2d(256, 1, 4, 1, 0, bias=False), # FC with Conv.\n )\n\n def forward(self, imgs):\n logits = self.model(imgs)\n out = torch.sigmoid(logits)\n \n return out.view(-1, 1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training"},{"metadata":{},"cell_type":"markdown","source":"**My training baseline**"},{"metadata":{"trusted":true},"cell_type":"code","source":"!mkdir results\n!ls","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"EPOCH = 0 # play with me\nLR = 0.001\ncriterion = nn.BCELoss()\noptimizerD = optim.Adam(netD.parameters(), lr=LR, betas=(0.5, 0.999))\noptimizerG = optim.Adam(netG.parameters(), lr=LR, betas=(0.5, 0.999))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This doesn't run because EPOCH = 0, change it and try ;)"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"for epoch in range(EPOCH):\n for i, data in enumerate(dataloader, 0):\n # 1st Step: Updating the weights of the neural network of the discriminator\n netD.zero_grad()\n \n # Training the discriminator with a real image of the dataset\n real,_ = data\n input = Variable(real)\n target = Variable(torch.ones(input.size()[0]))\n output = netD(input)\n errD_real = criterion(output, target)\n \n # Training the discriminator with a fake image generated by the generator\n noise = Variable(torch.randn(input.size()[0], 100, 1, 1))\n fake = netG(noise)\n target = Variable(torch.zeros(input.size()[0]))\n output = netD(fake.detach())\n errD_fake = criterion(output, target)\n \n # Backpropagating the total error\n errD = errD_real + errD_fake\n errD.backward()\n optimizerD.step()\n \n # 2nd Step: Updating the weights of the neural network of the generator\n netG.zero_grad()\n target = Variable(torch.ones(input.size()[0]))\n output = netD(fake)\n errG = criterion(output, target)\n errG.backward()\n optimizerG.step()\n \n # 3rd Step: Printing the losses and saving the real images and the generated images of the minibatch every 100 steps\n print('[%d/%d][%d/%d] Loss_D: %.4f; Loss_G: %.4f' % (epoch, EPOCH, i, len(dataloader), errD.item(), errG.item()))\n if i % 100 == 0:\n vutils.save_image(real, '%s/real_samples.png' % \"./results\", normalize=True)\n fake = netG(noise)\n vutils.save_image(fake.data, '%s/fake_samples_epoch_%03d.png' % (\"./results\", epoch), normalize=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Best public training\n- 06/29 [RaLSGAN dogs](https://www.kaggle.com/speedwagon/ralsgan-dogs) V9\n- 06/29 this kernel V5\n- some version of this kernel"},{"metadata":{},"cell_type":"markdown","source":"### Parameters"},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 32\nLR_G = 0.001\nLR_D = 0.0005\n\nbeta1 = 0.5\nepochs = 100\n\nreal_label = 0.9\nfake_label = 0\nnz = 128\n\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Initialize models and optimizers"},{"metadata":{"trusted":true},"cell_type":"code","source":"netG = Generator(nz).to(device)\nnetD = Discriminator().to(device)\n\ncriterion = nn.BCELoss()\n\noptimizerD = optim.Adam(netD.parameters(), lr=LR_D, betas=(beta1, 0.999))\noptimizerG = optim.Adam(netG.parameters(), lr=LR_G, betas=(beta1, 0.999))\n\nfixed_noise = torch.randn(25, nz, 1, 1, device=device)\n\nG_losses = []\nD_losses = []\nepoch_time = []","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Plot Loss per EPOCH**\n> plot_loss()"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"def plot_loss (G_losses, D_losses, epoch):\n plt.figure(figsize=(10,5))\n plt.title(\"Generator and Discriminator Loss - EPOCH \"+ str(epoch))\n plt.plot(G_losses,label=\"G\")\n plt.plot(D_losses,label=\"D\")\n plt.xlabel(\"iterations\")\n plt.ylabel(\"Loss\")\n plt.legend()\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Show generated images**\n> show_generated_img()"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"trusted":true},"cell_type":"code","source":"def show_generated_img(n_images=5):\n sample = []\n for _ in range(n_images):\n noise = torch.randn(1, nz, 1, 1, device=device)\n gen_image = netG(noise).to(\"cpu\").clone().detach().squeeze(0)\n gen_image = gen_image.numpy().transpose(1, 2, 0)\n sample.append(gen_image)\n \n figure, axes = plt.subplots(1, len(sample), figsize = (64,64))\n for index, axis in enumerate(axes):\n axis.axis('off')\n image_array = sample[index]\n axis.imshow(image_array)\n \n plt.show()\n plt.close()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Training Loop"},{"metadata":{"_kg_hide-output":false,"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"for epoch in range(epochs):\n \n start = time.time()\n for ii, (real_images, train_labels) in tqdm(enumerate(train_loader), total=len(train_loader)):\n ############################\n # (1) Update D network: maximize log(D(x)) + log(1 - D(G(z)))\n ###########################\n # train with real\n netD.zero_grad()\n real_images = real_images.to(device)\n batch_size = real_images.size(0)\n labels = torch.full((batch_size, 1), real_label, device=device)\n\n output = netD(real_images)\n errD_real = criterion(output, labels)\n errD_real.backward()\n D_x = output.mean().item()\n\n # train with fake\n noise = torch.randn(batch_size, nz, 1, 1, device=device)\n fake = netG(noise)\n labels.fill_(fake_label)\n output = netD(fake.detach())\n errD_fake = criterion(output, labels)\n errD_fake.backward()\n D_G_z1 = output.mean().item()\n errD = errD_real + errD_fake\n optimizerD.step()\n\n ############################\n # (2) Update G network: maximize log(D(G(z)))\n ###########################\n netG.zero_grad()\n labels.fill_(real_label) # fake labels are real for generator cost\n output = netD(fake)\n errG = criterion(output, labels)\n errG.backward()\n D_G_z2 = output.mean().item()\n optimizerG.step()\n \n # Save Losses for plotting later\n G_losses.append(errG.item())\n D_losses.append(errD.item())\n \n if (ii+1) % (len(train_loader)//2) == 0:\n print('[%d/%d][%d/%d] Loss_D: %.4f Loss_G: %.4f D(x): %.4f D(G(z)): %.4f / %.4f'\n % (epoch + 1, epochs, ii+1, len(train_loader),\n errD.item(), errG.item(), D_x, D_G_z1, D_G_z2))\n \n plot_loss (G_losses, D_losses, epoch)\n G_losses = []\n D_losses = []\n if epoch % 10 == 0:\n show_generated_img()\n\n epoch_time.append(time.time()- start)\n \n# valid_image = netG(fixed_noise)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"print (\">> average EPOCH duration = \", np.mean(epoch_time))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Generation example\n\n**WARNING,THIS CONTAINS IMAGES THAT MAY HURT THE SENSITIVITY OF SOME PEOPLE**"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"show_generated_img(7)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if not os.path.exists('../output_images'):\n os.mkdir('../output_images')\n \nim_batch_size = 50\nn_images=10000\n\nfor i_batch in tqdm(range(0, n_images, im_batch_size)):\n gen_z = torch.randn(im_batch_size, nz, 1, 1, device=device)\n gen_images = netG(gen_z)\n images = gen_images.to(\"cpu\").clone().detach()\n images = images.numpy().transpose(0, 2, 3, 1)\n for i_image in range(gen_images.size(0)):\n save_image(gen_images[i_image, :, :, :], os.path.join('../output_images', f'image_{i_batch+i_image:05d}.png'))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"fig = plt.figure(figsize=(25, 16))\n# display 10 images from each class\nfor i, j in enumerate(images[:32]):\n ax = fig.add_subplot(4, 8, i + 1, xticks=[], yticks=[])\n plt.imshow(j)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"import shutil\nshutil.make_archive('images', 'zip', '../output_images')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Save models"},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.save(netG.state_dict(), 'generator.pth')\ntorch.save(netD.state_dict(), 'discriminator.pth')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# MiFID metric"},{"metadata":{},"cell_type":"markdown","source":"Base code from [Demo MiFID metric for Dog image generation comp](https://www.kaggle.com/wendykan/demo-mifid-metric-for-dog-image-generation-comp)"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from __future__ import absolute_import, division, print_function\nimport numpy as np\nimport os\nimport gzip, pickle\nimport tensorflow as tf\nfrom scipy import linalg\nimport pathlib\nimport urllib\nimport warnings\nfrom PIL import Image\n\nclass KernelEvalException(Exception):\n pass\n\nmodel_params = {\n 'Inception': {\n 'name': 'Inception', \n 'imsize': 64,\n 'output_layer': 'Pretrained_Net/pool_3:0', \n 'input_layer': 'Pretrained_Net/ExpandDims:0',\n 'output_shape': 2048,\n 'cosine_distance_eps': 0.1\n }\n}\n\ndef create_model_graph(pth):\n \"\"\"Creates a graph from saved GraphDef file.\"\"\"\n # Creates graph from saved graph_def.pb.\n with tf.gfile.FastGFile( pth, 'rb') as f:\n graph_def = tf.GraphDef()\n graph_def.ParseFromString( f.read())\n _ = tf.import_graph_def( graph_def, name='Pretrained_Net')\n\ndef _get_model_layer(sess, model_name):\n # layername = 'Pretrained_Net/final_layer/Mean:0'\n layername = model_params[model_name]['output_layer']\n layer = sess.graph.get_tensor_by_name(layername)\n ops = layer.graph.get_operations()\n for op_idx, op in enumerate(ops):\n for o in op.outputs:\n shape = o.get_shape()\n if shape._dims != []:\n shape = [s.value for s in shape]\n new_shape = []\n for j, s in enumerate(shape):\n if s == 1 and j == 0:\n new_shape.append(None)\n else:\n new_shape.append(s)\n o.__dict__['_shape_val'] = tf.TensorShape(new_shape)\n return layer\n\ndef get_activations(images, sess, model_name, batch_size=50, verbose=False):\n \"\"\"Calculates the activations of the pool_3 layer for all images.\n\n Params:\n -- images : Numpy array of dimension (n_images, hi, wi, 3). The values\n must lie between 0 and 256.\n -- sess : current session\n -- batch_size : the images numpy array is split into batches with batch size\n batch_size. A reasonable batch size depends on the disposable hardware.\n -- verbose : If set to True and parameter out_step is given, the number of calculated\n batches is reported.\n Returns:\n -- A numpy array of dimension (num images, 2048) that contains the\n activations of the given tensor when feeding inception with the query tensor.\n \"\"\"\n inception_layer = _get_model_layer(sess, model_name)\n n_images = images.shape[0]\n if batch_size > n_images:\n print(\"warning: batch size is bigger than the data size. setting batch size to data size\")\n batch_size = n_images\n n_batches = n_images//batch_size + 1\n pred_arr = np.empty((n_images,model_params[model_name]['output_shape']))\n for i in tqdm(range(n_batches)):\n if verbose:\n print(\"\\rPropagating batch %d/%d\" % (i+1, n_batches), end=\"\", flush=True)\n start = i*batch_size\n if start+batch_size < n_images:\n end = start+batch_size\n else:\n end = n_images\n \n batch = images[start:end]\n pred = sess.run(inception_layer, {model_params[model_name]['input_layer']: batch})\n pred_arr[start:end] = pred.reshape(-1,model_params[model_name]['output_shape'])\n if verbose:\n print(\" done\")\n return pred_arr\n\n\n# def calculate_memorization_distance(features1, features2):\n# neigh = NearestNeighbors(n_neighbors=1, algorithm='kd_tree', metric='euclidean')\n# neigh.fit(features2) \n# d, _ = neigh.kneighbors(features1, return_distance=True)\n# print('d.shape=',d.shape)\n# return np.mean(d)\n\ndef normalize_rows(x: np.ndarray):\n \"\"\"\n function that normalizes each row of the matrix x to have unit length.\n\n Args:\n ``x``: A numpy matrix of shape (n, m)\n\n Returns:\n ``x``: The normalized (by row) numpy matrix.\n \"\"\"\n return np.nan_to_num(x/np.linalg.norm(x, ord=2, axis=1, keepdims=True))\n\n\ndef cosine_distance(features1, features2):\n # print('rows of zeros in features1 = ',sum(np.sum(features1, axis=1) == 0))\n # print('rows of zeros in features2 = ',sum(np.sum(features2, axis=1) == 0))\n features1_nozero = features1[np.sum(features1, axis=1) != 0]\n features2_nozero = features2[np.sum(features2, axis=1) != 0]\n norm_f1 = normalize_rows(features1_nozero)\n norm_f2 = normalize_rows(features2_nozero)\n\n d = 1.0-np.abs(np.matmul(norm_f1, norm_f2.T))\n print('d.shape=',d.shape)\n print('np.min(d, axis=1).shape=',np.min(d, axis=1).shape)\n mean_min_d = np.mean(np.min(d, axis=1))\n print('distance=',mean_min_d)\n return mean_min_d\n\n\ndef distance_thresholding(d, eps):\n if d < eps:\n return d\n else:\n return 1\n\ndef calculate_frechet_distance(mu1, sigma1, mu2, sigma2, eps=1e-6):\n \"\"\"Numpy implementation of the Frechet Distance.\n The Frechet distance between two multivariate Gaussians X_1 ~ N(mu_1, C_1)\n and X_2 ~ N(mu_2, C_2) is\n d^2 = ||mu_1 - mu_2||^2 + Tr(C_1 + C_2 - 2*sqrt(C_1*C_2)).\n \n Stable version by Dougal J. Sutherland.\n\n Params:\n -- mu1 : Numpy array containing the activations of the pool_3 layer of the\n inception net ( like returned by the function 'get_predictions')\n for generated samples.\n -- mu2 : The sample mean over activations of the pool_3 layer, precalcualted\n on an representive data set.\n -- sigma1: The covariance matrix over activations of the pool_3 layer for\n generated samples.\n -- sigma2: The covariance matrix over activations of the pool_3 layer,\n precalcualted on an representive data set.\n\n Returns:\n -- : The Frechet Distance.\n \"\"\"\n\n mu1 = np.atleast_1d(mu1)\n mu2 = np.atleast_1d(mu2)\n\n sigma1 = np.atleast_2d(sigma1)\n sigma2 = np.atleast_2d(sigma2)\n\n assert mu1.shape == mu2.shape, \"Training and test mean vectors have different lengths\"\n assert sigma1.shape == sigma2.shape, \"Training and test covariances have different dimensions\"\n\n diff = mu1 - mu2\n\n # product might be almost singular\n covmean, _ = linalg.sqrtm(sigma1.dot(sigma2), disp=False)\n if not np.isfinite(covmean).all():\n msg = \"fid calculation produces singular product; adding %s to diagonal of cov estimates\" % eps\n warnings.warn(msg)\n offset = np.eye(sigma1.shape[0]) * eps\n # covmean = linalg.sqrtm((sigma1 + offset).dot(sigma2 + offset))\n covmean = linalg.sqrtm((sigma1 + offset).dot(sigma2 + offset))\n \n # numerical error might give slight imaginary component\n if np.iscomplexobj(covmean):\n if not np.allclose(np.diagonal(covmean).imag, 0, atol=1e-3):\n m = np.max(np.abs(covmean.imag))\n raise ValueError(\"Imaginary component {}\".format(m))\n covmean = covmean.real\n\n # covmean = tf.linalg.sqrtm(tf.linalg.matmul(sigma1,sigma2))\n\n print('covmean.shape=',covmean.shape)\n # tr_covmean = tf.linalg.trace(covmean)\n\n tr_covmean = np.trace(covmean)\n return diff.dot(diff) + np.trace(sigma1) + np.trace(sigma2) - 2 * tr_covmean\n # return diff.dot(diff) + tf.linalg.trace(sigma1) + tf.linalg.trace(sigma2) - 2 * tr_covmean\n#-------------------------------------------------------------------------------\n\n\ndef calculate_activation_statistics(images, sess, model_name, batch_size=50, verbose=False):\n \"\"\"Calculation of the statistics used by the FID.\n Params:\n -- images : Numpy array of dimension (n_images, hi, wi, 3). The values\n must lie between 0 and 255.\n -- sess : current session\n -- batch_size : the images numpy array is split into batches with batch size\n batch_size. A reasonable batch size depends on the available hardware.\n -- verbose : If set to True and parameter out_step is given, the number of calculated\n batches is reported.\n Returns:\n -- mu : The mean over samples of the activations of the pool_3 layer of\n the incption model.\n -- sigma : The covariance matrix of the activations of the pool_3 layer of\n the incption model.\n \"\"\"\n act = get_activations(images, sess, model_name, batch_size, verbose)\n mu = np.mean(act, axis=0)\n sigma = np.cov(act, rowvar=False)\n return mu, sigma, act\n \ndef _handle_path_memorization(path, sess, model_name, is_checksize, is_check_png):\n path = pathlib.Path(path)\n files = list(path.glob('*.jpg')) + list(path.glob('*.png'))\n imsize = model_params[model_name]['imsize']\n\n # In production we don't resize input images. This is just for demo purpose. \n x = np.array([np.array(img_read_checks(fn, imsize, is_checksize, imsize, is_check_png)) for fn in files])\n m, s, features = calculate_activation_statistics(x, sess, model_name)\n del x #clean up memory\n return m, s, features\n\n# check for image size\ndef img_read_checks(filename, resize_to, is_checksize=False, check_imsize = 64, is_check_png = False):\n im = Image.open(str(filename))\n if is_checksize and im.size != (check_imsize,check_imsize):\n raise KernelEvalException('The images are not of size '+str(check_imsize))\n \n if is_check_png and im.format != 'PNG':\n raise KernelEvalException('Only PNG images should be submitted.')\n\n if resize_to is None:\n return im\n else:\n return im.resize((resize_to,resize_to),Image.ANTIALIAS)\n\ndef calculate_kid_given_paths(paths, model_name, model_path, feature_path=None):\n ''' Calculates the KID of two paths. '''\n tf.reset_default_graph()\n create_model_graph(str(model_path))\n with tf.Session() as sess:\n sess.run(tf.global_variables_initializer())\n m1, s1, features1 = _handle_path_memorization(paths[0], sess, model_name, is_checksize = True, is_check_png = True)\n if feature_path is None:\n m2, s2, features2 = _handle_path_memorization(paths[1], sess, model_name, is_checksize = False, is_check_png = False)\n else:\n with np.load(feature_path) as f:\n m2, s2, features2 = f['m'], f['s'], f['features']\n\n print('m1,m2 shape=',(m1.shape,m2.shape),'s1,s2=',(s1.shape,s2.shape))\n print('starting calculating FID')\n fid_value = calculate_frechet_distance(m1, s1, m2, s2)\n print('done with FID, starting distance calculation')\n distance = cosine_distance(features1, features2) \n return fid_value, distance","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"!ls ../output_images","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**I have to fix this :(**"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"user_images_unzipped_path = '../output_images'\nimages_path = [user_images_unzipped_path,'../all-dogs/all-dogs/']\n\nmodel_path = '../input/dog-face-generation-competition-kid-metric-input/classify_image_graph_def.pb'\n\nfid_epsilon = 10e-15\n\nfid_value_public, distance_public = calculate_kid_given_paths(images_path, 'Inception', model_path)\ndistance_public = distance_thresholding(distance_public, model_params['Inception']['cosine_distance_eps'])\nprint(\"FID_public: \", fid_value_public, \"distance_public: \", distance_public, \"multiplied_public: \", fid_value_public /(distance_public + fid_epsilon))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# References\n\n- [DCGAN baseline](https://www.kaggle.com/artgor/dcgan-baseline) by Andrew\n- [RaLSGAN dogs](https://www.kaggle.com/speedwagon/ralsgan-dogs) by Vlad\n- [GAN dogs starter](https://www.kaggle.com/wendykan/gan-dogs-starter)\n- [GitHub Achronus generative_adversarial_networks](https://github.com/Achronus/Machine-Learning-101/blob/master/coding_templates_and_data_files/computer_vision/2.%20generative_adversarial_networks.py)\n- [It’s Training Cats and Dogs: NVIDIA Research Uses AI to Turn Cats Into Dogs, Lions and Tigers, Too](https://blogs.nvidia.com/blog/2018/04/15/nvidia-research-image-translation/)\n- [A Beginner's Guide to Generative Adversarial Networks (GANs)](https://skymind.ai/wiki/generative-adversarial-network-gan)\n\n
\n
\n\n----\n\n## I think I'll keep updating this, I like this comp and GANs :D\n### btw, If you want to create a x5 team, I'm in :p\n\n\n\n"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}