{"cells":[{"metadata":{},"cell_type":"markdown","source":"# T3h d0g mak3rz GAN \n\n##### Authors: Shay Guterman, Yogev Heskia and Eviatar Levy\n\n________________________________\nThis is not our submission, but rather the model we used to evaluate our GAN and analyse it, best parameters and approaches were chosen and submitted. Also, our best score was calculated after the 187 epochs.\n\n### What worked for us?\n\n**Making the network larger**  Got us from 60 to ~55. The Architectures that works for us has ~18 million parameters for the generator and  ~17 million for the discriminator.\n<brr>**Adding FID calculations to the training**  We noticed  large fluctuations in FID between following epochs, probably due to the unstable nature of GANS and the use of cyclic learning rate so knowing when to stop your GAN is very important. In addition, we are confined to 3 submission a day and it is critical to be able to calculate the FID through your training to get better understanding of the network behavior .\n<brr>**Voodo** We don't know why, but using model.train() before the learning module reduced some of the fluctuations in the training process and improved our score. \n<br> Implementation of those 2 steps took us from ~55 to ~50.\n<brr>**Weight initialization** At first we implemented weight initialization with nn.init.normal_(m.weight.data, 0.0, 0.02) , the results were quite similar to our results without it. When we changed it to nn.init.normal_(m.weight.data, 0.0, 0.15-1) we improve our score to ~46.\n<brr> **Other parameters and observations** Try different seeds, some work better than others. Higher beta gets you smaller fluctuations in FID score, however gets slightly worse results.   \n<brr>\n**Other things we implemented** ColorJitter augmentation, small dropout.\n\n\n### What didn't work for us?\n\n**Post selection** We've tried to use the discriminator to improve our submission. Meaning, to generate 10,000 images that the discriminator labels them close to real images of dogs. I think this approach is valid, however I think we didn't understand the loss function good enough, and didnt have enough time to explore the idea\n<brr>\nAdding convolution layers, random search over many parameters,Larger network, CGANS (altough we really didn't try very hard).\n\n\n\n___________________\nOf course Nirjhar Roy deserves A lot of Credit: https://www.kaggle.com/phoenix9032/gan-dogs-starter-24-jul-custom-layers\nIf you upvote this kernel and haven't upvoted his, please do. \n\n\n"},{"metadata":{},"cell_type":"markdown","source":"## content Table and kernel flow\n\n1. ** Imports ** <br>\n2. ** Conf**<br>\n3. ** Pytorch initializations**<br>\n4. **Data generator**<br>\n5. **Defining Neural Nets**<br>\n> **5.1 Generator**<br>\n> **5.2 Discriminator**<br>\n> **5.3 Weight initialization**<br>\n6. **Functions **<br>\n> **6.1 Image show functions**<br>\n> **6.2 Truncate function**<br>\n7. **Calc FID**\n8. **Train Module**<br>\n9. **Define and train the GAN**<br>\n10. **Plot FID**<br>\n"},{"metadata":{},"cell_type":"markdown","source":"## 1. Imports \n"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import shutil\nimport os\nimport random\nfrom time import time\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport torch\nfrom torch.nn.utils import spectral_norm\nfrom torch.utils.data import Dataset\nimport torchvision\nimport xml.etree.ElementTree as ET\nfrom tqdm import tqdm_notebook as tqdm\nfrom scipy.stats import truncnorm\n\n\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2. Conf"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define RGB images\nCHANNEL = {'RGB': 3, 'BW': 1}\nIMG_CHANNEL = CHANNEL['RGB']\n\n# Defining path to dog images \n# Yogev, maybe add the annotions also the same way?\nDATA_LOCATION = '../input/generative-dog-images/all-dogs/all-dogs/'\n\n# Define batch size\nBATCH_SIZE = 32\n\n# Each image in the generator starts from LATENT_DIM points generated from a some noise distribution \nLATENT_DIM = 128\n\n# The amount of parameters in the networks scales with those number\nCONV_DEPTHS_G = 100\nCONV_DEPTHS_D = 160\n\n\n# Learning rate functions\n# The learning scheduale \nLEARNING_RATE_G = 0.00025 # was 0.0003 \nLEARNING_RATE_D = 0.00015 # was 0.0001\nBETA_1 = 0.503\n\nT0_interval = 100\nETA_MIN = 0.00003\n\n# Amount of epochs, EPOCHS/T0_interval has to be integer\n\nEPOCHS = 300\n\n# Start submitting from epoch\nSUBMIT_START = 45\n# Submit every # epochs\n# SUBMIT_INTERVAL = T0_interval/2\nSUBMIT_INTERVAL = 10\n\n# Define limit for training time\nTIME_FOR_TRAIN = 31000 # 8.3 hours (in seconds)\n# TIME_FOR_TRAIN = 7 * 60 * 60 # hours\n# TIME_FOR_TRAIN = 5 * 60\n\n\nreal_label = 0.6\nfake_label = 0.0","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### LR scheduale - cosine annealing \n\nRelevant paper: https://arxiv.org/pdf/1608.03983.pdf <br>\nAlso explained in: https://sidravi1.github.io/blog/2018/04/25/playing-with-sgdr\n<brr>\n<br>\n\\begin{align}\n\\eta_t = \\eta_{min}^i + \\frac{1}{2}(\\eta_{max}^i - \\eta_{min}^i)(1 + \\cos(\\frac{T_{cur}}{T_i} \\pi))\n\\end{align}\n\n![](https://sidravi1.github.io/assets/2014_04_25_sgdr_schedule.png)"},{"metadata":{},"cell_type":"markdown","source":"## 3. Pytorch initializations"},{"metadata":{"trusted":true},"cell_type":"code","source":"def seed_everything(seed=666):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n    \nseed_everything()\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4. Data generator\n\nAlongside data augmentation"},{"metadata":{"trusted":true},"cell_type":"code","source":"class DataGenerator(Dataset):\n    def __init__(self, directory, transform=None, n_samples=np.inf):\n        self.directory = directory\n        self.transform = transform\n        self.n_samples = n_samples\n        self.samples = self._load_subfolders_images(directory)\n        if len(self.samples) == 0:\n            raise RuntimeError(\"Found 0 files in subfolders of: {}\".format(directory))\n\n    def _load_subfolders_images(self, root):\n        IMG_EXTENSIONS = (\n            '.jpg', '.jpeg', '.png', '.ppm', '.bmp', '.pgm', '.tif', '.tiff', '.webp')\n\n        def is_valid_file(x):\n            return torchvision.datasets.folder.has_file_allowed_extension(x, IMG_EXTENSIONS)\n\n        \n        \n        required_transforms = torchvision.transforms.Compose([\n            torchvision.transforms.Resize(64),\n            torchvision.transforms.CenterCrop(64),\n        ])\n        imgs = []\n        paths = []\n        for root, _, fnames in sorted(os.walk(root)):\n            for fname in sorted(fnames)[:min(self.n_samples, 999999999999999)]:\n                path = os.path.join(root, fname)\n                paths.append(path)\n\n        for path in paths:\n            if is_valid_file(path):\n                # Load image\n                img = torchvision.datasets.folder.default_loader(path)\n\n                # Get bounding boxes\n                annotation_basename = os.path.splitext(os.path.basename(path))[0]\n                annotation_dirname = next(\n                    dirname for dirname in os.listdir('../input/generative-dog-images/annotation/Annotation/') if\n                    dirname.startswith(annotation_basename.split('_')[0]))\n                annotation_filename = os.path.join('../input/generative-dog-images/annotation/Annotation/',\n                                                   annotation_dirname, annotation_basename)\n                tree = ET.parse(annotation_filename)\n                root = tree.getroot()\n                objects = root.findall('object')\n                for o in objects:\n                    bndbox = o.find('bndbox')\n                    xmin = int(bndbox.find('xmin').text)\n                    ymin = int(bndbox.find('ymin').text)\n                    xmax = int(bndbox.find('xmax').text)\n                    ymax = int(bndbox.find('ymax').text)\n\n                    w = np.min((xmax - xmin, ymax - ymin))\n                    bbox = (xmin, ymin, xmin + w, ymin + w)\n                    object_img = required_transforms(img.crop(bbox))\n                    # object_img = object_img.resize((64,64), Image.ANTIALIAS)\n                    imgs.append(object_img)\n        return imgs\n\n    def __getitem__(self, index):\n        sample = self.samples[index]\n\n        if self.transform is not None:\n            sample = self.transform(sample)\n\n        return np.asarray(sample)\n\n    def __len__(self):\n        return len(self.samples)\n    \n# torchvision.transforms.ColorJitter(brightness=0, contrast=0, saturation=0, hue=0)\ntransform = torchvision.transforms.Compose([torchvision.transforms.RandomHorizontalFlip(p=0.2),\n                                            torchvision.transforms.RandomAffine(3, translate=None, scale=None, shear=None, resample=False, fillcolor=0),    \n                                            torchvision.transforms.ColorJitter(brightness=0.04, contrast=0.03, saturation=0, hue=0),\n                                            torchvision.transforms.ToTensor(),\n                                            torchvision.transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])\ndata_gen = DataGenerator(DATA_LOCATION, transform=transform, n_samples=85000)\ntrain_loader = torch.utils.data.DataLoader(data_gen, shuffle=True, batch_size=BATCH_SIZE, num_workers=4)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5. Defining Neural Nets"},{"metadata":{},"cell_type":"markdown","source":"### 5.1 Generator"},{"metadata":{"trusted":true},"cell_type":"code","source":"class PixelwiseNorm(torch.nn.Module):\n    def __init__(self):\n        super(PixelwiseNorm, self).__init__()\n\n    def forward(self, x, alpha=1e-8):\n        \"\"\"\n        forward pass of the module\n        :param x: input activations volume\n        :param alpha: small number for numerical stability\n        :return: y => pixel normalized activations\n        \"\"\"\n        y = x.pow(2.).mean(dim=1, keepdim=True).add(alpha).sqrt()  # [N1HW]\n        y = x / y  # normalize the input x volume\n        return y\n    \n    \nclass Generator(torch.nn.Module):\n    def __init__(self, latent_dim, nfeats, nchannels):  #was nz\n        super(Generator, self).__init__()\n\n        # input is Z, going into a convolution\n        self.conv1 = spectral_norm(torch.nn.ConvTranspose2d(latent_dim, nfeats * 8, 4, 1, 0, bias=False)) #was nz\n\n        self.conv2 = spectral_norm(torch.nn.ConvTranspose2d(nfeats * 8, nfeats * 8, 4, 2, 1, bias=False))\n\n        self.conv3 = spectral_norm(torch.nn.ConvTranspose2d(nfeats * 8, nfeats * 4, 4, 2, 1, bias=False))\n\n        self.conv4 = spectral_norm(torch.nn.ConvTranspose2d(nfeats * 4, nfeats * 2, 4, 2, 1, bias=False))\n\n        self.conv5 = spectral_norm(torch.nn.ConvTranspose2d(nfeats * 2, nfeats, 4, 2, 1, bias=False))\n\n        self.conv6 = spectral_norm(torch.nn.ConvTranspose2d(nfeats, nchannels, 3, 1, 1, bias=False))\n        self.pixnorm = PixelwiseNorm()\n\n    def forward(self, x):\n\n        x = torch.nn.functional.leaky_relu(self.conv1(x))\n        x = torch.nn.Dropout(0.03)(x)\n        x = torch.nn.functional.leaky_relu(self.conv2(x))\n        x = self.pixnorm(x)\n        x = torch.nn.functional.leaky_relu(self.conv3(x))\n        x = torch.nn.Dropout(0.03)(x)\n        x = self.pixnorm(x)        \n        x = torch.nn.functional.leaky_relu(self.conv4(x))\n        x = torch.nn.Dropout(0.03)(x)\n        x = self.pixnorm(x)\n        x = torch.nn.functional.leaky_relu(self.conv5(x))\n        x = self.pixnorm(x)\n        x = torch.tanh(self.conv6(x))\n\n        return x","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 5.2 Discriminator"},{"metadata":{"trusted":true},"cell_type":"code","source":"class MinibatchStdDev(torch.nn.Module):\n    \"\"\"\n    Minibatch standard deviation layer for the discriminator\n    \"\"\"\n\n    def __init__(self):\n        \"\"\"\n        derived class constructor\n        \"\"\"\n        super(MinibatchStdDev, self).__init__()\n\n    def forward(self, x, alpha=1e-8):\n        \"\"\"\n        forward pass of the layer\n        :param x: input activation volume\n        :param alpha: small number for numerical stability\n        :return: y => x appended with standard deviation constant map\n        \"\"\"\n        batch_size, _, height, width = x.shape\n        # [B x C x H x W] Subtract mean over batch.\n        y = x - x.mean(dim=0, keepdim=True)\n        # [1 x C x H x W]  Calc standard deviation over batch\n        y = torch.sqrt(y.pow(2.).mean(dim=0, keepdim=False) + alpha)\n\n        # [1]  Take average over feature_maps and pixels.\n        y = y.mean().view(1, 1, 1, 1)\n\n        # [B x 1 x H x W]  Replicate over group and pixels.\n        y = y.repeat(batch_size, 1, height, width)\n\n        # [B x C x H x W]  Append as new feature_map.\n        y = torch.cat([x, y], 1)\n        # return the computed values:\n        return y","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class Discriminator(torch.nn.Module):\n    def __init__(self, nchannels, nfeats):\n        super(Discriminator, self).__init__()\n\n        # input is (nchannels) x 64 x 64\n        self.conv1 = torch.nn.Conv2d(nchannels, nfeats, 4, 2, 1, bias=False)\n        # state size. (nfeats) x 32 x 32\n\n        self.conv2 = spectral_norm(torch.nn.Conv2d(nfeats, nfeats * 2, 4, 2, 1, bias=False))\n        self.bn2 = torch.nn.BatchNorm2d(nfeats * 2)\n        # state size. (nfeats*2) x 16 x 16\n\n        self.conv3 = spectral_norm(torch.nn.Conv2d(nfeats * 2, nfeats * 4, 4, 2, 1, bias=False))\n        self.bn3 = torch.nn.BatchNorm2d(nfeats * 4)\n        # state size. (nfeats*4) x 8 x 8\n\n        self.conv4 = spectral_norm(torch.nn.Conv2d(nfeats * 4, nfeats * 8, 4, 2, 1, bias=False))\n        self.bn4 = torch.nn.MaxPool2d(2)\n        # state size. (nfeats*8) x 4 x 4\n        self.batch_discriminator = MinibatchStdDev()\n\n        self.conv5 = spectral_norm(torch.nn.Conv2d(nfeats * 8 + 1, 1, 2, 1, 0, bias=False))\n        # state size. 1 x 1 x 1\n\n    def forward(self, x):\n        x = torch.nn.functional.leaky_relu(self.conv1(x), 0.1)\n        x = torch.nn.functional.leaky_relu(self.bn2(self.conv2(x)), 0.1)\n        x = torch.nn.functional.leaky_relu(self.bn3(self.conv3(x)), 0.1)\n        x = torch.nn.functional.leaky_relu(self.bn4(self.conv4(x)), 0.1)\n        x = self.batch_discriminator(x)\n        x = torch.sigmoid(self.conv5(x))\n        # x= self.conv5(x)\n        return x.view(-1, 1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 5.3 Weight initialization\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"def weights_init(m):\n    classname = m.__class__.__name__\n    if classname.find('Conv') != -1:\n        nn.init.normal_(m.weight.data, 0.0, 1.0)        \n    elif classname.find('BatchNorm') != -1:\n        nn.init.normal_(m.weight.data, 1.0, 0.02)\n        nn.init.constant_(m.bias.data, 0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6. Functions "},{"metadata":{},"cell_type":"markdown","source":"### 6.1 Image show functions"},{"metadata":{"trusted":true},"cell_type":"code","source":"def show_generated_img_all():\n    gen_z = torch.randn(32, LATENT_DIM, 1, 1, device=device)\n    gen_images = netG(gen_z).to(\"cpu\").clone().detach()\n    gen_images = gen_images.numpy().transpose(0, 2, 3, 1)\n    gen_images = (gen_images + 1.0) / 2.0\n    fig = plt.figure(figsize=(25, 16))\n    for ii, img in enumerate(gen_images):\n        ax = fig.add_subplot(4, 8, ii + 1, xticks=[], yticks=[])\n        plt.imshow(img)\n    # plt.savefig(filename)\n\n\ndef show_generated_img():\n    row_num = 1\n    col_num = 10 \n    gen_z = torch.randn(row_num * col_num , LATENT_DIM, 1, 1, device=device)\n    gen_images = netG(gen_z).to(\"cpu\").clone().detach()\n    gen_images = gen_images.numpy().transpose(0, 2, 3, 1)\n    gen_images = (gen_images + 1.0) / 2.0\n    fig = plt.figure(figsize=(20, 4))\n    for ii, img in enumerate(gen_images):\n        ax = fig.add_subplot(row_num, col_num, ii + 1, xticks=[], yticks=[])\n        plt.imshow(img)\n    plt.show()\n    # plt.savefig(filename)\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 6.2 Truncate function"},{"metadata":{"trusted":true},"cell_type":"code","source":"def truncated_normal(size, threshold=1):\n    values = truncnorm.rvs(-threshold, threshold, size=size)\n    return values","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 7. Calc FID\n\nCalc FID functions"},{"metadata":{"trusted":true},"cell_type":"code","source":"from __future__ import absolute_import, division, print_function\nimport numpy as np\nimport os\nimport gzip, pickle\nimport tensorflow as tf\nfrom scipy import linalg\nimport pathlib\nimport urllib\nimport warnings\nfrom tqdm import tqdm\nfrom PIL import Image\n\nclass KernelEvalException(Exception):\n    pass\n\nmodel_params = {\n    'Inception': {\n        'name': 'Inception', \n        'imsize': 64,\n        'output_layer': 'Pretrained_Net/pool_3:0', \n        'input_layer': 'Pretrained_Net/ExpandDims:0',\n        'output_shape': 2048,\n        'cosine_distance_eps': 0.1\n        }\n}\n\ndef create_model_graph(pth):\n    \"\"\"Creates a graph from saved GraphDef file.\"\"\"\n    # Creates graph from saved graph_def.pb.\n    with tf.gfile.FastGFile( pth, 'rb') as f:\n        graph_def = tf.GraphDef()\n        graph_def.ParseFromString( f.read())\n        _ = tf.import_graph_def( graph_def, name='Pretrained_Net')\n\ndef _get_model_layer(sess, model_name):\n    # layername = 'Pretrained_Net/final_layer/Mean:0'\n    layername = model_params[model_name]['output_layer']\n    layer = sess.graph.get_tensor_by_name(layername)\n    ops = layer.graph.get_operations()\n    for op_idx, op in enumerate(ops):\n        for o in op.outputs:\n            shape = o.get_shape()\n            if shape._dims != []:\n              shape = [s.value for s in shape]\n              new_shape = []\n              for j, s in enumerate(shape):\n                if s == 1 and j == 0:\n                  new_shape.append(None)\n                else:\n                  new_shape.append(s)\n              o.__dict__['_shape_val'] = tf.TensorShape(new_shape)\n    return layer\n\ndef get_activations(images, sess, model_name, batch_size=50, verbose=False):\n    \"\"\"Calculates the activations of the pool_3 layer for all images.\n\n    Params:\n    -- images      : Numpy array of dimension (n_images, hi, wi, 3). The values\n                     must lie between 0 and 256.\n    -- sess        : current session\n    -- batch_size  : the images numpy array is split into batches with batch size\n                     batch_size. A reasonable batch size depends on the disposable hardware.\n    -- verbose    : If set to True and parameter out_step is given, the number of calculated\n                     batches is reported.\n    Returns:\n    -- A numpy array of dimension (num images, 2048) that contains the\n       activations of the given tensor when feeding inception with the query tensor.\n    \"\"\"\n    inception_layer = _get_model_layer(sess, model_name)\n    n_images = images.shape[0]\n    if batch_size > n_images:\n        print(\"warning: batch size is bigger than the data size. setting batch size to data size\")\n        batch_size = n_images\n    n_batches = n_images//batch_size + 1\n    pred_arr = np.empty((n_images,model_params[model_name]['output_shape']))\n    for i in range(n_batches):\n        if verbose:\n            print(\"\\rPropagating batch %d/%d\" % (i+1, n_batches), end=\"\", flush=True)\n        start = i*batch_size\n        if start+batch_size < n_images:\n            end = start+batch_size\n        else:\n            end = n_images\n                    \n        batch = images[start:end]\n        pred = sess.run(inception_layer, {model_params[model_name]['input_layer']: batch})\n        pred_arr[start:end] = pred.reshape(-1,model_params[model_name]['output_shape'])\n    if verbose:\n        print(\" done\")\n    return pred_arr\n\n\n# def calculate_memorization_distance(features1, features2):\n#     neigh = NearestNeighbors(n_neighbors=1, algorithm='kd_tree', metric='euclidean')\n#     neigh.fit(features2) \n#     d, _ = neigh.kneighbors(features1, return_distance=True)\n#     print('d.shape=',d.shape)\n#     return np.mean(d)\n\ndef normalize_rows(x: np.ndarray):\n    \"\"\"\n    function that normalizes each row of the matrix x to have unit length.\n\n    Args:\n     ``x``: A numpy matrix of shape (n, m)\n\n    Returns:\n     ``x``: The normalized (by row) numpy matrix.\n    \"\"\"\n    return np.nan_to_num(x/np.linalg.norm(x, ord=2, axis=1, keepdims=True))\n\n\ndef cosine_distance(features1, features2):\n    # print('rows of zeros in features1 = ',sum(np.sum(features1, axis=1) == 0))\n    # print('rows of zeros in features2 = ',sum(np.sum(features2, axis=1) == 0))\n    features1_nozero = features1[np.sum(features1, axis=1) != 0]\n    features2_nozero = features2[np.sum(features2, axis=1) != 0]\n    norm_f1 = normalize_rows(features1_nozero)\n    norm_f2 = normalize_rows(features2_nozero)\n\n    d = 1.0-np.abs(np.matmul(norm_f1, norm_f2.T))\n    print('d.shape=',d.shape)\n    print('np.min(d, axis=1).shape=',np.min(d, axis=1).shape)\n    mean_min_d = np.mean(np.min(d, axis=1))\n    print('distance=',mean_min_d)\n    return mean_min_d\n\n\ndef distance_thresholding(d, eps):\n    if d < eps:\n        return d\n    else:\n        return 1\n\ndef calculate_frechet_distance(mu1, sigma1, mu2, sigma2, eps=1e-6):\n    \"\"\"Numpy implementation of the Frechet Distance.\n    The Frechet distance between two multivariate Gaussians X_1 ~ N(mu_1, C_1)\n    and X_2 ~ N(mu_2, C_2) is\n            d^2 = ||mu_1 - mu_2||^2 + Tr(C_1 + C_2 - 2*sqrt(C_1*C_2)).\n            \n    Stable version by Dougal J. Sutherland.\n\n    Params:\n    -- mu1 : Numpy array containing the activations of the pool_3 layer of the\n             inception net ( like returned by the function 'get_predictions')\n             for generated samples.\n    -- mu2   : The sample mean over activations of the pool_3 layer, precalcualted\n               on an representive data set.\n    -- sigma1: The covariance matrix over activations of the pool_3 layer for\n               generated samples.\n    -- sigma2: The covariance matrix over activations of the pool_3 layer,\n               precalcualted on an representive data set.\n\n    Returns:\n    --   : The Frechet Distance.\n    \"\"\"\n\n    mu1 = np.atleast_1d(mu1)\n    mu2 = np.atleast_1d(mu2)\n\n    sigma1 = np.atleast_2d(sigma1)\n    sigma2 = np.atleast_2d(sigma2)\n\n    assert mu1.shape == mu2.shape, \"Training and test mean vectors have different lengths\"\n    assert sigma1.shape == sigma2.shape, \"Training and test covariances have different dimensions\"\n\n    diff = mu1 - mu2\n\n    # product might be almost singular\n    covmean, _ = linalg.sqrtm(sigma1.dot(sigma2), disp=False)\n    if not np.isfinite(covmean).all():\n        msg = \"fid calculation produces singular product; adding %s to diagonal of cov estimates\" % eps\n        warnings.warn(msg)\n        offset = np.eye(sigma1.shape[0]) * eps\n        # covmean = linalg.sqrtm((sigma1 + offset).dot(sigma2 + offset))\n        covmean = linalg.sqrtm((sigma1 + offset).dot(sigma2 + offset))\n    \n    # numerical error might give slight imaginary component\n    if np.iscomplexobj(covmean):\n        if not np.allclose(np.diagonal(covmean).imag, 0, atol=1e-3):\n            m = np.max(np.abs(covmean.imag))\n            raise ValueError(\"Imaginary component {}\".format(m))\n        covmean = covmean.real\n\n    # covmean = tf.linalg.sqrtm(tf.linalg.matmul(sigma1,sigma2))\n\n    print('covmean.shape=',covmean.shape)\n    # tr_covmean = tf.linalg.trace(covmean)\n\n    tr_covmean = np.trace(covmean)\n    return diff.dot(diff) + np.trace(sigma1) + np.trace(sigma2) - 2 * tr_covmean\n    # return diff.dot(diff) + tf.linalg.trace(sigma1) + tf.linalg.trace(sigma2) - 2 * tr_covmean\n#-------------------------------------------------------------------------------\n\n\ndef calculate_activation_statistics(images, sess, model_name, batch_size=50, verbose=False):\n    \"\"\"Calculation of the statistics used by the FID.\n    Params:\n    -- images      : Numpy array of dimension (n_images, hi, wi, 3). The values\n                     must lie between 0 and 255.\n    -- sess        : current session\n    -- batch_size  : the images numpy array is split into batches with batch size\n                     batch_size. A reasonable batch size depends on the available hardware.\n    -- verbose     : If set to True and parameter out_step is given, the number of calculated\n                     batches is reported.\n    Returns:\n    -- mu    : The mean over samples of the activations of the pool_3 layer of\n               the incption model.\n    -- sigma : The covariance matrix of the activations of the pool_3 layer of\n               the incption model.\n    \"\"\"\n    act = get_activations(images, sess, model_name, batch_size, verbose)\n    mu = np.mean(act, axis=0)\n    sigma = np.cov(act, rowvar=False)\n    return mu, sigma, act\n    \ndef _handle_path_memorization(path, sess, model_name, is_checksize, is_check_png):\n    path = pathlib.Path(path)\n    files = list(path.glob('*.jpg')) + list(path.glob('*.png'))\n    imsize = model_params[model_name]['imsize']\n\n    # In production we don't resize input images. This is just for demo purpose. \n    x = np.array([np.array(img_read_checks(fn, imsize, is_checksize, imsize, is_check_png)) for fn in files])\n    m, s, features = calculate_activation_statistics(x, sess, model_name)\n    del x #clean up memory\n    return m, s, features\n\n# check for image size\ndef img_read_checks(filename, resize_to, is_checksize=False, check_imsize = 64, is_check_png = False):\n    im = Image.open(str(filename))\n    if is_checksize and im.size != (check_imsize,check_imsize):\n        raise KernelEvalException('The images are not of size '+str(check_imsize))\n    \n    if is_check_png and im.format != 'PNG':\n        raise KernelEvalException('Only PNG images should be submitted.')\n\n    if resize_to is None:\n        return im\n    else:\n        return im.resize((resize_to,resize_to),Image.ANTIALIAS)\n\ndef calculate_kid_given_paths(paths, model_name, model_path, feature_path=None, mm=[], ss=[], ff=[]):\n    ''' Calculates the KID of two paths. '''\n    tf.reset_default_graph()\n    create_model_graph(str(model_path))\n    with tf.Session() as sess:\n        sess.run(tf.global_variables_initializer())\n        m1, s1, features1 = _handle_path_memorization(paths[0], sess, model_name, is_checksize = True, is_check_png = True)\n        if len(mm) != 0:\n            m2 = mm\n            s2 = ss\n            features2 = ff\n        elif feature_path is None:\n            m2, s2, features2 = _handle_path_memorization(paths[1], sess, model_name, is_checksize = False, is_check_png = False)\n        else:\n            with np.load(feature_path) as f:\n                m2, s2, features2 = f['m'], f['s'], f['features']\n\n        print('m1,m2 shape=',(m1.shape,m2.shape),'s1,s2=',(s1.shape,s2.shape))\n        print('starting calculating FID')\n        fid_value = calculate_frechet_distance(m1, s1, m2, s2)\n        print('done with FID, starting distance calculation')\n        distance = cosine_distance(features1, features2)        \n        return fid_value, distance, m2, s2, features2\n    \n\n    \n    \nimport zipfile\nComputeLB = False\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 8. Train Module"},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_module(epochs):\n    FID_list = []\n    epoch_list = []\n    step = 0\n    start = time()\n    for epoch in range(epochs):\n        for ii, (real_images) in enumerate(train_loader):\n            end = time()\n            if (end - start) > TIME_FOR_TRAIN:\n                break\n            ############################\n            # (1) Update D network: maximize log(D(x)) + log(1 - D(G(z)))\n            ###########################\n            # train with real\n            netD.zero_grad()\n            real_images = real_images.to(device)\n            batch_size = real_images.size(0)\n            labels = torch.full((batch_size, 1), real_label, device=device) + np.random.uniform(-0.1, 0.1)\n\n            output = netD(real_images)\n            errD_real = criterion(output, labels)\n            errD_real.backward()\n            D_x = output.mean().item()\n\n            # train with fake\n            noise = torch.randn(batch_size, LATENT_DIM, 1, 1, device=device)\n            fake = netG(noise)\n            labels.fill_(fake_label) + np.random.uniform(0, 0.2)\n            output = netD(fake.detach())\n            errD_fake = criterion(output, labels)\n            errD_fake.backward()\n            D_G_z1 = output.mean().item()\n            errD = errD_real + errD_fake\n            optimizerD.step()\n\n            ############################\n            # (2) Update G network: maximize log(D(G(z)))\n            ###########################\n            netG.zero_grad()\n            labels.fill_(real_label)  # fake labels are real for generator cost\n            output = netD(fake)\n            errG = criterion(output, labels)\n            errG.backward()\n            D_G_z2 = output.mean().item()\n            optimizerG.step()\n\n            if step % 500 == 0:\n                print('[%d/%d][%d/%d] Loss_D: %.4f Loss_G: %.4f D(x): %.4f D(G(z)): %.4f / %.4f'\n                      % (epoch + 1, EPOCHS, ii, len(train_loader),\n                         errD.item(), errG.item(), D_x, D_G_z1, D_G_z2))\n\n                valid_image = netG(fixed_noise)\n            step += 1\n            lr_schedulerG.step(epoch)\n            lr_schedulerD.step(epoch)\n            \n        if epoch % 5 == 0:\n            show_generated_img()\n            print(end - start)\n            \n        if epoch>=SUBMIT_START and epoch%SUBMIT_INTERVAL==0:\n            netG.eval()\n            if not os.path.exists('../output_images'):\n                os.mkdir('../output_images')\n            im_batch_size = 50\n            n_images = 10000\n            for i_batch in range(0, n_images, im_batch_size):\n                z = truncated_normal((im_batch_size, LATENT_DIM, 1, 1), threshold=1)\n                gen_z = torch.from_numpy(z).float().to(device)\n                # gen_z = torch.randn(im_batch_size, 100, 1, 1, device=device)\n                gen_images = netG(gen_z)\n                images = gen_images.to(\"cpu\").clone().detach()\n                images = images.numpy().transpose(0, 2, 3, 1)\n                for i_image in range(gen_images.size(0)):\n                    torchvision.utils.save_image((gen_images[i_image, :, :, :] + 1.0) / 2.0,\n                                                 os.path.join('../output_images', f'image_{i_batch + i_image:05d}.png'))\n            shutil.make_archive('images', 'zip', '../output_images')\n            show_generated_img_all()\n            # UNCOMPRESS OUR IMGAES\n            with zipfile.ZipFile(\"../working/images.zip\",\"r\") as z:\n                z.extractall(\"../tmp/images2/\")\n\n            # COMPUTE LB SCORE\n            m2 = []; s2 =[]; f2 = []\n            user_images_unzipped_path = '../tmp/images2/'\n            images_path = [user_images_unzipped_path,'../input/generative-dog-images/all-dogs/all-dogs/']\n            public_path = '../input/dog-face-generation-competition-kid-metric-input/classify_image_graph_def.pb'\n            fid_epsilon = 10e-15\n            fid_value_public, distance_public, m2, s2, f2 = calculate_kid_given_paths(images_path, 'Inception', public_path, mm=m2, ss=s2, ff=f2)\n            distance_public = distance_thresholding(distance_public, model_params['Inception']['cosine_distance_eps'])\n            print(\"FID_public: \", fid_value_public, \"distance_public: \", distance_public, \"multiplied_public: \",\n                    fid_value_public /(distance_public + fid_epsilon))\n            FID_list.append(fid_value_public)\n            epoch_list.append(epoch)\n\n            netG.train()\n\n        if (end - start) > TIME_FOR_TRAIN:\n            break\n    return epoch_list,FID_list","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 9. Define and train the GAN"},{"metadata":{"trusted":true},"cell_type":"code","source":"netG = Generator(LATENT_DIM, CONV_DEPTHS_G, IMG_CHANNEL).to(device)\nnetD = Discriminator(IMG_CHANNEL, CONV_DEPTHS_D).to(device)   # was 3 here\n\nnetG.train()\nnetD.train()\n\n\nweights_init(netG)\nweights_init(netD)\n\nprint(\"Generator parameters:    \", sum(p.numel() for p in netG.parameters() if p.requires_grad))\nprint(\"Discriminator parameters:\", sum(p.numel() for p in netD.parameters() if p.requires_grad))\n\ncriterion = torch.nn.BCELoss()\noptimizerD = torch.optim.Adam(netD.parameters(), lr=LEARNING_RATE_G, betas=(BETA_1, 0.999))\noptimizerG = torch.optim.Adam(netG.parameters(), lr=LEARNING_RATE_D, betas=(BETA_1, 0.999))\nlr_schedulerG = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(optimizerG,\n                                                                     T_0= T0_interval, eta_min=ETA_MIN)\nlr_schedulerD = torch.optim.lr_scheduler.CosineAnnealingWarmRestarts(optimizerD,\n\n                                                                     T_0= T0_interval, eta_min=ETA_MIN)\nfixed_noise = torch.randn(25, LATENT_DIM, 1, 1, device=device)\nbatch_size = train_loader.batch_size\n# criterion = nn.MSELoss()\nepoch_list,FID_list =  train_module(EPOCHS)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 10. Plot FID again\n\nJust implemented this, Hopefully it will work"},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nepoch_num = np.array(epoch_list)\nfid_score = np.array(FID_list)\nplt.plot(epoch_num,fid_score,linewidth= 4.0)\nplt.title('Results',fontsize=18)\nplt.ylabel('FID score',fontsize=14)\nplt.xlabel('# Epchs',fontsize=14)\nplt.grid()\n\nprint(f\" Best fid = {best_fid}\")\nbest_fid = np.min(fid_score)\n\nplt.show()\n\n","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"}},"nbformat":4,"nbformat_minor":1}