{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Private LB Simulation\n## PART I: Private Dataset [5 min read]\n\n\n<img src=\"https://i.ibb.co/t4cCsqF/Captura-de-pantalla-de-2019-07-17-13-58-42.png\" alt=\"Captura-de-pantalla-de-2019-07-17-13-58-42\" border=\"0\">\n\n----\n\nI'm back! no more discussions about memorizers or rules, it's boring.\n\n**I think this can help everyone, no matter what kind of GAN you use: memorizer, GAN, CGAN, RaLSGAN etc**\n\n**In this kernel I simulate the private dataset!** And we'll see if the dataset we use to generate matters when it comes to getting a good result.\n\n> This took me some time, so I hope you'll appreciate it if it works for you. **Remember, the <span style=\"color:blue\">upvote</span> button is near the fork button.** And as you see, the title says *part I*, the *part II* is about the ```mysterious pretrained NN``` ;)\n\n\n### Datasets\n\n\n1. [PetFinder.my Adoption Prediction](https://www.kaggle.com/c/petfinder-adoption-prediction/data)\n2. [Dogs vs. Cats](https://www.kaggle.com/c/dogs-vs-cats)\n3. [Dog Breed Identification](https://www.kaggle.com/c/dog-breed-identification)  \n4. [CIFAR-10](https://www.cs.toronto.edu/~kriz/cifar.html)\n\n\n**Why this datasets could be the private dataset aka \"mysterious private dog images\" ?** \n\nWe are using [Stanford Dogs Dataset](http://vision.stanford.edu/aditya86/ImageNetDogs/), it's public, like these others datasets. For example, CIFAR-10 is also a well known dataset and has **dogs** images (among others). But keep reading, you'll see the best part soon ;)\n\n\n### My test\n\n1. Train using **these datasets** and generate ```User Generate N Dogs Images ```\n2. Use as ```Mysterious NN``` the inception model (risky hypothesis)\n3. Use as ```Mysterious Dog Dataset``` (private) the *Generative Dogs Dataset*\n\nIn other words, I do the opposite of what kaggle does.\n\n**Let's see if the dog dataset we use to generate images is important when it comes to getting a great LB score!**\n> If the results do not vary much, then perhaps the dataset is not important\n\n----\n\n**NOTE** \n1. I can't import all these datasets (the kernel doesn't start properly) so I only import **2** of them (*my winning horses*)\n2. You'll see this kernel has many versions, probably I'll try one different dataset in each version.\n\n<br>"},{"metadata":{},"cell_type":"markdown","source":"# [CIFAR-10](https://www.cs.toronto.edu/~kriz/cifar.html)\n\nThe CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images. \n\nThe dataset is divided into five training batches and one test batch, each with 10000 images. The test batch contains exactly 1000 randomly-selected images from each class. The training batches contain the remaining images in random order, but some training batches may contain more images from one class than another. Between them, the training batches contain exactly 5000 images from each class. Obviously, I'm going to use only **dog** images.\n\nHere are the classes in the dataset, as well as 10 random images from each:\n\n<img src=\"https://i.ibb.co/02xnm2X/Captura-de-pantalla-de-2019-07-18-01-27-11.png\" alt=\"Captura-de-pantalla-de-2019-07-18-01-27-11\" border=\"0\">\n\n"},{"metadata":{"trusted":true,"_kg_hide-output":false},"cell_type":"code","source":"import numpy as np\nimport keras\nfrom keras.datasets import cifar10\n(x_train, y_train), (x_test, y_test) = cifar10.load_data()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"```x_train, x_test:``` uint8 array of RGB image data with shape (num_samples, 3, 32, 32) or (num_samples, 32, 32, 3) based on the image_data_format backend setting of either channels_first or channels_last respectively.\n\n```y_train, y_test:``` uint8 array of category labels (integers in range 0-9) with shape (num_samples,)."},{"metadata":{"trusted":true},"cell_type":"code","source":"x_train.shape, y_train.shape, x_test.shape, y_test.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Labels** are encoded, the magic value is **dogs = 5**  (airplane=0 ...truck=9)"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"y_train[0:5]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"idx_dogs = [i for i in range(x_train.shape[0]) if y_train[i]==5]\nlen(idx_dogs), idx_dogs[0:5]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"imagesIn = x_train[idx_dogs]\nimagesIn.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now you have **5000 new dogs!** Resize the pictures to 64x64 and try your GAN with this dataset."},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"del x_train, x_test, y_train, y_test","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# [Dog Breed Identification](https://www.kaggle.com/c/dog-breed-identification)\n\n**Kaggle description**\n\n> In this playground competition, you are provided a strictly canine subset of ImageNet in order to practice fine-grained image categorization. How well you can tell your Norfolk Terriers from your Norwich Terriers? With 120 breeds of dogs and a limited number training images per class, you might find the problem more, err, ruff than you anticipated.\n\n>**Acknowledgments**\nWe extend our gratitude to the creators of the Stanford Dogs Dataset for making this competition possible: Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li.\n\n\n**This dataset is very similar to out dataset!**\n\n![](https://storage.googleapis.com/kaggle-competitions/kaggle/3333/media/border_collies.png)"},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls ../input/dog-breed-identification/","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Load and crop**"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"ComputeLB = True\nDogsOnly = False\n\nimport numpy as np, pandas as pd, os\nimport xml.etree.ElementTree as ET \nimport matplotlib.pyplot as plt, zipfile \nfrom PIL import Image \n\nROOT = '../input/dog-breed-identification/train/'\nif not ComputeLB: ROOT = '../input/'\nIMAGES = os.listdir(ROOT + 'train/')\n#breeds = os.listdir(ROOT + 'annotation/Annotation/') \n\nidxIn = 0; namesIn = []\nimagesIn = np.zeros((25000,64,64,3))\n\n# CROP WITH BOUNDING BOXES TO GET DOGS ONLY\n# https://www.kaggle.com/paulorzp/show-annotations-and-breeds\nif DogsOnly:\n    pass\n    \n# RANDOMLY CROP FULL IMAGES\nelse:\n    IMAGES = np.sort(IMAGES)\n    np.random.seed(810)\n    x = np.random.choice(np.arange(len(IMAGES)),10000)\n    np.random.seed(None)\n    for k in range(len(x)):\n        #print (ROOT + 'train/' + IMAGES[x[k]])\n        img = Image.open(ROOT + 'train/' + IMAGES[x[k]])\n        w = img.size[0]; h = img.size[1];\n        if (k%2==0)|(k%3==0):\n            w2 = 100; h2 = int(h/(w/100))\n            a = 18; b = 0          \n        else:\n            a=0; b=0\n            if w<h:\n                w2 = 64; h2 = int((64/w)*h)\n                b = (h2-64)//2\n            else:\n                h2 = 64; w2 = int((64/h)*w)\n                a = (w2-64)//2\n        img = img.resize((w2,h2), Image.ANTIALIAS)\n        img = img.crop((0+a, 0+b, 64+a, 64+b))    \n        imagesIn[idxIn,:,:,:] = np.asarray(img)\n        namesIn.append(IMAGES[x[k]])\n        #if idxIn%1000==0: print(idxIn)\n        idxIn += 1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Display cropped images**\n> **NOTE:** probably this crop is not the best, but this is just a quick test."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"x = np.random.randint(0,idxIn,25)\nfor k in range(5):\n    plt.figure(figsize=(37,3))\n    for j in range(5):\n        plt.subplot(1,5,j+1)\n        img = Image.fromarray( imagesIn[x[k*5+j],:,:,:].astype('uint8') )\n        plt.axis('off')\n        if not DogsOnly: plt.title(namesIn[x[k*5+j]],fontsize=11)\n        else: plt.title(namesIn[x[k*5+j]].split('-')[1],fontsize=11)\n        plt.imshow(img)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### <span style=\"color:red\"> DISCLAIMER </span>\n\nThe following code is from: [Dog Memorizer GAN](https://www.kaggle.com/cdeotte/dog-memorizer-gan) , I use this code because **it's fast!** (I'm not 100% if it's allowed or not) you should generate the 10k images using **your own gan** and check **[LB Score]()**\n"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"### Build Discriminator"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from keras.models import Model\nfrom keras.layers import Input, Dense, Conv2D, Reshape, Flatten, concatenate\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import LearningRateScheduler\nfrom keras.optimizers import SGD, Adam","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# BUILD DISCRIMINATIVE NETWORK\ndog = Input((12288,))\ndogName = Input((10000,))\nx = Dense(12288, activation='sigmoid')(dogName) \nx = Reshape((2,12288,1))(concatenate([dog,x]))\nx = Conv2D(1,(2,1),use_bias=False,name='conv')(x)\ndiscriminated = Flatten()(x)\n\n# COMPILE\ndiscriminator = Model([dog,dogName], discriminated)\ndiscriminator.get_layer('conv').trainable = False\ndiscriminator.get_layer('conv').set_weights([np.array([[[[-1.0 ]]],[[[1.0]]]])])\ndiscriminator.compile(optimizer='adam', loss='binary_crossentropy')\n\n# DISPLAY ARCHITECTURE\ndiscriminator.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Train Discriminator"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# TRAINING DATA\ntrain_y = (imagesIn[:10000,:,:,:]/255.).reshape((-1,12288))\ntrain_X = np.zeros((10000,10000))\nfor i in range(10000): train_X[i,i] = 1\nzeros = np.zeros((10000,12288))\n\n# TRAIN NETWORK\nlr = 0.5\nfor k in range(5):\n    annealer = LearningRateScheduler(lambda x: lr)\n    h = discriminator.fit([zeros,train_X], train_y, epochs = 10, batch_size=256, callbacks=[annealer], verbose=0)\n    print('Epoch',(k+1)*10,'/30 - loss =',h.history['loss'][-1] )\n    if h.history['loss'][-1]<0.533: lr = 0.1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Delete Training Images"},{"metadata":{"trusted":true},"cell_type":"code","source":"del imagesIn","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Discriminator Recalls from Memory Dogs**"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":" for k in range(5):\n    plt.figure(figsize=(15,3))\n    for j in range(5):\n        xx = np.zeros((10000))\n        xx[np.random.randint(10000)] = 1\n        plt.subplot(1,5,j+1)\n        img = discriminator.predict([zeros[0,:].reshape((-1,12288)),xx.reshape((-1,10000))]).reshape((-1,64,64,3))\n        img = Image.fromarray( (255*img).astype('uint8').reshape((64,64,3)))\n        plt.axis('off')\n        plt.imshow(img)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Build Generator and GAN"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# BUILD GENERATOR NETWORK\nseed = Input((10000,))\ngenerated = Dense(12288, activation='linear')(seed)\n\n# COMPILE\ngenerator = Model(seed, [generated,Reshape((10000,))(seed)])\n\n# DISPLAY ARCHITECTURE\ngenerator.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**BUILD GENERATIVE ADVERSARIAL NETWORK**"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"discriminator.trainable=False    \ngan_input = Input(shape=(10000,))\nx = generator(gan_input)\ngan_output = discriminator(x)\n\n# COMPILE GAN\ngan = Model(gan_input, gan_output)\ngan.get_layer('model_1').get_layer('conv').set_weights([np.array([[[[-1 ]]],[[[255.]]]])])\ngan.compile(optimizer=Adam(5), loss='mean_squared_error')\n\n# DISPLAY ARCHITECTURE\ngan.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Discriminator Coaches Generator"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# TRAINING DATA\ntrain = np.zeros((10000,10000))\nfor i in range(10000): train[i,i] = 1\nzeros = np.zeros((10000,12288))\n\n# TRAIN NETWORKS\nlr = 5.\nfor k in range(50):  \n\n    # BEGIN DISCRIMINATOR COACHES GENERATOR\n    annealer = LearningRateScheduler(lambda x: lr)\n    h = gan.fit(train, zeros, epochs = 1, batch_size=256, callbacks=[annealer], verbose=0)\n    if (k<10)|(k%5==4):\n        print('Epoch',(k+1)*10,'/500 - loss =',h.history['loss'][-1] )\n    if h.history['loss'][-1] < 25: lr = 1.\n    if h.history['loss'][-1] < 1.5: lr = 0.5\n        \n    # DISPLAY GENERATOR LEARNING PROGRESS\n    if k<10:        \n        plt.figure(figsize=(15,3))\n        for j in range(5):\n            xx = np.zeros((10000))\n            xx[np.random.randint(10000)] = 1\n            plt.subplot(1,5,j+1)\n            img = generator.predict(xx.reshape((-1,10000)))[0].reshape((-1,64,64,3))\n            img = Image.fromarray( (img).astype('uint8').reshape((64,64,3)))\n            plt.axis('off')\n            plt.imshow(img)\n        plt.show()  ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Build Generator Class"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"class DogGenerator:\n    index = 0   \n    def getDog(self,seed):\n        xx = np.zeros((10000))\n        xx[self.index] = 0.999\n        xx[np.random.randint(10000)] = 0.001\n        img = generator.predict(xx.reshape((-1,10000)))[0].reshape((64,64,3))\n        self.index = (self.index+1)%10000\n        return Image.fromarray( img.astype('uint8') ) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Submit to Kaggle"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# SAVE TO ZIP FILE NAMED IMAGES.ZIP\nz = zipfile.PyZipFile('images.zip', mode='w')\nd = DogGenerator()\nfor k in range(10000):\n    img = d.getDog(np.random.normal(0,1,100))\n    f = str(k)+'.png'\n    img.save(f,'PNG'); z.write(f); os.remove(f)\n    #if k % 1000==0: print(k)\nz.close()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# [LB Score]()\n\n1. We didn't use ```ImageNet Dog Images``` to generate ```User Generated dog Images```, **we used Dog Breed Identification dataset!**\n2. We use as ```Mysterious Dog Dataset``` (**private dataset**) the ```ImageNet Dog Images``` to generate ```User Generated dog Images```\n3. We use as ```Mysterious Pre-trained dataset``` the same inception model (a risky hypothesis). But, do you see the title, it says *part I*, if there is a lot of people interested I can do *part 2* and show my **private NN test**\n\nThat means **we do the opposite of kaggle**\n\n### The question is:\n\n- ```private dataset = all dogs```, generate using ```Dog Breed Identification``` gives us a LB around 7.5 - 8.5\n\n- **```private dataset = Dog Breed Identification```, generate using ```all dogs``` gives us a LB around 7.5 - 8.5 ? This is the real case, and I think the answer is yes!**\n\nYou can try with the others datasets too! I checked ```Dog Breed Identification``` and ```Cifar-10```, and it's pretty similar (using cifar-10 the score is around 10).\n\n<img src=\"https://i.ibb.co/DVn018V/Captura-de-pantalla-de-2019-07-18-02-55-09.png\" alt=\"Captura-de-pantalla-de-2019-07-18-02-55-09\" border=\"0\">\n"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from __future__ import absolute_import, division, print_function\nimport numpy as np\nimport os\nimport gzip, pickle\nimport tensorflow as tf\nfrom scipy import linalg\nimport pathlib\nimport urllib\nimport warnings\nfrom tqdm import tqdm\nfrom PIL import Image\n\nclass KernelEvalException(Exception):\n    pass\n\nmodel_params = {\n    'Inception': {\n        'name': 'Inception', \n        'imsize': 64,\n        'output_layer': 'Pretrained_Net/pool_3:0', \n        'input_layer': 'Pretrained_Net/ExpandDims:0',\n        'output_shape': 2048,\n        'cosine_distance_eps': 0.1\n        }\n}\n\ndef create_model_graph(pth):\n    \"\"\"Creates a graph from saved GraphDef file.\"\"\"\n    # Creates graph from saved graph_def.pb.\n    with tf.gfile.FastGFile( pth, 'rb') as f:\n        graph_def = tf.GraphDef()\n        graph_def.ParseFromString( f.read())\n        _ = tf.import_graph_def( graph_def, name='Pretrained_Net')\n\ndef _get_model_layer(sess, model_name):\n    # layername = 'Pretrained_Net/final_layer/Mean:0'\n    layername = model_params[model_name]['output_layer']\n    layer = sess.graph.get_tensor_by_name(layername)\n    ops = layer.graph.get_operations()\n    for op_idx, op in enumerate(ops):\n        for o in op.outputs:\n            shape = o.get_shape()\n            if shape._dims != []:\n              shape = [s.value for s in shape]\n              new_shape = []\n              for j, s in enumerate(shape):\n                if s == 1 and j == 0:\n                  new_shape.append(None)\n                else:\n                  new_shape.append(s)\n              o.__dict__['_shape_val'] = tf.TensorShape(new_shape)\n    return layer\n\ndef get_activations(images, sess, model_name, batch_size=50, verbose=False):\n    \"\"\"Calculates the activations of the pool_3 layer for all images.\n\n    Params:\n    -- images      : Numpy array of dimension (n_images, hi, wi, 3). The values\n                     must lie between 0 and 256.\n    -- sess        : current session\n    -- batch_size  : the images numpy array is split into batches with batch size\n                     batch_size. A reasonable batch size depends on the disposable hardware.\n    -- verbose    : If set to True and parameter out_step is given, the number of calculated\n                     batches is reported.\n    Returns:\n    -- A numpy array of dimension (num images, 2048) that contains the\n       activations of the given tensor when feeding inception with the query tensor.\n    \"\"\"\n    inception_layer = _get_model_layer(sess, model_name)\n    n_images = images.shape[0]\n    if batch_size > n_images:\n        print(\"warning: batch size is bigger than the data size. setting batch size to data size\")\n        batch_size = n_images\n    n_batches = n_images//batch_size + 1\n    pred_arr = np.empty((n_images,model_params[model_name]['output_shape']))\n    for i in tqdm(range(n_batches)):\n        if verbose:\n            print(\"\\rPropagating batch %d/%d\" % (i+1, n_batches), end=\"\", flush=True)\n        start = i*batch_size\n        if start+batch_size < n_images:\n            end = start+batch_size\n        else:\n            end = n_images\n                    \n        batch = images[start:end]\n        pred = sess.run(inception_layer, {model_params[model_name]['input_layer']: batch})\n        pred_arr[start:end] = pred.reshape(-1,model_params[model_name]['output_shape'])\n    if verbose:\n        print(\" done\")\n    return pred_arr\n\n\n# def calculate_memorization_distance(features1, features2):\n#     neigh = NearestNeighbors(n_neighbors=1, algorithm='kd_tree', metric='euclidean')\n#     neigh.fit(features2) \n#     d, _ = neigh.kneighbors(features1, return_distance=True)\n#     print('d.shape=',d.shape)\n#     return np.mean(d)\n\ndef normalize_rows(x: np.ndarray):\n    \"\"\"\n    function that normalizes each row of the matrix x to have unit length.\n\n    Args:\n     ``x``: A numpy matrix of shape (n, m)\n\n    Returns:\n     ``x``: The normalized (by row) numpy matrix.\n    \"\"\"\n    return np.nan_to_num(x/np.linalg.norm(x, ord=2, axis=1, keepdims=True))\n\n\ndef cosine_distance(features1, features2):\n    # print('rows of zeros in features1 = ',sum(np.sum(features1, axis=1) == 0))\n    # print('rows of zeros in features2 = ',sum(np.sum(features2, axis=1) == 0))\n    features1_nozero = features1[np.sum(features1, axis=1) != 0]\n    features2_nozero = features2[np.sum(features2, axis=1) != 0]\n    norm_f1 = normalize_rows(features1_nozero)\n    norm_f2 = normalize_rows(features2_nozero)\n\n    d = 1.0-np.abs(np.matmul(norm_f1, norm_f2.T))\n    print('d.shape=',d.shape)\n    print('np.min(d, axis=1).shape=',np.min(d, axis=1).shape)\n    mean_min_d = np.mean(np.min(d, axis=1))\n    print('distance=',mean_min_d)\n    return mean_min_d\n\n\ndef distance_thresholding(d, eps):\n    if d < eps:\n        return d\n    else:\n        return 1\n\ndef calculate_frechet_distance(mu1, sigma1, mu2, sigma2, eps=1e-6):\n    \"\"\"Numpy implementation of the Frechet Distance.\n    The Frechet distance between two multivariate Gaussians X_1 ~ N(mu_1, C_1)\n    and X_2 ~ N(mu_2, C_2) is\n            d^2 = ||mu_1 - mu_2||^2 + Tr(C_1 + C_2 - 2*sqrt(C_1*C_2)).\n            \n    Stable version by Dougal J. Sutherland.\n\n    Params:\n    -- mu1 : Numpy array containing the activations of the pool_3 layer of the\n             inception net ( like returned by the function 'get_predictions')\n             for generated samples.\n    -- mu2   : The sample mean over activations of the pool_3 layer, precalcualted\n               on an representive data set.\n    -- sigma1: The covariance matrix over activations of the pool_3 layer for\n               generated samples.\n    -- sigma2: The covariance matrix over activations of the pool_3 layer,\n               precalcualted on an representive data set.\n\n    Returns:\n    --   : The Frechet Distance.\n    \"\"\"\n\n    mu1 = np.atleast_1d(mu1)\n    mu2 = np.atleast_1d(mu2)\n\n    sigma1 = np.atleast_2d(sigma1)\n    sigma2 = np.atleast_2d(sigma2)\n\n    assert mu1.shape == mu2.shape, \"Training and test mean vectors have different lengths\"\n    assert sigma1.shape == sigma2.shape, \"Training and test covariances have different dimensions\"\n\n    diff = mu1 - mu2\n\n    # product might be almost singular\n    covmean, _ = linalg.sqrtm(sigma1.dot(sigma2), disp=False)\n    if not np.isfinite(covmean).all():\n        msg = \"fid calculation produces singular product; adding %s to diagonal of cov estimates\" % eps\n        warnings.warn(msg)\n        offset = np.eye(sigma1.shape[0]) * eps\n        # covmean = linalg.sqrtm((sigma1 + offset).dot(sigma2 + offset))\n        covmean = linalg.sqrtm((sigma1 + offset).dot(sigma2 + offset))\n    \n    # numerical error might give slight imaginary component\n    if np.iscomplexobj(covmean):\n        if not np.allclose(np.diagonal(covmean).imag, 0, atol=1e-3):\n            m = np.max(np.abs(covmean.imag))\n            raise ValueError(\"Imaginary component {}\".format(m))\n        covmean = covmean.real\n\n    # covmean = tf.linalg.sqrtm(tf.linalg.matmul(sigma1,sigma2))\n\n    print('covmean.shape=',covmean.shape)\n    # tr_covmean = tf.linalg.trace(covmean)\n\n    tr_covmean = np.trace(covmean)\n    return diff.dot(diff) + np.trace(sigma1) + np.trace(sigma2) - 2 * tr_covmean\n    # return diff.dot(diff) + tf.linalg.trace(sigma1) + tf.linalg.trace(sigma2) - 2 * tr_covmean\n#-------------------------------------------------------------------------------\n\n\ndef calculate_activation_statistics(images, sess, model_name, batch_size=50, verbose=False):\n    \"\"\"Calculation of the statistics used by the FID.\n    Params:\n    -- images      : Numpy array of dimension (n_images, hi, wi, 3). The values\n                     must lie between 0 and 255.\n    -- sess        : current session\n    -- batch_size  : the images numpy array is split into batches with batch size\n                     batch_size. A reasonable batch size depends on the available hardware.\n    -- verbose     : If set to True and parameter out_step is given, the number of calculated\n                     batches is reported.\n    Returns:\n    -- mu    : The mean over samples of the activations of the pool_3 layer of\n               the incption model.\n    -- sigma : The covariance matrix of the activations of the pool_3 layer of\n               the incption model.\n    \"\"\"\n    act = get_activations(images, sess, model_name, batch_size, verbose)\n    mu = np.mean(act, axis=0)\n    sigma = np.cov(act, rowvar=False)\n    return mu, sigma, act\n    \ndef _handle_path_memorization(path, sess, model_name, is_checksize, is_check_png):\n    path = pathlib.Path(path)\n    files = list(path.glob('*.jpg')) + list(path.glob('*.png'))\n    imsize = model_params[model_name]['imsize']\n\n    # In production we don't resize input images. This is just for demo purpose. \n    x = np.array([np.array(img_read_checks(fn, imsize, is_checksize, imsize, is_check_png)) for fn in files])\n    m, s, features = calculate_activation_statistics(x, sess, model_name)\n    del x #clean up memory\n    return m, s, features\n\n# check for image size\ndef img_read_checks(filename, resize_to, is_checksize=False, check_imsize = 64, is_check_png = False):\n    im = Image.open(str(filename))\n    if is_checksize and im.size != (check_imsize,check_imsize):\n        raise KernelEvalException('The images are not of size '+str(check_imsize))\n    \n    if is_check_png and im.format != 'PNG':\n        raise KernelEvalException('Only PNG images should be submitted.')\n\n    if resize_to is None:\n        return im\n    else:\n        return im.resize((resize_to,resize_to),Image.ANTIALIAS)\n\ndef calculate_kid_given_paths(paths, model_name, model_path, feature_path=None, mm=[], ss=[], ff=[]):\n    ''' Calculates the KID of two paths. '''\n    tf.reset_default_graph()\n    create_model_graph(str(model_path))\n    with tf.Session() as sess:\n        sess.run(tf.global_variables_initializer())\n        m1, s1, features1 = _handle_path_memorization(paths[0], sess, model_name, is_checksize = True, is_check_png = True)\n        if len(mm) != 0:\n            m2 = mm\n            s2 = ss\n            features2 = ff\n        elif feature_path is None:\n            m2, s2, features2 = _handle_path_memorization(paths[1], sess, model_name, is_checksize = False, is_check_png = False)\n        else:\n            with np.load(feature_path) as f:\n                m2, s2, features2 = f['m'], f['s'], f['features']\n\n        print('m1,m2 shape=',(m1.shape,m2.shape),'s1,s2=',(s1.shape,s2.shape))\n        print('starting calculating FID')\n        fid_value = calculate_frechet_distance(m1, s1, m2, s2)\n        print('done with FID, starting distance calculation')\n        distance = cosine_distance(features1, features2)        \n        return fid_value, distance, m2, s2, features2","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"if ComputeLB:\n  \n    # UNCOMPRESS OUR IMGAES\n    with zipfile.ZipFile(\"../working/images.zip\",\"r\") as z:\n        z.extractall(\"../tmp/images2/\")\n\n    # COMPUTE LB SCORE\n    m2 = []; s2 =[]; f2 = []\n    user_images_unzipped_path = '../tmp/images2/'\n    images_path = [user_images_unzipped_path,'../input/generative-dog-images/all-dogs/all-dogs/']\n    public_path = '../input/dog-face-generation-competition-kid-metric-input/classify_image_graph_def.pb'\n\n    fid_epsilon = 10e-15\n\n    fid_value_public, distance_public, m2, s2, f2 = calculate_kid_given_paths(images_path, 'Inception', public_path, mm=m2, ss=s2, ff=f2)\n    distance_public = distance_thresholding(distance_public, model_params['Inception']['cosine_distance_eps'])\n    print(\"FID_public: \", fid_value_public, \"distance_public: \", distance_public, \"multiplied_public: \",\n            fid_value_public /(distance_public + fid_epsilon))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# REMOVE FILES TO PREVENT KERNEL ERROR OF TOO MANY FILES\n! rm -r ../tmp","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}