{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Dog Autoencoder\nIn this kernel, we learn about autoencoders. By understanding autoencoders, we will better understand GANs (Generative Adversarial Networks) and VAEs (Variational Autoencoders). We will also learn how to use an autoencoder to generate images of dogs.\n  \n![image](http://playagricola.com/Kaggle/ae7119.jpg)\n  \nKaggle's \"Generative Dog Images\" competition asks us to generate dog images using **generative methods**. It is unclear whether we must use GANs. If we must use GANs, then this kernel's output is **not** a valid competition submission.\n# Load Data and Augment\nWe will randomly crop the original 20,000 images and make 500,000 new training images."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"ComputeLB = False\n\nimport os, gc, zipfile\nimport numpy as np, pandas as pd\nfrom PIL import Image\nimport matplotlib.pyplot as plt\n\nif ComputeLB: PATH = '../input/generative-dog-images/all-dogs/all-dogs/'\nelse: PATH = '../input/all-dogs/all-dogs/'\nIMAGES = os.listdir(PATH)\nprint('There are',len(IMAGES),'images. Here are 5 example filesnames:')\nprint(IMAGES[:5])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"os.mkdir('../tmp')\nos.mkdir('../tmp/images')\n\n# CREATE RANDOMLY CROPPED IMAGES\nfor i in range(500000):\n    img = Image.open(PATH + IMAGES[i%len(IMAGES)])\n    img = img.resize(( 100,int(img.size[1]/(img.size[0]/100) )), Image.ANTIALIAS)\n    w = img.size[0]; h = img.size[1]; a=0; b=0\n    if w>64: a = np.random.randint(0,w-64)\n    if h>64: b = np.random.randint(0,h-64)\n    img = img.crop((a, b, 64+a, 64+b))\n    img.save('../tmp/images/'+str(i)+'.png','PNG')\n    if i%100000==0: print('created',i,'cropped images')\nprint('created 500000 cropped images')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Build Autoencoder\nThis code is inspired by Keras' tutorial [here][1]\n\n[1]: https://blog.keras.io/building-autoencoders-in-keras.html"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.models import Model\nfrom keras.layers import Input, Dense, Conv2D, MaxPooling2D, UpSampling2D\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.callbacks import LearningRateScheduler\n\nBATCH_SIZE = 256; EPOCHS = 10\ntrain_datagen = ImageDataGenerator(rescale=1./255)\ntrain_batches = train_datagen.flow_from_directory('../tmp/',\n        target_size=(64,64), shuffle=True, class_mode='input', batch_size=BATCH_SIZE)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ENCODER\ninput_img = Input(shape=(64, 64, 3))  \nx = Conv2D(48, (3, 3), activation='relu', padding='same')(input_img)\nx = MaxPooling2D((2, 2), padding='same')(x)\nx = Conv2D(96, (3, 3), activation='relu', padding='same')(x)\nx = MaxPooling2D((2, 2), padding='same')(x)\nx = Conv2D(192, (3, 3), activation='relu', padding='same')(x)\nx = MaxPooling2D((2, 2), padding='same')(x)\nencoded = Conv2D(32, (1, 1), activation='relu', padding='same')(x)\n\n# LATENT SPACE\nlatentSize = (8,8,32)\n\n# DECODER\ndirect_input = Input(shape=latentSize)\nx = Conv2D(192, (1, 1), activation='relu', padding='same')(direct_input)\nx = UpSampling2D((2, 2))(x)\nx = Conv2D(192, (3, 3), activation='relu', padding='same')(x)\nx = UpSampling2D((2, 2))(x)\nx = Conv2D(96, (3, 3), activation='relu', padding='same')(x)\nx = UpSampling2D((2, 2))(x)\nx = Conv2D(48, (3, 3), activation='relu', padding='same')(x)\ndecoded = Conv2D(3, (3, 3), activation='sigmoid', padding='same')(x)\n\n# COMPILE\nencoder = Model(input_img, encoded)\ndecoder = Model(direct_input, decoded)\nautoencoder = Model(input_img, decoder(encoded))\n\nautoencoder.compile(optimizer='Adam', loss='binary_crossentropy')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train Autoencoder"},{"metadata":{"trusted":true},"cell_type":"code","source":"history = autoencoder.fit_generator(train_batches,\n        steps_per_epoch = train_batches.samples // BATCH_SIZE,\n        epochs = EPOCHS, verbose=2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# View Reconstruction\nOur encoder works by mapping images from 12288 dimensional space (64 x 64 x 3) into 2048 dimensional space (latent image). This is a 6x compression rate. Our decoder works by mapping our latent image back into 12288 dimensional space. Below are examples. (Note that a decoder is like a GAN generator)."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"images = next(iter(train_batches))[0]\nfor i in range(5):\n\n    plt.figure(figsize=(15,5))\n    plt.subplot(1,3,1)\n    \n    # ORIGINAL IMAGE\n    orig = images[i,:,:,:].reshape((-1,64,64,3))\n    img = Image.fromarray( (255*orig).astype('uint8').reshape((64,64,3)))\n    plt.title('Original')\n    plt.imshow(img)\n\n    # LATENT IMAGE\n    latent_img = encoder.predict(orig)\n    mx = np.max( latent_img[0] )\n    mn = np.min( latent_img[0] )\n    latent_flat = ((latent_img[0] - mn) * 255/(mx - mn)).flatten(order='F')\n    img = Image.fromarray( latent_flat[:2025].astype('uint8').reshape((45,45)), mode='L') \n    plt.subplot(1,3,2)\n    plt.title('Latent')\n    plt.xlim((-10,55))\n    plt.ylim((-10,55))\n    plt.axis('off')\n    plt.imshow(img)\n\n    # RECONSTRUCTED IMAGE\n    decoded_imgs = decoder.predict(latent_img[0].reshape((-1,latentSize[0],latentSize[1],latentSize[2])))\n    img = Image.fromarray( (255*decoded_imgs[0]).astype('uint8').reshape((64,64,3)))\n    plt.subplot(1,3,3)\n    plt.title('Reconstructed')\n    plt.imshow(img)\n    \n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Latent Space and Dog Generation\nWe choose the size of latent space. In our autoencoder, we chose for latent space to be 2048 dimensional (6x compression). If we map all 20,000 images into latent space, they would cluster inside a 2048 dimensional hyperellipsoid. That ellipsoid would represent dog images. (Cat images would form a different ellipsoid). Below I have plotted 256 of our dog images in latent space as blue dots and drew their ellipsoid. (Note that latent space is being projected onto 2D for display in this kernel). (Note because of ReLU, you may see clipping).\n\nIf we would like to generate a new dog image, we can chose a new random point (different from existing training image dots) inside this ellipsoid and then decode it. For example, we could choose the 9 red points below and then convert them into dog images."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from matplotlib.patches import Ellipse\n\n# PROJECT LATENT INTO 2D, AVOID DEAD RELU\nlatent_img = encoder.predict(images)\nlatent_img2 = latent_img.reshape((-1,latentSize[0]*latentSize[1]*latentSize[2]))\nd = 0; s = 0\nwhile s<0.1:\n    x = latent_img2[:,d]\n    s = np.std(x); d += 1\ns = 0\nwhile s<0.1:\n    y = latent_img2[:,d]\n    s = np.std(y); d += 1\n\n# CALCULATE ELLIPSOID FROM 256 IMAGES\ncov = np.cov(x, y)\nlambda_, v = np.linalg.eig(cov)\nlambda_ = np.sqrt(lambda_)\nfor j in [1,2,3]:\n    ell = Ellipse(xy=(np.mean(x), np.mean(y)), width=lambda_[0]*j*2, \n            height=lambda_[1]*j*2, angle=np.rad2deg(np.arccos(v[0, 0])))\n    ell.set_facecolor('None')\n    ell.set_edgecolor('black')\n    plt.gca().add_artist(ell)\n    \n# PLOT 256 IMAGES AS DOTS IN LATENT SPACE\nplt.scatter(x,y)\nd = np.random.multivariate_normal([np.mean(x),np.mean(y)],cov,9)\nplt.scatter(d[:,0],d[:,1],color='red',s=100)\nplt.title('Dog Images form an Ellipsoid in Latent Space')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# CREATE 10000 CROPPED IMAGES\nx = np.random.choice(np.arange(20000),10000)\nimages = np.zeros((10000,64,64,3))\nfor i in range(len(x)):\n    img = Image.open(PATH + IMAGES[x[i]])\n    img = img.resize((100,int(img.size[1]/(img.size[0]/100))), Image.ANTIALIAS)\n    img = img.crop((18, 0, 82, 64))\n    images[i,:,:,:] = np.asarray(img).astype('float32') / 255.\n    #if i%1000==0: print(i)\n        \n# CALCULATE ELLIPSOID FROM 10000 IMAGES        \nencoded_imgs = encoder.predict(images)\nsz = latentSize[0] * latentSize[1] * latentSize[2]\nencoded_imgs = encoded_imgs.reshape((-1,sz))\nmm = np.mean(encoded_imgs,axis=0)\nss = np.cov(encoded_imgs,rowvar=False)\n\n# GENERATE 9 RANDOM DOG IMAGES\ngenerated = np.random.multivariate_normal(mm,ss,9)\ngenerated = generated.reshape((-1,latentSize[0],latentSize[1],latentSize[2]))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"# PLOT 9 RANDOM DOG IMAGES\nfor k in range(3):\n    plt.figure(figsize=(15,5))\n    plt.subplot(1,3,1)\n    decoded_imgs = decoder.predict(generated[k*3].reshape((-1,latentSize[0],latentSize[1],latentSize[2])))\n    img = Image.fromarray( (255*decoded_imgs[0]).astype('uint8').reshape((64,64,3)))\n    plt.imshow(img)\n    plt.subplot(1,3,2)\n    decoded_imgs = decoder.predict(generated[k*3+1].reshape((-1,latentSize[0],latentSize[1],latentSize[2])))\n    img = Image.fromarray( (255*decoded_imgs[0]).astype('uint8').reshape((64,64,3)))\n    plt.imshow(img)\n    plt.subplot(1,3,3)\n    decoded_imgs = decoder.predict(generated[k*3+2].reshape((-1,latentSize[0],latentSize[1],latentSize[2])))\n    img = Image.fromarray( (255*decoded_imgs[0]).astype('uint8').reshape((64,64,3)))\n    plt.imshow(img)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# How is LB scored?\nWhat is the FID (Fréchet Inception Distance) metric? Now that we understand latent space, the FID metric is easy to understand. After we train our autoencoder, we can compute the latent hyper-ellipsoid of our dog training data. Then if someone generates dog images, we could feed them into our autoencoder and calculate the latent hyper-ellipsoid of their generated dog images. Then FID is basically the distance between the two latent hyper-ellipsoids squared.\n\nLet `e1` be the training dog ellipsoid and `e2` be the generated dog ellipsoid. Let `x0, x1, ..., x2047` be the 2048 features of latent space. Each vector `x_k` from training dataset `e1` has approx 20000 values, and `e2` has 10000. Then the distance between the ellipsoids is approx the sum of the difference of means (ellipse centers) and sum of the difference of standard deviations (ellipse widths) of each 2048 `x_k`, i.e. `(mean(x_k)_e1 - mean(x_k)_e2)` plus `(std(x_k)_e1 - std(x_k)_e2)` for each of the 2048 `k` values. Finally `FID = this sum^2`.\n\nIn the image below, `FID approx = ||c2-c1||^2 + (h2-h1)^2 + (w2-w1)^2`. (Note this is exact when features are uncorrelated). MiFID adds a multiplier penalty if your dog images are too similar to the training data. `MiFID = FID * penalty`.\n\n![image](http://playagricola.com/Kaggle/ellipsesb7119.jpg)\n\n# Generating Better Dog Images\nThe generated dog images above are not very recognizable. Instead of choosing completely random points in latent space, we can choose points near existing training images. Let's try that."},{"metadata":{"trusted":true},"cell_type":"code","source":"# DISTANCE TO MOVE AWAY FROM EXISTING TRAIN IMAGES\nbeta = 0.35\n# GENERATE 9 RANDOM DOG IMAGES\ngenerated = np.random.multivariate_normal(mm,ss,9)\ngenerated = beta*generated + (1-beta)*encoded_imgs[:9]","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"for k in range(3):\n    plt.figure(figsize=(15,5))\n    plt.subplot(1,3,1)\n    decoded_imgs = decoder.predict(generated[k*3].reshape((-1,latentSize[0],latentSize[1],latentSize[2])))\n    img = Image.fromarray( (255*decoded_imgs[0]).astype('uint8').reshape((64,64,3)))\n    plt.imshow(img)\n    plt.subplot(1,3,2)\n    decoded_imgs = decoder.predict(generated[k*3+1].reshape((-1,latentSize[0],latentSize[1],latentSize[2])))\n    img = Image.fromarray( (255*decoded_imgs[0]).astype('uint8').reshape((64,64,3)))\n    plt.imshow(img)\n    plt.subplot(1,3,3)\n    decoded_imgs = decoder.predict(generated[k*3+2].reshape((-1,latentSize[0],latentSize[1],latentSize[2])))\n    img = Image.fromarray( (255*decoded_imgs[0]).astype('uint8').reshape((64,64,3)))\n    plt.imshow(img)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"beta = 0.2\n# GENERATE 10000 RANDOM DOG IMAGES FOR KAGGLE\ngenerated = np.random.multivariate_normal(mm,ss,10000)\nencoded_imgs = beta*generated + (1-beta)*encoded_imgs\ndecoded_imgs = decoder.predict(encoded_imgs.reshape((-1,latentSize[0],latentSize[1],latentSize[2])))\ndecoded_imgs.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Submit to Kaggle"},{"metadata":{"trusted":true},"cell_type":"code","source":"# SAVE TO ZIP FILE NAMED IMAGES.ZIP\nz = zipfile.PyZipFile('images.zip', mode='w')\nfor k in range(10000):\n    img = Image.fromarray( (255*decoded_imgs[k]).astype('uint8').reshape((64,64,3)))   \n    f = str(k)+'.png'\n    img.save(f,'PNG'); z.write(f); os.remove(f)\n    #if k % 1000==0: print(k)\nz.close()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Calculate LB Score\nIf you wish to compute LB, you must add the LB metric dataset [here][1] to this kernel and change the boolean variable in the first cell block.\n\n[1]: https://www.kaggle.com/wendykan/dog-face-generation-competition-kid-metric-input"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"from __future__ import absolute_import, division, print_function\nimport numpy as np\nimport os\nimport gzip, pickle\nimport tensorflow as tf\nfrom scipy import linalg\nimport pathlib\nimport urllib\nimport warnings\nfrom tqdm import tqdm\nfrom PIL import Image\n\nclass KernelEvalException(Exception):\n    pass\n\nmodel_params = {\n    'Inception': {\n        'name': 'Inception', \n        'imsize': 64,\n        'output_layer': 'Pretrained_Net/pool_3:0', \n        'input_layer': 'Pretrained_Net/ExpandDims:0',\n        'output_shape': 2048,\n        'cosine_distance_eps': 0.1\n        }\n}\n\ndef create_model_graph(pth):\n    \"\"\"Creates a graph from saved GraphDef file.\"\"\"\n    # Creates graph from saved graph_def.pb.\n    with tf.gfile.FastGFile( pth, 'rb') as f:\n        graph_def = tf.GraphDef()\n        graph_def.ParseFromString( f.read())\n        _ = tf.import_graph_def( graph_def, name='Pretrained_Net')\n\ndef _get_model_layer(sess, model_name):\n    # layername = 'Pretrained_Net/final_layer/Mean:0'\n    layername = model_params[model_name]['output_layer']\n    layer = sess.graph.get_tensor_by_name(layername)\n    ops = layer.graph.get_operations()\n    for op_idx, op in enumerate(ops):\n        for o in op.outputs:\n            shape = o.get_shape()\n            if shape._dims != []:\n              shape = [s.value for s in shape]\n              new_shape = []\n              for j, s in enumerate(shape):\n                if s == 1 and j == 0:\n                  new_shape.append(None)\n                else:\n                  new_shape.append(s)\n              o.__dict__['_shape_val'] = tf.TensorShape(new_shape)\n    return layer\n\ndef get_activations(images, sess, model_name, batch_size=50, verbose=False):\n    \"\"\"Calculates the activations of the pool_3 layer for all images.\n\n    Params:\n    -- images      : Numpy array of dimension (n_images, hi, wi, 3). The values\n                     must lie between 0 and 256.\n    -- sess        : current session\n    -- batch_size  : the images numpy array is split into batches with batch size\n                     batch_size. A reasonable batch size depends on the disposable hardware.\n    -- verbose    : If set to True and parameter out_step is given, the number of calculated\n                     batches is reported.\n    Returns:\n    -- A numpy array of dimension (num images, 2048) that contains the\n       activations of the given tensor when feeding inception with the query tensor.\n    \"\"\"\n    inception_layer = _get_model_layer(sess, model_name)\n    n_images = images.shape[0]\n    if batch_size > n_images:\n        print(\"warning: batch size is bigger than the data size. setting batch size to data size\")\n        batch_size = n_images\n    n_batches = n_images//batch_size + 1\n    pred_arr = np.empty((n_images,model_params[model_name]['output_shape']))\n    for i in tqdm(range(n_batches)):\n        if verbose:\n            print(\"\\rPropagating batch %d/%d\" % (i+1, n_batches), end=\"\", flush=True)\n        start = i*batch_size\n        if start+batch_size < n_images:\n            end = start+batch_size\n        else:\n            end = n_images\n                    \n        batch = images[start:end]\n        pred = sess.run(inception_layer, {model_params[model_name]['input_layer']: batch})\n        pred_arr[start:end] = pred.reshape(-1,model_params[model_name]['output_shape'])\n    if verbose:\n        print(\" done\")\n    return pred_arr\n\n\n# def calculate_memorization_distance(features1, features2):\n#     neigh = NearestNeighbors(n_neighbors=1, algorithm='kd_tree', metric='euclidean')\n#     neigh.fit(features2) \n#     d, _ = neigh.kneighbors(features1, return_distance=True)\n#     print('d.shape=',d.shape)\n#     return np.mean(d)\n\ndef normalize_rows(x: np.ndarray):\n    \"\"\"\n    function that normalizes each row of the matrix x to have unit length.\n\n    Args:\n     ``x``: A numpy matrix of shape (n, m)\n\n    Returns:\n     ``x``: The normalized (by row) numpy matrix.\n    \"\"\"\n    return np.nan_to_num(x/np.linalg.norm(x, ord=2, axis=1, keepdims=True))\n\n\ndef cosine_distance(features1, features2):\n    # print('rows of zeros in features1 = ',sum(np.sum(features1, axis=1) == 0))\n    # print('rows of zeros in features2 = ',sum(np.sum(features2, axis=1) == 0))\n    features1_nozero = features1[np.sum(features1, axis=1) != 0]\n    features2_nozero = features2[np.sum(features2, axis=1) != 0]\n    norm_f1 = normalize_rows(features1_nozero)\n    norm_f2 = normalize_rows(features2_nozero)\n\n    d = 1.0-np.abs(np.matmul(norm_f1, norm_f2.T))\n    print('d.shape=',d.shape)\n    print('np.min(d, axis=1).shape=',np.min(d, axis=1).shape)\n    mean_min_d = np.mean(np.min(d, axis=1))\n    print('distance=',mean_min_d)\n    return mean_min_d\n\n\ndef distance_thresholding(d, eps):\n    if d < eps:\n        return d\n    else:\n        return 1\n\ndef calculate_frechet_distance(mu1, sigma1, mu2, sigma2, eps=1e-6):\n    \"\"\"Numpy implementation of the Frechet Distance.\n    The Frechet distance between two multivariate Gaussians X_1 ~ N(mu_1, C_1)\n    and X_2 ~ N(mu_2, C_2) is\n            d^2 = ||mu_1 - mu_2||^2 + Tr(C_1 + C_2 - 2*sqrt(C_1*C_2)).\n            \n    Stable version by Dougal J. Sutherland.\n\n    Params:\n    -- mu1 : Numpy array containing the activations of the pool_3 layer of the\n             inception net ( like returned by the function 'get_predictions')\n             for generated samples.\n    -- mu2   : The sample mean over activations of the pool_3 layer, precalcualted\n               on an representive data set.\n    -- sigma1: The covariance matrix over activations of the pool_3 layer for\n               generated samples.\n    -- sigma2: The covariance matrix over activations of the pool_3 layer,\n               precalcualted on an representive data set.\n\n    Returns:\n    --   : The Frechet Distance.\n    \"\"\"\n\n    mu1 = np.atleast_1d(mu1)\n    mu2 = np.atleast_1d(mu2)\n\n    sigma1 = np.atleast_2d(sigma1)\n    sigma2 = np.atleast_2d(sigma2)\n\n    assert mu1.shape == mu2.shape, \"Training and test mean vectors have different lengths\"\n    assert sigma1.shape == sigma2.shape, \"Training and test covariances have different dimensions\"\n\n    diff = mu1 - mu2\n\n    # product might be almost singular\n    covmean, _ = linalg.sqrtm(sigma1.dot(sigma2), disp=False)\n    if not np.isfinite(covmean).all():\n        msg = \"fid calculation produces singular product; adding %s to diagonal of cov estimates\" % eps\n        warnings.warn(msg)\n        offset = np.eye(sigma1.shape[0]) * eps\n        # covmean = linalg.sqrtm((sigma1 + offset).dot(sigma2 + offset))\n        covmean = linalg.sqrtm((sigma1 + offset).dot(sigma2 + offset))\n    \n    # numerical error might give slight imaginary component\n    if np.iscomplexobj(covmean):\n        if not np.allclose(np.diagonal(covmean).imag, 0, atol=1e-3):\n            m = np.max(np.abs(covmean.imag))\n            raise ValueError(\"Imaginary component {}\".format(m))\n        covmean = covmean.real\n\n    # covmean = tf.linalg.sqrtm(tf.linalg.matmul(sigma1,sigma2))\n\n    print('covmean.shape=',covmean.shape)\n    # tr_covmean = tf.linalg.trace(covmean)\n\n    tr_covmean = np.trace(covmean)\n    return diff.dot(diff) + np.trace(sigma1) + np.trace(sigma2) - 2 * tr_covmean\n    # return diff.dot(diff) + tf.linalg.trace(sigma1) + tf.linalg.trace(sigma2) - 2 * tr_covmean\n#-------------------------------------------------------------------------------\n\n\ndef calculate_activation_statistics(images, sess, model_name, batch_size=50, verbose=False):\n    \"\"\"Calculation of the statistics used by the FID.\n    Params:\n    -- images      : Numpy array of dimension (n_images, hi, wi, 3). The values\n                     must lie between 0 and 255.\n    -- sess        : current session\n    -- batch_size  : the images numpy array is split into batches with batch size\n                     batch_size. A reasonable batch size depends on the available hardware.\n    -- verbose     : If set to True and parameter out_step is given, the number of calculated\n                     batches is reported.\n    Returns:\n    -- mu    : The mean over samples of the activations of the pool_3 layer of\n               the incption model.\n    -- sigma : The covariance matrix of the activations of the pool_3 layer of\n               the incption model.\n    \"\"\"\n    act = get_activations(images, sess, model_name, batch_size, verbose)\n    mu = np.mean(act, axis=0)\n    sigma = np.cov(act, rowvar=False)\n    return mu, sigma, act\n    \ndef _handle_path_memorization(path, sess, model_name, is_checksize, is_check_png):\n    path = pathlib.Path(path)\n    files = list(path.glob('*.jpg')) + list(path.glob('*.png'))\n    imsize = model_params[model_name]['imsize']\n\n    # In production we don't resize input images. This is just for demo purpose. \n    x = np.array([np.array(img_read_checks(fn, imsize, is_checksize, imsize, is_check_png)) for fn in files])\n    m, s, features = calculate_activation_statistics(x, sess, model_name)\n    del x #clean up memory\n    return m, s, features\n\n# check for image size\ndef img_read_checks(filename, resize_to, is_checksize=False, check_imsize = 64, is_check_png = False):\n    im = Image.open(str(filename))\n    if is_checksize and im.size != (check_imsize,check_imsize):\n        raise KernelEvalException('The images are not of size '+str(check_imsize))\n    \n    if is_check_png and im.format != 'PNG':\n        raise KernelEvalException('Only PNG images should be submitted.')\n\n    if resize_to is None:\n        return im\n    else:\n        return im.resize((resize_to,resize_to),Image.ANTIALIAS)\n\ndef calculate_kid_given_paths(paths, model_name, model_path, feature_path=None, mm=[], ss=[], ff=[]):\n    ''' Calculates the KID of two paths. '''\n    tf.reset_default_graph()\n    create_model_graph(str(model_path))\n    with tf.Session() as sess:\n        sess.run(tf.global_variables_initializer())\n        m1, s1, features1 = _handle_path_memorization(paths[0], sess, model_name, is_checksize = True, is_check_png = True)\n        if len(mm) != 0:\n            m2 = mm\n            s2 = ss\n            features2 = ff\n        elif feature_path is None:\n            m2, s2, features2 = _handle_path_memorization(paths[1], sess, model_name, is_checksize = False, is_check_png = False)\n        else:\n            with np.load(feature_path) as f:\n                m2, s2, features2 = f['m'], f['s'], f['features']\n\n        print('m1,m2 shape=',(m1.shape,m2.shape),'s1,s2=',(s1.shape,s2.shape))\n        print('starting calculating FID')\n        fid_value = calculate_frechet_distance(m1, s1, m2, s2)\n        print('done with FID, starting distance calculation')\n        distance = cosine_distance(features1, features2)        \n        return fid_value, distance, m2, s2, features2\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"if ComputeLB:\n    # FREE MEMORY\n    del decoded_imgs, encoded_imgs, images, encoder, decoder, autoencoder, generated\n    x = gc.collect()\n    \n    # UNCOMPRESS OUR IMGAES\n    with zipfile.ZipFile(\"../working/images.zip\",\"r\") as z:\n        z.extractall(\"../tmp/images2/\")\n\n    # COMPUTE LB SCORE\n    m2 = []; s2 =[]; f2 = []\n    user_images_unzipped_path = '../tmp/images2/'\n    images_path = [user_images_unzipped_path,'../input/generative-dog-images/all-dogs/all-dogs/']\n    public_path = '../input/dog-face-generation-competition-kid-metric-input/classify_image_graph_def.pb'\n\n    fid_epsilon = 10e-15\n\n    fid_value_public, distance_public, m2, s2, f2 = calculate_kid_given_paths(images_path, 'Inception', public_path, mm=m2, ss=s2, ff=f2)\n    distance_public = distance_thresholding(distance_public, model_params['Inception']['cosine_distance_eps'])\n    print(\"FID_public: \", fid_value_public, \"distance_public: \", distance_public, \"multiplied_public: \",\n            fid_value_public /(distance_public + fid_epsilon))\n    \n# REMOVE FILES TO PREVENT KERNEL ERROR OF TOO MANY FILES\n! rm -r ../tmp","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}