{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Pretrained Big GAN 128 - Scores LB 12\nIn this kernel, we load Pretrained Big GAN 128 and have fun generating some dog images. \n\nIt's amazing how accurate Big GAN generated dogs appear. Big GAN 128 was trained on the same set of dogs that we are using in Kaggle's Dog Competition. (Stanford Dogs are a subset of ImageNet Dogs). If we were allowed more than 9 hours to train, we could train Big GAN 128 in a Kaggle notebook/kernel and submit it as our competition solution!! Big GAN 128 scores an amazing LB 12!! (Since we need to turn on internet to download pretrained network weights, the outputted images from this notebook/kernel cannot be submitted).\n\nBig GAN 128 can produce 1000 different classes of images. A full list is [here][4]. For example class 472 is a `canoe` and class 71 is a `scorpion`. Try changing the code below to see images besides dogs. Furthermore you can do latent walks between different classes. As an example I morph dogs into cats below. Dogs are classes 151 thru 280 and cats are 281 thru 293.\n\nYou can read a blog about Big GAN [here][1]. A paper descibing Big GAN is [here][3]. We download Big GAN from GitHub [here][2]\n\n[1]: https://medium.com/syncedreview/biggan-a-new-state-of-the-art-in-image-synthesis-cf2ec5694024\n[2]: https://github.com/huggingface/pytorch-pretrained-BigGAN\n[3]: https://openreview.net/forum?id=B1xsqj09Fm\n[4]: https://gist.github.com/yrevar/942d3a0ac09ec9e5eb3a"},{"metadata":{},"cell_type":"markdown","source":"# Load Pretrained Big GAN 128"},{"metadata":{"_kg_hide-output":true,"trusted":true},"cell_type":"code","source":"! pip install pytorch-pretrained-biggan","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np, pandas as pd, os\nimport os, torch, gc\nfrom PIL import Image\nimport matplotlib.pyplot as plt\nfrom pytorch_pretrained_biggan import BigGAN, truncated_noise_sample\n\n# LOAD PRETRAINED BIGGAN-DEEP-128\nmodel = BigGAN.from_pretrained('biggan-deep-128')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Fun Latent Walk\nBelow are latent walks from dogs to cats. Big GAN recognizes 1000 different category classes. There is a complete list of classes [here][1]. For example, `Labrador retriever` is class 208 and `Tiger` is class 292. By changing the `start class` and `end class` for `class_vector` below you can control what you would like to see morph into what. \n\nThe smooth transistions of the latent walks below demonstrate that Big GAN 128 is not a memorizer GAN.\n\n[1]: https://gist.github.com/yrevar/942d3a0ac09ec9e5eb3a"},{"metadata":{"trusted":true,"_kg_hide-input":false},"cell_type":"code","source":"COLS = 10\nROWS = 10\ntruncation = 0.5\nclass_vector = np.zeros((ROWS*COLS,1000),dtype=np.float32)\nnoise_vector = np.zeros((ROWS*COLS,128),dtype=np.float32)\n\nfor j in range(ROWS):\n    \n    # CHANGE THE FOLLOWING TWO LINES TO CONTROL WHAT CLASS MORPHS TO WHAT CLASS\n    #################\n    # DOGS ARE CLASSES 151 THRU 281, CATS ARE 281 THRU 294\n    # CATEGORY LIST HERE: https://gist.github.com/yrevar/942d3a0ac09ec9e5eb3a\n    class_vector[j*COLS,np.random.randint(151,281)]=1 # START CLASS\n    class_vector[(j+1)*COLS-1,np.random.randint(281,294)]=1 # END CLASS   \n    #################\n    \n    step = class_vector[(j+1)*COLS-1,] - class_vector[j*COLS,]\n    for k in range(1,COLS-1):\n        class_vector[j*COLS+k,] = class_vector[j*COLS,] + (k/(COLS-1))*step\n    noise_vector[j*COLS,:] = truncated_noise_sample(truncation=truncation, batch_size=1)\n    noise_vector[(j+1)*COLS-1,:] = truncated_noise_sample(truncation=truncation, batch_size=1)\n    step = noise_vector[(j+1)*COLS-1,]-noise_vector[j*COLS,]\n    for k in range(1,COLS-1):\n        noise_vector[j*COLS+k,] = noise_vector[j*COLS,] + (k/(COLS-1))*step\n\n# All in tensors\nnoise_vector = torch.from_numpy(noise_vector)\nclass_vector = torch.from_numpy(class_vector)\n\n# If you have a GPU, put everything on cuda\nnoise_vector = noise_vector.to('cuda')\nclass_vector = class_vector.to('cuda')\nmodel.to('cuda')\n\n# Generate an image\nwith torch.no_grad():\n    output = model(noise_vector, class_vector, truncation)\n\n# If you have a GPU put back on CPU\noutput = output.to('cpu')\noutput = output.numpy().transpose(0, 2, 3, 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# DISPLAY LATENT WALK\nplt.figure(figsize=(int(2*COLS),int(2*ROWS)))\nplt.subplots_adjust(hspace=0,wspace=0)\nfor k in range(ROWS*COLS):\n    plt.subplot(ROWS,COLS,k+1)\n    img = Image.fromarray( ((output[k,]+1.)/2.*255).astype('uint8') )\n    plt.axis('off')\n    plt.imshow(img)\nplt.show(img)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Generate 10000 Dog Images\nIf we set the variable `truncation` below to `0.5` then the dog images will look photo realistic but they will also have less variation in the background and pose. The competition metric rewards variation more than quality. Therefore `truncation=0.5` yields LB 34, whereas `trucation=1.0` yields LB 12 even though the latter are less photo realistic. (In the latent walks above, we used `truncation=0.5` and you can see the image quality difference)."},{"metadata":{"trusted":true},"cell_type":"code","source":"os.mkdir('../tmp')\nos.mkdir('../tmp/images/')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# GENERATE SAMPLES\n# TRUNCATION VARIABLE AFFECTS QUALITY AND DIVERSITY\ntruncation=1.0\nfor k in range(100):\n    SAMPLES = 100\n    class_vector = np.zeros((SAMPLES,1000),dtype=np.float32)\n    for x in range(SAMPLES): \n        class_vector[x,np.random.randint(151,281)]=1 #(281,294) are cats\n    # CATEGORY LIST HERE: https://gist.github.com/yrevar/942d3a0ac09ec9e5eb3a\n    noise_vector = truncated_noise_sample(truncation=truncation, batch_size=SAMPLES)\n\n    # All in tensors\n    noise_vector = torch.from_numpy(noise_vector)\n    class_vector = torch.from_numpy(class_vector)\n\n    # If you have a GPU, put everything on cuda\n    noise_vector = noise_vector.to('cuda')\n    class_vector = class_vector.to('cuda')\n    model.to('cuda')\n\n    # Generate an image\n    with torch.no_grad():\n        output = model(noise_vector, class_vector, truncation)\n\n    # If you have a GPU put back on CPU\n    output = output.to('cpu')\n    output = output.numpy().transpose(0, 2, 3, 1)\n    \n    for j in range(100):\n        img = Image.fromarray( ((output[j,]+1.)/2.*255).astype('uint8') )\n        img = img.resize((64,64), Image.ANTIALIAS)\n        img.save('../tmp/images/'+str(k*100+j)+'.png','PNG')\n    #if k%10==0: print(k)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Display Examples"},{"metadata":{"trusted":true},"cell_type":"code","source":"# DISPLAY GENERATED DOGS\nplt.figure(figsize=(15,15))\nplt.subplots_adjust(hspace=0,wspace=0)\nfor k in range(100):\n    plt.subplot(10,10,k+1)\n    img = Image.fromarray( ((output[k,]+1.)/2.*255).astype('uint8') )\n    plt.axis('off')\n    plt.imshow(img)\nplt.show(img)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Calculate LB Score"},{"metadata":{"trusted":true},"cell_type":"code","source":"# CLEAR GPU MEMORY\ndel model, output, class_vector, noise_vector, img; gc.collect()\ntorch.cuda.empty_cache()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"from __future__ import absolute_import, division, print_function\nimport numpy as np\nimport os\nimport gzip, pickle\nimport tensorflow as tf\nfrom scipy import linalg\nimport pathlib\nimport urllib\nimport warnings\nfrom tqdm import tqdm\nfrom PIL import Image\n\nclass KernelEvalException(Exception):\n    pass\n\nmodel_params = {\n    'Inception': {\n        'name': 'Inception', \n        'imsize': 64,\n        'output_layer': 'Pretrained_Net/pool_3:0', \n        'input_layer': 'Pretrained_Net/ExpandDims:0',\n        'output_shape': 2048,\n        'cosine_distance_eps': 0.1\n        }\n}\n\ndef create_model_graph(pth):\n    \"\"\"Creates a graph from saved GraphDef file.\"\"\"\n    # Creates graph from saved graph_def.pb.\n    with tf.gfile.FastGFile( pth, 'rb') as f:\n        graph_def = tf.GraphDef()\n        graph_def.ParseFromString( f.read())\n        _ = tf.import_graph_def( graph_def, name='Pretrained_Net')\n\ndef _get_model_layer(sess, model_name):\n    # layername = 'Pretrained_Net/final_layer/Mean:0'\n    layername = model_params[model_name]['output_layer']\n    layer = sess.graph.get_tensor_by_name(layername)\n    ops = layer.graph.get_operations()\n    for op_idx, op in enumerate(ops):\n        for o in op.outputs:\n            shape = o.get_shape()\n            if shape._dims != []:\n              shape = [s.value for s in shape]\n              new_shape = []\n              for j, s in enumerate(shape):\n                if s == 1 and j == 0:\n                  new_shape.append(None)\n                else:\n                  new_shape.append(s)\n              o.__dict__['_shape_val'] = tf.TensorShape(new_shape)\n    return layer\n\ndef get_activations(images, sess, model_name, batch_size=50, verbose=False):\n    \"\"\"Calculates the activations of the pool_3 layer for all images.\n\n    Params:\n    -- images      : Numpy array of dimension (n_images, hi, wi, 3). The values\n                     must lie between 0 and 256.\n    -- sess        : current session\n    -- batch_size  : the images numpy array is split into batches with batch size\n                     batch_size. A reasonable batch size depends on the disposable hardware.\n    -- verbose    : If set to True and parameter out_step is given, the number of calculated\n                     batches is reported.\n    Returns:\n    -- A numpy array of dimension (num images, 2048) that contains the\n       activations of the given tensor when feeding inception with the query tensor.\n    \"\"\"\n    inception_layer = _get_model_layer(sess, model_name)\n    n_images = images.shape[0]\n    if batch_size > n_images:\n        print(\"warning: batch size is bigger than the data size. setting batch size to data size\")\n        batch_size = n_images\n    n_batches = n_images//batch_size + 1\n    pred_arr = np.empty((n_images,model_params[model_name]['output_shape']))\n    for i in tqdm(range(n_batches)):\n        if verbose:\n            print(\"\\rPropagating batch %d/%d\" % (i+1, n_batches), end=\"\", flush=True)\n        start = i*batch_size\n        if start+batch_size < n_images:\n            end = start+batch_size\n        else:\n            end = n_images\n                    \n        batch = images[start:end]\n        pred = sess.run(inception_layer, {model_params[model_name]['input_layer']: batch})\n        pred_arr[start:end] = pred.reshape(-1,model_params[model_name]['output_shape'])\n    if verbose:\n        print(\" done\")\n    return pred_arr\n\n\n# def calculate_memorization_distance(features1, features2):\n#     neigh = NearestNeighbors(n_neighbors=1, algorithm='kd_tree', metric='euclidean')\n#     neigh.fit(features2) \n#     d, _ = neigh.kneighbors(features1, return_distance=True)\n#     print('d.shape=',d.shape)\n#     return np.mean(d)\n\ndef normalize_rows(x: np.ndarray):\n    \"\"\"\n    function that normalizes each row of the matrix x to have unit length.\n\n    Args:\n     ``x``: A numpy matrix of shape (n, m)\n\n    Returns:\n     ``x``: The normalized (by row) numpy matrix.\n    \"\"\"\n    return np.nan_to_num(x/np.linalg.norm(x, ord=2, axis=1, keepdims=True))\n\n\ndef cosine_distance(features1, features2):\n    # print('rows of zeros in features1 = ',sum(np.sum(features1, axis=1) == 0))\n    # print('rows of zeros in features2 = ',sum(np.sum(features2, axis=1) == 0))\n    features1_nozero = features1[np.sum(features1, axis=1) != 0]\n    features2_nozero = features2[np.sum(features2, axis=1) != 0]\n    norm_f1 = normalize_rows(features1_nozero)\n    norm_f2 = normalize_rows(features2_nozero)\n\n    d = 1.0-np.abs(np.matmul(norm_f1, norm_f2.T))\n    print('d.shape=',d.shape)\n    print('np.min(d, axis=1).shape=',np.min(d, axis=1).shape)\n    mean_min_d = np.mean(np.min(d, axis=1))\n    print('distance=',mean_min_d)\n    return mean_min_d\n\n\ndef distance_thresholding(d, eps):\n    if d < eps:\n        return d\n    else:\n        return 1\n\ndef calculate_frechet_distance(mu1, sigma1, mu2, sigma2, eps=1e-6):\n    \"\"\"Numpy implementation of the Frechet Distance.\n    The Frechet distance between two multivariate Gaussians X_1 ~ N(mu_1, C_1)\n    and X_2 ~ N(mu_2, C_2) is\n            d^2 = ||mu_1 - mu_2||^2 + Tr(C_1 + C_2 - 2*sqrt(C_1*C_2)).\n            \n    Stable version by Dougal J. Sutherland.\n\n    Params:\n    -- mu1 : Numpy array containing the activations of the pool_3 layer of the\n             inception net ( like returned by the function 'get_predictions')\n             for generated samples.\n    -- mu2   : The sample mean over activations of the pool_3 layer, precalcualted\n               on an representive data set.\n    -- sigma1: The covariance matrix over activations of the pool_3 layer for\n               generated samples.\n    -- sigma2: The covariance matrix over activations of the pool_3 layer,\n               precalcualted on an representive data set.\n\n    Returns:\n    --   : The Frechet Distance.\n    \"\"\"\n\n    mu1 = np.atleast_1d(mu1)\n    mu2 = np.atleast_1d(mu2)\n\n    sigma1 = np.atleast_2d(sigma1)\n    sigma2 = np.atleast_2d(sigma2)\n\n    assert mu1.shape == mu2.shape, \"Training and test mean vectors have different lengths\"\n    assert sigma1.shape == sigma2.shape, \"Training and test covariances have different dimensions\"\n\n    diff = mu1 - mu2\n\n    # product might be almost singular\n    covmean, _ = linalg.sqrtm(sigma1.dot(sigma2), disp=False)\n    if not np.isfinite(covmean).all():\n        msg = \"fid calculation produces singular product; adding %s to diagonal of cov estimates\" % eps\n        warnings.warn(msg)\n        offset = np.eye(sigma1.shape[0]) * eps\n        # covmean = linalg.sqrtm((sigma1 + offset).dot(sigma2 + offset))\n        covmean = linalg.sqrtm((sigma1 + offset).dot(sigma2 + offset))\n    \n    # numerical error might give slight imaginary component\n    if np.iscomplexobj(covmean):\n        if not np.allclose(np.diagonal(covmean).imag, 0, atol=1e-3):\n            m = np.max(np.abs(covmean.imag))\n            raise ValueError(\"Imaginary component {}\".format(m))\n        covmean = covmean.real\n\n    # covmean = tf.linalg.sqrtm(tf.linalg.matmul(sigma1,sigma2))\n\n    print('covmean.shape=',covmean.shape)\n    # tr_covmean = tf.linalg.trace(covmean)\n\n    tr_covmean = np.trace(covmean)\n    return diff.dot(diff) + np.trace(sigma1) + np.trace(sigma2) - 2 * tr_covmean\n    # return diff.dot(diff) + tf.linalg.trace(sigma1) + tf.linalg.trace(sigma2) - 2 * tr_covmean\n#-------------------------------------------------------------------------------\n\n\ndef calculate_activation_statistics(images, sess, model_name, batch_size=50, verbose=False):\n    \"\"\"Calculation of the statistics used by the FID.\n    Params:\n    -- images      : Numpy array of dimension (n_images, hi, wi, 3). The values\n                     must lie between 0 and 255.\n    -- sess        : current session\n    -- batch_size  : the images numpy array is split into batches with batch size\n                     batch_size. A reasonable batch size depends on the available hardware.\n    -- verbose     : If set to True and parameter out_step is given, the number of calculated\n                     batches is reported.\n    Returns:\n    -- mu    : The mean over samples of the activations of the pool_3 layer of\n               the incption model.\n    -- sigma : The covariance matrix of the activations of the pool_3 layer of\n               the incption model.\n    \"\"\"\n    act = get_activations(images, sess, model_name, batch_size, verbose)\n    mu = np.mean(act, axis=0)\n    sigma = np.cov(act, rowvar=False)\n    return mu, sigma, act\n    \ndef _handle_path_memorization(path, sess, model_name, is_checksize, is_check_png):\n    path = pathlib.Path(path)\n    files = list(path.glob('*.jpg')) + list(path.glob('*.png'))\n    imsize = model_params[model_name]['imsize']\n\n    # In production we don't resize input images. This is just for demo purpose. \n    x = np.array([np.array(img_read_checks(fn, imsize, is_checksize, imsize, is_check_png)) for fn in files])\n    m, s, features = calculate_activation_statistics(x, sess, model_name)\n    del x #clean up memory\n    return m, s, features\n\n# check for image size\ndef img_read_checks(filename, resize_to, is_checksize=False, check_imsize = 64, is_check_png = False):\n    im = Image.open(str(filename))\n    if is_checksize and im.size != (check_imsize,check_imsize):\n        raise KernelEvalException('The images are not of size '+str(check_imsize))\n    \n    if is_check_png and im.format != 'PNG':\n        raise KernelEvalException('Only PNG images should be submitted.')\n\n    if resize_to is None:\n        return im\n    else:\n        return im.resize((resize_to,resize_to),Image.ANTIALIAS)\n\ndef calculate_kid_given_paths(paths, model_name, model_path, feature_path=None, mm=[], ss=[], ff=[]):\n    ''' Calculates the KID of two paths. '''\n    tf.reset_default_graph()\n    create_model_graph(str(model_path))\n    with tf.Session() as sess:\n        sess.run(tf.global_variables_initializer())\n        m1, s1, features1 = _handle_path_memorization(paths[0], sess, model_name, is_checksize = True, is_check_png = True)\n        if len(mm) != 0:\n            m2 = mm\n            s2 = ss\n            features2 = ff\n        elif feature_path is None:\n            m2, s2, features2 = _handle_path_memorization(paths[1], sess, model_name, is_checksize = False, is_check_png = False)\n        else:\n            with np.load(feature_path) as f:\n                m2, s2, features2 = f['m'], f['s'], f['features']\n\n        print('m1,m2 shape=',(m1.shape,m2.shape),'s1,s2=',(s1.shape,s2.shape))\n        print('starting calculating FID')\n        fid_value = calculate_frechet_distance(m1, s1, m2, s2)\n        print('done with FID, starting distance calculation')\n        distance = cosine_distance(features1, features2)        \n        return fid_value, distance, m2, s2, features2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# COMPUTE LB SCORE\nmm = []; ss =[]; ff = []\nuser_images_unzipped_path = '../tmp/images/'\nimages_path = [user_images_unzipped_path,'../input/generative-dog-images/all-dogs/all-dogs/']\npublic_path = '../input/dog-face-generation-competition-kid-metric-input/classify_image_graph_def.pb'\n\nfid_epsilon = 10e-15\n\nfid_value_public, distance_public, mm, ss, ff = calculate_kid_given_paths(images_path, 'Inception', public_path, mm=mm, ss=ss, ff=ff)\ndistance_public = distance_thresholding(distance_public, model_params['Inception']['cosine_distance_eps'])\nprint(\"FID_public: \", fid_value_public, \"distance_public: \", distance_public, \"multiplied_public: \",\n          fid_value_public /(distance_public + fid_epsilon))","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}