{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Conditional Dog Breed GAN\nThis kernel is a A-C-Ra-LS-DC-GAN. Whoa that's a lot of letters! The A if for [Auxiliary Classifier][8]. The C is for [Conditional GAN][1]. The Ra is for [Relativistic Average GAN][2]. The LS is for [Least Squares GAN][3]. The DC is for [Deep Convolutional GAN][4]!\n![image](http://playagricola.com/Kaggle/GAN2.jpg)\n\nConditional GANs are fun! When we train our GAN, we can associate each image (and seed) with one or more labels (classes). Afterwards, we can request our Generator to draw a dog with certain labels. For example, we can label every training image as either \"facing left\", \"facing center\", or \"facing right\". This is categorical feature one. Next we label every training image as either \"short hair\", \"long hair\", or \"no hair\". This is categorical feature two. Then we can ask our Generator to draw a dog that is \"facing left\" and has \"long hair\". Fun, right?!\n\nIn this kernel, we will use one categorial feature, namely breed. After training our GAN, we can ask our Generator to draw a specific breed of dog! Keep in mind that this kernel is a **work in progress**. The GAN architecture and/or hyperparameters are not neccessarily optimal. Similar to most of you, I'm learning this stuff too. I encourage everyone to fork this kernel and improve it.\n# UPDATE v17\nKernel version 16 scores LB 100. This version scores LB 52. In addition to small changes, the following big changes were made:\n* Crop original images with square inside bounding box plus padding (80x80)\n* Use data augmentation, random crops (64x64)\n* Use dense layer of 121 sigmoid units before output unit\n* Compute classification error on dense layer and add to discriminator's loss\n* Compile training loop as `tf.function` for 2x speedup\n\n# Load and Crop Images\n\n[1]: https://arxiv.org/abs/1411.1784\n[2]: https://arxiv.org/pdf/1807.00734.pdf\n[3]: https://arxiv.org/abs/1611.04076\n[4]: https://arxiv.org/abs/1511.06434\n[5]: https://www.kaggle.com/c/generative-dog-images/discussion/99215\n[6]: https://www.kaggle.com/c/generative-dog-images/discussion/99485\n[7]: https://www.kaggle.com/cdeotte/dog-memorizer-gan\n[8]: https://arxiv.org/abs/1610.09585"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np, pandas as pd, os, time, gc\nimport xml.etree.ElementTree as ET , time\nimport matplotlib.pyplot as plt, zipfile \nfrom PIL import Image \n\n# STOP KERNEL IF IT RUNS FOR MORE THAN 8 HOURS\nkernel_start = time.time()\nLIMIT = 8\n\n# PREPROCESS IMAGES\nComputeLB = True\nDogsOnly = True\nROOT = '../input/generative-dog-images/'\nif not ComputeLB: ROOT = '../input/'\nIMAGES = os.listdir(ROOT + 'all-dogs/all-dogs/')\nbreeds = os.listdir(ROOT + 'annotation/Annotation/') \n\nidxIn = 0; namesIn = []\nimagesIn = np.zeros((25000,80,80,3))\n\n# CROP WITH BOUNDING BOXES TO GET DOGS ONLY\n# https://www.kaggle.com/paulorzp/show-annotations-and-breeds\nif DogsOnly:\n for breed in breeds:\n for dog in os.listdir(ROOT+'annotation/Annotation/'+breed):\n try: img = Image.open(ROOT+'all-dogs/all-dogs/'+dog+'.jpg') \n except: continue \n ww,hh = img.size\n tree = ET.parse(ROOT+'annotation/Annotation/'+breed+'/'+dog)\n root = tree.getroot()\n objects = root.findall('object')\n for o in objects:\n bndbox = o.find('bndbox') \n xmin = int(bndbox.find('xmin').text)\n ymin = int(bndbox.find('ymin').text)\n xmax = int(bndbox.find('xmax').text)\n ymax = int(bndbox.find('ymax').text)\n w = np.min((xmax - xmin, ymax - ymin))\n # ADD PADDING TO CROPS\n EXTRA = w//8\n a1 = EXTRA; a2 = EXTRA; b1 = EXTRA; b2 = EXTRA\n a1 = np.min((a1,xmin)); a2 = np.min((a2,ww-xmin-w))\n b1 = np.min((b1,ymin)); b2 = np.min((b2,hh-ymin-w))\n img2 = img.crop((xmin-a1, ymin-b1, xmin+w+a2, ymin+w+b2))\n img2 = img2.resize((80,80), Image.ANTIALIAS)\n imagesIn[idxIn,:,:,:] = np.asarray(img2)\n namesIn.append(breed) \n #if idxIn%1000==0: print(idxIn)\n idxIn += 1\n \n idx = np.arange(idxIn)\n np.random.shuffle(idx)\n imagesIn = imagesIn[idx,:,:,:]\n namesIn = np.array(namesIn)[idx]\n \n# RANDOMLY CROP FULL IMAGES\nelse:\n for k in range(len(IMAGES)):\n img = Image.open(ROOT + 'all-dogs/all-dogs/' + IMAGES[k])\n w = img.size[0]\n h = img.size[1]\n sz = np.min((w,h))\n a=0; b=0\n if w (tf.Tensor,tf.Tensor):\n x = tf.image.random_flip_left_right(x)\n return (x,y)\n\n# FUNCTION FOR DATA AUGMENTATION\ndef crop(x: tf.Tensor, y:tf.Tensor) -> (tf.Tensor,tf.Tensor):\n x = tf.random_crop(x,size=[64,64,3])\n return (x,y)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"BATCH_SIZE = 32\n\nfrom sklearn import preprocessing\n\nfor i in range(len(namesIn)):\n namesIn[i] = namesIn[i].split('-')[1].lower()\nle = preprocessing.LabelEncoder()\nnamesIn = le.fit_transform(namesIn)\n\nimagesIn = (imagesIn[:idxIn,:,:,:]-127.5)/127.5\nnamesIn = namesIn[:idxIn]\nimagesIn = imagesIn.astype('float32')\nnamesIn = namesIn.astype('int8')\nds = tf.data.Dataset.from_tensor_slices((imagesIn,namesIn)).map(flip).map(crop).batch(BATCH_SIZE,drop_remainder=True)\nprint('TF Version',tf.__version__); print()\nprint('Our TF data pipeline has been built')\nprint(ds)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Generator\nThe Generator receives both a random seed `Z` and class label(s) `C`. It then attempts to draw a dog matching the label(s). We use an `Embedding 120x120` to input the categorical variable breed into our net. We then concatenate it with our length 128 seed and then fully connect both to our first group of 1024x4x4 maps."},{"metadata":{"trusted":true},"cell_type":"code","source":"MAPS = 128\nnoise_dim = 128\n\nfrom tensorflow.keras import layers\nfrom tensorflow.keras.initializers import RandomNormal\ninit = RandomNormal(mean=0.0, stddev=0.02)\n\ndef make_generator():\n seed = tf.keras.Input(shape=((noise_dim,)))\n label = tf.keras.Input(shape=((1,)))\n x = layers.Embedding(120, 120, input_length=1,name='emb')(label)\n x = layers.Flatten()(x)\n x = layers.concatenate([seed,x])\n x = layers.Dense(4*4*MAPS*8, use_bias=False)(x)\n x = layers.Reshape((4, 4, MAPS*8))(x)\n x = layers.BatchNormalization()(x)\n x = layers.ReLU()(x)\n \n x = layers.Conv2DTranspose(MAPS*4, (5, 5), strides=(2, 2), padding='same', kernel_initializer=init, use_bias=False)(x)\n x = layers.BatchNormalization()(x)\n x = layers.ReLU()(x)\n \n x = layers.Conv2DTranspose(MAPS*2, (5, 5), strides=(2, 2), padding='same', kernel_initializer=init, use_bias=False)(x)\n x = layers.BatchNormalization()(x)\n x = layers.ReLU()(x)\n \n x = layers.Conv2DTranspose(MAPS, (5, 5), strides=(2, 2), padding='same', kernel_initializer=init, use_bias=False)(x)\n x = layers.BatchNormalization()(x)\n x = layers.ReLU()(x)\n \n x = layers.Conv2DTranspose(3, (5, 5), strides=(2, 2), padding='same', kernel_initializer=init, use_bias=False, activation='tanh')(x)\n\n model = tf.keras.Model(inputs=[seed,label], outputs=x) \n return model\n\ngenerator = make_generator()\n#generator.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Discriminator\nThe Discriminator receives both an image and class label(s). We can design it however we want. Two examples are pictured below. In this kernel, we build option 1. Likewise, we could have designed the Generator in different ways. Option 1 knows the class while it extracts image features. Option 2 extracts features first and then receives the class information for classfication. (Kernel version 13 uses option 2).\n![image](http://playagricola.com/Kaggle/disc3b.jpg)"},{"metadata":{"trusted":true},"cell_type":"code","source":"GNOISE = 0.25\n\ndef make_discriminator():\n image = tf.keras.Input(shape=((64,64,3)))\n label = tf.keras.Input(shape=((1,)))\n x = layers.Embedding(120, 64*64, input_length=1)(label)\n x = layers.Reshape((64,64,1))(x)\n x = layers.concatenate([image,x])\n \n x = layers.Conv2D(MAPS, (5, 5), strides=(2, 2), padding='same', kernel_initializer=init, use_bias=False)(x)\n x = layers.BatchNormalization()(x)\n #x = layers.GaussianNoise(GNOISE)(x)\n x = layers.LeakyReLU()(x)\n\n x = layers.Conv2D(MAPS*2, (5, 5), strides=(2, 2), padding='same', kernel_initializer=init, use_bias=False)(x)\n x = layers.BatchNormalization()(x)\n #x = layers.GaussianNoise(GNOISE)(x)\n x = layers.LeakyReLU()(x)\n\n x = layers.Conv2D(MAPS*4, (5, 5), strides=(2, 2), padding='same', kernel_initializer=init, use_bias=False)(x)\n x = layers.BatchNormalization()(x)\n #x = layers.GaussianNoise(GNOISE)(x)\n x = layers.LeakyReLU()(x)\n\n x = layers.Conv2D(MAPS*8, (5, 5), strides=(2, 2), padding='same', kernel_initializer=init, use_bias=False)(x)\n x = layers.BatchNormalization()(x)\n #x = layers.GaussianNoise(GNOISE)(x)\n x = layers.LeakyReLU()(x)\n \n x = layers.Flatten()(x)\n x = layers.Dense(121, activation='sigmoid')(x)\n x2 = layers.Dense(1, activation='linear')(x)\n \n model = tf.keras.Model(inputs=[image,label], outputs=[x,x2])\n return model\n\ndiscriminator = make_discriminator()\n#discriminator.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Losses and Optimizers\nTo balance the Generator's and Discriminator's CGAN learning, we found that making the Generator's learning rate larger than the Discriminator's helps. (For basic RaLSGAN, we found that setting them equal was fine). With the additional class information the job of the Discriminator is easier. If the Discriminator knows that it should be receiving an image of a Black Lab and the image is white than it is clearly fake."},{"metadata":{"trusted":true},"cell_type":"code","source":"# THESE LOSS FUNCTIONS ARE UNUSED. THEY ARE REWRITTEN INSIDE TRAINING LOOP BELOW\nfrom tensorflow.contrib.eager.python import tfe\n\n# RaLS Discriminator Loss\ndef RaLS_errD(fake,real):\n return (tf.reduce_mean( (real - tf.reduce_mean(fake,0) - tf.ones_like(real))**2,0 )\n + tf.reduce_mean( (fake - tf.reduce_mean(real,0) + tf.ones_like(real))**2,0 ) )/2.\n\n# RaLS Generator Loss\ndef RaLS_errG(fake,real):\n return (tf.reduce_mean( (real - tf.reduce_mean(fake,0) + tf.ones_like(real))**2,0 )\n + tf.reduce_mean( (fake - tf.reduce_mean(real,0) - tf.ones_like(real))**2,0 ) )/2.\n\n# OPTIMIZER - ADAM\nlearning_rate = tfe.Variable(0.0002)\ngenerator_optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate, beta1=0.5)\ndiscriminator_optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate, beta1=0.5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training Loop"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"DISPLAY_EVERY = 10\n\ndef display_images(model, test_input, labs):\n predictions = model([test_input,labs], training=False)\n fig = plt.figure(figsize=(16,4))\n for i in range(predictions.shape[0]):\n plt.subplot(2, 8, i+1)\n plt.imshow( (predictions[i, :, :, :]+1.)/2. )\n plt.axis('off')\n plt.show()\n \ndef generate_latent_points(latent_dim, n_samples):\n return tf.random.truncated_normal((n_samples,latent_dim))\n\ndef train(dataset, epochs):\n all_gl = np.array([]); all_dl = np.array([])\n \n for epoch in range(epochs):\n start = time.time()\n gl = []; dl = []\n \n # TENSOR FLOW DATA.DATASET HAS A BUG AND WONT SHUFFLE ON ITS OWN :-(\n # https://github.com/tensorflow/tensorflow/issues/27680\n idx = np.arange(idxIn)\n np.random.shuffle(idx)\n dataset = (tf.data.Dataset.from_tensor_slices((imagesIn[idx,:,:,:],namesIn[idx]))\n .map(flip).map(crop).batch(BATCH_SIZE,drop_remainder=True))\n \n # TRAIN ACGAN\n for i,image_batch in enumerate(dataset):\n gg,dd = train_step(image_batch,generator,discriminator,\n generator_optimizer, discriminator_optimizer)\n gl.append(gg); dl.append(dd)\n all_gl = np.append(all_gl,np.array([gl]))\n all_dl = np.append(all_dl,np.array([dl]))\n \n # EXPONENTIALLY DECAY LEARNING RATES\n if epoch>180: learning_rate.assign(learning_rate*0.95)\n \n # DISPLAY PROGRESS\n if epoch%DISPLAY_EVERY==0:\n # PLOT EPOCH LOSS\n plt.figure(figsize=(16,2))\n plt.plot(np.arange(len(gl)),gl,label='Gen_loss')\n plt.plot(np.arange(len(dl)),dl,label='Disc_loss')\n plt.legend()\n plt.title('Epoch '+str(epoch)+' Loss')\n ymax = plt.ylim()[1]\n plt.show()\n \n # PLOT ALL TIME LOSS\n plt.figure(figsize=(16,2))\n plt.plot(np.arange(len(all_gl)),all_gl,label='Gen_loss')\n plt.plot(np.arange(len(all_dl)),all_dl,label='Disc_loss')\n plt.legend()\n plt.ylim((0,np.min([1.1*np.max(all_gl),2*ymax])))\n plt.title('All Time Loss')\n plt.show()\n\n # DISPLAY IMAGES FROM TRAIN PROGRESS\n seed = generate_latent_points(noise_dim, num_examples)\n labs = tf.cast(120*tf.random.uniform((num_examples,1)),tf.int8)\n display_images(generator, seed, labs)\n \n # PRINT STATS\n print('EPOCH',epoch,'took',np.round(time.time()-start,1),'sec')\n print('Gen_loss mean=',np.mean(gl),'std=',np.std(gl))\n print('Disc_loss mean=',np.mean(dl),'std=',np.std(dl))\n print('Learning rate = ',end='')\n tf.print(discriminator_optimizer._lr)\n \n x = gc.collect()\n tt = np.round( (time.time() - kernel_start)/60,1 )\n if tt > LIMIT*60: break","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"EPOCHS = 250\nnum_examples = 16\n\n@ tf.function\ndef train_step(images,generator,discriminator,generator_optimizer,discriminator_optimizer):\n \n bce = tf.keras.losses.BinaryCrossentropy(from_logits=True,label_smoothing=0.4)\n bce2 = tf.keras.losses.BinaryCrossentropy(from_logits=False,label_smoothing=0.4)\n noise = tf.random.normal((32,128)) # update noise_dim here\n labs = tf.cast(120*tf.random.uniform((32,)),tf.int32)\n \n # USE GRADIENT TAPE TO CALCULATE GRADIENTS\n with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape: \n generated_images = generator([noise,labs], training=True)\n real_cat, real_output = discriminator([images[0],images[1]], training=True)\n fake_cat, fake_output = discriminator([generated_images,labs], training=True)\n \n # GENERATOR LOSS \n gen_loss = (tf.reduce_mean( (real_output - tf.reduce_mean(fake_output,0) + tf.ones_like(real_output))**2,0 )\n + tf.reduce_mean( (fake_output - tf.reduce_mean(real_output,0) - tf.ones_like(real_output))**2,0 ) )/2.\n \n # DISCRIMINATOR LOSS\n disc_loss = bce(tf.ones_like(real_output), real_output) + bce(tf.zeros_like(fake_output), fake_output) \n real_cat2 = tf.one_hot(tf.cast(images[1],tf.int32),121,dtype=tf.int32)\n fake_cat2 = tf.one_hot(120*tf.ones((32,),tf.int32),121,dtype=tf.int32)\n disc_loss += bce2(real_cat2,real_cat) + bce2(fake_cat2,fake_cat) \n \n # BACK PROPAGATE ERROR\n gradients_of_generator = gen_tape.gradient(gen_loss, generator.trainable_variables)\n gradients_of_discriminator = disc_tape.gradient(disc_loss, discriminator.trainable_variables)\n generator_optimizer.apply_gradients(zip(gradients_of_generator, generator.trainable_variables))\n discriminator_optimizer.apply_gradients(zip(gradients_of_discriminator, discriminator.trainable_variables))\n \n return gen_loss, disc_loss\n\nprint('Training started. Displaying every '+str(DISPLAY_EVERY)+'th epoch.')\ntrain(ds, EPOCHS)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Display Generated Dog Breeds\nFor each breed, we will ask our Generator to draw us 10 dog pictures of that breed by feeding into our Generator both 10 random seeds `Z` of length 100 and a breed number `C` from 0 to 119 inclusive. Because this CGAN isn't optimal yet, some breeds may only output 1 image repeatedly. This is called `Mode Collapse`. Below we won't display the breeds with mode collapse. For each breed below we display one row of real pictures above two row of fake pictures. Pretty cool, huh?!"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"mse = tf.keras.losses.MeanSquaredError()\n\nprint('Display Random Dogs by Breed')\nprint()\nfor j in np.random.randint(0,120,25):\n # GENERATE DOGS\n seed = generate_latent_points(noise_dim, 10)\n labs = tf.cast( j*np.ones((10,1)), tf.int8)\n predictions = generator([seed,labs], training=False); d = 0 \n # GET BREED NAME \n br = np.argwhere( namesIn==j ).flatten()\n bd = le.inverse_transform(np.array([j]))[0].capitalize()\n # CALCULATE VARIETY\n for k in range(4): d += mse(predictions[k,:,:,:],predictions[k+1,:,:,:]) \n d = np.round( np.array(d),1 )\n if d<1.0: \n print(bd,'had mode collapse. No display. (variety =',d,')')\n continue\n # DISPLAY DOGS\n print(bd,'REAL DOGS on top. FAKE DOGS on bottom. (variety =',d,')')\n plt.figure(figsize=(15,9))\n for i in range(5):\n plt.subplot(3,5,i+1)\n plt.imshow( (imagesIn[br[i],:,:,:]+1.)/2. )\n plt.axis('off')\n for i in range(10):\n plt.subplot(3,5,i+6)\n plt.imshow( (predictions[i,:,:,:]+1.)/2. )\n plt.axis('off')\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Submit to Kaggle\nWe will ask our Generative Network to draw 10000 random dog images from whatever random breeds. Alternatively we could ask for specifically 83 of each breed to guarentee variety and increased FID score."},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"seed = generate_latent_points(noise_dim, 100)\nlabs = tf.cast(120*tf.random.uniform((100,1)),tf.int8)\npredictions = generator([seed,labs], training=False)\nplt.figure(figsize=(20,20))\nplt.subplots_adjust(wspace=0,hspace=0)\nfor k in range(100):\n plt.subplot(10,10,k+1)\n plt.imshow( (predictions[k,:,:,:]+1.)/2. )\n plt.axis('off')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# SUBMIT 84 IMAGES OF EACH OF 119 BREEDS and 4 of breed 120\nz = zipfile.PyZipFile('images.zip', mode='w')\nfor i in range(120):\n ct = 84\n if i==119: ct=4\n seed = generate_latent_points(noise_dim, ct)\n labs = tf.cast( i*np.ones((ct,1)), tf.int8)\n predictions = generator([seed,labs], training=False)\n predictions = 255*((np.array(predictions)+1.)/2.)\n for j in range(ct):\n img = Image.fromarray( predictions[j,:,:,:].astype('uint8') )\n f = str(i*84+j)+'.png'\n img.save(f,'PNG'); z.write(f); os.remove(f)\n #if (i%10==0)|(i==119): print(i*84)\nz.close()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Calculate LB Score\nIf you wish to compute LB, you must add the LB metric dataset [here][1] to this kernel and change the boolean variable in the first cell block.\n\n[1]: https://www.kaggle.com/wendykan/dog-face-generation-competition-kid-metric-input"},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"from __future__ import absolute_import, division, print_function\nimport numpy as np\nimport os\nimport gzip, pickle\nimport tensorflow as tf\nfrom scipy import linalg\nimport pathlib\nimport urllib\nimport warnings\nfrom tqdm import tqdm\nfrom PIL import Image\n\nclass KernelEvalException(Exception):\n pass\n\nmodel_params = {\n 'Inception': {\n 'name': 'Inception', \n 'imsize': 64,\n 'output_layer': 'Pretrained_Net/pool_3:0', \n 'input_layer': 'Pretrained_Net/ExpandDims:0',\n 'output_shape': 2048,\n 'cosine_distance_eps': 0.1\n }\n}\n\ndef create_model_graph(pth):\n \"\"\"Creates a graph from saved GraphDef file.\"\"\"\n # Creates graph from saved graph_def.pb.\n with tf.gfile.FastGFile( pth, 'rb') as f:\n graph_def = tf.GraphDef()\n graph_def.ParseFromString( f.read())\n _ = tf.import_graph_def( graph_def, name='Pretrained_Net')\n\ndef _get_model_layer(sess, model_name):\n # layername = 'Pretrained_Net/final_layer/Mean:0'\n layername = model_params[model_name]['output_layer']\n layer = sess.graph.get_tensor_by_name(layername)\n ops = layer.graph.get_operations()\n for op_idx, op in enumerate(ops):\n for o in op.outputs:\n shape = o.get_shape()\n if shape._dims != []:\n shape = [s.value for s in shape]\n new_shape = []\n for j, s in enumerate(shape):\n if s == 1 and j == 0:\n new_shape.append(None)\n else:\n new_shape.append(s)\n o.__dict__['_shape_val'] = tf.TensorShape(new_shape)\n return layer\n\ndef get_activations(images, sess, model_name, batch_size=50, verbose=False):\n \"\"\"Calculates the activations of the pool_3 layer for all images.\n\n Params:\n -- images : Numpy array of dimension (n_images, hi, wi, 3). The values\n must lie between 0 and 256.\n -- sess : current session\n -- batch_size : the images numpy array is split into batches with batch size\n batch_size. A reasonable batch size depends on the disposable hardware.\n -- verbose : If set to True and parameter out_step is given, the number of calculated\n batches is reported.\n Returns:\n -- A numpy array of dimension (num images, 2048) that contains the\n activations of the given tensor when feeding inception with the query tensor.\n \"\"\"\n inception_layer = _get_model_layer(sess, model_name)\n n_images = images.shape[0]\n if batch_size > n_images:\n print(\"warning: batch size is bigger than the data size. setting batch size to data size\")\n batch_size = n_images\n n_batches = n_images//batch_size + 1\n pred_arr = np.empty((n_images,model_params[model_name]['output_shape']))\n for i in tqdm(range(n_batches)):\n if verbose:\n print(\"\\rPropagating batch %d/%d\" % (i+1, n_batches), end=\"\", flush=True)\n start = i*batch_size\n if start+batch_size < n_images:\n end = start+batch_size\n else:\n end = n_images\n \n batch = images[start:end]\n pred = sess.run(inception_layer, {model_params[model_name]['input_layer']: batch})\n pred_arr[start:end] = pred.reshape(-1,model_params[model_name]['output_shape'])\n if verbose:\n print(\" done\")\n return pred_arr\n\n\n# def calculate_memorization_distance(features1, features2):\n# neigh = NearestNeighbors(n_neighbors=1, algorithm='kd_tree', metric='euclidean')\n# neigh.fit(features2) \n# d, _ = neigh.kneighbors(features1, return_distance=True)\n# print('d.shape=',d.shape)\n# return np.mean(d)\n\ndef normalize_rows(x: np.ndarray):\n \"\"\"\n function that normalizes each row of the matrix x to have unit length.\n\n Args:\n ``x``: A numpy matrix of shape (n, m)\n\n Returns:\n ``x``: The normalized (by row) numpy matrix.\n \"\"\"\n return np.nan_to_num(x/np.linalg.norm(x, ord=2, axis=1, keepdims=True))\n\n\ndef cosine_distance(features1, features2):\n # print('rows of zeros in features1 = ',sum(np.sum(features1, axis=1) == 0))\n # print('rows of zeros in features2 = ',sum(np.sum(features2, axis=1) == 0))\n features1_nozero = features1[np.sum(features1, axis=1) != 0]\n features2_nozero = features2[np.sum(features2, axis=1) != 0]\n norm_f1 = normalize_rows(features1_nozero)\n norm_f2 = normalize_rows(features2_nozero)\n\n d = 1.0-np.abs(np.matmul(norm_f1, norm_f2.T))\n print('d.shape=',d.shape)\n print('np.min(d, axis=1).shape=',np.min(d, axis=1).shape)\n mean_min_d = np.mean(np.min(d, axis=1))\n print('distance=',mean_min_d)\n return mean_min_d\n\n\ndef distance_thresholding(d, eps):\n if d < eps:\n return d\n else:\n return 1\n\ndef calculate_frechet_distance(mu1, sigma1, mu2, sigma2, eps=1e-6):\n \"\"\"Numpy implementation of the Frechet Distance.\n The Frechet distance between two multivariate Gaussians X_1 ~ N(mu_1, C_1)\n and X_2 ~ N(mu_2, C_2) is\n d^2 = ||mu_1 - mu_2||^2 + Tr(C_1 + C_2 - 2*sqrt(C_1*C_2)).\n \n Stable version by Dougal J. Sutherland.\n\n Params:\n -- mu1 : Numpy array containing the activations of the pool_3 layer of the\n inception net ( like returned by the function 'get_predictions')\n for generated samples.\n -- mu2 : The sample mean over activations of the pool_3 layer, precalcualted\n on an representive data set.\n -- sigma1: The covariance matrix over activations of the pool_3 layer for\n generated samples.\n -- sigma2: The covariance matrix over activations of the pool_3 layer,\n precalcualted on an representive data set.\n\n Returns:\n -- : The Frechet Distance.\n \"\"\"\n\n mu1 = np.atleast_1d(mu1)\n mu2 = np.atleast_1d(mu2)\n\n sigma1 = np.atleast_2d(sigma1)\n sigma2 = np.atleast_2d(sigma2)\n\n assert mu1.shape == mu2.shape, \"Training and test mean vectors have different lengths\"\n assert sigma1.shape == sigma2.shape, \"Training and test covariances have different dimensions\"\n\n diff = mu1 - mu2\n\n # product might be almost singular\n covmean, _ = linalg.sqrtm(sigma1.dot(sigma2), disp=False)\n if not np.isfinite(covmean).all():\n msg = \"fid calculation produces singular product; adding %s to diagonal of cov estimates\" % eps\n warnings.warn(msg)\n offset = np.eye(sigma1.shape[0]) * eps\n # covmean = linalg.sqrtm((sigma1 + offset).dot(sigma2 + offset))\n covmean = linalg.sqrtm((sigma1 + offset).dot(sigma2 + offset))\n \n # numerical error might give slight imaginary component\n if np.iscomplexobj(covmean):\n if not np.allclose(np.diagonal(covmean).imag, 0, atol=1e-3):\n m = np.max(np.abs(covmean.imag))\n raise ValueError(\"Imaginary component {}\".format(m))\n covmean = covmean.real\n\n # covmean = tf.linalg.sqrtm(tf.linalg.matmul(sigma1,sigma2))\n\n print('covmean.shape=',covmean.shape)\n # tr_covmean = tf.linalg.trace(covmean)\n\n tr_covmean = np.trace(covmean)\n return diff.dot(diff) + np.trace(sigma1) + np.trace(sigma2) - 2 * tr_covmean\n # return diff.dot(diff) + tf.linalg.trace(sigma1) + tf.linalg.trace(sigma2) - 2 * tr_covmean\n#-------------------------------------------------------------------------------\n\n\ndef calculate_activation_statistics(images, sess, model_name, batch_size=50, verbose=False):\n \"\"\"Calculation of the statistics used by the FID.\n Params:\n -- images : Numpy array of dimension (n_images, hi, wi, 3). The values\n must lie between 0 and 255.\n -- sess : current session\n -- batch_size : the images numpy array is split into batches with batch size\n batch_size. A reasonable batch size depends on the available hardware.\n -- verbose : If set to True and parameter out_step is given, the number of calculated\n batches is reported.\n Returns:\n -- mu : The mean over samples of the activations of the pool_3 layer of\n the incption model.\n -- sigma : The covariance matrix of the activations of the pool_3 layer of\n the incption model.\n \"\"\"\n act = get_activations(images, sess, model_name, batch_size, verbose)\n mu = np.mean(act, axis=0)\n sigma = np.cov(act, rowvar=False)\n return mu, sigma, act\n \ndef _handle_path_memorization(path, sess, model_name, is_checksize, is_check_png):\n path = pathlib.Path(path)\n files = list(path.glob('*.jpg')) + list(path.glob('*.png'))\n imsize = model_params[model_name]['imsize']\n\n # In production we don't resize input images. This is just for demo purpose. \n x = np.array([np.array(img_read_checks(fn, imsize, is_checksize, imsize, is_check_png)) for fn in files])\n m, s, features = calculate_activation_statistics(x, sess, model_name)\n del x #clean up memory\n return m, s, features\n\n# check for image size\ndef img_read_checks(filename, resize_to, is_checksize=False, check_imsize = 64, is_check_png = False):\n im = Image.open(str(filename))\n if is_checksize and im.size != (check_imsize,check_imsize):\n raise KernelEvalException('The images are not of size '+str(check_imsize))\n \n if is_check_png and im.format != 'PNG':\n raise KernelEvalException('Only PNG images should be submitted.')\n\n if resize_to is None:\n return im\n else:\n return im.resize((resize_to,resize_to),Image.ANTIALIAS)\n\ndef calculate_kid_given_paths(paths, model_name, model_path, feature_path=None, mm=[], ss=[], ff=[]):\n ''' Calculates the KID of two paths. '''\n tf.reset_default_graph()\n create_model_graph(str(model_path))\n with tf.Session() as sess:\n sess.run(tf.global_variables_initializer())\n m1, s1, features1 = _handle_path_memorization(paths[0], sess, model_name, is_checksize = True, is_check_png = True)\n if len(mm) != 0:\n m2 = mm\n s2 = ss\n features2 = ff\n elif feature_path is None:\n m2, s2, features2 = _handle_path_memorization(paths[1], sess, model_name, is_checksize = False, is_check_png = False)\n else:\n with np.load(feature_path) as f:\n m2, s2, features2 = f['m'], f['s'], f['features']\n\n print('m1,m2 shape=',(m1.shape,m2.shape),'s1,s2=',(s1.shape,s2.shape))\n print('starting calculating FID')\n fid_value = calculate_frechet_distance(m1, s1, m2, s2)\n print('done with FID, starting distance calculation')\n distance = cosine_distance(features1, features2) \n return fid_value, distance, m2, s2, features2","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"if ComputeLB:\n \n # UNCOMPRESS OUR IMGAES\n with zipfile.ZipFile(\"../working/images.zip\",\"r\") as z:\n z.extractall(\"../tmp/images2/\")\n\n # COMPUTE LB SCORE\n m2 = []; s2 =[]; f2 = []\n user_images_unzipped_path = '../tmp/images2/'\n images_path = [user_images_unzipped_path,'../input/generative-dog-images/all-dogs/all-dogs/']\n public_path = '../input/dog-face-generation-competition-kid-metric-input/classify_image_graph_def.pb'\n\n fid_epsilon = 10e-15\n\n fid_value_public, distance_public, m2, s2, f2 = calculate_kid_given_paths(images_path, 'Inception', public_path, mm=m2, ss=s2, ff=f2)\n distance_public = distance_thresholding(distance_public, model_params['Inception']['cosine_distance_eps'])\n print(\"FID_public: \", fid_value_public, \"distance_public: \", distance_public, \"multiplied_public: \",\n fid_value_public /(distance_public + fid_epsilon))\n \n # REMOVE FILES TO PREVENT KERNEL ERROR OF TOO MANY FILES\n ! rm -r ../tmp","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}