{"cells":[{"metadata":{},"cell_type":"markdown","source":"<span style=\"font-family:Papyrus; font-size:2.5em;\">DCGAN (Deep convolutional generative adversarial networks)</span>\n---\n![](https://cdn-images-1.medium.com/max/1600/0*yeBUm7SfMaHjnFd2.gif)\n<br>\n\n### <span style=\"color:green\">Estable version V50</span>"},{"metadata":{},"cell_type":"markdown","source":"# What is DCGAN?\n\n[Original Paper](https://arxiv.org/abs/1511.06434) proposed by Radford.\n\nOne of the most interesting parts of GANs is the design of the **Generator**. The Generator network is able to take random noise and map it into images such that the discriminator cannot tell which images came from the dataset and which images came from the generator.\n\nThis is a very interesting application of neural networks. Typically neural nets map input into a binary output, (1 or 0), maybe a regression output, (some real-valued number), or even multiple categorical outputs, (such as MNIST or CIFAR-10/100).\nWe will see how a neural net maps from random noise to an image matrix and how using Convolutional Layers in the generator network produces better results.\n\nIn addition, although GAN is known for its difficulty in learning, [this paper](https://arxiv.org/abs/1511.06434) proposed by Radford. introduces various techniques for successful learning:\n\n- Convert max-pooling layers to convolution layers\n- Convert fully connected layers to global average pooling layers in the discriminator\n- Use batch normalization layers in the generator and the discriminator\n- Use leaky ReLU activation functions in the discriminator\n\n---\n\nDCGAN is one of the popular and successful network design for GAN. It mainly composes of convolution layers without max pooling or fully connected layers. It uses convolutional stride and transposed convolution for the downsampling and the upsampling. The figure below is the network design for the generator.\n\n![](https://cdn-images-1.medium.com/max/800/1*rdXKdyfNjorzP10ZA3yNmQ.png)\n\nThis is the DCGAN generator presented in the LSUN scene modeling paper. This network takes in a 100x1 noise vector, denoted z, and maps it into the G(Z) output which is 64x64x3.\n\nThis architecture is especially interesting the way the first layer expands the random noise. The network goes from 100x1 to 1024x4x4! This layer is denoted ‘project and reshape’.\n\nWe see that following this layer, classical convolutional layers are applied which reshape the network with the (N+P — F)/S + 1 equation classically taught with convolutional layers. In the diagram above we can see that the N parameter, (Height/Width), goes from 4 to 8 to 16 to 32, it doesn’t appear that there is any padding, the kernel filter parameter F is 5x5, and the stride is 2. You may find this equation to be useful for designing your own convolutional layers for customized output sizes.\n\nWe see the network goes from:\n\n```100x1 → 1024x4x4 → 512x8x8 → 256x16x16 → 128x32x32 → 64x64x3```\n\n![](https://cdn-images-1.medium.com/max/800/1*bNjBJm6827sRXvzmcjGTDQ.png)\n\nAbove is the output from the network presented in the paper, citing that this came after 5 epochs of training. Pretty impressive stuff.\n\n---\n\nHere is the **summary** of DCGAN:\n\n- Replace all max pooling with convolutional stride\n- Use transposed convolution for upsampling.\n- Eliminate fully connected layers.\n- Use Batch normalization except the output layer for the generator and the input layer of the discriminator.\n- Use ReLU in the generator except for the output which uses tanh.\n- Use LeakyReLU in the discriminator.\n\n\n---\n\nHere are the **tuning tips** quote directly from the paper.\n\n> All models were trained with mini-batch stochastic gradient descent (SGD) with a mini-batch size of 128. All weights were initialized from a zero-centered Normal distribution with standard deviation 0.02. In the LeakyReLU, the slope of the leak was set to 0.2 in all models. While previous GAN work has used momentum to accelerate training, we used the Adam optimizer with tuned hyperparameters. We found the suggested learning rate of 0.001, to be too high, using 0.0002 instead. Additionally, we found leaving the momentum term β1 at the suggested value of 0.9 resulted in training oscillation and instability while reducing it to 0.5 helped stabilize training."},{"metadata":{},"cell_type":"markdown","source":"<br>\n# Tensorflow DCGAN"},{"metadata":{},"cell_type":"markdown","source":"### Load packages"},{"metadata":{"_kg_hide-input":true,"_uuid":"267d5504009eca2b809a2691131366baebe98436","trusted":false},"cell_type":"code","source":"import os\nimport time\nimport tensorflow as tf\nimport numpy as np\nfrom glob import glob\nimport datetime\nimport random\nimport PIL\nfrom PIL import Image\nimport matplotlib.pyplot as plt\nfrom tqdm import tqdm_notebook as tqdm\n%matplotlib inline\nimport urllib\nimport tarfile\nimport xml.etree.ElementTree as ET\nfrom imageio import imread, imsave, mimsave\nimport shutil\nimport cv2\nimport glob\nfrom imageio import imread, imsave, mimsave","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load and Crop Data\n\nCheck this kernels:\n\n- [Crop images using bounding box](https://www.kaggle.com/whizzkid/crop-images-using-bounding-box)\n- [Dog Memorizer GAN](https://www.kaggle.com/cdeotte/dog-memorizer-gan)"},{"metadata":{"_kg_hide-input":true,"trusted":false},"cell_type":"code","source":"!ls ../input/","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"root_images = \"../input/generative-dog-images/all-dogs/all-dogs/\"\nroot_annots = \"../input/generative-dog-images/annotation/Annotation/\"\nINPUT_DATA_DIR = \"../input/generative-dog-images/all-dogs/all-dogs/\"\nIMG_DIR = \"images\"\nComputeLB = False\nDogsOnly = True","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Processing**"},{"metadata":{"_kg_hide-input":true,"trusted":false},"cell_type":"code","source":"import numpy as np, pandas as pd, os\nimport xml.etree.ElementTree as ET \nimport matplotlib.pyplot as plt, zipfile \nfrom PIL import Image \nfrom glob import glob\n\nROOT = '../input/generative-dog-images/'\nif not ComputeLB: ROOT = '../input/'\nIMAGES = os.listdir(ROOT + 'all-dogs/all-dogs/')\nbreeds = os.listdir(ROOT + 'annotation/Annotation/') \n\nidxIn = 0; namesIn = []\nimagesIn = np.zeros((25000,64,64,3))\n\n# CROP WITH BOUNDING BOXES TO GET DOGS ONLY\n# https://www.kaggle.com/paulorzp/show-annotations-and-breeds\nif DogsOnly:\n    for breed in breeds:\n        for dog in os.listdir(ROOT+'annotation/Annotation/'+breed):\n            try: img = Image.open(ROOT+'all-dogs/all-dogs/'+dog+'.jpg') \n            except: continue           \n            tree = ET.parse(ROOT+'annotation/Annotation/'+breed+'/'+dog)\n            root = tree.getroot()\n            objects = root.findall('object')\n            for o in objects:\n                bndbox = o.find('bndbox') \n                xmin = int(bndbox.find('xmin').text)\n                ymin = int(bndbox.find('ymin').text)\n                xmax = int(bndbox.find('xmax').text)\n                ymax = int(bndbox.find('ymax').text)\n                w = np.min((xmax - xmin, ymax - ymin))\n                img2 = img.crop((xmin, ymin, xmin+w, ymin+w))\n                img2 = img2.resize((64,64), Image.ANTIALIAS)\n                imagesIn[idxIn,:,:,:] = np.asarray(img2)\n                #if idxIn%1000==0: print(idxIn)\n                namesIn.append(breed)\n                idxIn += 1\n    idx = np.arange(idxIn)\n    np.random.shuffle(idx)\n    imagesIn = imagesIn[idx,:,:,:]\n    namesIn = np.array(namesIn)[idx]\n    \n# RANDOMLY CROP FULL IMAGES\nelse:\n    x = np.random.choice(np.arange(20579),10000)\n    for k in range(len(x)):\n        img = Image.open(ROOT + 'all-dogs/all-dogs/' + IMAGES[x[k]])\n        w = img.size[0]\n        h = img.size[1]\n        sz = np.min((w,h))\n        a=0; b=0\n        if w<h: b = (h-sz)//2\n        else: a = (w-sz)//2\n        img = img.crop((0+a, 0+b, sz+a, sz+b))  \n        img = img.resize((64,64), Image.ANTIALIAS)   \n        imagesIn[idxIn,:,:,:] = np.asarray(img)\n        namesIn.append(IMAGES[x[k]])\n        if idxIn%1000==0: print(idxIn)\n        idxIn += 1\n    \n# DISPLAY CROPPED IMAGES\nx = np.random.randint(0,idxIn,25)\nfor k in range(5):\n    plt.figure(figsize=(15,3))\n    for j in range(5):\n        plt.subplot(1,5,j+1)\n        img = Image.fromarray( imagesIn[x[k*5+j],:,:,:].astype('uint8') )\n        plt.axis('off')\n        if not DogsOnly: plt.title(namesIn[x[k*5+j]],fontsize=11)\n        else: plt.title(namesIn[x[k*5+j]].split('-')[1],fontsize=11)\n        plt.imshow(img)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"imagesIn.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<br>\n# Generator"},{"metadata":{},"cell_type":"markdown","source":"The original **Generator** (from the Paper) was:\n\n```100x1 → 1024x4x4 → 512x8x8 → 256x16x16 → 128x32x32 → 64x64x3```\n![](https://cdn-images-1.medium.com/max/800/1*rdXKdyfNjorzP10ZA3yNmQ.png)\n\nMy version is:\n\n```.... → 512x4x4 → 256x8x8 → 128x16x16 → 64x32x32 → 64x64x32 → 64x64x32```"},{"metadata":{"_uuid":"fc376df46433261bfeb643a95793718a9d969ed1","trusted":false},"cell_type":"code","source":"def generator(z, output_channel_dim, training):\n    with tf.variable_scope(\"generator\", reuse= not training):\n        \n        # 4x4x512\n        fully_connected = tf.layers.dense(z, 4*4*512)\n        fully_connected = tf.reshape(fully_connected, (-1, 4, 4, 512))\n        fully_connected = tf.nn.leaky_relu(fully_connected)\n\n        # 4x4x512 -> 8x8x256\n        trans_conv1 = tf.layers.conv2d_transpose(inputs=fully_connected,\n                                                 filters=256,\n                                                 kernel_size=[5,5],\n                                                 strides=[2,2],\n                                                 padding=\"SAME\",\n                                                 kernel_initializer=tf.truncated_normal_initializer(stddev=WEIGHT_INIT_STDDEV),\n                                                 name=\"trans_conv1\")\n        batch_trans_conv1 = tf.layers.batch_normalization(inputs = trans_conv1,\n                                                          training=training,\n                                                          epsilon=EPSILON,\n                                                          name=\"batch_trans_conv1\")\n        trans_conv1_out = tf.nn.leaky_relu(batch_trans_conv1,\n                                           name=\"trans_conv1_out\")\n        \n        # 8x8x256 -> 16x16x128\n        trans_conv2 = tf.layers.conv2d_transpose(inputs=trans_conv1_out,\n                                                 filters=128,\n                                                 kernel_size=[5,5],\n                                                 strides=[2,2],\n                                                 padding=\"SAME\",\n                                                 kernel_initializer=tf.truncated_normal_initializer(stddev=WEIGHT_INIT_STDDEV),\n                                                 name=\"trans_conv2\")\n        batch_trans_conv2 = tf.layers.batch_normalization(inputs = trans_conv2,\n                                                          training=training,\n                                                          epsilon=EPSILON,\n                                                          name=\"batch_trans_conv2\")\n        trans_conv2_out = tf.nn.leaky_relu(batch_trans_conv2,\n                                           name=\"trans_conv2_out\")\n        \n        # 16x16x128 -> 32x32x64\n        trans_conv3 = tf.layers.conv2d_transpose(inputs=trans_conv2_out,\n                                                 filters=64,\n                                                 kernel_size=[5,5],\n                                                 strides=[2,2],\n                                                 padding=\"SAME\",\n                                                 kernel_initializer=tf.truncated_normal_initializer(stddev=WEIGHT_INIT_STDDEV),\n                                                 name=\"trans_conv3\")\n        batch_trans_conv3 = tf.layers.batch_normalization(inputs = trans_conv3,\n                                                          training=training,\n                                                          epsilon=EPSILON,\n                                                          name=\"batch_trans_conv3\")\n        trans_conv3_out = tf.nn.leaky_relu(batch_trans_conv3,\n                                           name=\"trans_conv3_out\")\n        \n\n        # 32x32x64 -> 64x64x32\n        trans_conv4 = tf.layers.conv2d_transpose(inputs=trans_conv3_out,\n                                                 filters=32,\n                                                 kernel_size=[5,5],\n                                                 strides=[2,2],\n                                                 padding=\"SAME\",\n                                                 kernel_initializer=tf.truncated_normal_initializer(stddev=WEIGHT_INIT_STDDEV),\n                                                 name=\"trans_conv4\")\n        batch_trans_conv4 = tf.layers.batch_normalization(inputs = trans_conv4,\n                                                          training=training,\n                                                          epsilon=EPSILON,\n                                                          name=\"batch_trans_conv4\")\n        trans_conv4_out = tf.nn.leaky_relu(batch_trans_conv4,\n                                           name=\"trans_conv4_out\")\n        \n        # 64x64x32 -> 64x64x3\n        logits = tf.layers.conv2d_transpose(inputs=trans_conv4_out,\n                                            filters=3,\n                                            kernel_size=[5,5],\n                                            strides=[1,1],\n                                            padding=\"SAME\",\n                                            kernel_initializer=tf.truncated_normal_initializer(stddev=WEIGHT_INIT_STDDEV),\n                                            name=\"logits\")\n        out = tf.tanh(logits, name=\"out\")\n        return out","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Discriminator"},{"metadata":{"_uuid":"ba53a4bb09dbcd57d3e3392f74ccd054ecf23ecb","trusted":false},"cell_type":"code","source":"def discriminator(x, reuse):\n    with tf.variable_scope(\"discriminator\", reuse=reuse): \n        \n        # 64x64x3 -> 32x32x32\n        conv1 = tf.layers.conv2d(inputs=x,\n                                 filters=32,\n                                 kernel_size=[5,5],\n                                 strides=[2,2],\n                                 padding=\"SAME\",\n                                 kernel_initializer=tf.truncated_normal_initializer(stddev=WEIGHT_INIT_STDDEV),\n                                 name='conv1')\n        batch_norm1 = tf.layers.batch_normalization(conv1,\n                                                    training=True,\n                                                    epsilon=EPSILON,\n                                                    name='batch_norm1')\n        conv1_out = tf.nn.leaky_relu(batch_norm1,\n                                     name=\"conv1_out\")\n        \n        # 32x32x32 -> 16x16x64\n        conv2 = tf.layers.conv2d(inputs=conv1_out,\n                                 filters=64,\n                                 kernel_size=[5, 5],\n                                 strides=[2, 2],\n                                 padding=\"SAME\",\n                                 kernel_initializer=tf.truncated_normal_initializer(stddev=WEIGHT_INIT_STDDEV),\n                                 name='conv2')\n        batch_norm2 = tf.layers.batch_normalization(conv2,\n                                                    training=True,\n                                                    epsilon=EPSILON,\n                                                    name='batch_norm2')\n        conv2_out = tf.nn.leaky_relu(batch_norm2,\n                                     name=\"conv2_out\")\n        \n        # 16x16x64 -> 8x8x128\n        conv3 = tf.layers.conv2d(inputs=conv2_out,\n                                 filters=128,\n                                 kernel_size=[5, 5],\n                                 strides=[2, 2],\n                                 padding=\"SAME\",\n                                 kernel_initializer=tf.truncated_normal_initializer(stddev=WEIGHT_INIT_STDDEV),\n                                 name='conv3')\n        batch_norm3 = tf.layers.batch_normalization(conv3,\n                                                    training=True,\n                                                    epsilon=EPSILON,\n                                                    name='batch_norm3')\n        conv3_out = tf.nn.leaky_relu(batch_norm3,\n                                     name=\"conv3_out\")\n        \n        # 8x8x128 -> 8x8x256\n        conv4 = tf.layers.conv2d(inputs=conv3_out,\n                                 filters=256,\n                                 kernel_size=[5, 5],\n                                 strides=[1, 1],\n                                 padding=\"SAME\",\n                                 kernel_initializer=tf.truncated_normal_initializer(stddev=WEIGHT_INIT_STDDEV),\n                                 name='conv4')\n        batch_norm4 = tf.layers.batch_normalization(conv4,\n                                                    training=True,\n                                                    epsilon=EPSILON,\n                                                    name='batch_norm4')\n        conv4_out = tf.nn.leaky_relu(batch_norm4,\n                                     name=\"conv4_out\")\n        \n        # 8x8x256 -> 4x4x512\n        \n        conv5 = tf.layers.conv2d(inputs=conv4_out,\n                                filters=512,\n                                kernel_size=[5, 5],\n                                strides=[2, 2],\n                                padding=\"SAME\",\n                                kernel_initializer=tf.truncated_normal_initializer(stddev=WEIGHT_INIT_STDDEV),\n                                name='conv5')\n        batch_norm5 = tf.layers.batch_normalization(conv5,\n                                                    training=True,\n                                                    epsilon=EPSILON,\n                                                    name='batch_norm5')\n        conv5_out = tf.nn.leaky_relu(batch_norm5,\n                                     name=\"conv5_out\")\n\n        flatten = tf.reshape(conv5_out, (-1, 4*4*512))\n        logits = tf.layers.dense(inputs=flatten,\n                                 units=1,\n                                 activation=None)\n        out = tf.sigmoid(logits)\n        return out, logits","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Loss and Optimizer"},{"metadata":{"_uuid":"8ef5bbb8e4d157577f1b15600aa64cb40289a754","trusted":false},"cell_type":"code","source":"def model_loss(input_real, input_z, output_channel_dim):\n    g_model = generator(input_z, output_channel_dim, True)\n\n    noisy_input_real = input_real + tf.random_normal(shape=tf.shape(input_real),\n                                                     mean=0.0,\n                                                     stddev=random.uniform(0.0, 0.1),\n                                                     dtype=tf.float32)\n    \n    d_model_real, d_logits_real = discriminator(noisy_input_real, reuse=False)\n    d_model_fake, d_logits_fake = discriminator(g_model, reuse=True)\n    \n    d_loss_real = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=d_logits_real,\n                                                                         labels=tf.ones_like(d_model_real)*random.uniform(0.9, 1.0)))\n    d_loss_fake = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=d_logits_fake,\n                                                                         labels=tf.zeros_like(d_model_fake)))\n    d_loss = tf.reduce_mean(0.5 * (d_loss_real + d_loss_fake))\n    g_loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=d_logits_fake,\n                                                                    labels=tf.ones_like(d_model_fake)))\n    return d_loss, g_loss","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"01820b197de1b6d0ca39592043308557d941937a","trusted":false},"cell_type":"code","source":"def model_optimizers(d_loss, g_loss):\n    t_vars = tf.trainable_variables()\n    g_vars = [var for var in t_vars if var.name.startswith(\"generator\")]\n    d_vars = [var for var in t_vars if var.name.startswith(\"discriminator\")]\n    \n    update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)\n    gen_updates = [op for op in update_ops if op.name.startswith('generator')]\n    \n    with tf.control_dependencies(gen_updates):\n        d_train_opt = tf.train.AdamOptimizer(learning_rate=LR_D, beta1=BETA1_D).minimize(d_loss, var_list=d_vars)\n        g_train_opt = tf.train.AdamOptimizer(learning_rate=LR_G, beta1=BETA1_G).minimize(g_loss, var_list=g_vars)  \n    return d_train_opt, g_train_opt","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4e8d59264d04344fd0c284c1d41193bd111d4c3f","trusted":false},"cell_type":"code","source":"def model_inputs(real_dim, z_dim):\n    inputs_real = tf.placeholder(tf.float32, (None, *real_dim), name='inputs_real')\n    inputs_z = tf.placeholder(tf.float32, (None, z_dim), name=\"input_z\")\n    learning_rate_G = tf.placeholder(tf.float32, name=\"lr_g\")\n    learning_rate_D = tf.placeholder(tf.float32, name=\"lr_d\")\n    return inputs_real, inputs_z, learning_rate_G, learning_rate_D","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Utils"},{"metadata":{"trusted":false},"cell_type":"code","source":"def show_samples(sample_images, name, epoch):\n    figure, axes = plt.subplots(1, len(sample_images), figsize = (IMAGE_SIZE, IMAGE_SIZE))\n    for index, axis in enumerate(axes):\n        axis.axis('off')\n        image_array = sample_images[index].astype('uint8') \n        axis.imshow(image_array)\n    plt.show()\n    plt.close()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def save_samples(sample_images, name, epoch):\n    # save images\n    for index,img in enumerate(sample_images):\n        image = Image.fromarray(img.astype('uint8') )\n        image.save(name+\"_\"+str(epoch)+\"_\"+str(index)+\".png\") ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"43677a2f90b65e39110c1f547a928bb1b5f9d892","trusted":false},"cell_type":"code","source":"def test(sess, input_z, out_channel_dim, epoch):\n    example_z = np.random.uniform(-1, 1, size=[SAMPLES_TO_SHOW, input_z.get_shape().as_list()[-1]])\n    samples = sess.run(generator(input_z, out_channel_dim, False), feed_dict={input_z: example_z})\n    sample_images = [((sample + 1.0) * 127.5).astype(np.uint8) for sample in samples]\n    show_samples(sample_images, IMG_DIR + \"samples\", epoch)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def generate (sess, input_z, out_channel_dim):\n    print (\">> Generating 1k images ...\")\n    for i in tqdm(range(100)):\n        example_z = np.random.uniform(-1, 1, size=[100, 100]).astype(np.float32)\n        imgs = sess.run(generator(input_z, out_channel_dim, False), feed_dict={input_z: example_z})\n        imgs = [((img + 1.0) * 127.5).astype(np.uint8) for img in imgs]\n        for j in range(len(imgs)):\n            imsave(os.path.join(IMG_DIR, f'dog_{i}_{j}.png'), imgs[j])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0ed8f1c378f936ac81ae89b35d5b6914cd6efbbb","trusted":false},"cell_type":"code","source":"def summarize_epoch(epoch, duration, sess, d_losses, g_losses, input_z, data_shape):\n    minibatch_size = int(data_shape[0]//BATCH_SIZE)\n    print(\"Epoch {}/{}\".format(epoch, EPOCHS),\n          \"\\nDuration: {:.5f}\".format(duration),\n          \"\\nD Loss: {:.5f}\".format(np.mean(d_losses[-minibatch_size:])),\n          \"\\nG Loss: {:.5f}\".format(np.mean(g_losses[-minibatch_size:])))\n    \n    fig, ax = plt.subplots()\n    plt.plot(d_losses, label='Discriminator', alpha=0.6)\n    plt.plot(g_losses, label='Generator', alpha=0.6)\n    plt.title(\"Losses\")\n    plt.legend()\n    #plt.savefig(OUTPUT_DIR + \"losses_\" + str(epoch) + \".png\")\n    plt.show()\n    plt.close()\n    test(sess, input_z, data_shape[3], epoch)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5977b8035723edd4a013406cb5376848d62dcc0a","trusted":false},"cell_type":"code","source":"def get_batches(data):\n    batches = []\n    for i in range(int(data.shape[0]//BATCH_SIZE)):\n        batch = data[i * BATCH_SIZE:(i + 1) * BATCH_SIZE]\n        augmented_images = []\n        for img in batch:\n            image = Image.fromarray(img.astype('uint8'))\n            if random.choice([True, False]):\n                image = image.transpose(Image.FLIP_LEFT_RIGHT)\n            augmented_images.append(np.asarray(image))\n        batch = np.asarray(augmented_images)\n        normalized_batch = (batch / 127.5) - 1.0\n        batches.append(normalized_batch)\n    return batches","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<br>\n# Training"},{"metadata":{"trusted":false},"cell_type":"code","source":"!mkdir images\n!ls","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"578f639e8a70bed5c797f0f4655d318f1a07f6fc","trusted":false},"cell_type":"code","source":"def train(get_batches, data_shape, LR_G = 2e-4, LR_D = 0.0005):\n    input_images, input_z, lr_G, lr_D = model_inputs(data_shape[1:], NOISE_SIZE)\n    d_loss, g_loss = model_loss(input_images, input_z, data_shape[3])\n    d_opt, g_opt = model_optimizers(d_loss, g_loss)\n    generator_epoch_loss = 0\n    train_d_losses = []\n    train_g_losses = []\n    generator_epoch_loss = 999\n    \n    with tf.Session() as sess:\n        sess.run(tf.global_variables_initializer())\n        epoch = 0\n        iteration = 0\n        d_losses = []\n        g_losses = []\n        \n        for epoch in tqdm(range(EPOCHS)):        \n            epoch += 1\n            start_time = time.time()\n                \n            for batch_images in get_batches:\n                iteration += 1\n                batch_z = np.random.uniform(-1, 1, size=(BATCH_SIZE, NOISE_SIZE))\n                _ = sess.run(d_opt, feed_dict={input_images: batch_images, input_z: batch_z, lr_D: LR_D})\n                _ = sess.run(g_opt, feed_dict={input_images: batch_images, input_z: batch_z, lr_G: LR_G})\n                d_losses.append(d_loss.eval({input_z: batch_z, input_images: batch_images}))\n                g_losses.append(g_loss.eval({input_z: batch_z}))\n\n            summarize_epoch(epoch, time.time()-start_time, sess, d_losses, g_losses, input_z, data_shape)\n            minibatch_size = int(data_shape[0]//BATCH_SIZE)\n            generator_epoch_loss = np.mean(g_losses[-minibatch_size:])\n            train_d_losses.append(np.mean(d_losses[-minibatch_size:]))\n            train_g_losses.append(np.mean(g_losses[-minibatch_size:]))\n            \n            if epoch == EPOCHS:\n                generate (sess, input_z, out_channel_dim=3)\n            \n    fig, ax = plt.subplots()\n    plt.plot(train_d_losses, label='Discriminator', alpha=0.5)\n    plt.plot(train_g_losses, label='Generator', alpha=0.5)\n    plt.title(\"Training Losses\")\n    plt.legend()\n    plt.savefig('train_losses.png')\n    plt.show()\n    plt.close()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Hyperparameters**"},{"metadata":{"_uuid":"86335745ed2e6b406839c5d85a1085aab276e73a","trusted":false},"cell_type":"code","source":"IMAGE_SIZE = 64\nNOISE_SIZE = 100\nLR_D = 0.0005\nLR_G = 2e-4\nBATCH_SIZE = 64\nEPOCHS = 100\nBETA1_G = 0.5\nBETA1_D = 0.5\nWEIGHT_INIT_STDDEV = 0.02\nMOMENTUM = 0.9\nEPSILON = 0.0005\nSAMPLES_TO_SHOW = 5 # each epoch","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Training loop**"},{"metadata":{"_kg_hide-input":true,"_uuid":"1ee6349afccb4d2f29f36d6605dd2f156350821a","trusted":false},"cell_type":"code","source":"start = time.time()\n\nprint(\">> Start training...\")\nwith tf.Graph().as_default():\n    train(get_batches(imagesIn), imagesIn.shape)\n    \nprint(\">> train time = \",time.time() - start)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<br>\n# Generated Dogs"},{"metadata":{"_kg_hide-input":true,"trusted":false},"cell_type":"code","source":"imgs = os.listdir('/kaggle/working/images')\nprint (\"images = \",len(imgs))\n\nplt.figure(figsize=(10,10))\nfor i,image in enumerate(imgs):\n    im= Image.open('images/'+image)\n    plt.subplot(3,3,i+1)\n    plt.axis(\"off\")\n    plt.imshow(im)    \n    if(i==8):\n        break","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"shutil.make_archive('images', 'zip', 'images')\n!ls","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Generated images information**"},{"metadata":{"_kg_hide-input":true,"trusted":false},"cell_type":"code","source":"imgs = os.listdir('/kaggle/working/images')\nprint (\"images = \",len(imgs))\n\n# read image\nimg = cv2.imread('/kaggle/working/images/'+imgs[0], cv2.IMREAD_UNCHANGED)\n \n# get dimensions of image\ndimensions = img.shape\n \n# height, width, number of channels in image\nheight = img.shape[0]\nwidth = img.shape[1]\nchannels = img.shape[2]\n \nprint('Image Dimension    : ',dimensions)\nprint('Image Height       : ',height)\nprint('Image Width        : ',width)\nprint('Number of Channels : ',channels)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<br>\n# LB Score\n\n- You need to set ```ComputeLB = True``` and import the dataset [Dog face generation competition MiFID metric input\n](https://www.kaggle.com/wendykan/dog-face-generation-competition-kid-metric-input) "},{"metadata":{"_kg_hide-input":true,"trusted":false},"cell_type":"code","source":"from __future__ import absolute_import, division, print_function\nimport numpy as np\nimport os\nimport gzip, pickle\nimport tensorflow as tf\nfrom scipy import linalg\nimport pathlib\nimport urllib\nimport warnings\nfrom tqdm import tqdm\nfrom PIL import Image\n\nclass KernelEvalException(Exception):\n    pass\n\nmodel_params = {\n    'Inception': {\n        'name': 'Inception', \n        'imsize': 64,\n        'output_layer': 'Pretrained_Net/pool_3:0', \n        'input_layer': 'Pretrained_Net/ExpandDims:0',\n        'output_shape': 2048,\n        'cosine_distance_eps': 0.1\n        }\n}\n\ndef create_model_graph(pth):\n    \"\"\"Creates a graph from saved GraphDef file.\"\"\"\n    # Creates graph from saved graph_def.pb.\n    with tf.gfile.FastGFile( pth, 'rb') as f:\n        graph_def = tf.GraphDef()\n        graph_def.ParseFromString( f.read())\n        _ = tf.import_graph_def( graph_def, name='Pretrained_Net')\n\ndef _get_model_layer(sess, model_name):\n    # layername = 'Pretrained_Net/final_layer/Mean:0'\n    layername = model_params[model_name]['output_layer']\n    layer = sess.graph.get_tensor_by_name(layername)\n    ops = layer.graph.get_operations()\n    for op_idx, op in enumerate(ops):\n        for o in op.outputs:\n            shape = o.get_shape()\n            if shape._dims != []:\n              shape = [s.value for s in shape]\n              new_shape = []\n              for j, s in enumerate(shape):\n                if s == 1 and j == 0:\n                  new_shape.append(None)\n                else:\n                  new_shape.append(s)\n              o.__dict__['_shape_val'] = tf.TensorShape(new_shape)\n    return layer\n\ndef get_activations(images, sess, model_name, batch_size=50, verbose=False):\n    \"\"\"Calculates the activations of the pool_3 layer for all images.\n\n    Params:\n    -- images      : Numpy array of dimension (n_images, hi, wi, 3). The values\n                     must lie between 0 and 256.\n    -- sess        : current session\n    -- batch_size  : the images numpy array is split into batches with batch size\n                     batch_size. A reasonable batch size depends on the disposable hardware.\n    -- verbose    : If set to True and parameter out_step is given, the number of calculated\n                     batches is reported.\n    Returns:\n    -- A numpy array of dimension (num images, 2048) that contains the\n       activations of the given tensor when feeding inception with the query tensor.\n    \"\"\"\n    inception_layer = _get_model_layer(sess, model_name)\n    n_images = images.shape[0]\n    if batch_size > n_images:\n        print(\"warning: batch size is bigger than the data size. setting batch size to data size\")\n        batch_size = n_images\n    n_batches = n_images//batch_size + 1\n    pred_arr = np.empty((n_images,model_params[model_name]['output_shape']))\n    for i in tqdm(range(n_batches)):\n        if verbose:\n            print(\"\\rPropagating batch %d/%d\" % (i+1, n_batches), end=\"\", flush=True)\n        start = i*batch_size\n        if start+batch_size < n_images:\n            end = start+batch_size\n        else:\n            end = n_images\n                    \n        batch = images[start:end]\n        pred = sess.run(inception_layer, {model_params[model_name]['input_layer']: batch})\n        pred_arr[start:end] = pred.reshape(-1,model_params[model_name]['output_shape'])\n    if verbose:\n        print(\" done\")\n    return pred_arr\n\n\n# def calculate_memorization_distance(features1, features2):\n#     neigh = NearestNeighbors(n_neighbors=1, algorithm='kd_tree', metric='euclidean')\n#     neigh.fit(features2) \n#     d, _ = neigh.kneighbors(features1, return_distance=True)\n#     print('d.shape=',d.shape)\n#     return np.mean(d)\n\ndef normalize_rows(x: np.ndarray):\n    \"\"\"\n    function that normalizes each row of the matrix x to have unit length.\n\n    Args:\n     ``x``: A numpy matrix of shape (n, m)\n\n    Returns:\n     ``x``: The normalized (by row) numpy matrix.\n    \"\"\"\n    return np.nan_to_num(x/np.linalg.norm(x, ord=2, axis=1, keepdims=True))\n\n\ndef cosine_distance(features1, features2):\n    # print('rows of zeros in features1 = ',sum(np.sum(features1, axis=1) == 0))\n    # print('rows of zeros in features2 = ',sum(np.sum(features2, axis=1) == 0))\n    features1_nozero = features1[np.sum(features1, axis=1) != 0]\n    features2_nozero = features2[np.sum(features2, axis=1) != 0]\n    norm_f1 = normalize_rows(features1_nozero)\n    norm_f2 = normalize_rows(features2_nozero)\n\n    d = 1.0-np.abs(np.matmul(norm_f1, norm_f2.T))\n    print('d.shape=',d.shape)\n    print('np.min(d, axis=1).shape=',np.min(d, axis=1).shape)\n    mean_min_d = np.mean(np.min(d, axis=1))\n    print('distance=',mean_min_d)\n    return mean_min_d\n\n\ndef distance_thresholding(d, eps):\n    if d < eps:\n        return d\n    else:\n        return 1\n\ndef calculate_frechet_distance(mu1, sigma1, mu2, sigma2, eps=1e-6):\n    \"\"\"Numpy implementation of the Frechet Distance.\n    The Frechet distance between two multivariate Gaussians X_1 ~ N(mu_1, C_1)\n    and X_2 ~ N(mu_2, C_2) is\n            d^2 = ||mu_1 - mu_2||^2 + Tr(C_1 + C_2 - 2*sqrt(C_1*C_2)).\n            \n    Stable version by Dougal J. Sutherland.\n\n    Params:\n    -- mu1 : Numpy array containing the activations of the pool_3 layer of the\n             inception net ( like returned by the function 'get_predictions')\n             for generated samples.\n    -- mu2   : The sample mean over activations of the pool_3 layer, precalcualted\n               on an representive data set.\n    -- sigma1: The covariance matrix over activations of the pool_3 layer for\n               generated samples.\n    -- sigma2: The covariance matrix over activations of the pool_3 layer,\n               precalcualted on an representive data set.\n\n    Returns:\n    --   : The Frechet Distance.\n    \"\"\"\n\n    mu1 = np.atleast_1d(mu1)\n    mu2 = np.atleast_1d(mu2)\n\n    sigma1 = np.atleast_2d(sigma1)\n    sigma2 = np.atleast_2d(sigma2)\n\n    assert mu1.shape == mu2.shape, \"Training and test mean vectors have different lengths\"\n    assert sigma1.shape == sigma2.shape, \"Training and test covariances have different dimensions\"\n\n    diff = mu1 - mu2\n\n    # product might be almost singular\n    covmean, _ = linalg.sqrtm(sigma1.dot(sigma2), disp=False)\n    if not np.isfinite(covmean).all():\n        msg = \"fid calculation produces singular product; adding %s to diagonal of cov estimates\" % eps\n        warnings.warn(msg)\n        offset = np.eye(sigma1.shape[0]) * eps\n        # covmean = linalg.sqrtm((sigma1 + offset).dot(sigma2 + offset))\n        covmean = linalg.sqrtm((sigma1 + offset).dot(sigma2 + offset))\n    \n    # numerical error might give slight imaginary component\n    if np.iscomplexobj(covmean):\n        if not np.allclose(np.diagonal(covmean).imag, 0, atol=1e-3):\n            m = np.max(np.abs(covmean.imag))\n            raise ValueError(\"Imaginary component {}\".format(m))\n        covmean = covmean.real\n\n    # covmean = tf.linalg.sqrtm(tf.linalg.matmul(sigma1,sigma2))\n\n    print('covmean.shape=',covmean.shape)\n    # tr_covmean = tf.linalg.trace(covmean)\n\n    tr_covmean = np.trace(covmean)\n    return diff.dot(diff) + np.trace(sigma1) + np.trace(sigma2) - 2 * tr_covmean\n    # return diff.dot(diff) + tf.linalg.trace(sigma1) + tf.linalg.trace(sigma2) - 2 * tr_covmean\n#-------------------------------------------------------------------------------\n\n\ndef calculate_activation_statistics(images, sess, model_name, batch_size=50, verbose=False):\n    \"\"\"Calculation of the statistics used by the FID.\n    Params:\n    -- images      : Numpy array of dimension (n_images, hi, wi, 3). The values\n                     must lie between 0 and 255.\n    -- sess        : current session\n    -- batch_size  : the images numpy array is split into batches with batch size\n                     batch_size. A reasonable batch size depends on the available hardware.\n    -- verbose     : If set to True and parameter out_step is given, the number of calculated\n                     batches is reported.\n    Returns:\n    -- mu    : The mean over samples of the activations of the pool_3 layer of\n               the incption model.\n    -- sigma : The covariance matrix of the activations of the pool_3 layer of\n               the incption model.\n    \"\"\"\n    act = get_activations(images, sess, model_name, batch_size, verbose)\n    mu = np.mean(act, axis=0)\n    sigma = np.cov(act, rowvar=False)\n    return mu, sigma, act\n    \ndef _handle_path_memorization(path, sess, model_name, is_checksize, is_check_png):\n    path = pathlib.Path(path)\n    files = list(path.glob('*.jpg')) + list(path.glob('*.png'))\n    imsize = model_params[model_name]['imsize']\n\n    # In production we don't resize input images. This is just for demo purpose. \n    x = np.array([np.array(img_read_checks(fn, imsize, is_checksize, imsize, is_check_png)) for fn in files])\n    m, s, features = calculate_activation_statistics(x, sess, model_name)\n    del x #clean up memory\n    return m, s, features\n\n# check for image size\ndef img_read_checks(filename, resize_to, is_checksize=False, check_imsize = 64, is_check_png = False):\n    im = Image.open(str(filename))\n    if is_checksize and im.size != (check_imsize,check_imsize):\n        raise KernelEvalException('The images are not of size '+str(check_imsize))\n    \n    if is_check_png and im.format != 'PNG':\n        raise KernelEvalException('Only PNG images should be submitted.')\n\n    if resize_to is None:\n        return im\n    else:\n        return im.resize((resize_to,resize_to),Image.ANTIALIAS)\n\ndef calculate_kid_given_paths(paths, model_name, model_path, feature_path=None):\n    ''' Calculates the KID of two paths. '''\n    tf.reset_default_graph()\n    create_model_graph(str(model_path))\n    with tf.Session() as sess:\n        sess.run(tf.global_variables_initializer())\n        m1, s1, features1 = _handle_path_memorization(paths[0], sess, model_name, is_checksize = True, is_check_png = True)\n        if feature_path is None:\n            m2, s2, features2 = _handle_path_memorization(paths[1], sess, model_name, is_checksize = False, is_check_png = False)\n        else:\n            with np.load(feature_path) as f:\n                m2, s2, features2 = f['m'], f['s'], f['features']\n\n        print('m1,m2 shape=',(m1.shape,m2.shape),'s1,s2=',(s1.shape,s2.shape))\n        print('starting calculating FID')\n        fid_value = calculate_frechet_distance(m1, s1, m2, s2)\n        print('done with FID, starting distance calculation')\n        distance = cosine_distance(features1, features2)        \n        return fid_value, distance","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":false},"cell_type":"code","source":"user_images_unzipped_path = '/kaggle/working/images/'\nimages_path = [user_images_unzipped_path,'../input/generative-dog-images/all-dogs/all-dogs/']\npublic_path = '../input/dog-face-generation-competition-kid-metric-input/classify_image_graph_def.pb'\nfid_epsilon = 10e-15\nfid_value_public, distance_public = calculate_kid_given_paths(images_path, 'Inception', public_path)\ndistance_public = distance_thresholding(distance_public, model_params['Inception']['cosine_distance_eps'])\nprint(\"FID_public: \", fid_value_public, \"distance_public: \", distance_public, \"multiplied_public: \", fid_value_public /(distance_public + fid_epsilon))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":false},"cell_type":"code","source":"!rm -r images\n!ls","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# How can you improve this basic code?\n\n1. Data Augmentation\n2. Improve the Generator and Discriminator NN\n3. Increase EPOCH = better score ¿?\n4. Use ```DogsOnly``` pictures. (but carefully)\n5. Use [All you need is GAN Hacks](https://www.kaggle.com/c/generative-dog-images/discussion/98595#latest-570614)"},{"metadata":{},"cell_type":"markdown","source":"<br>\n# References:\n\n- [GAN — DCGAN (Deep convolutional generative adversarial networks)](https://medium.com/@jonathan_hui/gan-dcgan-deep-convolutional-generative-adversarial-networks-df855c438f)\n- [DCGANs (Deep Convolutional Generative Adversarial Networks)](https://towardsdatascience.com/dcgans-deep-convolutional-generative-adversarial-networks-c7f392c2c8f8)\n- [DCGAN Chainer](https://github.com/chainer/chainer/tree/master/examples/dcgan)\n- [DCGAN: Generate the images with Deep Convolutional GAN](https://medium.com/@keisukeumezawa/dcgan-generate-the-images-with-deep-convolutinal-gan-55edf947c34b)\n- [Image Generator (DCGAN) 'The Simpsons' Dataset](https://www.kaggle.com/greg115/image-generator-dcgan-the-simpsons-dataset)\n\n**Thanks for reading!**\n\nLet me know if you find something wrong or how to improve this, I'll keep updating."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"},"widgets":{"application/vnd.jupyter.widget-state+json":{"state":{"2219ce26505546e68290269fb59bc249":{"model_module":"@jupyter-widgets/base","model_module_version":"1.1.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.1.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.1.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"26bdbe4d7fd84a148fded880a1c05a21":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.4.0","model_name":"HBoxModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.4.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.4.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_5f1ad56b87c045248e3ffd1d442967e0","IPY_MODEL_e14c4bbbeee34f5b97799a7575ab1515"],"layout":"IPY_MODEL_81cfda54ddea439a80b1039726b6c710"}},"2e581a432db14305a67203fa43ab84d2":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.4.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.4.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.1.0","_view_name":"StyleView","description_width":""}},"47b7ec44b11c4799b53b1949f3ad761f":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.4.0","model_name":"ProgressStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.4.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.1.0","_view_name":"StyleView","bar_color":null,"description_width":""}},"5f1ad56b87c045248e3ffd1d442967e0":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.4.0","model_name":"IntProgressModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.4.0","_model_name":"IntProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.4.0","_view_name":"ProgressView","bar_style":"success","description":"","description_tooltip":null,"layout":"IPY_MODEL_2219ce26505546e68290269fb59bc249","max":100,"min":0,"orientation":"horizontal","style":"IPY_MODEL_7c823aa341164114a68f49a35dec7c0e","value":100}},"6157075e75b34c6795b3f6ac2366ddaf":{"model_module":"@jupyter-widgets/base","model_module_version":"1.1.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.1.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.1.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"74bab2c4fa3a4480bff2c9c71390f1d1":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.4.0","model_name":"IntProgressModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.4.0","_model_name":"IntProgressModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.4.0","_view_name":"ProgressView","bar_style":"success","description":"","description_tooltip":null,"layout":"IPY_MODEL_6157075e75b34c6795b3f6ac2366ddaf","max":100,"min":0,"orientation":"horizontal","style":"IPY_MODEL_47b7ec44b11c4799b53b1949f3ad761f","value":100}},"7c62745ecb4c4d8486f4e55a2f18e4e4":{"model_module":"@jupyter-widgets/base","model_module_version":"1.1.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.1.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.1.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"7c823aa341164114a68f49a35dec7c0e":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.4.0","model_name":"ProgressStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.4.0","_model_name":"ProgressStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.1.0","_view_name":"StyleView","bar_color":null,"description_width":""}},"81cfda54ddea439a80b1039726b6c710":{"model_module":"@jupyter-widgets/base","model_module_version":"1.1.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.1.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.1.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"897c4c79e4924f8db426b6109ed4e532":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.4.0","model_name":"HBoxModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.4.0","_model_name":"HBoxModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.4.0","_view_name":"HBoxView","box_style":"","children":["IPY_MODEL_74bab2c4fa3a4480bff2c9c71390f1d1","IPY_MODEL_ee3c96fb544042c99fa2d7eba09caed3"],"layout":"IPY_MODEL_7c62745ecb4c4d8486f4e55a2f18e4e4"}},"9ef8cec5f3784461a894b4401808f244":{"model_module":"@jupyter-widgets/base","model_module_version":"1.1.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.1.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.1.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}},"a2ca9337357d4b24a99cb4ab69da693f":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.4.0","model_name":"DescriptionStyleModel","state":{"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.4.0","_model_name":"DescriptionStyleModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.1.0","_view_name":"StyleView","description_width":""}},"e14c4bbbeee34f5b97799a7575ab1515":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.4.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.4.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.4.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_9ef8cec5f3784461a894b4401808f244","placeholder":"​","style":"IPY_MODEL_a2ca9337357d4b24a99cb4ab69da693f","value":"100% 100/100 [1:08:21&lt;00:00, 81.00s/it]"}},"ee3c96fb544042c99fa2d7eba09caed3":{"model_module":"@jupyter-widgets/controls","model_module_version":"1.4.0","model_name":"HTMLModel","state":{"_dom_classes":[],"_model_module":"@jupyter-widgets/controls","_model_module_version":"1.4.0","_model_name":"HTMLModel","_view_count":null,"_view_module":"@jupyter-widgets/controls","_view_module_version":"1.4.0","_view_name":"HTMLView","description":"","description_tooltip":null,"layout":"IPY_MODEL_f4ef6e5c91394884bbff56756f53d525","placeholder":"​","style":"IPY_MODEL_2e581a432db14305a67203fa43ab84d2","value":"100% 100/100 [02:16&lt;00:00,  1.71s/it]"}},"f4ef6e5c91394884bbff56756f53d525":{"model_module":"@jupyter-widgets/base","model_module_version":"1.1.0","model_name":"LayoutModel","state":{"_model_module":"@jupyter-widgets/base","_model_module_version":"1.1.0","_model_name":"LayoutModel","_view_count":null,"_view_module":"@jupyter-widgets/base","_view_module_version":"1.1.0","_view_name":"LayoutView","align_content":null,"align_items":null,"align_self":null,"border":null,"bottom":null,"display":null,"flex":null,"flex_flow":null,"grid_area":null,"grid_auto_columns":null,"grid_auto_flow":null,"grid_auto_rows":null,"grid_column":null,"grid_gap":null,"grid_row":null,"grid_template_areas":null,"grid_template_columns":null,"grid_template_rows":null,"height":null,"justify_content":null,"left":null,"margin":null,"max_height":null,"max_width":null,"min_height":null,"min_width":null,"order":null,"overflow":null,"overflow_x":null,"overflow_y":null,"padding":null,"right":null,"top":null,"visibility":null,"width":null}}},"version_major":2,"version_minor":0}}},"nbformat":4,"nbformat_minor":1}