{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction and Setup\n\nThis notebook is used for the course Machine Learning in Practice from the Radboud Univerity in Nijmegen. This notebook is the tutorial notebook by Amy Jang with modifications. Code from Erik Linder-Noren is used to convert the CycleGAN into a DualGAN.\n\nThis notebook utilizes a DualGAN architecture to add Monet-style to photos. The TFRecord dataset will be used. Import the following packages and change the accelerator to TPU.\n\nThe imports are needed to make the code work, these imports are combined of the imports needed for both the tutorial notebook by Amy Jang and the code by Erik Linder-Noren.\n\nTutorial notebook by Amy Jang: https://www.kaggle.com/amyjang/monet-cyclegan-tutorial\n\nErik Linder-Noren's Github page: https://github.com/eriklindernoren/Keras-GAN"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nimport tensorflow_addons as tfa\nimport tensorflow_datasets as tfds\n\nfrom kaggle_datasets import KaggleDatasets\nimport matplotlib.pyplot as plt\nimport numpy as np\n\nfrom keras.datasets import mnist\nfrom keras.layers import Input, Dense, Reshape, Flatten, Dropout, Concatenate\nfrom keras.layers import BatchNormalization, Activation, ZeroPadding2D\nfrom keras.layers.advanced_activations import LeakyReLU\nfrom keras.layers.convolutional import UpSampling2D, Conv2D\nfrom keras.models import Sequential, Model\nfrom keras.optimizers import RMSprop, Adam\nfrom keras.utils import to_categorical\nimport keras.backend as K\n\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Device:', tpu.master())\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nexcept:\n    strategy = tf.distribute.get_strategy()\nprint('Number of replicas:', strategy.num_replicas_in_sync)\n\nAUTOTUNE = tf.data.experimental.AUTOTUNE\n    \nprint(tf.__version__)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load in the data\n\nThe following section contains code that has been taken from the tutorial notebook by Amy Jang. \n\nWe want to keep our photo dataset and our Monet dataset separate. First, load in the filenames of the TFRecords."},{"metadata":{"trusted":true},"cell_type":"code","source":"GCS_PATH = KaggleDatasets().get_gcs_path()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"MONET_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/monet_tfrec/*.tfrec'))\nprint('Monet TFRecord Files:', len(MONET_FILENAMES))\n\nPHOTO_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/photo_tfrec/*.tfrec'))\nprint('Photo TFRecord Files:', len(PHOTO_FILENAMES))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"All the images for the competition are already sized to 256x256. As these images are RGB images, set the channel to 3. Additionally, we need to scale the images to a [-1, 1] scale. Because we are building a generative model, we don't need the labels or the image id so we'll only return the image from the TFRecord."},{"metadata":{"trusted":true},"cell_type":"code","source":"IMAGE_SIZE = [256, 256]\n\ndef decode_image(image):\n    image = tf.image.decode_jpeg(image, channels=3)\n    image = (tf.cast(image, tf.float32) / 127.5) - 1\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    return image\n\ndef read_tfrecord(example):\n    tfrecord_format = {\n        \"image_name\": tf.io.FixedLenFeature([], tf.string),\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"target\": tf.io.FixedLenFeature([], tf.string)\n    }\n    example = tf.io.parse_single_example(example, tfrecord_format)\n    image = decode_image(example['image'])\n    return image","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Define the function to extract the image from the files."},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_dataset(filenames, labeled=True, ordered=False):\n    dataset = tf.data.TFRecordDataset(filenames)\n    dataset = dataset.map(read_tfrecord, num_parallel_calls=AUTOTUNE)\n    return dataset","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's load in our datasets."},{"metadata":{"trusted":true},"cell_type":"code","source":"monet_ds = load_dataset(MONET_FILENAMES, labeled=True).batch(1)\nphoto_ds = load_dataset(PHOTO_FILENAMES, labeled=True).batch(1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"monet_iter = iter(monet_ds)\nphoto_iter = iter(photo_ds)\n\nfor _ in range(4):\n    example_monet = next(monet_iter)\n    example_photo = next(photo_iter)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's  visualize a photo example and a Monet example."},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.subplot(221)\nplt.title('Photo')\nplt.imshow(example_photo[0] * 0.5 + 0.5) \n\nplt.subplot(223)\nplt.imshow(example_photo[0])\n\nplt.subplot(222)\nplt.title('Monet')\nplt.imshow(example_monet[0] * 0.5 + 0.5)\n\nplt.subplot(224)\nplt.imshow(example_monet[0])\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Build the generator\n\nThe code in this section has been taken from the notebook by Amy Jang. \n\nWe'll be using a UNET architecture for our CycleGAN. To build our generator, let's first define our `downsample` and `upsample` methods.\n\nThe `downsample`, as the name suggests, reduces the 2D dimensions, the width and height, of the image by the stride. The stride is the length of the step the filter takes. Since the stride is 2, the filter is applied to every other pixel, hence reducing the weight and height by 2.\n\nWe'll be using an instance normalization instead of batch normalization. As the instance normalization is not standard in the TensorFlow API, we'll use the layer from TensorFlow Add-ons."},{"metadata":{"trusted":true},"cell_type":"code","source":"def downsample(filters, kernel_size, apply_instancenorm=True):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    result = keras.Sequential()\n    result.add(layers.Conv2D(filters, kernel_size, strides=2, padding='same',\n                             kernel_initializer=initializer, use_bias=False))\n\n    if apply_instancenorm:\n        result.add(tfa.layers.InstanceNormalization(gamma_initializer=gamma_init))\n\n    result.add(layers.LeakyReLU())\n\n    return result","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`Upsample` does the opposite of downsample and increases the dimensions of the of the image. `Conv2DTranspose` does basically the opposite of a `Conv2D` layer."},{"metadata":{"trusted":true},"cell_type":"code","source":"def upsample(filters, kernel_size, apply_dropout=False):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    result = keras.Sequential()\n    result.add(layers.Conv2DTranspose(filters, kernel_size, strides=2,\n                                      padding='same',\n                                      kernel_initializer=initializer,\n                                      use_bias=False))\n\n    result.add(tfa.layers.InstanceNormalization(gamma_initializer=gamma_init))\n\n    if apply_dropout:\n        result.add(layers.Dropout(0.5))\n\n    result.add(layers.ReLU())\n\n    return result","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's build our generator!\n\nThe generator first downsamples the input image and then upsample while establishing long skip connections. Skip connections are a way to help bypass the vanishing gradient problem by concatenating the output of a layer to multiple layers instead of only one. Here we concatenate the output of the downsample layer to the upsample layer in a symmetrical fashion."},{"metadata":{"trusted":true},"cell_type":"code","source":"def Generator(output_channels = 3):\n    inputs = layers.Input(shape=[256,256,3])\n\n    # bs = batch size\n    down_stack = [\n        downsample(64, 4, apply_instancenorm=False), # (bs, 128, 128, 64)\n        downsample(128, 4), # (bs, 64, 64, 128)\n        downsample(256, 4), # (bs, 32, 32, 256)\n        downsample(512, 4), # (bs, 16, 16, 512)\n        downsample(512, 4), # (bs, 8, 8, 512)\n        downsample(512, 4), # (bs, 4, 4, 512)\n        downsample(512, 4), # (bs, 2, 2, 512)\n        downsample(512, 4), # (bs, 1, 1, 512)\n    ]\n    up_stack = [\n        upsample(512, 4, apply_dropout=True), # (bs, 2, 2, 1024)\n        upsample(512, 4, apply_dropout=True), # (bs, 4, 4, 1024)\n        upsample(512, 4, apply_dropout=True), # (bs, 8, 8, 1024)\n        upsample(512, 4), # (bs, 16, 16, 1024)\n        upsample(256, 4), # (bs, 32, 32, 512)\n        upsample(128, 4), # (bs, 64, 64, 256)\n        upsample(64, 4), # (bs, 128, 128, 128)\n    ]\n\n    initializer = tf.random_normal_initializer(0., 0.02)\n    last = layers.Conv2DTranspose(output_channels, 4,\n                                  strides=2,\n                                  padding='same',\n                                  kernel_initializer=initializer,\n                                  activation='tanh') # (bs, 256, 256, 3)\n    \n\n    x = inputs\n\n    skips = []\n    for down in down_stack:\n        x = down(x)\n        skips.append(x)\n\n    skips = reversed(skips[:-1]) \n\n    # Upsampling and establishing the skip connections\n    for up, skip in zip(up_stack, skips):\n        x = up(x)\n        # concatenates output of upsample layer and skip connection\n        x = layers.Concatenate()([x, skip])\n        \n    prediction = last(x)\n\n    return keras.Model(inputs=inputs, outputs=prediction)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Added by our team to check the architecture of the Generator. \ntest_gen = Generator()\ntest_gen.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Build the discriminator\n\nThe code in this section has been taken from the notebook by Amy Jang. Some modifications have been made by our team. \n\nThe discriminator takes in the input image and classifies it as real or fake (generated). Instead of outputing a single node, the discriminator outputs a smaller 2D image with higher pixel values indicating a real classification and lower values indicating a fake classification."},{"metadata":{"trusted":true},"cell_type":"code","source":"def Discriminator():\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    inp = layers.Input(shape=[256, 256, 3], name='input_image')\n\n    x = inp\n\n    down1 = downsample(64, 4, False)(x) # (bs, 128, 128, 64)\n    down2 = downsample(128, 4)(down1) # (bs, 64, 64, 128)\n    down3 = downsample(256, 4)(down2) # (bs, 32, 32, 256)\n\n    zero_pad1 = layers.ZeroPadding2D()(down3) # (bs, 34, 34, 256)\n    conv = layers.Conv2D(512, 4, strides=1,\n                         kernel_initializer=initializer,\n                         use_bias=False)(zero_pad1) # (bs, 31, 31, 512)\n\n    norm1 = tfa.layers.InstanceNormalization(gamma_initializer=gamma_init)(conv)\n\n    leaky_relu = layers.LeakyReLU()(norm1)\n\n    zero_pad2 = layers.ZeroPadding2D()(leaky_relu) # (bs, 33, 33, 512)\n\n    #Modification by our team: removing the last layer. \n    #last = layers.Conv2D(1, 4, strides=1,\n    #                     kernel_initializer=initializer)(zero_pad2) # (bs, 30, 30, 1)\n    \n    flatten = layers.Flatten()(zero_pad2)\n    \n    last = layers.Dense(1)(flatten)\n\n    return tf.keras.Model(inputs=inp, outputs=last)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Added by our team to check the architecture of the Discriminator. \ntest_disc = Discriminator()\ntest_disc.summary()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Training procedure \n\nThe code in this section has been taken from the github from Erik Linder-Norén. Several modifications have been made by our team. "},{"metadata":{"trusted":true},"cell_type":"code","source":"class DUALGAN(keras.Model):\n    def __init__(self,\n                 monet_generator, \n                 photo_generator,\n                 monet_discriminator, \n                 photo_discriminator,\n                 optimizer, \n                 disc_loss, \n                 gen_loss):\n        \n        super(DUALGAN, self).__init__()\n        # Modification made by our team: initialisation outside the class. \n        self.gen_loss = gen_loss\n        self.disc_loss = disc_loss\n        self.optimizer = optimizer\n        self.img_rows = 256\n        self.img_cols = 256\n        self.img_channels = 3\n        self.img_dim = self.img_rows*self.img_cols\n        \n        #Monet discriminator\n        self.D_A = monet_discriminator \n        self.D_A.compile(loss=self.disc_loss, optimizer=self.optimizer, metrics=['accuracy'])\n\n        #photo discriminator\n        self.D_B = photo_discriminator \n        self.D_B.compile(loss=self.disc_loss, optimizer=self.optimizer, metrics=['accuracy'])\n\n        #-------------------------\n        # Construct Computational\n        #   Graph of Generators\n        #-------------------------\n\n        # Build the generators\n        self.G_AB = photo_generator #monet to photo\n        self.G_BA = monet_generator #photo to monet\n\n        # For the combined model we will only train the generators\n        self.D_A.trainable = False\n        self.D_B.trainable = False\n\n        # The generator takes images from their respective domains as inputs\n        imgs_A = Input(shape=[self.img_rows, self.img_cols, self.img_channels]) \n        imgs_B = Input(shape=[self.img_rows, self.img_cols, self.img_channels])\n\n        # Generators translates the images to the opposite domain\n        fake_B = self.G_AB(imgs_A)\n        fake_A = self.G_BA(imgs_B)\n\n        # The discriminators determines validity of translated images\n        valid_A = self.D_A(fake_A)\n        valid_B = self.D_B(fake_B)\n\n        # Generators translate the images back to their original domain\n        recov_A = self.G_BA(fake_B)\n        recov_B = self.G_AB(fake_A)\n\n        # Modification made by our team: different loss-weights.\n        # The combined model  (stacked generators and discriminators)\n        self.combined = Model(inputs=[imgs_A, imgs_B], outputs=[valid_A, valid_B, recov_A, recov_B])\n        self.combined.compile(loss=[self.gen_loss, self.gen_loss, 'mae', 'mae'],\n                                optimizer=self.optimizer,\n                                loss_weights=[1, 1, 500, 500])\n\n    \n\n    def sample_generator_input(self, X, n_samples, same=False):\n        #Samples a number of images from X given n_samples. If same=True,\n        #the sampled images are the images at range [0,n_samples). \n        if same:\n            idx = np.arange(n_samples)\n        else:\n            idx = np.random.randint(0, X.shape[0], n_samples)\n        return X[idx]\n\n    def train(self, X_A, X_B, epochs, batch_size=128, sample_interval=10, plot = True, clip_value = 0.01, n_critic = 3):\n\n        valid = np.ones((batch_size, 1))\n        fake = np.zeros((batch_size, 1))\n\n        for epoch in range(epochs):\n            # Train the discriminator for n_critic iterations\n            for _ in range(n_critic):\n\n                # ----------------------\n                #  Train Discriminators\n                # ----------------------\n\n                # Sample generator inputs\n                imgs_A = self.sample_generator_input(X_A, batch_size) #monet\n                imgs_B = self.sample_generator_input(X_B, batch_size)\n\n                # Translate images to their opposite domain\n                fake_B = self.G_AB.predict(imgs_A)\n                fake_A = self.G_BA.predict(imgs_B)\n\n                # Train the discriminators # TODO: ensure that the losses are (lists of) scalars?\n                D_A_loss_real = self.D_A.train_on_batch(imgs_A, valid)\n                D_A_loss_fake = self.D_A.train_on_batch(fake_A, fake)\n\n                D_B_loss_real = self.D_B.train_on_batch(imgs_B, valid)\n                D_B_loss_fake = self.D_B.train_on_batch(fake_B, fake)\n\n                D_A_loss = np.add(D_A_loss_real, D_A_loss_fake)\n                D_B_loss = np.add(D_B_loss_real, D_B_loss_fake)\n\n                # Clip discriminator weights\n                for d in [self.D_A, self.D_B]:\n                    for l in d.layers:\n                        weights = l.get_weights()\n                        weights = [np.clip(w, -clip_value, clip_value) for w in weights]\n                        l.set_weights(weights)\n\n                # ------------------\n                #  Train Generators\n                # ------------------\n\n                # Train the generators\n            g_loss = self.combined.train_on_batch([imgs_A, imgs_B], [valid, valid, imgs_A, imgs_B])\n   \n            # Print the progress\n            print (\"%d [D1 loss: %f] [D2 loss: %f] [G loss: %f]\" \\\n                       % (epoch, D_A_loss[0], D_B_loss[0], g_loss[0]))\n            \n            if plot and epoch % sample_interval == 0:\n                self.plot_predictions(X_A, X_B)\n\n            \n    def get_predictions(self, X_A, X_B, n_samples, same):\n        imgs_A = self.sample_generator_input(X_A, n_samples, same)\n        imgs_B = self.sample_generator_input(X_B, n_samples, same)\n\n        # Images translated to their opposite domain\n        fake_B = self.G_AB.predict(imgs_A)\n        fake_A = self.G_BA.predict(imgs_B)\n            \n        return imgs_A, imgs_B, fake_A, fake_B\n    \n    def plot_predictions(self, monet, photo, n_samples = 4, figsize=(10,15), same = True, scale = 0.5):\n        \n        real_monet, real_picture, fake_monet, fake_picture = self.get_predictions(monet, photo, n_samples, same)\n        \n        # show picture -> Monet\n        plt.figure(figsize=figsize)\n        for i in range(n_samples):\n            plt.subplot(n_samples,2,1+i*2)\n            plt.imshow(real_picture[i] * scale + scale)\n            plt.title(\"picture\")\n\n            plt.subplot(n_samples,2,2+i*2)\n            plt.imshow(fake_monet[i] * scale + scale)\n            plt.title(\"picture to Monet\")\n        plt.show()\n\n        # show Monet -> picture\n        plt.figure(figsize=figsize)\n        for i in range(n_samples):\n            plt.subplot(n_samples,2,1+i*2)\n            plt.imshow(real_monet[i] * scale + scale)\n            plt.title(\"Monet\")\n\n            plt.subplot(n_samples,2,2+i*2)\n            plt.imshow(fake_picture[i] * scale + scale)\n            plt.title(\"Monet to picture\")\n        plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Modification by our team: Here the BatchDataset objects are prosessed \n# such that they can be used when training.\nmonet = np.array([m.squeeze() for m in monet_ds.as_numpy_iterator()])\nphoto = np.array([p.squeeze() for p in photo_ds.as_numpy_iterator()])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Modification by our team: Data augmentation\n\ndef hor_flip(image):\n    image = tf.image.flip_left_right(image)\n    return image\n\ndef augment_data(original_data):\n\n    data = original_data.copy()\n    aug_data = np.zeros((data.shape[0]*2, data.shape[1], data.shape[2], data.shape[3])) \n    \n    for i in range(0,data.shape[0]):\n        aug_data[i+data.shape[0]] = hor_flip(data[i])\n        aug_data[i] = data[i] \n    \n    return aug_data\naugmented_monet = augment_data(monet)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Initializing and training\n\nThe code in this section is inspired on the notebook by Amy Jang."},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    monet_generator = Generator() # transforms photos to Monet-esque paintings = A\n    photo_generator = Generator() # transforms Monet paintings to be more like photos = B\n\n    monet_discriminator = Discriminator() # differentiates real Monet paintings and generated Monet paintings = AB\n    photo_discriminator = Discriminator() # differentiates real photos and generated photos = BA","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Modification by our team: Different loss functions.\nwith strategy.scope():\n\n    def disc_loss(validation_prediction, truth):\n        mse = tf.keras.losses.MeanSquaredError(reduction=tf.keras.losses.Reduction.NONE)\n        return mse(truth, validation_prediction)\n    \n    def gen_loss(validation_prediction, truth):\n        mse = tf.keras.losses.MeanSquaredError(reduction=tf.keras.losses.Reduction.NONE)\n        return mse(truth, validation_prediction)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    gan = DUALGAN(monet_generator,\n                  photo_generator, \n                  monet_discriminator, \n                  photo_discriminator, \n                  optimizer, \n                  disc_loss, \n                  gen_loss)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    gan.train(augmented_monet, photo, epochs=111, batch_size=32)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualize our Monet-esque photos\n\nThe code in this section is taken from the notebook by Amy Jang."},{"metadata":{"trusted":true},"cell_type":"code","source":"_, ax = plt.subplots(5, 2, figsize=(12, 12))\nfor i, img in enumerate(photo_ds.take(4)):\n    prediction = monet_generator(img, training=False)[0].numpy()\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    img = (img[0] * 127.5 + 127.5).numpy().astype(np.uint8)\n\n    ax[i, 0].imshow(img)\n    ax[i, 1].imshow(prediction)\n    ax[i, 0].set_title(\"Input Photo\")\n    ax[i, 1].set_title(\"Monet-esque\")\n    ax[i, 0].axis(\"off\")\n    ax[i, 1].axis(\"off\")\n#plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Create submission file\n\nThe code in this section has been taken from the notebook by Amy Jang."},{"metadata":{"trusted":true},"cell_type":"code","source":"import PIL\n! mkdir ../images","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"i = 1\nfor img in photo_ds:\n    prediction = monet_generator(img, training=False)[0].numpy()\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    im = PIL.Image.fromarray(prediction)\n    im.save(\"../images/\" + str(i) + \".jpg\")\n    i += 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import shutil\nshutil.make_archive(\"/kaggle/working/images\", 'zip', \"/kaggle/images\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}