{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Generative Dog Images\n*A GAN machine learning project* \n\n## OVERVIEW \nCreating artifical images is an exciting concept in the machine learning community. This competition was aimed to give new data scientists the ability to practice and learn how to create their first artifical images using Generative Adversial Network (GAN) models. This notebook is to practice keys skills and learn how GANs work using real images of dogs. Essentially, we have two neural networks - a generator which generates fake images and a discriminator which tries to always say which images are real and which are fake. As the generator learns, it will create better and better images and (hopefully) start to fool the discriminator into thinking its images are real. Our goal is to generate images which are real enough that it can convince the discriminator as such as well. \n\nStructure of this notebook:\n1. Description of the problem and data\n2. Exploratory data analysis (EDA) \n3. Analysis - model building and training\n4. Result \n5. Conclusion","metadata":{}},{"cell_type":"code","source":"#import libraries \nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt, zipfile\nimport seaborn as sns\nimport cv2\nimport random\nimport os\nimport xml.etree.ElementTree as ET \nimport PIL\nfrom PIL import Image\nimport time\nimport shutil\nfrom IPython import display\nimport tensorflow as tf\nfrom tensorflow.keras import layers\nfrom tensorflow.keras.models import Sequential, Model\nfrom tensorflow.keras.layers import Dense, LeakyReLU, Dropout, BatchNormalization, Input\nfrom tensorflow.keras.preprocessing.image import ImageDataGenerator\n\nSEED = tf.random.set_seed(49)","metadata":{"execution":{"iopub.status.busy":"2022-09-20T16:48:54.092807Z","iopub.execute_input":"2022-09-20T16:48:54.093287Z","iopub.status.idle":"2022-09-20T16:49:00.448268Z","shell.execute_reply.started":"2022-09-20T16:48:54.093199Z","shell.execute_reply":"2022-09-20T16:49:00.447602Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Description of the problem and data\n\nThe goal of this competition is to be able to confuse the computer into thinking our generated images are real. We are given two zipped folders, 'Annotation' and 'all-dogs'. Annotation contains the class labels and bounding boxes for the images. All-dogs contains the images. Images are initially different sizes. We are not given a dataframe to view or plot values. Instead, we will move on to the EDA and view some images. Prior to training, we will set our training data (images) as \"train_images\" and we will normalize all images.","metadata":{}},{"cell_type":"code","source":"#get filenames\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"execution":{"iopub.status.busy":"2022-09-20T16:49:00.449569Z","iopub.execute_input":"2022-09-20T16:49:00.450121Z","iopub.status.idle":"2022-09-20T16:49:00.457533Z","shell.execute_reply.started":"2022-09-20T16:49:00.450090Z","shell.execute_reply":"2022-09-20T16:49:00.456730Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#set directory and unzip folders\nif not os.path.exists('generative-dog-images'):\n    !mkdir /kaggle/working/generative-dog-images\n!unzip /kaggle/input/generative-dog-images/all-dogs.zip -d /kaggle/working/generative-dog-images > /dev/null 2>&1\n!unzip /kaggle/input/generative-dog-images/Annotation.zip -d /kaggle/working/generative-dog-images > /dev/null 2>&1","metadata":{"execution":{"iopub.status.busy":"2022-09-20T16:49:00.458792Z","iopub.execute_input":"2022-09-20T16:49:00.459064Z","iopub.status.idle":"2022-09-20T16:49:18.772119Z","shell.execute_reply.started":"2022-09-20T16:49:00.459007Z","shell.execute_reply":"2022-09-20T16:49:18.770970Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#set directories and paths\nall_dogs_dir = '/kaggle/working/generative-dog-images/all-dogs/'\nannotation_dir = '/kaggle/working/generative-dog-images/Annotation/'\n\nIMAGES = os.listdir(all_dogs_dir)\nBREEDS = os.listdir(annotation_dir)","metadata":{"execution":{"iopub.status.busy":"2022-09-20T16:49:18.774393Z","iopub.execute_input":"2022-09-20T16:49:18.774677Z","iopub.status.idle":"2022-09-20T16:49:18.793031Z","shell.execute_reply.started":"2022-09-20T16:49:18.774643Z","shell.execute_reply":"2022-09-20T16:49:18.792314Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Setup GPU accelerator - configure Strategy. Assume TPU...if not set default for GPU/CPU\ntpu = None\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.TPUStrategy(tpu)\nexcept ValueError:\n    strategy = tf.distribute.get_strategy()","metadata":{"execution":{"iopub.status.busy":"2022-09-20T16:49:18.794206Z","iopub.execute_input":"2022-09-20T16:49:18.794418Z","iopub.status.idle":"2022-09-20T16:49:24.991940Z","shell.execute_reply.started":"2022-09-20T16:49:18.794394Z","shell.execute_reply":"2022-09-20T16:49:24.991030Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Dog Count:', len(os.listdir(all_dogs_dir)))\nprint('Breed Count:', len(os.listdir(annotation_dir)))","metadata":{"execution":{"iopub.status.busy":"2022-09-20T16:49:24.993426Z","iopub.execute_input":"2022-09-20T16:49:24.993824Z","iopub.status.idle":"2022-09-20T16:49:25.014437Z","shell.execute_reply.started":"2022-09-20T16:49:24.993784Z","shell.execute_reply":"2022-09-20T16:49:25.013599Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Exploratory data analysis (EDA) \nWe will crop the images to the bounding boxes and save them. Each image prior to crop and after will be a different size based on the bounding box and original image size. Then we will resize the images to be 100x100 pixels with 3 color channels. After cropping the images, we visualize them below. ","metadata":{}},{"cell_type":"code","source":"# Crop to bounding boxes such that we only have the dog(s) in the picture\n# https://www.kaggle.com/paulorzp/show-annotations-and-breeds\nidxIn = 0\nnamesIn = []\nimagesIn = np.zeros((25000,100,100,3))\n\nfor breed in BREEDS: \n    for dog in os.listdir(annotation_dir+breed):\n        try: img = Image.open(all_dogs_dir+dog+'.jpg') #open image\n        except: continue           \n        tree = ET.parse(annotation_dir+breed+'/'+dog)\n        root = tree.getroot()\n        objects = root.findall('object')\n        for o in objects: #bounding box\n            bndbox = o.find('bndbox') \n            xmin = int(bndbox.find('xmin').text)\n            ymin = int(bndbox.find('ymin').text)\n            xmax = int(bndbox.find('xmax').text)\n            ymax = int(bndbox.find('ymax').text)\n            w = np.min((xmax - xmin, ymax - ymin))\n            img2 = img.crop((xmin, ymin, xmin+w, ymin+w)) #crop to bounding box\n            img2 = img2.resize((100,100), Image.ANTIALIAS) #resize to 100x100\n            imagesIn[idxIn,:,:,:] = np.asarray(img2)\n            namesIn.append(breed)\n            idxIn += 1\nidx = np.arange(idxIn)\nnp.random.shuffle(idx)\nimagesIn = imagesIn[idx,:,:,:]\nnamesIn = np.array(namesIn)[idx]","metadata":{"execution":{"iopub.status.busy":"2022-09-20T16:49:25.015689Z","iopub.execute_input":"2022-09-20T16:49:25.015926Z","iopub.status.idle":"2022-09-20T16:51:12.658514Z","shell.execute_reply.started":"2022-09-20T16:49:25.015899Z","shell.execute_reply":"2022-09-20T16:51:12.657562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#delete files now that we have images \n# and names files to make room\nshutil.rmtree(all_dogs_dir)\nshutil.rmtree(annotation_dir)","metadata":{"execution":{"iopub.status.busy":"2022-09-20T16:51:12.659713Z","iopub.execute_input":"2022-09-20T16:51:12.660504Z","iopub.status.idle":"2022-09-20T16:51:14.149259Z","shell.execute_reply.started":"2022-09-20T16:51:12.660456Z","shell.execute_reply":"2022-09-20T16:51:14.148160Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#view array shape (make sure images are 100x100x3)\nimagesIn.shape","metadata":{"execution":{"iopub.status.busy":"2022-09-20T16:51:14.150392Z","iopub.execute_input":"2022-09-20T16:51:14.150656Z","iopub.status.idle":"2022-09-20T16:51:14.158989Z","shell.execute_reply.started":"2022-09-20T16:51:14.150627Z","shell.execute_reply":"2022-09-20T16:51:14.157824Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"namesIn.shape","metadata":{"execution":{"iopub.status.busy":"2022-09-20T16:51:14.162862Z","iopub.execute_input":"2022-09-20T16:51:14.163169Z","iopub.status.idle":"2022-09-20T16:51:14.171520Z","shell.execute_reply.started":"2022-09-20T16:51:14.163132Z","shell.execute_reply":"2022-09-20T16:51:14.170668Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# visualize cropped images\nx = np.random.randint(0,idxIn,25)\nfor k in range(5):\n    plt.figure(figsize=(15,3))\n    for j in range(5):\n        plt.subplot(1,5,j+1)\n        img = Image.fromarray(imagesIn[x[k*5+j],:,:,:].astype('uint8') )\n        plt.axis('off')\n        plt.title(namesIn[x[k*5+j]].split('-')[1],fontsize=11)\n        plt.imshow(img)\n    plt.show();","metadata":{"execution":{"iopub.status.busy":"2022-09-20T16:51:14.172811Z","iopub.execute_input":"2022-09-20T16:51:14.173109Z","iopub.status.idle":"2022-09-20T16:51:16.498061Z","shell.execute_reply.started":"2022-09-20T16:51:14.173074Z","shell.execute_reply":"2022-09-20T16:51:16.497195Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# normalize to [-1, 1]\ntrain_images = imagesIn / 127.5 - 127.5\ndel imagesIn  \ntrain_images.shape","metadata":{"execution":{"iopub.status.busy":"2022-09-20T16:51:40.326130Z","iopub.execute_input":"2022-09-20T16:51:40.326628Z","iopub.status.idle":"2022-09-20T16:51:42.688559Z","shell.execute_reply.started":"2022-09-20T16:51:40.326593Z","shell.execute_reply":"2022-09-20T16:51:42.687716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Make sure we can return to 'normal' color fotos after having normalized\nx = np.random.randint(0,idxIn,25)\nfor k in range(5):\n    plt.figure(figsize=(15,3))\n    for j in range(5):\n        plt.subplot(1,5,j+1)\n        img = Image.fromarray((train_images[x[k*5+j],:,:,:]*127.5 +127.5).astype('uint8') )\n        plt.axis('off')\n        plt.title(namesIn[x[k*5+j]].split('-')[1],fontsize=11)\n        plt.imshow(img)\n    plt.show();","metadata":{"execution":{"iopub.status.busy":"2022-09-20T16:51:45.706121Z","iopub.execute_input":"2022-09-20T16:51:45.706669Z","iopub.status.idle":"2022-09-20T16:51:48.035777Z","shell.execute_reply.started":"2022-09-20T16:51:45.706623Z","shell.execute_reply":"2022-09-20T16:51:48.035108Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Analysis - model building and training\n\nNow for the fun part! We will begin to train our GAN model. We have cropped and normalized our data above, which is what we will use in the model. We will declare two models: generator and discriminator. The generator will create images and try to \"trick\" the discriminator. The discriminator will decide which images are real or fake. Our goal is to fool the discriminator such that it thinks our generated images are actually real. We will use 200 epochs, which we hope will be enough to train the models. Next, we will create 16 images per epoch so that we can see what the generator is creating. Our generator will have multiple layers using activation functions such as ReLU and Leaky ReLU. Leaky ReLU will help the function from becoming saturated at 0. \n\n![Screen Shot 2022-09-19 at 17.07.43.jpg](attachment:9437718c-3721-47d1-9a78-18e854a4f430.jpg)\n\nhttps://www.researchgate.net/figure/ReLU-activation-function-vs-LeakyReLU-activation-function_fig2_358306930\n\nWe will also use dropout layers which randomly sets input units to 0 during each training step. We will use batch normalization which normalizes its inputs by keeping the mean close to 0. In the generator model, we will use 3 as our output so that we have 3 color channels, and we will use a 'tanh' activation function which gives higher gradient values during training and more weight changes in network. In the discriminator, we will use 1 so that we have a yes/no type of response (is or isn't real), and we will use a linear activation function. As suggested by other Kagglers. For our loss functions, we will use Binary Cross Entropy. When compiling the models, we will use the Adam optimizer with a learning rate of 0.0001.","metadata":{},"attachments":{"9437718c-3721-47d1-9a78-18e854a4f430.jpg":{"image/jpeg":"/9j/4AAQSkZJRgABAgEASABIAAD/4QDoRXhpZgAATU0AKgAAAAgABgESAAMAAAABAAEAAAEaAAUAAAABAAAAVgEbAAUAAAABAAAAXgEoAAMAAAABAAIAAAITAAMAAAABAAEAAIdpAAQAAAABAAAAZgAAAAAAAABIAAAAAQAAAEgAAAABAAiQAAAHAAAABDAyMjGRAQAHAAAABAECAwCShgAHAAAAEgAAAMygAAAHAAAABDAxMDCgAQADAAAAAQABAACgAgAEAAAAAQAAAYWgAwAEAAAAAQAAAIikBgADAAAAAQAAAAAAAAAAQVNDSUkAAABTY3JlZW5zaG90AAD/4gIQSUNDX1BST0ZJTEUAAQEAAAIAYXBwbAQAAABtbnRyUkdCIFhZWiAH5gAJABMAEAAmACxhY3NwQVBQTAAAAABBUFBMAAAAAAAAAAAAAAAAAAAAAAAA9tYAAQAAAADTLWFwcGyLFfVImp6xqZvkrY9U6DLEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAApkZXNjAAAA/AAAAChjcHJ0AAABJAAAAFB3dHB0AAABdAAAABRyWFlaAAABiAAAABRnWFlaAAABnAAAABRiWFlaAAABsAAAABRyVFJDAAABxAAAABBjaGFkAAAB1AAAACxiVFJDAAABxAAAABBnVFJDAAABxAAAABBtbHVjAAAAAAAAAAEAAAAMZW5VUwAAAAwAAAAcAEcAMgA0ADEAVgBDbWx1YwAAAAAAAAABAAAADGVuVVMAAAA0AAAAHABDAG8AcAB5AHIAaQBnAGgAdAAgAEEAcABwAGwAZQAgAEkAbgBjAC4ALAAgADIAMAAyADJYWVogAAAAAAAA9tYAAQAAAADTLVhZWiAAAAAAAACDAAAAQ20AAAFXWFlaIAAAAAAAAE5BAACvbgAAC6dYWVogAAAAAAAAJZUAAA0lAADGL3BhcmEAAAAAAAAAAAAB9gRzZjMyAAAAAAABDNYAAAYj///yxQAAB/IAAP1p///7f////ZQAAAPzAAC+y//AABEIAIgBhQMBIgACEQEDEQH/xAAfAAABBQEBAQEBAQAAAAAAAAAAAQIDBAUGBwgJCgv/xAC1EAACAQMDAgQDBQUEBAAAAX0BAgMABBEFEiExQQYTUWEHInEUMoGRoQgjQrHBFVLR8CQzYnKCCQoWFxgZGiUmJygpKjQ1Njc4OTpDREVGR0hJSlNUVVZXWFlaY2RlZmdoaWpzdHV2d3h5eoOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4eLj5OXm5+jp6vHy8/T19vf4+fr/xAAfAQADAQEBAQEBAQEBAAAAAAAAAQIDBAUGBwgJCgv/xAC1EQACAQIEBAMEBwUEBAABAncAAQIDEQQFITEGEkFRB2FxEyIygQgUQpGhscEJIzNS8BVictEKFiQ04SXxFxgZGiYnKCkqNTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqCg4SFhoeIiYqSk5SVlpeYmZqio6Slpqeoqaqys7S1tre4ubrCw8TFxsfIycrS09TV1tfY2dri4+Tl5ufo6ery8/T19vf4+fr/2wBDAAEBAQEBAQIBAQIDAgICAwQDAwMDBAUEBAQEBAUGBQUFBQUFBgYGBgYGBgYHBwcHBwcICAgICAkJCQkJCQkJCQn/2wBDAQEBAQICAgQCAgQJBgUGCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQkJCQn/3QAEABn/2gAMAwEAAhEDEQA/AP7+KKKKACiiigAooooAKyNe1/Q/C2kT6/4ku4bCxtV3SzzuI40GcZZmIA5IA9616/PX/goJo+g/E7Sfh5+zhr0t9Hb+PvFdpb3H9n3M1pL9m05X1CXMkDK4Um3Rc5GCRjnFAH2lY/EvwDqUWly2WrWz/wBtbvsKlwr3GzIbYhwx24+bjjvW5q/iXw/oFzZWet3sNpJqM32e2WVwnmy4JCJnGWIHA6ntX5peMLm88H/tleMh4mtNSv7ZPAOnQ+FINOOb55PtVympizdjkXBzamR87goDE8Zr5VGhfHfT/wDgnh44vf2l77xNc/En7UmmWFvqUoZUvLS7xpUmleWo3ySK0TSXALPJLuy2F2qAftr8T/il8Pfgv4Gv/iX8VNXttC0HS0D3V9eOI4YlZgi7mPHLEAepNdva3VtfW0d7ZyLLDModHQ5VlYZBBHBBHINfHv7aHwM+G37Qn7K+q+Afjvo0PiDTUigvJrG4LeRJcwEFDIisBIqudwVsruAOMgV9S+DvCuieBvCmm+DPDUbQ6dpVtFaWsbO0hSGFQiJucsx2qAASScCgDpKKKKACiiigAoorzuX4t/DKD4nw/BWbXbJfFs9g2px6SZV+1tZq/ltOIs7tgf5c+tAHolFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABSMyqCzHAHUmlr8+f2/Lz4qa5H8LPgv8JtWXSJ/G/jaytNVlMPnk6PY29xqN4pXfH8khto4pBnDJIUPDUAfoGssTqrowIboQev0p25Q23PJ7V+Ucnja18CfFH4q3upaxB4XtPhvp2maT4ZiuYZZ7C2bUIi5uVso5EM0ssxESAMCcBARmvCY/2hfj78Qf2KrP8AbG8Zajpsfi/RtS0pLLStHt7q0ezuhPFZanp995k7G4Mzs7mEooh4T94U3kA/afx3468JfDLwbqfxB8eX8Wl6Lo1vJd3t3O22OGGIbndj2AAzWj4Z8SaH4x8O2HizwzcpeadqdvHdWs8ZyksMyh0dT6MpBFfOX7Yvwz8E/Gv9lPxh4G+JtgNQ0i+0mSS5tGd0SQxr5gR/LZSyblG5CdrDhgQSK7z9m7wV4c+HfwB8G+DfCULW2m2Oj2aW8Jd5PLTyVIUM5ZtozwM4A4HFAHtlFFFABRRRQAUUVwWsfFL4c+H/ABzpfwz1vWrO18Q63HLLYadJKq3FxHDzI0cZO5gueSKAO9ooooAKKKKACiiigD//0P7+KKKKACiiigAooooAKwLnwt4cvdft/FV5ZQy6jaRtFBcOoMkaP94IT93PfHWt+igDG1Dw7oWq39rqmpWcM9zZMWt5XQF4iepRuoz3xUmo6Fo2r3Frd6paxXEllJ5sDSKGMb4xuXPRsd+tatFAHjv7QP8AyRjxF/16H/0IV7An3B9K8f8A2gP+SM+If+vQ/wDoQr2BPuD6UDHUUUUCCiivnb9qv9qL4Rfsb/ArXv2gvjbqK6foehQGQjIMtxMeIraBDy80z4RFHUn0oA8Z/wCCgn7dPgj9hT4LN411K2k1zxXrUo03wv4etRvu9V1Ob5YYI0HO3cQZHOFVckkV4T/wTU/Yb8X/AAc/tn9rz9q6ZNd+PXxNVbnXr3dvi0u1bDw6RYcYjt7ccNt+++SSRgDxH/gn5+y78XP2lPjE3/BU39vDTWs/FmsxFfAfhG5BMfhTRZP9UzxtwNQuk2STtgMh+Xg5A/dOgAooooAKKKKACiiigAooooAKKKKACiiigAooooAK881X4W+Ddc+JOkfFnVbd59a0G1urOwkaR/LgjvTH55WLOze4iRd+NwXIBwTn0OigDzC/+Dvw91L4hr8UrmwB1n7OtpJKGIWaKNt8ayx52SeW/wAyFgSp6EUvij4O/D3xje6Xd67YCRNIvP7Qgt0Yx25us5E0kSYSRwckFwcHnrXp1FAHj/7QPHwO8W4/6BN1/wCimrf+En/JKvDP/YJsv/RCVgftBY/4Ub4tz/0Cbv8A9FNW98JP+SU+GP8AsE2X/ohKAPQqKKKACiivJfjt8cvhb+zV8Itf+Onxp1eDQvDHhq1a8v724YKkca4AHuzsQiKOWYhRyaAPGv24P2z/AIW/sKfAXUfjZ8S/Nu5EZbTStKtVMl3qeozZW2s7eNQSXlfC5xhRya+JP+Cbf7GnxnfxlqP/AAUO/b3dL742eObcx22mI2+y8K6MzbrfS7ReVEu3DXEgyWkJGTglvG/2IPgx8Uf+Chnxzs/+CoH7Xulz6X4ctlY/C7wbfLj7BYucpq15Ccr9suFwydTGuMHkGv3+oAKKKKACiiigAooooA//0f7+KKKKACiiigAooooAKKKKACiiigDx39oH/kjHiL/r0P8A6EK9gT7g+leP/tAf8kZ8Q/8AXof/AEIV7An3B9KAHUUVXu7u1sLWS+vZFihhUu7uQFVVGSSTwAB1oA5nx9488HfC7wVqnxG+IWpW+j6HoltJeX17dOI4YIIlLO7uxAAAFfgP+zp4D8Tf8Fh/2hdO/bm+O+nXNj8DvA955nw28NXqFBqt1C3/ACHrqM4JXcP9GRsgAbsVl+MLjWf+C4n7SR+GmhXVxbfsrfC/U1fXJ4GZI/G+s2z8WG4bd+m27A+eMlZWwMdCv9Eui6LonhjR7Xw/4ftYbCwso1gt7a3RYooo0G1URFAVVUcAAYFAGoAFAUcAUtQrcW7ztbI6mRACyAjcAehI6gHBxQlzbySvbxyK0keN6gglc9MjtmgCaiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAPIP2gf8Akh3i3/sE3X/opq3fhJ/ySnwx/wBgmy/9EJWD+0Fx8DfFv/YJuv8A0U1b3wk/5JT4Y/7BNl/6ISgD0KiimSSJEjSyEKqgkk8AAdSaAMjxH4j0LwhoF74q8UXcVhpunQvcXNzO4SKKKNSzu7NgBVAySa/nc8D6H4k/4LgftD2/xa+IVrNbfss/DrUmbQtGuFKx+MdVtiQmoXA4LWMD8wrna7KCQRWh8ffH3j//AILG/tHXX7F37P2oT6Z+z34EvlHxJ8V2Tsv9u3cTbv8AhHbF1ZMxHBF44LKVYKQQMN++vhjw18Pvg94I0/wd4WtbLw9oGkQR2lpbRBLe3giQBURB8qgdAPWgDtkjSJBHGAqqMADgACnVmR63o82rSaDFdRNexRiV4A4MioxwGK9QCehPWk/tvR/7Y/4R77VF9v8AK8/7PvHm+Vnbv2ZztzxnGM0AalFFFABRRRQAUUUUAf/S/v4ooooAKKKKACiiigAooooAKKKKAPHf2gf+SMeIv+vQ/wDoQr2BPuD6V4/+0B/yRnxD/wBeh/8AQhXsCfcH0oGOr+fv9sv45/Ef/go7+0bcf8Ev/wBjXWJ9M8MaEyzfF3xpZEp/Z1ox/d6JYS5Ae9usETFd3lxnnktj2T/gpL+138XNX+Iei/8ABNz9h2ZJPjB49tnuNQ1TKtB4X0QECfULk5yszqSLZMEuwJ7V94fsY/se/Cj9iP4IWHwa+FluzbWN1qeoznfd6lfy8z3d1KfmkkkbuxOBgDgUCPWPgf8ABD4X/s4/CzRvgz8G9Ig0Tw7oVuttaWsChVVVGCzH+J2PLMeSSSa+QfF1n43+MH7f8Pw/vL+F/BHgvw3b6pd6e8EjGXUru5P2d/MWdArIkW5SY2AGRjnI/RavOPCfwu8M+EPGXiDx9YeZLqviV4DeTStu+S2QpDGg/hRAzYA7kmgD8TfDn7XPiX4FfsPa7+1dFqOl6h8Ste8W+I4ryy1eOaWa9XS9dvbCHTYBDJG8S2VtGiK+11RFLMp3En6Q8SeLvHmk/G79mr4jT6ppt14l+Iep6jpOvNoMUsNhqGkjRr7UIgUllmdhZzRQskhf7zNgKH21+gek/s+/CDRvFGseLLTQ7Vp9ckaa5iljSSDzpFKTyxxMpWN7gH9+Vx5uAXyck3tN+CPw00nx/b/Eqz01F1OxsRp1h/zxsbbnclpCMJB5mcOUALAAE4GKAPkr9pj4+/tEfDn4jnw98MvsX9mi3jk/f+EfEmtNvYfN/pWlsLfH+z94d67D4m/HT4o+D/2NLj4u2d9o9l42fT5GsU1PTL+1tLrUfn+z2q6fPLFfK87qsaoz78nIBGBX27WVq2g6Hr8cMOu2UF6lvKk8SzxrIElj5SRQwOHXsw5HagD5e/Yo+LHxS+O/wE0j4y/FK50N5/EsMV/a2mhxzqlhBNEji0uJZ5ZDNcxMSJXVIlDfLs+XJ+tqx9B8O+H/AArpiaL4YsbfTbONnZYLWJIYlZ2LuQiAKCzEsTjkkk81sUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFAHj/7QRI+B3i0j/oE3X/opq3/AISf8kp8MZ/6BNl/6ISsD9oL/kh3i3/sE3X/AKKat/4Sf8kp8Mf9gmy/9EJQB6DX4U/t+/tM/E/9qH45x/8ABKz9iPUpLbxDfRpL8RfFFmSP+EX0SUAlI5Nyr9uuVYLGg3FVJZh0r2H/AIKX/txePPhff6B+xZ+yHCusfHb4oB7bSYxh4dEsT8txrF9g5SK3U7kG072GAPX6T/YJ/YY+G/7CnwaTwL4Zlk1nxLq8n9oeJ/EV4TJfaxqkvzT3M8rZYqXLeWmcRphRQB7N+zL+zV8I/wBkf4MaN8CfgnpiaXoejRBFVRmSaU/6yeZ+sksrfM7nkk18ifG3wfJ+0B/wUB8J/CfX9QuW8KeDfC1z4g1HS1S3ktLy7v7g2UEdys0MjECFJiNjIwyCCMnP6b1514c+Fng7wv441z4j6ZAzaz4h8lbu5lcuxjgXbHGmfuRrkkKOMknqaAPxi/4X34r+D/7Ofxr+LWg6hct8TovHevQWVu1k17cXi6YwlstNCCN2S2e0VcOANgdmUg16v4x8aeMdbsv2cPinbeJRr/iPxN4wijmvdPtPscUmnXUE73dn5Yy5tYApAMrFiUDMS1fq/p/gbwlpPiW88YaZYRQalqKqtzNGNpl2fdLgfKWA43YzjjOKdL4H8J3HiqDxtPYxvqlrAbeCdgSYoySSEBO1SckEgAkcE44oA8R+NXin4y6Fr1tb/DiYx2rw7pANAm1b58n/AJax6jZhOP4Sjeue1Zvjv4pav4H/AGbrzxt8Rdefw/qTRi3XUY9IdJY7ieQRQCPTmlu2MjOyqil5AWIyCOK+qqayI4w4BGc8+ooA+B/+CdfxQ+N3xn+A1r8Rf2gNatrzxROEs9T0uysvskGl39mDDdwBmLPI7SKWcklAeE+XGfvqmpGkYIQBcnPHqadQAUUUUAf/0/7+KK5bxf4pj8IaT/a0ljeagNwXyrKIzS899o7Dua8s/wCF+2n/AEK3iP8A8Fz/AONAHvlFfCPx/wD2/Ph9+zh4ITx9488H+MJ7J7hLYLYaLcXUu+QEg+XEGbHHJxgV8Wf8P6v2Wv8AoRfiT/4Smo//ABugD9waK/D0/wDBev8AZaH/ADIvxJ/8JTUf/jdH/D+v9lv/AKET4k/+EpqP/wAbp2A/cKivw+/4f1fstf8AQi/En/wlNR/+N0n/AA/r/Za/6ET4k/8AhKaj/wDG6LAfuFRX4ff8P6v2Wv8AoRfiT/4Smo//ABuk/wCH9f7LX/QifEn/AMJTUf8A43SA/WD9oH/kjPiH/r0P/oQr4k/4KSft4ap+yv4R0f4R/ArTR4r+NPxFkOm+EtBiIZhKww99dKDujtLYEu7kbTjGa+Ifjt/wXL+Bfin4P+IfDfw7+HHxE1HX72zeLTbWfw3fW0U902BDE88kYSJXfCl2OFzk8V0H/BOT4O/EXwf4y1/9uj9trw5r+s/HDx+oV4vsEktt4b0sHMOk2GXKqqrgzOiqXfOc0DPtr/gm/wDsDWX7Fnw71TxH4/1V/F/xa+IFwNX8ceKbjJl1DUHyxjjB4jtoCxSCNQAFA9sfpDXgf/C/bQ/8yr4j/wDBc/8AjSf8L9s/+hW8R/8Aguf/ABoFY99orwL/AIX7Zj/mVvEf/guf/Gl/4X7af9Ct4j/8Fz/40Ae+UV4H/wAL9tP+hW8R/wDguf8Axo/4X7af9Ct4j/8ABc/+NA7HvlFeBf8AC/bPr/wi3iP/AMFz/wCNL/wv20/6FbxH/wCC5/8AGgR75RXgf/C/bT/oVfEf/guf/GvnXxr/AMFANG8E/tH+Bf2erzwB4ruZPHkd01vqEGnu0Nm1qU3NdjP7uJhINsnTIYHpQB+g1FFFABRRRQAUVz3ijxCnhbRZdaktLm9ERUeTZxmaVtxx8qDk47+1eR/8L9s/+hW8R/8Aguf/ABoA99orwP8A4X7af9Ct4j/8Fz/40f8AC/bTp/wi3iP/AMFz/wCNAWPfKK8D/wCF+Wn/AEK3iP8A8Fz/AONH/C/LT/oVfEf/AILn/wAaB2PfKK8Af9oGwjXfJ4Y8RKPU6e4H864bXf2z/hN4XUv4lgv9OA6m5iSLH/fcgoEfXFFfmp4r/wCCtf7EHgeFrjxV4tt7NVzndNbsRjrwspP6V806j/wcUf8ABKLTbhrST4i+dMpI2QWk8zZHp5aNmgD9Yf2gcf8ACjvFuef+JTdf+imr4R/a7/b78KfsL/speD73TtOm8V/EPxZZWGleEPClh+8vtV1CWBAu2NcuIYhl5pMbVUYJyRXyL8bf+C9P7Cfi/wCDHiTSvhqPFHiO9vtMuIreOy8P6gytI8bBA0hhCqCerE4HWvPf+CZHwb+JGo+NE/4KFft0+F9d1T4oatpkWm+GdN+wNPZ+F9DVFWOGyyxRZ7hAHnlVVY7iuSC2QD79/wCCa37BfiH9mjSNZ+P37R+pr4r+OfxGK3firXDnbGvWHT7VfupbWw+VQANxyTX6m14H/wAL8tP+hW8R/wDguf8Axo/4X5af9Cr4j/8ABc/+NAz3yivA/wDhflp/0K3iP/wXP/jSf8L9s/8AoVvEf/guf/GgR77RXgf/AAvy0/6FbxH/AOC5/wDGj/hflp/0KviP/wAFz/40Dse+UV4H/wAL9tOn/CLeI/8AwXP/AI0f8L9tP+hW8R/+C5/8aBHvlFeB/wDC/LT/AKFXxH/4Ln/xr50+L3/BQLRfhF8TPAvw6v8A4f8Ai3UT45vpNPintdOd/srou/zJ1zkQ44Z+inGetA7H6D0UituUN0zzzS0CP//U/v4ooooAa6JIMSAMPeszUbrQ9HtWvtWkgtYF6ySlUUfUtgVq1+ZX7R+i658Yv27/AIYfBqPV7i00PQtG1HxPqVnHHDJBdN5iWcEc6yo2VZXlGO3UYPNAH6Ht4h8Ir4gi8KtdW39pzwm4S13L5rRDguE67c9+lF34g8Jafr9r4Vvrq2h1K+jklt7Z2VZZUiKh2RTywUsM46Zr8c9N+JWreAdN/aU8Vavq1xonjweOJdE8NzRWYvbryG0yyOkWttCyOvkzSMx52xhi5ZhgkYHjH4ifEf4n/sAaB8f9R1/+3viDqevaLcaMsNmlpJpuoPdW8M+m24VQ7hWEqzFmYsS4J2jAAP3K+zW3/PNfyH+FH2a2/wCea/kP8K+V/wBprxb8bfC8ejn4QaT4g1NpvN+1f2FaaRdbMbNnm/2rfWe3OTt8vfnndjjPLeHdf/aG8V/s2eN5tV03W9N8W/2ffJpMesW+m21w05tn8gxrpl3dxEeZjBZw2e2KdwPsi2k0m83i0MUvlsVfYVbaw6g4zg+1Wfs1t/zzX8h/hX4Y/wDBLG9u9f8AiDoWseDZL59FtPhL4WsfEQuvOATxJEZPPimEvP25B5gucjeMrvOcV+6tIDxf4/29uvwa8Qssagi1PYf3hXsyfcH0rx/9oD/kjPiH/r0P/oQr2BPuD6UAOooooAKKKKACiiigAprMEUuegGadWfq2l2Wt6Xc6LqSeZbXkTwSrkjckilWGRgjIJ5HNAHw1+zL+2Z4t/aN+Kviz4bJ4E/siDwLqV5pGt6iNVhuooryPy5bSOBEiRpxPbyiSRlwsD/uyXOSPu421sbj7WY180Lt34G7HXGeuK8k+G37P/wAHfhBq97r3w10G20e71KGCC6kt1KmZbddkZfnDMFABc/M2OSa9ioAKKKKACiiigAooooA+cv2qtR/ao0v4N393+xpp/h7U/HnmRLZw+KHnTTdhcCVpTbOkuVTJUKwya/MO38A/8HDvjmxX/hIPiL8F/AzuORpXh/V9SkT8bq/VCRX2T8Qfjv8AFXWP224P2ZPBltPp3h/QfCkXivWdXgNs7E3N5Lb21t5cwZtjraz7toD5KkEAV5r4c/bI8QfDD9m2D9rz4u2GqeIdJ8UaxHHJBpSwtFounTXn2O3k8l5Ell271aXylkkY7iBgAUAfJ9//AME7/wDgs746nEvjz9t5dJiY/PF4d8E2tpgeiSSXzkfUg13Wm/8ABHj4x6jbJ/wsj9sn45ancEZl/s/WbDTYWPfbHHYOyr6DeT719eSftAfELQvjz8Lk8SaZqGiab8VotRs5NB1JreSfTrvT4TcxTb7Z5UHnQg70EjAHb0bIr2n4qftGa38M/FbeGLD4ceJvE0axJJ9t0qXRUtyWz8mL7VLSbcuOf3W30JoA/M2X/g39/Y98S6kdZ+L/AI0+Jnj28f782u+LLyRn/wB4W4gH6V6pon/BCL/glfo8ax3vwrtdYK/xard3l8T9fPmcH8q+nf2oP2jfGXw5/Z20b4i+D9Ol0TWPE2r6NpEMOpLBNJYtqt5FbM8ot5ZoGeJXJG2R0LAckV81Qftp/GLTv2Qtd8bagbObxhY+O4vAdrfNDi2L3etQaTDfTRKQP3aT+Y6rhSy4xg4oA9w8B/8ABKf/AIJt/DK/XVfAvwR8HaddIMCaPSbcyY/3mQn9a+q9K+BPwS0KNYtG8IaLaqgwoisbdcAemEr8ufGf7Zv7R/hH4NeJ9U0250261LwL8R9M8H319dWp3X1lfX1jD5sccTJFHKI7tgx2lcqMKDX2R8Tf2yU+HH7Qmkfs2W3gXXtd13X7I3+my2LWX2ee3gkiS7kZ5bmMxLbCZWbzApfBEYYgZAPXPjn4d8P6Z8DfFi6ZYW9uBpN1jyokTH7pv7oFdt8JP+SU+GP+wTZf+iErn/2gCT8C/FhIwTpN1x/2yauh+En/ACSnwx/2CbL/ANEJQB6DRRRQAUUUUAFFFFABRRRQB8KfFj9sXxV8Pf2ntC/Zh8PeBzruo+JLT7dY3Q1SG3U28EqJfPIjRsYxbxuHUFt0x+VQOtfcpt4JJVuJI1MiAhWIBIz1ANeMXn7OXwV1D4iRfFq98P28viSCc3MeoNuM6OUEZ2tnhSoAKj5T1xmvbaACiiigD//V/v4ooooAK880f4XeENG+IOqfFG3haTXNWhitpbiVixW3hJKRRjoibiWIHUnJr0OigDjh8P8AwYvjdviQmnRJrj2ws3vFBWR4FYssbkEBlViSu4HBJxjJo1j4f+DfEPiTSvF+t6fFdajohkawlkBb7O0o2u8ak7Q5XjfjcASAQCc9jRQAUUUUANVVX7oAzzxTqKKAPHf2gf8AkjHiL/r0P/oQr2BPuD6VV1DT7HVbOTTtThS4glG145AGVh6EHg1cAxxQMKKKKBBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQB82fCP4Hav4M+LfxF+MnjLUI9R1Hxvd2aW8caFUstM0+1WGC1BJJYmZp52PTdKQBxXjY/4J8/CDW9Avfh18RGn17wgr37aVpcsskaWQ1PJuR+7YLIUZj9mdl3QAnac81970UAfN9j+znoqfFXQfiLq15JeW/g+xkstBs5CX+zNcKFuJ5JXLPLLIqhQSeFz6163rvw2+HXii//ALU8TaBpuo3RUL511awzSbR0G51JwOwzXa0UAeV/EX4K/DT4p/Dx/hX4v0uKTQy0Lx20P7gQvbyLLC8Rj27GjdVZCuMECvHLj9h79nO88I3HgO90iabRryzntLi0a7n8qY3F0l69w6hwDdfaI1kWf76kcEDivreigD5F8Z/sR/Avx18L4/hDrEN/HpA1OPWZvIu5I57m/hmS4juLiYfPI6yxo4JPVRxgYrZsP2PvgjY/E+0+NL2l5c+KrOQOmpz3k7zbRCIfKPz7fJKjc0YAUuSxBNfUNFAHj/7QX/JDvFv/AGCbr/0U1b/wk/5JT4Y/7BNl/wCiErtr+wstUspdN1KFLi3nUpJHIoZHVhgqwOQQR1BqS1tbaxtY7KyjWKGFQkaIAFVVGAoA4AA4AoAnooooAKKKKACiiigAooooAKKKKACiiigD/9b+/iiiigAqNZoXcxo4LDqARmiaaO3heeY7UQFiT2A5NfyyfAb4geMNW/aVb4z6/bz6ZDaan4z8eWl3Z31xcza7pelGbTo9JW22CKFeFnwWct8pUDnAB/U6zoi7nIA9TxSGWJSAzAE9Oetfzq/EL9qr48/HL9kiPVPjNdaPoum/EO58LaZHBpt5EbixuNTmWe/C3NtNKi28dvhIzJIJshmdVUqK9n+MP7R/wr8W/tS+AtZ8KeIIz4O8JfD/AMV6rYXjyyQQ6hqsF5b6PDBGX2C4MZWcrjcGyrqSpBIB+4iOjjchDD1FOr8rNA8ceNf2Jf8AgnR4J1DXI/P8UXkmi6XNNes8kVte+IL+K3NzdMWyIbY3PmSZYfKmMjrXkfw//b2+IHja/j8OxeK9It9L1XxTrek2+v3kCwvbWOjW6fOYSwRpp7lj5YIw0KllBJBoA/a+ivwe+IHxw+OkGtfHq7134hW8sPhrT9J8LWOk/Zlhhe/kt4mvtR27jJDEz3qgMHxhMbvlzXqnif8AaV+P3hH4o+IfgZ8LdV0m8/4RzT/DUunwPZzTyypcR3U+ofvFkOYktII385uN7FFJdlFAH7G0V/PXqn/BS741+JPAHi/xp4T1bTYprX4Wa14vh09Lb95Zap5jR6VbLI7YlZUjka6ByEOM7BxXTeKv2u/H/wADf7POsa0nxE1XwP8ADddZFxgR/b9VvbmKwU5hIWSKN8vMwBwgLZBoA/e6ivxA8Xftg/th2fxLufhz4CutH1S0tfE2gaKmrPZM0Nybyye71lY9koGzTwEKkZOW2MxZTX1N8Bv2pPGPxA/4J5TftLeNr6zTW4dI1K7kmtkCxo9uZfJ3xbm2PsCFkJ4JxQB+iMV5aXErwQSo7xcOqsCVz6gdKsV+IPgLxj8Wv2Sfg38Cdb8Sa1YT6p8QNQS48Y3epwbZmszp9zqNxtkVgPNhKrHEoUs5YAA4xXhkX/BTD9qTxD8JNI+KWh3mi20V74Jk8T3cbWbS/Zrm51P7JptszCXAMivGs+7HlgM3B6AH9GNFflJ4g/a3+IWuWPxI8R6P4n0nR4/AKvFBpcdv9pu9REOjwX010MvuWIz3IWBlG0ohY7twx8waN+0R8Svh9b/CuLx74tXx9qfhjwD4h8eak20Rte3ttDFaWtpA8JUTMGnuS4If5Y84BANAH79UV+El/wDt2ftZaho8dt4IOkXWr6pqXgux09pbSQ2jX/iIrLfWHmI4zHZ2xeRpRlxtwea/S39jn4v+JfjT8LL/AMS+LL+01G6ste1jTEmtY/J3QWF9LbRNLFuby3ZY844ypBxzQB9XUUUUAFFFFABRRRQAUxZI3JCMCRwcHpSu6xqXc4UDJJ7Cv5/fg7+294X+DPwN+Knxp0u9h17xl478c69eeHtKMzTL/Z0N+ml2U0vzhYLYKgmYs8YO84IzkAH9AlVzd2ouRZGVBMy7xHuG4qOCcdce9fin4h/bi/aF8B+FHm+I2paFpa38fiDW7C9RUupV0vRra1jjtXFtJLbPfT6hO+FieQLbKM/vAxHz5D+1t8T/AIgftOfDrWRd2Wj65qHhLR9Fu/Er7G0XTdQ1vztYu4XhaTc809pYLDbdUWVsO6khWAP6N6jSaGRisbBiOoBzX5Vz/tL/ALRei/tPah+z/rF9pdzFo17Frl7eQ2xjSHwubKWSRpN8jYl+0Kkasp5yxxgcfkd+y58U/iB8P5rr9snXdPltxBoXin4l29suozXU3iOw8RahM2nWyQkLHbR2ETw70bcyhgVIG7AB/WO7pGu6QhR6nikMkYIUsAT0Getfgb8dPjt8Vfi/+zTa+Hfi1q2n6Ovj3xr4c0bSXtLmCOeKNTDfan50lrcTRRxrFHJ5StKZAmfM5ZRXofxN/aE+Eetfta/8L+s/FMUfg74bfDPVZLSIySxLqV/qN4Iw1vE237QbdLB412qxLTrszuoA/bBHSRdyEMPUc06vym0Lxn47/YZ/4J3fDbT9TVbnxVeS6Jotxcak7vFbXutXSLPcXLFslLfzXbBYAlQuQDXiPwf/AOCinj74o6wnw7XxJo9kdT8WeK9MtfEF1AscUOleHZPssTvEZFU3F1cKzRg4DRfMq96AP3Hor8FfE37SHxmtYv2jPGuv/ECCZPBcdr4V07Rltljglu4rG3lkv1TeZY0muNREbMGwVjX5vlzXpmt/tKftGeGvHGs/s5/B7U9GvNY0TTfDC6TDJazTmV7xHkvS0nnHMEdtH5rzscDlBmRkBAP2eor+e+0/4Kd/Fvx/4J1Tx34W1TTLMv8AD/xB4rOk/Z8zWE0DyQ6VCJmb95MTGXuF52+igc9Hrf7Ufjf4N2vw6j8d+I1+IV54T+Hup+LL6eLbHJfalai3sY4UaFwsp82WbzOG4QtwRQB+9tFfhR4v/bq/av0O+13QtCk0e/e2ufAllYXws5GtptY8U3E8d5pqskuClpEIJWfl0VyGOSMffH7Gf7Qvif40/AnW/iJ4+v7K4udI1fV7Iz2sfkqLewmeNHkiLMUbCE4J5GDQB9rreWj3DWaSo0qDLIGG4A9yOoqxX4C/BH4r/F74S/sffDX9qq41SzuNe+MfiqxvdQk1aDb5Gma5cvcGESeYBGsNoAUznBAUD109E/4KHftEfFD4V6L8Vfhpf6HFa6zP4zu5Lee1kee00jQZrqOymceYD50qx25MbqpPnZOOBQB+89FfkJ8Df2z/AIvfHfX38FX+s6J4U1Pw1o3hnUb7dELk6vc6tbve3sNonmgrFFbx7EZSxErEklUKn56+Dv7R3xg8aeHvgpbfErxfb+IB8S/HGqXeoREJD/Z+nabDdXdrbLJE4IkMsdoAjEkhypU0Af0AUV+BPjP/AIKMftQ/8Kqu/ix4MtNNEN74U1vxLHatavK+li2uvI0m2nAlUm5uyVWVW+62cKAK+9/2Kf2hfif8YfGnxM8CfFe80241HwRqOm2DJYRNB5c9zpVrfXUYV3dnije5VUlP3juH8PAB9/0UUUAf/9f+/iiiigBkkaSoYpQGVhgg8gg9jXOWPgvwfpn2f+zdKtLf7Jv8jy4UXyvM+/swBt3fxY6966aigDi7r4b/AA8vtJj0G90LT5rGFzJHbvbRNErt1ZUK7QT3IGat/wDCDeCza2Vl/ZFl5OmndaR+RHtgPrEMYQ/7uK6migCnqGnafq1m+napBHc28ow8Uqh0YehVgQayG8H+Em2BtLtDsm+0LmFOJQMeYOPv4/i610dFAGJN4Z8OXEl1NcWFvI96FW4ZokJlCfdEhI+YL2znFWodH0m3vX1GC2iS4kRYnlVFDsi52qWAyVGTgdBWjRQBzKeC/B8cckKaVZqksckLqIIwGjlOZEPy8q55YdD3q2PDXhwEEWFuMQfZh+6TiH/nn0+5/s9K26KAMaz8OeH9Ogt7WwsbeCK0z5CRxqqx7hg7AAAuR1xXmnxY+Bvgj4v/AAyuvhBrsbWeg37ILq3ssQiSNXEjR/KOFcj5sckZHevZKKAMi48P6FeQWtreWcM0dkyvbrIisImQYVkyDtKjgEciqUfg7wjDZtp0Ol2iW7x+U0QhQIY87thXGNuTnHTNdJRQBiJ4Z8ORXU99Hp9ss91GsM0giQNJGowqOcZZQOADwBSQeGPDdsYDbafbR/ZYmhh2xIPLjbG5EwPlU4GQODW5RQBh2vhnw5Y28NpZafbQxW0nnRIkSKscmCN6gDAbk8jnk1fsdO0/TI3i06CO3WR2kYRqFBdzlmIHUk8k96u0UAFFFFABRRRQAUUUUARyxRTxNDModHBDKRkEHqCK8y0b4H/Bjw6tynh/wno9iL2Iw3AgsYI/Njbqj7UG5T3B4r1GigDkpfAPgaeystMn0axe304MLSI28ZSAMpRhEu3CZUkHbjI4qA/DjwCLUWcWi2KIrwyKFt4gFktxiBwNuA0Q+4eq9sV2lFAHz18JP2cvC3wxi1261W+u/FOqeJHb+0b/AFcpNLLB8wS2wFCiCNWKqmMYznJNep2fw68AadDa29holhBHYwPa26x28aiKCT78UYCjajY5UYB9K7KigDzfUvg58JNZ0K18Mav4Y0q602xYvbWktnC8MLNwTHGy7VJ7kAVeb4X/AA2b+zN3h/Tj/Yv/ACD/APRYv9F/64fL+7/4Diu6ooAyta0LRPEmmvo/iGzhvrSXBeG4jWSNtpBGVYEHBAI461z5+Gvw7ZURtC08iO5N6o+zRYFy3WYfLxIe7/e967WigDlpvA3gq4k1CW40izdtWCi9LQRk3IQYUTZX59o4G7OK0LXw74fstRbV7Oxgiu3iWBpkjVZDEn3ULAZKr2HQVs0UAcRD8M/hzbrItvoGnIJUljcLbRDck/MqnC8iT+MHhu9aEXgrwdA1s8GlWaGztzaQFYEHlQNgGJOPljOBlRxwOK6eigDl7PwP4M0/T4NJsNJs4bW1mFzDCkEapHMpyJFUDAcHkMOa5Hx/8IvDnjf4Za58LbAnQrLxDDNBdSacqQy7LnicqQMB5FLAvgkZz1r1aigDi7D4deBtO8IaZ4Ch0q1bR9Highs7SSJXihW1ULDsVgQDGFG0jkdqsWfgLwPp8TQWGj2UCOsysscEagrcNumBAUZEjcuP4jyc11lFAHJWXgHwNpupQazp+jWUF5a2/wBkhnjt41kjg/55I4XKx/7I49qW08BeB7BbVbHR7KEWMz3FsEgjUQyyAh5I8L8rsCcsME5Oa6yigDlYPAvgq1gu7a20izjjv5POuVWCMCaTdu3yAL8zbucnJzzWrY6Doml311qem2cNvc3zK9zLHGqvMyKEUyMBliqgAE5wBitWigAooooA/9k="}}},{"cell_type":"code","source":"# set globals for training\nEPOCHS = 100 \nNOISE_DIM = 100\nBATCH_SIZE = 256\nLATENT_DIM = 100*100*3 \nONES = np.ones(BATCH_SIZE)\nZEROS = np.zeros(BATCH_SIZE)","metadata":{"execution":{"iopub.status.busy":"2022-09-20T16:51:51.364381Z","iopub.execute_input":"2022-09-20T16:51:51.365256Z","iopub.status.idle":"2022-09-20T16:51:51.372143Z","shell.execute_reply.started":"2022-09-20T16:51:51.365214Z","shell.execute_reply":"2022-09-20T16:51:51.371254Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#reshape images for model\ntrain_images = train_images.reshape(-1, 100*100*3)\ntrain_images.shape","metadata":{"execution":{"iopub.status.busy":"2022-09-20T16:51:51.727426Z","iopub.execute_input":"2022-09-20T16:51:51.728176Z","iopub.status.idle":"2022-09-20T16:51:51.734613Z","shell.execute_reply.started":"2022-09-20T16:51:51.728139Z","shell.execute_reply":"2022-09-20T16:51:51.733519Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#build generator model\ndef create_generator(latentDim):\n    with strategy.scope():\n\n        inputs = Input(shape=(latentDim, ))\n        gen = layers.Dense(256, activation=LeakyReLU(alpha=0.2))(inputs) \n        gen = layers.BatchNormalization(momentum=0.8)(gen) \n        gen = layers.Dense(512, activation=LeakyReLU(alpha=0.2))(gen)\n        gen = layers.BatchNormalization(momentum=0.8)(gen)\n        gen = layers.Dense(1024, activation=LeakyReLU(alpha=0.2))(gen)\n        gen = layers.BatchNormalization(momentum=0.8)(gen)\n        gen = Dense(100*100*3, activation='tanh')(gen)\n        model = Model(inputs, gen)\n        return model","metadata":{"execution":{"iopub.status.busy":"2022-09-20T16:55:52.550596Z","iopub.execute_input":"2022-09-20T16:55:52.550877Z","iopub.status.idle":"2022-09-20T16:55:52.558374Z","shell.execute_reply.started":"2022-09-20T16:55:52.550848Z","shell.execute_reply":"2022-09-20T16:55:52.557446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#build discriminator model\ndef create_discriminator(image_size):\n    with strategy.scope():\n\n        inputs = Input(shape=(image_size, ))\n        disc = layers.Dense(512, activation=LeakyReLU(alpha=0.2))(inputs)\n        disc = layers.Dense(256, activation=LeakyReLU(alpha=0.2))(disc)\n        disc = layers.Dense(1, activation='sigmoid')(disc)\n        model = Model(inputs, disc)\n        return model","metadata":{"execution":{"iopub.status.busy":"2022-09-20T16:55:52.745652Z","iopub.execute_input":"2022-09-20T16:55:52.745923Z","iopub.status.idle":"2022-09-20T16:55:52.752185Z","shell.execute_reply.started":"2022-09-20T16:55:52.745898Z","shell.execute_reply":"2022-09-20T16:55:52.751186Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n\n    #build discriminator \n    discriminator = create_discriminator(100*100*3) \n    discriminator.compile(loss=tf.keras.losses.BinaryCrossentropy(), \n                          optimizer=tf.keras.optimizers.Adam(0.0001), \n                          metrics=['accuracy'])","metadata":{"execution":{"iopub.status.busy":"2022-09-20T16:55:52.949054Z","iopub.execute_input":"2022-09-20T16:55:52.949528Z","iopub.status.idle":"2022-09-20T16:55:53.283197Z","shell.execute_reply.started":"2022-09-20T16:55:52.949484Z","shell.execute_reply":"2022-09-20T16:55:53.282399Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#build combined model\ngenerator = create_generator(LATENT_DIM)","metadata":{"execution":{"iopub.status.busy":"2022-09-20T16:55:53.386271Z","iopub.execute_input":"2022-09-20T16:55:53.386883Z","iopub.status.idle":"2022-09-20T16:55:54.166130Z","shell.execute_reply.started":"2022-09-20T16:55:53.386848Z","shell.execute_reply":"2022-09-20T16:55:54.165421Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(' ############## GENERATOR MODEL ##############')\ngenerator.summary()\nprint('\\n', '############## DISCRIMINATOR MODEL ##############')\ndiscriminator.summary()","metadata":{"execution":{"iopub.status.busy":"2022-09-20T16:55:54.167594Z","iopub.execute_input":"2022-09-20T16:55:54.167951Z","iopub.status.idle":"2022-09-20T16:55:54.182699Z","shell.execute_reply.started":"2022-09-20T16:55:54.167924Z","shell.execute_reply":"2022-09-20T16:55:54.182117Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#visualize example and make sure its working correct\nnoise = tf.random.normal([1, LATENT_DIM])\nimage = generator(noise)\ndiscriminator.trainable = False\nprediction = discriminator(image)\nimage = (tf.reshape(image, (100,100,3)).numpy()* 127.5 + 127.5).astype(int) \nplt.imshow(image);\nplt.axis('off')\nprint(\"Prediction = \", prediction)","metadata":{"execution":{"iopub.status.busy":"2022-09-20T16:55:54.183822Z","iopub.execute_input":"2022-09-20T16:55:54.184022Z","iopub.status.idle":"2022-09-20T16:55:54.548961Z","shell.execute_reply.started":"2022-09-20T16:55:54.183999Z","shell.execute_reply":"2022-09-20T16:55:54.548121Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n\n    #compile model completely using noise image as our starting point\n    noise = Input(shape=(LATENT_DIM, ))\n    image = generator(noise)\n    prediction = discriminator(image)\n\n    GAN = Model(noise, prediction) \n\n    GAN.compile(loss=tf.keras.losses.BinaryCrossentropy(), \n                          optimizer=tf.keras.optimizers.Adam(0.0001))","metadata":{"execution":{"iopub.status.busy":"2022-09-20T16:55:54.550586Z","iopub.execute_input":"2022-09-20T16:55:54.550834Z","iopub.status.idle":"2022-09-20T16:55:54.719413Z","shell.execute_reply.started":"2022-09-20T16:55:54.550806Z","shell.execute_reply":"2022-09-20T16:55:54.718541Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#create folder to save generator images\nif not os.path.exists('GAN_images'):\n    os.makedirs('GAN_images')","metadata":{"execution":{"iopub.status.busy":"2022-09-20T16:55:54.720649Z","iopub.execute_input":"2022-09-20T16:55:54.721557Z","iopub.status.idle":"2022-09-20T16:55:54.725485Z","shell.execute_reply.started":"2022-09-20T16:55:54.721467Z","shell.execute_reply":"2022-09-20T16:55:54.724575Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#produce sample images\ndef sample_images(epoch):\n    rows = 5\n    cols = 5\n    noise = np.random.randn(rows*cols, LATENT_DIM)\n    images = generator.predict(noise)\n    images = (tf.reshape(images, (images.shape[0], 100,100,3)).numpy()* 127.5 + 127.5).astype(int) \n    fig, axs = plt.subplots(rows, cols)\n    index = 0\n    for r in range(rows):\n        for c in range(cols): \n            axs[r,c].imshow(images[index])\n            axs[r,c].axis('off')\n            index+=1\n    plt.show()\n    fig.savefig('GAN_images/%d.png' % epoch) \n    plt.close()","metadata":{"execution":{"iopub.status.busy":"2022-09-20T16:55:55.099523Z","iopub.execute_input":"2022-09-20T16:55:55.100229Z","iopub.status.idle":"2022-09-20T16:55:55.108950Z","shell.execute_reply.started":"2022-09-20T16:55:55.100181Z","shell.execute_reply":"2022-09-20T16:55:55.108317Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#create losses\ndiscriminator_loss = []\ngenerator_loss = []\n\n# training\nfor epoch in range(EPOCHS):\n    #get batches of real images\n    index = np.random.randint(0, train_images.shape[0], BATCH_SIZE) ###################################\n    real_images = train_images[index]\n    \n    #get fake images\n    noise = np.random.randn(BATCH_SIZE, LATENT_DIM)\n    fake_images = generator.predict(noise)\n    \n    #train \n    disc_loss_real, disc_acc_real = discriminator.train_on_batch(real_images, ONES)\n    disc_loss_fake, disc_acc_fake = discriminator.train_on_batch(fake_images, ZEROS)\n    disc_loss = 0.5 * (disc_loss_real + disc_loss_fake)\n    disc_acc = 0.5 * (disc_loss_real + disc_loss_fake)\n    \n    #train generator \n    noise = np.random.randn(BATCH_SIZE, LATENT_DIM)\n    gen_loss = GAN.train_on_batch(noise, ONES)\n    noise = np.random.randn(BATCH_SIZE, LATENT_DIM)\n    gen_loss = GAN.train_on_batch(noise, ZEROS)\n    \n    #save loss\n    discriminator_loss.append(disc_loss)\n    generator_loss.append(gen_loss)\n    \n    print(f\"epoch: {epoch+1}/{EPOCHS}, disc_loss: {disc_loss:.4f}, gen_loss: {gen_loss:.4f}\")\n    \n    if epoch % 10 == 0:\n        print(f'Fake Images at Epoch: {epoch+1}/{EPOCHS}')\n        sample_images(epoch)","metadata":{"execution":{"iopub.status.busy":"2022-09-20T16:55:55.327360Z","iopub.execute_input":"2022-09-20T16:55:55.328147Z","iopub.status.idle":"2022-09-20T16:57:10.484953Z","shell.execute_reply.started":"2022-09-20T16:55:55.328114Z","shell.execute_reply":"2022-09-20T16:57:10.483613Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(generator_loss, label='g_losses')\nplt.plot(discriminator_loss, label='d_losses')\nplt.legend();","metadata":{"execution":{"iopub.status.busy":"2022-09-20T16:51:17.072097Z","iopub.status.idle":"2022-09-20T16:51:17.072442Z","shell.execute_reply.started":"2022-09-20T16:51:17.072248Z","shell.execute_reply":"2022-09-20T16:51:17.072269Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Result ","metadata":{}},{"cell_type":"markdown","source":"It looks like our model learned something! Great! Since we can't submit the project anymore (past competition dates), we can't see how well our model is performing so far. However, we can see that the model started to get more browns and started to form some sort of shape-like figures. However, it is obvious these are not dogs, or anything like it. As a first GAN project, I'm satisified that the model was able to run and start to learn something. After 200 epochs and many hours of waiting for the model to train, I've decided this is as far as I'll take it for this mini-project. We can see that the model was learning and (hopefully) improving. During this project, I tried modifying many parts of the model to try and get a better outcome such as different activation functions, different momentum values, different number of Convolutional layers, and different values of dropout. In all, since these models take so long to train, I tried 3 different combinations of values and structure and this was the best result thus far. In the conclusion, we'll discuss ways to improve the model. ","metadata":{}},{"cell_type":"code","source":"#save images in zip\nz = zipfile.PyZipFile('images.zip', mode='w')\n\n#save model\nmodel_filename = 'generator_model.h5'\ntf.keras.models.save_model(generator, model_filename, overwrite=True, include_optimizer=True, save_format=None)","metadata":{"execution":{"iopub.status.busy":"2022-09-20T16:51:17.073521Z","iopub.status.idle":"2022-09-20T16:51:17.074125Z","shell.execute_reply.started":"2022-09-20T16:51:17.073936Z","shell.execute_reply":"2022-09-20T16:51:17.073958Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Conclusion\n\nOur model did learn and begin to take shape. However, it is obvious our model did not create convincing dog images. There are a few suggestions: increase the number of epochs, modify the architecture and/or try transfer learning. Something that I would have liked to try was transfer learning during the generator part so that the model doesn't have to learn from 0. In addition, I think increasing the number of epochs would have useful. As we can see in the generated images, we are just now starting to get more \"dog-like\" colors and begining to create shapes. GANs are complex models which require much patience and knowledge. This project just scrapes the surface of GANs and how much they can accomplish. \n\nFor some cool inspiration for GANs and what they can do: https://machinelearningmastery.com/impressive-applications-of-generative-adversarial-networks/","metadata":{}},{"cell_type":"markdown","source":"  \n   \n.\n\n.\n\n.\n\n.\n\n.\n\n.\n\n.\n       ","metadata":{}},{"cell_type":"markdown","source":"This notebook was inspired from the Tensorflow website (https://www.tensorflow.org/tutorials/generative/dcgan) because this was my first GAN project and I needed guidance on how to make it work, as well as https://www.kaggle.com/code/jpdurham/tensorflow-dcgan and https://www.analyticsvidhya.com/blog/2021/10/an-end-to-end-introduction-to-generative-adversarial-networksgans/.\n","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}