{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Imagine if we had access to the true data distribution $P_{data}(x)$ we could sample from that distribution in order to generate new samples, however there is no direct way to do this as typically this distribution is complex and high-dimensional. What if we could instead sample from a random noise (e.g. Normal distribution) and then learn to transform that to $P_{data}(x)$. Neural networks are a prime candidate to capture functions with high complexity and we can use to to capture this transformation. This is exactly what the do. They train the transformer network or Generator along with another network, called the Discriminator, in a game theoretic way. Going back to our image generation example:\n\nThe Generator network ($G$), tries to fool the discriminator in thinking that the generated images are real,meaning that they are taken from $P_{data}$, and\nThe Discriminator network ($D$), tries to differentiate between real ($x\\sim P_{data}$) and fake images.\n\nRandom noise is fed into the Generator that transforms it into a \"fake image\". The Discriminator is fed both from the training set images ($p_{data}(x)$) and the fake images coming from the Generator and it has to tell them apart. The idea behind GAN, is to train both of these networks alternatively to do the best they can in generating and discriminating images. The intuition is that by improving one of these networks, in this game theoretic manner, the other network has to do a better job to win the game, and that in turn improves its performance and this loop continues.","metadata":{}},{"cell_type":"markdown","source":"<img src=\"https://4.bp.blogspot.com/-YTGLQjhch-Q/Wz475NU2TSI/AAAAAAAAsu4/zaC_wfZBX80dePLflgQUaAaxE72od3VCgCEwYBhgL/s1600/Figura_2.png\" height=\"500\" width=\"500\">","metadata":{}},{"cell_type":"markdown","source":"<pre><b>Credits to Chris Deotte's Kernel</b>\nhttps://www.kaggle.com/cdeotte/dog-memorizer-gan</pre>\n\n<pre><b>Credits to Nanashi's Ideas and Initiatives</b>\nhttps://www.kaggle.com/jesucristo/memorizer-cgan-for-dummies</pre>","metadata":{}},{"cell_type":"markdown","source":"# Import Libraries","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd \nimport keras\nfrom keras.layers import Input, Dense, Reshape, Flatten, Dropout,Concatenate\nfrom keras.backend import random_normal,ones_like,zeros_like,mean\nfrom keras.backend import get_session\nfrom keras.layers import BatchNormalization, Activation, ZeroPadding2D\nfrom keras.layers.advanced_activations import LeakyReLU\nfrom keras.layers.convolutional import UpSampling2D, Conv2D\nfrom keras.models import Sequential, Model\nfrom keras.optimizers import Adam\nfrom keras.layers import concatenate\nfrom keras.initializers import TruncatedNormal\nfrom keras.callbacks import LearningRateScheduler, EarlyStopping, History\nfrom PIL import Image\nimport warnings\nimport os\nimport time\nfrom glob import glob\nimport datetime\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport numpy as np, pandas as pd, os\nimport xml.etree.ElementTree as ET \nimport matplotlib.pyplot as plt, zipfile \nfrom PIL import Image \nfrom glob import glob","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5"},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(os.listdir(\"../input/\"))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load Images","metadata":{}},{"cell_type":"code","source":"ComputeLB = False\nDogsOnly = False\n\nimport numpy as np, pandas as pd, os\nimport xml.etree.ElementTree as ET \nimport matplotlib.pyplot as plt, zipfile \nfrom PIL import Image \n\nROOT = '../input/generative-dog-images/'\nif not ComputeLB: ROOT = '../input/'\nIMAGES = os.listdir(ROOT + 'all-dogs/all-dogs/')\nbreeds = os.listdir(ROOT + 'annotation/Annotation/') \n\nidxIn = 0; namesIn = []\nimagesIn = np.zeros((25000,64,64,3))\n\n# CROP WITH BOUNDING BOXES TO GET DOGS ONLY\n# https://www.kaggle.com/paulorzp/show-annotations-and-breeds\nif DogsOnly:\n    for breed in breeds:\n        for dog in os.listdir(ROOT+'annotation/Annotation/'+breed):\n            try: img = Image.open(ROOT+'all-dogs/all-dogs/'+dog+'.jpg') \n            except: continue           \n            tree = ET.parse(ROOT+'annotation/Annotation/'+breed+'/'+dog)\n            root = tree.getroot()\n            objects = root.findall('object')\n            for o in objects:\n                bndbox = o.find('bndbox') \n                xmin = int(bndbox.find('xmin').text)\n                ymin = int(bndbox.find('ymin').text)\n                xmax = int(bndbox.find('xmax').text)\n                ymax = int(bndbox.find('ymax').text)\n                w = np.min((xmax - xmin, ymax - ymin))\n                img2 = img.crop((xmin, ymin, xmin+w, ymin+w))\n                img2 = img2.resize((64,64), Image.ANTIALIAS)\n                imagesIn[idxIn,:,:,:] = np.asarray(img2)\n                #if idxIn%1000==0: print(idxIn)\n                namesIn.append(breed)\n                idxIn += 1\n    idx = np.arange(idxIn)\n    np.random.shuffle(idx)\n    imagesIn = imagesIn[idx,:,:,:]\n    namesIn = np.array(namesIn)[idx]\n    \n# RANDOMLY CROP FULL IMAGES\nelse:\n    IMAGES = np.sort(IMAGES)\n    np.random.seed(810)\n    x = np.random.choice(np.arange(20579),10000)\n    np.random.seed(None)\n    for k in range(len(x)):\n        img = Image.open(ROOT + 'all-dogs/all-dogs/' + IMAGES[x[k]])\n        w = img.size[0]; h = img.size[1];\n        if (k%2==0)|(k%3==0):\n            w2 = 100; h2 = int(h/(w/100))\n            a = 18; b = 0          \n        else:\n            a=0; b=0\n            if w<h:\n                w2 = 64; h2 = int((64/w)*h)\n                b = (h2-64)//2\n            else:\n                h2 = 64; w2 = int((64/h)*w)\n                a = (w2-64)//2\n        img = img.resize((w2,h2), Image.ANTIALIAS)\n        img = img.crop((0+a, 0+b, 64+a, 64+b))    \n        imagesIn[idxIn,:,:,:] = np.asarray(img)\n        namesIn.append(IMAGES[x[k]])\n        #if idxIn%1000==0: print(idxIn)\n        idxIn += 1\n    \n# DISPLAY CROPPED IMAGES\nx = np.random.randint(0,idxIn,25)\nfor k in range(5):\n    plt.figure(figsize=(15,3))\n    for j in range(5):\n        plt.subplot(1,5,j+1)\n        img = Image.fromarray( imagesIn[x[k*5+j],:,:,:].astype('uint8') )\n        plt.axis('off')\n        if not DogsOnly: plt.title(namesIn[x[k*5+j]],fontsize=11)\n        else: plt.title(namesIn[x[k*5+j]].split('-')[1],fontsize=11)\n        plt.imshow(img)\n    plt.show()","metadata":{"_kg_hide-output":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Variables used","metadata":{}},{"cell_type":"code","source":"IMG_SIZE = Input((12288,))\nIMG_SIZE_2 = Input((10000,))\nNOISE_SIZE = 10000\n#BATCH_SIZE = 256 # orig gives ~7.24\n#BATCH_SIZE = 512 # gives ~7.25\n#BATCH_SIZE = 128 # gives 7.22594\n#BATCH_SIZE = 128 # gives ~7.222\nBATCH_SIZE = 64","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Build Discriminator","metadata":{}},{"cell_type":"code","source":"def discriminatorFunction():\n    \n    input_layer = Dense(12288, activation='sigmoid')(IMG_SIZE_2) \n    input_layer = Reshape((2,12288,1))(concatenate([IMG_SIZE,input_layer]))\n    discriminator = Conv2D(filters = 1, kernel_size=[2,1],use_bias=False, name = 'layer_1')(input_layer)\n    out = Flatten()(discriminator)\n    return out","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Discriminator\")\nmodel = discriminatorFunction()\nmodel_discriminator = Model([IMG_SIZE,IMG_SIZE_2], model)\nmodel_discriminator.get_layer('layer_1').trainable = False\nmodel_discriminator.get_layer('layer_1').set_weights([np.array([[[[-1.0 ]]],[[[1.0]]]])])\nmodel_discriminator.summary()\nmodel_discriminator.compile(optimizer='adam', loss='binary_crossentropy')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Build Generator","metadata":{}},{"cell_type":"code","source":"def GeneratorFunction(noise_shape=(NOISE_SIZE,)):\n    input_layer = Input(noise_shape)\n    generated = Dense(12288, activation='linear')(input_layer)\n        \n# COMPILE\n    model = Model(inputs=input_layer,outputs = [generated,Reshape((10000,))(input_layer)])\n    model.summary()\n  \n    return model","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Generator\")\nmodel_generator = GeneratorFunction(noise_shape=(NOISE_SIZE,))","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train the Discriminator","metadata":{}},{"cell_type":"code","source":"# TRAINING DATA\ntrain_y = (imagesIn[:10000,:,:,:]/255.).reshape((-1,12288))\ntrain_X = np.zeros((10000,10000))\nfor i in range(10000): train_X[i,i] = 1\nzeros = np.zeros((10000,12288))\n\n# ---------------------\n#  Train Discriminator\n# ---------------------\n\n# orig\nlr = 0.5\n# Let's play with lr\n#lr = 0.3 # gives ~7.251\nfor k in range(5):\n    LR_Scheduler = LearningRateScheduler(lambda x: lr)\n    h = model_discriminator.fit([zeros,train_X], train_y, epochs = 100,batch_size = BATCH_SIZE, callbacks=[LR_Scheduler], verbose=0)\n    print('Epoch',(k+1)*10,'/50 - loss =',h.history['loss'][-1] )\n    if h.history['loss'][-1]<0.533: lr = 0.1","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"del train_X, train_y, imagesIn","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Discriminator Recalls from Memory Dogs')    \nfor k in range(5):\n    plt.figure(figsize=(15,3))\n    for j in range(5):\n        xx = np.zeros((10000))\n        xx[np.random.randint(10000)] = 1\n        plt.subplot(1,5,j+1)\n        img = model_discriminator.predict([zeros[0,:].reshape((-1,12288)),xx.reshape((-1,10000))]).reshape((-1,64,64,3))\n        img = Image.fromarray( (255*img).astype('uint8').reshape((64,64,3)))\n        plt.axis('off')\n        plt.imshow(img)\n    plt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Build the GAN","metadata":{}},{"cell_type":"code","source":"# BUILD GENERATIVE ADVERSARIAL NETWORK\nmodel_discriminator.trainable = False #discriminator is not trainable for GANs\nz = Input(shape=(NOISE_SIZE,))\nimg = model_generator(z)\nreal = model_discriminator(img)\n\n# COMPILE GAN\ngan = Model(z, real)\ngan.get_layer('model_1').get_layer('layer_1').set_weights([np.array([[[[-1 ]]],[[[255.]]]])])\ngan.compile(optimizer=Adam(5), loss='mean_squared_error')\n\n# DISPLAY ARCHITECTURE\nprint(\"Model created based on Discriminator and Generator\")\ngan.summary()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train the GAN","metadata":{}},{"cell_type":"code","source":"train = np.zeros((10000,10000))\nfor i in range(10000): train[i,i] = 1\nzeros = np.zeros((10000,12288))\n\nSteps_per_epoch = 50\n\nprint(\"Training begins... Total steps per epoch: {}\".format(Steps_per_epoch ))\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"warnings.filterwarnings(\"ignore\")\n\nlr = 5.\n\nfor step in range(Steps_per_epoch):\n        \n    # ---------------------\n    #  Train GAN\n    # ---------------------\n    LR_Scheduler = LearningRateScheduler(lambda x: lr)\n    h = gan.fit(train, zeros, epochs = 1, batch_size=256, callbacks=[LR_Scheduler], verbose=0)\n\n    # DISPLAY GENERATOR LEARNING PROGRESS \n    if (step<10)|(step%5==4):\n        \n        print (\"Step: {}/{} [G loss: {:.4f}]\".format(\n                     (step+1)*10, Steps_per_epoch*10, h.history['loss'][-1]))\n        \n    if h.history['loss'][-1] < 25: lr = 1.\n    #if h.history['loss'][-1] < 1.5: lr = 0.5 # orig\n    #if h.history['loss'][-1] < 1.5: lr = 0.05 # test gaves ~7.23\n    if h.history['loss'][-1] < 1.5: lr = 0.01 # test gaves ~7.23 \n        \n    # DISPLAY GENERATOR LEARNING PROGRESS\n    \n    if step<10: \n    \n        # Plot images\n        plt.figure(figsize=(15,3))\n        for j in range(5):\n            zz = np.zeros((10000))\n            zz[np.random.randint(10000)] = 1\n            plt.subplot(1,5,j+1)\n            img = model_generator.predict(zz.reshape((-1,10000)))[0].reshape((-1,64,64,3))\n            img = Image.fromarray( (img).astype('uint8').reshape((64,64,3)))\n            plt.axis('off')\n            plt.imshow(img)\n        plt.show()   ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submit to Kaggle","metadata":{}},{"cell_type":"code","source":"class DogGenerator:\n    index = 0   \n    def getDog(self,seed):\n        xx = np.zeros((10000))\n        xx[self.index] = 0.999\n        xx[np.random.randint(10000)] = 0.001\n        img = model_generator.predict(xx.reshape((-1,10000)))[0].reshape((64,64,3))\n        self.index = (self.index+1)%10000\n        return Image.fromarray( img.astype('uint8') ) ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# SAVE TO ZIP FILE NAMED IMAGES.ZIP\nz = zipfile.PyZipFile('images.zip', mode='w')\nd = DogGenerator()\nfor k in range(10000):\n    img = d.getDog(np.random.normal(0,1,100))\n    f = str(k)+'.png'\n    img.save(f,'PNG'); z.write(f); os.remove(f)\n    #if k % 1000==0: print(k)\nz.close()","metadata":{},"execution_count":null,"outputs":[]}]}