{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"!pip install tabgan","metadata":{"execution":{"iopub.status.busy":"2021-09-17T14:18:03.353162Z","iopub.execute_input":"2021-09-17T14:18:03.353565Z","iopub.status.idle":"2021-09-17T14:18:13.356271Z","shell.execute_reply.started":"2021-09-17T14:18:03.353482Z","shell.execute_reply":"2021-09-17T14:18:13.355037Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from tabgan.sampler import OriginalGenerator, GANGenerator\nimport pandas as pd\nimport numpy as np\n\ntrain = pd.read_csv(\"../input/tabular-playground-series-sep-2021/train.csv\").dropna()\ntest = pd.read_csv(\"../input/tabular-playground-series-sep-2021/test.csv\").dropna()","metadata":{"execution":{"iopub.status.busy":"2021-09-17T14:23:10.283573Z","iopub.execute_input":"2021-09-17T14:23:10.284107Z","iopub.status.idle":"2021-09-17T14:24:03.386866Z","shell.execute_reply.started":"2021-09-17T14:23:10.284071Z","shell.execute_reply":"2021-09-17T14:24:03.385571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.isnull().sum().sum()\ntest.isnull().sum().sum()","metadata":{"execution":{"iopub.status.busy":"2021-09-17T14:28:14.637863Z","iopub.execute_input":"2021-09-17T14:28:14.638891Z","iopub.status.idle":"2021-09-17T14:28:14.783991Z","shell.execute_reply.started":"2021-09-17T14:28:14.638832Z","shell.execute_reply":"2021-09-17T14:28:14.783094Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n\n# random input data\n\ntarget = pd.DataFrame(train['claim'])\ntrain=train.drop(['claim','id'],axis=1)\n\ntest=test.drop('id',axis=1)\n\n\n\n\n# generate data\n#new_train1, new_target1 = OriginalGenerator().generate_data_pipe(train, target, test, )\n\n\nnew_train3, new_target3 = GANGenerator(gen_x_times=1.1, cat_cols=None, bot_filter_quantile=0.001,\n                                       top_filter_quantile=0.999,\n                                       is_post_process=True,\n                                       adversaial_model_params={\n                                           \"metrics\": \"AUC\", \"max_depth\": 2,\n                                           \"max_bin\": 100, \"n_estimators\": 500,\n                                          \"learning_rate\": 0.02, \"random_state\": 42,\n                                      }, pregeneration_frac=2, only_generated_data=False,\n                                      epochs=500).generate_data_pipe(train, target,\n                                                                    test, deep_copy=True,\n                                                                    only_adversarial=False,\n                                                                     use_adversarial=True)","metadata":{"execution":{"iopub.status.busy":"2021-09-17T16:08:34.456311Z","iopub.execute_input":"2021-09-17T16:08:34.456694Z","iopub.status.idle":"2021-09-17T16:08:34.478931Z","shell.execute_reply.started":"2021-09-17T16:08:34.45666Z","shell.execute_reply":"2021-09-17T16:08:34.476825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"new_train3.to_csv('train.csv',index=False) \nnew_target3.to_csv('targetcsv',index=False) ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# I’m Something of a Painter Myself\n\n**Warm UP**\n\nsuppose that a smuggler wants to make fake currency, but dont know how the real currency look. there is a police whos job is to accuratly distinguise fake or real currency.\n\nnow the task of smuggler is to generate fake currency as good as real ones.\nand the task of police is to correctly identify fake currency.\n\nGAN's story is almost similar. a genral GAN composed of two counterparts \n1. Generator whose job is to create fake data i.e (images).\n2. Discriminator whose job is to differenciate fake and real data\n\nIn summary for this notebook , the game follows with:\nThe generator trying to maximize the probability of making the discriminator mistakes its inputs as real.\nAnd the discriminator guiding the generator to produce more realistic images.\n\nIn the perfect equilibrium, the generator would capture the general training data distribution. As a result, the discriminator would be always unsure of whether its inputs are real or not.\n\n# Below is the very basic architecture of GAN's","metadata":{}},{"cell_type":"markdown","source":"![](http://cdn-media-1.freecodecamp.org/images/m41LtQVUf3uk5IOYlHLpPazxI3pWDwG8VEvU)","metadata":{}},{"cell_type":"markdown","source":"# AIM OF THIS NOTEBOOK IS TO DEVELOP BASIC UNDERSTANDING OF GANS BEFORE DIVING INTO COMPETATION\n\nWe will be using MNIST dataset, the mnist dataset have image data of numerical digits 0-9\nso our gan will be generating fake images of digits. \n\n**DCGAN on MNIST**\n**DCGAN is a Generative Adversarial Network (GAN) using CNN.**\n\n* The discriminator learns to discriminate real from fake images.\n* The generator tries to fool the discriminator by generating fake images.\n* The generator + discriminator form an adversarial network.\n* DCGAN trains the discriminator and adversarial networks alternately.","metadata":{}},{"cell_type":"markdown","source":"# PRE-REQUISITES\n\n1. basic inderstanding of neural networks.\n2. terminologies like optimaization,upsampling, downsampling etc.\n3. little bit of coding related to tensorflow","metadata":{}},{"cell_type":"markdown","source":"# IMPORT LIBRARIES","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nimport numpy as np\nimport math\nimport matplotlib.pyplot as plt\nimport os\nfrom tensorflow import keras\nfrom tensorflow.keras import layers","metadata":{"execution":{"iopub.status.busy":"2021-09-08T09:33:25.708175Z","iopub.execute_input":"2021-09-08T09:33:25.708565Z","iopub.status.idle":"2021-09-08T09:33:29.909418Z","shell.execute_reply.started":"2021-09-08T09:33:25.70846Z","shell.execute_reply":"2021-09-08T09:33:29.90861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Buiding a Generator Model¶\n**Build a model using layers of BatchNorm-ReLU-Conv2DTranpose to generate fake images**\n\n![](https://cdn-media-1.freecodecamp.org/images/cKZw7BBTj3gfDCc8xiSKzf6PyHZkhHEVrFy3)\n\n**The Generator network implemented here**\n\nThe network has 4 convolutional layers, all followed by BN (except for the output layer) and Rectified Linear unit (ReLU) activations.\n\nIt takes as an input a random vector z (drawn from a normal distribution). After reshaping z to have a 4D shape, we feed it to the generator that starts a series of upsampling layers.\n\nEach upsampling layer represents a transpose convolution operation with strides 2. Transpose convolutions are similar to the regular convolutions.\n\nThis final output shape is defined by the size of the training images. In this case for MNIST, it would generate a 28x28 greyscale image.\n\n**WHAT IS UP SAMPLING**\nIn the Upsampling network, the abstract image representations are upsampled using various techniques to make their spatial dimensions equal to the input image.\n","metadata":{}},{"cell_type":"code","source":"def build_generator(image_size=28, input_size=100):\n    \n    #Build an input layer\n    gen_input = keras.Input(shape=(input_size,))\n    \n    #Increase dimensions and resize to 3D to feed it to Conv2DTranspose layer\n    x = layers.Dense(7 * 7 * 128)(gen_input)\n    x = layers.Reshape((7, 7, 128))(x)\n    \n    #Use ConvTranspose\n    x = layers.BatchNormalization()(x)\n    x = layers.Activation('relu')(x)\n    x = layers.Conv2DTranspose(128, kernel_size=[5,5], strides=2, padding='same')(x)\n    \n    x = layers.BatchNormalization()(x)\n    x = layers.Activation('relu')(x)\n    x = layers.Conv2DTranspose(64, kernel_size=[5,5], strides=2, padding='same')(x)\n    \n    x = layers.BatchNormalization()(x)\n    x = layers.Activation('relu')(x)\n    x = layers.Conv2DTranspose(32, kernel_size=[5,5], strides=1, padding='same')(x)\n    \n    x = layers.BatchNormalization()(x)\n    x = layers.Activation('relu')(x)\n    x = layers.Conv2DTranspose(1, kernel_size=[5,5], strides=1, padding='same')(x)\n    \n    #Output layer for Generator\n    x = layers.Activation('sigmoid')(x)\n    \n    #Build model using Model API\n    generator = keras.Model(gen_input, x, name='generator')\n    \n    return generator","metadata":{"execution":{"iopub.status.busy":"2021-09-08T09:33:29.910829Z","iopub.execute_input":"2021-09-08T09:33:29.911132Z","iopub.status.idle":"2021-09-08T09:33:29.924416Z","shell.execute_reply.started":"2021-09-08T09:33:29.911098Z","shell.execute_reply":"2021-09-08T09:33:29.920675Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Discriminator\n\nplease note generator and dicriminator network seems to be same. the difference is that in generator we were using convolution2DTranspose while in discrimator Convolution2D.\n\nfor generator we need to increase size from input(100) to target image (28,28)\nfor discriminator we need to decrease size from 28,28 to 1\n\nThe discriminator is also a 4 layer CNN with BN (except its input layer) and leaky ReLU activations. Many activation functions will work fine with this basic GAN architecture.\nHowever, leaky ReLUs are very popular because they help the gradients flow easier through the architecture.\n\nA regular ReLU function works by truncating negative values to 0. This has the effect of blocking the gradients to flow through the network. Instead of the function being zero, leaky ReLUs allow a small negative value to pass through. That is, the function computes the greatest value between the features and a small factor.\n\n![](https://i0.wp.com/androidkt.com/wp-content/uploads/2020/05/Selection_019.png?resize=760%2C364)\n\nFinally, the discriminator needs to output probabilities. For that, we use the Logistic Sigmoid activation function on the final logits.","metadata":{}},{"cell_type":"code","source":"def build_discriminator(data_shape=[28,28,1,]):\n    \n    #Build the network\n    dis_input = keras.Input(data_shape)\n    x = layers.LeakyReLU(alpha=0.2)(dis_input)\n    x = layers.Conv2D(32, kernel_size=[5,5], strides=2, padding='same')(x)\n    \n    x = layers.LeakyReLU(alpha=0.2)(x)\n    x = layers.Conv2D(64, kernel_size=[5,5], strides=2, padding='same')(x)\n    \n    x = layers.LeakyReLU(alpha=0.2)(x)\n    x = tf.keras.layers.Conv2D(128, kernel_size=[5,5], strides=2, padding='same')(x)\n    \n    x = layers.LeakyReLU(alpha=0.2)(x)\n    x = tf.keras.layers.Conv2D(256, kernel_size=[5,5], strides=1, padding='same')(x)\n    \n    #Flatten the output and build an output layer\n    x = layers.Flatten()(x)\n    x = layers.Dense(1, activation='sigmoid')(x)\n    \n    #Build Model\n    discriminator = keras.Model(dis_input, x, name='discriminator')\n    \n    return discriminator","metadata":{"execution":{"iopub.status.busy":"2021-09-08T09:33:29.926897Z","iopub.execute_input":"2021-09-08T09:33:29.927367Z","iopub.status.idle":"2021-09-08T09:33:29.93654Z","shell.execute_reply.started":"2021-09-08T09:33:29.927314Z","shell.execute_reply":"2021-09-08T09:33:29.935563Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Build Adversarial model\n\ntill now we have only created skeleton for our GAN. as of  we need to add compiling part, optimizer, forward pass we need parameters like noise learning rate decay.\n\n**Adversarial = generator + discriminator**\n\nwe will be calling build_generator and build_discriminator functions, i am using Adam optimizer which is popular one, experement with using different optimizers\n[here is a link to know more abut optimizers](https://machinelearningknowledge.ai/keras-optimizers-explained-with-examples-for-beginners/#:~:text=Keras%20Adam%20Optimizer%20%28Adaptive%20Moment%20Estimation%29%20The%20adam,of%20data%20and%20parameters%20are%20available%20for%20usage.)\n\n**IMPORTANT**\ni have not compiled generator, while i have compiled discriminator why because generator will be using some learning of dicriminator. so we will be using frozen disciminator to train the whole adversarial network.\n\n**-->so the ultimate equation be like**\nadversarial network = generator + frozen discriminator.\n\n","metadata":{}},{"cell_type":"code","source":"def build_models():\n    \n    noise_size = 100\n    lr = 2e-4\n    decay = 6e-8\n    \n    #Build Base Discriminator model\n    base_discriminator = build_discriminator(data_shape=(28,28,1,))\n    \n    #Define optimizer and compile model\n    discriminator = keras.Model(inputs=base_discriminator.inputs, \n                                          outputs=base_discriminator.outputs)\n    optimizer = keras.optimizers.RMSprop(lr=lr, decay=decay)\n    discriminator.compile(loss='binary_crossentropy',\n                          optimizer=optimizer,\n                          metrics=['accuracy'])\n    \n    #Build Generator model\n    generator = build_generator(image_size=28, input_size=noise_size)\n    \n    #Build Frozen Discriminator\n    frozen_discriminator = keras.Model(inputs=base_discriminator.inputs, \n                                          outputs=base_discriminator.outputs)\n    #Freeze the weights of discriminator during adversarial training\n    frozen_discriminator.trainable = False\n\n    #Build Adversarial model\n    optimizer = keras.optimizers.RMSprop(lr=lr * 0.6, decay=decay * 0.5)\n    #Adversarial = generator + discriminator\n    adversarial = keras.Model(generator.input, \n                        frozen_discriminator(generator.output))\n    \n    adversarial.compile(loss='binary_crossentropy',\n                        optimizer=optimizer,\n                        metrics=['accuracy'])    \n    \n    return generator, discriminator, adversarial","metadata":{"execution":{"iopub.status.busy":"2021-09-08T09:33:29.938467Z","iopub.execute_input":"2021-09-08T09:33:29.938987Z","iopub.status.idle":"2021-09-08T09:33:29.949934Z","shell.execute_reply.started":"2021-09-08T09:33:29.93895Z","shell.execute_reply":"2021-09-08T09:33:29.949104Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# NOW COMES THE TRAINING PART\n\n**JUST IN CASE** \nthis notebook is not about classification, its about generating fake images from existing ones.\n\nright now load the data and then start training, carefully if you see we have only loaded train part from dataset not the test part and labels. right because we are only interested in creating new images.\n\nfirst we will be training disciminator followed by training the Adversial network(training generator)  next at evrey iteration we see how well our generator is doing. specifically after 500 iteration we will see the output from generator.\n\nso before begining the training check the architecture of Generator, discriminator and adversarial network and try to analyse how inputs are conveyed in generator network.\n\ndiscriminator and adversarial network are general purpose neural network.","metadata":{}},{"cell_type":"code","source":"def train_gan(generator, discriminator, adversarial, noise_size=100):\n    \n    #Training parameters\n    batch_size = 64\n    train_steps = 10000\n    image_size = 28\n    \n    # load MNIST dataset\n    (train_x, _), (_, _) = tf.keras.datasets.mnist.load_data()\n    #Make it 3D dataset\n    train_x = np.reshape(train_x, [-1, image_size, image_size, 1])\n    #Standardize data : 0 to 1\n    train_x = train_x.astype('float32') / 255\n    \n    #Input for testing generator at different intervals, we will generate 16 images\n    test_noise_input = np.random.uniform(-1.0,1.0, size=[16, noise_size])\n    \n    #Start training\n    for i in range(train_steps):\n        \n        #Train DISCRIMATOR\n        \n        #1. Get fake images from Generator\n        noise_input = np.random.uniform(-1.0,1.0, size=[batch_size, noise_size])\n        fake_images = generator.predict(noise_input)\n        \n        #2. Get real images from training set\n        img_indexes = np.random.randint(0, train_x.shape[0], size=batch_size)\n        real_images = train_x[img_indexes]\n        \n        #3. Prepare input for training Discriminator\n        X = np.concatenate((real_images, fake_images))\n        \n        #4. Labels for training\n        y_real = np.ones((batch_size, 1))\n        y_fake = np.zeros((batch_size, 1))\n        y = np.concatenate((y_real, y_fake))\n        \n        #5. Train Discriminator\n        d_loss, d_acc = discriminator.train_on_batch(X, y)\n        \n        \n        #Train ADVERSARIAL Network\n        \n        #1. Prepare input - create a new batch of noise\n        X = noise_input = np.random.uniform(-1.0,1.0, size=[batch_size, noise_size])\n        \n        #2. Prepare labels - training Adversarial network to lie :) - All 1s\n        y = np.ones((batch_size, 1))\n        \n        #3. Train - Pls note Discrimator is not getting trained here\n        a_loss, a_acc = adversarial.train_on_batch(X, y)\n        \n        if i % 100 == 0:\n            #Print loss and Accuracy for both networks\n            print(\"%s [Discriminator loss: %f, acc: %f, Adversarial loss: %f, acc: %f]\" % (i, d_loss, d_acc, a_loss, a_acc) )\n        \n        #Save generated images to see how well Generator is doing\n        if (i+1) % 500 == 0:\n            \n            #Generate 16 images\n            fake_images = generator.predict(test_noise_input)\n            \n            #Display images\n            plot_images(fake_images, i+1)\n            \n    #Save Generator model\n    generator.save('mnist_generator_dcgan.h5')    ","metadata":{"execution":{"iopub.status.busy":"2021-09-08T09:33:29.951202Z","iopub.execute_input":"2021-09-08T09:33:29.951618Z","iopub.status.idle":"2021-09-08T09:33:29.964903Z","shell.execute_reply.started":"2021-09-08T09:33:29.951577Z","shell.execute_reply":"2021-09-08T09:33:29.963996Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# This is a utility function to draw images","metadata":{}},{"cell_type":"code","source":"def plot_images(fake_images, step):\n    \n    plt.figure(figsize=(2.5,2.5))\n    num_images = fake_images.shape[0]\n    \n    image_size = fake_images.shape[1]\n    rows = int(math.sqrt(fake_images.shape[0]))\n    \n    for i in range(num_images):\n        plt.subplot(rows, rows, i + 1)\n        image = np.reshape(fake_images[i], [image_size, image_size])\n        plt.imshow(image, cmap='gray')\n        plt.axis('off')\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2021-09-08T09:33:29.966185Z","iopub.execute_input":"2021-09-08T09:33:29.9666Z","iopub.status.idle":"2021-09-08T09:33:29.97569Z","shell.execute_reply.started":"2021-09-08T09:33:29.966563Z","shell.execute_reply":"2021-09-08T09:33:29.974861Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# MODEL SUMMARY","metadata":{}},{"cell_type":"code","source":"G, D, A = build_models()\nG.summary()\nD.summary()\nA.summary()\n","metadata":{"execution":{"iopub.status.busy":"2021-09-08T09:33:29.977199Z","iopub.execute_input":"2021-09-08T09:33:29.97749Z","iopub.status.idle":"2021-09-08T09:33:31.983177Z","shell.execute_reply.started":"2021-09-08T09:33:29.977447Z","shell.execute_reply":"2021-09-08T09:33:31.982242Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Lets begin the training\n\n**-first iteration-**\ninitially the generator have no idea, after 500 epochs the generated images doesnt seems like digits.\n\n**-second iteration- Onward** \nthe generator have developed little bit understanding of data, geerated images have so similarity to minst dataset\n\nafter every cycle it can be visually confirmed that quality of generated images are increasing.\n\n***THANKS FOR PATIENTLY READING TILL END***","metadata":{}},{"cell_type":"code","source":"train_gan(G, D, A)","metadata":{"execution":{"iopub.status.busy":"2021-09-08T09:33:31.986059Z","iopub.execute_input":"2021-09-08T09:33:31.986355Z","iopub.status.idle":"2021-09-08T09:44:05.034038Z","shell.execute_reply.started":"2021-09-08T09:33:31.986325Z","shell.execute_reply":"2021-09-08T09:44:05.03287Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"execution":{"iopub.status.busy":"2021-09-17T04:08:56.303767Z","iopub.execute_input":"2021-09-17T04:08:56.30406Z","iopub.status.idle":"2021-09-17T04:12:14.490544Z","shell.execute_reply.started":"2021-09-17T04:08:56.30403Z","shell.execute_reply":"2021-09-17T04:12:14.489282Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}