{"metadata":{"kernelspec":{"display_name":"Python 3.9 (tensorflow)","language":"python","name":"tensorflow"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.9.15"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 👨‍🎨 Paint With DCGAN","metadata":{}},{"cell_type":"markdown","source":"## Use DCGAN to Monet-esque Paintings","metadata":{}},{"cell_type":"markdown","source":"<div align=\"center\">\n    <img src=\"https://raw.githubusercontent.com/justinsiowqi/-Paint-With-DCGAN/main/Monet%20Haystacks.gif\" alt=\"Monet Haystacks\" style=\"width: 500px;\"> \n</div>\n<div align=\"center\">\n  Monet Haystacks (source: WiffleGif)\n</div>","metadata":{}},{"cell_type":"markdown","source":"Ever wanted to leverage the power of code to bring paintings to life? \n\nIntroducing the **Generative Adversarial Network** (GAN). It's a powerful neural network that creates new, realistic images just from learning from some examples. Specifically, we'll focus on creating a **Deep Convolutional GAN** (DCGAN) to create monet-esque paintings from scratch. \n\nGet your paint brushes ready as we learn from the legendary Claude Monet...","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"### <font color='000000'>Table of contents<font><a class='anchor' id='top'></a>\n\n1. [Introduction](#section-one)  \n    \n2. [Get Data](#section-two)\n    \n3. [Build Model](#section-three)  \n 3.1. [Build the Generator](#section-three-one)  \n 3.2. [Build the Discriminator](#section-three-two)  \n 3.3. [Defne Loss Functions & Optimizer](#section-three-three)  \n    \n4. [Train Model](#section-four)  \n 4.1. [Custom Training Loop](#section-four-one)  \n 4.2. [Sample & Display Images](#section-four-two)  \n\n5. [Submit Images](#section-five)\n    \n6. [Conclusion](#section-six)","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-one\"></a>\n## 1. Introduction\n\nWe will begin by loading, parsing and displaying the images from the Monet TFRecord file. Note that we won't be using the photo TFRecord file since the DCGAN creates images from scratch.\n\nThe next thing we need to do is build the DCGAN model. This involves building two separate neural networks - the generator and the discriminator. We'll go into detail on how these two networks work below.\n\nFinally, we'll create a custom training loop for the model and then we can view our monet-esque art piece!","metadata":{}},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-two\"></a>\n## 2. Get Data\n\n* Load and extract images from monet TFRecord files.\n* Display 16 images from the monet dataset in a 4x4 grid.","metadata":{}},{"cell_type":"markdown","source":"### Understanding TFRecords:\n\nThis section deals with TFRecord files. If you prefer to work with jpeg files, head to the data tab and download the monet_jpg and photo_jpeg files instead.\n\nYou may be wondering: what are TFRecord files? They're simply a more efficient way of **storing** and **transferring** data. Each TFRecord file contains multiple data samples (called examples). Each example stores values for certain features.\n\nLet's look at the code below. Since the TFRecord files are stored in a folder, we need to use the **glob module** to access each individual file. We only want files that end with .tfrec. \n\nNext, TFRecordDataset is used to load these files as bytes. A dictionary is also created to **describe the features**, like image_name and image. After that, we'll create a function that will **parse a single example** from a TFRecord file and extract the data contained inside it. We apply this function to the dataset and shuffle + batch.\n\nFinally, let's create a function to display the images. We convert the monet dataset into an iterable object and retrieve the first item. Then, we use a loop to plot 16 images in a 4x4 grid.","metadata":{}},{"cell_type":"code","source":"# Import libraries\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport glob\n\nimport tensorflow as tf\nfrom keras.models import Sequential\nfrom keras.layers import Conv2D, Conv2DTranspose, Dense, Flatten, Reshape\nfrom keras.layers import BatchNormalization, Dropout\nfrom keras.layers import ReLU, LeakyReLU, Activation\nfrom tensorflow.keras.optimizers import Adam","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Set the buffer size and batch size\n\nBUFFER_SIZE = 60000\nBATCH_SIZE = 32\nIMAGE_SIZE = [256, 256]","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Path to all the TFRecord files inside the Monet folders\n\nmonet_filepath = glob.glob(f'monet_tfrec/*.tfrec')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load TFRecord file from the folder as bytes\n\nraw_monet_dataset = tf.data.TFRecordDataset(monet_filepath)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create a dictionary describing the features\n\nfeature_description = {\n    'image_name': tf.io.FixedLenFeature([], tf.string),\n    'image': tf.io.FixedLenFeature([], tf.string),\n    'target': tf.io.FixedLenFeature([], tf.string)\n}","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create a function to extract images from the TFRecord file\n\ndef _parse_image_function(example_proto):\n    example = tf.io.parse_single_example(example_proto, feature_description)\n    image = tf.io.decode_jpeg(example['image'])\n    image = (tf.cast(image, tf.float32) / 127.5) - 1.0\n    image = tf.image.resize(image, IMAGE_SIZE)\n    return image","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Extract images, shuffle and select a batch\n\nmonet_dataset = raw_monet_dataset.map(_parse_image_function)\nmonet_dataset = monet_dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create a function to display the 16 images from the dataset\n\nmonet_example = next(iter(monet_dataset))\n\ndef display_images(example):\n    plt.figure(figsize = (7, 7))\n    for n in range(16):\n        ax = plt.subplot(4, 4, n + 1)\n        plt.imshow(example[n] * 0.5 + 0.5)\n        plt.axis(\"off\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Display the first 16 images from the monet dataset\n\ndisplay_images(monet_example)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-three\"></a>\n## 3. Build Model\n\n* Build the generator that takes a noise vector and outputs a tensor of 256x256x3.\n* Build the discriminator that takes the tensor of 256x256x3 and outputs a probability that an image is real or fake.\n* Create two separate loss functions and optimizers for the generator and discriminator.","metadata":{}},{"cell_type":"markdown","source":"<div align=\"center\">\n    <img src=\"https://raw.githubusercontent.com/justinsiowqi/-Paint-With-DCGAN/main/GAN%20diagram.png\" alt=\"GAN Diagram\" style=\"width: 500px;\"> \n</div>\n<div align=\"center\">\n  GAN Architecture (source: GANs in Action)\n</div>","metadata":{}},{"cell_type":"markdown","source":"### An Overview on GANs:\n\nGANs are a type of machine learning algorithm that uses two neural networks to generate new data. The two networks **play a game** against each other. The generator's goal is to **produce fake data**, while the discriminator's goal is to **tell fake data apart from real data**. \n\nYou can imagine the generator to be a con-artist that forges paintings and resells them for a living. On the other hand, the discriminator is a museum purveyor who purchases art pieces for his gallery. \n\nLet's take a look at the diagram above. The generator uses the random noise vector (z) to produce a bunch of fake images (x*). The discriminator is also given a bunch of real images (x) from the training set. The discriminator must now decide which is real and fake. This process is repeated many times until the generator gets really good at producing fake images, whereas the disriminator gets better at distinguishing real from fake. ","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-three-one\"></a>\n### 3.1. Build the Generator","metadata":{}},{"cell_type":"markdown","source":"<div align=\"center\">\n    <img src=\"https://raw.githubusercontent.com/justinsiowqi/-Paint-With-DCGAN/main/DCGAN%20Generator%20diagram.png\" alt=\"DCGAN Generator Diagram\" style=\"width: 500px;\"> \n</div>\n<div align=\"center\">\n  DCGAN Generator (source: GANs in Action)\n</div>","metadata":{}},{"cell_type":"markdown","source":"### An Overview on the Generator:\n\nWe will start by defining a function called build_generator which uses the **Sequential API** to stack layers.\n\nThe generator takes in a random noise vector as input and reshapes it into a 8x8x1024 tensor through a fully connected layer. \n\nWe use **transposed convolutions** to reshape tensors until we get a tensor output of 256x256x3. Transposed convolutions upsizes a vetor into an image by **increasing the height and width** and **decreasing the depth**.\n\nBetween the transposed convolution layers, we applied **batch normalization** to stabilize the process. After batch normalization, the **ReLU** activation function is applied to help gradients flow through properly. \n\nAs for the final layer, we do not apply batch normalization. We also use the **tanh** activation function instead of leaky ReLU.","metadata":{}},{"cell_type":"code","source":"# Create a function to build the generator model\n\ndef build_generator():\n    model = Sequential()\n    model.add(Dense(8*8*1024, input_shape=(100,)))\n    model.add(Reshape((8, 8, 1024)))\n\n    model.add(Conv2DTranspose(512, (3, 3), strides=(2, 2), padding='same'))\n    model.add(BatchNormalization())\n    model.add(ReLU())\n\n    model.add(Conv2DTranspose(256, (3, 3), strides=(2, 2), padding='same'))\n    model.add(BatchNormalization())\n    model.add(ReLU())\n    \n    model.add(Conv2DTranspose(128, (3, 3), strides=(2, 2), padding='same'))\n    model.add(BatchNormalization())\n    model.add(ReLU())\n    \n    model.add(Conv2DTranspose(64, (3, 3), strides=(2, 2), padding='same'))\n    model.add(BatchNormalization())\n    model.add(ReLU())\n    \n    model.add(Conv2DTranspose(32, (3, 3), strides=(2, 2), padding='same'))\n    model.add(BatchNormalization())\n    model.add(ReLU())\n\n    model.add(Conv2DTranspose(3, (3, 3), strides=(1, 1), padding='same'))\n    model.add(Activation('tanh'))\n\n    return model","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Use the noise vector to create an image. The generator is still untrained here!\n\ngenerator = build_generator()\n\nnoise = tf.random.normal([1, 100])\ngenerated_image = generator(noise, training=False)\n\nplt.imshow(generated_image[0, :, :, 0])","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-three-two\"></a>\n### 3.2. Build the Discriminator","metadata":{}},{"cell_type":"markdown","source":"<div align=\"center\">\n    <img src=\"https://raw.githubusercontent.com/justinsiowqi/-Paint-With-DCGAN/main/DCGAN%20Discriminator%20Diagram.png\" alt=\"DCGAN Discriminator Diagram\" style=\"width: 500px;\"> \n</div>\n<div align=\"center\">\n  DCGAN Discriminator (source: GANs in Action)\n</div>","metadata":{}},{"cell_type":"markdown","source":"### An Overview on the Discriminator:\n\nNow that we've finished the generator, we can move on to the discriminator. You'll find that this is oddly similar to a regular classifier.\n\nLet's begin by creating a function called build_discriminator and using the **Sequential API**.\n\nThe discriminator receives the 256x256x3 image as input from the generator. \n\nNext, we apply several **convolutional layers** to downsize the image. Convolutional layers are like transposed convolutional layers but it works in reverse: they **decrease the height and width** and **increase the depth**.\n\nAlso, we apply **batch normalization**, **leaky ReLU** activation function and **dropout** after each convolutional layer. Take note that the very first layer does not have batch normalization.\n\nThe final layer is a fully connected layer that uses a **sigmoid** activation function and outputs the probability that an image is real rather than fake.","metadata":{}},{"cell_type":"code","source":"# Create a function to build the discriminator model\n\ndef build_discriminator():\n    model = Sequential()\n    model.add(Conv2D(32, (3, 3), strides=(1, 1), padding='same', input_shape=[256, 256, 3]))\n    model.add(LeakyReLU(0.2))\n    model.add(Dropout(0.3))\n    \n    model.add(Conv2D(64, (3, 3), strides=(2, 2), padding='same'))\n    model.add(BatchNormalization())\n    model.add(LeakyReLU(0.2))\n    model.add(Dropout(0.3))\n\n    model.add(Conv2D(128, (3, 3), strides=(2, 2), padding='same'))\n    model.add(BatchNormalization())\n    model.add(LeakyReLU(0.2))\n    model.add(Dropout(0.3))\n    \n    model.add(Conv2D(256, (3, 3), strides=(2, 2), padding='same'))\n    model.add(BatchNormalization())\n    model.add(LeakyReLU(0.2))\n    model.add(Dropout(0.3))\n    \n    model.add(Conv2D(512, (3, 3), strides=(2, 2), padding='same'))\n    model.add(BatchNormalization())\n    model.add(LeakyReLU(0.2))\n    model.add(Dropout(0.3))\n\n    model.add(Flatten())\n    model.add(Dense(1))\n    model.add(Activation('sigmoid'))\n\n    return model","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Use the discriminator to classify the image above (1 for real and 0 for fake)\n\ndiscriminator = build_discriminator()\ndecision = discriminator(generated_image)\nprint (decision)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-three-three\"></a>\n### 3.3. Define Loss Functions & Optimizers\n\n* Create two separate loss functions\n    * The generator performs well when the discriminator outputs 1 for a fake image.\n    * The discriminator performs well when it outputs 1 for a real image and 0 for a fake image.\n* Create two separate optimzers\n    * Both models use an Adam optimizer with learning rate of 0.0001 and beta of 0.5","metadata":{}},{"cell_type":"code","source":"# Create two separate loss functions for the generator and discriminator\n\ncross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits=True)\n\ndef generator_loss(fake_output):\n    return cross_entropy(tf.ones_like(fake_output), fake_output)\n\ndef discriminator_loss(real_output, fake_output):\n    real_loss = cross_entropy(tf.ones_like(real_output), real_output)\n    fake_loss = cross_entropy(tf.zeros_like(fake_output), fake_output)\n    total_loss = real_loss + fake_loss\n    return total_loss","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create two separate optimizers for the generator and discriminator\n\ngenerator_optimizer = tf.keras.optimizers.Adam(learning_rate=0.00015, beta_1=0.5)\ndiscriminator_optimizer = tf.keras.optimizers.Adam(learning_rate=0.0001, beta_1=0.5)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-four\"></a>\n## 4. Train Model\n\n* Create a custom training loop that trains the models and compute the losses.\n* Create a function to display the sampled images in a 4x4 grid.","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-four-one\"></a>\n### 4.1. Custom Training Loop","metadata":{}},{"cell_type":"markdown","source":"### An Overview on Custom Training Loops\n\nIn some cases, the regular fit() method is simply not flexible enough. Since we're using separate loss functions and optimizers for the generator and discriminator, it's best that we create a custom training loop.\n\nTraining the GAN requires a **ton** of trial and error so feel free to play around with the hyperparameters here!\n\nThere are two functions we need to create. The first function trains **one step** of a GAN. The generator uses the random noise vector to generate a bunch of images. The discriminator evaluates both the real and fake images. We compute the generator and discriminator loss using the loss functions we created earlier. Finally, we update the optimizers and return the losses for both models.\n\nThe second function involves training the GAN on **multiple epochs**. For each epoch, we will loop through the image batches and calculate the losses. Finally, we will output one sample image every 25 epochs (250 epochs / 25 sample interval).","metadata":{}},{"cell_type":"code","source":"# Set the hyperparameters to be used for training\n\nEPOCHS = 500\nnoise_dim = 100\nsample_interval = EPOCHS / 10\nnum_examples_to_generate = 16","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create a function that trains one step of a GAN\n\n@tf.function\ndef train_step(images):\n    \n    # Create random noise vect\n    noise = tf.random.normal([BATCH_SIZE, noise_dim])\n\n    with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:\n        \n        # Use random noise vector to generate images\n        generated_images = generator(noise, training=True)\n\n        # Discriminator evaluates the real and fake images\n        real_output = discriminator(images, training=True)\n        fake_output = discriminator(generated_images, training=True)\n\n        # Losses are computed for both the generator and discriminator\n        gen_loss = generator_loss(fake_output)\n        disc_loss = discriminator_loss(real_output, fake_output)\n\n    # Compute gradients\n    gradients_of_generator = gen_tape.gradient(gen_loss, generator.trainable_variables)\n    gradients_of_discriminator = disc_tape.gradient(disc_loss, discriminator.trainable_variables)\n\n    # Update optimizers\n    generator_optimizer.apply_gradients(zip(gradients_of_generator, generator.trainable_variables))\n    discriminator_optimizer.apply_gradients(zip(gradients_of_discriminator, discriminator.trainable_variables))\n    \n    return gen_loss, disc_loss","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create a function that trains the GAN on multiple epochs\n\ndef train(dataset, epochs):\n    \n    for epoch in range(epochs):\n        \n        # Create an empty list to store losses \n        gen_loss_list = []\n        disc_loss_list = []\n\n        for image_batch in dataset:\n            t = train_step(image_batch)\n            gen_loss_list.append(t[0])\n            disc_loss_list.append(t[1])\n            \n            # Calculate total loss \n            g_loss = sum(gen_loss_list) / len(gen_loss_list)\n            d_loss = sum(disc_loss_list) / len(disc_loss_list)\n\n        print (f'Epoch {epoch + 1} [G loss: {g_loss}] [D loss: {d_loss}]')\n        \n        # For every 25 epochs, we will output an image\n        if (epoch + 1) % sample_interval == 0:\n            sample_images(generator)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-four-two\"></a>\n### 4.2. Sample & Display Images\n\n* We use the sampled random noise vectors to generate the images, then we rescale the images to [0, 1].\n* Finally, we plot the images in a 4x4 grid.","metadata":{}},{"cell_type":"code","source":"# Create a function to plot the generated images from the GAN model\n\ndef sample_images(generator, image_grid_rows=4, image_grid_columns=4):\n\n    # Sample random noise vector\n    noise = tf.random.normal([BATCH_SIZE, noise_dim])\n\n    # Generate images\n    gen_imgs = generator.predict(noise)\n\n    # Rescale image pixel values to [0, 1]\n    gen_imgs = 0.5 * gen_imgs + 0.5\n\n    # Set grid and plot images\n    fig, axs = plt.subplots(image_grid_rows, image_grid_columns, figsize=(7, 7), sharey=True, sharex=True)\n\n    count = 0\n    for i in range(image_grid_rows):\n        for j in range(image_grid_columns):\n            axs[i, j].imshow(gen_imgs[count, :, :, :])\n            axs[i, j].axis('off')\n            count += 1","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train(monet_dataset, EPOCHS)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-five\"></a>\n## 5. Submit Images","metadata":{}},{"cell_type":"code","source":"# Import libraries\n\nimport os\nfrom PIL import Image\nimport shutil","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create new directory and generate images\n\nimage_dir =  '../DCGAN_images'\nif not os.path.exists(image_dir):\n    os.makedirs(image_dir)\n\nfor i in range(7000):\n    noise = tf.random.normal([BATCH_SIZE, noise_dim])\n    submission_imgs = generator.predict(noise)\n    submission_imgs = 0.5 * submission_imgs + 0.5\n    submission_imgs = (submission_imgs * 255).astype('uint8')\n    submission = Image.fromarray(submission_imgs[0, :, :, :])\n    submission.save(f\"{image_dir}/{i}.jpg\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create a zip file of the images\n\nshutil.make_archive('DCGAN_images', 'zip', 'DCGAN_images')","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"---","metadata":{}},{"cell_type":"markdown","source":"<a class=\"anchor\" id=\"section-six\"></a>\n## 6. Conclusion\n\nI hope you enjoyed learning how to create these Monet-esque paintings! \n\nHere's a quick summary of what we learnt today. We started off by parsing and extracting images from a TFRecord file. Then, we learnt that GANs are simply made of two separate models: the generator and the discriminator. The former creates fake images, while the latter attempts to tell apart those fake images from real images from the training set. We built both models step-by-step and created the separate loss functions and optimizers. Lastly, we learnt how to flexibly train a GAN model through a custom training loop. \n\nIf you have any questions or feedback, feel free to reach out in the comments section. Also, don't hesitate to upvote and drop a follow when you can. Have a great week ahead!","metadata":{}},{"cell_type":"markdown","source":"### References:\n\n* [A Hands-On Guide to TFRecords](https://towardsdatascience.com/a-practical-guide-to-tfrecords-584536bc786c)\n* [Write the TFRecord File](https://www.tensorflow.org/tutorials/load_data/tfrecord#write_the_tfrecord_file)\n* [Deep Convolutional Generative Adversarial Network](https://www.tensorflow.org/tutorials/generative/dcgan?hl=en)\n* [Monitor GAN Training Progress and Identify Common Failure Modes](https://www.mathworks.com/help/deeplearning/ug/monitor-gan-training-progress-and-identify-common-failure-modes.html)\n* [Deconvolution and Checkerboard Artifacts](https://distill.pub/2016/deconv-checkerboard/)\n* [How to Train a GAN? Tips and tricks to make GANs work](https://github.com/soumith/ganhacks)","metadata":{}}]}