{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# GAN's From Scratch \n\nby Kris Smith","metadata":{}},{"cell_type":"markdown","source":"# Problem Statement\n\n***This notebook is working towards generating images in the style of a famous artist painter [Claude Monet](https://en.wikipedia.org/wiki/Claude_Monet). This problem is presented as a competition on the Kaggle platform for the sake of learning. We will explore in particular one common approach to this task in the form of [Generative Adversarial Networks(GANs)](https://en.wikipedia.org/wiki/Generative_adversarial_network).***","metadata":{}},{"cell_type":"code","source":"!pip install albumentations\nfrom kaggle_datasets import KaggleDatasets\n\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nimport matplotlib.pyplot as plt\nimport os\nfrom zipfile import ZipFile\n\nimport os\nimport math\nimport random\n\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport cv2\nimport albumentations as A\nfrom albumentations import (\n    Compose, RandomBrightness, JpegCompression, HueSaturationValue, RandomContrast, HorizontalFlip,\n    Rotate\n)\nAUTOTUNE = tf.data.experimental.AUTOTUNE","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-06-24T04:14:30.365649Z","iopub.execute_input":"2023-06-24T04:14:30.365999Z","iopub.status.idle":"2023-06-24T04:14:56.302025Z","shell.execute_reply.started":"2023-06-24T04:14:30.365938Z","shell.execute_reply":"2023-06-24T04:14:56.300988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Overview\n\n### Dataset Description\n<i>The dataset contains four directories: monet_tfrec, photo_tfrec, monet_jpg, and photo_jpg. The monet_tfrec and monet_jpg directories contain the same painting images, and the photo_tfrec and photo_jpg directories contain the same photos.<i>\n\n<i>We recommend using TFRecords as a Getting Started competition is a great way to become more familiar with a new data format, but JPEG images have also been provided.\n\n<i>The monet directories contain Monet paintings. Use these images to train your model.\n\n<i>The photo directories contain photos. Add Monet-style to these images and submit your generated jpeg images as a zip file. Other photos outside of this dataset can be transformed but keep your submission file limited to 10,000 images.\n\n<i>Note: Monet-style art can be created from scratch using other GAN architectures like DCGAN. The submitted image files do not necessarily have to be transformed photos.\n\n<i>Check out the CycleGAN dataset to experiment with the artistic style of other artists.</i>\n\n    \n### Files\n* monet_jpg - 300 Monet paintings sized 256x256 in JPEG format\n* monet_tfrec - 300 Monet paintings sized 256x256 in TFRecord format\n* photo_jpg - 7028 photos sized 256x256 in JPEG format\n* photo_tfrec - 7028 photos sized 256x256 in TFRecord format\n\n### Submission format\n* Your kernel's output must be called images.zip and contain 7,000-10,000 images sized 256x256.\n    \n    \n### Evaluation\n    \nMiFID\n    \nSubmissions are evaluated on MiFID (Memorization-informed Fréchet Inception Distance), which is a modification from Fréchet Inception Distance (FID).\n\nThe smaller MiFID is, the better your generated images are.\n\nWhat is FID?\nOriginally published here (github), FID, along with Inception Score (IS), are both commonly used in recent publications as the standard for evaluation methods of GANs.\n\nIn FID, we use the Inception network to extract features from an intermediate layer. Then we model the data distribution for these features using a multivariate Gaussian distribution with mean µ and covariance Σ. The FID between the real images 𝑟\n and generated images 𝑔\n is computed as:\n$$\nFID = ||\\mu_{r} - \\mu_{g}||^{2} + Tr(\\Sigma_{r} + \\Sigma_{g} - 2(\\Sigma_{r}\\Sigma_{g})^{1/2})\n$$\n\n    \n    where 𝑇𝑟 sums up all the diagonal elements. FID is calculated by computing the Fréchet distance between two Gaussians fitted to feature representations of the Inception network.\n\nWhat is MiFID (Memorization-informed FID)?\nIn addition to FID, Kaggle takes training sample memorization into account.\n\nThe memorization distance is defined as the minimum cosine distance of all training samples in the feature space, averaged across all user generated image samples. This distance is thresholded, and it's assigned to 1.0 if the distance exceeds a pre-defined epsilon.\n\nIn mathematical form:\n    \n$$\nd_{ij} = 1 - \\cos(f_{gi}, f_{rj}) = 1 - \\frac{f_{gi} \\cdot f_{rj}}{||f_{gi}|| ||f_{rj}||}\n$$\n    \nwhere 𝑓𝑔\n and 𝑓𝑟\n represent the generated/real images in feature space (defined in pre-trained networks); and 𝑓𝑔𝑖\n and 𝑓𝑟𝑗\n represent the 𝑖𝑡ℎ\n and 𝑗𝑡ℎ\n vectors of 𝑓𝑔\n and 𝑓𝑟\n, respectively.\n\n$$\nd = \\frac{1}{N} \\sum_{i} \\min_{j} d_{ij}\n$$\n    \ndefines the minimum distance of a certain generated image (𝑖\n) across all real images ((𝑗\n), then averaged across all the generated images.\n\nenter image description here\ndefines the threshold of the weight only applies when the (𝑑\n) is below a certain empirically determined threshold.\n\nFinally, this memorization term is applied to the FID:\n    \n$$\nMiFID = FID * \\frac{1}{d_thr}\n$$\n    \n**Kaggle's workflow calculating MiFID for public and private scores**\n    \nKaggle calculates public MiFID scores with the pre-train neural network Inception, and the public images used for evaluation are the rest of the TFDS Monet paintings. ***Note that as a Getting Started competition there is no private leaderboard.***\n\n    \n### Submission File\n    \nYou are going to generate 7,000-10,000 Monet-style images that are in jpg format. Their sizes should be 256x256x3 (RGB). Then you need to zip those images and your output from your Kernel should only have ONE output file named images.zip.\n\nPlease note that Kaggle Kernels has a number of output files capped at 500. We highly encourage you to either directly write to a zip file as you generate images, or create a folder at ../tmp as your temporary directory.\n","metadata":{}},{"cell_type":"markdown","source":"---\n\n# Data","metadata":{}},{"cell_type":"markdown","source":"## EDA","metadata":{}},{"cell_type":"code","source":"BASE_PATH = \"../input/gan-getting-started/\"\nMONET_PATH = os.path.join(BASE_PATH, \"monet_jpg\")\nPHOTO_PATH = os.path.join(BASE_PATH, \"photo_jpg\")","metadata":{"execution":{"iopub.status.busy":"2023-06-24T04:14:56.325256Z","iopub.execute_input":"2023-06-24T04:14:56.325636Z","iopub.status.idle":"2023-06-24T04:14:56.330352Z","shell.execute_reply.started":"2023-06-24T04:14:56.325604Z","shell.execute_reply":"2023-06-24T04:14:56.329317Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"***Image statistics. The following function was provided by the following notebook: https://www.kaggle.com/code/ihelon/monet-visualization-and-augmentation. This function displays the number of images in the monet and photo folders along with their attributes.***","metadata":{}},{"cell_type":"code","source":"def print_folder_statistics(path):\n    d_image_sizes = {}\n    for image_name in os.listdir(path):\n        image = cv2.imread(os.path.join(path, image_name))\n        d_image_sizes[image.shape] = d_image_sizes.get(image.shape, 0) + 1\n        \n    for size, count in d_image_sizes.items():\n        print(f\"shape: {size}\\tcount: {count}\")\n\n\nprint(f\"Monet images:\")\nprint_folder_statistics(MONET_PATH)\nprint(\"-\" * 10)\nprint(f\"Photo images:\")\nprint_folder_statistics(PHOTO_PATH)\nprint(\"-\" * 10)","metadata":{"execution":{"iopub.status.busy":"2023-06-24T04:14:56.332992Z","iopub.execute_input":"2023-06-24T04:14:56.333588Z","iopub.status.idle":"2023-06-24T04:15:40.781063Z","shell.execute_reply.started":"2023-06-24T04:14:56.333557Z","shell.execute_reply":"2023-06-24T04:15:40.780106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"***We will focus mainly on the monet images for now. We can see that we are working with 300 of them and each image is 256x256 pixels with 3 channels of color(RGB).***","metadata":{}},{"cell_type":"markdown","source":"### Load Data","metadata":{}},{"cell_type":"code","source":"dataset = keras.utils.image_dataset_from_directory(\n    MONET_PATH, \n    label_mode=None, \n    image_size=(256, 256), \n    batch_size=8\n)\ndataset = dataset.map(lambda x: x / 255.0)","metadata":{"execution":{"iopub.status.busy":"2023-06-24T04:15:40.782408Z","iopub.execute_input":"2023-06-24T04:15:40.784587Z","iopub.status.idle":"2023-06-24T04:15:43.772531Z","shell.execute_reply.started":"2023-06-24T04:15:40.784537Z","shell.execute_reply":"2023-06-24T04:15:43.771580Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Inspecting Images","metadata":{}},{"cell_type":"code","source":"## Choose the number of images to display\nn = 4\n\n## Create a temporary dataset to stop iteration after the first batch\ndataset_temp = iter(dataset)\n\n## Fetch the first batch\nx = next(dataset_temp)\n\n## Make sure the data is in the correct format\nif isinstance(x, tuple):\n    x = x[0]  \n\nfig, axes = plt.subplots(n, 1, figsize=(20, 20))\n\nfor i in range(n):\n    img = x[i].numpy()\n    img = (img * 255).astype(\"int32\")\n    axes[i].axis(\"off\")\n    axes[i].imshow(img)\n\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-06-24T04:15:43.774015Z","iopub.execute_input":"2023-06-24T04:15:43.774372Z","iopub.status.idle":"2023-06-24T04:15:44.553414Z","shell.execute_reply.started":"2023-06-24T04:15:43.774325Z","shell.execute_reply":"2023-06-24T04:15:44.552598Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Feature Engineering\n\nSince we only have 300 images to train on this is not very many for a deep learning model to make use of. There are techniques we can apply to create more training data such as augmentation techniques. Cropping, jittering, rotating, flipping, and normalizing are techniques we can use to apply to the data we have to create new images out of the same data. \n\nI have used some methods mentioned above which someone had published in this notebook here: https://www.kaggle.com/code/dimitreoliveira/improving-cyclegan-monet-paintings.","metadata":{}},{"cell_type":"code","source":"IMAGE_SIZE = [256, 256]\n\ndef normalize(image):\n    return (tf.cast(image, tf.float32) / 127.5) - 1\n\ndef decode_image(image):\n    #image = tf.image.decode_jpeg(image, channels=3)\n    #image = tf.reshape(image, [256, 256, 3])\n    image = tf.image.decode_jpeg(image, channels=3)\n    #image = (tf.cast(image, tf.float32) / 127.5) - 1\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    return image\n\ndef random_crop(image):\n    cropped_image = tf.image.random_crop(image, size=[256, 256, 3])\n    return cropped_image\n\ndef random_jitter(image):\n    # resizing to 286 x 286 x 3 \n    image = tf.image.resize(image, [int(256*1.3), int(256*1.3)],\n                          method=tf.image.ResizeMethod.NEAREST_NEIGHBOR)\n    # randomly cropping to 256 x 256 x 3\n    image = random_crop(image)\n    # random mirroring\n    return image\n\ndef flip(image):\n    return tf.image.flip_left_right(image)\n\ndef preprocess_image_train(image, label=None):\n    image = random_jitter(image)\n    return image\n\ndef read_tfrecord(example):\n    tfrecord_format = {\n        \"image_name\": tf.io.FixedLenFeature([], tf.string),\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"target\": tf.io.FixedLenFeature([], tf.string)}\n    example = tf.io.parse_single_example(example, tfrecord_format)\n    image = decode_image(example['image'])\n    return image\n\ndef load_dataset(filenames, labeled=False, ordered=False, repeats=200):\n    dataset = tf.data.TFRecordDataset(filenames)\n    dataset = dataset.map(read_tfrecord, num_parallel_calls=AUTOTUNE)\n    dataset = dataset.concatenate(dataset.map(flip, num_parallel_calls=AUTOTUNE).shuffle(100000))\n    dataset = dataset.concatenate(dataset.map(random_jitter, num_parallel_calls=AUTOTUNE).shuffle(100000, reshuffle_each_iteration=True).repeat(repeats))\n    dataset = dataset.map(normalize, num_parallel_calls=AUTOTUNE).shuffle(100000)\n    return dataset\n\n\n# dataset = load_dataset(tf.io.gfile.glob(str(KaggleDatasets().get_gcs_path() + '/monet_tfrec/*.tfrec')),\n#                         labeled=True, \n#                         repeats=100).batch(100, drop_remainder=True)\n# dataset","metadata":{"execution":{"iopub.status.busy":"2023-06-24T04:15:44.554481Z","iopub.execute_input":"2023-06-24T04:15:44.554790Z","iopub.status.idle":"2023-06-24T04:15:44.571488Z","shell.execute_reply.started":"2023-06-24T04:15:44.554762Z","shell.execute_reply":"2023-06-24T04:15:44.570709Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model\n\nAfter trying a traditional vanilla GAN, and an AutoEncoder and not having much luck with either, I implemented a BiGAN and had some results finally. A BiGAN has a generator and discriminator just like a traditional GAN but you also train an Encoder along side with them. This allows the model to learn how to encode images in order to assist the generator.","metadata":{}},{"cell_type":"markdown","source":"## Create the discriminator\n\nIt maps a 256x256 image to a binary classification score.\n\nI added Gaussian noise to the input images because the generator was producing the same output for every prediction at inference time. This is know as `mode collapse` and I dealt with it using the noise adding technique along with dropout layers in the discriminator.\n\nAfter training with sigmoid activation function in the final layer I switched to Tanh which normalizes the outputs to the range of [-1, 1] as opposed to sigmoid which is [0, 1]. This seems to allow the model to learn the RGB scheme better and I think sigmoid would probably work better with black and white images.","metadata":{}},{"cell_type":"code","source":"## Function to add Gaussian noise to the inputs\ndef add_noise(layer):\n    noise = keras.layers.GaussianNoise(0.2)\n    return noise(layer)","metadata":{"execution":{"iopub.status.busy":"2023-06-24T04:15:44.572842Z","iopub.execute_input":"2023-06-24T04:15:44.573430Z","iopub.status.idle":"2023-06-24T04:15:44.589204Z","shell.execute_reply.started":"2023-06-24T04:15:44.573401Z","shell.execute_reply":"2023-06-24T04:15:44.588165Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"latent_dim = 512\ninput_img = layers.Input(shape=(256, 256, 3))\ninput_img = add_noise(input_img) ## Add noise\n\ninput_z = layers.Input(shape=(latent_dim,))\n\n# image processing\nx = layers.Conv2D(128, kernel_size=4, strides=2, padding=\"same\")(input_img)\nx = layers.LeakyReLU(alpha=0.2)(x)\nx = layers.Dropout(0.4)(x)\nx = layers.Conv2D(256, kernel_size=4, strides=2, padding=\"same\")(x)\nx = layers.LeakyReLU(alpha=0.2)(x)\nx = layers.Dropout(0.4)(x)\nx = layers.Conv2D(512, kernel_size=4, strides=2, padding=\"same\")(x)\nx = layers.LeakyReLU(alpha=0.2)(x)\n\nx = layers.Dropout(0.4)(x)\nx = layers.Flatten()(x)\n\n# latent vector processing\ny = layers.Dense(512)(input_z)\ny = layers.LeakyReLU(alpha=0.2)(y)\ny = layers.Dropout(0.3)(y)\n\n# combining the outputs\ncombined = layers.Concatenate()([x, y])\noutput = layers.Dense(1, activation=\"sigmoid\")(combined)\n\ndiscriminator = keras.Model([input_img, input_z], output, name=\"discriminator\")\ndiscriminator.summary()","metadata":{"execution":{"iopub.status.busy":"2023-06-24T04:15:44.590609Z","iopub.execute_input":"2023-06-24T04:15:44.591119Z","iopub.status.idle":"2023-06-24T04:15:44.776351Z","shell.execute_reply.started":"2023-06-24T04:15:44.591088Z","shell.execute_reply":"2023-06-24T04:15:44.775640Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Create the generator\n\nIt mirrors the discriminator, replacing `Conv2D` layers with `Conv2DTranspose` layers.","metadata":{}},{"cell_type":"code","source":"latent_dim = 512\n\ngenerator = keras.Sequential(\n    [\n        keras.Input(shape=(latent_dim,)),\n        layers.Dense(8 * 8 * 512),\n        layers.Reshape((8, 8, 512)),\n        layers.Conv2DTranspose(512, kernel_size=4, strides=2, padding=\"same\"),\n        layers.LeakyReLU(alpha=0.2),\n        layers.Conv2DTranspose(256, kernel_size=4, strides=2, padding=\"same\"),\n        layers.LeakyReLU(alpha=0.2),\n        layers.Conv2DTranspose(128, kernel_size=4, strides=2, padding=\"same\"),\n        layers.LeakyReLU(alpha=0.2),\n        layers.Conv2DTranspose(64, kernel_size=4, strides=2, padding=\"same\"),\n        layers.LeakyReLU(alpha=0.2),\n        layers.Conv2DTranspose(32, kernel_size=4, strides=2, padding=\"same\"),\n        layers.LeakyReLU(alpha=0.2),\n        \n        layers.Conv2D(3, kernel_size=5, padding=\"same\", activation=\"tanh\"),\n    ],\n    name=\"generator\",\n)\ngenerator.summary()","metadata":{"execution":{"iopub.status.busy":"2023-06-24T04:15:44.780078Z","iopub.execute_input":"2023-06-24T04:15:44.781170Z","iopub.status.idle":"2023-06-24T04:15:44.979895Z","shell.execute_reply.started":"2023-06-24T04:15:44.781137Z","shell.execute_reply":"2023-06-24T04:15:44.979207Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Create the Encoder\n\nBiGAN model requires an encoder be trained along side the generator.","metadata":{}},{"cell_type":"code","source":"encoder = keras.Sequential(\n    [\n        keras.Input(shape=(256, 256, 3)),\n        layers.Conv2D(128, kernel_size=4, strides=2, padding=\"same\"),\n        layers.LeakyReLU(alpha=0.2),\n        layers.Conv2D(256, kernel_size=4, strides=2, padding=\"same\"),\n        layers.LeakyReLU(alpha=0.2),\n        layers.Conv2D(512, kernel_size=4, strides=2, padding=\"same\"),\n        layers.LeakyReLU(alpha=0.2),\n        layers.Flatten(),\n        layers.Dense(latent_dim),\n    ],\n    name=\"encoder\",\n)\n\nencoder.summary()","metadata":{"execution":{"iopub.status.busy":"2023-06-24T04:15:44.980913Z","iopub.execute_input":"2023-06-24T04:15:44.981474Z","iopub.status.idle":"2023-06-24T04:15:45.082995Z","shell.execute_reply.started":"2023-06-24T04:15:44.981441Z","shell.execute_reply":"2023-06-24T04:15:45.082261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Create the BiGAN Class","metadata":{}},{"cell_type":"markdown","source":"Just like the traditional GAN with the BiGAN the discriminator does not get weights updated with gradient descent like the generator does in training. The encoder also gets updated weights throughout the process.\n\nThe class below contains logic to initialize and compile the model as well as train the BiGAN. \n\nFor each batch, training steps include:\n* Sample random points in the latent space from embedding model\n* Decode them into fake images\n* Encode the real images into latent space vectors\n* Combine the generated images with the real images\n* Combine the encoded latent vectors from the real images with the random latent vectors\n* Assemble labels discriminating real from fake image\n* Add random noise to the labels\n* Train the discriminator\n* Sample random points in latent space again\n* Assemble labels which are all real\n* Train the Generator\n* Update weights of generator\n* Train the encoder\n* Update weights of the encoder","metadata":{}},{"cell_type":"code","source":"class BiGAN(keras.Model):\n    def __init__(self, discriminator, generator, encoder, latent_dim):\n        super().__init__()\n        self.discriminator = discriminator\n        self.generator = generator\n        self.encoder = encoder\n        self.latent_dim = latent_dim\n\n    def compile(self, d_optimizer, g_optimizer, e_optimizer, loss_fn):\n        super().compile()\n        self.d_optimizer = d_optimizer\n        self.g_optimizer = g_optimizer\n        self.e_optimizer = e_optimizer\n        self.loss_fn = loss_fn\n        self.d_loss_metric = keras.metrics.Mean(name=\"d_loss\")\n        self.g_loss_metric = keras.metrics.Mean(name=\"g_loss\")\n        self.e_loss_metric = keras.metrics.Mean(name=\"e_loss\")\n\n    @property\n    def metrics(self):\n        return [self.d_loss_metric, self.g_loss_metric, self.e_loss_metric]\n\n    def train_step(self, real_images):\n        # Sample random points in the latent space\n        batch_size = tf.shape(real_images)[0]\n        random_latent_vectors = tf.random.normal(shape=(batch_size, self.latent_dim))\n\n        # Decode them to fake images\n        generated_images = self.generator(random_latent_vectors)\n        # Encode real images to latent vectors\n        encoded_latent_vectors = self.encoder(real_images)\n\n        # Combine them with real images\n        fake_and_real_images = tf.concat([generated_images, real_images], axis=0)\n        # Combine them with latent vectors\n        encoded_and_random_latent_vectors = tf.concat([encoded_latent_vectors, random_latent_vectors], axis=0)\n\n        # Assemble labels discriminating real from fake images\n        labels = tf.concat(\n            [tf.ones((batch_size, 1)), tf.zeros((batch_size, 1))], axis=0\n        )\n        # Add random noise to the labels - important trick!\n        labels += 0.05 * tf.random.uniform(tf.shape(labels))\n\n        # Train the discriminator but do not update weights\n        with tf.GradientTape() as tape:\n            predictions = self.discriminator([fake_and_real_images, encoded_and_random_latent_vectors])\n            d_loss = self.loss_fn(labels, predictions)\n#         grads = tape.gradient(d_loss, self.discriminator.trainable_weights)\n#         self.d_optimizer.apply_gradients(\n#             zip(grads, self.discriminator.trainable_weights)\n#         )\n\n        # Sample random points in the latent space\n        random_latent_vectors = tf.random.normal(shape=(batch_size, self.latent_dim))\n\n        # Assemble labels that say \"all real images\"\n        misleading_labels = tf.zeros((batch_size, 1))\n\n        # Train the generator\n        with tf.GradientTape() as tape:\n            predictions = self.discriminator([self.generator(random_latent_vectors), random_latent_vectors])\n            g_loss = self.loss_fn(misleading_labels, predictions)\n        grads = tape.gradient(g_loss, self.generator.trainable_weights)\n        self.g_optimizer.apply_gradients(zip(grads, self.generator.trainable_weights))\n\n        # Train the encoder\n        with tf.GradientTape() as tape:\n            predictions = self.discriminator([real_images, self.encoder(real_images)])\n            e_loss = self.loss_fn(misleading_labels, predictions)\n        grads = tape.gradient(e_loss, self.encoder.trainable_weights)\n        self.e_optimizer.apply_gradients(zip(grads, self.encoder.trainable_weights))\n\n        # Update metrics\n        self.d_loss_metric.update_state(d_loss)\n        self.g_loss_metric.update_state(g_loss)\n        self.e_loss_metric.update_state(e_loss)\n        \n        return {\n            \"d_loss\": self.d_loss_metric.result(),\n            \"g_loss\": self.g_loss_metric.result(),\n            \"e_loss\": self.e_loss_metric.result(),\n        }","metadata":{"execution":{"iopub.status.busy":"2023-06-24T04:15:45.083965Z","iopub.execute_input":"2023-06-24T04:15:45.084275Z","iopub.status.idle":"2023-06-24T04:15:45.108768Z","shell.execute_reply.started":"2023-06-24T04:15:45.084250Z","shell.execute_reply":"2023-06-24T04:15:45.107885Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Class below is a helper monitor for training the BiGAN. This class helps to handle the testing data and helps encode images for each training batch. It also contains the logic for call backs after training to find the most optimized epoch from training and revert the model to that specific checkpoint.","metadata":{}},{"cell_type":"code","source":"class GANMonitor(keras.callbacks.Callback):\n    def __init__(self, num_img=3, latent_dim=512):\n        self.num_img = num_img\n        self.latent_dim = latent_dim\n        self.test_data = None\n\n    def set_test_data(self, test_data):\n        self.test_data = test_data\n\n    def on_epoch_end(self, epoch, logs=None):\n        # Check if test_data is set\n        if self.test_data is not None:\n            # Retrieve a batch of data from the dataset\n            real_images = next(iter(self.test_data.take(self.num_img)))\n            \n            # Encode the real images to obtain the latent vectors\n            encoded_latent_vectors = self.model.encoder(real_images)\n            \n            # Generate images from the encoded latent vectors\n            generated_images = self.model.generator(encoded_latent_vectors)\n            generated_images *= 255\n            generated_images.numpy()\n            for i in range(self.num_img):\n                img = keras.utils.array_to_img(generated_images[i])\n                img.save(\"generated_img_%03d_%d.png\" % (epoch, i))","metadata":{"execution":{"iopub.status.busy":"2023-06-24T04:15:45.110079Z","iopub.execute_input":"2023-06-24T04:15:45.111053Z","iopub.status.idle":"2023-06-24T04:15:45.123103Z","shell.execute_reply.started":"2023-06-24T04:15:45.111016Z","shell.execute_reply":"2023-06-24T04:15:45.122359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Training\n\nAfter training and finally getting some outputs which were not just blobs of random color, I finally tried inference and made some submissions. After my 10th submission the best score I could get with the time I had and this approach was 311 which by the looks of the leader board is not great. But this could probably be improved drastically by switching approaches to a more modern architecture like a WGAN or CycleGAN which are the new SOTA for this type of problem.","metadata":{}},{"cell_type":"code","source":"epochs = 25\n\ngan = BiGAN(discriminator=discriminator, generator=generator, encoder=encoder, latent_dim=latent_dim)\n\ngan.compile(\n    e_optimizer=keras.optimizers.Adam(learning_rate=0.0002),\n    d_optimizer=keras.optimizers.Adam(learning_rate=0.0002),\n    g_optimizer=keras.optimizers.Adam(learning_rate=0.0002),\n    loss_fn=keras.losses.BinaryCrossentropy(),\n)\n\nhistory = gan.fit(\n    dataset, \n    epochs=epochs, \n    callbacks=[GANMonitor(num_img=10, latent_dim=latent_dim)]\n)","metadata":{"execution":{"iopub.status.busy":"2023-06-24T04:17:40.805851Z","iopub.execute_input":"2023-06-24T04:17:40.806574Z","iopub.status.idle":"2023-06-24T04:21:23.787806Z","shell.execute_reply.started":"2023-06-24T04:17:40.806538Z","shell.execute_reply":"2023-06-24T04:21:23.786756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.plot(history.history['g_loss'])\nplt.title('Model loss')\nplt.ylabel('Loss')\nplt.xlabel('Epoch')\nplt.legend(['gen_loss'], loc='upper left')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-06-24T04:21:38.651767Z","iopub.execute_input":"2023-06-24T04:21:38.652136Z","iopub.status.idle":"2023-06-24T04:21:38.919746Z","shell.execute_reply.started":"2023-06-24T04:21:38.652104Z","shell.execute_reply":"2023-06-24T04:21:38.918902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"PHOTO_FILENAMES = tf.io.gfile.glob(str(KaggleDatasets().get_gcs_path() + '/photo_tfrec/*.tfrec'))\nprint('Photo TFRecord Files:', len(PHOTO_FILENAMES))\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    dataset = tf.data.TFRecordDataset(filenames)\n    dataset = dataset.map(read_tfrecord, num_parallel_calls=AUTOTUNE)\n    return dataset\n\n\nphoto_ds = load_dataset(PHOTO_FILENAMES, labeled=True).batch(1)","metadata":{"execution":{"iopub.status.busy":"2023-06-24T04:21:54.010792Z","iopub.execute_input":"2023-06-24T04:21:54.011482Z","iopub.status.idle":"2023-06-24T04:21:54.674992Z","shell.execute_reply.started":"2023-06-24T04:21:54.011446Z","shell.execute_reply":"2023-06-24T04:21:54.673991Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"_, ax = plt.subplots(5, 2, figsize=(12, 12))\nfor i, img in enumerate(photo_ds.take(5)):\n    # Encode the input image\n    encoded_img = gan.encoder(img)\n\n    # Generate the prediction using the generator\n    prediction = gan.generator(encoded_img)[0].numpy()\n\n    # Post-process the images\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    img = img[0].numpy().astype('int32')\n    prediction = (img * 127.5 + 127.5).astype(np.uint8)\n\n    ax[i, 0].imshow(img)\n    ax[i, 1].imshow(prediction)\n    ax[i, 1].set_title(\"Monet-esque\")\n    ax[i, 0].set_title(\"Real Photo\")\n    ax[i, 1].axis(\"off\")\n    ax[i, 0].axis(\"off\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-06-24T04:28:51.517531Z","iopub.execute_input":"2023-06-24T04:28:51.517898Z","iopub.status.idle":"2023-06-24T04:28:53.085812Z","shell.execute_reply.started":"2023-06-24T04:28:51.517869Z","shell.execute_reply":"2023-06-24T04:28:53.084888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Conclusion\n\nThe biggest takeaway for me on this project was that I should spend a little more time researching how others are approaching this problem first. I wanted to learn some traditional GANs as I was brand new to them so I started with a vanilla GAN and after I did not get much results from it I tried implementing an AeGAN which I had better results with but still not getting much more than blobs output. My final model was a BiGAN implementation which would have probably worked if we were just making images of paintings look like Monet but the problem is to train on painted style images but then photos were imput to the encoder and generator and I just could not get much better performance after a lot of attempts and hours. Many people it looks like are using StyleGAN or CycleGAN and I should have tried one of those instead...I learned to spend more time researching the problem and approaches others use first.\n\n","metadata":{}},{"cell_type":"code","source":"import PIL\n! mkdir ../images","metadata":{"execution":{"iopub.status.busy":"2023-06-24T04:17:35.965508Z","iopub.status.idle":"2023-06-24T04:17:35.966129Z","shell.execute_reply.started":"2023-06-24T04:17:35.965854Z","shell.execute_reply":"2023-06-24T04:17:35.965876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Inference\n\nGenerate Monet like images from the photos provided in the photo dataset.","metadata":{}},{"cell_type":"code","source":"i = 1\nfor img in photo_ds:\n    if i % 350 == 0:\n        print(f\"{round(i/7038*100)}% Complete\")\n\n    # Encode the input image\n    encoded_img = gan.encoder(img)\n    \n    # Generate the prediction using the generator\n    prediction = gan.generator(encoded_img)[0].numpy()\n    \n    # Post-process the images\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    \n    # Convert numpy array to image and save it\n    im = PIL.Image.fromarray(prediction)\n    im.save(\"../images/\" + str(i) + \".jpg\")\n    i += 1","metadata":{"execution":{"iopub.status.busy":"2023-06-24T04:17:35.967826Z","iopub.status.idle":"2023-06-24T04:17:35.969305Z","shell.execute_reply.started":"2023-06-24T04:17:35.969061Z","shell.execute_reply":"2023-06-24T04:17:35.969084Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import shutil\nshutil.make_archive(\"/kaggle/working/images\", 'zip', \"/kaggle/images\")","metadata":{"execution":{"iopub.status.busy":"2023-06-24T04:17:35.970641Z","iopub.status.idle":"2023-06-24T04:17:35.971409Z","shell.execute_reply.started":"2023-06-24T04:17:35.971169Z","shell.execute_reply":"2023-06-24T04:17:35.971192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Save Models","metadata":{}},{"cell_type":"code","source":"# Save model weights\ngan.save_weights('bigan_weights')\ngan.generator.save_weights('bigan_generator_weights')\ngan.discriminator.save_weights('bigan_discriminator_weights')\ngan.encoder.save_weights('bigan_encoder_weights')\n\n# Load model weights\n# gan.load_weights('bigan_weights')\n# gan.generator.load_weights('bigan_generator_weights')\n# gan.discriminator.load_weights('bigan_discriminator_weights')\n# gan.encoder.load_weights('bigan_encoder_weights')","metadata":{"execution":{"iopub.status.busy":"2023-06-24T04:17:35.972763Z","iopub.status.idle":"2023-06-24T04:17:35.973560Z","shell.execute_reply.started":"2023-06-24T04:17:35.973283Z","shell.execute_reply":"2023-06-24T04:17:35.973305Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}