{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"nvidiaTeslaT4","dataSources":[{"sourceId":21755,"databundleVersionId":1475600,"sourceType":"competition"},{"sourceId":7800428,"sourceType":"datasetVersion","datasetId":4567376}],"dockerImageVersionId":30666,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"!pip install -U tensorflow\n!pip install imageio","metadata":{"execution":{"iopub.status.busy":"2024-03-12T02:54:42.652852Z","iopub.execute_input":"2024-03-12T02:54:42.653187Z","iopub.status.idle":"2024-03-12T02:55:07.195632Z","shell.execute_reply.started":"2024-03-12T02:54:42.653154Z","shell.execute_reply":"2024-03-12T02:55:07.194497Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Monet Paintings Generator Using CycleGAN\n\nEvery painter has its own style, as it is the case of Van Gogh, Michelangelo, Frida Kahlo, Monet, etc. Replicating this style can be challenging to any person with no experience in the field of painting. Identify the patterns and replicate them into an artwork it's not the easiest task for us as humans. Nevertheless, neural networks are especially effective performing this tasks, this time talking specificly about Generative Adversarial Networks (GAN). In this document we will explore the uses of GAN, generating Monet's styled photos.\n\n\"Monet\n\n***(NGA Gov, 2024)***","metadata":{}},{"cell_type":"markdown","source":"## How does a GAN works?\n\nThis type of Neural Network consist basicly in a generator and a discriminator. The generator is a neural network which its output is meant to be a styled image good enough to \"fool\" the discriminator and make him believe that is a \"real\" image. The discriminator is in charge of deciding whether and image is \"real\" or not.\nWhat do we mean with real? A \"real\" image is in this case a real monet painting.\n\nThis two Neural Networks compete each other to generate a realistically styled photo, hence the name.\n\n\"GAN\n\n***(Google for developers, 2022)***\n\n### CycleGAN a variant of GANs\n\nHowever, in this document we will focus on a variant of GANs called CycleGAN, which consists on the same principles of the GANs, a discriminator and a generator. The main difference of this variant is that once the image is styled, it is then unstyled back. This is done to ensure the preservation of the characteristics of the original image, computing and minimizing the loss of the four operations performed (Monet Discriminator, Monet Generator, Photo Discriminator, Photo Generator). This approach leads to a significant improvement of the performance of the model, as preserving the essence of the original photo its necessary to have a measurement of the similarity between these two.","metadata":{}},{"cell_type":"markdown","source":"# Exploratory Data Analysis - (EDA)","metadata":{}},{"cell_type":"markdown","source":"## Data Loading\nIn this project we will use TPUs to train our model, these devices can significantly improve the time of training. To make use of them it's necessary to feed our model with tfrecords, the format used to train models with TPU.","metadata":{}},{"cell_type":"code","source":"import os\nimport tensorflow as tf\n#import tensorflow_datasets as tfds\nfrom tensorflow.keras.layers import LayerNormalization\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\n\ntry:\n # Get the connected TPUs\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver(\"local\")\n print('Device:', tpu.master())\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n\n # Set strategy\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\nexcept:\n strategy = tf.distribute.get_strategy()\nprint('Number of replicas:', strategy.num_replicas_in_sync)\n\nAUTOTUNE = tf.data.experimental.AUTOTUNE\n \nprint(tf.__version__)","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-03-12T02:55:07.197993Z","iopub.execute_input":"2024-03-12T02:55:07.198381Z","iopub.status.idle":"2024-03-12T02:55:14.018322Z","shell.execute_reply.started":"2024-03-12T02:55:07.198347Z","shell.execute_reply":"2024-03-12T02:55:14.017376Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Read the tfrecords files\nDS_PATH = '/kaggle/input/gan-getting-started'\nMONET_FILENAMES = tf.io.gfile.glob(str(DS_PATH + '/monet_tfrec/*.tfrec'))\nprint('Monet TFRecord Files:', len(MONET_FILENAMES))\n\nPHOTO_FILENAMES = tf.io.gfile.glob(str(DS_PATH + '/photo_tfrec/*.tfrec'))\nprint('Photo TFRecord Files:', len(PHOTO_FILENAMES))","metadata":{"execution":{"iopub.status.busy":"2024-03-12T02:55:14.019912Z","iopub.execute_input":"2024-03-12T02:55:14.020274Z","iopub.status.idle":"2024-03-12T02:55:14.056303Z","shell.execute_reply.started":"2024-03-12T02:55:14.020242Z","shell.execute_reply":"2024-03-12T02:55:14.055458Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Read the number of records in jpg\nMONET_JPG = \"/kaggle/input/gan-getting-started/monet_jpg\"\nPHOTO_JPG = \"/kaggle/input/gan-getting-started/photo_jpg\"\n\nprint(f\"Monet records {len(os.listdir(MONET_JPG))}\")\nprint(f\"Photo records {len(os.listdir(PHOTO_JPG))}\")","metadata":{"execution":{"iopub.status.busy":"2024-03-12T02:55:14.058672Z","iopub.execute_input":"2024-03-12T02:55:14.059282Z","iopub.status.idle":"2024-03-12T02:55:14.990237Z","shell.execute_reply.started":"2024-03-12T02:55:14.059247Z","shell.execute_reply":"2024-03-12T02:55:14.989317Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"IMAGE_SIZE = [256, 256]\n\ndef decode_image(image):\n \"\"\" \n Decode a tfrecotd to image \n \"\"\"\n image = tf.image.decode_jpeg(image, channels=3)\n image = (tf.cast(image, tf.float32) / 127.5) - 1\n image = tf.reshape(image, [*IMAGE_SIZE, 3])\n return image\n\ndef read_tfrecord(example):\n \"\"\" \n Read a tfrecords and return an image \n \"\"\"\n tfrecord_format = {\n \"image_name\": tf.io.FixedLenFeature([], tf.string),\n \"image\": tf.io.FixedLenFeature([], tf.string),\n \"target\": tf.io.FixedLenFeature([], tf.string)\n }\n example = tf.io.parse_single_example(example, tfrecord_format)\n image = decode_image(example['image'])\n return image","metadata":{"execution":{"iopub.status.busy":"2024-03-12T02:55:14.991539Z","iopub.execute_input":"2024-03-12T02:55:14.991907Z","iopub.status.idle":"2024-03-12T02:55:14.999492Z","shell.execute_reply.started":"2024-03-12T02:55:14.991874Z","shell.execute_reply":"2024-03-12T02:55:14.998633Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_dataset(filenames, labeled=True, ordered=False):\n \"\"\" \n Load the dataset and configures it for the TPUs \n \"\"\"\n dataset = tf.data.TFRecordDataset(filenames)\n dataset = dataset.map(read_tfrecord, num_parallel_calls=AUTOTUNE)\n return dataset","metadata":{"execution":{"iopub.status.busy":"2024-03-12T02:55:15.000685Z","iopub.execute_input":"2024-03-12T02:55:15.000959Z","iopub.status.idle":"2024-03-12T02:55:15.013374Z","shell.execute_reply.started":"2024-03-12T02:55:15.000935Z","shell.execute_reply":"2024-03-12T02:55:15.012605Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Datasets of monet and real photos\nmonet_ds = load_dataset(MONET_FILENAMES, labeled=True).batch(1)\nphoto_ds = load_dataset(PHOTO_FILENAMES, labeled=True).batch(1)","metadata":{"execution":{"iopub.status.busy":"2024-03-12T02:55:15.014399Z","iopub.execute_input":"2024-03-12T02:55:15.014705Z","iopub.status.idle":"2024-03-12T02:55:15.487410Z","shell.execute_reply.started":"2024-03-12T02:55:15.014673Z","shell.execute_reply":"2024-03-12T02:55:15.486382Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# First example of each dataset\nexample_monet = next(iter(monet_ds))\nexample_photo = next(iter(photo_ds))","metadata":{"execution":{"iopub.status.busy":"2024-03-12T02:55:15.488669Z","iopub.execute_input":"2024-03-12T02:55:15.488943Z","iopub.status.idle":"2024-03-12T02:55:15.579549Z","shell.execute_reply.started":"2024-03-12T02:55:15.488919Z","shell.execute_reply":"2024-03-12T02:55:15.578757Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Analysis\n\nThe mean intensity, and the standard deviation of the pixels are two metrics that can tell us a lot about the distribution of the data. The distribution can give us a hint of the differences between real photos and monet paintings.","metadata":{}},{"cell_type":"code","source":"import cv2\nimport numpy as np\nimport os\n\ndef calculate_dataset_statistics(dataset_path):\n \"\"\"\n Calculate the pixel intensity std deviation and mean\n \"\"\"\n total_images = 0\n pixel_intensities = []\n\n for filename in os.listdir(dataset_path):\n if filename.endswith(('.jpg', '.jpeg', '.png')):\n image = cv2.imread(os.path.join(dataset_path, filename), cv2.IMREAD_GRAYSCALE)\n\n total_images += 1\n pixel_intensities.extend(image.flatten())\n\n pixel_intensities = np.array(pixel_intensities)\n\n mean_intensity = np.mean(pixel_intensities)\n std_dev_intensity = np.std(pixel_intensities)\n\n return total_images, mean_intensity, std_dev_intensity","metadata":{"execution":{"iopub.status.busy":"2024-03-12T02:55:15.580662Z","iopub.execute_input":"2024-03-12T02:55:15.580924Z","iopub.status.idle":"2024-03-12T02:55:15.810876Z","shell.execute_reply.started":"2024-03-12T02:55:15.580902Z","shell.execute_reply":"2024-03-12T02:55:15.809952Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Print statistics of the photos\ntotal_images, mean_intensity, std_dev_intensity = calculate_dataset_statistics(PHOTO_JPG)\nmon_total_images, mon_mean_intensity, mon_std_dev_intensity = calculate_dataset_statistics(MONET_JPG)\n\nprint(\"Real Photos \\n----------------------\")\nprint(f\"Image Number: {total_images}\")\nprint(f\"Mean pixel intensity: {mean_intensity}\")\nprint(f\"Pixel std deviation: {std_dev_intensity}\")\n\nprint(\"\\nMonet \\n----------------------\")\nprint(f\"Image Number: {mon_total_images}\")\nprint(f\"Mean pixel intensity: {mon_mean_intensity}\")\nprint(f\"Pixel std deviation: {mon_std_dev_intensity}\")","metadata":{"execution":{"iopub.status.busy":"2024-03-12T02:55:15.814483Z","iopub.execute_input":"2024-03-12T02:55:15.814764Z","iopub.status.idle":"2024-03-12T02:58:28.401754Z","shell.execute_reply.started":"2024-03-12T02:55:15.814741Z","shell.execute_reply":"2024-03-12T02:58:28.400801Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In this case the images present a very similar distributions. This can contribute to an improvement of the training process and give a more natural and coherent look to the generated images.","metadata":{}},{"cell_type":"markdown","source":"### Examples\n\nFor a better understanding of the data here are some examples of the training photos:","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\n#Print the examples of images\nplt.subplot(121)\nplt.title('Photo')\nplt.imshow(example_photo[0] * 0.5 + 0.5)\n\nplt.subplot(122)\nplt.title('Monet Painting')\nplt.imshow(example_monet[0] * 0.5 + 0.5)","metadata":{"execution":{"iopub.status.busy":"2024-03-12T02:58:28.402982Z","iopub.execute_input":"2024-03-12T02:58:28.403404Z","iopub.status.idle":"2024-03-12T02:58:28.950025Z","shell.execute_reply.started":"2024-03-12T02:58:28.403372Z","shell.execute_reply":"2024-03-12T02:58:28.948896Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Both of the types of photos we have have the same size (256 x 256) and three channels which are Red, Green and Blue (RGB).","metadata":{}},{"cell_type":"markdown","source":"# Model","metadata":{}},{"cell_type":"markdown","source":"## Architecture\n\nFirst, we define a downsample layer consisting of:\n1. 2d Convolution - Reduces the size by two because of the stride of two.\n2. Instance Normalization - Only in first layer, to prevent overfitting and improve the convergence.\n3. Leaky ReLU - To prevent the dying ReLU\n\nThen, a upsample Layer consisting of:\n1. 2d Transposed Convolution - Used to augment dimensionality.\n2. Instance Normalization - Only in first three layers, to prevent overfitting and improve the convergence.\n3. Dropout - To improve the generalization of the model.\n4. ReLU - ReLU activation function.","metadata":{}},{"cell_type":"code","source":"OUTPUT_CHANNELS = 3\nwith strategy.scope():\n class InstanceNormalization(layers.Layer):\n \"\"\" \n Normalization layer, unlike the batch normalization, the normalization done\n by this layer is done on every image and not the entire batch \n \"\"\"\n def __init__(self, gamma_initializer='random_normal', epsilon=1e-5):\n super(InstanceNormalization, self).__init__()\n self.epsilon = epsilon\n self.gamma_initializer = gamma_initializer\n\n def build(self, input_shape):\n self.scale = self.add_weight(\n name='scale',\n shape=input_shape[-1:],\n initializer=self.gamma_initializer,\n trainable=True\n )\n self.offset = self.add_weight(\n name='offset',\n shape=input_shape[-1:],\n initializer='zeros',\n trainable=True\n )\n\n def call(self, inputs):\n mean, variance = tf.nn.moments(inputs, axes=[1, 2], keepdims=True)\n inv = tf.math.rsqrt(variance + self.epsilon)\n normalized = (inputs - mean) * inv\n return self.scale * normalized + self.offset\n\n def downsample(filters, size, apply_instancenorm=True):\n initializer = tf.random_normal_initializer(0., 0.02)\n gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n result = keras.Sequential()\n result.add(layers.Conv2D(filters, size, strides=2, padding='same',\n kernel_initializer=initializer, use_bias=False))\n\n if apply_instancenorm:\n result.add(InstanceNormalization(gamma_initializer=gamma_init))\n\n result.add(layers.LeakyReLU())\n\n return result\n\n def upsample(filters, size, apply_dropout=False):\n initializer = tf.random_normal_initializer(0., 0.02)\n gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n result = keras.Sequential()\n result.add(layers.Conv2DTranspose(filters, size, strides=2,\n padding='same',\n kernel_initializer=initializer,\n use_bias=False))\n\n result.add(InstanceNormalization(gamma_initializer=gamma_init))\n\n if apply_dropout:\n result.add(layers.Dropout(0.5))\n\n result.add(layers.ReLU())\n\n return result","metadata":{"execution":{"iopub.status.busy":"2024-03-12T02:58:28.951889Z","iopub.execute_input":"2024-03-12T02:58:28.952257Z","iopub.status.idle":"2024-03-12T02:58:28.968437Z","shell.execute_reply.started":"2024-03-12T02:58:28.952223Z","shell.execute_reply":"2024-03-12T02:58:28.967652Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Generator\nThe generator doesn´t have only sequential connections, instead it has skip connections between the down_stack and the up_stack. This prevents the vanishing gradient and helps the model to recover specific details from the original image.\n\nIt consists of:\n1. Input Layer - The input should have the shape 256x256x3, the size of the images previously defined.\n2. Down Stack - Eight downsample layers with skip connections to the Up Stack.\n3. Up Stack - Seven upsample layers\n4. 2d Transposed Convolution - Used to augment dimensionality and get a 256x256x3 result.","metadata":{}},{"cell_type":"code","source":"def Generator():\n inputs = layers.Input(shape=[256,256,3])\n\n # bs = batch size\n down_stack = [\n downsample(64, 4, apply_instancenorm=False), # (bs, 128, 128, 64)\n downsample(128, 4), # (bs, 64, 64, 128)\n downsample(256, 4), # (bs, 32, 32, 256)\n downsample(512, 4), # (bs, 16, 16, 512)\n downsample(512, 4), # (bs, 8, 8, 512)\n downsample(512, 4), # (bs, 4, 4, 512)\n downsample(512, 4), # (bs, 2, 2, 512)\n downsample(512, 4), # (bs, 1, 1, 512)\n ]\n\n up_stack = [\n upsample(512, 4, apply_dropout=True), # (bs, 2, 2, 1024)\n upsample(512, 4, apply_dropout=True), # (bs, 4, 4, 1024)\n upsample(512, 4, apply_dropout=True), # (bs, 8, 8, 1024)\n upsample(512, 4), # (bs, 16, 16, 1024)\n upsample(256, 4), # (bs, 32, 32, 512)\n upsample(128, 4), # (bs, 64, 64, 256)\n upsample(64, 4), # (bs, 128, 128, 128)\n ]\n\n initializer = tf.random_normal_initializer(0., 0.02)\n last = layers.Conv2DTranspose(OUTPUT_CHANNELS, 4,\n strides=2,\n padding='same',\n kernel_initializer=initializer,\n activation='tanh') # (bs, 256, 256, 3)\n\n x = inputs\n\n # Downsampling through the model\n skips = []\n for down in down_stack:\n x = down(x)\n skips.append(x)\n\n skips = reversed(skips[:-1])\n\n # Upsampling and establishing the skip connections\n for up, skip in zip(up_stack, skips):\n x = up(x)\n x = layers.Concatenate()([x, skip])\n\n x = last(x)\n\n return keras.Model(inputs=inputs, outputs=x)","metadata":{"execution":{"iopub.status.busy":"2024-03-12T02:58:28.969487Z","iopub.execute_input":"2024-03-12T02:58:28.969776Z","iopub.status.idle":"2024-03-12T02:58:28.987008Z","shell.execute_reply.started":"2024-03-12T02:58:28.969752Z","shell.execute_reply":"2024-03-12T02:58:28.986134Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Discriminator\n\nThe discriminator consists of:\n1. Input Layer - The input should have the shape 256x256x3, the size of the images previously defined.\n2. Down Stach - The down stack consists of 3 downsample layers.\n3. Zero Padding - The zero padding layer fill with zeros the borders of the image to avoid loosing information on the convolution layer.\n4. 2d Convolution - Reduces the size and catch complex patterns.\n5. Instance Normalization - Only in first three layers, to prevent overfitting and improve the convergence.\n6. Leaky Relu - To prevent the dying ReLU.\n7. Zero Padding - The zero padding layer fill with zeros the borders of the image to avoid loosing information on the convolution layer.\n8. 2d Convolution - Generates the activation map to be compared with the generated by real Monet paintings.","metadata":{}},{"cell_type":"code","source":"def Discriminator():\n initializer = tf.random_normal_initializer(0., 0.02)\n gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n inp = layers.Input(shape=[256, 256, 3], name='input_image')\n\n x = inp\n\n down1 = downsample(64, 4, False)(x) # (bs, 128, 128, 64)\n down2 = downsample(128, 4)(down1) # (bs, 64, 64, 128)\n down3 = downsample(256, 4)(down2) # (bs, 32, 32, 256)\n\n zero_pad1 = layers.ZeroPadding2D()(down3) # (bs, 34, 34, 256)\n conv = layers.Conv2D(512, 4, strides=1,\n kernel_initializer=initializer,\n use_bias=False)(zero_pad1) # (bs, 31, 31, 512)\n\n norm1 = InstanceNormalization(gamma_initializer=gamma_init)(conv)\n \n\n leaky_relu = layers.LeakyReLU()(norm1)\n\n zero_pad2 = layers.ZeroPadding2D()(leaky_relu) # (bs, 33, 33, 512)\n\n last = layers.Conv2D(1, 4, strides=1,\n kernel_initializer=initializer)(zero_pad2) # (bs, 30, 30, 1)\n\n return tf.keras.Model(inputs=inp, outputs=last)","metadata":{"execution":{"iopub.status.busy":"2024-03-12T02:58:28.988264Z","iopub.execute_input":"2024-03-12T02:58:28.988572Z","iopub.status.idle":"2024-03-12T02:58:29.002293Z","shell.execute_reply.started":"2024-03-12T02:58:28.988544Z","shell.execute_reply":"2024-03-12T02:58:29.001395Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n monet_generator = Generator() # transforms photos to Monet-esque paintings\n photo_generator = Generator() # transforms Monet paintings to be more like photos\n\n monet_discriminator = Discriminator() # differentiates real Monet paintings and generated Monet paintings\n photo_discriminator = Discriminator() # differentiates real photos and generated photos","metadata":{"execution":{"iopub.status.busy":"2024-03-12T03:02:15.001707Z","iopub.execute_input":"2024-03-12T03:02:15.002097Z","iopub.status.idle":"2024-03-12T03:02:15.966990Z","shell.execute_reply.started":"2024-03-12T03:02:15.002066Z","shell.execute_reply":"2024-03-12T03:02:15.966156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The image below correspond to a non-trained generator. The image is just random noise caused by the random initialization of the kernels around zero.","metadata":{}},{"cell_type":"code","source":"to_monet = monet_generator(example_photo)\n\nplt.subplot(1, 2, 1)\nplt.title(\"Original Photo\")\nplt.imshow(example_photo[0] * 0.5 + 0.5)\n\nplt.subplot(1, 2, 2)\nplt.title(\"Monet-esque Photo\")\nplt.imshow(to_monet[0] * 0.5 + 0.5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-03-12T03:02:17.243771Z","iopub.execute_input":"2024-03-12T03:02:17.244498Z","iopub.status.idle":"2024-03-12T03:02:17.732537Z","shell.execute_reply.started":"2024-03-12T03:02:17.244467Z","shell.execute_reply":"2024-03-12T03:02:17.731537Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### CycleGAN\n\nThe CycleGAN combines the previously defined architectures and apply loss functions to adjust the gradients","metadata":{}},{"cell_type":"code","source":"class CycleGan(keras.Model):\n def __init__(\n self,\n monet_generator,\n photo_generator,\n monet_discriminator,\n photo_discriminator,\n lambda_cycle=10,\n ):\n super(CycleGan, self).__init__()\n self.m_gen = monet_generator\n self.p_gen = photo_generator\n self.m_disc = monet_discriminator\n self.p_disc = photo_discriminator\n self.lambda_cycle = lambda_cycle\n \n def compile(\n self,\n m_gen_optimizer,\n p_gen_optimizer,\n m_disc_optimizer,\n p_disc_optimizer,\n gen_loss_fn,\n disc_loss_fn,\n cycle_loss_fn,\n identity_loss_fn\n ):\n super(CycleGan, self).compile()\n self.m_gen_optimizer = m_gen_optimizer\n self.p_gen_optimizer = p_gen_optimizer\n self.m_disc_optimizer = m_disc_optimizer\n self.p_disc_optimizer = p_disc_optimizer\n self.gen_loss_fn = gen_loss_fn\n self.disc_loss_fn = disc_loss_fn\n self.cycle_loss_fn = cycle_loss_fn\n self.identity_loss_fn = identity_loss_fn\n \n def train_step(self, batch_data):\n real_monet, real_photo = batch_data\n\n with tf.GradientTape(persistent=True) as tape:\n # Cycle photo to monet and back\n fake_monet = self.m_gen(real_photo, training=True)\n cycled_photo = self.p_gen(fake_monet, training=True)\n\n # Cycle monet to photo and back\n fake_photo = self.p_gen(real_monet, training=True)\n cycled_monet = self.m_gen(fake_photo, training=True)\n\n # Self generation\n same_monet = self.m_gen(real_monet, training=True)\n same_photo = self.p_gen(real_photo, training=True)\n\n # Discriminator for real images\n disc_real_monet = self.m_disc(real_monet, training=True)\n disc_real_photo = self.p_disc(real_photo, training=True)\n\n # Discriminator for fake images\n disc_fake_monet = self.m_disc(fake_monet, training=True)\n disc_fake_photo = self.p_disc(fake_photo, training=True)\n\n # Generator loss functions\n monet_gen_loss = self.gen_loss_fn(disc_fake_monet)\n photo_gen_loss = self.gen_loss_fn(disc_fake_photo)\n\n # Cycle loss function\n total_cycle_loss = self.cycle_loss_fn(real_monet, cycled_monet, self.lambda_cycle) + self.cycle_loss_fn(real_photo, cycled_photo, self.lambda_cycle)\n\n # Total loss combining past two\n total_monet_gen_loss = monet_gen_loss + total_cycle_loss + self.identity_loss_fn(real_monet, same_monet, self.lambda_cycle)\n total_photo_gen_loss = photo_gen_loss + total_cycle_loss + self.identity_loss_fn(real_photo, same_photo, self.lambda_cycle)\n\n # Discriminator loss\n monet_disc_loss = self.disc_loss_fn(disc_real_monet, disc_fake_monet)\n photo_disc_loss = self.disc_loss_fn(disc_real_photo, disc_fake_photo)\n\n # Calculate the gradients for generator and discriminator\n monet_generator_gradients = tape.gradient(total_monet_gen_loss,\n self.m_gen.trainable_variables)\n photo_generator_gradients = tape.gradient(total_photo_gen_loss,\n self.p_gen.trainable_variables)\n\n monet_discriminator_gradients = tape.gradient(monet_disc_loss,\n self.m_disc.trainable_variables)\n photo_discriminator_gradients = tape.gradient(photo_disc_loss,\n self.p_disc.trainable_variables)\n \n #Apply gradients to the generators and discriminators\n self.m_gen_optimizer.apply_gradients(zip(monet_generator_gradients,\n self.m_gen.trainable_variables))\n\n self.p_gen_optimizer.apply_gradients(zip(photo_generator_gradients,\n self.p_gen.trainable_variables))\n\n self.m_disc_optimizer.apply_gradients(zip(monet_discriminator_gradients,\n self.m_disc.trainable_variables))\n\n self.p_disc_optimizer.apply_gradients(zip(photo_discriminator_gradients,\n self.p_disc.trainable_variables))\n return {\n \"monet_gen_loss\": total_monet_gen_loss,\n \"photo_gen_loss\": total_photo_gen_loss,\n \"monet_disc_loss\": monet_disc_loss,\n \"photo_disc_loss\": photo_disc_loss\n }\n \n def call(self, real_photo):\n # Generate Monet styled photo\n fake_monet = self.m_gen(real_photo, training=False)\n return fake_monet","metadata":{"execution":{"iopub.status.busy":"2024-03-12T03:02:19.405127Z","iopub.execute_input":"2024-03-12T03:02:19.405492Z","iopub.status.idle":"2024-03-12T03:02:19.423363Z","shell.execute_reply.started":"2024-03-12T03:02:19.405461Z","shell.execute_reply":"2024-03-12T03:02:19.422440Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Loss Functions","metadata":{}},{"cell_type":"markdown","source":"#### Discriminator Loss\n\nThe discriminator loss function consists on:\n1. Calculate the real loss using binary crossentropy comparing the data with a ones vector.\n2. Calculte the generated loss using binary crossentropy comparing the data with a zeros vector.\n3. Get the total loss.\n4. Multiply by .5 to normalize the contribution of each loss.","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n def discriminator_loss(real, generated):\n real_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(real), real)\n\n generated_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.zeros_like(generated), generated)\n\n total_disc_loss = real_loss + generated_loss\n\n return total_disc_loss * 0.5","metadata":{"execution":{"iopub.status.busy":"2024-03-12T03:02:22.510731Z","iopub.execute_input":"2024-03-12T03:02:22.511072Z","iopub.status.idle":"2024-03-12T03:02:22.517577Z","shell.execute_reply.started":"2024-03-12T03:02:22.511047Z","shell.execute_reply":"2024-03-12T03:02:22.516631Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Generator Loss\n\nThe generator loss consist on:\n1. Calculte the generated loss using binary crossentropy comparing the data with a ones vector. ","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n def generator_loss(generated):\n return tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(generated), generated)","metadata":{"execution":{"iopub.status.busy":"2024-03-12T03:02:23.738990Z","iopub.execute_input":"2024-03-12T03:02:23.739368Z","iopub.status.idle":"2024-03-12T03:02:23.744782Z","shell.execute_reply.started":"2024-03-12T03:02:23.739337Z","shell.execute_reply":"2024-03-12T03:02:23.743785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Cycle Loss\n\nThe Cycle Loss consist of:\n1. Calculate the difference between the real image and the cycled image.\n2. Calculate the mean of the difference.\n3. Multiply by lambda.","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n def calc_cycle_loss(real_image, cycled_image, LAMBDA):\n loss1 = tf.reduce_mean(tf.abs(real_image - cycled_image))\n\n return LAMBDA * loss1","metadata":{"execution":{"iopub.status.busy":"2024-03-12T03:02:25.487273Z","iopub.execute_input":"2024-03-12T03:02:25.487662Z","iopub.status.idle":"2024-03-12T03:02:25.492685Z","shell.execute_reply.started":"2024-03-12T03:02:25.487630Z","shell.execute_reply":"2024-03-12T03:02:25.491759Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Identity Loss\n\nThe Identity Loss function takes as parameters a real image (It can be styled or unstyled) and a self-generated image (Doesn´t change style, it´s generated with the same generator).\n1. Calculate the absolute difference between real image and same image.\n2. Calculate the mean of the difference.\n3. Multiply by lambda an by .5.","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n def identity_loss(real_image, same_image, LAMBDA):\n loss = tf.reduce_mean(tf.abs(real_image - same_image))\n return LAMBDA * 0.5 * loss","metadata":{"execution":{"iopub.status.busy":"2024-03-12T03:02:27.596072Z","iopub.execute_input":"2024-03-12T03:02:27.596443Z","iopub.status.idle":"2024-03-12T03:02:27.601788Z","shell.execute_reply.started":"2024-03-12T03:02:27.596412Z","shell.execute_reply":"2024-03-12T03:02:27.600650Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Optimizers\n\nThe optimizer used in this model is Adam with a beta of .5 and an initial learning rate of 2e-4.","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n monet_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n photo_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n\n monet_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n photo_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)","metadata":{"execution":{"iopub.status.busy":"2024-03-12T03:02:28.983722Z","iopub.execute_input":"2024-03-12T03:02:28.984783Z","iopub.status.idle":"2024-03-12T03:02:28.997421Z","shell.execute_reply.started":"2024-03-12T03:02:28.984736Z","shell.execute_reply":"2024-03-12T03:02:28.996499Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model declaration and compilation","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n cycle_gan_model = CycleGan(\n monet_generator, photo_generator, monet_discriminator, photo_discriminator\n )\n\n cycle_gan_model.compile(\n m_gen_optimizer = monet_generator_optimizer,\n p_gen_optimizer = photo_generator_optimizer,\n m_disc_optimizer = monet_discriminator_optimizer,\n p_disc_optimizer = photo_discriminator_optimizer,\n gen_loss_fn = generator_loss,\n disc_loss_fn = discriminator_loss,\n cycle_loss_fn = calc_cycle_loss,\n identity_loss_fn = identity_loss\n )\n","metadata":{"execution":{"iopub.status.busy":"2024-03-12T03:02:30.543464Z","iopub.execute_input":"2024-03-12T03:02:30.544339Z","iopub.status.idle":"2024-03-12T03:02:30.555917Z","shell.execute_reply.started":"2024-03-12T03:02:30.544304Z","shell.execute_reply":"2024-03-12T03:02:30.554939Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Model Training","metadata":{}},{"cell_type":"markdown","source":"#### Personalized Callback\n\nThe Callback declared below has two main functions:\n1. Save images on each epoch of training.\n2. Save the progress of the weights every five epochs.","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras import callbacks\ndef tensor_to_image(tensor):\n return tf.keras.preprocessing.image.array_to_img(tensor[0])\n\nclass SaveOutputImages(callbacks.Callback):\n def __init__(self, generator_model, input_data):\n super(SaveOutputImages, self).__init__()\n self.generator_model = generator_model\n self.input_data = input_data\n self.epoch_images = [example_photo]\n\n def on_epoch_end(self, epoch, logs=None):\n generated_images = self.generator_model.predict(self.input_data)\n\n epoch_image = tensor_to_image(generated_images)\n self.epoch_images.append(epoch_image)\n if epoch % 5 == 0:\n self.generator_model.m_gen.save_weights(f\"model_{epoch}.weights.h5\")\n\nwith strategy.scope():\n output_saver = SaveOutputImages(generator_model=cycle_gan_model, input_data=example_photo)\n #cycle_gan_model.load_weights(\"/kaggle/input/epoch14gam/model_14.weights.h5\")","metadata":{"execution":{"iopub.status.busy":"2024-03-12T03:02:32.816934Z","iopub.execute_input":"2024-03-12T03:02:32.817301Z","iopub.status.idle":"2024-03-12T03:02:32.825490Z","shell.execute_reply.started":"2024-03-12T03:02:32.817272Z","shell.execute_reply":"2024-03-12T03:02:32.824357Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Training Process\nThe model is trained in 25 epochs using the output_saver callback.","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n history = cycle_gan_model.fit(\n tf.data.Dataset.zip((monet_ds, photo_ds)),\n epochs=25,\n callbacks=[output_saver]\n )","metadata":{"execution":{"iopub.status.busy":"2024-03-12T03:02:35.705420Z","iopub.execute_input":"2024-03-12T03:02:35.706321Z","iopub.status.idle":"2024-03-12T03:30:18.785503Z","shell.execute_reply.started":"2024-03-12T03:02:35.706284Z","shell.execute_reply":"2024-03-12T03:30:18.784406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Results\n\nThe train of the model is visible on the images below, which contains the progress of the generated images every five epochs and the original image.","metadata":{}},{"cell_type":"code","source":"axis = 231\nplt.figure(figsize=(10, 5))\nplt.subplots_adjust(hspace=.5, wspace = .2)\nplt.subplot(axis)\nplt.title('Original')\nplt.imshow(output_saver.epoch_images[0][0]*.5 + .5)\n\nfor i in range(5, 26, 5):\n try:\n plt.subplot(axis + int(i/5))\n plt.title(f\"Epoch {i}\")\n plt.imshow(output_saver.epoch_images[i])\n except:\n print(f\"No epoch {i}\")","metadata":{"execution":{"iopub.status.busy":"2024-03-12T03:32:26.602828Z","iopub.execute_input":"2024-03-12T03:32:26.603578Z","iopub.status.idle":"2024-03-12T03:32:27.659449Z","shell.execute_reply.started":"2024-03-12T03:32:26.603547Z","shell.execute_reply":"2024-03-12T03:32:27.658574Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Model Evolution\n\nThe gif above shows the evolution of the generator. And how the model learns to transfer an image from one domain to another. We can observe a significant improvement of the model performance on each epoch. Aditionally, we can observe a notorious change of the color and texture of the photo, looking more like the monet style.","metadata":{}},{"cell_type":"code","source":"import imageio\nimport numpy as np\n\ngif_path = '/kaggle/working/animated.gif'\nimages_gif = [np.uint8(output_saver.epoch_images[0][0]*125.5+125.5)] + [np.uint8(image) for image in output_saver.epoch_images[1:]]\nimageio.mimsave(gif_path, images_gif, duration=1000)\n\nfrom IPython.display import Image, display\ndisplay(Image(filename=gif_path))","metadata":{"execution":{"iopub.status.busy":"2024-03-12T03:38:39.762821Z","iopub.execute_input":"2024-03-12T03:38:39.763178Z","iopub.status.idle":"2024-03-12T03:38:42.984091Z","shell.execute_reply.started":"2024-03-12T03:38:39.763151Z","shell.execute_reply":"2024-03-12T03:38:42.981168Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Submission File Creation\n\nCreate a zip with the generated images.","metadata":{}},{"cell_type":"code","source":"! mkdir ../images","metadata":{"execution":{"iopub.status.busy":"2024-03-12T03:38:52.931642Z","iopub.execute_input":"2024-03-12T03:38:52.932446Z","iopub.status.idle":"2024-03-12T03:38:54.009257Z","shell.execute_reply.started":"2024-03-12T03:38:52.932413Z","shell.execute_reply":"2024-03-12T03:38:54.008275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import numpy as np\nimport PIL\nimport shutil\n\ni = 1\nfor img in photo_ds:\n prediction = monet_generator(img, training=False)[0].numpy()\n prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n im = PIL.Image.fromarray(prediction)\n im.save(\"../images/\" + str(i) + \".jpg\")\n i += 1\n \nshutil.make_archive(\"/kaggle/working/images\", 'zip', \"/kaggle/images\")","metadata":{"execution":{"iopub.status.busy":"2024-03-12T03:39:00.026922Z","iopub.execute_input":"2024-03-12T03:39:00.028059Z","iopub.status.idle":"2024-03-12T03:45:00.050686Z","shell.execute_reply.started":"2024-03-12T03:39:00.028010Z","shell.execute_reply":"2024-03-12T03:45:00.049666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Conclusions\n\nThe objective of this document was satisfactorily fulfilled, obtaining a generator of monet style images from scratch. Applying machine learning models and concepts like Generative Adversarial Networks. This is a field with a great importance nowadays, the generative AI is in constant growth.\n\nOne possible improvement for future projects is expand the dataset by using images of higher quality and sectioning the images in 256x256 chunks. Applyin this generator we could generate higher quality images using the same generator, applying it all over the image divided in chuncks, concatenating it just after.\n\nThis projects improves my hability to construct neural networks solutions from scratch for different purposes. This Neural Network will be deployed to a web app using cloud computing as the final stage of this project.","metadata":{}},{"cell_type":"markdown","source":"**Bibliography**\n\nClaude Monet. (2024). Retrieved March 10, 2024, from Nga.gov website: https://www.nga.gov/features/slideshows/claude-monet.html#slide_2\n\nEl generador. (2022). Retrieved March 10, 2024, from Google for Developers website: https://developers.google.com/machine-learning/gan/generator?hl=es-419\n\namyjang. (2020, August 29). Monet CycleGAN Tutorial. Retrieved March 10, 2024, from Kaggle.com website: https://www.kaggle.com/code/amyjang/monet-cyclegan-tutorial","metadata":{}}]}