\n\n\n### This notebook was created to try several experiments to the original CycleGan (or at least a vey close one) implementation, and see what works or not for this specific task.\n\n#### Experiments (Starting from the original paper architecture):\n- Transformer with residual blocks [++] \n- Residual connections between Generator and Discriminator [++]\n- Not using `InstanceNorm` at the first layer of both generator and discriminator [++]\n- Better `InstanceNorm` layer initialization [++]\n- Training a lot longer [++]\n- Better `Conv` layer initialization [+]\n- Residual connection with `Concatenate` instead of `Add` [+]\n- Data augmentations (flips, rotations, and crops) [+]\n- Discriminator with label smoothing [+]\n- Using [external data](https://www.kaggle.com/dimitreoliveira/tfrecords-monet-paintings-256x256) (1193 files) [+-]\n- Train on crops [+-] \n- Decoder with resize-convolution [+-]\n- 9 transformer blocks [+-] \n- Patch discriminator [+-] \n- Lager batch size [+-] \n\n\n#### Some references\n- [My previous work with a CycleGAN introduction and baseline](https://www.kaggle.com/dimitreoliveira/introduction-to-cyclegan-monet-paintings)\n- [CycleGAN homepage](https://junyanz.github.io/CycleGAN/)\n- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks - Original paper](https://arxiv.org/pdf/1703.10593.pdf)\n- [CycleGAN: Learning to Translate Images (Without Paired Training Data)](https://towardsdatascience.com/cyclegan-learning-to-translate-images-without-paired-training-data-5b4e93862c8d)\n- [One nice implementation of TF CycleGAN](https://github.com/architrathore/CycleGAN)\n- [How to Develop a CycleGAN for Image-to-Image Translation with Keras](https://machinelearningmastery.com/cyclegan-tutorial-with-keras/)\n- [How to Implement GAN Hacks in Keras to Train Stable Models](https://machinelearningmastery.com/how-to-code-generative-adversarial-network-hacks/)\n- [How to Train a GAN? Tips and tricks to make GANs work](https://github.com/soumith/ganhacks)\n- [Deconvolution and Checkerboard Artifacts](https://distill.pub/2016/deconv-checkerboard/)\n- [Keras implementation of CycleGAN](https://keras.io/examples/generative/cyclegan/)\n- [CycleGAN with Better Cycles](https://ssnl.github.io/better_cycles/report.pdf)","metadata":{}},{"cell_type":"markdown","source":"## Dependencies","metadata":{}},{"cell_type":"code","source":"import os, random, json, PIL, shutil, re, imageio, glob\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nfrom PIL import ImageDraw\nimport matplotlib.pyplot as plt\nfrom kaggle_datasets import KaggleDatasets\nimport tensorflow as tf\nimport tensorflow.keras.layers as L\nimport tensorflow.keras.backend as K\nimport tensorflow_addons as tfa\nfrom tensorflow.keras import Model, losses, optimizers\nfrom tensorflow.keras.callbacks import Callback\nimport cv2\n","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-06-10T12:27:19.044122Z","iopub.execute_input":"2024-06-10T12:27:19.044459Z","iopub.status.idle":"2024-06-10T12:27:25.795636Z","shell.execute_reply.started":"2024-06-10T12:27:19.044385Z","shell.execute_reply":"2024-06-10T12:27:25.794817Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!python --version","metadata":{"execution":{"iopub.status.busy":"2024-06-07T13:09:53.798768Z","iopub.execute_input":"2024-06-07T13:09:53.799759Z","iopub.status.idle":"2024-06-07T13:09:54.76462Z","shell.execute_reply.started":"2024-06-07T13:09:53.799722Z","shell.execute_reply":"2024-06-07T13:09:54.763537Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pip show tensorflow","metadata":{"execution":{"iopub.status.busy":"2024-06-07T13:09:57.561772Z","iopub.execute_input":"2024-06-07T13:09:57.562732Z","iopub.status.idle":"2024-06-07T13:10:09.378188Z","shell.execute_reply.started":"2024-06-07T13:09:57.56268Z","shell.execute_reply":"2024-06-07T13:10:09.376817Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model parameters","metadata":{}},{"cell_type":"code","source":"HEIGHT = 256\nWIDTH = 256\nHEIGHT_RESIZE = 128\nWIDTH_RESIZE = 128\nCHANNELS = 3\nBATCH_SIZE = 16\nEPOCHS = 9\nTRANSFORMER_BLOCKS = 6\nGENERATOR_LR = 2e-4\nDISCRIMINATOR_LR = 2e-4","metadata":{"_kg_hide-input":false,"execution":{"iopub.status.busy":"2024-06-10T12:27:47.621257Z","iopub.execute_input":"2024-06-10T12:27:47.62225Z","iopub.status.idle":"2024-06-10T12:27:47.627539Z","shell.execute_reply.started":"2024-06-10T12:27:47.622212Z","shell.execute_reply":"2024-06-10T12:27:47.626569Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load data","metadata":{}},{"cell_type":"code","source":"def get_data(filenames, augment=None, repeat=True, shuffle=True, batch_size=1):\n image=cv2.imread(filenames)\n image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n image=cv2.resize(image,(HEIGHT,WIDTH))\n image=tf.convert_to_tensor(image, dtype=tf.float32)\n image = tf.reshape(image, [1,HEIGHT, WIDTH, CHANNELS])\n img = tf.cast(image, dtype=tf.float32)\n# print(img.shape)\n img = data_augment(img[0])\n# print(type(img))\n return (img / 127.5) - 1.0","metadata":{"execution":{"iopub.status.busy":"2024-06-10T16:12:14.570438Z","iopub.execute_input":"2024-06-10T16:12:14.571191Z","iopub.status.idle":"2024-06-10T16:12:14.578622Z","shell.execute_reply.started":"2024-06-10T16:12:14.571157Z","shell.execute_reply":"2024-06-10T16:12:14.577609Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"filepath=\"/kaggle/input/japanart\"","metadata":{"execution":{"iopub.status.busy":"2024-06-10T16:12:19.213478Z","iopub.execute_input":"2024-06-10T16:12:19.21384Z","iopub.status.idle":"2024-06-10T16:12:19.218513Z","shell.execute_reply.started":"2024-06-10T16:12:19.21381Z","shell.execute_reply":"2024-06-10T16:12:19.217446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import tensorflow as tf\n# import os\n\n# def _bytes_feature(value):\n# \"\"\"Returns a bytes_list from a string / byte.\"\"\"\n# return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))\n\n# def _int64_feature(value):\n# \"\"\"Returns an int64_list from a bool / enum / int / uint.\"\"\"\n# return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))\n\n# def image_example(image_string):\n# image_shape = tf.image.decode_jpeg(image_string).shape\n# feature = {\n# 'height': _int64_feature(image_shape[0]),\n# 'width': _int64_feature(image_shape[1]),\n# 'depth': _int64_feature(image_shape[2]),\n# 'image_raw': _bytes_feature(image_string),\n# }\n# return tf.train.Example(features=tf.train.Features(feature=feature))\n\n# def convert_images_to_tfrecords(image_folder, output_file):\n# writer = tf.io.TFRecordWriter(output_file)\n\n# for image_name in os.listdir(image_folder):\n# image_path = os.path.join(image_folder, image_name)\n# if os.path.isfile(image_path): # Ensure it's a file\n# try:\n# image_string = open(image_path, 'rb').read()\n# tf_example = image_example(image_string)\n# writer.write(tf_example.SerializeToString())\n# except Exception as e:\n# print(f\"Error processing file {image_path}: {e}\")\n\n# writer.close()\n\n# # Example usage:\n# image_folder = '/kaggle/input/japanart'\n# output_file = '/kaggle/working/japan.tfrec'\n# convert_images_to_tfrecords(image_folder, output_file)","metadata":{"execution":{"iopub.status.busy":"2024-06-10T14:36:29.761332Z","iopub.execute_input":"2024-06-10T14:36:29.762102Z","iopub.status.idle":"2024-06-10T14:37:13.997145Z","shell.execute_reply.started":"2024-06-10T14:36:29.762064Z","shell.execute_reply":"2024-06-10T14:37:13.99605Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"l=[]\nc=0\nfor file in os.listdir(filepath):\n im=cv2.imread(filepath+\"/\"+file)\n c+=1\n img=get_data(filepath+\"/\"+file)\n print(c)\n l.append(img)","metadata":{"execution":{"iopub.status.busy":"2024-06-10T16:12:24.179113Z","iopub.execute_input":"2024-06-10T16:12:24.179996Z","iopub.status.idle":"2024-06-10T16:15:26.798087Z","shell.execute_reply.started":"2024-06-10T16:12:24.179956Z","shell.execute_reply":"2024-06-10T16:15:26.797113Z"},"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(l)","metadata":{"execution":{"iopub.status.busy":"2024-06-10T16:16:23.261859Z","iopub.execute_input":"2024-06-10T16:16:23.26225Z","iopub.status.idle":"2024-06-10T16:16:23.268513Z","shell.execute_reply.started":"2024-06-10T16:16:23.262216Z","shell.execute_reply":"2024-06-10T16:16:23.267635Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"GCS_PATH = KaggleDatasets().get_gcs_path('gan-getting-started')\n\n# PHOTO_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/photo_jpg/*.jpg'))\nPHOTO_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/photo_tfrec/photo*.tfrec'))","metadata":{"execution":{"iopub.status.busy":"2024-06-10T16:08:10.732784Z","iopub.execute_input":"2024-06-10T16:08:10.733165Z","iopub.status.idle":"2024-06-10T16:08:11.231146Z","shell.execute_reply.started":"2024-06-10T16:08:10.733134Z","shell.execute_reply":"2024-06-10T16:08:11.230077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(PHOTO_FILENAMES)\n# GCS_PATH","metadata":{"execution":{"iopub.status.busy":"2024-06-10T16:09:44.784222Z","iopub.execute_input":"2024-06-10T16:09:44.784595Z","iopub.status.idle":"2024-06-10T16:09:44.790584Z","shell.execute_reply.started":"2024-06-10T16:09:44.784562Z","shell.execute_reply":"2024-06-10T16:09:44.789705Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"/kaggle/input/japanart/MET-36461.jpg","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"JAPANART_FILENAMES=tf.io.gfile.glob('/kaggle/working' + '/japan.tfrec')","metadata":{"execution":{"iopub.status.busy":"2024-06-10T14:37:37.82355Z","iopub.execute_input":"2024-06-10T14:37:37.824495Z","iopub.status.idle":"2024-06-10T14:37:37.829169Z","shell.execute_reply.started":"2024-06-10T14:37:37.824455Z","shell.execute_reply":"2024-06-10T14:37:37.828201Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"len(JAPANART_FILENAMES)","metadata":{"execution":{"iopub.status.busy":"2024-06-10T14:37:41.946796Z","iopub.execute_input":"2024-06-10T14:37:41.947171Z","iopub.status.idle":"2024-06-10T14:37:41.953695Z","shell.execute_reply.started":"2024-06-10T14:37:41.947141Z","shell.execute_reply":"2024-06-10T14:37:41.952732Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"GCS_PATH = KaggleDatasets().get_gcs_path('gan-getting-started')\n# EXT_PATH = KaggleDatasets().get_gcs_path('tfrecords-monet-paintings-256x256')\n# EXT_PATH = KaggleDatasets().get_gcs_path('monet-tfrecords-256x256')\n\nMONET_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/monet_tfrec/monet*.tfrec'))\nPHOTO_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/photo_tfrec/photo*.tfrec'))\n# MONET_FILENAMES = tf.io.gfile.glob(str(EXT_PATH + '/monet*.tfrec'))\n\ndef count_data_items(filenames):\n n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n return np.sum(n)\n\nn_monet_samples = count_data_items(MONET_FILENAMES)\nn_photo_samples = count_data_items(PHOTO_FILENAMES)\n\nprint(f'Monet TFRecord files: {len(MONET_FILENAMES)}')\nprint(f'Monet image files: {n_monet_samples}')\nprint(f'Photo TFRecord files: {len(PHOTO_FILENAMES)}')\nprint(f'Photo image files: {n_photo_samples}')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-06-10T16:41:00.547957Z","iopub.execute_input":"2024-06-10T16:41:00.548403Z","iopub.status.idle":"2024-06-10T16:41:01.158093Z","shell.execute_reply.started":"2024-06-10T16:41:00.548371Z","shell.execute_reply":"2024-06-10T16:41:01.157074Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Augmentations\n\nData augmentation for GANs should be done very carefully, especially for tasks similar to style transfer, if we apply transformations that can change too much the style of the data (e.g. brightness, contrast, saturation) it can cause the generator to do not efficiently learn the base style, so in this case, we are using only spatial transformations like, flips, rotates and crops.","metadata":{}},{"cell_type":"code","source":"def data_augment(image):\n p_spatial = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_rotate = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n\n # 90º rotations\n if p_rotate > .8:\n image = tf.image.rot90(image, k=3) # rotate 270º\n elif p_rotate > .6:\n image = tf.image.rot90(image, k=2) # rotate 180º\n elif p_rotate > .4:\n image = tf.image.rot90(image, k=1) # rotate 90º\n \n # Flips\n image = tf.image.random_flip_left_right(image)\n image = tf.image.random_flip_up_down(image)\n if p_spatial > .75:\n image = tf.image.transpose(image)\n \n\n # Train on crops\n image = tf.image.random_crop(image, size=[HEIGHT_RESIZE, WIDTH_RESIZE, CHANNELS])\n \n \n return image","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-06-10T16:09:49.025281Z","iopub.execute_input":"2024-06-10T16:09:49.025653Z","iopub.status.idle":"2024-06-10T16:09:49.035209Z","shell.execute_reply.started":"2024-06-10T16:09:49.025619Z","shell.execute_reply":"2024-06-10T16:09:49.034192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Auxiliar functions","metadata":{}},{"cell_type":"code","source":"def normalize_img(img):\n img = tf.cast(img, dtype=tf.float32)\n # Map values in the range [-1, 1]\n return (img / 127.5) - 1.0\n\ndef decode_image(image):\n image = tf.image.decode_jpeg(image, channels=CHANNELS)\n image = tf.reshape(image, [HEIGHT, WIDTH, CHANNELS])\n return image\n\ndef read_tfrecord(example):\n tfrecord_format = {\n 'image': tf.io.FixedLenFeature([], tf.string)\n }\n example = tf.io.parse_single_example(example, tfrecord_format)\n image = decode_image(example['image'])\n return image\n\ndef load_dataset(filenames):\n dataset = tf.data.TFRecordDataset(filenames)\n dataset = dataset.map(read_tfrecord)\n return dataset\n\ndef get_dataset(filenames, augment=None, repeat=True, shuffle=True, batch_size=1):\n dataset = load_dataset(filenames)\n\n if augment:\n dataset = dataset.map(augment)\n dataset = dataset.map(normalize_img) \n if repeat:\n dataset = dataset.repeat()\n if shuffle:\n dataset = dataset.shuffle(512)\n \n dataset = dataset.batch(batch_size)\n dataset = dataset.cache()\n \n \n return dataset\n\ndef display_samples(ds, n_samples):\n ds_iter = iter(ds)\n for n_sample in range(n_samples):\n example_sample = next(ds_iter)\n plt.subplot(121)\n plt.imshow(example_sample[0] * 0.5 + 0.5)\n plt.axis('off')\n plt.show()\n \ndef display_generated_samples(ds, model, n_samples):\n ds_iter = iter(ds)\n for n_sample in range(n_samples):\n example_sample = next(ds_iter)\n generated_sample = model.predict(example_sample)\n \n f = plt.figure(figsize=(12, 12))\n \n plt.subplot(121)\n plt.title('Input image')\n plt.imshow(example_sample[0] * 0.5 + 0.5)\n plt.axis('off')\n \n plt.subplot(122)\n plt.title('Generated image')\n plt.imshow(generated_sample[0] * 0.5 + 0.5)\n plt.axis('off')\n plt.show()\n \ndef evaluate_cycle(ds, generator_a, generator_b, n_samples=1):\n fig, axes = plt.subplots(n_samples, 3, figsize=(22, (n_samples*6)))\n axes = axes.flatten()\n \n ds_iter = iter(ds)\n for n_sample in range(n_samples):\n idx = n_sample*3\n example_sample = next(ds_iter)\n generated_a_sample = generator_a.predict(example_sample)\n generated_b_sample = generator_b.predict(generated_a_sample)\n \n axes[idx].set_title('Input image', fontsize=18)\n axes[idx].imshow(example_sample[0] * 0.5 + 0.5)\n axes[idx].axis('off')\n \n axes[idx+1].set_title('Generated image', fontsize=18)\n axes[idx+1].imshow(generated_a_sample[0] * 0.5 + 0.5)\n axes[idx+1].axis('off')\n \n axes[idx+2].set_title('Cycled image', fontsize=18)\n axes[idx+2].imshow(generated_b_sample[0] * 0.5 + 0.5)\n axes[idx+2].axis('off')\n \n plt.show()\n \ndef predict_and_save(input_ds, generator_model, output_path):\n i = 1\n for img in input_ds:\n prediction = generator_model(img, training=False)[0].numpy() # make predition\n prediction = (prediction * 127.5 + 127.5).astype(np.uint8) # re-scale\n im = PIL.Image.fromarray(prediction)\n im.save(f'{output_path}{str(i)}.jpg')\n i += 1","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-06-10T16:09:51.772813Z","iopub.execute_input":"2024-06-10T16:09:51.773199Z","iopub.status.idle":"2024-06-10T16:09:51.797931Z","shell.execute_reply.started":"2024-06-10T16:09:51.773164Z","shell.execute_reply":"2024-06-10T16:09:51.796908Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"photo_ds = get_dataset(PHOTO_FILENAMES, augment=data_augment, batch_size=BATCH_SIZE)\n# art_dataset=get_dataset(JAPANART_FILENAMES, augment=data_augment, batch_size=BATCH_SIZE)\nart_dataset = tf.data.Dataset.from_tensor_slices(l)\nart_ds=art_dataset.batch(1)\nart_ds = art_ds.cache()\n# art_ds=art_ds.map(data_augment)\n# len(photo_ds)\n# len(art_ds)","metadata":{"execution":{"iopub.status.busy":"2024-06-10T16:30:59.281159Z","iopub.execute_input":"2024-06-10T16:30:59.281547Z","iopub.status.idle":"2024-06-10T16:30:59.625184Z","shell.execute_reply.started":"2024-06-10T16:30:59.281515Z","shell.execute_reply":"2024-06-10T16:30:59.624349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import tensorflow as tf\n\n# # Path to your TFRecord file\n# tfrecord_file = '/kaggle/working/japan.tfrec'\n\n# # Define a function to parse a TFRecord example\n# def parse_tfrecord_fn(example):\n# feature_description = {\n# 'height': tf.io.FixedLenFeature([], tf.int64),\n# 'width': tf.io.FixedLenFeature([], tf.int64),\n# 'depth': tf.io.FixedLenFeature([], tf.int64),\n# 'image_raw': tf.io.FixedLenFeature([], tf.string),\n# }\n# return tf.io.parse_single_example(example, feature_description)\n\n# # Read the TFRecord file\n# raw_dataset = tf.data.TFRecordDataset(tfrecord_file)\n\n# # Parse the TFRecord examples\n# parsed_dataset = raw_dataset.map(parse_tfrecord_fn)\n\n# # Iterate over the dataset and print the contents\n# for parsed_example in parsed_dataset.take(1): # Adjust 5 to the number of examples you want to inspect\n# print(parsed_example)\n","metadata":{"execution":{"iopub.status.busy":"2024-06-10T16:31:03.529957Z","iopub.execute_input":"2024-06-10T16:31:03.530339Z","iopub.status.idle":"2024-06-10T16:31:03.535695Z","shell.execute_reply.started":"2024-06-10T16:31:03.530307Z","shell.execute_reply":"2024-06-10T16:31:03.534634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"photo_ds","metadata":{"execution":{"iopub.status.busy":"2024-06-10T16:31:03.927831Z","iopub.execute_input":"2024-06-10T16:31:03.928768Z","iopub.status.idle":"2024-06-10T16:31:03.93491Z","shell.execute_reply.started":"2024-06-10T16:31:03.928729Z","shell.execute_reply":"2024-06-10T16:31:03.933879Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"art_ds","metadata":{"execution":{"iopub.status.busy":"2024-06-10T16:31:04.94464Z","iopub.execute_input":"2024-06-10T16:31:04.945029Z","iopub.status.idle":"2024-06-10T16:31:04.951601Z","shell.execute_reply.started":"2024-06-10T16:31:04.944996Z","shell.execute_reply":"2024-06-10T16:31:04.950538Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"gan_ds = tf.data.Dataset.zip((art_ds, photo_ds))","metadata":{"execution":{"iopub.status.busy":"2024-06-10T16:31:08.565103Z","iopub.execute_input":"2024-06-10T16:31:08.565453Z","iopub.status.idle":"2024-06-10T16:31:08.570934Z","shell.execute_reply.started":"2024-06-10T16:31:08.565424Z","shell.execute_reply":"2024-06-10T16:31:08.56997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Auxiliar functions (model)\n\nHere we the building blocks of our models:\n- Encoder block: Apply convolutional filters while also reducing data resolution and increasing features.\n- Decoder block: Apply convolutional filters while also increasing data resolution and decreasing features.\n- Transformer block: Apply convolutional filters to find relevant data patterns and keeps features constant.","metadata":{}},{"cell_type":"code","source":"conv_initializer = tf.random_normal_initializer(mean=0.0, stddev=0.02)\ngamma_initializer = tf.keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n \ndef encoder_block(input_layer, filters, size=3, strides=2, apply_instancenorm=True, activation=L.ReLU(), name='block_x'):\n block = L.Conv2D(filters, size, \n strides=strides, \n padding='same', \n use_bias=False, \n kernel_initializer=conv_initializer, \n name=f'encoder_{name}')(input_layer)\n\n if apply_instancenorm:\n block = tfa.layers.InstanceNormalization(gamma_initializer=gamma_initializer)(block)\n \n block = activation(block)\n\n return block\n\ndef transformer_block(input_layer, size=3, strides=1, name='block_x'):\n filters = input_layer.shape[-1]\n \n block = L.Conv2D(filters, size, strides=strides, padding='same', use_bias=False, \n kernel_initializer=conv_initializer, name=f'transformer_{name}_1')(input_layer)\n\n block = L.ReLU()(block)\n \n block = L.Conv2D(filters, size, strides=strides, padding='same', use_bias=False, \n kernel_initializer=conv_initializer, name=f'transformer_{name}_2')(block)\n \n block = L.Add()([block, input_layer])\n\n return block\n\ndef decoder_block(input_layer, filters, size=3, strides=2, apply_instancenorm=True, name='block_x'):\n block = L.Conv2DTranspose(filters, size, \n strides=strides, \n padding='same', \n use_bias=False, \n kernel_initializer=conv_initializer, \n name=f'decoder_{name}')(input_layer)\n\n if apply_instancenorm:\n block = tfa.layers.InstanceNormalization(gamma_initializer=gamma_initializer)(block)\n\n block = L.ReLU()(block)\n \n return block\n\n# Resized convolution\ndef decoder_rc_block(input_layer, filters, size=3, strides=1, apply_instancenorm=True, name='block_x'):\n block = tf.image.resize(images=input_layer, method='bilinear', \n size=(input_layer.shape[1]*2, input_layer.shape[2]*2))\n \n\n block = L.Conv2D(filters, size, \n strides=strides, \n padding='same', \n use_bias=False, \n kernel_initializer=conv_initializer, \n name=f'decoder_{name}')(block)\n\n if apply_instancenorm:\n block = tfa.layers.InstanceNormalization(gamma_initializer=gamma_initializer)(block)\n\n block = L.ReLU()(block)\n \n return block","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-06-10T16:41:36.652862Z","iopub.execute_input":"2024-06-10T16:41:36.653777Z","iopub.status.idle":"2024-06-10T16:41:36.672065Z","shell.execute_reply.started":"2024-06-10T16:41:36.653736Z","shell.execute_reply":"2024-06-10T16:41:36.670933Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Generator model\n\nThe `generator` is responsible for generating images from a specific domain. `CycleGAN` architecture has two generators, in this context we will have one `generator` that will take `photos` and generate `Monet paints`, and the other `generator` will take `Monet paintings` and generate `photos`.\n\nBellow, we have the architecture of the original `CycleGAN` `generator`, ours have some changes to improve performance on this task.\n\n
","metadata":{}},{"cell_type":"code","source":"def generator_fn(height=HEIGHT, width=WIDTH, channels=CHANNELS, transformer_blocks=TRANSFORMER_BLOCKS):\n OUTPUT_CHANNELS = 3\n inputs = L.Input(shape=[height, width, channels], name='input_image')\n\n # Encoder\n enc_1 = encoder_block(inputs, 64, 7, 1, apply_instancenorm=False, activation=L.ReLU(), name='block_1') # (bs, 256, 256, 64)\n enc_2 = encoder_block(enc_1, 128, 3, 2, apply_instancenorm=True, activation=L.ReLU(), name='block_2') # (bs, 128, 128, 128)\n enc_3 = encoder_block(enc_2, 256, 3, 2, apply_instancenorm=True, activation=L.ReLU(), name='block_3') # (bs, 64, 64, 256)\n \n # Transformer\n x = enc_3\n for n in range(transformer_blocks):\n x = transformer_block(x, 3, 1, name=f'block_{n+1}') # (bs, 64, 64, 256)\n\n # Decoder\n x_skip = L.Concatenate(name='enc_dec_skip_1')([x, enc_3]) # encoder - decoder skip connection\n \n dec_1 = decoder_block(x_skip, 128, 3, 2, apply_instancenorm=True, name='block_1') # (bs, 128, 128, 128)\n x_skip = L.Concatenate(name='enc_dec_skip_2')([dec_1, enc_2]) # encoder - decoder skip connection\n \n dec_2 = decoder_block(x_skip, 64, 3, 2, apply_instancenorm=True, name='block_2') # (bs, 256, 256, 64)\n x_skip = L.Concatenate(name='enc_dec_skip_3')([dec_2, enc_1]) # encoder - decoder skip connection\n\n outputs = last = L.Conv2D(OUTPUT_CHANNELS, 7, \n strides=1, padding='same', \n kernel_initializer=conv_initializer, \n use_bias=False, \n activation='tanh', \n name='decoder_output_block')(x_skip) # (bs, 256, 256, 3)\n\n generator = Model(inputs, outputs)\n \n return generator\n\nsample_generator = generator_fn()\nsample_generator.summary()","metadata":{"_kg_hide-input":false,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-06-10T16:41:40.301813Z","iopub.execute_input":"2024-06-10T16:41:40.302217Z","iopub.status.idle":"2024-06-10T16:41:40.568654Z","shell.execute_reply.started":"2024-06-10T16:41:40.302181Z","shell.execute_reply":"2024-06-10T16:41:40.567712Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Discriminator model\n\n\nThe `discriminator` is responsible for differentiating real images from images that have been generated by a `generator` model.\n\nBellow, we have the architecture of the original `CycleGAN` `discriminator`, again, ours have some changes to improve performance on this task.\n\n
","metadata":{}},{"cell_type":"code","source":"def discriminator_fn(height=HEIGHT, width=WIDTH, channels=CHANNELS):\n inputs = L.Input(shape=[height, width, channels], name='input_image')\n \n # Encoder \n x = encoder_block(inputs, 64, 4, 2, apply_instancenorm=False, activation=L.LeakyReLU(0.2), name='block_1') # (bs, 128, 128, 64)\n x = encoder_block(x, 128, 4, 2, apply_instancenorm=True, activation=L.LeakyReLU(0.2), name='block_2') # (bs, 64, 64, 128)\n x = encoder_block(x, 256, 4, 2, apply_instancenorm=True, activation=L.LeakyReLU(0.2), name='block_3') # (bs, 32, 32, 256)\n x = encoder_block(x, 512, 4, 1, apply_instancenorm=True, activation=L.LeakyReLU(0.2), name='block_4') # (bs, 32, 32, 512)\n\n outputs = L.Conv2D(1, 4, strides=1, padding='valid', kernel_initializer=conv_initializer)(x) # (bs, 29, 29, 1)\n \n discriminator = Model(inputs, outputs)\n \n return discriminator\n\n\nsample_discriminator = discriminator_fn()\nsample_discriminator.summary()","metadata":{"_kg_hide-input":false,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-06-10T16:41:41.384246Z","iopub.execute_input":"2024-06-10T16:41:41.38516Z","iopub.status.idle":"2024-06-10T16:41:41.513815Z","shell.execute_reply.started":"2024-06-10T16:41:41.385119Z","shell.execute_reply":"2024-06-10T16:41:41.512826Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Build model (CycleGAN)","metadata":{}},{"cell_type":"code","source":"# with strategy.scope():\nmonet_generator = generator_fn(height=None, width=None, transformer_blocks=TRANSFORMER_BLOCKS) # transforms photos to Monet-esque paintings\nphoto_generator = generator_fn(height=None, width=None, transformer_blocks=TRANSFORMER_BLOCKS) # transforms Monet paintings to be more like photos\n\nmonet_discriminator = discriminator_fn(height=None, width=None) # differentiates real Monet paintings and generated Monet paintings\nphoto_discriminator = discriminator_fn(height=None, width=None) # differentiates real photos and generated photos\n\n\nclass CycleGan(Model):\n def __init__(\n self,\n monet_generator,\n photo_generator,\n monet_discriminator,\n photo_discriminator,\n lambda_cycle=10,\n ):\n super(CycleGan, self).__init__()\n self.m_gen = monet_generator\n self.p_gen = photo_generator\n self.m_disc = monet_discriminator\n self.p_disc = photo_discriminator\n self.lambda_cycle = lambda_cycle\n \n def compile(\n self,\n m_gen_optimizer,\n p_gen_optimizer,\n m_disc_optimizer,\n p_disc_optimizer,\n gen_loss_fn,\n disc_loss_fn,\n cycle_loss_fn,\n identity_loss_fn\n ):\n super(CycleGan, self).compile()\n self.m_gen_optimizer = m_gen_optimizer\n self.p_gen_optimizer = p_gen_optimizer\n self.m_disc_optimizer = m_disc_optimizer\n self.p_disc_optimizer = p_disc_optimizer\n self.gen_loss_fn = gen_loss_fn\n self.disc_loss_fn = disc_loss_fn\n self.cycle_loss_fn = cycle_loss_fn\n self.identity_loss_fn = identity_loss_fn\n \n def train_step(self, batch_data):\n real_monet, real_photo = batch_data\n \n with tf.GradientTape(persistent=True) as tape:\n # photo to monet back to photo\n fake_monet = self.m_gen(real_photo, training=True)\n cycled_photo = self.p_gen(fake_monet, training=True)\n\n # monet to photo back to monet\n fake_photo = self.p_gen(real_monet, training=True)\n cycled_monet = self.m_gen(fake_photo, training=True)\n\n # generating itself\n same_monet = self.m_gen(real_monet, training=True)\n same_photo = self.p_gen(real_photo, training=True)\n\n # discriminator used to check, inputing real images\n disc_real_monet = self.m_disc(real_monet, training=True)\n disc_real_photo = self.p_disc(real_photo, training=True)\n\n # discriminator used to check, inputing fake images\n disc_fake_monet = self.m_disc(fake_monet, training=True)\n disc_fake_photo = self.p_disc(fake_photo, training=True)\n\n # evaluates generator loss\n monet_gen_loss = self.gen_loss_fn(disc_fake_monet)\n photo_gen_loss = self.gen_loss_fn(disc_fake_photo)\n\n # evaluates total cycle consistency loss\n total_cycle_loss = self.cycle_loss_fn(real_monet, cycled_monet, self.lambda_cycle) + self.cycle_loss_fn(real_photo, cycled_photo, self.lambda_cycle)\n\n # evaluates total generator loss\n total_monet_gen_loss = monet_gen_loss + total_cycle_loss + self.identity_loss_fn(real_monet, same_monet, self.lambda_cycle)\n total_photo_gen_loss = photo_gen_loss + total_cycle_loss + self.identity_loss_fn(real_photo, same_photo, self.lambda_cycle)\n\n # evaluates discriminator loss\n monet_disc_loss = self.disc_loss_fn(disc_real_monet, disc_fake_monet)\n photo_disc_loss = self.disc_loss_fn(disc_real_photo, disc_fake_photo)\n\n # Calculate the gradients for generator and discriminator\n monet_generator_gradients = tape.gradient(total_monet_gen_loss,\n self.m_gen.trainable_variables)\n photo_generator_gradients = tape.gradient(total_photo_gen_loss,\n self.p_gen.trainable_variables)\n\n monet_discriminator_gradients = tape.gradient(monet_disc_loss,\n self.m_disc.trainable_variables)\n photo_discriminator_gradients = tape.gradient(photo_disc_loss,\n self.p_disc.trainable_variables)\n\n # Apply the gradients to the optimizer\n self.m_gen_optimizer.apply_gradients(zip(monet_generator_gradients,\n self.m_gen.trainable_variables))\n\n self.p_gen_optimizer.apply_gradients(zip(photo_generator_gradients,\n self.p_gen.trainable_variables))\n\n self.m_disc_optimizer.apply_gradients(zip(monet_discriminator_gradients,\n self.m_disc.trainable_variables))\n\n self.p_disc_optimizer.apply_gradients(zip(photo_discriminator_gradients,\n self.p_disc.trainable_variables))\n \n return {'monet_gen_loss': total_monet_gen_loss,\n 'photo_gen_loss': total_photo_gen_loss,\n 'monet_disc_loss': monet_disc_loss,\n 'photo_disc_loss': photo_disc_loss\n }","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-06-10T16:41:44.439606Z","iopub.execute_input":"2024-06-10T16:41:44.439963Z","iopub.status.idle":"2024-06-10T16:41:45.163216Z","shell.execute_reply.started":"2024-06-10T16:41:44.439934Z","shell.execute_reply":"2024-06-10T16:41:45.162427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Loss functions","metadata":{}},{"cell_type":"code","source":"# with strategy.scope():\n# Discriminator loss {0: fake, 1: real} (The discriminator loss outputs the average of the real and generated loss)\ndef discriminator_loss(real, generated):\n real_loss = losses.BinaryCrossentropy(from_logits=True, reduction=losses.Reduction.NONE)(tf.ones_like(real), real)\n\n generated_loss = losses.BinaryCrossentropy(from_logits=True, reduction=losses.Reduction.NONE)(tf.zeros_like(generated), generated)\n\n total_disc_loss = real_loss + generated_loss\n\n return total_disc_loss * 0.5\n\n# Generator loss\ndef generator_loss(generated):\n return losses.BinaryCrossentropy(from_logits=True, reduction=losses.Reduction.NONE)(tf.ones_like(generated), generated)\n\n\n# Cycle consistency loss (measures if original photo and the twice transformed photo to be similar to one another)\n#with strategy.scope():\ndef calc_cycle_loss(real_image, cycled_image, LAMBDA):\n loss1 = tf.reduce_mean(tf.abs(real_image - cycled_image))\n\n return LAMBDA * loss1\n\n# Identity loss (compares the image with its generator (i.e. photo with photo generator))\n#with strategy.scope():\ndef identity_loss(real_image, same_image, LAMBDA):\n loss = tf.reduce_mean(tf.abs(real_image - same_image))\n return LAMBDA * 0.5 * loss","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-06-10T16:41:47.958658Z","iopub.execute_input":"2024-06-10T16:41:47.959063Z","iopub.status.idle":"2024-06-10T16:41:47.969161Z","shell.execute_reply.started":"2024-06-10T16:41:47.959029Z","shell.execute_reply":"2024-06-10T16:41:47.96826Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train","metadata":{}},{"cell_type":"code","source":"# with strategy.scope():\n\nmonet_generator_optimizer = optimizers.Adam(2e-4, beta_1=0.5)\nphoto_generator_optimizer = optimizers.Adam(2e-4, beta_1=0.5)\n\nmonet_discriminator_optimizer = optimizers.Adam(2e-4, beta_1=0.5)\nphoto_discriminator_optimizer = optimizers.Adam(2e-4, beta_1=0.5)\n\n\n# Create GAN\ngan_model = CycleGan(monet_generator, photo_generator, \n monet_discriminator, photo_discriminator)\n\ngan_model.compile(m_gen_optimizer=monet_generator_optimizer,\n p_gen_optimizer=photo_generator_optimizer,\n m_disc_optimizer=monet_discriminator_optimizer,\n p_disc_optimizer=photo_discriminator_optimizer,\n gen_loss_fn=generator_loss,\n disc_loss_fn=discriminator_loss,\n cycle_loss_fn=calc_cycle_loss,\n identity_loss_fn=identity_loss)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-06-10T16:41:50.372619Z","iopub.execute_input":"2024-06-10T16:41:50.373475Z","iopub.status.idle":"2024-06-10T16:41:50.389858Z","shell.execute_reply.started":"2024-06-10T16:41:50.373442Z","shell.execute_reply":"2024-06-10T16:41:50.389024Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create dataset\nmonet_ds = get_dataset(MONET_FILENAMES, augment=data_augment, batch_size=BATCH_SIZE)\nphoto_ds = get_dataset(PHOTO_FILENAMES, augment=data_augment, batch_size=BATCH_SIZE)\ngan_ds = tf.data.Dataset.zip((monet_ds, photo_ds))\n\nphoto_ds_eval = get_dataset(PHOTO_FILENAMES, repeat=False, shuffle=False, batch_size=1)\nmonet_ds_eval = get_dataset(MONET_FILENAMES, repeat=False, shuffle=False, batch_size=1)\n\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-06-10T16:41:52.956812Z","iopub.execute_input":"2024-06-10T16:41:52.957551Z","iopub.status.idle":"2024-06-10T16:41:53.42043Z","shell.execute_reply.started":"2024-06-10T16:41:52.957511Z","shell.execute_reply":"2024-06-10T16:41:53.419628Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"monet_generator.load_weights(\"/kaggle/input/generator_monet/tensorflow2/generator4/1/monet_generator4.h5\")\nmonet_discriminator.load_weights(\"/kaggle/input/monet_discriminator/tensorflow2/discriminator4/1/monet_discriminator4.h5\")\nphoto_generator.load_weights(\"/kaggle/input/photo_generator/tensorflow2/photogenerator4/1/photo_generator4.h5\")\nphoto_discriminator.load_weights(\"/kaggle/input/photo_discriminator/tensorflow2/photodiscriminator4/1/photo_discriminator4.h5\")","metadata":{"execution":{"iopub.status.busy":"2024-06-07T05:26:26.308672Z","iopub.execute_input":"2024-06-07T05:26:26.309071Z","iopub.status.idle":"2024-06-07T05:26:27.544504Z","shell.execute_reply.started":"2024-06-07T05:26:26.309039Z","shell.execute_reply":"2024-06-07T05:26:27.543674Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"history = gan_model.fit(gan_ds, \n epochs=7, \n steps_per_epoch=(7038//BATCH_SIZE), \n verbose=2)","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2024-06-10T16:41:58.162137Z","iopub.execute_input":"2024-06-10T16:41:58.162513Z","iopub.status.idle":"2024-06-10T17:26:31.049544Z","shell.execute_reply.started":"2024-06-10T16:41:58.162478Z","shell.execute_reply":"2024-06-10T17:26:31.047472Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Evaluating generator models\n\nHere we are going to evaluate the generator models including how good is the generator cycle, this means that we will get a photo to generate a Monet picture from it, then use the generated picture to generate the original photo.\n\n## Photo (input) -> Monet (generated) -> Photo (generated)","metadata":{}},{"cell_type":"code","source":"evaluate_cycle(photo_ds_eval.take(2), monet_generator, photo_generator, n_samples=2)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-06-07T07:16:43.985582Z","iopub.execute_input":"2024-06-07T07:16:43.986001Z","iopub.status.idle":"2024-06-07T07:16:47.571109Z","shell.execute_reply.started":"2024-06-07T07:16:43.985971Z","shell.execute_reply":"2024-06-07T07:16:47.570188Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here we will do the same process but starting with a Monet picture.\n\n## Monet (input) -> Photo (generated) -> Monet (generated)","metadata":{}},{"cell_type":"code","source":"evaluate_cycle(monet_ds_eval.take(2), photo_generator, monet_generator, n_samples=2)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-06-07T07:16:47.573609Z","iopub.execute_input":"2024-06-07T07:16:47.573955Z","iopub.status.idle":"2024-06-07T07:16:51.142591Z","shell.execute_reply.started":"2024-06-07T07:16:47.573923Z","shell.execute_reply":"2024-06-07T07:16:51.141193Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Visualize predictions\n\nA common issue with images generated by GANs is that the often show some undisered artifacts, a very common on is known as \"[checkerboard artifacts](https://distill.pub/2016/deconv-checkerboard/)\", a good practice is to inspect some of the images to see its quality and if some of these undisered artifacts are present.","metadata":{}},{"cell_type":"code","source":"display_generated_samples(photo_ds_eval.take(8), monet_generator, 8)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-06-07T07:16:51.144398Z","iopub.execute_input":"2024-06-07T07:16:51.144798Z","iopub.status.idle":"2024-06-07T07:16:57.308363Z","shell.execute_reply.started":"2024-06-07T07:16:51.144759Z","shell.execute_reply":"2024-06-07T07:16:57.307375Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Make predictions","metadata":{}},{"cell_type":"code","source":"%%time\nos.makedirs('../images/') # Create folder to save generated images\npredict_and_save(photo_ds_eval, monet_generator, '../images/')","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission file","metadata":{}},{"cell_type":"code","source":"shutil.make_archive('/kaggle/working/images/', 'zip', '../images')\n\nprint(f\"Generated samples: {len([name for name in os.listdir('../images/') if os.path.isfile(os.path.join('../images/', name))])}\")","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Output models","metadata":{}},{"cell_type":"code","source":"monet_generator.save('monet_generator5.h5')\nphoto_generator.save('photo_generator5.h5')\nmonet_discriminator.save('monet_discriminator5.h5')\nphoto_discriminator.save('photo_discriminator5.h5')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2024-04-15T08:13:47.488843Z","iopub.execute_input":"2024-04-15T08:13:47.489935Z","iopub.status.idle":"2024-04-15T08:13:47.783873Z","shell.execute_reply.started":"2024-04-15T08:13:47.489892Z","shell.execute_reply":"2024-04-15T08:13:47.783021Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}
