{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":21755,"databundleVersionId":1475600,"sourceType":"competition"}],"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Introduction\n\nThe task of the competition [I’m Something of a Painter Myself](https://www.kaggle.com/competitions/gan-getting-started) is to build a generative adversarial networks (GAN) that generates 7,000 to 10,000 Monet-style images. I'll try to build the CycleGAN model, which is particularly well-suited for unpaired image-to-image translation tasks like this one, as described in [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://arxiv.org/abs/1703.10593). \n\nAs we know, a GAN consists of at least two neural networks: a *generator* model and a *discriminator* model. The generator is a neural network that creates the images. this generator is trained using a discriminator. The two models will work against each other, with the generator trying to trick the discriminator, and the discriminator trying to accurately classify the real vs. generated images.\n\nThis implementation consisted of several key components:\n\n- Data Preparation: We utilized two distinct datasets:\n    - Monet-style images dataset for training.\n    - Photos dataset for testing and transformation.\n- Model Architecture: The CycleGAN comprised two generator-discriminator pairs:\n    - A generator to transform photos into Monet-style images, and its corresponding discriminator.\n    - A generator to transform Monet-style images into photos, and its corresponding discriminator.\n- Loss Functions: We implemented multiple loss functions to train the model effectively:\n    - Adversarial loss to train generators to produce realistic images in each domain. \n    - Cycle consistency loss to ensure the preservation of content and structure during transformations.\n    - Identity loss to maintain color composition when processing images already in the target domain.\n- Training Process: The model was trained using a custom training loop, allowing for fine-grained control over the learning process and detailed monitoring of different loss components.\n\nFinally, based on the competition requirements, we'll use the CycleGAN model to apply Monet-style to the testing photos for submission.","metadata":{}},{"cell_type":"markdown","source":"### Setup","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow.keras import layers, models\n\nimport os\nimport pathlib\nimport time\nimport datetime\n\nfrom matplotlib import pyplot as plt\nfrom IPython import display\n\nimport PIL\n\nAUTOTUNE = tf.data.experimental.AUTOTUNE\n\ntf.__version__","metadata":{"execution":{"iopub.status.busy":"2024-09-16T00:50:23.701011Z","iopub.execute_input":"2024-09-16T00:50:23.701436Z","iopub.status.idle":"2024-09-16T00:50:37.312856Z","shell.execute_reply.started":"2024-09-16T00:50:23.701374Z","shell.execute_reply":"2024-09-16T00:50:37.311749Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Dataset\n\nThe dataset contains four directories: `monet_tfrec`, `photo_tfrec`, `monet_jpg`, and `photo_jpg`. \n\nThe monet directories contain Monet paintings, we'll use these images to train our model.\n\nThe photo directories contain photos, we'll add Monet-style to these images and submit generated jpeg images as a zip file, and the submission files limited to 10,000 images.\n\nAlso, we'll use [TFRecords](https://www.tensorflow.org/tutorials/load_data/tfrecord) to efficiently read and store datasets.","metadata":{}},{"cell_type":"markdown","source":"## Exploratory Data Analysis\n\n### Load the dataset","metadata":{}},{"cell_type":"code","source":"TRAIN_FILES = []\nTEST_FILES = []\nfor dirname, _, filenames in os.walk('../input/gan-getting-started/monet_tfrec'):\n    for filename in filenames:\n         TRAIN_FILES.append(os.path.join(dirname, filename))\n            \nfor dirname, _, filenames in os.walk('../input/gan-getting-started/photo_tfrec'):\n    for filename in filenames:\n         TEST_FILES.append(os.path.join(dirname, filename))\n\nTRAIN_FILES","metadata":{"execution":{"iopub.status.busy":"2024-09-16T00:50:37.317580Z","iopub.execute_input":"2024-09-16T00:50:37.319000Z","iopub.status.idle":"2024-09-16T00:50:37.350067Z","shell.execute_reply.started":"2024-09-16T00:50:37.318946Z","shell.execute_reply":"2024-09-16T00:50:37.348764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we have the `images.tfrecords` files, let's use the `tf.data.TFRecordDataset` to read the TFRecord files:","metadata":{}},{"cell_type":"code","source":"raw_train_ds = tf.data.TFRecordDataset(TRAIN_FILES)\nraw_test_ds = tf.data.TFRecordDataset(TEST_FILES)\n\nraw_train_ds","metadata":{"execution":{"iopub.status.busy":"2024-09-16T00:50:37.352531Z","iopub.execute_input":"2024-09-16T00:50:37.353445Z","iopub.status.idle":"2024-09-16T00:50:38.129454Z","shell.execute_reply.started":"2024-09-16T00:50:37.353374Z","shell.execute_reply":"2024-09-16T00:50:38.128432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We'll use the `tf.train.Example.ParseFromString` function to parse a tensor from dataset and return a `tf.train.Example` object, then we read the features key and value from the `Example` object:","metadata":{}},{"cell_type":"code","source":"import numpy as np\n\nfor raw_record in raw_train_ds.take(1):\n    example = tf.train.Example()\n    example.ParseFromString(raw_record.numpy())\n\n    result = {}\n\n# example.features.feature is the dictionary\nfor key, feature in example.features.feature.items():\n    # The values are the Feature objects which contain a `kind` which contains:\n    # one of three fields: bytes_list, float_list, int64_list\n    kind = feature.WhichOneof('kind')\n    result[key] = np.array(getattr(feature, kind).value).dtype\n\nresult","metadata":{"execution":{"iopub.status.busy":"2024-09-16T00:50:38.132597Z","iopub.execute_input":"2024-09-16T00:50:38.133317Z","iopub.status.idle":"2024-09-16T00:50:38.199155Z","shell.execute_reply.started":"2024-09-16T00:50:38.133278Z","shell.execute_reply":"2024-09-16T00:50:38.198046Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Based on above results, we know that the dataset have 3 features: `image_name`, `target`, `image`, and all features have a string data type.\n\nSo, we'll create a description dictionary of the features and create parse function to parse `Example` proto using the description dictonary.","metadata":{}},{"cell_type":"code","source":"# Create a description of the features.\nfeature_description = {\n    'target': tf.io.FixedLenFeature([], tf.string, default_value=''),\n    'image_name': tf.io.FixedLenFeature([], tf.string, default_value=''),\n    'image': tf.io.FixedLenFeature([], tf.string, default_value=''),\n}\n\ndef _parse_image(example_proto):\n  # Parse the input tf.train.Example proto using the dictionary above.\n  return tf.io.parse_single_example(example_proto, feature_description)\n\ntrain_ds = raw_train_ds.map(_parse_image)\ntest_ds = raw_test_ds.map(_parse_image)\n\ntrain_ds","metadata":{"execution":{"iopub.status.busy":"2024-09-16T00:50:38.200380Z","iopub.execute_input":"2024-09-16T00:50:38.200756Z","iopub.status.idle":"2024-09-16T00:50:38.279196Z","shell.execute_reply.started":"2024-09-16T00:50:38.200718Z","shell.execute_reply":"2024-09-16T00:50:38.278235Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Recover the image from the TFRecord file:","metadata":{}},{"cell_type":"code","source":"for image_features in train_ds.take(1):\n    image = image_features['image'].numpy()\n    \n    image_name = image_features['image_name'].numpy()\n    filename = image_name.decode('utf-8')\n    \n    target = image_features['target'].numpy()\n    label = target.decode('utf-8')\n    \n    display.display(display.Image(data=image,alt=label))\n\n# Decode an image file to a uint8 tensor\nsample_image = tf.io.decode_jpeg(image)\nsample_image.shape","metadata":{"execution":{"iopub.status.busy":"2024-09-16T00:50:38.280326Z","iopub.execute_input":"2024-09-16T00:50:38.280660Z","iopub.status.idle":"2024-09-16T00:50:38.322406Z","shell.execute_reply.started":"2024-09-16T00:50:38.280616Z","shell.execute_reply":"2024-09-16T00:50:38.321420Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"According to the above results, we know the image size is 256x256.","metadata":{}},{"cell_type":"markdown","source":"### Preprocessing\n\nAs mentioned in the [CycleGAN paper](https://arxiv.org/abs/1703.10593), apply random jittering and mirroring to the training dataset. These are some of the image augmentation techniques that avoids overfitting.\n\n- In random jittering, the image is resized to 286 x 286 and then randomly cropped to 256 x 256.\n- In random mirroring, the image is randomly flipped horizontally i.e., left to right.\n- Normalize the images to the [-1, 1] range","metadata":{}},{"cell_type":"code","source":"# Each image is 256x256 in size\nIMAGE_SIZE = 256\nCHANNELS = 3\nEPOCHS = 30\n\ndef random_crop(image):\n    cropped_image = tf.image.random_crop(image, size=[IMAGE_SIZE, IMAGE_SIZE, CHANNELS])\n\n    return cropped_image\n\n# normalizing the images to [-1, 1]\ndef normalize(image):\n    image = tf.cast(image, tf.float32)\n    image = (image / 127.5) - 1\n    return image\n\ndef random_jitter(image):\n    # resizing to 286 x 286 x 3\n    image = tf.image.resize(image, [286, 286], method=tf.image.ResizeMethod.NEAREST_NEIGHBOR)\n\n    # randomly cropping to 256 x 256 x 3\n    image = random_crop(image)\n\n    # random mirroring\n    image = tf.image.random_flip_left_right(image)\n\n    return image","metadata":{"execution":{"iopub.status.busy":"2024-09-16T00:50:38.323866Z","iopub.execute_input":"2024-09-16T00:50:38.324603Z","iopub.status.idle":"2024-09-16T00:50:38.332541Z","shell.execute_reply.started":"2024-09-16T00:50:38.324550Z","shell.execute_reply":"2024-09-16T00:50:38.331351Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- Batch and shuffle dataset","metadata":{}},{"cell_type":"code","source":"# The facade training set consist of 300 images\nBUFFER_SIZE = 300\nBATCH_SIZE = 1\n\ndef preprocess_image_train(batch):\n    image = batch['image']\n    image = tf.io.decode_jpeg(image, channels=CHANNELS)\n    image = random_jitter(image)\n    image = normalize(image)\n    return image\n\ndef preprocess_image_test(batch):\n    image = batch['image']\n    image = tf.io.decode_jpeg(image, channels=CHANNELS)\n    image = normalize(image)\n    return image\n\ndef create_dataset_train(dataset):\n    ds = dataset.map(lambda batch: preprocess_image_train(batch), num_parallel_calls=AUTOTUNE)\n    ds = ds.cache().shuffle(BUFFER_SIZE).batch(BATCH_SIZE)\n    return ds.prefetch(AUTOTUNE)\n\ndef create_dataset_test(dataset):\n    ds = dataset.map(lambda batch: preprocess_image_test(batch), num_parallel_calls=AUTOTUNE)\n    ds = ds.cache().shuffle(BUFFER_SIZE).batch(BATCH_SIZE)\n    return ds.prefetch(AUTOTUNE)\n\ntrain_ds = create_dataset_train(train_ds)\ntest_ds = create_dataset_test(test_ds)","metadata":{"execution":{"iopub.status.busy":"2024-09-16T00:50:38.333937Z","iopub.execute_input":"2024-09-16T00:50:38.334350Z","iopub.status.idle":"2024-09-16T00:50:38.677447Z","shell.execute_reply.started":"2024-09-16T00:50:38.334303Z","shell.execute_reply":"2024-09-16T00:50:38.676461Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.subplot(121)\nplt.title('Monet art')\nplt.imshow(sample_image)\n\nplt.subplot(122)\nplt.title('Monet art with random jitter')\nplt.imshow(random_jitter(sample_image))","metadata":{"execution":{"iopub.status.busy":"2024-09-16T00:50:38.678796Z","iopub.execute_input":"2024-09-16T00:50:38.679213Z","iopub.status.idle":"2024-09-16T00:50:39.350565Z","shell.execute_reply.started":"2024-09-16T00:50:38.679167Z","shell.execute_reply":"2024-09-16T00:50:39.349470Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Build the CycleGAN Model","metadata":{}},{"cell_type":"markdown","source":"### Build the generator\n\nThe [CycleGAN paper](https://arxiv.org/abs/1703.10593) uses a modified ResNet based generator. For simplicity, we'll use a modified [U-Net](https://arxiv.org/abs/1505.04597) for generator. A U-Net consists of an downsampler (encoder) and upsampler (decoder).","metadata":{}},{"cell_type":"markdown","source":"#### Build the downsampler (encoder)\n\nStructure: Convolution ->  Instance normalization -> Leaky ReLU\n\n> Note that there is no `InstanceNormalization` defined in the keras layers, here we use the `GroupNormalization` with the `group=-1` is the same as the `InstanceNormalization`","metadata":{}},{"cell_type":"code","source":"def downsample(filters, size, apply_norm=True):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    result = models.Sequential()\n    result.add(layers.Conv2D(filters, size, strides=2, padding='same',\n                             kernel_initializer=initializer, use_bias=False))\n    if apply_norm:\n        result.add(layers.GroupNormalization(groups=-1))\n    result.add(layers.LeakyReLU())\n    return result","metadata":{"execution":{"iopub.status.busy":"2024-09-16T00:50:39.354428Z","iopub.execute_input":"2024-09-16T00:50:39.354857Z","iopub.status.idle":"2024-09-16T00:50:39.361611Z","shell.execute_reply.started":"2024-09-16T00:50:39.354816Z","shell.execute_reply":"2024-09-16T00:50:39.360472Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Build the upsampler(decoder)\n\nStructure: Transposed convolution -> Instance normalization -> Dropout (applied to the first 3 blocks) -> ReLU","metadata":{}},{"cell_type":"code","source":"def upsample(filters, size, apply_dropout=False):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    result = models.Sequential()\n    result.add(layers.Conv2DTranspose(filters, size, strides=2, padding='same',\n                                      kernel_initializer=initializer, use_bias=False))\n    result.add(layers.GroupNormalization(groups=-1))\n    if apply_dropout:\n        result.add(layers.Dropout(0.5))\n    result.add(layers.ReLU())\n    return result","metadata":{"execution":{"iopub.status.busy":"2024-09-16T00:50:39.362861Z","iopub.execute_input":"2024-09-16T00:50:39.363186Z","iopub.status.idle":"2024-09-16T00:50:39.371245Z","shell.execute_reply.started":"2024-09-16T00:50:39.363140Z","shell.execute_reply":"2024-09-16T00:50:39.370266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Generator\n\nBuild generator with encoder and decoder:","metadata":{}},{"cell_type":"code","source":"def make_generator():\n    inputs = layers.Input(shape=[IMAGE_SIZE, IMAGE_SIZE, CHANNELS])\n\n    down_stack = [\n        downsample(64, 4, apply_norm=False),\n        downsample(128, 4),\n        downsample(256, 4),\n        downsample(512, 4),\n        downsample(512, 4),\n        downsample(512, 4),\n        downsample(512, 4),\n        downsample(512, 4),\n    ]\n    \n    up_stack = [\n        upsample(512, 4, apply_dropout=True),\n        upsample(512, 4, apply_dropout=True),\n        upsample(512, 4, apply_dropout=True),\n        upsample(512, 4),\n        upsample(256, 4),\n        upsample(128, 4),\n        upsample(64, 4),\n    ]\n    \n    initializer = tf.random_normal_initializer(0., 0.02)\n    last = layers.Conv2DTranspose(3, 4, strides=2, padding='same',\n                                  kernel_initializer=initializer, activation='tanh')\n    \n    x = inputs\n    skips = []\n    for down in down_stack:\n        x = down(x)\n        skips.append(x)\n    \n    skips = reversed(skips[:-1])\n    for up, skip in zip(up_stack, skips):\n        x = up(x)\n        x = layers.Concatenate()([x, skip])\n    \n    x = last(x)\n    return models.Model(inputs=inputs, outputs=x)","metadata":{"execution":{"iopub.status.busy":"2024-09-16T00:50:39.372665Z","iopub.execute_input":"2024-09-16T00:50:39.372983Z","iopub.status.idle":"2024-09-16T00:50:39.383967Z","shell.execute_reply.started":"2024-09-16T00:50:39.372951Z","shell.execute_reply":"2024-09-16T00:50:39.383023Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Build the discriminator\n\nThe discriminator is a CNN-based image classifier:","metadata":{}},{"cell_type":"code","source":"def make_discriminator():\n    initializer = tf.random_normal_initializer(0., 0.02)\n    inputs = layers.Input(shape=[IMAGE_SIZE, IMAGE_SIZE, 3], name='input_image')\n    \n    x = inputs\n    x = downsample(64, 4, False)(x)\n    x = downsample(128, 4)(x)\n    x = downsample(256, 4)(x)\n    x = layers.ZeroPadding2D()(x)\n    x = layers.Conv2D(512, 4, strides=1, kernel_initializer=initializer, use_bias=False)(x)\n    x = layers.GroupNormalization(groups=-1)(x)\n    x = layers.LeakyReLU()(x)\n    x = layers.ZeroPadding2D()(x)\n    x = layers.Conv2D(1, 4, strides=1, kernel_initializer=initializer)(x)\n    return models.Model(inputs=inputs, outputs=x)","metadata":{"execution":{"iopub.status.busy":"2024-09-16T00:50:39.385207Z","iopub.execute_input":"2024-09-16T00:50:39.385522Z","iopub.status.idle":"2024-09-16T00:50:39.398020Z","shell.execute_reply.started":"2024-09-16T00:50:39.385487Z","shell.execute_reply":"2024-09-16T00:50:39.397072Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Create Generators and Discriminators","metadata":{}},{"cell_type":"code","source":"monet_generator = make_generator()\nphoto_generator = make_generator()\nmonet_discriminator = make_discriminator()\nphoto_discriminator = make_discriminator()","metadata":{"execution":{"iopub.status.busy":"2024-09-16T00:50:39.399352Z","iopub.execute_input":"2024-09-16T00:50:39.400334Z","iopub.status.idle":"2024-09-16T00:50:40.303094Z","shell.execute_reply.started":"2024-09-16T00:50:39.400283Z","shell.execute_reply":"2024-09-16T00:50:40.301951Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Use the discriminator to classify the generated images as reak or fake. The model will be trained to output positive values for real images, and negative values for fake images.","metadata":{}},{"cell_type":"markdown","source":"### Define the loss and optimizers\n\nDefine loss functions for both models:","metadata":{}},{"cell_type":"code","source":"# Loss function\ncross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits=True)","metadata":{"execution":{"iopub.status.busy":"2024-09-16T00:50:40.304390Z","iopub.execute_input":"2024-09-16T00:50:40.304738Z","iopub.status.idle":"2024-09-16T00:50:40.309689Z","shell.execute_reply.started":"2024-09-16T00:50:40.304675Z","shell.execute_reply":"2024-09-16T00:50:40.308560Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Discriminator Loss\n\nThis method quantifies how well the discriminator is able to distinguish real images from fakes. it compares the discriminator's predictions on real images to an array of 1s, and the discriminator's predictions on fake (generated) images to an array os 0s.","metadata":{}},{"cell_type":"code","source":"def discriminator_loss(real_output, fake_output):\n    real_loss = cross_entropy(tf.ones_like(real_output), real_output)\n    fake_loss = cross_entropy(tf.zeros_like(fake_output), fake_output)\n    total_loss = real_loss + fake_loss\n    return total_loss * 0.5","metadata":{"execution":{"iopub.status.busy":"2024-09-16T00:50:40.310950Z","iopub.execute_input":"2024-09-16T00:50:40.311318Z","iopub.status.idle":"2024-09-16T00:50:40.323105Z","shell.execute_reply.started":"2024-09-16T00:50:40.311281Z","shell.execute_reply":"2024-09-16T00:50:40.322020Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Generator Loss\n\nThe generator's loss function quantifies how well it was able to trick the discriminator. That is, if the generator is performing well, the discriminator will classofy the fake images as real (or 1). It compares the discriminiators decisions on the generated images to an array of 1s.","metadata":{}},{"cell_type":"code","source":"def generator_loss(fake_output):\n    return cross_entropy(tf.ones_like(fake_output), fake_output)","metadata":{"execution":{"iopub.status.busy":"2024-09-16T00:50:40.324361Z","iopub.execute_input":"2024-09-16T00:50:40.325112Z","iopub.status.idle":"2024-09-16T00:50:40.333592Z","shell.execute_reply.started":"2024-09-16T00:50:40.325063Z","shell.execute_reply":"2024-09-16T00:50:40.332744Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"LAMBDA = 10\n\ndef compute_cycle_loss(real_img, cycled_img):\n    loss = tf.reduce_mean(tf.abs(real_img - cycled_img))\n    return LAMBDA * loss","metadata":{"execution":{"iopub.status.busy":"2024-09-16T00:50:40.334774Z","iopub.execute_input":"2024-09-16T00:50:40.335071Z","iopub.status.idle":"2024-09-16T00:50:40.344739Z","shell.execute_reply.started":"2024-09-16T00:50:40.335038Z","shell.execute_reply":"2024-09-16T00:50:40.343638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def identity_loss(real_img, same_img):\n    loss = tf.reduce_mean(tf.abs(real_img - same_img))\n    return LAMBDA * 0.5 * loss","metadata":{"execution":{"iopub.status.busy":"2024-09-16T00:50:40.346217Z","iopub.execute_input":"2024-09-16T00:50:40.346674Z","iopub.status.idle":"2024-09-16T00:50:40.361693Z","shell.execute_reply.started":"2024-09-16T00:50:40.346627Z","shell.execute_reply":"2024-09-16T00:50:40.360837Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Optimizers\n\nInitialize the optimizers for all the generators and the discriminators:","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras import optimizers\n\n# Optimizers\nmonet_generator_optimizer = optimizers.Adam(2e-4, beta_1=0.5)\nphoto_generator_optimizer = optimizers.Adam(2e-4, beta_1=0.5)\nmonet_discriminator_optimizer = optimizers.Adam(2e-4, beta_1=0.5)\nphoto_discriminator_optimizer = optimizers.Adam(2e-4, beta_1=0.5)","metadata":{"execution":{"iopub.status.busy":"2024-09-16T00:50:40.363053Z","iopub.execute_input":"2024-09-16T00:50:40.363465Z","iopub.status.idle":"2024-09-16T00:50:40.382300Z","shell.execute_reply.started":"2024-09-16T00:50:40.363416Z","shell.execute_reply":"2024-09-16T00:50:40.381589Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Checkpoints\n\nThis method try to save and restore models, this is helpful in case a long running training task is interrupted.","metadata":{}},{"cell_type":"code","source":"CHECKPOINT_DIR = '../working/checkpoints/train'\n\nckpt = tf.train.Checkpoint(monet_generator=monet_generator,\n                           photo_generator=photo_generator,\n                           monet_discriminator=monet_discriminator,\n                           photo_discriminator=photo_discriminator,\n                           monet_generator_optimizer=monet_generator_optimizer,\n                           photo_generator_optimizer=photo_generator_optimizer,\n                           monet_discriminator_optimizer=monet_discriminator_optimizer,\n                           photo_discriminator_optimizer=photo_discriminator_optimizer)\n\nckpt_manager = tf.train.CheckpointManager(ckpt, CHECKPOINT_DIR, max_to_keep=5)\n\n# if a checkpoint exists, restore the latest checkpoint.\nif ckpt_manager.latest_checkpoint:\n  ckpt.restore(ckpt_manager.latest_checkpoint)\n  print ('Latest checkpoint restored!!')","metadata":{"execution":{"iopub.status.busy":"2024-09-16T00:50:40.383382Z","iopub.execute_input":"2024-09-16T00:50:40.383720Z","iopub.status.idle":"2024-09-16T00:50:40.392034Z","shell.execute_reply.started":"2024-09-16T00:50:40.383659Z","shell.execute_reply":"2024-09-16T00:50:40.390924Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Training","metadata":{}},{"cell_type":"markdown","source":"#### Training loop\n\nThe training loop consists of four basic steps:\n- Get the predictions\n- Calculate the loss\n- Calculate the gradients using backpropagation\n- Apply the gradients to the optimizer","metadata":{}},{"cell_type":"code","source":"@tf.function\ndef train_step(real_monet, real_photo):\n    with tf.GradientTape(persistent=True) as tape:\n        # real monet -> fake photo\n        fake_photo = monet_generator(real_monet, training=True)\n        # fake photo -> monet\n        cycled_monet = photo_generator(fake_photo, training=True)\n        # real photo -> fake monet\n        fake_monet = photo_generator(real_photo, training=True)\n        # fake monet -> photo\n        cycled_photo = monet_generator(fake_monet, training=True)\n        # use for identity loss\n        same_monet = monet_generator(real_monet, training=True)\n        same_photo = photo_generator(real_photo, training=True)\n        \n        # The discriminators evaluate both real and fake images in each domain\n        disc_real_monet = monet_discriminator(real_monet, training=True)\n        disc_real_photo = photo_discriminator(real_photo, training=True)\n        disc_fake_monet = monet_discriminator(fake_monet, training=True)\n        disc_fake_photo = photo_discriminator(fake_photo, training=True)\n        \n        # Generator losses: How well the generators fool the discriminators\n        monet_gen_loss = generator_loss(disc_fake_photo)\n        photo_gen_loss = generator_loss(disc_fake_monet)\n        # Cycle consistency loss: Ensures that cycled images are similar to originals\n        total_cycle_loss = compute_cycle_loss(real_monet, cycled_monet) + compute_cycle_loss(real_photo, cycled_photo)\n        # Identity loss: Encourages generators to preserve input when it's already in the target domain\n        total_monet_gen_loss = monet_gen_loss + total_cycle_loss + identity_loss(real_monet, same_monet)\n        total_photo_gen_loss = photo_gen_loss + total_cycle_loss + identity_loss(real_photo, same_photo)\n        # Discriminator losses: How well the discriminators distinguish real from fake images\n        monet_disc_loss = discriminator_loss(disc_real_monet, disc_fake_monet)\n        photo_disc_loss = discriminator_loss(disc_real_photo, disc_fake_photo)\n    \n    # Computes the gradients for each model with respect to its loss.\n    monet_generator_gradients = tape.gradient(total_monet_gen_loss, monet_generator.trainable_variables)\n    photo_generator_gradients = tape.gradient(total_photo_gen_loss, photo_generator.trainable_variables)\n    monet_discriminator_gradients = tape.gradient(monet_disc_loss, monet_discriminator.trainable_variables)\n    photo_discriminator_gradients = tape.gradient(photo_disc_loss, photo_discriminator.trainable_variables)\n    \n    # Updates the model parameters using the calculated gradients.\n    monet_generator_optimizer.apply_gradients(zip(monet_generator_gradients, monet_generator.trainable_variables))\n    photo_generator_optimizer.apply_gradients(zip(photo_generator_gradients, photo_generator.trainable_variables))\n    monet_discriminator_optimizer.apply_gradients(zip(monet_discriminator_gradients, monet_discriminator.trainable_variables))\n    photo_discriminator_optimizer.apply_gradients(zip(photo_discriminator_gradients, photo_discriminator.trainable_variables))","metadata":{"execution":{"iopub.status.busy":"2024-09-16T00:50:40.393879Z","iopub.execute_input":"2024-09-16T00:50:40.394675Z","iopub.status.idle":"2024-09-16T00:50:40.407727Z","shell.execute_reply.started":"2024-09-16T00:50:40.394608Z","shell.execute_reply":"2024-09-16T00:50:40.406734Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from IPython.display import clear_output\n\ndef train(monet_dataset, photo_dataset, epochs):\n    n = 0\n    for epoch in range(epochs):\n        start = time.time()\n        for monet_image, photo_image in tf.data.Dataset.zip((monet_dataset, photo_dataset)):\n            train_step(monet_image, photo_image)\n            if n % 10 == 0:\n                print ('.', end='')\n            n += 1\n        clear_output(wait=True)\n        if (epoch + 1) % 5 == 0:\n            ckpt_save_path = ckpt_manager.save()\n            print ('Saving checkpoint for epoch {} at {}'.format(epoch+1,ckpt_save_path))\n        \n        print(f'Time taken for epoch {epoch + 1} is {time.time() - start} sec')","metadata":{"execution":{"iopub.status.busy":"2024-09-16T00:50:40.408939Z","iopub.execute_input":"2024-09-16T00:50:40.409242Z","iopub.status.idle":"2024-09-16T00:50:40.422195Z","shell.execute_reply.started":"2024-09-16T00:50:40.409208Z","shell.execute_reply":"2024-09-16T00:50:40.421158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def generate_images(model, test_input):\n    prediction = model(test_input)\n    plt.figure(figsize=(12, 12))\n    display_list = [test_input[0], prediction[0]]\n    title = ['Input Image', 'Monet-style Image']\n    for i in range(2):\n        plt.subplot(1, 2, i+1)\n        plt.title(title[i])\n        plt.imshow(display_list[i] * 0.5 + 0.5)\n        plt.axis('off')\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2024-09-16T00:50:40.423316Z","iopub.execute_input":"2024-09-16T00:50:40.423892Z","iopub.status.idle":"2024-09-16T00:50:40.432951Z","shell.execute_reply.started":"2024-09-16T00:50:40.423853Z","shell.execute_reply":"2024-09-16T00:50:40.431916Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Train the model\n\n","metadata":{}},{"cell_type":"code","source":"# Train the model\ntrain(train_ds, test_ds, epochs=EPOCHS)","metadata":{"execution":{"iopub.status.busy":"2024-09-16T00:50:40.434325Z","iopub.execute_input":"2024-09-16T00:50:40.434781Z","iopub.status.idle":"2024-09-16T01:28:30.264755Z","shell.execute_reply.started":"2024-09-16T00:50:40.434729Z","shell.execute_reply":"2024-09-16T01:28:30.263706Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Results","metadata":{}},{"cell_type":"markdown","source":"### Generate Monet-style Images","metadata":{}},{"cell_type":"code","source":"# Generate Monet-style image from 5 test photos\nfor photo in test_ds.take(5):\n    generate_images(monet_generator, photo)","metadata":{"execution":{"iopub.status.busy":"2024-09-16T01:28:30.266185Z","iopub.execute_input":"2024-09-16T01:28:30.266499Z","iopub.status.idle":"2024-09-16T01:28:33.717674Z","shell.execute_reply.started":"2024-09-16T01:28:30.266466Z","shell.execute_reply":"2024-09-16T01:28:33.716581Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The implemented CycleGAN model demonstrated good performance in transforming test photos into Monet-style images. Here is my observations:\n\n- Style Transfer Quality: The model successfully captured characteristic elements of Monet's style, including brushstrokes, color palettes, and overall aesthetic.\n- Content Preservation: Thanks to the cycle consistency loss, the transformed images maintained the structural integrity and key features of the original photographs.\n- Color Composition: The identity loss helped in preserving appropriate color compositions, particularly important when dealing with landscapes and natural scenes.","metadata":{}},{"cell_type":"markdown","source":"## Submission","metadata":{}},{"cell_type":"code","source":"! mkdir ../images","metadata":{"execution":{"iopub.status.busy":"2024-09-16T01:28:33.719403Z","iopub.execute_input":"2024-09-16T01:28:33.720142Z","iopub.status.idle":"2024-09-16T01:28:34.890473Z","shell.execute_reply.started":"2024-09-16T01:28:33.720093Z","shell.execute_reply":"2024-09-16T01:28:34.889113Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import shutil\n\ni = 1\nfor photo in test_ds:\n    prediction = monet_generator(photo, training=False)[0].numpy()\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    im = PIL.Image.fromarray(prediction)\n    im.save(\"../images/\" + str(i) + \".jpg\")\n    i += 1\n    \nshutil.make_archive(\"/kaggle/working/images\", 'zip', \"/kaggle/images\")","metadata":{"execution":{"iopub.status.busy":"2024-09-16T01:28:34.896708Z","iopub.execute_input":"2024-09-16T01:28:34.897115Z","iopub.status.idle":"2024-09-16T01:37:21.939952Z","shell.execute_reply.started":"2024-09-16T01:28:34.897071Z","shell.execute_reply":"2024-09-16T01:37:21.938848Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Conclusion\n\nIn this project, we implemented a CycleGAN to transform photos into the Claude Monet style images.\n\nWe start with using the TFRecord to read the Monet photos, its benefits include efficiency, flexibility, scalability, integration with TensorFlow, performance, and portability, making it a popular choice for data storage in many machine learning projects. We also explored using the ` tf.train.Example` to read messages from `.tfrecord` files. Also, as per the [CycleGAN paper](https://arxiv.org/abs/1703.10593) mentioned, we apply random jittering and mirroring techniques to the training dataset to avoid overfitting.\n\nIn the training phase, the complex training step allows the CycleGAN to learn to translate images between the two domains (Monet-style and photo-style) in both directions, while preserving the content and structure of the original images. Note that we implemented multiple loss functions to train the model effectively. Note that the training can take a long time, especially on CPU, after configured to training with TPU, training is significantly faster than before, the training step shortens a lot of time.\n\nIn conclusion, the CycleGAN implementation has successfully bridged the gap between photographs and Monet's artistic style, demonstrating the power of deep learning in creative applications. This project achieves its primary goal of style transfer.","metadata":{}},{"cell_type":"markdown","source":"## Reference\n\n- [Tensorflow: TFRecord and tf.train.Example](https://www.tensorflow.org/tutorials/load_data/tfrecord)\n- [Tensorflow: CycleGAN](https://www.tensorflow.org/tutorials/generative/cyclegan)\n- [Tensorflow: Pix2Pix](https://www.tensorflow.org/tutorials/generative/pix2pix)\n- [Paper: Generative Adversarial Networks (GAN)](https://arxiv.org/abs/1406.2661)\n- [Paper: Image-to-Image Translation with Conditional Adversarial Networks (Pix2Pix)](https://arxiv.org/abs/1611.07004)\n- [Paper: Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks (CycleGAN)](https://arxiv.org/abs/1703.10593)\n","metadata":{}}]}