{"cells":[{"metadata":{},"cell_type":"markdown","source":"# CycleGAN - Style Transfer (Photos to Monet Paintings)\n\n\nThis Notebook intends to follow the development of the CycleGAN architecture to capturing special characteristics of Monet paintings and figuring out how these characteristics could be translated into the other image collection, all in the absence of any paired training examples.\n\nThis is a Kaggle competition to generate images in the style of Oscar-Claude Monet using generative adversarial networks (GANs). A Kaggle competition provides datasets contain Monet paintings \"monet_tfrec\", and real photos \"photo_tfrec\". The images are provided in TFRecord format as well as in JPEG format. In this competition, we are asked to use Oscar-Claude Monet paintings images to train our model and adding Monet-style to the real images, and submit the generated jpeg images as a zip file.\n\nThis problem is an Image-to-image translation and iThis Notebook intends to follow the development of the CycleGAN architecture to capturing special characteristics of Monet paintings and figuring out how these characteristics could be translated into the other image collection, all in the absence of any paired training examples.n order to tackle this problem, in general, we have two approaches Paired approach and an Unpaired approach. In Paired approach, it is necessary to have paired representations of the data in both domains. In In the Unpaired approach, both domains can be completely different as no resemblance between them such as in our case in this project (Converting real images to Monet's masterpiece). There are several methods that work on the principle of the unpaired approach, and one of them that performs very well and has shown impressive results is a CycleGAN.  \n\n"},{"metadata":{},"cell_type":"markdown","source":"Sources:\n* https://arxiv.org/pdf/1703.10593.pdf (paper)*\n\n* https://www.kaggle.com/amyjang/monet-cyclegan-tutorial (baseline competition)\n\n* https://www.kaggle.com/dimitreoliveira/introduction-to-cyclegan-monet-paintings\n\n* https://junyanz.github.io/CycleGAN/\n\n* https://hardikbansal.github.io/CycleGANBlog/\n\n* https://www.tensorflow.org/tutorials/generative/cyclegan\n\n* https://towardsdatascience.com/cyclegan-learning-to-translate-images-without-paired-training-data-5b4e93862c8d"},{"metadata":{},"cell_type":"markdown","source":"# CycleGAN \nUnpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks\n\nIt uses a generator and a discriminator,The generator has to generate images that are accepted by the discriminator, and the discriminator tries to discover the images that are not real and reject the images generated by the generator. CycleGAN uses a loss of cycle consistency to allow training without the need for paired data.\n\n"},{"metadata":{},"cell_type":"markdown","source":"# Implementation"},{"metadata":{},"cell_type":"markdown","source":"Below, we are setting up the input pipeline and importing all the dependencies.\n\nIt is necessary to enable the TPUs during the implementation, to carry out the implementation."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nimport tensorflow_addons as tfa\n\nfrom kaggle_datasets import KaggleDatasets\nimport matplotlib.pyplot as plt\nimport numpy as np\n\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Device:', tpu.master())\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nexcept:\n    strategy = tf.distribute.get_strategy()\nprint('Number of replicas:', strategy.num_replicas_in_sync)\n\nAUTO = tf.data.experimental.AUTOTUNE\nprint(tf.__version__)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Load the dataset"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"GCS_PATH_MONET = KaggleDatasets().get_gcs_path('monet-tfrecords-extdata')\nGCS_PATH = KaggleDatasets().get_gcs_path('gan-getting-started')\nGCS_PATH_MONET, GCS_PATH","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#GCS_PATH_tareq = KaggleDatasets().get_gcs_path('tareq-picture')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.image as mpimg \nimport matplotlib.pyplot as plt \n  \n# Read Images \n# img = mpimg.imread() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# To read external data (my picture )\n# from PIL import Image  \n  \n# # Opens a image in RGB mode  \n# im = Image.open('../input/tareq-picture/image_6487327.JPG') ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# To resize my picture and make it (256,256,3)\n# im2 = im.resize((256,256))\n# im2 = np.asarray(im2)\n# im2 = im2[:,:, 0:3]\n\n# plt.imshow(im2)\n# plt.show()\n\n# print(im2.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Reading the data"},{"metadata":{"trusted":true},"cell_type":"code","source":"import re\nMONET_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/monet_tfrec/*.tfrec'))\nprint('Monet TFRecord Files:', len(MONET_FILENAMES))\n\nPHOTO_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/photo_tfrec/*.tfrec'))\nprint('Photo TFRecord Files:', len(PHOTO_FILENAMES))\n\ndef count_data_items(filenames):\n    n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n    return np.sum(n)\n\nn_monet_samples = count_data_items(MONET_FILENAMES)\nn_photo_samples = count_data_items(PHOTO_FILENAMES)\n\nBATCH_SIZE =  4\nEPOCHS_NUM = 30\n\nprint(f'Monet TFRecord files: {len(MONET_FILENAMES)}')\nprint(f'Monet image files: {n_monet_samples}')\nprint(f'Photo TFRecord files: {len(PHOTO_FILENAMES)}')\nprint(f'Photo image files: {n_photo_samples}')\nprint(f\"Batch_size: {BATCH_SIZE}\")\nprint(f\"Epochs number: {EPOCHS_NUM}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"IMAGE PRE-PROCESSING\n\n* Resizing image\n* Normalizing the images to [-1, 1]"},{"metadata":{"trusted":true},"cell_type":"code","source":"IMAGE_SIZE = [256, 256]\n\ndef decode_image(image):\n    image = tf.image.decode_jpeg(image, channels=3)\n    image = (tf.cast(image, tf.float32) / 127.5) - 1\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    return image\n\ndef read_tfrecord(example):\n    tfrecord_format = {\n        \"image_name\": tf.io.FixedLenFeature([], tf.string),\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"target\": tf.io.FixedLenFeature([], tf.string)\n    }\n    example = tf.io.parse_single_example(example, tfrecord_format)\n    image = decode_image(example['image'])\n    return image","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Random jittering and mirroring to the training dataset. These are some of the image augmentation techniques that avoids overfitting, Random jittering performs:\n\n* Resize an image to bigger height and width\n* Randomly crop to the target size\n* Randomly rotate the image"},{"metadata":{"trusted":true},"cell_type":"code","source":"def data_augment(image):\n    p_rotate = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_spatial = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    p_crop = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n    if p_crop > .5:\n        image = tf.image.resize(image, [286, 286]) #resizing to 286 x 286 x 3\n        image = tf.image.random_crop(image, size=[256, 256, 3]) # randomly cropping to 256 x 256 x 3\n        if p_crop > .9:\n            image = tf.image.resize(image, [300, 300])\n            image = tf.image.random_crop(image, size=[256, 256, 3])\n    \n    if p_rotate > .9:\n        image = tf.image.rot90(image, k=3) # rotate 270º\n    elif p_rotate > .7:\n        image = tf.image.rot90(image, k=2) # rotate 180º\n    elif p_rotate > .5:\n        image = tf.image.rot90(image, k=1) # rotate 90º\n        \n        ## random mirroring\n    if p_spatial > .6:\n        image = tf.image.random_flip_left_right(image)\n        image = tf.image.random_flip_up_down(image)\n        if p_spatial > .9:\n            image = tf.image.transpose(image)\n    \n    return image\n\ndef load_dataset(filenames):\n    dataset = tf.data.TFRecordDataset(filenames)\n    dataset = dataset.map(read_tfrecord, num_parallel_calls=AUTO)\n    return dataset\n\ndef get_gan_dataset(monet_files, photo_files, augment=None, repeat=True, shuffle=True, batch_size=1):\n\n    monet_ds = load_dataset(monet_files)\n    photo_ds = load_dataset(photo_files)\n    \n    if augment:\n        monet_ds = monet_ds.map(augment, num_parallel_calls=AUTO)\n        photo_ds = photo_ds.map(augment, num_parallel_calls=AUTO)\n        \n    if repeat:\n        monet_ds = monet_ds.repeat()\n        photo_ds = photo_ds.repeat()\n    if shuffle:\n        monet_ds = monet_ds.shuffle(2048)\n        photo_ds = photo_ds.shuffle(2048)\n        \n    monet_ds = monet_ds.batch(batch_size, drop_remainder=True)\n    photo_ds = photo_ds.batch(batch_size, drop_remainder=True)\n    monet_ds = monet_ds.cache()\n    photo_ds = photo_ds.cache()\n    monet_ds = monet_ds.prefetch(AUTO)\n    photo_ds = photo_ds.prefetch(AUTO)\n    \n    gan_ds = tf.data.Dataset.zip((monet_ds, photo_ds))\n    \n    return gan_ds","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Adding the image preprocessing to the pipeline."},{"metadata":{"trusted":true},"cell_type":"code","source":"full_dataset = get_gan_dataset(MONET_FILENAMES, PHOTO_FILENAMES, augment=data_augment, repeat=True, shuffle=True, batch_size=BATCH_SIZE)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Visualizing the data and checking that the upload is successful."},{"metadata":{"trusted":true},"cell_type":"code","source":"example_monet , example_photo = next(iter(full_dataset))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Visualizing the real photo\nplt.subplot(121)\nplt.title('Real photo')\nplt.imshow(example_photo[2] * 0.5 + 0.5)\n\n# Visualizing the Monet painting\nplt.subplot(122)\nplt.title('Monet painting')\nplt.imshow(example_monet[2]* 0.5 + 0.5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"MODEL\n\nTo build the model, we will follow the following steps:\n\n* Build the Generator\n* Build the Discriminador\n* loss functions\n\n> Discriminator loss\n\n> Generator loss\n\n> Adversary loss\n\n> Cycle loss\n\n> Identity loss\n\n* Define the optimizers"},{"metadata":{},"cell_type":"markdown","source":"Build the Generator:\nThe architecture of the generator is a modified U-Net, consisting of an encoder block and a decoder block, each of them is made up of simpler blocks of layers:\nEach block of the encoder, we call it downsample-k where k denotes the number of filters, consisting of the following layers:\n* Convolution\n* Instance Normalization (not apply to the first block)\n* Leaky ReLU"},{"metadata":{"trusted":true},"cell_type":"code","source":"OUTPUT_CHANNELS = 3\n\ndef downsample(filters, size, apply_instancenorm=True):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n     \n    result = keras.Sequential()\n    # Convolutional layer\n    result.add(layers.Conv2D(filters, size, strides=2, padding='same',\n                             kernel_initializer=initializer, use_bias=False))\n # Normalization layer\n    if apply_instancenorm:\n        result.add(tfa.layers.InstanceNormalization(gamma_initializer=gamma_init))\n # Activation layer\n    result.add(layers.LeakyReLU())\n\n    return result","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The decoder(upsampling) is made up of:\n* Transposed Convolution\n* Instance Normalization\n* Dropout (applied to the first 3 blocks)\n* ReLU\n\nSkip connections exist between encoder and decoder."},{"metadata":{"trusted":true},"cell_type":"code","source":"def upsample(filters, size, apply_dropout=False):\n     # Normalization layer\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n     # Transpose convolutional layer\n    result = keras.Sequential()\n    result.add(layers.Conv2DTranspose(filters, size, strides=2,\n                                      padding='same',\n                                      kernel_initializer=initializer,\n                                      use_bias=False))\n#Instance Normalization\n    result.add(tfa.layers.InstanceNormalization(gamma_initializer=gamma_init))\n# Dropout layer\n    if apply_dropout:\n        result.add(layers.Dropout(0.5))\n# Activation layer\n    result.add(layers.ReLU())\n\n    return result","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Generator:\n* The execution of a downsample-k block\n* Then, the execution of an upsampleD-k block (with Dropout)\n* Then, the execution of an upsample-k block (without Dropout)\n* The last thing is the Transposed Convolution of 3 filters to convert the output into a 256X256 image by 3 channels"},{"metadata":{},"cell_type":"markdown","source":"\nAll convolutional layers of downsample, have the parameter strides = 2, which causes the dimensions to be reduced by half, likewise, the Transposed Convolution layers of upsample also have the parameter strides = 2 so the dimensions are doubled. In the first two dimensions, not counting the Batch size dimension."},{"metadata":{"trusted":true},"cell_type":"code","source":"def Generator():\n    inputs = layers.Input(shape=[256,256,3])\n\n    # bs = batch size\n    down_stack = [\n        downsample(64, 4, apply_instancenorm=False), # (bs, 128, 128, 64)\n        downsample(128, 4), # (bs, 64, 64, 128)\n        downsample(256, 4), # (bs, 32, 32, 256)\n        downsample(512, 4), # (bs, 16, 16, 512)\n        downsample(512, 4), # (bs, 8, 8, 512)\n        downsample(512, 4), # (bs, 4, 4, 512)\n        downsample(512, 4), # (bs, 2, 2, 512)\n        downsample(512, 4), # (bs, 1, 1, 512)\n    ]\n\n    up_stack = [\n        upsample(512, 4, apply_dropout=True), # (bs, 2, 2, 1024)\n        upsample(512, 4, apply_dropout=True), # (bs, 4, 4, 1024)\n        upsample(512, 4, apply_dropout=True), # (bs, 8, 8, 1024)\n        upsample(512, 4), # (bs, 16, 16, 1024)\n        upsample(256, 4), # (bs, 32, 32, 512)\n        upsample(128, 4), # (bs, 64, 64, 256)\n        upsample(64, 4), # (bs, 128, 128, 128)\n    ]\n\n    initializer = tf.random_normal_initializer(0., 0.02)\n    last = layers.Conv2DTranspose(OUTPUT_CHANNELS, 4,\n                                  strides=2,\n                                  padding='same',\n                                  kernel_initializer=initializer,\n                                  activation='tanh') # (bs, 256, 256, 3)\n\n    x = inputs\n\n    # Downsampling through the model\n    skips = []\n    for down in down_stack:\n        x = down(x)\n        skips.append(x)\n\n    skips = reversed(skips[:-1])\n\n    # Upsampling and establishing the skip connections\n    for up, skip in zip(up_stack, skips):\n        x = up(x)\n        x = layers.Concatenate()([x, skip])\n\n    x = last(x)\n\n    return keras.Model(inputs=inputs, outputs=x)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Testing the model and show the architecture of generator "},{"metadata":{"trusted":true},"cell_type":"code","source":"generator_g = Generator()\ntf.keras.utils.plot_model(generator_g, show_shapes=True, dpi=64)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"photo = (example_photo[0,...] * 0.5 + 0.5)\nplt.imshow(photo, vmin=0, vmax=255) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"example_gen_output_y = generator_g(photo[tf.newaxis,...], training=False)\nplt.imshow(example_gen_output_y[0]) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# We pass the denormalized photo so that some result can be seen, since the model is not trained\n#photo = example_photo[0,...] (example_photo[2] * 0.5 + 0.5)\nphoto = (example_photo[0,...] * 0.5 + 0.5)\nexample_gen_output_y = generator_g(photo[tf.newaxis,...], training=False)\n\nplt.subplot(1,2,1)\nplt.imshow(photo, vmin=0, vmax=255) \n\nplt.subplot(1,2,2)\nplt.imshow(example_gen_output_y[0]) \n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Build the Discriminator\n\nThe task of the discriminator is whether an input image, which is (the output of a generator), is original or fake!\n\nIt can be seen that the architecture of the discriminator is a convolution network of the PatchGAN type, instead of returning whether the image is real or not, this architecture returns whether pieces of the image can be considered real or false. \nAs we mentioned in the generator, the encoder is made up of downsample-k blocks, the block performs an image compression operation (downsample). It consists of the following layers:\n* Convolution\n* Instance Normalization (not apply to the first block)\n* Leaky ReLU\n\nAll convolutional layers of downsampling, have the parameter strides = 2, which causes the dimensions to be reduced by half.\nThe shape of the discriminator output layer is (batch_size, 30, 30, 1), each 30x30 patch of the output sorts a 70x70 portion of the input image"},{"metadata":{"trusted":true},"cell_type":"code","source":"def Discriminator():\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    inp = layers.Input(shape=[256, 256, 3], name='input_image')\n\n    x = inp\n\n    down1 = downsample(64, 4, False)(x) # (bs, 128, 128, 64)\n    down2 = downsample(128, 4)(down1) # (bs, 64, 64, 128)\n    down3 = downsample(256, 4)(down2) # (bs, 32, 32, 256)\n\n    zero_pad1 = layers.ZeroPadding2D()(down3) # (bs, 34, 34, 256)\n    conv = layers.Conv2D(512, 4, strides=1,\n                         kernel_initializer=initializer,\n                         use_bias=False)(zero_pad1) # (bs, 31, 31, 512)\n\n    norm1 = tfa.layers.InstanceNormalization(gamma_initializer=gamma_init)(conv)\n\n    leaky_relu = layers.LeakyReLU()(norm1)\n\n    zero_pad2 = layers.ZeroPadding2D()(leaky_relu) # (bs, 33, 33, 512)\n\n    last = layers.Conv2D(1, 4, strides=1,\n                         kernel_initializer=initializer)(zero_pad2) # (bs, 30, 30, 1)\n\n    return tf.keras.Model(inputs=inp, outputs=last)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Testing the model and show the architecture of the discriminator\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"discriminator_y = Discriminator()\ntf.keras.utils.plot_model(discriminator_y, show_shapes=True, dpi=64)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"photo = example_photo[0,...]* 0.5 + 0.5\nphoto_1=photo \nplt.imshow(photo_1, vmin=0, vmax=255) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nexample_gen_output_y = generator_g(photo_1[tf.newaxis,...], training=False)\nplt.imshow(example_gen_output_y[0,...])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# example_disc_out = discriminator_y([example_photo, example_gen_output_y], training=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# example_disc_out = discriminator_y([example_photo, example_gen_output_y], training=False)\n# m = example_disc_out[0,...,-1].numpy()*1000\n# im = plt.imshow(m, vmin=-20, vmax=20, cmap='RdBu_r')\n# plt.colorbar(im,fraction=0.046, pad=0.04)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# print(example_disc_out.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# # We pass the denormalized photo so that some result can be seen, since the model is not trained\n# photo = example_photo[0,...]* 0.5 + 0.5\n# example_gen_output_y = generator_g(photo[tf.newaxis,...], training=False)\n# example_disc_out = discriminator_y([example_photo, example_gen_output_y], training=False)\n\n# print(example_disc_out.shape)\n\n# plt.figure(figsize=(10,10))\n\n# plt.subplot(1,3,1)\n# plt.imshow(photo, vmin=0, vmax=255) \n\n# plt.subplot(1,3,2)\n# plt.imshow(example_gen_output_y[0,...]) \n\n# plt.subplot(1,3,3)\n# m = example_disc_out[0,...,-1].numpy()*1000\n# im = plt.imshow(m, vmin=-20, vmax=20, cmap='RdBu_r')\n# plt.colorbar(im,fraction=0.046, pad=0.04)\n\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Build the complete Model\nThe complete model consists of:  \n* 2 generators (G_MONET and G_PHOTO)\nThe G_MONET generator learns how to transform a photograph into a Monet painting.\n\nThe G_PHOTO generator learns how to transform a Monet painting into a photograph.\n\nAs the images are not paired, it is necessary to use two cycles\nInput Photo -> G_MONET -> Fake Monet -> G_PHOTO -> Cycle Photo\nInput Monet -> G_PHOTO -> Fake Photo -> G_MONET -> Cycle Monet  \n\n* 2 discriminators (D_MONET AND D_PHOTO)\nThe D_MONET discriminator learns to differentiate if a Monet painting is real or fake, serving to calculate the adversary loss and improve the G_MONET generator.\n\nThe D_PHOTO discriminator learns to differentiate if a photo is real or fake, serving to calculate the adversary loss and improve the G_PHOTO generator.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    monet_generator = Generator() # transforms photos to Monet-esque paintings\n    photo_generator = Generator() # transforms Monet paintings to be more like photos\n\n    monet_discriminator = Discriminator() # differentiates real Monet paintings and generated Monet paintings\n    photo_discriminator = Discriminator() # differentiates real photos and generated photos\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"to_monet = monet_generator(example_photo)\n\nplt.subplot(1, 2, 1)\nplt.title(\"Original Photo\")\nplt.imshow(example_photo[0] * 0.5 + 0.5)\n\nplt.subplot(1, 2, 2)\nplt.title(\"Monet-esque Photo\")\nplt.imshow(to_monet[0] * 0.5 + 0.5)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# to_photo = photo_generator(example_monet)\n\n# plt.subplot(1, 2, 1)\n# plt.title(\"Original Monet\")\n# plt.imshow(example_monet[0] * 0.5 + 0.5)\n\n# plt.subplot(1, 2, 2)\n# plt.title(\"Monet-esque Photo\")\n# plt.imshow(to_photo [0] * 0.5 + 0.5)\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#example_disc_out = discriminator_y([example_photo, example_gen_output_y], training=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plt.subplot(1,3,3)\n# m = example_disc_out[0,...,-1].numpy()*1000\n# im = plt.imshow(m, vmin=-20, vmax=20, cmap='RdBu_r')\n# plt.colorbar(im,fraction=0.046, pad=0.04)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Example\n\nGenerator part:\n* Starting with the photo, a simulation of a Monet painting is generated and then from this simulation, an attempt is made to generate the original photo\n\n* Starting from the Monet, a photo simulation is generated and then from this simulation, an attempt is made to generate the original Monet"},{"metadata":{"trusted":true},"cell_type":"code","source":"photo = (example_photo[0,...] * 0.5 + 0.5)\nmonet = (example_monet[0,...] * 0.5 + 0.5)\n\n# From photo we generate Monet (fake) and regenerate the photo (cycle) again\nexample_gen_output_monet_fake = monet_generator(photo[tf.newaxis,...], training=False)\nexample_gen_output_photo_cycle =  photo_generator(example_gen_output_monet_fake, training=False)\n\n# We run the discriminator for Monet (fake)\nexample_disc_out_monet = monet_discriminator(example_gen_output_monet_fake, training=False)\n\n\n# From Monet we generate photo (fake) and regenerate Monet (cycle) again\nexample_gen_output_photo_fake =  photo_generator(monet[tf.newaxis,...], training=False)\nexample_gen_output_monet_cycle = monet_generator(example_gen_output_photo_fake, training=False)\n\n# We execute the discriminator for Photo (fake)\nexample_disc_out_photo = photo_discriminator(example_gen_output_photo_fake, training=False)\n\n\n# We present results, as the network is not trained, the outputs are not good, \n# but we modify the scala to be able to have some example images\n\nplt.figure(figsize=(10,10))\n\n# Input Photo\nplt.subplot(2,4,1)\nplt.imshow(photo, vmin=0, vmax=255) \n\n# Fake Monet\nplt.subplot(2,4,2)\n#m = example_gen_output_monet_fake[0,...].numpy()\n#print(np.min(m), np.max(m))\ncontrast = 100 \nplt.imshow(example_gen_output_monet_fake[0,...]*contrast) \n\n# Photo Cycle\nplt.subplot(2,4,3)\n#m = example_gen_output_photo_cycle[0,...].numpy()\n#print(np.min(m), np.max(m))\ncontrast = 100\nplt.imshow(example_gen_output_photo_cycle[0,...]*contrast) \n\n# Monet discriminator result\n#plt.subplot(2,4,4)\n#m = example_disc_out_monet[0,...,-1].numpy()\n#print(np.min(m), np.max(m))\n#contrast = 1000\n#im = plt.imshow(m*contrast, vmin=-20, vmax=20, cmap='RdBu_r')\n#plt.colorbar(im,fraction=0.046, pad=0.04)\n\n\n\n# Input Monet\nplt.subplot(2,4,5)\nplt.imshow(monet, vmin=0, vmax=255) \n\n# Fake Photo\nplt.subplot(2,4,6)\n#m = example_gen_output_photo_fake[0,...].numpy()\n#print(np.min(m), np.max(m))\ncontrast = 100\nplt.imshow(example_gen_output_photo_fake[0,...]*contrast) \n\n# Monet Cycle\nplt.subplot(2,4,7)\n#m = example_gen_output_monet_cycle[0,...].numpy()\n#print(np.min(m), np.max(m))\ncontrast = 100\nplt.imshow(example_gen_output_monet_cycle[0,...]*contrast) \n\n# Photo discriminator result  \n#plt.subplot(2,4,8)\n#m = example_disc_out_photo[0,...,-1].numpy()\n#print(np.min(m), np.max(m))\n#contrast = 1000\n#im = plt.imshow(contrast, vmin=-20, vmax=20, cmap='RdBu_r')\n#plt.colorbar(im,fraction=0.046, pad=0.04)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class CycleGan(keras.Model):\n    def __init__(\n        self,\n        monet_generator,\n        photo_generator,\n        monet_discriminator,\n        photo_discriminator,\n        lambda_cycle=15,\n    ):\n        super(CycleGan, self).__init__()\n        self.m_gen = monet_generator\n        self.p_gen = photo_generator\n        self.m_disc = monet_discriminator\n        self.p_disc = photo_discriminator\n        self.lambda_cycle = lambda_cycle\n        \n    def compile(\n        self,\n        m_gen_optimizer,\n        p_gen_optimizer,\n        m_disc_optimizer,\n        p_disc_optimizer,\n        gen_loss_fn,\n        disc_loss_fn,\n        cycle_loss_fn,\n        identity_loss_fn\n    ):\n        super(CycleGan, self).compile()\n        self.m_gen_optimizer = m_gen_optimizer\n        self.p_gen_optimizer = p_gen_optimizer\n        self.m_disc_optimizer = m_disc_optimizer\n        self.p_disc_optimizer = p_disc_optimizer\n        self.gen_loss_fn = gen_loss_fn\n        self.disc_loss_fn = disc_loss_fn\n        self.cycle_loss_fn = cycle_loss_fn\n        self.identity_loss_fn = identity_loss_fn\n        \n    def train_step(self, batch_data):\n        real_monet, real_photo = batch_data\n        \n        with tf.GradientTape(persistent=True) as tape:\n            # photo to monet back to photo\n            fake_monet = self.m_gen(real_photo, training=True)\n            cycled_photo = self.p_gen(fake_monet, training=True)\n\n            # monet to photo back to monet\n            fake_photo = self.p_gen(real_monet, training=True)\n            cycled_monet = self.m_gen(fake_photo, training=True)\n\n            # generating itself\n            same_monet = self.m_gen(real_monet, training=True)\n            same_photo = self.p_gen(real_photo, training=True)\n\n            # discriminator used to check, inputing real images\n            disc_real_monet = self.m_disc(real_monet, training=True)\n            disc_real_photo = self.p_disc(real_photo, training=True)\n\n            # discriminator used to check, inputing fake images\n            disc_fake_monet = self.m_disc(fake_monet, training=True)\n            disc_fake_photo = self.p_disc(fake_photo, training=True)\n\n            # evaluates generator loss\n            monet_gen_loss = self.gen_loss_fn(disc_fake_monet)\n            photo_gen_loss = self.gen_loss_fn(disc_fake_photo)\n\n            # evaluates total cycle consistency loss\n            total_cycle_loss = self.cycle_loss_fn(real_monet, cycled_monet, self.lambda_cycle) + self.cycle_loss_fn(real_photo, cycled_photo, self.lambda_cycle)\n\n            # evaluates total generator loss\n            total_monet_gen_loss = monet_gen_loss + total_cycle_loss + self.identity_loss_fn(real_monet, same_monet, self.lambda_cycle)\n            total_photo_gen_loss = photo_gen_loss + total_cycle_loss + self.identity_loss_fn(real_photo, same_photo, self.lambda_cycle)\n\n            # evaluates discriminator loss\n            monet_disc_loss = self.disc_loss_fn(disc_real_monet, disc_fake_monet)\n            photo_disc_loss = self.disc_loss_fn(disc_real_photo, disc_fake_photo)\n\n        # Calculate the gradients for generator and discriminator\n        monet_generator_gradients = tape.gradient(total_monet_gen_loss,\n                                                  self.m_gen.trainable_variables)\n        photo_generator_gradients = tape.gradient(total_photo_gen_loss,\n                                                  self.p_gen.trainable_variables)\n\n        monet_discriminator_gradients = tape.gradient(monet_disc_loss,\n                                                      self.m_disc.trainable_variables)\n        photo_discriminator_gradients = tape.gradient(photo_disc_loss,\n                                                      self.p_disc.trainable_variables)\n\n        # Apply the gradients to the optimizer\n        self.m_gen_optimizer.apply_gradients(zip(monet_generator_gradients,\n                                                 self.m_gen.trainable_variables))\n\n        self.p_gen_optimizer.apply_gradients(zip(photo_generator_gradients,\n                                                 self.p_gen.trainable_variables))\n\n        self.m_disc_optimizer.apply_gradients(zip(monet_discriminator_gradients,\n                                                  self.m_disc.trainable_variables))\n\n        self.p_disc_optimizer.apply_gradients(zip(photo_discriminator_gradients,\n                                                  self.p_disc.trainable_variables))\n        \n        return {\n            \"monet_gen_loss\": total_monet_gen_loss,\n            \"photo_gen_loss\": total_photo_gen_loss,\n            \"monet_disc_loss\": monet_disc_loss,\n            \"photo_disc_loss\": photo_disc_loss\n        }\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Sigmoid cross entropy is used to calculate the adversary losses in the discriminator and generator."},{"metadata":{},"cell_type":"markdown","source":"**Discriminator loss**\nThe discriminator loss function takes 2 inputs:\n\nFor the discriminator_monet will take as input:\nThe output of the discriminator_monet whose input is the real Monet of the training set\nThe output of the discriminator_monet whose input is the fake Monet generated by the generator_monet\n\nFor the discriminator_photo will take as input:\nThe output of the discriminator_photo whose input is the real photo of the training set\nThe output of the discriminator_photo whose input is the fake photo generated by the generator_photo\n\nThe calculation of the loss has two components:\nreal_loss compare the real image with a matrix of 1. (Real)\ngenerate_loss compare the fake image with a matrix of 0 (Fake)\nSo the total_loss is the sum of the real_loss and the generate_loss times 0.5"},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    def discriminator_loss(real, generated):\n        real_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(real), real)\n\n        generated_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.zeros_like(generated), generated)\n\n        total_disc_loss = real_loss + generated_loss\n\n        return total_disc_loss * 0.4\n    \n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Generator loss**\nThe generator loss has 3 terms:\n\n* Adversary loss\n* Cycle loss\n* Identity loss  "},{"metadata":{},"cell_type":"markdown","source":"**Generator adversary loss**\n\nThe output of the discriminator will be as input \n* For the loss of the generator_monet, the function will take the output of the discriminator_monet executed with fake Monet\n* For the loss of the generator_photo, the function will take the output of the discriminator_photo executed with a fake photo\n\nThe perfect generator will have the discriminator output only ones(REAL) Therefore, compare the generated image with a matrix of 1 to find the loss."},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    def generator_loss(generated):\n        return tf.keras.losses.BinaryCrossentropy(from_logits=True,\n                                                  reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(generated), generated)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Generator cycle consistency loss**\nTo make the network learn the correct mapping and the result is similar to the original input.\n\n\nInput Photo -> G_MONET -> Fake Monet -> G_PHOTO -> Cycle Photo\nGenerate a Monet style image from a photo, this generated image is passed as input to the second generator, which should generate a photo from a fake Monet style image.\n\nInput Monet -> G_PHOTO -> Fake Photo -> G_MONET -> Cycle Monet\nOn the other hand, an image is generated that aims to imitate a real photo from a Monet painting, this generated image is passed as an input to the second generator, which should generate the Monet painting again from the fake photo.\n\n\nTo calculate the cycle consistency loss:\nThe average absolute error for a photo is calculated between Input Photo and Cycle Photo\nThe mean absolute error for Monet is calculated between Input Monet and Cycle Monet\nThe cycle error will be the sum of both terms.\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    def calc_cycle_loss(real_image, cycled_image, LAMBDA):\n        loss1 = tf.reduce_mean(tf.abs(real_image - cycled_image))\n\n        return LAMBDA * loss1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Identity loss**\nThe loss of identity forces what the generator generates to resemble the input.\n\nFor the loss of the generator_monet, the function will take the training monet image and the output of the generator_monet with the same input (same_monet)\n\nFor the loss of the generator_photo, the function will take the training photo image and the output of the generator_photo with the same input (same_photo)\n\nThe loss will be the mean absolute error between the real image and the generated one."},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    def identity_loss(real_image, same_image, LAMBDA):\n        loss = tf.reduce_mean(tf.abs(real_image - same_image))\n        return LAMBDA * 0.5 * loss","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Initializing the optimizers for all the generators and the discriminators."},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    monet_generator_optimizer = tf.keras.optimizers.Adam(0.002, decay = 0, beta_1=0.5)\n    photo_generator_optimizer = tf.keras.optimizers.Adam(0.002, beta_1=0.5)\n\n    monet_discriminator_optimizer = tf.keras.optimizers.Adam(0.002, beta_1=0.5)\n    photo_discriminator_optimizer = tf.keras.optimizers.Adam(0.002, beta_1=0.5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n    cycle_gan_model = CycleGan(\n        monet_generator, photo_generator, monet_discriminator, photo_discriminator\n    )\n\n    cycle_gan_model.compile(\n        m_gen_optimizer = monet_generator_optimizer,\n        p_gen_optimizer = photo_generator_optimizer,\n        m_disc_optimizer = monet_discriminator_optimizer,\n        p_disc_optimizer = photo_discriminator_optimizer,\n        gen_loss_fn = generator_loss,\n        disc_loss_fn = discriminator_loss,\n        cycle_loss_fn = calc_cycle_loss,\n        identity_loss_fn = identity_loss\n    )\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cycle_gan_model.fit(\n    full_dataset,\n    epochs=EPOCHS_NUM,\n    steps_per_epoch=(max(n_monet_samples, n_photo_samples)//BATCH_SIZE),\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"example_gen_output_monet_fake = monet_generator(example_photo, training=False)\nexample_gen_output_photo_cycle = photo_generator(example_gen_output_monet_fake, training=False)\nexample_gen_output_photo_same = photo_generator(example_photo, training=False)\n\n\n# We execute the discriminator for Photo (real)\nexample_disc_out_photo_real = photo_discriminator(example_photo, training=False)\n\n# We run the discriminator for Monet (fake)\nexample_disc_out_monet_fake = monet_discriminator(example_gen_output_monet_fake, training=False)\n\n\n# from Monet we generate photo (fake) and regenerate Monet (cycle) again\nexample_gen_output_photo_fake = photo_generator(example_monet, training=False)\nexample_gen_output_monet_cycle = monet_generator(example_gen_output_photo_fake, training=False)\nexample_gen_output_monet_same = monet_generator(example_monet, training=False)\n\n# We run the discriminator for Monet (real)\nexample_disc_out_monet_real = monet_discriminator(example_monet, training=False)\n\n# We execute the discriminator for Photo (fake)\nexample_disc_out_photo_fake = photo_discriminator(example_gen_output_photo_fake, training=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"example_photo.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,10))\n\n# photo\nplt.subplot(4,4,1)\nplt.imshow(example_photo[0] * 0.5 + 0.5) \n\n# Monet \nplt.subplot(4,4,2)\nplt.imshow(example_gen_output_monet_fake[0] * 0.5 + 0.5) \n\n# Photo Cycle\nplt.subplot(4,4,3)\nplt.imshow(example_gen_output_photo_cycle[0] * 0.5 + 0.5) \n\n# Photo Same\nplt.subplot(4,4,4)\nplt.imshow(example_gen_output_photo_same[0] * 0.5 + 0.5) \n\n# Discriminador Photo (real)\nplt.subplot(4,4,5)\nm = example_disc_out_photo_real[0,...,-1].numpy()\nplt.imshow(m, vmin=np.min(m), vmax=np.max(m), cmap='RdBu_r')\n#plt.colorbar(im,fraction=0.046, pad=0.04)\n\n# Discriminador Monet (fake)\nplt.subplot(4,4,7)\nm = example_disc_out_monet_fake[0,...,-1].numpy()\nplt.imshow(m, vmin=np.min(m), vmax=np.max(m), cmap='RdBu_r')\n#plt.colorbar(im,fraction=0.046, pad=0.04)\n\n# Monet \nplt.subplot(4,4,9)\nplt.imshow(example_monet[0] * 0.5 + 0.5) \n\n# Foto generado\nplt.subplot(4,4,10)\nplt.imshow(example_gen_output_photo_fake[0] * 0.5 + 0.5) \n\n# Monet Cycle\nplt.subplot(4,4,11)\nplt.imshow(example_gen_output_monet_cycle[0] * 0.5 + 0.5) \n\n# Monet Same\nplt.subplot(4,4,12)\nplt.imshow(example_gen_output_monet_same[0] * 0.5 + 0.5) \n\n# Discriminador Monet (real)\nplt.subplot(4,4,13)\nm = example_disc_out_monet_real[0,...,-1].numpy()\nplt.imshow(m, vmin=np.min(m), vmax=np.max(m), cmap='RdBu_r')\n#plt.colorbar(im,fraction=0.046, pad=0.04)\n\n# Discriminador photo(fake)\nplt.subplot(4,4,15)\nm = example_disc_out_photo_fake[0,...,-1].numpy()\nplt.imshow(m, vmin=np.min(m), vmax=np.max(m), cmap='RdBu_r')\n#plt.colorbar(im,fraction=0.046, pad=0.04)\n\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# AUTOTUNE = tf.data.experimental.AUTOTUNE","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# _, ax = plt.subplots(2, 2, figsize=(12, 12))\n# for i, img in enumerate(example_photo.take(2)):\n#     prediction = monet_generator(img, training=False)[0].numpy()\n#     prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n#     img = (img[1] * 127.5 + 127.5).numpy().astype(np.uint8)\n\n#     ax[i, 0].imshow(img)\n#     ax[i, 1].imshow(prediction)\n#     ax[i, 0].set_title(\"Input Photo\")\n#     ax[i, 1].set_title(\"Monet-esque\")\n#     ax[i, 0].axis(\"off\")\n#     ax[i, 1].axis(\"off\")\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import PIL\ndef predict_and_save(input_ds, generator_model, output_path):\n    i = 1\n    for img in input_ds:\n        prediction = generator_model(img, training=False)[0].numpy() # make predition\n        prediction = (prediction * 127.5 + 127.5).astype(np.uint8)   # re-scale\n        im = PIL.Image.fromarray(prediction)\n        im.save(f'{output_path}{str(i)}.jpg')\n        i += 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# im2 = im2.reshape(1,256,256,3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# scaled = (im2.astype(np.float32) - 127.5) / 127.5\n# scaled = (im2.astype(np.float32) / 127.5) - 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plt.imshow(scaled[0,:,:,:])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# prediction = monet_generator(scaled, training=False)[0].numpy() # make predition\n# prediction = (prediction * 127.5 + 127.5).astype(np.uint8)   # re-scale\n# im = PIL.Image.fromarray(prediction)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# im","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# for img in load_dataset(PHOTO_FILENAMES).batch(1):\n#     img = img\n#     break;","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# s = np.asarray(img[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# s.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# plt.imshow(s)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# s = s.reshape(1,256,256,3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# prediction = monet_generator(s, training=False)[0].numpy() # make predition\n# prediction = (prediction * 127.5 + 127.5).astype(np.uint8)   # re-scale\n# im = PIL.Image.fromarray(prediction)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# im","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# im = PIL.Image.fromarray(img)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nos.makedirs('../images/') # Create folder to save generated images\n\npredict_and_save(load_dataset(PHOTO_FILENAMES).batch(1), monet_generator, '../images/')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import shutil\nshutil.make_archive('/kaggle/working/images/', 'zip', '../images')\n\nprint(f\"Generated samples: {len([name for name in os.listdir('../images/') if os.path.isfile(os.path.join('../images/', name))])}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}