\n
"},{"metadata":{},"cell_type":"markdown","source":"# Implementation"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\n\nimport tensorflow_addons as tfa\n\nfrom kaggle_datasets import KaggleDatasets\n\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\n\nimport numpy as np\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Set up the input pipeline\n"},{"metadata":{},"cell_type":"markdown","source":"Using TPUs\n\n* https://www.kaggle.com/philculliton/a-simple-petals-tf-2-2-notebook\n* https://www.kaggle.com/c/tpu-getting-started\n\n\nIt is not necessary to enable the TPUs during the implementation, to carry out the implementation I have left the notebook without any accelerator, and for the first tests of very few epoch and used GPU, only for large trainings I have enabled the TPU."},{"metadata":{"trusted":true},"cell_type":"code","source":"try:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n print('Device:', tpu.master())\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\n # strategy = tf.distribute.Strategy.experimental_run_v2(tpu)\nexcept:\n strategy = tf.distribute.get_strategy()\nprint('Number of replicas:', strategy.num_replicas_in_sync)\n \nAUTOTUNE = tf.data.experimental.AUTOTUNE\n\nprint(tf.__version__)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Load the dataset"},{"metadata":{},"cell_type":"markdown","source":"The data is read from the Tfrec files"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Google Cloud Storage \nGCS_PATH = KaggleDatasets().get_gcs_path()\nprint(GCS_PATH)\n\nMONET_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/monet_tfrec/*.tfrec'))\nprint('Monet files: ', str(len(MONET_FILENAMES))) \n#print(*MONET_FILENAMES, sep = \"\\n\")\n\nPHOTO_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/photo_tfrec/*.tfrec'))\nprint('Photo files: ', str(len(PHOTO_FILENAMES)))\n#print(*PHOTO_FILENAMES, sep = \"\\n\")\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Data set size"},{"metadata":{"trusted":true},"cell_type":"code","source":"# https://www.kaggle.com/dimitreoliveira/introduction-to-cyclegan-monet-paintings\n \nimport re #Regular expression operations\n\ndef count_data_items(filenames):\n n = [int(re.compile(r\"-([0-9]*)\\.\").search(filename).group(1)) for filename in filenames]\n return np.sum(n)\n\nMONET_DATASET_SIZE = count_data_items(MONET_FILENAMES)\nPHOTO_DATASET_SIZE = count_data_items(PHOTO_FILENAMES)\n\nprint('Monet image files: ', str(MONET_DATASET_SIZE))\nprint('Photo image files: ', str(PHOTO_DATASET_SIZE))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Functions are defined to read the images as they are, without any type of transformation"},{"metadata":{"trusted":true},"cell_type":"code","source":"def read_tfrecord(example):\n tfrecord_format = {\n \"image_name\": tf.io.FixedLenFeature([], tf.string),\n \"image\": tf.io.FixedLenFeature([], tf.string),\n \"target\": tf.io.FixedLenFeature([], tf.string)\n }\n example = tf.io.parse_single_example(example, tfrecord_format)\n image = tf.image.decode_jpeg(example['image'], channels=3)\n return image\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n dataset = tf.data.TFRecordDataset(filenames)\n dataset = dataset.map(read_tfrecord, num_parallel_calls=AUTOTUNE)\n return dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"monet_ds = load_dataset(MONET_FILENAMES, labeled=True)\nphoto_ds = load_dataset(PHOTO_FILENAMES, labeled=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We check that the upload will be done correctly"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,10))\n\nfor i, img in enumerate(monet_ds.take(4)):\n plt.subplot(2,4,i+1)\n plt.imshow(img) \n\nfor i, img in enumerate(photo_ds.take(4)):\n plt.subplot(2,4,i+5)\n plt.imshow(img) \n \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The photo file is divided into train, test and validation\n\nIn this notebook we will only make a first approach to the architecture, so we will not use the test and validation sets to evaluate the model."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_size = int(0.7 * PHOTO_DATASET_SIZE)\ntest_size = int(0.15 * PHOTO_DATASET_SIZE)\nval_size = int(0.15 * PHOTO_DATASET_SIZE)\nBUFFER_SIZE = 1000\n\nphoto_ds = photo_ds.shuffle(BUFFER_SIZE)\n\ntrain_photo_3dim_ds = photo_ds.take(train_size)\n\ntest_photo_3dim_ds = photo_ds.skip(train_size)\ntest_photo_3dim_ds = photo_ds.take(test_size)\n\nval_photo_3dim_ds = photo_ds.skip(train_size+test_size)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\nMonet's paints data will all be used for training"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_size = int(0.7 * MONET_DATASET_SIZE)\ntest_size = int(0.15 * MONET_DATASET_SIZE)\nval_size = int(0.15 * MONET_DATASET_SIZE)\nBUFFER_SIZE = 1000\n\nmonet_ds = monet_ds.shuffle(BUFFER_SIZE)\n\ntrain_monet_3dim_ds = monet_ds\n\n#train_monet_3dim_ds = monet_ds.take(train_size)\n\n#test_monet_3dim_ds = monet_ds.skip(train_size)\n#test_monet_3dim_ds = monet_ds.take(test_size)\n\n#val_monet_3dim_ds = monet_ds.skip(train_size+test_size)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### IMAGE PRE-PROCESSING\n\n* Resizing image (In this case it would not be necessary to do it, because the images are already in the necessary size. But with this step if we wanted to add new images it would not be necessary to scale them previously)\n\n* Normalizing the images to [-1, 1] \n\n* Random jittering and mirroring to the training dataset. These are some of the image augmentation techniques that avoids overfitting, Random jittering performs:\n * Resize an image to bigger height and width\n * Randomly crop to the target size\n * Randomly flip the image horizontally"},{"metadata":{},"cell_type":"markdown","source":"We define preprocessing functions."},{"metadata":{"trusted":true},"cell_type":"code","source":"IMG_HEIGHT = 256\nIMG_WIDTH = 256\n\n\ndef resize(image):\n image = tf.image.resize(image, [IMG_HEIGHT, IMG_WIDTH])\n return image\n\n# normalizing the images to [-1, 1]\ndef normalize(image):\n image = tf.cast(image, tf.float32)\n image = (image / 127.5) - 1\n return image\n\ndef denormalize(image):\n image = tf.cast(image, tf.float32)\n image = (image * 0.5 + 0.5) * 255 # Range 0 to 1 and then to range 0..255\n image = tf.cast(image, tf.int32)\n return image\n\ndef random_crop(image):\n cropped_image = tf.image.random_crop(\n image, size=[IMG_HEIGHT, IMG_WIDTH, 3])\n\n return cropped_image\n\ndef random_jitter(image):\n # resizing to 286 x 286 x 3\n image = tf.image.resize(image, [286, 286],\n method=tf.image.ResizeMethod.NEAREST_NEIGHBOR)\n\n # randomly cropping to 256 x 256 x 3\n image = random_crop(image)\n\n # random mirroring\n image = tf.image.random_flip_left_right(image)\n\n return image\n\ndef preprocess_image_train(image):\n image = resize(image)\n image = random_jitter(image)\n image = normalize(image)\n return image\n\ndef preprocess_image_test(image):\n image = resize(image)\n image = normalize(image)\n return image","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Image preprocessing is added to the pipeline, depending on the set, train or test/validation\n\nThe batch dimension is added"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_photo_ds = train_photo_3dim_ds.map(\n preprocess_image_train, num_parallel_calls=AUTOTUNE).cache().shuffle(\n BUFFER_SIZE).batch(1)\n\ntest_photo_ds = test_photo_3dim_ds.map(\n preprocess_image_test, num_parallel_calls=AUTOTUNE).cache().shuffle(\n BUFFER_SIZE).batch(1)\n\nval_photo_ds = val_photo_3dim_ds.map(\n preprocess_image_test, num_parallel_calls=AUTOTUNE).cache().shuffle(\n BUFFER_SIZE).batch(1)\n\n\ntrain_monet_ds = train_monet_3dim_ds.map(\n preprocess_image_train, num_parallel_calls=AUTOTUNE).cache().shuffle(\n BUFFER_SIZE).batch(1)\n\n# test_monet_ds = test_monet_3dim_ds.map(\n# preprocess_image_test, num_parallel_calls=AUTOTUNE).cache().shuffle(\n# BUFFER_SIZE).batch(1)\n\n# val_monet_ds = val_monet_3dim_ds.map(\n# preprocess_image_test, num_parallel_calls=AUTOTUNE).cache().shuffle(\n# BUFFER_SIZE).batch(1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It is checked again that the upload will be successful"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,10))\n\nfor i, img in enumerate(train_monet_ds.take(4)):\n plt.subplot(2,4,i+1)\n plt.imshow(denormalize(img[0,...]), vmin=0, vmax=255) # first dimension (batch) is eliminated and we denormalize the image \n \nfor i, img in enumerate(train_photo_ds.take(4)):\n plt.subplot(2,4,i+5)\n plt.imshow(denormalize(img[0,...]), vmin=0, vmax=255) # first dimension (batch) is eliminated and we denormalize the image\n \nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Examples are saved for implementation testing"},{"metadata":{"trusted":true},"cell_type":"code","source":"example_photo = next(iter(test_photo_ds)) # includes batch dimension and the photo processed according to the pipeline (test set only normalization)\nexample_monet = next(iter(train_monet_ds)) # includes batch dimension and the photo processed according to the pipeline\n\nplt.subplot(1,2,1)\nplt.imshow(denormalize(example_photo[0,...]), vmin=0, vmax=255) # first dimension (batch) is eliminated and we denormalize the image\n\nplt.subplot(1,2,2)\nplt.imshow(denormalize(example_monet[0,...]), vmin=0, vmax=255) # first dimension (batch) is eliminated and we denormalize the image\n\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Test random jitter"},{"metadata":{"trusted":true},"cell_type":"code","source":"example_monet_origin = denormalize(example_monet[0,...])\n\nplt.figure(figsize=(15,15))\nfor i in range(4):\n image = random_jitter(example_monet_origin)\n plt.subplot(1, 4, i+1)\n plt.imshow(image)\n plt.axis('off')\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## MODEL\n\nTo build the model we will follow the following steps:\n1. Build the Generator\n1. Build the Discriminador\n1. Definition of loss functions\n * Discriminator loss\n * Generator loss\n * Adversary loss\n * Cycle loss\n * Identity loss\n1. Define the optimizers\n"},{"metadata":{},"cell_type":"markdown","source":"### Build the Generator\n\nThe original paper CycleGAN uses a modified based on resnet generator. In this Notebook, a modified unet generator is used for simplicity, as in \n\nhttps://www.tensorflow.org/tutorials/generative/cyclegan\n\nThe architecture of the generator is a modified U-Net, consisting of an encoder block and a decoder block, each of them is made up of simpler blocks of layers:\n\n* Each block of the encoder, we call it downsample-k where k denotes the number of filters, this block performs an image compression operation (downsample). It consists of the following layers \n * Convolution \n * Instance Normalization (not apply to the first block)\n * Leaky ReLU\n* In the decoder we can find two types of blocks, depending on whether a dropout operation is performed or not, we will call each one of them upsampleD-k and upsample-k, since they are performing a decompression. Each of these blocks is made up of:\n * Transposed Convolution\n * Instance Normalization\n * Dropout (applied to the first 3 blocks) \n * ReLU \n* Skip connections exist between encoder and decoder.\n\n\n\n","attachments":{"image.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAOwAAACdCAYAAAC3rZRwAAAQaUlEQVR4Ae2df2xUVRbH32g18UdijIl/q/EfEy2JhOgfGnbUPxYwVFiD0zqsho2EFWOqu7DURIwo4KBZHGOg0YQGobGERIqJW+PC7tKYWDSjMrtSYUtola6h0oEOQoUpPZt7X9/0vd43M2c6787cmflOMn1vzjvv3nO/53zm/ZjX9yzCCwpAgapRwKqaSGs0UMuyyP1Oy3GmPLaUz9g7o9PrDbqWu9va0Oe3pssZs1WnAIA1IGUdTSGyrKgSiYAv32u8vyMLttuvL9ZI425DjvnGAu3nWA3mCiqQvyIqGFg9dS2ADVmLlSGHrbBicxvSyXZKZjIS2nta92cXJeL513McC30hOH6YmqMAgDUgF6UAm0gTpRLbJbTOHvBMYDvaojbU4ZVytBOpkxSe2hWPx+MUi3UYoAJC4CgAYDkqafYpFVgRXlvIPqYV825gm0IWrezslyMYPriBLMuGljL9EmLNQ0PzASsAYAMWdDbNBQGs6FeecGpqp3+//dBUGGlpE6eeMpkMEdkns8TCyfEEgJ1Nsiq8DoCtcAJE972x3/gew1pTx7ADnSttGKd2Y+0zyUTiGFbsEjuvzGC39GtqtNPqQJmhDI2Pj2ffwt9ZJubHRoadJjA1XAEAa0CCBHjqCaAUWU2deaObCaxwbm+a3jUmsk9IHRUb16nXgbZ75Jwb2PbGkLMYU8MVALCGJKhtjkVWo318mUkP+wDsDXS4P0HiZxmrMUr/u+hd5oZ/pDcu2xIb4uH+HrKanBNMNsxi22pZjd4G8MlYBQCsQamZSA1TZ+cO2t/zReBRDQ4O0Ejatamd6mFgYCDwvtCgPgUArD5t0TIUCFwBABu4pGgQCuhTAMDq0xYtQ4HAFQCwgUuKBqGAPgUArD5t0TIUCFwBABu4pGgQCuhTAMDq0xYtQ4HAFQCwgUuKBqGAPgWqAtiGhgZ5tY68uH3GHRqKsQ0NDelTEi2XTYHTp08HUg/uK8LKFnyJHVUNsKmzY1TKO9LcTAC2xGoxZHUBbCQSKTkaAFuyhP4NiC1sKbCKdQGsv7bVaAWwhmcNwBqeoDKHB2DLLHix3QHYYhWrbX8Aa3h+AazhCSpzeAC2zIIX2x2ALVax2vAfGRmhkydPKu+vvvoKJ51MTjGANTk7+mJraWmhRx55hFpanvS8ly79HYDVJ3vpLQPY0jWsxhYEsP3fH1d+ITh2fADAmpxQAGtydvTFBmBVbXHhhKoJLIYoAGDVRABYVRNYDFEAwKqJALCqJrAYogCAVRMBYFVNYDFEAQCrJgLAqprAYogCAFZNBIBVNYHFEAUArJoIAKtqAoshCgBYNREAVtUEFkMUALBqIgCsqgkshigAYNVEAFhVE1gMUQDAqokAsKomsBiiAIBVEwFgVU1gMUQBAKsmAsCqmsBiiAIAVk0EgFU18bU8+OCDgdxas6ury7f9ajHu27cvEB3mzZtXcMgAVpUIwKqa+FoEsKXeufGvW9+mWgA2Ho/7alSMEcAWo9a0L4Cd1iLvHIC15RFbWACbt1S0LgSwTHkBLIBllopWNwDLlBfAAlhmqWh1A7BMeQEsgGWWilY3AMuUt56A7e3tlWeCb731Vpr5vummm3AMy6wZHW4AlqlqvQG7Zu1ffM+K79rdCWCZNaPDDcAyVQWw9tMDASyzYDS5AVimsAAWwDJLRasbgGXKC2ABLLNUtLoBWKa8ABbAMktFqxuAZcoLYAEss1S0ugFYprwAFsAyS0WrG4BlygtgASyzVLS6AVimvAAWwDJLRasbgGXKC2ABLLNUtLoBWKa8ABbAMktFqxuAZcoLYAEss1S0ugFYprwAFsAyS0WrG4BlygtgASyzVLS6AVimvAAWwDJLRasbgGXKC2ABLLNUtLoBWKa8ABbAMktFqxuAZcoLYAEss1S0ugFYprwAFsAyS0WrG4BlygtgASyzVLS6AVimvAAWwDJLRasbgGXKC2ABLLNUtLoBWKa8JgC7du3aQB5E9fzzz+cdtbjNKe6amFeiii0EsEzpTQH2yy+/ZEbs75ZMJgnA2tpYVlWUvyeRVRFxQ0OD7z1yi3maXKS5mYaGhjyDL+YDgMUucTH1cqKnXe4NrYztVVZLHuymE6MZxc4xsIHt2xuXAbTGOux20wPUaFkUXdlG45yeSvABsLZ4YpcYW9gxOnZ8gCKRSAkVZa+qaws70huje1o7ZSeij85+NyEpyVE4nphV/CxgWy2LdvQOyg7iTRZZVqN8C4MIKGQtnlXn3JUArK0UgLW38qYD63wRTI4nJR/tibFsqaeT9pb34EjWVNRMQWBFB4vjfdlG++KPySCGhSXTL+cb23qyy3XMAFhbVQBrPrCT4wkKtdo8bA+LjZtFaRcU7XKDVxA71xre2YJrxhtDnjVWWnYQHmMAH8STyaPRqG9L5Qb2hhtuoMOHD3tiKfcx7N1330179uzxxFBuYJ98Mkovv7zec/6g3I/q2PDa67R8+e89MVRiC3v99dfTpUuXPPko9EHAalmtHjfbFvbYivlQENiZjYkOQ+H2meZAPr/33nvyG+nQoUOe9soNrOh8zpw5dPPNN9PExISMpdzAik4XL15MYuznz5+XMZQbWHFSb9WqP8qcJJP/kdCUG9j+74/TG2/EZAyf/f2gjKESwJ47d46uuuoqevzxx2UuCv4Z3i9jdu8OE6WlLbyht+DquRyKBPaU7DDel8rVHo2NjdG+ffuKfu/cuTO7jv0tZNHk5KTsJyhgjx07lu0jX4yvvPKK9Gtvt483br/9dgoS2Hx9O8s2bdokY/jwww+l5kKTIIF1+vGbbty4kR57bAm9EdtCAtAPdu3OxhAksH59u21C842bNskYRL9OXQQJrNiTcveZa/7999+XfkuXLpVxdHd352RALDjSHpV+7nPBYwm7ng7lxidvm2JhUcAOdrfKINz75G0h7z76mTNn5OMI4/F4UdNXX3016+8k5sqVK3IAQQH73XffZfvIF19zc7P027x5c7ZIggQ2X9/OshUrVsgYtm7dmo0hSGCdfvym4nfahx5+mFatepY2b47Jt5OTIIH169ttmzt3Lj377Golhu+P/Tews8SffPIJqyZeeukl6bdkyRKZj3fffTcvXM7xq9spPnVM67YVO18A2MGpYrH3ucXPOCJx0y+xxfU/7pz24c/19PTI9js77VPizppBAVvM77ALFiyga6+9li5cuCDDCBJYZ1yFpqtXr5Z6nDp1SroGCWy+vt1XOr32+kYZwz/++a+K7RI7W9edH+yq2C7x5cuX6cYbb6T7778/n3TZZcN/W+dhJZV09hBmf/wqGnfTl+0sOzN1Fjj2xTB1tDZS2377NHVnYoROHTnoCSi7zixntm3bRvPnz/ddu9zAXnPNNfTRRx95Yik3sLfddhu99dZbnhjKDezChYtoxYo/SEici1SC3MJ6BufzoaWlhf705zUUDj/kiSHIXWKfbn1NoVCIRkdHfZflMsai4udPeyMXbbX3TsOx2R+/in7yAys8Mmnq6+2l/uHpHeFEXy/1JfpzxRm4vdzA+g2g3MD6xVBuYB1I3dNyAytOOrn7F/OVANYvH/lsiT7vhRFHOuxj2qPug9p8DeRYVhjYHCuW0wxgbbUBrPm/w4pMNYXsrar7Cid7S+v9iWc2DAFYpmrYwuJaYmap2MBGN2Tdw8q5n+yiomcALFMyAAtgmaVClHFO1tpb2rb24K4EBLDMLABYAMssFa1uAJYprynAOmcdS5ni/2HtpAsNq+1VFRHjpFN5y8r9O+zMM7Q4S1zeXMzsDcDOVCTHZxO2sDlCC9wMYAOXNLAGASxTSgCLY1hmqWh1A7BMeQEsgGWWilY3AMuUF8ACWGapaHUDsEx5ASyAZZaKVjcAy5QXwAJYZqlodQOwTHkBLIBllopWNwDLlBfAAlhmqWh1A7BMeQEsgGWWilY3AMuUF8ACWGapaHUDsEx5ASyAZZaKVjcAy5QXwAJYZqlodQOwTHlrEdiREf/nReBaYmZRVMANwDJFr0VgP/74Y3mTsL17vU9YA7DMoqiAG4Blil6LwDpDX7RoEYk7Rf7yyy/SBGAdZcybAlhmToIE9s477yTxVAHx7JxbbrlFRuD8M/V9991H4s77Bw4cyN5GViy7ePEiLVy4kF588UX65ptvPMt+/PFHWr58OT311FP0ww8/eJYdOXKEWltb6dFHH5X3WHb6EdOnn35a+op5553JZAjAMouiAm4Alil6kMAyuyybm3j8xNVXXy0fsyI6BbBlk77ojgAsU7JaBPbTTz+VW9bdu3d7VACwHjmM+gBgmemoRWB/+ukn39EDWF9ZjDACWGYaahHYXEMHsLmUqbwdwDJzAGBx4QSzVLS6AVimvAAWwDJLRasbgGXKC2ABLLNUtLoBWKa8ABbAMktFqxuAZcoLYAEss1S0ugFYprwAFsAyS0WrG4BlygtgASyzVLS6AVimvAJY53rbUqZdXV3MHivnVuh32FLG76w7b968ggNsaWmhan0Ce8HBzdIBwM5SuFpeLR+wMx+Ole/zHXfcUZJMAFaVD8CqmtS9BcCaWwIA1tzcVCwyAFsx6Qt2DGALSlR/DgDW3JwDWHNzU7HIAGzFpC/YMYAtKFH9OQhgnbO5pUxx0in42gGwwWuKFgNSAGeJVSEBrKoJLIYoAGDVRABYVRNYDFEAwKqJALCqJrAYogCAVRMBYFVNYDFEAQCrJgLAqprAUgEFtm3bRj09PZ6eAaxHDvkBwKqawFIhBebPn0/XXXcd/frrrzICAKsmom6AfSISoWQySefPn6fLly+T8yAoMZ2YmJA30U6n03LevUzcCV+sc+7cObpy5YpnvUuXLsnHW5w9e1Yq66z3888/y6ITd+sfHR31LDtz5oy8i//4+DgJP/Fy1kulUvLu/KJdxyamk5OTJPrwi13ENDY2RtUeu4hfjHVgYED+BvzAAw8QgJXl4flTN8Aue+IJWrNmDe3atYtOnDhBy5Ytk0KIqQDlnXfeoe3bt9Pp06c9y8SjL3bs2EFbtmyRoLnXO3r0KIkHSa1fvz7blpgRj8w4fPiw3MV74YUXPMuee+45+RiOzz//nJ555hnPsnXr1lF3dzd9++23FIlEssvEl4Z4fIdf7AJkTuwXLlzwjKsSsQsY3fq5YxdjE8vE27lYA8DKEvD8qRtgI83NNDQ05Bk8PpilwNdffy1hFV+O4gVg1fwAWFUTWCqggHi+z1133eXpGcB65JAfAKyqCSyGKABg1UQAWFUTWAxRAMCqiQCwqiawGKIAgFUTAWBVTWAxRAEAqyYCwKqawGKIAgBWTUTVAOv8NlfKFD/rqAVgsiUfsKXUgXtdk8fvF1tVAOsXOGy1r0AuYPPdWnXmsq49e+nNN9+sGbEAbM2ksvYGAmDVnAJYVRNYDFEAwKqJALCqJrAYogCAVRMBYFVNYDFEAQCrJgLAqprAYogCAth7751Lv12wYNbvhoYGnHQyJJ8Io8YVEP8zLP4HuNS3+P/iWnlhC1srmcQ46kIBAFsXacYga0UBAFsrmcQ46kIBAFsXacYga0UBAFsrmcQ46kIBAFsXacYga0UBAFsrmcQ46kIBAFsXacYga0UBAFsrmcQ46kKB/wPTXFv/+Xo8fAAAAABJRU5ErkJggg=="}}},{"metadata":{},"cell_type":"markdown","source":"For the implementation of the blocks that make up the coding part of the generator, a \"downsample\" function will be created that passing the number of filters to it and if normalization is applied, it will create a keras.Sequential object"},{"metadata":{"trusted":true},"cell_type":"code","source":"def downsample(filters, apply_norm=True):\n result = keras.Sequential()\n\n initializer = tf.random_normal_initializer(0,0.02)\n\n # Convolutional layer\n result.add(layers.Conv2D(filters,\n kernel_size=4,\n strides=2,\n padding=\"same\",\n kernel_initializer=initializer,\n use_bias=not apply_norm)) # when applying Normalization you already have the bias implicit\n\n # Normalization layer\n if apply_norm:\n gamma_init = tf.keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n result.add(tfa.layers.InstanceNormalization(gamma_initializer=gamma_init)) \n\n # Activation layer\n result.add(layers.LeakyReLU())\n\n return result","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# A test is performed to verify that it does not give an error\ndown_model = downsample(3)\ndown_result = down_model(example_photo, 0)\nprint (down_result.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For the implementation of the blocks that make up the decoding part of the generator, an \"upsample\" function will be created that passing the number of filters to it and if dropout is applied, it will create a keras.Sequential object"},{"metadata":{"trusted":true},"cell_type":"code","source":"\ndef upsample(filters, apply_dropout=True):\n result = keras.Sequential()\n\n initializer = tf.random_normal_initializer(0,0.02)\n\n # Transpose convolutional layer\n result.add(layers.Conv2DTranspose(filters,\n kernel_size=4,\n strides=2,\n padding=\"same\",\n kernel_initializer=initializer,\n use_bias=False)) # al aplicar BatchNormalization ya tiene implicito el sesgo\n\n # Normalization layer\n gamma_init = tf.keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n result.add(tfa.layers.InstanceNormalization(gamma_initializer=gamma_init)) # Cyclegan usa la normalización de instancias\n \n \n # Dropout layer\n if apply_dropout:\n result.add(layers.Dropout(0.5))\n\n # Activation layer\n result.add(layers.ReLU())\n\n return result","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# A test is performed to verify that it does not give an error\nup_model = upsample(3)\nup_result = up_model(down_result)\nprint (up_result.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, a diagram of the Generator is presented, in which the inputs and outputs of the different blocks can be seen.\n\n* The red arrows represent the execution of a downsample-k block\n* The blue arrows represent the execution of an upsampleD-k block (with Dropout)\n* The green arrows represent the execution of an upsample-k block (without Dropout)\n* Lastly, an output layer, represented by a pink arrow is executed, which is simply Transposed Convolution of 3 filters to convert the output into a 256X256 image by 3 channels\n\n\nAll convolutional layers of downsample, have the parameter strides = 2, which causes the dimensions to be reduced by half, likewise the Transposed Convolution layers of upsample also have the parameter strides = 2 so the dimensions are doubled. In the first two dimensions, not counting the Batch size dimension.\n\nThe size of the 3 dimension will coincide with the number of filters applied.\n\n"},{"metadata":{},"cell_type":"markdown","source":"
"},{"metadata":{"trusted":true},"cell_type":"code","source":"def Generator():\n\n inputs = layers.Input(shape=[256,256,3])\n\n down_stack = [ \n downsample(64, apply_norm=False), # [bs, 128, 128, 64] \n downsample(128), # [bs, 64, 64, 128]\n downsample(256), # [bs, 32, 32, 256]\n downsample(512), # [bs, 16, 16, 512]\n downsample(512), # [bs, 8, 8, 512]\n downsample(512), # [bs, 4, 4, 512]\n downsample(512), # [bs, 2, 2, 512]\n downsample(512), # [bs, 1, 1, 512]\n ]\n\n up_stack = [\n upsample(512), # [bs, 2, 2, 1024] \n upsample(512), # [bs, 4, 4, 1024]\n upsample(512), # [bs, 8, 8, 1024]\n upsample(512, apply_dropout=False), # [bs, 16, 16, 1024]\n upsample(256, apply_dropout=False), # [bs, 32, 32, 512]\n upsample(128, apply_dropout=False), # [bs, 64, 64, 256]\n upsample(64, apply_dropout=False), # [bs, 128, 128, 128]\n ] \n\n # Output layer\n initializer = tf.random_normal_initializer(0,0.02)\n last = layers.Conv2DTranspose(filters = 3, # Number of image channels\n kernel_size=4,\n strides=2,\n padding=\"same\",\n kernel_initializer=initializer,\n activation=\"tanh\" # output -1 to 1\n )\n \n x = inputs\n skips = []\n \n # We add the Encoder blocks to the model and save the outputs to later perform the Skip Connections\n for down in down_stack:\n x = down(x) \n skips.append(x)\n \n\n # We eliminate the last layer of the Skips Connections, \n # since it will be a direct input in the first block of the Decoder \n # and we turn the Skips around since the second layer must connect \n # with the penultimate output of the Encoder, the third with the penultimate . \n # and so on successively\n skips = reversed(skips[:-1]) \n\n # We add the Decoder blocks to the model and sthe Skip Connections\n for up, skip in zip(up_stack, skips):\n x = up(x)\n x = layers.Concatenate()([x,skip])\n x = last(x)\n\n return keras.Model(inputs=inputs, outputs=x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":" # We test what creates the model and show architecture\ngenerator_g = Generator()\ntf.keras.utils.plot_model(generator_g, show_shapes=True, dpi=64)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Generator initial test"},{"metadata":{"trusted":true},"cell_type":"code","source":"# We pass the denormalized photo so that some result can be seen, since the model is not trained\nphoto = denormalize(example_photo[0,...])\nexample_gen_output_y = generator_g(photo[tf.newaxis,...], training=False)\n\nplt.subplot(1,2,1)\nplt.imshow(photo, vmin=0, vmax=255) \n\nplt.subplot(1,2,2)\nplt.imshow(example_gen_output_y[0]) \n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Build the Discriminator\n\n\nThe task of the discriminator is whether an input image is original or fake (the output of a generator)\n\nThe architecture of the discriminator is a convolution network of the PatchGAN type, instead of returning whether the image is real or not, this architecture returns whether pieces of the image can be considered real or false. consisting of an encoder blockmade up of simpler blocks of layers:\n\n\n
\n\n\nAs in the generator, the encoder is made up of downsample-k blocks, this block performs an image compression operation (downsample). It consists of the following layers \n * Convolution \n * Instance Normalization (not apply to the first block)\n * Leaky ReLU\n\nSo the already implemented function is used \"downsample\", all convolutional layers of downsample, have the parameter strides = 2, which causes the dimensions to be reduced by half.\n\n\nThe shape of the discriminator output layer is (batch_size, 30, 30, 1), each 30x30 patch of the output sorts a 70x70 portion of the input image\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"def Discriminator():\n \n ini = layers.Input(shape=[256, 256, 3], name=\"input_img\") \n\n down1 = downsample(64, apply_norm=False)(ini) # [bs, 128, 128, 64] \n down2 = downsample(128)(down1) # [bs, 64, 64, 128]\n down3 = downsample(256)(down2) # [bs, 32, 32, 256]\n \n zero_pad1 = tf.keras.layers.ZeroPadding2D()(down3) # (bs, 34, 34, 256)\n\n initializer = tf.random_normal_initializer(0., 0.02)\n conv = tf.keras.layers.Conv2D(512, \n kernel_size=4, \n strides=1,\n kernel_initializer=initializer,\n use_bias=False)(zero_pad1) # (bs, 31, 31, 512)\n\n gamma_init = tf.keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n norm1 = tfa.layers.InstanceNormalization(gamma_initializer=gamma_init)(conv)\n\n leaky_relu = tf.keras.layers.LeakyReLU()(norm1)\n\n zero_pad2 = tf.keras.layers.ZeroPadding2D()(leaky_relu) # (bs, 33, 33, 512)\n\n last = tf.keras.layers.Conv2D(1, \n kernel_size=4, \n strides=1,\n kernel_initializer=initializer)(zero_pad2) # (bs, 30, 30, 1)\n\n return keras.Model(inputs=ini, outputs=last)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":" # We test what creates the model and show architecture\ndiscriminator_y = Discriminator()\ntf.keras.utils.plot_model(discriminator_y, show_shapes=True, dpi=64)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"Discriminator initial test"},{"metadata":{"trusted":true},"cell_type":"code","source":"# We pass the denormalized photo so that some result can be seen, since the model is not trained\nphoto = denormalize(example_photo[0,...])\nexample_gen_output_y = generator_g(photo[tf.newaxis,...], training=False)\nexample_disc_out = discriminator_y([example_photo, example_gen_output_y], training=False)\n\nprint(example_disc_out.shape)\n\nplt.figure(figsize=(10,10))\n\nplt.subplot(1,3,1)\nplt.imshow(photo, vmin=0, vmax=255) \n\nplt.subplot(1,3,2)\nplt.imshow(example_gen_output_y[0,...]) \n\nplt.subplot(1,3,3)\nm = example_disc_out[0,...,-1].numpy()*1000\nim = plt.imshow(m, vmin=-20, vmax=20, cmap='RdBu_r')\nplt.colorbar(im,fraction=0.046, pad=0.04)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Build the complete Model \n\n\nAs seen in the introduction, the complete model consists of:\n \n* 2 generators (G_MONET and G_PHOTO) \n * The G_MONET generator learns how to transform a photograph into a Monet painting \n * The G_PHOTO generator learns how to transform a Monet painting into a photograph \n * As the images are not paired, it is necessary to use two cycles\n * Input Photo -> G_MONET -> Fake Monet -> G_PHOTO -> Cycle Photo\n * Input Monet -> G_PHOTO -> Fake Photo -> G_MONET -> Cycle Monet\n \n* 2 discriminators (D_MONET AND D_PHOTO) \n * The D_MONET discriminator learns to differentiate if a Monet painting is real or fake, serving to calculate the adversary loss and improve the G_MONET generator\n * The D_PHOTO discriminator learns to differentiate if a photo is real or fake, serving to calculate the adversary loss and improve the G_PHOTO generator\n * As the images are not paired, to calculate the loss of the discriminators, each one of them is executed twice, one with a real image and another with the fake image generated.\n "},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n \n generator_monet = Generator()\n generator_photo = Generator()\n\n discriminator_monet = Discriminator()\n discriminator_photo = Discriminator()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Example\n\nGenerator part:\n\n* Starting from the photo, a simulation of a Monet painting is generated and later from this simulation an attempt is made to generate the original photo\n\n* Starting from the monet, a photo simulation is generated and later from this simulation an attempt is made to generate the original monet\n\nDiscriminator part:\n\n* Discriminator so that the fake photo looks like a real photo\n* Discriminator so that the monet fake looks like a Monet painting"},{"metadata":{"trusted":true},"cell_type":"code","source":"photo = denormalize(example_photo[0,...])\nmonet = denormalize(example_monet[0,...])\n\n# From photo we generate Monet (fake) and regenerate the photo (cycle) again\nexample_gen_output_monet_fake = generator_monet(photo[tf.newaxis,...], training=False)\nexample_gen_output_photo_cycle = generator_photo(example_gen_output_monet_fake, training=False)\n\n# We run the discriminator for Monet (fake)\nexample_disc_out_monet = discriminator_monet(example_gen_output_monet_fake, training=False)\n\n\n# From Monet we generate photo (fake) and regenerate Monet (cycle) again\nexample_gen_output_photo_fake = generator_photo(monet[tf.newaxis,...], training=False)\nexample_gen_output_monet_cycle = generator_monet(example_gen_output_photo_fake, training=False)\n\n# We execute the discriminator for Photo (fake)\nexample_disc_out_photo = discriminator_photo(example_gen_output_photo_fake, training=False)\n\n\n# We present results, as the network is not trained, the outputs are not good, \n# but we modify the scala to be able to have some example images\n\nplt.figure(figsize=(10,10))\n\n# Input Photo\nplt.subplot(2,4,1)\nplt.imshow(photo, vmin=0, vmax=255) \n\n# Fake Monet\nplt.subplot(2,4,2)\n#m = example_gen_output_monet_fake[0,...].numpy()\n#print(np.min(m), np.max(m))\ncontrast = 100 \nplt.imshow(example_gen_output_monet_fake[0,...]*contrast) \n\n# Photo Cycle\nplt.subplot(2,4,3)\n#m = example_gen_output_photo_cycle[0,...].numpy()\n#print(np.min(m), np.max(m))\ncontrast = 100\nplt.imshow(example_gen_output_photo_cycle[0,...]*contrast) \n\n# Monet discriminator result\nplt.subplot(2,4,4)\n#m = example_disc_out_monet[0,...,-1].numpy()\n#print(np.min(m), np.max(m))\ncontrast = 1000\nim = plt.imshow(m*contrast, vmin=-20, vmax=20, cmap='RdBu_r')\nplt.colorbar(im,fraction=0.046, pad=0.04)\n\n\n\n# Input Monet\nplt.subplot(2,4,5)\nplt.imshow(monet, vmin=0, vmax=255) \n\n# Fake Photo\nplt.subplot(2,4,6)\n#m = example_gen_output_photo_fake[0,...].numpy()\n#print(np.min(m), np.max(m))\ncontrast = 100\nplt.imshow(example_gen_output_photo_fake[0,...]*contrast) \n\n# Monet Cycle\nplt.subplot(2,4,7)\n#m = example_gen_output_monet_cycle[0,...].numpy()\n#print(np.min(m), np.max(m))\ncontrast = 100\nplt.imshow(example_gen_output_monet_cycle[0,...]*contrast) \n\n# Photo discriminator result \nplt.subplot(2,4,8)\n#m = example_disc_out_photo[0,...,-1].numpy()\n#print(np.min(m), np.max(m))\ncontrast = 1000\nim = plt.imshow(m*contrast, vmin=-20, vmax=20, cmap='RdBu_r')\nplt.colorbar(im,fraction=0.046, pad=0.04)\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Definition of loss functions\n\nSigmoid cross entropy is used to calculate the adversary losses in the discriminator and generator."},{"metadata":{"trusted":true},"cell_type":"code","source":"loss_object = keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Discriminator loss\n\n* The discriminator loss function takes 2 inputs;\n * For the discriminator_photo will take as input: \n * The output of the discriminator_photo whose input is the real photo of the training set \n * The output of the discriminator_photo whose input is the fake photo generated by the generator_photo\n * For the discriminator_monet will take as input: \n * The output of the discriminator_monet whose input is the real Monet of the training set \n * The output of the discriminator_monet whose input is the fake Monet generated by the generator_monet \n* The calculation of the loss has two components: \n * real_loss compare the real image with a matrix of 1. (All Ok)\n * generate_loss compare the fake image with a matrix of 0 (All Fake)\n* So the total_loss is 1/2 of the sum of the real_loss and the generate_loss"},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n def discriminator_loss(disc_real_output, disc_generated_output):\n #compare the real image with a matrix of 1. (All Ok)\n real_loss = loss_object(tf.ones_like(disc_real_output), disc_real_output)\n\n #compare the fake image with a matrix of 0 (All Fake)\n generated_loss = loss_object(tf.zeros_like(disc_generated_output), disc_generated_output) \n\n total_disc_loss = (real_loss + generated_loss) / 2\n\n return total_disc_loss","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Generator loss\n> The generator loss has 3 terms:\n> * Adversary loss\n> * Cycle loss\n> * Identity loss\n \n\n**Generator adversary loss**\n* Takes as input the output of the dicriminator\n * For the loss of the generator_monet the function will take the output of the discriminator_monet executed with fake monet\n * For the loss of the generator_photo the function will take the output of the discriminator_photo executed with fake photo\n* The perfect generator will have the discriminator output only ones, all Ok. Therefore, compare the generated image with a matrix of 1 to find the loss."},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n def generator_adversarial_loss(generated):\n return loss_object(tf.ones_like(generated), generated)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Generator cycle consistency loss**\n\nAs the training data is not paired, to make the network learn the correct mapping and the result is similar to the original input. The authors of the paper add another term for the calculation of the loss which they call the loss of consistency of the cycle.\n\nThe cycle consists of:\n* Generate a monet style image from a photo, this generated image is passed as input to the second generator, which should generate a photo from a fake monet style image.\n * Input Photo -> G_MONET -> Fake Monet -> G_PHOTO -> Cycle Photo\n* On the other hand, an image is generated that aims to imitate a real photo from a Monet painting, this generated image is passed as an input to the second generator, which should generate the Monet painting again from the fake photo.\n * Input Monet -> G_PHOTO -> Fake Photo -> G_MONET -> Cycle Monet \n\nTo calculate the cycle consistency loss, two terms are calculated\n* Average absolute error for photo, is calculated between Input Photo and Cycle Photo\n* Mean absolute error for monet, is calculated between Input Monet and Cycle Monet\n\nThe cyclo error will be the sum of both terms.\n\nThis term is applied to the total loss calculation for both generators (generator_monet and generator_photo)\n\n"},{"metadata":{},"cell_type":"markdown","source":"Since the calculation of the two terms is the same, then a function is created that allows each of the terms to be calculated, varying only the inputs."},{"metadata":{"trusted":true},"cell_type":"code","source":"\nwith strategy.scope():\n def generator_calc_cycle_loss(real_image, cycled_image, param_lambda):\n loss1 = tf.reduce_mean(tf.abs(real_image - cycled_image))\n\n return param_lambda * loss1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Identity loss**\n\nIt is desired that if the generator_monet is executed with the image of the original monet painting, the result should be a very similar image, and also with the generator_photo and the original photo.\n\nTherefore, the loss of identity forces what the generator generates to resemble the input.\n\n\n* It takes as inputs, 2 images, the original image and the output of running the generator with the original image.\n * For the loss of the generator_monet the function will take the training monet image and the output of the generator_monet with the same input (same_monet)\n * For the loss of the generator_photo the function will take the training photo image and the output of the generator_photo with the same input (same_photo)\n \n\nThe loss will be the mean absolute error between the real image and the generated one.\n"},{"metadata":{},"cell_type":"markdown","source":"Given that the calculation of the losses is the same for both generators, then a function is created that allows it to be calculated, varying only the inputs."},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n def generator_identity_loss(real_image, same_image, param_lambda):\n loss = tf.reduce_mean(tf.abs(real_image - same_image))\n return param_lambda * 0.5 * loss","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Initialize the optimizers \n\nInitialize the optimizers for all the generators and the discriminators."},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n generator_monet_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n generator_photo_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n\n discriminator_monet_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n discriminator_photo_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Modelo completo\n\nWe define the CycleGan class that inherits from Keras.model, this will allow overwriting the train_step function that is used in the fit method in such a way that performance can be maximized with the execution in TPU.\n\n"},{"metadata":{},"cell_type":"markdown","source":"Next, the complete diagram is presented again, which allows to understand the implementation of the train_step function"},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{"trusted":true},"cell_type":"code","source":"\nclass CycleGan(keras.Model):\n def __init__(\n self,\n generator_monet,\n generator_photo,\n discriminator_monet,\n discriminator_photo,\n lambda_cycle=10,\n lambda_identity=10\n ):\n super(CycleGan, self).__init__()\n self.generator_monet = generator_monet\n self.generator_photo = generator_photo\n self.discriminator_monet = discriminator_monet\n self.discriminator_photo = discriminator_photo\n self.lambda_cycle = lambda_cycle\n self.lambda_identity = lambda_identity\n \n \n def compile(\n self,\n generator_monet_optimizer,\n generator_photo_optimizer,\n discriminator_monet_optimizer,\n discriminator_photo_optimizer,\n discriminator_loss, \n generator_adversarial_loss,\n generator_calc_cycle_loss,\n generator_identity_loss\n ):\n super(CycleGan, self).compile()\n self.generator_monet_optimizer = generator_monet_optimizer\n self.generator_photo_optimizer = generator_photo_optimizer\n self.discriminator_monet_optimizer = discriminator_monet_optimizer\n self.discriminator_photo_optimizer = discriminator_photo_optimizer\n self.discriminator_loss = discriminator_loss \n self.generator_adversarial_loss = generator_adversarial_loss\n self.generator_calc_cycle_loss = generator_calc_cycle_loss\n self.generator_identity_loss = generator_identity_loss\n \n \n\n def train_step(self, batch_data):\n\n monet, photo = batch_data\n \n with tf.GradientTape(persistent=True) as tape:\n\n # Obtener predicciones\n # desde photo generamos Monet (fake) y regeneramos de nuevo la foto (cycle)\n gen_output_monet_fake = self.generator_monet(photo, training=False)\n gen_output_photo_cycle = self.generator_photo(gen_output_monet_fake, training=False)\n\n # desde Monet generamos photo (fake) y regeneramos de nuevo Monet (cycle) \n gen_output_photo_fake = self.generator_photo(monet, training=False)\n gen_output_monet_cycle = self.generator_monet(gen_output_photo_fake, training=False)\n\n # generating itself are used for identity loss.\n gen_output_monet_same = self.generator_monet(monet, training=False)\n gen_output_photo_same = self.generator_photo(photo, training=False)\n\n\n # Ejecutamos el discriminador para Monet real (entrada) y Monet generado (fake)\n disc_out_monet_real = self.discriminator_monet(monet, training=False) \n disc_out_monet_fake = self.discriminator_monet(gen_output_monet_fake, training=False)\n\n # Ejecutamos el discriminador para Foto real (entrada) y Foto (fake)\n disc_out_photo_real = self.discriminator_photo(photo, training=False)\n disc_out_photo_fake = self.discriminator_photo(gen_output_photo_fake, training=False)\n\n\n # Calculate the loss\n \n gen_monet_adversarial_loss = self.generator_adversarial_loss(disc_out_monet_fake)\n gen_photo_adversarial_loss = self.generator_adversarial_loss(disc_out_photo_fake)\n\n # evaluates total cycle consistency loss\n total_cycle_loss = (self.generator_calc_cycle_loss(monet, gen_output_monet_cycle, self.lambda_cycle) \n + self.generator_calc_cycle_loss(photo, gen_output_photo_cycle, self.lambda_cycle))\n\n gen_monet_identity_loss = self.generator_identity_loss(monet, gen_output_monet_same, self.lambda_identity)\n gen_photo_identity_loss = self.generator_identity_loss(photo, gen_output_photo_same, self.lambda_identity)\n \n # Total generator loss = adversarial loss + cycle loss + identity loss\n total_gen_monet_loss = (gen_monet_adversarial_loss + total_cycle_loss + gen_monet_identity_loss)\n total_gen_photo_loss = (gen_photo_adversarial_loss + total_cycle_loss + gen_photo_identity_loss)\n\n disc_monet_loss = self.discriminator_loss(disc_out_monet_real, disc_out_monet_fake)\n disc_photo_loss = self.discriminator_loss(disc_out_photo_real, disc_out_photo_fake)\n\n # Calculate the gradients for generator and discriminator\n gen_monet_gradients = tape.gradient(total_gen_monet_loss, \n self.generator_monet.trainable_variables)\n gen_photo_gradients = tape.gradient(total_gen_photo_loss, \n self.generator_photo.trainable_variables)\n\n disc_monet_gradients = tape.gradient(disc_monet_loss, \n self.discriminator_monet.trainable_variables)\n disc_photo_gradients = tape.gradient(disc_photo_loss, \n self.discriminator_photo.trainable_variables)\n\n\n # Apply the gradients to the optimizer\n\n\n self.generator_monet_optimizer.apply_gradients(zip(gen_monet_gradients,\n self.generator_monet.trainable_variables))\n self.generator_photo_optimizer.apply_gradients(zip(gen_photo_gradients,\n self.generator_photo.trainable_variables)) \n\n self.discriminator_monet_optimizer.apply_gradients(zip(disc_monet_gradients,\n self.discriminator_monet.trainable_variables)) \n self.discriminator_photo_optimizer.apply_gradients(zip(disc_photo_gradients,\n self.discriminator_photo.trainable_variables)) \n\n \n return {\n \"total_gen_monet_loss\": total_gen_monet_loss,\n \"total_gen_photo_loss\": total_gen_photo_loss,\n \"disc_monet_loss\": disc_monet_loss,\n \"disc_photo_loss\": disc_photo_loss\n }\n ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n cycle_gan_model = CycleGan(\n generator_monet, generator_photo, discriminator_monet, discriminator_photo\n )\n\n cycle_gan_model.compile(\n generator_monet_optimizer = generator_monet_optimizer,\n generator_photo_optimizer = generator_photo_optimizer,\n discriminator_monet_optimizer = discriminator_monet_optimizer,\n discriminator_photo_optimizer = discriminator_photo_optimizer,\n discriminator_loss = discriminator_loss, \n generator_adversarial_loss = generator_adversarial_loss,\n generator_calc_cycle_loss = generator_calc_cycle_loss,\n generator_identity_loss = generator_identity_loss\n )\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Model test\n\nWe have tested the model with CPU, GPU and TPU\n\n2 epochs\n* Accelerator = NONE ----- Computation time = 10391658.411232ms\n* Accelerator = GPU ----- Computation time = 270945.03528ms\n* Accelerator = TPU Number of replicas: 8 ----- Computation time = 74406.024123ms\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"import time\n\nt1 = time.process_time()\n\ncycle_gan_model.fit(\n tf.data.Dataset.zip((train_photo_ds, train_monet_ds)),\n epochs=20\n)\n\nt2 = time.process_time()\n\nprint (\"Accelerator = ----- Computation time = \" + str(1000*(t2 - t1)) + \"ms\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def generate_images(model, test_input):\n prediction = model(test_input)\n\n plt.figure(figsize=(12, 12))\n\n plt.subplot(1, 2, 1)\n plt.imshow(test_input[0] * 0.5 + 0.5)\n plt.title('Input Image')\n plt.axis('off')\n \n plt.subplot(1, 2, 2)\n plt.imshow(prediction[0] * 0.5 + 0.5)\n plt.title('Predicted Image')\n plt.axis('off')\n\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"generate_images(generator_monet, example_photo)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Execution example"},{"metadata":{"trusted":true},"cell_type":"code","source":"# From photo we generate Monet (fake) and regenerate the photo (cycle) again\nexample_gen_output_monet_fake = generator_monet(example_photo, training=False)\nexample_gen_output_photo_cycle = generator_photo(example_gen_output_monet_fake, training=False)\nexample_gen_output_photo_same = generator_photo(example_photo, training=False)\n\n\n# We execute the discriminator for Photo (real)\nexample_disc_out_photo_real = discriminator_photo(example_photo, training=False)\n\n# We run the discriminator for Monet (fake)\nexample_disc_out_monet_fake = discriminator_monet(example_gen_output_monet_fake, training=False)\n\n\n# from Monet we generate photo (fake) and regenerate Monet (cycle) again\nexample_gen_output_photo_fake = generator_photo(example_monet, training=False)\nexample_gen_output_monet_cycle = generator_monet(example_gen_output_photo_fake, training=False)\nexample_gen_output_monet_same = generator_monet(example_monet, training=False)\n\n# We run the discriminator for Monet (real)\nexample_disc_out_monet_real = discriminator_monet(example_monet, training=False)\n\n# We execute the discriminator for Photo (fake)\nexample_disc_out_photo_fake = discriminator_photo(example_gen_output_photo_fake, training=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We visualize example"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,10))\n\n# Foto entrada\nplt.subplot(4,4,1)\nplt.imshow(example_photo[0] * 0.5 + 0.5) \n\n# Monet generado\nplt.subplot(4,4,2)\nplt.imshow(example_gen_output_monet_fake[0] * 0.5 + 0.5) \n\n# Photo Cycle\nplt.subplot(4,4,3)\nplt.imshow(example_gen_output_photo_cycle[0] * 0.5 + 0.5) \n\n# Photo Same\nplt.subplot(4,4,4)\nplt.imshow(example_gen_output_photo_same[0] * 0.5 + 0.5) \n\n# Discriminador para Photo (real)\nplt.subplot(4,4,5)\nm = example_disc_out_photo_real[0,...,-1].numpy()\nplt.imshow(m, vmin=np.min(m), vmax=np.max(m), cmap='RdBu_r')\nplt.colorbar(im,fraction=0.046, pad=0.04)\n\n# Discriminador para Monet (fake)\nplt.subplot(4,4,7)\nm = example_disc_out_monet_fake[0,...,-1].numpy()\nplt.imshow(m, vmin=np.min(m), vmax=np.max(m), cmap='RdBu_r')\nplt.colorbar(im,fraction=0.046, pad=0.04)\n\n# Monet entrada\nplt.subplot(4,4,9)\nplt.imshow(example_monet[0] * 0.5 + 0.5) \n\n# Foto generado\nplt.subplot(4,4,10)\nplt.imshow(example_gen_output_photo_fake[0] * 0.5 + 0.5) \n\n# Monet Cycle\nplt.subplot(4,4,11)\nplt.imshow(example_gen_output_monet_cycle[0] * 0.5 + 0.5) \n\n# Monet Same\nplt.subplot(4,4,12)\nplt.imshow(example_gen_output_monet_same[0] * 0.5 + 0.5) \n\n# Discriminador para Monet (real)\nplt.subplot(4,4,13)\nm = example_disc_out_monet_real[0,...,-1].numpy()\nplt.imshow(m, vmin=np.min(m), vmax=np.max(m), cmap='RdBu_r')\nplt.colorbar(im,fraction=0.046, pad=0.04)\n\n# Discriminador para Photo (fake)\nplt.subplot(4,4,15)\nm = example_disc_out_photo_fake[0,...,-1].numpy()\nplt.imshow(m, vmin=np.min(m), vmax=np.max(m), cmap='RdBu_r')\nplt.colorbar(im,fraction=0.046, pad=0.04)\n\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for img in test_photo_ds.take(5):\n generate_images(generator_monet, img)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In this Notebook we have tried to carry out the step-by-step implementation of CycleGAN to generate a model that allows a style transfer, on photographs with a painting style like Monet.\n\nIn a next Notebook I will make the adjustment of the model."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}