{"cells":[{"metadata":{},"cell_type":"markdown","source":"CycleGAN without identity loss"},{"metadata":{},"cell_type":"markdown","source":"## Identifying problems\n1. CycleGAN learning is not stable. The more epochs, the worse the result.\n* If one of the sets of images is relatively small (only 300 Monet paintings), then overfitting is observed - the discriminator remembers them and accurately determines them. In the case of testing with other real paintings, Monet does not recognize them as the author, confidently considers them to be generated fakes.\n* Objective function problem. The correspondence of the restored drawing to the original, as well as the preservation of colors in the generated drawing, distracts too much from the real goal - to generate a Monet painting (or photograph) similar to the real one\n* Too many hyperparameters. Also, initial initialization affects the learning outcome.\n\n## Fail: CycleGAN without identity loss\n\nCycleGAN model problem. If we submit a photo of horses along with zebras at the entrance, waiting for zebras in the output (or photos of men and women waiting for photos of women), then the logical result is to take into account the identity, that is, if you submit photos of some zebras (some women) at the entrance, you should get the same photo at the exit (preferably without changes). Just as the basline CycleGAN notebook does. \n\nBut if the changes are significant (for example, we make a topographic plan from satellite images or a cartoon from photographs), then submitting Monet's paintings (or a topographic plan) to the entrance instead of a photo (or satellite images) does not make sense.\n\nMy hypothesis: if we remove the identity check that the generator should produce practically unchanged input result if input already meets the requirements (that is, it is already a Monet painting or a topographic plan), then the learning process will look more natural. Why “torment” CycleGAN with an unnecessary component of the loss function?\n\nIdentity loss deleted from the model. \n\nIt was hard to estimate result of identity loss absence when FID was growing up after some epoch on basic BCE loss.\n\nAfter change to WGAN-GP and LSE loss it became clear that identity loss makes better results. Added lambda_id to control it.\n\n\n## Fail: MSE cycle loss\n\nLoss function problem. I tried to use the mean square error (MSE) for the recovered image (instead of MAE in baseline notebook). If the color difference between the point of the reconstructed drawing is insignificant, it is indistinguishable to the human eye. Squaring will penalize large deviations and reduce the significance of small deviations. At the same time, the disadvantage of MSE - intolerance to outliers in the training data - is completely inapplicable to our problem, since the training data does not contain the “ground truth” answer.\n\nFail: it is bad idea punish for cycled image deviation. Good transformation means some information lost that cant be reconstructed. Changed to Huber loss. \n\n\n\n## Early stopping \n\nFail: I spent two weeks looking for inception layers MIFID metric evaluated. I inspected pretrained inception/xception/resnet/vgg and their early layers (with max/average pooling) from tensorflow and tensorflow_hub. I didn't find anything related to the competition metric. \n\n\n## Additinal metric\n\nIn class CycleGan implemented function test_step method to evaluate metrics on validation set.\n\nI use for training 6038 photos and 300 Monet paintings. Validation set consists of 1000 photos and 1367 Monet paintings from monet-tfrecords-256x256 kaggle dataset\n\nI observe parts of total_cycle_loss: cycle_loss_mpm and cycle_loss_pmp - deviation of the Monet-Photo-Monet and Photo-Monet-Photo transformations\n\nObservation: when comparing the MAE deviation of the Monet-Photo-Monet and Photo-Monet-Photo transformations for the training and test data, I noticed the following:\n* for the training data, MAE of the transformations are approximately equal (cycle_loss_mpm ~= cycle_loss_pmp);\n* for the test data, the MAE of the Monet-Photo-Monet transformation is almost twice as high as the MAE of the Photo-Monet-Photo (val_cycle_loss_mpm >> val_cycle_loss_pmp);\n\n\nFail: excluding 1000 photos from training set is a very bad idea for the competition metric. I removed model evaluation on the test set\n\n\n## Fail: count Dicriminator and Generator loss for Real_Monet <-> Cycled_Monet and Real_Photo <-> Cycled_Photo\n\nIt cant help with overfitting Discriminator on Real_Monet<->Fake_Monet \n\n## Fail: Wasserstain loss WGAN-GP for discriminator and generator\n\nI tried to use WGAN-GP loss (versions 56 - 78). \nPositive changes: FID metric does not grow into infinity after some epoch.\nNegative: competition metric is higher then 44\n\n## Current experiment: LSE loss for discriminator and generator\n\nCompetition score is 41 from the first version. FID metric is decreasing with each epoch. \n\n\n\n## Other minor changes\n\n* fast generation of submission images (fast_photo_ds dataset with other batchsize than for training)\n\n\n## Result\n\nCycleGAN without identity loss is bad idea for this competition. The reasons to use the identity loss: to preserve the color composition. \nFinal version of the notebook uses identity loss with lambda_id\n\n"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import tensorflow as tf\nimport tensorflow_hub as hub\nimport tensorflow_probability as tfp\nimport os, random, json, PIL, shutil, re, gc\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nimport tensorflow_addons as tfa\nfrom scipy import linalg\nfrom kaggle_datasets import KaggleDatasets\nimport matplotlib.pyplot as plt\nimport numpy as np\ndef seed_everything(seed=0):\n random.seed(seed)\n np.random.seed(seed)\n tf.random.set_seed(seed)\n os.environ['PYTHONHASHSEED'] = str(seed)\n os.environ['TF_DETERMINISTIC_OPS'] = '1'\nSEED = 0\nseed_everything(SEED)\n\n\n%matplotlib inline\ntry:\n tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n print('Device:', tpu.master())\n tf.config.experimental_connect_to_cluster(tpu)\n tf.tpu.experimental.initialize_tpu_system(tpu)\n strategy = tf.distribute.experimental.TPUStrategy(tpu)\nexcept:\n strategy = tf.distribute.get_strategy()\nprint('Number of replicas:', strategy.num_replicas_in_sync)\n\nAUTOTUNE = tf.data.experimental.AUTOTUNE\ntf.__version__\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"GCS_PATH = KaggleDatasets().get_gcs_path(\"gan-getting-started\")\n\n# GCS_PATH_1367 = KaggleDatasets().get_gcs_path(\"monet-tfrecords-256x256\")\n# GCS_PATH_1193 = KaggleDatasets().get_gcs_path(\"tfrecords-monet-paintings-256x256\")\n\nMONET_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/monet_tfrec/*.tfrec'))\nPHOTO_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/photo_tfrec/*.tfrec'))\n# MONET_FILENAMES_1193 = tf.io.gfile.glob(str(GCS_PATH_1193 + '/mon*.tfrec'))\n# MONET_FILENAMES_1367 = tf.io.gfile.glob(str(GCS_PATH_1367 + '/mon*.tfrec'))\n\n\nprint('Monet TFRecord Files:', len(MONET_FILENAMES))\nprint('Monet TFRecord Files:', MONET_FILENAMES)\n\nprint('Photo TFRecord Files:', len(PHOTO_FILENAMES))\nprint('Photo Files:', PHOTO_FILENAMES)\n\n\n# print('Monet 1193 TFRecord Files:', len(MONET_FILENAMES_1193))\n# print('Monet 1193 TFRecord Files:', MONET_FILENAMES_1193)\n\n# print('Monet 1367 TFRecord Files:', len(MONET_FILENAMES_1367))\n# print('Monet 1367 TFRecord Files:', MONET_FILENAMES_1367)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"IMAGE_SIZE = [256, 256]\n\ndef decode_image(image):\n image = tf.image.decode_jpeg(image, channels=3)\n image = (tf.cast(image, tf.float32) / 127.5) - 1\n image = tf.reshape(image, [*IMAGE_SIZE, 3])\n return image\n\ndef read_tfrecord(example):\n tfrecord_format = {\n \"image\": tf.io.FixedLenFeature([], tf.string)\n }\n example = tf.io.parse_single_example(example, tfrecord_format)\n image = decode_image(example['image'])\n return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def data_augment_color(image):\n image = tf.image.random_flip_left_right(image)\n image = (image + 1) / 2\n image = tf.image.random_saturation(image, 0.7, 1.2)\n image = tf.clip_by_value(image, 0, 1) \n image = (image - 0.5) * 2 \n return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"###### from pats notebook https://www.kaggle.com/swepat/cyclegan-to-generate-monet-style-images #############\ndef data_augment(image):\n p_rotate = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_spatial = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n p_crop = tf.random.uniform([], 0, 1.0, dtype=tf.float32)\n \n # Apply jitter\n if p_crop > .5:\n image = tf.image.resize(image, [286, 286])\n image = tf.image.random_crop(image, size=[256, 256, 3])\n if p_crop > .9:\n image = tf.image.resize(image, [300, 300])\n image = tf.image.random_crop(image, size=[256, 256, 3])\n \n # Random rotation\n if p_rotate > .9:\n image = tf.image.rot90(image, k=3) # rotate 270º\n elif p_rotate > .7:\n image = tf.image.rot90(image, k=2) # rotate 180º\n elif p_rotate > .5:\n image = tf.image.rot90(image, k=1) # rotate 90º\n \n # Random mirroring\n if p_spatial > .6:\n image = tf.image.random_flip_left_right(image)\n image = tf.image.random_flip_up_down(image)\n if p_spatial > .9:\n image = tf.image.transpose(image)\n \n return image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"BATCH_SIZE = 1\n# EPOCHS = 5\n\ndef load_dataset(filenames):\n dataset = tf.data.TFRecordDataset(filenames)\n dataset = dataset.map(read_tfrecord, num_parallel_calls=AUTOTUNE)\n return dataset\n\nphoto_ds = load_dataset(PHOTO_FILENAMES)\nmonet_ds = load_dataset(MONET_FILENAMES)\n\n# photo_ds_val=photo_ds.skip(6038)\n# monet_ds_val = load_dataset(MONET_FILENAMES_1367)\n# val_ds=tf.data.Dataset.zip((monet_ds_val, photo_ds_val)).batch(64)\n\n# photo_ds=photo_ds.take(6038)\n\nmonet_ds = monet_ds.repeat()\nphoto_ds = photo_ds.repeat()\n\n# monet_ds = monet_ds.map(data_augment_color, num_parallel_calls=AUTOTUNE)\n# photo_ds = photo_ds.map(data_augment, num_parallel_calls=AUTOTUNE)\n\n# photo_ds=photo_ds.batch(BATCH_SIZE)\n# monet_ds = monet_ds.batch(BATCH_SIZE)\n \ngan_ds = tf.data.Dataset.zip((monet_ds, photo_ds)).batch(BATCH_SIZE).prefetch(AUTOTUNE)\n\nfast_photo_ds = load_dataset(PHOTO_FILENAMES).batch(32*strategy.num_replicas_in_sync).prefetch(32)\n\nmonet_ds_fid = load_dataset(MONET_FILENAMES).batch(32*strategy.num_replicas_in_sync).prefetch(32)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n# inception_model = tf.keras.applications.InceptionV3(input_shape=(256,256,3),pooling=\"avg\",include_top=False)\n inception_model = tf.keras.applications.InceptionV3(input_shape=(256,256,3),pooling=\"avg\",include_top=False)\n\n mix3 = inception_model.get_layer(\"mixed3\").output\n f0 = tf.keras.layers.GlobalMaxPooling2D()(mix3)\n\n inception_model = tf.keras.Model(inputs=inception_model.input, outputs=f0)\n inception_model.trainable = False\n \ndef calculate_activation_statistics_mod(images,fid_model):\n act=fid_model.predict(images)\n mu = np.mean(act, axis=0)\n sigma = np.cov(act, rowvar=False)\n return mu, sigma\nmyFID_mu2, myFID_sigma2 = calculate_activation_statistics_mod(monet_ds_fid,inception_model)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(myFID_mu2.shape,myFID_sigma2.shape)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def calculate_frechet_distance(mu1,sigma1,mu2,sigma2):\n fid_epsilon = 1e-14\n mu1 = np.atleast_1d(mu1)\n mu2 = np.atleast_1d(mu2)\n sigma1 = np.atleast_2d(sigma1)\n sigma2 = np.atleast_2d(sigma2)\n\n assert mu1.shape == mu2.shape, 'Training and test mean vectors have different lengths'\n assert sigma1.shape == sigma2.shape, 'Training and test covariances have different dimensions'\n\n # product might be almost singular\n covmean, _ = linalg.sqrtm(sigma1.dot(sigma2), disp=False)\n if not np.isfinite(covmean).all():\n msg = f'fid calculation produces singular product; adding {fid_epsilon} to diagonal of cov estimates'\n warnings.warn(msg)\n offset = np.eye(sigma1.shape[0]) * fid_epsilon\n covmean = linalg.sqrtm((sigma1 + offset).dot(sigma2 + offset))\n \n # numerical error might give slight imaginary component\n if np.iscomplexobj(covmean):\n if not np.allclose(np.diagonal(covmean).imag, 0, atol=1e-3):\n m = np.max(np.abs(covmean.imag))\n raise ValueError(f'Imaginary component {m}')\n covmean = covmean.real\n tr_covmean = np.trace(covmean)\n return (mu1 - mu2).dot(mu1 - mu2) + np.trace(sigma1) + np.trace(sigma2) - 2 * tr_covmean\n\n\n \ndef FID(images,gen_model,inception_model=inception_model,myFID_mu2=myFID_mu2, myFID_sigma2=myFID_sigma2):\n with strategy.scope():\n inp = layers.Input(shape=[256, 256, 3], name='input_image')\n x = gen_model(inp)\n x=inception_model(x)\n fid_model = tf.keras.Model(inputs=inp, outputs=x)\n \n mu1, sigma1 = calculate_activation_statistics_mod(images,fid_model)\n\n fid_value = calculate_frechet_distance(mu1, sigma1,myFID_mu2, myFID_sigma2)\n\n return fid_value","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nOUTPUT_CHANNELS = 3\n\ndef downsample(filters, size, apply_instancenorm=True):\n initializer = tf.random_normal_initializer(0., 0.02)\n gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n result = keras.Sequential()\n result.add(layers.Conv2D(filters, size, strides=2, padding='same',\n kernel_initializer=initializer, use_bias=False))\n\n if apply_instancenorm:\n result.add(tfa.layers.InstanceNormalization(gamma_initializer=gamma_init))\n\n result.add(layers.LeakyReLU())\n\n return result","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def upsample(filters, size, apply_dropout=False):\n initializer = tf.random_normal_initializer(0., 0.02)\n gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n result = keras.Sequential()\n result.add(layers.Conv2DTranspose(filters, size, strides=2,\n padding='same',\n kernel_initializer=initializer,\n use_bias=False))\n\n result.add(tfa.layers.InstanceNormalization(gamma_initializer=gamma_init))\n\n if apply_dropout:\n result.add(layers.Dropout(0.5))\n\n result.add(layers.ReLU())\n\n return result","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def Generator():\n inputs = layers.Input(shape=[256,256,3])\n\n # bs = batch size\n down_stack = [\n downsample(64, 4, apply_instancenorm=False), # (bs, 128, 128, 64)\n downsample(128, 4), # (bs, 64, 64, 128)\n downsample(256, 4), # (bs, 32, 32, 256)\n downsample(512, 4), # (bs, 16, 16, 512)\n downsample(512, 4), # (bs, 8, 8, 512)\n downsample(512, 4), # (bs, 4, 4, 512)\n downsample(512, 4), # (bs, 2, 2, 512)\n downsample(512, 4), # (bs, 1, 1, 512)\n ]\n\n up_stack = [\n upsample(512, 4, apply_dropout=True), # (bs, 2, 2, 1024)\n upsample(512, 4, apply_dropout=True), # (bs, 4, 4, 1024)\n upsample(512, 4, apply_dropout=True), # (bs, 8, 8, 1024)\n upsample(512, 4), # (bs, 16, 16, 1024)\n upsample(256, 4), # (bs, 32, 32, 512)\n upsample(128, 4), # (bs, 64, 64, 256)\n upsample(64, 4), # (bs, 128, 128, 128)\n ]\n\n initializer = tf.random_normal_initializer(0., 0.02)\n last = layers.Conv2DTranspose(OUTPUT_CHANNELS, 4,\n strides=2,\n padding='same',\n kernel_initializer=initializer,\n activation='tanh') # (bs, 256, 256, 3)\n\n x = inputs\n\n # Downsampling through the model\n skips = []\n for down in down_stack:\n x = down(x)\n skips.append(x)\n\n skips = reversed(skips[:-1])\n\n # Upsampling and establishing the skip connections\n for up, skip in zip(up_stack, skips):\n x = up(x)\n x = layers.Concatenate()([x, skip])\n\n x = last(x)\n\n return keras.Model(inputs=inputs, outputs=x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def Discriminator():\n initializer = tf.random_normal_initializer(0., 0.02)\n gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n inp = layers.Input(shape=[256, 256, 3], name='input_image')\n\n x = inp\n\n down1 = downsample(64, 4, False)(x) # (bs, 128, 128, 64)\n down2 = downsample(128, 4)(down1) # (bs, 64, 64, 128)\n down3 = downsample(256, 4)(down2) # (bs, 32, 32, 256)\n\n zero_pad1 = layers.ZeroPadding2D()(down3) # (bs, 34, 34, 256)\n conv = layers.Conv2D(512, 4, strides=1,\n kernel_initializer=initializer,\n use_bias=False)(zero_pad1) # (bs, 31, 31, 512)\n\n norm1 = tfa.layers.InstanceNormalization(gamma_initializer=gamma_init)(conv)\n\n leaky_relu = layers.LeakyReLU()(norm1)\n\n zero_pad2 = layers.ZeroPadding2D()(leaky_relu) # (bs, 33, 33, 512)\n\n last = layers.Conv2D(1, 4, strides=1,\n kernel_initializer=initializer)(zero_pad2) # (bs, 30, 30, 1)\n \n \n\n return tf.keras.Model(inputs=inp, outputs=last)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n monet_generator = Generator() # transforms photos to Monet-esque paintings\n photo_generator = Generator() # transforms Monet paintings to be more like photos\n\n monet_discriminator = Discriminator() # differentiates real Monet paintings and generated Monet paintings\n photo_discriminator = Discriminator() # differentiates real photos and generated photos","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class CycleGan(keras.Model):\n def __init__(\n self,\n monet_generator,\n photo_generator,\n monet_discriminator,\n photo_discriminator,\n lambda_cycle=1,\n lambda_id=0.3,\n# lambda_GP=10, \n ):\n super(CycleGan, self).__init__()\n self.m_gen = monet_generator\n self.p_gen = photo_generator\n self.m_disc = monet_discriminator\n self.p_disc = photo_discriminator\n self.lambda_cycle = lambda_cycle\n self.lambda_id = lambda_id\n# self.lambda_GP = lambda_GP\n\n\n \n def compile(\n self,\n m_gen_optimizer,\n p_gen_optimizer,\n m_disc_optimizer,\n p_disc_optimizer,\n gen_loss_fn,\n disc_loss_fn,\n cycle_loss_fn,\n identity_loss_fn\n ):\n super(CycleGan, self).compile()\n self.m_gen_optimizer = m_gen_optimizer\n self.p_gen_optimizer = p_gen_optimizer\n self.m_disc_optimizer = m_disc_optimizer\n self.p_disc_optimizer = p_disc_optimizer\n self.gen_loss_fn = gen_loss_fn\n self.disc_loss_fn = disc_loss_fn\n self.cycle_loss_fn = cycle_loss_fn\n self.identity_loss_fn = identity_loss_fn\n \n \n def train_step(self, batch_data):\n real_monet, real_photo = batch_data\n \n with tf.GradientTape(persistent=True) as tape:\n # photo to monet back to photo\n \n# noise = tf.random.normal(shape = (256,256,3), mean = 0.0, stddev = 0.02, dtype = tf.float32) \n \n fake_monet = self.m_gen(real_photo, training=True)\n# cycled_photo = self.p_gen(fake_monet+noise, training=True)\n cycled_photo = self.p_gen(fake_monet, training=True)\n\n # monet to photo back to monet\n fake_photo = self.p_gen(real_monet, training=True)\n# cycled_monet = self.m_gen(fake_photo+noise, training=True)\n cycled_monet = self.m_gen(fake_photo, training=True)\n\n # generating itself\n# same_monet = self.m_gen(real_monet, training=True)\n# same_photo = self.p_gen(real_photo, training=True)\n\n # discriminator used to check, inputing real images\n disc_real_monet = self.m_disc(real_monet, training=True)\n disc_real_photo = self.p_disc(real_photo, training=True)\n\n # discriminator used to check, inputing fake images\n disc_fake_monet = self.m_disc(fake_monet, training=True)\n disc_fake_photo = self.p_disc(fake_photo, training=True)\n\n # discriminator used to check, inputing cycled images\n# disc_cycled_monet = self.m_disc(cycled_monet, training=True)\n# disc_cycled_photo = self.p_disc(cycled_photo, training=True)\n\n \n # evaluates generator loss\n monet_gen_loss = self.gen_loss_fn(disc_fake_monet)\n photo_gen_loss = self.gen_loss_fn(disc_fake_photo)\n# monet_gen_loss = self.gen_loss_fn(disc_fake_monet)+self.gen_loss_fn(disc_cycled_monet)\n# photo_gen_loss = self.gen_loss_fn(disc_fake_photo)+self.gen_loss_fn(disc_cycled_photo)\n# monet_gen_loss = -disc_fake_monet-disc_cycled_monet # W\n# photo_gen_loss = -disc_fake_photo-disc_cycled_photo # W\n# monet_gen_loss = -disc_fake_monet # W\n# photo_gen_loss = -disc_fake_photo # W\n\n\n # evaluates total cycle consistency loss\n cycle_loss_mpm = self.cycle_loss_fn(real_monet, cycled_monet, self.lambda_cycle)\n cycle_loss_pmp = self.cycle_loss_fn(real_photo, cycled_photo, self.lambda_cycle)\n total_cycle_loss = cycle_loss_mpm + cycle_loss_pmp\n\n # evaluates total generator loss\n# total_monet_gen_loss = monet_gen_loss + total_cycle_loss + self.identity_loss_fn(real_photo+noise, fake_monet, self.lambda_id)\n# total_photo_gen_loss = photo_gen_loss + total_cycle_loss + self.identity_loss_fn(real_monet+noise, fake_photo, self.lambda_id)\n# total_monet_gen_loss = monet_gen_loss + total_cycle_loss \n# total_photo_gen_loss = photo_gen_loss + total_cycle_loss \n total_monet_gen_loss = monet_gen_loss + total_cycle_loss + self.identity_loss_fn(real_monet, fake_photo, self.lambda_id)\n total_photo_gen_loss = photo_gen_loss + total_cycle_loss + self.identity_loss_fn(real_photo, fake_monet, self.lambda_id)\n\n # evaluates discriminator loss\n monet_disc_loss = self.disc_loss_fn(disc_real_monet, disc_fake_monet)\n# monet_disc_loss2 = self.disc_loss_fn(disc_real_monet, disc_cycled_monet)\n photo_disc_loss = self.disc_loss_fn(disc_real_photo, disc_fake_photo)\n# photo_disc_loss2 = self.disc_loss_fn(disc_real_photo, disc_cycled_photo)\n \n # evaluates discriminator loss gradient punishment\n# alpha_m_f = tf.random.uniform(shape=[BATCH_SIZE,1,1,1], minval=0.,maxval=1.)\n# alpha_p_f = tf.random.uniform(shape=[BATCH_SIZE,1,1,1], minval=0.,maxval=1.)\n# alpha_m_c = tf.random.uniform(shape=[BATCH_SIZE,1,1,1], minval=0.,maxval=1.)\n# alpha_p_c = tf.random.uniform(shape=[BATCH_SIZE,1,1,1], minval=0.,maxval=1.)\n \n# m_f_hat = alpha_m_f * real_monet+ (1.0-alpha_m_f) * fake_monet\n# m_c_hat = alpha_m_c * real_monet+ (1.0-alpha_m_c) * cycled_monet\n# p_f_hat = alpha_p_f * real_photo+ (1.0-alpha_p_f) * fake_photo\n# p_c_hat = alpha_p_c * real_photo+ (1.0-alpha_p_c) * cycled_photo\n \n# d_m_f_hat = self.m_disc(m_f_hat, training=False) # changed from False\n# d_m_c_hat = self.m_disc(m_c_hat, training=False)\n# d_p_f_hat = self.p_disc(p_f_hat, training=False)\n# d_p_c_hat = self.p_disc(p_c_hat, training=False)\n\n \n # mayby maximum with 0?\n# GP_m_f = tf.reduce_mean((tf.sqrt(tf.reduce_sum(tf.gradients(d_m_f_hat,m_f_hat)[0]**2,axis=[1,2,3]))-1.0)**2)\n# GP_m_f = tf.reduce_mean(tf.maximum(tf.sqrt(tf.reduce_sum(tf.gradients(d_m_f_hat,m_f_hat)[0]**2,axis=[1,2,3]))-1.0,0)**2)\n\n# GP_m_c = tf.reduce_mean(tf.maximum(tf.sqrt(tf.reduce_sum(tf.gradients(d_m_c_hat,m_c_hat)[0]**2,axis=[1,2,3]))-1.0,0)**2)\n# GP_p_f = tf.reduce_mean((tf.sqrt(tf.reduce_sum(tf.gradients(d_p_f_hat,p_f_hat)[0]**2,axis=[1,2,3]))-1.0)**2)\n# GP_p_f = tf.reduce_mean(tf.maximum(tf.sqrt(tf.reduce_sum(tf.gradients(d_p_f_hat,p_f_hat)[0]**2,axis=[1,2,3]))-1.0,0)**2)\n\n # GP_p_c = tf.reduce_mean(tf.maximum(tf.sqrt(tf.reduce_sum(tf.gradients(d_p_c_hat,p_c_hat)[0]**2,axis=[1,2,3]))-1.0,0)**2)\n\n # evaluates discriminator loss\n# monet_disc_loss = -2*disc_real_monet + disc_fake_monet + disc_cycled_monet + self.lambda_GP*GP_m_f + self.lambda_GP*GP_m_c # W\n# photo_disc_loss = -2*disc_real_photo + disc_fake_photo + disc_cycled_photo + self.lambda_GP*GP_p_f + self.lambda_GP*GP_p_c # W \n\n# monet_disc_loss = -disc_real_monet + disc_fake_monet + self.lambda_GP*GP_m_f # W\n# photo_disc_loss = -disc_real_photo + disc_fake_photo + self.lambda_GP*GP_p_f # W \n\n\n \n # Calculate the gradients for generator and discriminator\n monet_generator_gradients = tape.gradient(total_monet_gen_loss,\n self.m_gen.trainable_variables)\n photo_generator_gradients = tape.gradient(total_photo_gen_loss,\n self.p_gen.trainable_variables)\n\n monet_discriminator_gradients = tape.gradient(monet_disc_loss,\n self.m_disc.trainable_variables)\n photo_discriminator_gradients = tape.gradient(photo_disc_loss,\n self.p_disc.trainable_variables)\n\n # Apply the gradients to the optimizer\n self.m_gen_optimizer.apply_gradients(zip(monet_generator_gradients,\n self.m_gen.trainable_variables))\n\n self.p_gen_optimizer.apply_gradients(zip(photo_generator_gradients,\n self.p_gen.trainable_variables))\n\n self.m_disc_optimizer.apply_gradients(zip(monet_discriminator_gradients,\n self.m_disc.trainable_variables))\n\n self.p_disc_optimizer.apply_gradients(zip(photo_discriminator_gradients,\n self.p_disc.trainable_variables))\n \n return {\n \"monet_disc_loss\": monet_disc_loss,\n# \"monet_disc_loss2\": -disc_real_monet+disc_cycled_monet,\n \"photo_disc_loss\": photo_disc_loss,\n# \"photo_disc_loss2\": -disc_real_photo+disc_cycled_photo,\n \"cycle_loss_mpm\": cycle_loss_mpm,\n \"cycle_loss_pmp\": cycle_loss_pmp,\n \"disc_real_monet\": disc_real_monet,\n \"disc_fake_monet\": disc_fake_monet, \n \"disc_real_photo\": disc_real_photo, \n \"disc_fake_photo\": disc_fake_photo, \n }\n ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n def discriminator_loss(real, generated):\n real_loss = tf.square(tf.ones_like(real) - real)\n\n generated_loss = tf.square(tf.zeros_like(generated) - generated)\n\n total_disc_loss = real_loss + generated_loss\n\n return total_disc_loss * 0.5\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n def generator_loss(generated):\n return tf.square(tf.ones_like(generated) - generated)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n def calc_cycle_loss(real_image, cycled_image, LAMBDA):\n loss1 = tf.reduce_mean(tf.abs(inception_model(real_image) - inception_model(cycled_image)))\n return LAMBDA * loss1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n def identity_loss(real_image, translated_image, LAMBDA):\n loss = tf.reduce_mean(tf.abs(tf.nn.avg_pool2d(real_image, ksize=16, strides=4, padding=\"VALID\") - tf.nn.avg_pool2d(translated_image, ksize=16, strides=4, padding=\"VALID\")))\n return LAMBDA * loss","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n\n monet_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n photo_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n\n monet_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n photo_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with strategy.scope():\n cycle_gan_model = CycleGan(\n monet_generator, photo_generator, monet_discriminator, photo_discriminator\n )\n\n cycle_gan_model.compile(\n m_gen_optimizer = monet_generator_optimizer,\n p_gen_optimizer = photo_generator_optimizer,\n m_disc_optimizer = monet_discriminator_optimizer,\n p_disc_optimizer = photo_discriminator_optimizer,\n gen_loss_fn = generator_loss,\n disc_loss_fn = discriminator_loss,\n cycle_loss_fn = calc_cycle_loss,\n identity_loss_fn = identity_loss\n )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# callbacks = [keras.callbacks.ModelCheckpoint(filepath='monet.h5',save_weights_only=True,save_best_only=True, monitor='val_total_cycle_loss', verbose=1)]\ndisc_m_loss1=[]\ndisc_p_loss1=[]\n# cycle_gan_model.built = True\n# cycle_gan_model.load_weights('../input/cyclegan-with-dg-pretraining/monet.h5')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nfids=[]\nbest_fid=999999999\nfor epoch in range(1,32):\n\n print(\"Epoch = \",epoch)\n hist=cycle_gan_model.fit(gan_ds,steps_per_epoch=1500, epochs=1).history\n cur_fid=FID(fast_photo_ds,monet_generator)\n# disc_m_loss1.append(hist[\"monet_disc_loss1\"][0][0][0])\n# disc_p_loss1.append(hist[\"photo_disc_loss1\"][0][0][0])\n fids.append(cur_fid)\n print(\"After epoch #{} FID = {}\\n\".format(epoch,cur_fid))\n if cur_fid