{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"tpu1vmV38","dataSources":[{"sourceId":21755,"databundleVersionId":1475600,"sourceType":"competition"}],"dockerImageVersionId":30299,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"Note: The markdown texts are my notes from understanding the code blocks and from ChatGPT's explanations of some of the blocks\n- Kaggle competition \"[I'm Something of a Painter Myself](https://www.kaggle.com/competitions/gan-getting-started/overview)\"\n- Original file copied in Feb. 2024 from Saravana Kumar, which was copied from UnfriendlyAI","metadata":{}},{"cell_type":"markdown","source":"# GAN\n- **GAN** = Generative Adversarial Network\n- Comprised of:\n    1. **Generator**: takes random noise as input and generates images that resemble the training data\n    2. **Discriminator**: like a classifier that determines between real images of Monet from the training data and replications from the generator\n        - Trained on real and fake to learn to differentiate between them\n- Training a GAN: **Adversarial training**\n    - Generator tries to create realistic images that trick the discriminator, and the discriminator tries to catch the fakes from the generator\n    - Both improve iteratively","metadata":{}},{"cell_type":"markdown","source":"## Two-Objective Discriminator\n- Used with adversarial training\n- Two discriminators in this GAN\n    1. Real vs. Fake Discriminator: distinguishes between real images from dataset and fake ones from generator\n    2. Attribute/Class Discriminator: focuses on specific attribute/class of images, discriminating between images with certain style/color/ect.\n- Constraining the generator\n    - Must generate images that meet criteria of *both* discriminators","metadata":{}},{"cell_type":"code","source":"BATCH_SIZE = 15 # Setting the batch size as 32 (I changed this to help with the OOM error received later on)\n\n# Importing nesessary packages\nimport tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\nimport tensorflow_addons as tfa\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport re\ntry:\n    from kaggle_datasets import KaggleDatasets\nexcept:\n    pass","metadata":{"papermill":{"duration":6.682951,"end_time":"2021-05-03T04:05:28.79917","exception":false,"start_time":"2021-05-03T04:05:22.116219","status":"completed"},"tags":[],"id":"individual-handy","execution":{"iopub.status.busy":"2024-04-29T16:30:08.126458Z","iopub.execute_input":"2024-04-29T16:30:08.126715Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Checking the Runtime TPU\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Device:', tpu.master())\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nexcept:\n    strategy = tf.distribute.get_strategy()\nprint('Number of replicas:', strategy.num_replicas_in_sync)\n\nAUTOTUNE = tf.data.experimental.AUTOTUNE\nAUTO = tf.data.experimental.AUTOTUNE    \nprint(tf.__version__)","metadata":{"papermill":{"duration":6.164365,"end_time":"2021-05-03T04:05:34.988047","exception":false,"start_time":"2021-05-03T04:05:28.823682","status":"completed"},"tags":[],"id":"external-girlfriend","outputId":"83bc1eee-0a75-4189-a125-72368285feb2","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Getting the images and separating between Monet and photo images\nGCS_PATH = KaggleDatasets().get_gcs_path(\"gan-getting-started\")\n\nMONET_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/monet_tfrec/*.tfrec'))\nPHOTO_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/photo_tfrec/*.tfrec'))\n\nprint(len(MONET_FILENAMES))\nprint(len(PHOTO_FILENAMES))","metadata":{"papermill":{"duration":1.039093,"end_time":"2021-05-03T04:05:36.052642","exception":false,"start_time":"2021-05-03T04:05:35.013549","status":"completed"},"tags":[],"id":"atmospheric-vampire","outputId":"5b6d0bd6-57be-41df-ec98-b620b7909610","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### def data_augment_flip\n- Function that flips images with a 50% chance\n    - Image has 50% chance that flipped\n    - Imcreases randomness => increases diversity in training data\n    - Improves discriminator's performance","metadata":{}},{"cell_type":"code","source":"def data_augment_flip(image):\n    image = tf.image.random_flip_left_right(image)\n    return image","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### def decode_image & def read_tfrecord\n- Used to decode JPEG images from TFRecord format\n- Resizes, normalizes pixel values, prepares them for training","metadata":{}},{"cell_type":"code","source":"IMAGE_SIZE = [256, 256]\n\ndef decode_image(image):\n    image = tf.image.decode_jpeg(image, channels=3)\n    image = (tf.cast(image, tf.float32) / 127.5) - 1\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    return image\n\ndef read_tfrecord(example):\n    tfrecord_format = {\n        \"image\": tf.io.FixedLenFeature([], tf.string)\n    }\n    example = tf.io.parse_single_example(example, tfrecord_format)\n    image = decode_image(example['image'])\n    return image","metadata":{"papermill":{"duration":0.04111,"end_time":"2021-05-03T04:05:36.119243","exception":false,"start_time":"2021-05-03T04:05:36.078133","status":"completed"},"tags":[],"id":"labeled-aerospace","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### def load_dataset\n- Takes a list of filenames TFRecord files\n- Sends to previous function (read_tfrecord) => decodes image to JPEG\n- Returns Tensorflow Dataset object\n\n### ds Variables\n- Loading Monet dataset\n    - Batching the Monet dataset\n- Loading photo dataset w. larger batch size\n    - Batching the photo dataset\n- Basically preprocessing datasets and batching to improve training performance","metadata":{}},{"cell_type":"code","source":"def load_dataset(filenames):\n    dataset = tf.data.TFRecordDataset(filenames)\n    dataset = dataset.map(read_tfrecord, num_parallel_calls=AUTOTUNE)\n    return dataset\n\nmonet_ds = load_dataset(MONET_FILENAMES).batch(1)\nphoto_ds = load_dataset(PHOTO_FILENAMES).batch(1)\n\n\nfast_photo_ds = load_dataset(PHOTO_FILENAMES).batch(32*strategy.num_replicas_in_sync).prefetch(32)\nfid_photo_ds = load_dataset(PHOTO_FILENAMES).take(1024).batch(32*strategy.num_replicas_in_sync).prefetch(32)\nfid_monet_ds = load_dataset(MONET_FILENAMES).batch(32*strategy.num_replicas_in_sync).prefetch(32)","metadata":{"papermill":{"duration":0.266134,"end_time":"2021-05-03T04:05:36.475007","exception":false,"start_time":"2021-05-03T04:05:36.208873","status":"completed"},"tags":[],"id":"automatic-kernel","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### def get_gan_dataset\n- Preprocessing the Monet & photo datasets, mapping vectors to train on information in photos later\n- Zips Monet & photo datasets together (makes them a pair to train)\n- Combining Monet and photo datasets","metadata":{}},{"cell_type":"code","source":"def get_gan_dataset(monet_files, photo_files, augment=None, repeat=True, shuffle=True, batch_size=1):\n\n    monet_ds = load_dataset(monet_files)\n    photo_ds = load_dataset(photo_files)\n    \n\n        \n    if repeat:\n        monet_ds = monet_ds.repeat()\n        photo_ds = photo_ds.repeat()\n    if shuffle:\n        monet_ds = monet_ds.shuffle(2048)\n        photo_ds = photo_ds.shuffle(2048)\n        \n    monet_ds = monet_ds.batch(batch_size, drop_remainder=True)\n    photo_ds = photo_ds.batch(batch_size, drop_remainder=True)\n#     monet_ds = monet_ds.cache()\n#     photo_ds = photo_ds.cache()\n    if augment:\n        monet_ds = monet_ds.map(augment, num_parallel_calls=AUTO)\n        photo_ds = photo_ds.map(augment, num_parallel_calls=AUTO)\n        \n    monet_ds = monet_ds.prefetch(AUTO)\n    photo_ds = photo_ds.prefetch(AUTO)\n    \n    gan_ds = tf.data.Dataset.zip((monet_ds, photo_ds))\n    \n    return gan_ds","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### def get_photo_dataset\n- Basically same function as above, except this is just for photo preprocessing\n- Does not zip with Monet images","metadata":{}},{"cell_type":"code","source":"def get_photo_dataset(photo_files, augment=None, repeat=False, shuffle=False, batch_size=1):\n    photo_ds = load_dataset(photo_files)\n        \n    if repeat:\n        photo_ds = photo_ds.repeat()\n    if shuffle:\n        photo_ds = photo_ds.shuffle(2048)\n\n    photo_ds = photo_ds.batch(batch_size, drop_remainder=True)\n#     monet_ds = monet_ds.cache()\n#     photo_ds = photo_ds.cache()\n    if augment:\n        photo_ds = photo_ds.map(augment, num_parallel_calls=AUTO)\n    \n    photo_ds = photo_ds.prefetch(AUTO)\n    \n    return photo_ds\n\n","metadata":{"papermill":{"duration":0.037798,"end_time":"2021-05-03T04:05:36.538454","exception":false,"start_time":"2021-05-03T04:05:36.500656","status":"completed"},"tags":[],"id":"dynamic-consumption","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Getting a gan dataset with monet and photo\n# Pairing the monet to the photos\nfinal_dataset = get_gan_dataset(MONET_FILENAMES, PHOTO_FILENAMES, augment=data_augment_flip, repeat=True, shuffle=True, batch_size=BATCH_SIZE)","metadata":{"papermill":{"duration":0.086784,"end_time":"2021-05-03T04:05:36.651728","exception":false,"start_time":"2021-05-03T04:05:36.564944","status":"completed"},"tags":[],"id":"collaborative-impression","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# FID (Frechet Inception Distance): How performance is scored in this Kaggle Competition\n- Calculates FID scores (feature extraction with InceptionV3)","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n\n    inception_model = tf.keras.applications.InceptionV3(input_shape=(256,256,3),pooling=\"avg\",include_top=False)\n\n    mix3  = inception_model.get_layer(\"mixed9\").output\n    f0 = tf.keras.layers.GlobalAveragePooling2D()(mix3)\n\n    inception_model = tf.keras.Model(inputs=inception_model.input, outputs=f0)\n    inception_model.trainable = False\n\n    \n    \n    def calculate_activation_statistics_mod(images,fid_model):\n\n            act=tf.cast(fid_model.predict(images), tf.float32)\n\n            mu = tf.reduce_mean(act, axis=0)\n            mean_x = tf.reduce_mean(act, axis=0, keepdims=True)\n            mx = tf.matmul(tf.transpose(mean_x), mean_x)\n            vx = tf.matmul(tf.transpose(act), act)/tf.cast(tf.shape(act)[0], tf.float32)\n            sigma = vx - mx\n            return mu, sigma\n    myFID_mu2, myFID_sigma2 = calculate_activation_statistics_mod(fid_monet_ds,inception_model)        \n    fids=[]\n","metadata":{"papermill":{"duration":30.966971,"end_time":"2021-05-03T04:06:07.64481","exception":false,"start_time":"2021-05-03T04:05:36.677839","status":"completed"},"tags":[],"id":"enabling-rehabilitation","outputId":"a74dd122-3c29-49e3-d0e1-b489226d014e","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n    def calculate_frechet_distance(mu1,sigma1,mu2,sigma2):\n        fid_epsilon = 1e-14\n       \n        covmean = tf.linalg.sqrtm(tf.cast(tf.matmul(sigma1,sigma2),tf.complex64))\n#         isgood=tf.cast(tf.math.is_finite(covmean), tf.int32)\n#         if tf.size(isgood)!=tf.math.reduce_sum(isgood):\n#             return 0\n\n        covmean = tf.cast(tf.math.real(covmean),tf.float32)\n  \n        tr_covmean = tf.linalg.trace(covmean)\n\n        return tf.matmul(tf.expand_dims(mu1 - mu2, axis=0),tf.expand_dims(mu1 - mu2, axis=1)) + tf.linalg.trace(sigma1) + tf.linalg.trace(sigma2) - 2 * tr_covmean\n\n    \n    def FID(images,gen_model,inception_model=inception_model,myFID_mu2=myFID_mu2, myFID_sigma2=myFID_sigma2):\n                inp = layers.Input(shape=[256, 256, 3], name='input_image')\n                x  = gen_model(inp)\n                x=inception_model(x)\n                fid_model = tf.keras.Model(inputs=inp, outputs=x)\n                \n                mu1, sigma1= calculate_activation_statistics_mod(images,fid_model)\n\n                fid_value = calculate_frechet_distance(mu1, sigma1,myFID_mu2, myFID_sigma2)\n\n                return fid_value","metadata":{"papermill":{"duration":0.045472,"end_time":"2021-05-03T04:06:07.720198","exception":false,"start_time":"2021-05-03T04:06:07.674726","status":"completed"},"tags":[],"id":"subsequent-franklin","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### def down_sample\n- Creates downsampling layer for CNN (convolutional neural network) (used in the CycleGAN later)\n- Lowers the quality of the images to extract feature information\n    - Used in discriminator\n- Combines convolutonal layer with normalization","metadata":{}},{"cell_type":"code","source":"OUTPUT_CHANNELS = 3\n\ndef down_sample(filters, size, apply_instancenorm=True):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    layer = keras.Sequential()\n    layer.add(layers.Conv2D(filters, size, strides=2, padding='same', kernel_initializer=initializer, use_bias=False))\n\n    if apply_instancenorm:\n        layer.add(tfa.layers.InstanceNormalization(gamma_initializer=gamma_init))\n\n    layer.add(layers.LeakyReLU())\n\n    return layer","metadata":{"papermill":{"duration":0.039575,"end_time":"2021-05-03T04:06:07.790381","exception":false,"start_time":"2021-05-03T04:06:07.750806","status":"completed"},"tags":[],"id":"static-fluid","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### def up_sample\n- Creates upsampling layer for CNN\n- Increases quality/dimensions of photos \n    - Used in generator\n- Combines transposed convolutional layer w. normalization","metadata":{}},{"cell_type":"code","source":"def up_sample(filters, size, apply_dropout=False):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    layer = keras.Sequential()\n    layer.add(layers.Conv2DTranspose(filters, size, strides=2, padding='same', kernel_initializer=initializer,use_bias=False))\n    layer.add(tfa.layers.InstanceNormalization(gamma_initializer=gamma_init))\n\n    if apply_dropout:\n        layer.add(layers.Dropout(0.5))\n\n    layer.add(layers.ReLU())\n\n    return layer","metadata":{"papermill":{"duration":0.04203,"end_time":"2021-05-03T04:06:07.862571","exception":false,"start_time":"2021-05-03T04:06:07.820541","status":"completed"},"tags":[],"id":"fewer-playlist","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Generator\n- Image to Image translation\n- Downsampling layers --> Encoder\n    - down_stack\n    - Increasingly reducing spatial dimentions of the image\n    - Extracts features\n- Upsampling layers --> Decoder\n    - up_stack (sequence of upsampling layers)\n    - Increases the spatial dimensions\n    - Reconstructs the image from the extracted features\n- Generates realistic Monet images","metadata":{}},{"cell_type":"code","source":"def Generator():\n    inputs = layers.Input(shape=[256,256,3])\n    down_stack = [\n        down_sample(64, 4, apply_instancenorm=False),# (size, 128, 128, 64)\n        down_sample(128, 4),                         # (size, 64, 64, 128)\n        down_sample(256, 4),                         # (size, 32, 32, 256)\n        down_sample(512, 4),                         # (size, 16, 16, 512)\n        down_sample(512, 4),                         # (size, 8, 8, 512)\n        down_sample(512, 4),                         # (size, 4, 4, 512)\n        down_sample(512, 4),                         # (size, 2, 2, 512)\n        down_sample(512, 4),                         # (size, 1, 1, 512)\n    ]\n\n    up_stack = [\n        up_sample(512, 4, apply_dropout=True),       # (size, 2, 2, 1024)\n        up_sample(512, 4, apply_dropout=True),       # (size, 4, 4, 1024)\n        up_sample(512, 4, apply_dropout=True),       # (size, 8, 8, 1024)\n        up_sample(512, 4),                           # (size, 16, 16, 1024)\n        up_sample(256, 4),                           # (size, 32, 32, 512)\n        up_sample(128, 4),                           # (size, 64, 64, 256)\n        up_sample(64, 4),                            # (size, 128, 128, 128)\n    ]\n\n    initializer = tf.random_normal_initializer(0., 0.02)\n    last = layers.Conv2DTranspose(3, 4, strides=2, padding='same', kernel_initializer=initializer, activation='tanh') \n    # (size, 256, 256, 3)\n\n    x = inputs\n\n    # Downsampling through the model\n    skips = []\n    for down in down_stack:\n        x = down(x)\n        skips.append(x)\n\n    skips = reversed(skips[:-1])\n\n    # Upsampling and establishing the skip connections\n    for up, skip in zip(up_stack, skips):\n        x = up(x)\n        x = layers.Concatenate()([x, skip])\n\n    x = last(x)\n\n    return keras.Model(inputs=inputs, outputs=x)","metadata":{"papermill":{"duration":0.047878,"end_time":"2021-05-03T04:06:07.940505","exception":false,"start_time":"2021-05-03T04:06:07.892627","status":"completed"},"tags":[],"id":"judicial-petite","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Discriminator (Headless)\n- Takes images\n- Downsampling layers --> Encoder\n- Zero padding (x2) & convolutional layer\n- Constructs Keras Model w. image and outputs a zero-padded Leaky ReLU output\n- Discriminates between real Monet paintings and fake reproductions from GAN's Generator (above)\n\n- **GOAL** = for the discriminator to become *so* good at distinguishing real and fake Monet images that the generator has trouble fooling it\n    - When discriminator is good, the generator is also (results from the adversarial training: blessing & curse)\n        - Issues that arise include the generator becoming good at producing one aspect of the image and forgetting the others (Model collapse)","metadata":{}},{"cell_type":"markdown","source":"## Headless?\n- Means that the network doesn't have a final classification or output layer that produces specific prediction/decision\n- Outputs intermediate features that are used later for processing/analysis\n- This discriminator doesn't make a final output layer (=> binary classification: real or fake)\n    - Feature extraction\n    - **DOESN'T MEMORIZE CLASSIFICATION RESULTS, BUT INSTEAD LEARNS THE FEATURES THAT MAKE AN IMAGE REAL/FAKE**\n        - This is a lot better, because it won't just memorize the results; it will actually learn","metadata":{}},{"cell_type":"code","source":"# Discriminator learns to determine between real and fake images\n# Eventually helps generate images produced by GAN\ndef Discriminator():\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n    \n    inp = layers.Input(shape=[256, 256, 3], name='input_image')\n    x = inp\n    \n    down1 = down_sample(64, 4, False)(x)       # (size, 128, 128, 64)\n    down2 = down_sample(128, 4)(down1)         # (size, 64, 64, 128)\n    down3 = down_sample(256, 4)(down2)         # (size, 32, 32, 256)\n\n    zero_pad1 = layers.ZeroPadding2D()(down3) # (size, 34, 34, 256)\n    conv = layers.Conv2D(512, 4, strides=1, kernel_initializer=initializer, use_bias=False)(zero_pad1) # (size, 31, 31, 512)\n\n    norm1 = tfa.layers.InstanceNormalization(gamma_initializer=gamma_init)(conv)\n    leaky_relu = layers.LeakyReLU()(norm1)\n    zero_pad2 = layers.ZeroPadding2D()(leaky_relu) # (size, 33, 33, 512)\n\n    return tf.keras.Model(inputs=inp, outputs=zero_pad2) # Returns attributes of fake and real images that learned\n    # Doesn't return a prediction = headless","metadata":{"papermill":{"duration":0.043275,"end_time":"2021-05-03T04:06:08.014899","exception":false,"start_time":"2021-05-03T04:06:07.971624","status":"completed"},"tags":[],"id":"sudden-directory","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Head for two-objective discriminator for Monet\n- Defines discriminator head = part of discriminator\n- takes the feature maps from discriminator and => decision/score whether input is real/fake","metadata":{}},{"cell_type":"code","source":"def DHead():\n    initializer = tf.random_normal_initializer(0., 0.02)\n    \n    inp = layers.Input(shape=[33, 33, 512], name='input_image')\n    x = inp\n    \n    last = layers.Conv2D(1, 4, strides=1, kernel_initializer=initializer)(x) # (size, 30, 30, 1)\n\n    return tf.keras.Model(inputs=inp, outputs=last)","metadata":{"papermill":{"duration":0.039997,"end_time":"2021-05-03T04:06:08.085191","exception":false,"start_time":"2021-05-03T04:06:08.045194","status":"completed"},"tags":[],"id":"extreme-invention","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Why headless and then define a head?? Why not just have a head-discriminator?\n\n1. Headless: \n    - Lacks classification layers => training for generator\n    - Doesn't make any final decisions\n    - Keeps the generator from being influenced by the discriminator's classifications too early\n\n2. Head\n    - After the generator has been trained with headless, classification layers (**head**) added to discriminator\n    - Makes final decisions (real vs fake) during adversarial training process\n        - When both discriminator and generator trained together\n        \n        \n- - - - - - \n- Basically, we use the headless first so the generator actually learns the features \n- So it's not misled by the discriminator's classifications too early on in training (since they're both learning together)\n    - Don't want to lead the blind if you're blind yourself\n- Adding the head later helps with fine tuning the classification decisions","metadata":{}},{"cell_type":"markdown","source":"## Discriminator for Photos\n- Analyzes the features specifically in photos\n- Decides if it's real/fake","metadata":{}},{"cell_type":"code","source":"def DiscriminatorP():\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n    \n    inp = layers.Input(shape=[256, 256, 3], name='input_image')\n    x = inp\n    \n    down1 = down_sample(64, 4, False)(x)       # (size, 128, 128, 64)\n    down2 = down_sample(128, 4)(down1)         # (size, 64, 64, 128)\n    down3 = down_sample(256, 4)(down2)         # (size, 32, 32, 256)\n\n    zero_pad1 = layers.ZeroPadding2D()(down3) # (size, 34, 34, 256)\n    conv = layers.Conv2D(512, 4, strides=1, kernel_initializer=initializer, use_bias=False)(zero_pad1) # (size, 31, 31, 512)\n\n    norm1 = tfa.layers.InstanceNormalization(gamma_initializer=gamma_init)(conv)\n    leaky_relu = layers.LeakyReLU()(norm1)\n    zero_pad2 = layers.ZeroPadding2D()(leaky_relu) # (size, 33, 33, 512)\n    last = layers.Conv2D(1, 4, strides=1, kernel_initializer=initializer)(zero_pad2) # (size, 30, 30, 1)\n\n    return tf.keras.Model(inputs=inp, outputs=last)","metadata":{"papermill":{"duration":0.044783,"end_time":"2021-05-03T04:06:08.16026","exception":false,"start_time":"2021-05-03T04:06:08.115477","status":"completed"},"tags":[],"id":"protecting-fusion","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Why have two discriminators?\n- They're each tailored to different input types\n- Better analyze and decide authenticity ","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    monet_generator = Generator() # transforms photos to Monet-esque paintings\n    photo_generator = Generator() # transforms Monet paintings to be more like photos\n\n    monet_discriminator = Discriminator() # differentiates real Monet paintings and generated Monet paintings\n    photo_discriminator = DiscriminatorP() # differentiates real photos and generated photos\n    dHead1 = DHead() # Head for BCE\n    dHead2 = DHead() # Head for hinge loss\n","metadata":{"papermill":{"duration":10.251708,"end_time":"2021-05-03T04:06:18.443074","exception":false,"start_time":"2021-05-03T04:06:08.191366","status":"completed"},"tags":[],"id":"compliant-strike","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# CycleGAN with DiffAugment and two-objective discriminator\n- Training logic for the CycleGAN\n    - Compiles\n    - Train step method: custom, calculates losses and gradients","metadata":{}},{"cell_type":"code","source":"class CycleGan(keras.Model):\n    def __init__(\n        self,\n        monet_generator,\n        photo_generator,\n        monet_discriminator,\n        photo_discriminator,\n        dhead1,        \n        dhead2,        \n        lambda_cycle=3,\n        lambda_id=3,\n    ):\n        super(CycleGan, self).__init__()\n        self.m_gen = monet_generator\n        self.p_gen = photo_generator\n        self.m_disc = monet_discriminator\n        self.p_disc = photo_discriminator\n        self.lambda_cycle = lambda_cycle\n        self.lambda_id = lambda_id\n        self.dhead1 = dhead1\n        self.dhead2 = dhead2\n        \n    def compile(\n        self,\n        m_gen_optimizer,\n        p_gen_optimizer,\n        m_disc_optimizer,\n        p_disc_optimizer,\n        gen_loss_fn1,\n        gen_loss_fn2,\n        disc_loss_fn1,\n        disc_loss_fn2,\n        cycle_loss_fn,\n        identity_loss_fn,\n        aug_fn,\n\n    ):\n        super(CycleGan, self).compile()\n        self.m_gen_optimizer = m_gen_optimizer\n        self.p_gen_optimizer = p_gen_optimizer\n        self.m_disc_optimizer = m_disc_optimizer\n        self.p_disc_optimizer = p_disc_optimizer\n        self.gen_loss_fn1 = gen_loss_fn1\n        self.gen_loss_fn2 = gen_loss_fn2\n        self.disc_loss_fn1 = disc_loss_fn1\n        self.disc_loss_fn2 = disc_loss_fn2\n        self.cycle_loss_fn = cycle_loss_fn\n        self.identity_loss_fn = identity_loss_fn\n        self.aug_fn = aug_fn\n\n        self.step_num = 0\n        \n    def train_step(self, batch_data):\n        real_monet, real_photo = batch_data\n        batch_size = tf.shape(real_monet)[0]\n        with tf.GradientTape(persistent=True) as tape:\n        \n            # photo to monet back to photo\n            fake_monet = self.m_gen(real_photo, training=True)\n            cycled_photo = self.p_gen(fake_monet, training=True)\n\n            # monet to photo back to monet\n            fake_photo = self.p_gen(real_monet, training=True)\n            cycled_monet = self.m_gen(fake_photo, training=True)\n\n            # generating itself\n            same_monet = self.m_gen(real_monet, training=True)\n            same_photo = self.p_gen(real_photo, training=True)\n\n            \n            \n            # Diffaugment\n            both_monet = tf.concat([real_monet, fake_monet], axis=0)            \n            \n            aug_monet = self.aug_fn(both_monet)\n            \n            aug_real_monet = aug_monet[:batch_size]\n            aug_fake_monet = aug_monet[batch_size:]\n            \n            \n            # two-objective discriminator\n            disc_fake_monet1 = self.dhead1(self.m_disc(aug_fake_monet, training=True), training=True)\n            disc_real_monet1 = self.dhead1(self.m_disc(aug_real_monet, training=True), training=True)\n            disc_fake_monet2 = self.dhead2(self.m_disc(aug_fake_monet, training=True), training=True)\n            disc_real_monet2 = self.dhead2(self.m_disc(aug_real_monet, training=True), training=True)\n\n            monet_gen_loss1 = self.gen_loss_fn1(disc_fake_monet1) \n            monet_head_loss1 = self.disc_loss_fn1(disc_real_monet1, disc_fake_monet1)\n            monet_gen_loss2 = self.gen_loss_fn2(disc_fake_monet2)\n            monet_head_loss2 = self.disc_loss_fn2(disc_real_monet2, disc_fake_monet2)\n\n            monet_gen_loss = (monet_gen_loss1 + monet_gen_loss2) * 0.4\n            monet_disc_loss = monet_head_loss1 + monet_head_loss2\n\n            \n            # discriminator used to check, inputing real images\n\n            disc_real_photo = self.p_disc(real_photo, training=True)\n            # discriminator used to check, inputing fake images\n\n            disc_fake_photo = self.p_disc(fake_photo, training=True)\n\n            # evaluates generator loss\n\n            photo_gen_loss = self.gen_loss_fn1(disc_fake_photo)\n            \n            # evaluates discriminator loss\n\n            photo_disc_loss = self.disc_loss_fn1(disc_real_photo, disc_fake_photo)\n\n\n            # evaluates total generator loss\n            total_cycle_loss = self.cycle_loss_fn(real_monet, cycled_monet, self.lambda_cycle/ tf.cast(batch_size,tf.float32)) + self.cycle_loss_fn(real_photo, cycled_photo, self.lambda_cycle/ tf.cast(batch_size,tf.float32))\n\n            # evaluates total generator loss\n            total_monet_gen_loss = monet_gen_loss + total_cycle_loss +  self.identity_loss_fn(real_monet, same_monet, self.lambda_id / tf.cast(batch_size,tf.float32))\n            total_photo_gen_loss = photo_gen_loss + total_cycle_loss + self.identity_loss_fn(real_photo, same_photo, self.lambda_id/ tf.cast(batch_size,tf.float32))\n\n\n\n        # Calculate the gradients for generator and discriminator\n        monet_generator_gradients = tape.gradient(total_monet_gen_loss,\n                                                  self.m_gen.trainable_variables)\n        photo_generator_gradients = tape.gradient(total_photo_gen_loss,\n                                                  self.p_gen.trainable_variables)\n\n        monet_discriminator_gradients = tape.gradient(monet_disc_loss,\n                                                      self.m_disc.trainable_variables)\n        photo_discriminator_gradients = tape.gradient(photo_disc_loss,\n                                                      self.p_disc.trainable_variables)\n        \n\n        # Heads gradients\n        monet_head_gradients = tape.gradient(monet_head_loss1,\n                                                      self.dhead1.trainable_variables)\n\n        self.m_disc_optimizer.apply_gradients(zip(monet_head_gradients,\n                                                  self.dhead1.trainable_variables))       \n\n        monet_head_gradients = tape.gradient(monet_head_loss2,\n                                                      self.dhead2.trainable_variables)\n        self.m_disc_optimizer.apply_gradients(zip(monet_head_gradients,\n                                                  self.dhead2.trainable_variables))     \n        \n        \n        \n        # Apply the gradients to the optimizer\n        self.m_gen_optimizer.apply_gradients(zip(monet_generator_gradients,\n                                                 self.m_gen.trainable_variables))\n\n        self.p_gen_optimizer.apply_gradients(zip(photo_generator_gradients,\n                                                 self.p_gen.trainable_variables))\n\n        self.m_disc_optimizer.apply_gradients(zip(monet_discriminator_gradients,\n                                                  self.m_disc.trainable_variables))\n\n        self.p_disc_optimizer.apply_gradients(zip(photo_discriminator_gradients,\n                                                  self.p_disc.trainable_variables))\n        \n        return {\n            \"monet_head_loss1\": monet_head_loss1, \n            \"monet_head_loss2\": monet_head_loss2, \n            \"disc_real_monet\": disc_real_monet1, \n            \"disc_fake_monet\": disc_fake_monet1, \n            \"disc_real_monet2\": disc_real_monet2, \n            \"disc_fake_monet2\": disc_fake_monet2, \n            \"monet_gen_loss\": monet_gen_loss, \n            \"photo_disc_loss\": photo_disc_loss, \n            }\n","metadata":{"papermill":{"duration":0.069666,"end_time":"2021-05-03T04:06:18.54469","exception":false,"start_time":"2021-05-03T04:06:18.475024","status":"completed"},"tags":[],"id":"surrounded-seven","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Two loss functions for discriminator\n1. Loss function #1: uses hinge loss formulation\n    - Real and generated data as inputs, and penalizes discriminator when incorecctly classifies incorrectly\n    - Total loss = sum of losses\n    \n2. Loss funciton #2: uses Binary Cross Entropy (BCE) loss\n    - Handles logits (unnormalized outputs) and applies sigmoid activation internally\n    - Same as other loss funciton: penalizes when classifies incorrectly\n\n- - - - \n- We have two loss functions defined because they are both helpful in different areas of training.\n\n### Binary Cross Entropy Loss Explanation\n- Binary classification tasks\n- Measures difference bw 2 probability distrobutions: predicted probabilities & actual target probabilities","metadata":{}},{"cell_type":"code","source":"with strategy.scope(): # for TPU\n\n    def discriminator_loss1(real, generated):\n        real_loss = tf.math.minimum(tf.zeros_like(real), real-tf.ones_like(real))\n\n        generated_loss = tf.math.minimum(tf.zeros_like(generated), -generated-tf.ones_like(generated))\n\n        total_disc_loss = real_loss + generated_loss\n\n        return tf.reduce_mean(-total_disc_loss * 0.5)\n\n    def discriminator_loss2(real, generated):\n        generated_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(generated), generated)\n        real_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.zeros_like(real), real)\n        total_disc_loss = real_loss + generated_loss\n\n        return tf.reduce_mean(total_disc_loss * 0.5)\n\n","metadata":{"papermill":{"duration":0.043363,"end_time":"2021-05-03T04:06:18.621276","exception":false,"start_time":"2021-05-03T04:06:18.577913","status":"completed"},"tags":[],"id":"alpine-physics","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Two loss functions for generator\n\n1. Generator loss function #1: Simple negative mean formulation\n    - Takes generated data as input => negative mean of generated data\n    - Encourages the generator to produce data that discriminator classifies as real\n    \n    Soooo....\n    1. Generates fake data w. generator\n    2. Calculates the \"distance\" bw generated data & real data (w. binary cross entropy usually)\n    3. Take negative mean of distance measure\n    4. Basically penalizes the generator for generated data that's '\"farther\" than the real, input data\n    \n2. Generator loss function #2: Binary cross entropy (BCE) loss\n    - Same as before basically...\n    - Considered more sophisticated by ChatGPT","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    def generator_loss1(generated):\n        return  tf.reduce_mean(-generated)\n\n    def generator_loss2(generated):\n        return tf.reduce_mean(tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.zeros_like(generated), generated))","metadata":{"papermill":{"duration":0.042475,"end_time":"2021-05-03T04:06:18.695758","exception":false,"start_time":"2021-05-03T04:06:18.653283","status":"completed"},"tags":[],"id":"immune-novelty","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## So we're calculating loss for both discriminators and generators?\n- Yep.\n- Loss functions for both the discriminator and the generator\n- Help train CycleGAN by => gradients that update parameters of generator & discriminator => better performance","metadata":{}},{"cell_type":"markdown","source":"## calc_cycle_loss\n- Calculates cycle consistency loss\n- Ensures that image translated from domain A -> domain B and then back again should keep ~ original image from A\n- Can update LAMBDA (hyperparameter) to control influence of cycle gan consistency loss","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    def calc_cycle_loss(real_image, cycled_image, LAMBDA):\n        loss1 = tf.reduce_sum(tf.abs(real_image - cycled_image))\n\n        return LAMBDA * loss1 * 0.0000152587890625","metadata":{"papermill":{"duration":0.04152,"end_time":"2021-05-03T04:06:18.770802","exception":false,"start_time":"2021-05-03T04:06:18.729282","status":"completed"},"tags":[],"id":"violent-miami","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## identity_loss\n- Similar to calc_cycle_loss, but *preserves important features of original image during translation*","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    def identity_loss(real_image, same_image, LAMBDA):\n        loss = tf.reduce_sum(tf.abs(real_image - same_image))\n        return LAMBDA * 0.5 * loss * 0.0000152587890625","metadata":{"papermill":{"duration":0.041199,"end_time":"2021-05-03T04:06:18.843205","exception":false,"start_time":"2021-05-03T04:06:18.802006","status":"completed"},"tags":[],"id":"permanent-weekly","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Why 'with strategy.scope()'?\n- Ensures that loss functions executed within specified distributed training context (TPU's / GPU's)\n- Keeps training consistent","metadata":{}},{"cell_type":"markdown","source":"## Differentiable Augmentation for Data-Efficient GAN Training\n- Data augmentation in training (brightness, saturation, contrast....ect.)\n    - Combined => training data to make model more robust","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n# Differentiable Augmentation for Data-Efficient GAN Training\n# Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han\n# https://arxiv.org/pdf/2006.10738\n# from https://github.com/mit-han-lab/data-efficient-gans/blob/master/DiffAugment_tf.py\n\n\n\n    def DiffAugment(x, policy='', channels_first=False):\n        if policy:\n            if channels_first:\n                x = tf.transpose(x, [0, 2, 3, 1])\n            for p in policy.split(','):\n                for f in AUGMENT_FNS[p]:\n                    x = f(x)\n            if channels_first:\n                x = tf.transpose(x, [0, 3, 1, 2])\n        return x\n\n\n    def rand_brightness(x):\n        magnitude = tf.random.uniform([tf.shape(x)[0], 1, 1, 1]) - 0.5\n        x = x + magnitude\n        return x\n\n\n    def rand_saturation(x):\n        magnitude = tf.random.uniform([tf.shape(x)[0], 1, 1, 1]) * 2\n        x_mean = tf.reduce_sum(x, axis=3, keepdims=True) * 0.3333333333333333333\n        x = (x - x_mean) * magnitude + x_mean\n        return x\n\n\n    def rand_contrast(x):\n        magnitude = tf.random.uniform([tf.shape(x)[0], 1, 1, 1]) + 0.5\n        x_mean = tf.reduce_sum(x, axis=[1, 2, 3], keepdims=True) * 5.086e-6\n        x = (x - x_mean) * magnitude + x_mean\n        return x\n\n    def rand_translation(x, ratio=0.125):\n        batch_size = tf.shape(x)[0]\n        image_size = tf.shape(x)[1:3]\n        shift = tf.cast(tf.cast(image_size, tf.float32) * ratio + 0.5, tf.int32)\n        translation_x = tf.random.uniform([batch_size, 1], -shift[0], shift[0] + 1, dtype=tf.int32)\n        translation_y = tf.random.uniform([batch_size, 1], -shift[1], shift[1] + 1, dtype=tf.int32)\n        grid_x = tf.clip_by_value(tf.expand_dims(tf.range(image_size[0], dtype=tf.int32), 0) + translation_x + 1, 0, image_size[0] + 1)\n        grid_y = tf.clip_by_value(tf.expand_dims(tf.range(image_size[1], dtype=tf.int32), 0) + translation_y + 1, 0, image_size[1] + 1)\n        x = tf.gather_nd(tf.pad(x, [[0, 0], [1, 1], [0, 0], [0, 0]]), tf.expand_dims(grid_x, -1), batch_dims=1)\n        x = tf.transpose(tf.gather_nd(tf.pad(tf.transpose(x, [0, 2, 1, 3]), [[0, 0], [1, 1], [0, 0], [0, 0]]), tf.expand_dims(grid_y, -1), batch_dims=1), [0, 2, 1, 3])\n        return x\n\n\n    def rand_cutout(x, ratio=0.5):\n        batch_size = tf.shape(x)[0]\n        image_size = tf.shape(x)[1:3]\n        cutout_size = tf.cast(tf.cast(image_size, tf.float32) * ratio + 0.5, tf.int32)\n        offset_x = tf.random.uniform([tf.shape(x)[0], 1, 1], maxval=image_size[0] + (1 - cutout_size[0] % 2), dtype=tf.int32)\n        offset_y = tf.random.uniform([tf.shape(x)[0], 1, 1], maxval=image_size[1] + (1 - cutout_size[1] % 2), dtype=tf.int32)\n        grid_batch, grid_x, grid_y = tf.meshgrid(tf.range(batch_size, dtype=tf.int32), tf.range(cutout_size[0], dtype=tf.int32), tf.range(cutout_size[1], dtype=tf.int32), indexing='ij')\n        cutout_grid = tf.stack([grid_batch, grid_x + offset_x - cutout_size[0] // 2, grid_y + offset_y - cutout_size[1] // 2], axis=-1)\n        mask_shape = tf.stack([batch_size, image_size[0], image_size[1]])\n        cutout_grid = tf.maximum(cutout_grid, 0)\n        cutout_grid = tf.minimum(cutout_grid, tf.reshape(mask_shape - 1, [1, 1, 1, 3]))\n        mask = tf.maximum(1 - tf.scatter_nd(cutout_grid, tf.ones([batch_size, cutout_size[0], cutout_size[1]], dtype=tf.float32), mask_shape), 0)\n        x = x * tf.expand_dims(mask, axis=3)\n        return x\n\n\n    AUGMENT_FNS = {\n        'color': [rand_brightness, rand_saturation, rand_contrast],\n        'translation': [rand_translation],\n        'cutout': [rand_cutout],\n}\n    def aug_fn(image):\n        return DiffAugment(image,\"color,translation,cutout\")","metadata":{"papermill":{"duration":0.063213,"end_time":"2021-05-03T04:06:18.938375","exception":false,"start_time":"2021-05-03T04:06:18.875162","status":"completed"},"tags":[],"id":"located-trout","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Optimizers\n- Adam = Adaptive Moment Estimation = combo of momentum and Root Mean Square Propogation (RMSProp)\n    - Dynamically changes learing rate for each parameter\n    - Momentum to accelerate gradient decent\n    - Bias correction to adjust estimates for moments of gradients\n    - Efficient\n    \n1. Learning rate = 2e-4\n    - Step size that optimizer updates weights during training\n    - Stable, gradual updates to model parameters\n2. Beta value = 0.5\n    - Controls exponential decay rate for mean of gradients\n    - Past gradients have less influence on current update","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    monet_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    photo_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n\n    monet_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    photo_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)","metadata":{"papermill":{"duration":0.04133,"end_time":"2021-05-03T04:06:19.011148","exception":false,"start_time":"2021-05-03T04:06:18.969818","status":"completed"},"tags":[],"id":"joined-qualification","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## cycle_gan_model\n- Defining cycle gan!\n- Takes the generators and discriminators we defined earlier and combo's with head from discriminator!\n- Learns mapping bw 2 different domains (Monet & photos) and generates realistic images that mimic Monet","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    cycle_gan_model = CycleGan(\n        monet_generator, photo_generator, monet_discriminator, photo_discriminator, dHead1,  dHead2\n    )\n\n","metadata":{"papermill":{"duration":0.100669,"end_time":"2021-05-03T04:06:19.143362","exception":false,"start_time":"2021-05-03T04:06:19.042693","status":"completed"},"tags":[],"id":"underlying-diabetes","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Testing with different epoch #'s and alpha and beta values\n- Compile to prepare model for training by specifying how it should be optimized and what loss functions to use","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n\n    cycle_gan_model.compile(\n        m_gen_optimizer = monet_generator_optimizer,\n        p_gen_optimizer = photo_generator_optimizer,\n        m_disc_optimizer = monet_discriminator_optimizer,\n        p_disc_optimizer = photo_discriminator_optimizer,\n        gen_loss_fn1 = generator_loss1,\n        gen_loss_fn2 = generator_loss2,\n        disc_loss_fn1 = discriminator_loss1,\n        disc_loss_fn2 = discriminator_loss2,\n        cycle_loss_fn = calc_cycle_loss,\n        identity_loss_fn = identity_loss,\n        aug_fn = aug_fn ,\n\n\n    )","metadata":{"id":"unIOk8SNJz_y","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## cycle_gan_model.fit\n- Fitting w. specific training configuration\n- Evaluates FID distance bw photos and generated images w. generator","metadata":{}},{"cell_type":"code","source":"cycle_gan_model.fit(final_dataset,steps_per_epoch=1407, epochs=26)\nFID(fid_photo_ds, monet_generator) ","metadata":{"id":"Ms4V35Ar2ZNS","outputId":"b4004129-a7fe-4e21-8a02-f4d759fdfa65","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# OOM Error:\n- My GPU often ran out of memory at this point\n- - - \nSolutions produced by ChatGPT:\n\n1. **Reduce Batch Size:** Decrease the batch size used during training. A smaller batch size requires less memory but may also slow down training.\n\n2. **Reduce Model Complexity:** Simplify the architecture of your neural network, especially in terms of the number of layers, units, and parameters.\n\n3. **Use a Different GPU:** If possible, switch to a GPU with more memory or use a cloud-based service that provides access to more powerful GPUs.\n\n4. **Memory Management:** Check if there are any memory leaks in your code or if there are unnecessary tensors being stored in memory during training.\n\n5. **Gradient Accumulation:** Implement gradient accumulation techniques to simulate larger batch sizes without increasing memory usage.\n\n6. **Memory Profiling:** Use TensorFlow's memory profiler to analyze memory usage and identify areas where memory is being consumed excessively.\n\n7. **Limit Data Loading:** If using large datasets, consider loading data in batches rather than all at once to reduce memory consumption.","metadata":{}},{"cell_type":"markdown","source":"## Compiling w. a Lower Learning Rate (2e-4 -> 1e-4)\n- Updates parameters more slowly during training","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    monet_generator_optimizer = tf.keras.optimizers.Adam(1e-4, beta_1=0.5)\n    photo_generator_optimizer = tf.keras.optimizers.Adam(1e-4, beta_1=0.5)\n\n    monet_discriminator_optimizer = tf.keras.optimizers.Adam(1e-4, beta_1=0.5)\n    photo_discriminator_optimizer = tf.keras.optimizers.Adam(1e-4, beta_1=0.5)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n\n    cycle_gan_model.compile(\n        m_gen_optimizer = monet_generator_optimizer,\n        p_gen_optimizer = photo_generator_optimizer,\n        m_disc_optimizer = monet_discriminator_optimizer,\n        p_disc_optimizer = photo_discriminator_optimizer,\n        gen_loss_fn1 = generator_loss1,\n        gen_loss_fn2 = generator_loss2,\n        disc_loss_fn1 = discriminator_loss1,\n        disc_loss_fn2 = discriminator_loss2,\n        cycle_loss_fn = calc_cycle_loss,\n        identity_loss_fn = identity_loss,\n        aug_fn = aug_fn ,\n    )","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cycle_gan_model.fit(final_dataset,steps_per_epoch=1407, epochs=18)\nFID(fid_photo_ds,monet_generator) ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Compiling w. an even Smaller Learning Rate (1e-4 -> 1e-5)","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    monet_generator_optimizer = tf.keras.optimizers.Adam(1e-5, beta_1=0.5)\n    photo_generator_optimizer = tf.keras.optimizers.Adam(1e-5, beta_1=0.5)\n\n    monet_discriminator_optimizer = tf.keras.optimizers.Adam(1e-5, beta_1=0.5)\n    photo_discriminator_optimizer = tf.keras.optimizers.Adam(1e-5, beta_1=0.5)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n\n    cycle_gan_model.compile(\n        m_gen_optimizer = monet_generator_optimizer,\n        p_gen_optimizer = photo_generator_optimizer,\n        m_disc_optimizer = monet_discriminator_optimizer,\n        p_disc_optimizer = photo_discriminator_optimizer,\n        gen_loss_fn1 = generator_loss1,\n        gen_loss_fn2 = generator_loss2,\n        disc_loss_fn1 = discriminator_loss1,\n        disc_loss_fn2 = discriminator_loss2,\n        cycle_loss_fn = calc_cycle_loss,\n        identity_loss_fn = identity_loss,\n        aug_fn = aug_fn ,\n    )","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cycle_gan_model.fit(final_dataset,steps_per_epoch=1407, epochs=8)\nFID(fid_photo_ds,monet_generator) ","metadata":{"papermill":{"duration":345.558625,"end_time":"2021-05-03T08:27:47.892184","exception":false,"start_time":"2021-05-03T08:22:02.333559","status":"completed"},"tags":[],"id":"acquired-breed","outputId":"e4e223b3-e6db-4424-9710-ae4766e691bf","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Showing the images","metadata":{}},{"cell_type":"code","source":"ds_iter = iter(photo_ds)\nfor n_sample in range(8):\n        example_sample = next(ds_iter)\n        generated_sample = monet_generator(example_sample)\n        \n        f = plt.figure(figsize=(32, 32))\n        \n        plt.subplot(121)\n        plt.title('Input image')\n        plt.imshow(example_sample[0] * 0.5 + 0.5)\n        plt.axis('off')\n        \n        plt.subplot(122)\n        plt.title('Generated image')\n        plt.imshow(generated_sample[0] * 0.5 + 0.5)\n        plt.axis('off')\n        plt.show()","metadata":{"papermill":{"duration":31.920165,"end_time":"2021-05-03T08:28:46.022003","exception":false,"start_time":"2021-05-03T08:28:14.101838","status":"completed"},"tags":[],"id":"clear-subsection","outputId":"3569fd63-e08e-488a-c023-4f5c6828e9cb","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Showing the monet images","metadata":{}},{"cell_type":"code","source":"ds_iter = iter(monet_ds)\nfor n_sample in range(8):\n\n        example_sample = next(ds_iter)\n        generated_sample = photo_generator(example_sample)\n        \n        f = plt.figure(figsize=(24, 24))\n        \n        plt.subplot(121)\n        plt.title('Input image')\n        plt.imshow(example_sample[0] * 0.5 + 0.5)\n        plt.axis('off')\n        \n        plt.subplot(122)\n        plt.title('Generated image')\n        plt.imshow(generated_sample[0] * 0.5 + 0.5)\n        plt.axis('off')\n        plt.show()","metadata":{"papermill":{"duration":36.884887,"end_time":"2021-05-03T08:29:49.326769","exception":false,"start_time":"2021-05-03T08:29:12.441882","status":"completed"},"tags":[],"id":"actual-reader","outputId":"82845e69-16b4-4ac1-f34f-fc5e28938c70","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import PIL\n! mkdir ../images","metadata":{"papermill":{"duration":27.421043,"end_time":"2021-05-03T08:30:43.35362","exception":false,"start_time":"2021-05-03T08:30:15.932577","status":"completed"},"tags":[],"id":"educational-taxation","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Saving Predictions as jpg's\n- Iterates over images in photo and produces predictions from generator\n- Converts predictions to image format and saves them","metadata":{}},{"cell_type":"code","source":"%%time\ni = 1\nfor img in fast_photo_ds:\n    prediction = monet_generator(img, training=False).numpy()\n    prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n    for pred in prediction:\n        im = PIL.Image.fromarray(pred)\n        im.save(\"../images/\" + str(i) + \".jpg\")\n        i += 1\n    ","metadata":{"papermill":{"duration":308.75455,"end_time":"2021-05-03T08:36:18.774144","exception":false,"start_time":"2021-05-03T08:31:10.019594","status":"completed"},"tags":[],"id":"attended-venture","outputId":"a22cd9cf-e6b1-4e28-e1b8-c49d7cd6d07f","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import shutil\nshutil.make_archive(\"/kaggle/working/images\", 'zip', \"/kaggle/images\")","metadata":{"papermill":{"duration":30.599746,"end_time":"2021-05-03T08:37:15.966355","exception":false,"start_time":"2021-05-03T08:36:45.366609","status":"completed"},"tags":[],"id":"expired-battery","outputId":"fab9dc38-ce09-4de4-ec3e-36ecb491b3ae","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Saving weights\n- Weights = trainable parameters of specified models, internal state of model that adjusted during training to minimize loss function & improve performance\n- Later loaded to restore model's states\n- Can reuse wo. retraining models from scratch","metadata":{}},{"cell_type":"code","source":"\ndHead1.save_weights(\"dHead1.h5\")\ndHead2.save_weights(\"dHead2.h5\")\n\nphoto_generator.save_weights(\"photo_generator.h5\")\nmonet_generator.save_weights(\"monet_generator.h5\")\n\nphoto_discriminator.save_weights(\"photo_discriminator.h5\")\nmonet_discriminator.save_weights(\"monet_discriminator.h5\")","metadata":{"papermill":{"duration":26.776775,"end_time":"2021-05-03T08:38:09.478891","exception":false,"start_time":"2021-05-03T08:37:42.702116","status":"completed"},"tags":[],"id":"sharp-thesis","trusted":true},"execution_count":null,"outputs":[]}]}