{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":21755,"databundleVersionId":1475600,"sourceType":"competition"}],"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"USE_CACHED_MODELS = True","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:54:25.233708Z","iopub.execute_input":"2024-08-05T16:54:25.234623Z","iopub.status.idle":"2024-08-05T16:54:25.245872Z","shell.execute_reply.started":"2024-08-05T16:54:25.234585Z","shell.execute_reply":"2024-08-05T16:54:25.244949Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Introduction\n\n<casey.durfee@colorado.edu>\n\nFor this assignment, we've been given the task of completing the \"I'm something of a painter myself\" challenge on kaggle. We were instructed we could use the [recommended CycleGAN tutorial](https://www.kaggle.com/code/amyjang/monet-cyclegan-tutorial) as a jumping off point.\n\n## The Big Idea behind GANs\nIn the Beatles song \"You Never Give Me Your Money\", Paul McCartney laments that instead of money, you always give him \"your funny paper\". In this situation, Paul has to discriminate whether the bill you gave him was \"funny paper\" or not. He was able to identify it as fake, but perhaps, you might try making another fake bill. If you manage to fool Paul, the next time he will know what to look for, and thus be harder to deceive. \n\nIf the digital equivalents of you and Paul kept repeating this cycle over and over, eventually the digital you will get extremely good at creating fake money, and the digital Paul will get extremely good at detecting funny paper. The digital you is called the Generator, and the digital Paul is called the Discriminator. The two together form a Generative Adversarial Network (GAN). You have two AI models that start off not knowing anything, but they end up teaching each other.\n\nIn this problem, we are given a set of photos of the real world, and a set of Monet paintings. The Generator tries to create fake Monets based on real images, and the Discriminator tries to tell whether they are real or fake. Over time, the Generator will get good at art forgery, which will be useful for what I swear are completely legal reasons.\n\n## CycleGAN\nCycleGAN is a GAN designed to translate images from one domain to another. It actually trains two discriminators and two generators. Although we're only interested in turning photos into monets, CycleGAN will also learn to turn monets into photos.\n\nUnsupervised learning requires guardrails to keep from going way off track. There's no expert or supervisor to judge whether the art is actually good. The CycleGAN architecture [Zhu 2017] uses penalties, cycle loss and identity loss, to keep things from getting too weird. \n\nCycle loss is the amount of error when transforming from photo to monet and back (or from monet to photo and back). In a perfect situation, the monet to photo operation (M) and the photo to monet operation (P) would be inverses of each other, so if X is a photo, P(M(X)) = X. \n\nIdentity loss is the amount of change from passing a real photo through the \"make a fake photo\" filter or passing a real Monet through the \"make a fake monet\" filter. The authors of the original paper found that cycle loss alone leads to poor results. Intuitively, it makes sense: a real Monet is already as Monet-like as an image can be. So trying to Monet-ify it even further should ideally lead to zero changes to the image.\n\nI am going to start with the tutorial code, then try and improve on the Generator and Discriminator models.\n\n## A note on sources\nThe next section is taken directly from the tutorial. A complete list of references is at the end. My work starts with the \"Original Work Starts Here\" heading.\n\nI had to change code a bit because tensorflow_addons is no longer supported. According to this page, the group normalization layer with groups=-1 should be equivalent: https://keras.io/api/layers/normalization_layers/group_normalization/\n\nI was also unable to get the code to run with TPUs. it will only work on GPUs. I get the following error when I use the TPU:\n\n```Shapes of all inputs must match: values[0].shape = [1,30,30] != values[1].shape = [0,30,30]\n\t [[{{node Mean_3/input}}]] [Op:__inference_one_step_on_iterator_103553]```","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers\n#import tensorflow_addons as tfa\n\nfrom kaggle_datasets import KaggleDatasets\nimport matplotlib.pyplot as plt\nimport numpy as np\n\ntry:\n    tpu = tf.distribute.cluster_resolver.TPUClusterResolver()\n    print('Device:', tpu.master())\n    tf.config.experimental_connect_to_cluster(tpu)\n    tf.tpu.experimental.initialize_tpu_system(tpu)\n    strategy = tf.distribute.experimental.TPUStrategy(tpu)\nexcept:\n    if tf.config.list_physical_devices('GPU'):\n        strategy = tf.distribute.MirroredStrategy()\n        print(\"Running on GPU.\")\n    else:\n        strategy = tf.distribute.get_strategy()\n        print(\"Running on CPU.\")\nprint('Number of replicas:', strategy.num_replicas_in_sync)\n\nAUTOTUNE = tf.data.experimental.AUTOTUNE\n    \nprint(tf.__version__)","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","execution":{"iopub.status.busy":"2024-08-05T16:54:25.247328Z","iopub.execute_input":"2024-08-05T16:54:25.247588Z","iopub.status.idle":"2024-08-05T16:54:29.388105Z","shell.execute_reply.started":"2024-08-05T16:54:25.247566Z","shell.execute_reply":"2024-08-05T16:54:29.387149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"GCS_PATH = KaggleDatasets().get_gcs_path()\nMONET_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/monet_tfrec/*.tfrec'))\nprint('Monet TFRecord Files:', len(MONET_FILENAMES))\n\nPHOTO_FILENAMES = tf.io.gfile.glob(str(GCS_PATH + '/photo_tfrec/*.tfrec'))\nprint('Photo TFRecord Files:', len(PHOTO_FILENAMES))\nIMAGE_SIZE = [256, 256]\n\ndef decode_image(image):\n    image = tf.image.decode_jpeg(image, channels=3)\n    image = (tf.cast(image, tf.float32) / 127.5) - 1\n    image = tf.reshape(image, [*IMAGE_SIZE, 3])\n    return image\n\ndef read_tfrecord(example):\n    tfrecord_format = {\n        \"image_name\": tf.io.FixedLenFeature([], tf.string),\n        \"image\": tf.io.FixedLenFeature([], tf.string),\n        \"target\": tf.io.FixedLenFeature([], tf.string)\n    }\n    example = tf.io.parse_single_example(example, tfrecord_format)\n    image = decode_image(example['image'])\n    return image\n\ndef load_dataset(filenames, labeled=True, ordered=False):\n    dataset = tf.data.TFRecordDataset(filenames)\n    dataset = dataset.map(read_tfrecord, num_parallel_calls=AUTOTUNE)\n    return dataset","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:54:29.390314Z","iopub.execute_input":"2024-08-05T16:54:29.390984Z","iopub.status.idle":"2024-08-05T16:54:30.028781Z","shell.execute_reply.started":"2024-08-05T16:54:29.390949Z","shell.execute_reply":"2024-08-05T16:54:30.027850Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"monet_ds = load_dataset(MONET_FILENAMES, labeled=True).batch(1)\nphoto_ds = load_dataset(PHOTO_FILENAMES, labeled=True).batch(1)\nexample_monet = next(iter(monet_ds))\nexample_photo = next(iter(photo_ds))\nplt.subplot(121)\nplt.title('Photo')\nplt.imshow(example_photo[0] * 0.5 + 0.5)\n\nplt.subplot(122)\nplt.title('Monet')\nplt.imshow(example_monet[0] * 0.5 + 0.5)","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:54:30.029900Z","iopub.execute_input":"2024-08-05T16:54:30.030233Z","iopub.status.idle":"2024-08-05T16:54:31.881017Z","shell.execute_reply.started":"2024-08-05T16:54:30.030207Z","shell.execute_reply":"2024-08-05T16:54:31.880071Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"OUTPUT_CHANNELS = 3\n\ndef downsample(filters, size, apply_instancenorm=True):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    result = keras.Sequential()\n    result.add(layers.Conv2D(filters, size, strides=2, padding='same',\n                             kernel_initializer=initializer, use_bias=False))\n\n    if apply_instancenorm:\n        # casey: replaced defunct tfa\n        result.add(layers.GroupNormalization(groups=-1, gamma_initializer=gamma_init))\n        #result.add(tfa.layers.InstanceNormalization(gamma_initializer=gamma_init))\n        \n    result.add(layers.LeakyReLU())\n\n    return result\n\ndef upsample(filters, size, apply_dropout=False):\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    result = keras.Sequential()\n    result.add(layers.Conv2DTranspose(filters, size, strides=2,\n                                      padding='same',\n                                      kernel_initializer=initializer,\n                                      use_bias=False))\n\n    # casey: replaced defunct tfa\n    result.add(layers.GroupNormalization(groups=-1, gamma_initializer=gamma_init))\n    #result.add(tfa.layers.InstanceNormalization(gamma_initializer=gamma_init))\n    if apply_dropout:\n        result.add(layers.Dropout(0.5))\n\n    result.add(layers.ReLU())\n\n    return result\n\ndef Generator():\n    inputs = layers.Input(shape=[256,256,3])\n\n    # bs = batch size\n    down_stack = [\n        downsample(64, 4, apply_instancenorm=False), # (bs, 128, 128, 64)\n        downsample(128, 4), # (bs, 64, 64, 128)\n        downsample(256, 4), # (bs, 32, 32, 256)\n        downsample(512, 4), # (bs, 16, 16, 512)\n        downsample(512, 4), # (bs, 8, 8, 512)\n        downsample(512, 4), # (bs, 4, 4, 512)\n        downsample(512, 4), # (bs, 2, 2, 512)\n        downsample(512, 4), # (bs, 1, 1, 512)\n    ]\n\n    up_stack = [\n        upsample(512, 4, apply_dropout=True), # (bs, 2, 2, 1024)\n        upsample(512, 4, apply_dropout=True), # (bs, 4, 4, 1024)\n        upsample(512, 4, apply_dropout=True), # (bs, 8, 8, 1024)\n        upsample(512, 4), # (bs, 16, 16, 1024)\n        upsample(256, 4), # (bs, 32, 32, 512)\n        upsample(128, 4), # (bs, 64, 64, 256)\n        upsample(64, 4), # (bs, 128, 128, 128)\n    ]\n\n    initializer = tf.random_normal_initializer(0., 0.02)\n    last = layers.Conv2DTranspose(OUTPUT_CHANNELS, 4,\n                                  strides=2,\n                                  padding='same',\n                                  kernel_initializer=initializer,\n                                  activation='tanh') # (bs, 256, 256, 3)\n\n    x = inputs\n\n    # Downsampling through the model\n    skips = []\n    for down in down_stack:\n        x = down(x)\n        skips.append(x)\n\n    skips = reversed(skips[:-1])\n\n    # Upsampling and establishing the skip connections\n    for up, skip in zip(up_stack, skips):\n        x = up(x)\n        x = layers.Concatenate()([x, skip])\n\n    x = last(x)\n\n    return keras.Model(inputs=inputs, outputs=x)\n\ndef Discriminator():\n    initializer = tf.random_normal_initializer(0., 0.02)\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n\n    inp = layers.Input(shape=[256, 256, 3], name='input_image')\n\n    x = inp\n\n    down1 = downsample(64, 4, False)(x) # (bs, 128, 128, 64)\n    down2 = downsample(128, 4)(down1) # (bs, 64, 64, 128)\n    down3 = downsample(256, 4)(down2) # (bs, 32, 32, 256)\n\n    zero_pad1 = layers.ZeroPadding2D()(down3) # (bs, 34, 34, 256)\n    conv = layers.Conv2D(512, 4, strides=1,\n                         kernel_initializer=initializer,\n                         use_bias=False)(zero_pad1) # (bs, 31, 31, 512)\n    \n    # casey: replace defunct tfa\n    #norm1 = tfa.layers.InstanceNormalization(gamma_initializer=gamma_init)(conv)\n    norm1 = layers.GroupNormalization(groups=-1, gamma_initializer=gamma_init)(conv)\n   \n    leaky_relu = layers.LeakyReLU()(norm1)\n\n    zero_pad2 = layers.ZeroPadding2D()(leaky_relu) # (bs, 33, 33, 512)\n\n    last = layers.Conv2D(1, 4, strides=1,\n                         kernel_initializer=initializer)(zero_pad2) # (bs, 30, 30, 1)\n\n    return tf.keras.Model(inputs=inp, outputs=last)\n\nwith strategy.scope():\n    monet_generator = Generator() # transforms photos to Monet-esque paintings\n    photo_generator = Generator() # transforms Monet paintings to be more like photos\n\n    monet_discriminator = Discriminator() # differentiates real Monet paintings and generated Monet paintings\n    photo_discriminator = Discriminator() # differentiates real photos and generated photos\n    to_monet = monet_generator(example_photo)\n\nplt.subplot(1, 2, 1)\nplt.title(\"Original Photo\")\nplt.imshow(example_photo[0] * 0.5 + 0.5)\n\nplt.subplot(1, 2, 2)\nplt.title(\"Monet-esque Photo\")\nplt.imshow(to_monet[0] * 0.5 + 0.5)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:54:31.883440Z","iopub.execute_input":"2024-08-05T16:54:31.883820Z","iopub.status.idle":"2024-08-05T16:54:34.551218Z","shell.execute_reply.started":"2024-08-05T16:54:31.883793Z","shell.execute_reply":"2024-08-05T16:54:34.550303Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class CycleGan(keras.Model):\n    def __init__(\n        self,\n        monet_generator,\n        photo_generator,\n        monet_discriminator,\n        photo_discriminator,\n        lambda_cycle=10,\n    ):\n        super(CycleGan, self).__init__()\n        self.m_gen = monet_generator\n        self.p_gen = photo_generator\n        self.m_disc = monet_discriminator\n        self.p_disc = photo_discriminator\n        self.lambda_cycle = lambda_cycle\n        \n    def compile(\n        self,\n        m_gen_optimizer,\n        p_gen_optimizer,\n        m_disc_optimizer,\n        p_disc_optimizer,\n        gen_loss_fn,\n        disc_loss_fn,\n        cycle_loss_fn,\n        identity_loss_fn\n    ):\n        super(CycleGan, self).compile()\n        self.m_gen_optimizer = m_gen_optimizer\n        self.p_gen_optimizer = p_gen_optimizer\n        self.m_disc_optimizer = m_disc_optimizer\n        self.p_disc_optimizer = p_disc_optimizer\n        self.gen_loss_fn = gen_loss_fn\n        self.disc_loss_fn = disc_loss_fn\n        self.cycle_loss_fn = cycle_loss_fn\n        self.identity_loss_fn = identity_loss_fn\n        \n    def train_step(self, batch_data):\n        real_monet, real_photo = batch_data\n        \n        with tf.GradientTape(persistent=True) as tape:\n            # photo to monet back to photo\n            fake_monet = self.m_gen(real_photo, training=True)\n            cycled_photo = self.p_gen(fake_monet, training=True)\n\n            # monet to photo back to monet\n            fake_photo = self.p_gen(real_monet, training=True)\n            cycled_monet = self.m_gen(fake_photo, training=True)\n\n            # generating itself\n            same_monet = self.m_gen(real_monet, training=True)\n            same_photo = self.p_gen(real_photo, training=True)\n\n            # discriminator used to check, inputing real images\n            disc_real_monet = self.m_disc(real_monet, training=True)\n            disc_real_photo = self.p_disc(real_photo, training=True)\n\n            # discriminator used to check, inputing fake images\n            disc_fake_monet = self.m_disc(fake_monet, training=True)\n            disc_fake_photo = self.p_disc(fake_photo, training=True)\n\n            # evaluates generator loss\n            monet_gen_loss = self.gen_loss_fn(disc_fake_monet)\n            photo_gen_loss = self.gen_loss_fn(disc_fake_photo)\n\n            # evaluates total cycle consistency loss\n            total_cycle_loss = self.cycle_loss_fn(real_monet, cycled_monet, self.lambda_cycle) + self.cycle_loss_fn(real_photo, cycled_photo, self.lambda_cycle)\n\n            # evaluates total generator loss\n            monet_identity_loss = self.identity_loss_fn(real_monet, same_monet, self.lambda_cycle)\n            photo_identity_loss = self.identity_loss_fn(real_photo, same_photo, self.lambda_cycle)\n            total_monet_gen_loss = monet_gen_loss + total_cycle_loss + monet_identity_loss\n            total_photo_gen_loss = photo_gen_loss + total_cycle_loss + photo_identity_loss\n\n            # evaluates discriminator loss\n            monet_disc_loss = self.disc_loss_fn(disc_real_monet, disc_fake_monet)\n            photo_disc_loss = self.disc_loss_fn(disc_real_photo, disc_fake_photo)\n\n        # Calculate the gradients for generator and discriminator\n        monet_generator_gradients = tape.gradient(total_monet_gen_loss,\n                                                  self.m_gen.trainable_variables)\n        photo_generator_gradients = tape.gradient(total_photo_gen_loss,\n                                                  self.p_gen.trainable_variables)\n\n        monet_discriminator_gradients = tape.gradient(monet_disc_loss,\n                                                      self.m_disc.trainable_variables)\n        photo_discriminator_gradients = tape.gradient(photo_disc_loss,\n                                                      self.p_disc.trainable_variables)\n\n        # Apply the gradients to the optimizer\n        self.m_gen_optimizer.apply_gradients(zip(monet_generator_gradients,\n                                                 self.m_gen.trainable_variables))\n\n        self.p_gen_optimizer.apply_gradients(zip(photo_generator_gradients,\n                                                 self.p_gen.trainable_variables))\n\n        self.m_disc_optimizer.apply_gradients(zip(monet_discriminator_gradients,\n                                                  self.m_disc.trainable_variables))\n\n        self.p_disc_optimizer.apply_gradients(zip(photo_discriminator_gradients,\n                                                  self.p_disc.trainable_variables))\n        \n        return {\n            \"monet_gen_loss\": total_monet_gen_loss,\n            \"photo_gen_loss\": total_photo_gen_loss,\n            \"monet_disc_loss\": monet_disc_loss,\n            \"photo_disc_loss\": photo_disc_loss,\n            \"cycle_loss\": total_cycle_loss,\n            \"monet_identity_loss\": monet_identity_loss,\n            \"photo_identity_loss\": photo_identity_loss\n        }\n\nwith strategy.scope():\n    def discriminator_loss(real, generated):\n        real_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(real), real)\n\n        generated_loss = tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.zeros_like(generated), generated)\n\n        total_disc_loss = real_loss + generated_loss\n\n        return total_disc_loss * 0.5\n\n    def generator_loss(generated):\n        return tf.keras.losses.BinaryCrossentropy(from_logits=True, reduction=tf.keras.losses.Reduction.NONE)(tf.ones_like(generated), generated)\n\n    def calc_cycle_loss(real_image, cycled_image, LAMBDA):\n        loss1 = tf.reduce_mean(tf.abs(real_image - cycled_image))\n\n        return LAMBDA * loss1\n    def identity_loss(real_image, same_image, LAMBDA):\n        loss = tf.reduce_mean(tf.abs(real_image - same_image))\n        return LAMBDA * 0.5 * loss\n\n    monet_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    photo_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n\n    monet_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    photo_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n\n    cycle_gan_model = CycleGan(\n        monet_generator, photo_generator, monet_discriminator, photo_discriminator\n    )\n\n    cycle_gan_model.compile(\n        m_gen_optimizer = monet_generator_optimizer,\n        p_gen_optimizer = photo_generator_optimizer,\n        m_disc_optimizer = monet_discriminator_optimizer,\n        p_disc_optimizer = photo_discriminator_optimizer,\n        gen_loss_fn = generator_loss,\n        disc_loss_fn = discriminator_loss,\n        cycle_loss_fn = calc_cycle_loss,\n        identity_loss_fn = identity_loss\n    )","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:54:34.552718Z","iopub.execute_input":"2024-08-05T16:54:34.552989Z","iopub.status.idle":"2024-08-05T16:54:34.598071Z","shell.execute_reply.started":"2024-08-05T16:54:34.552966Z","shell.execute_reply":"2024-08-05T16:54:34.597316Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_results(monet_generator):\n    _, ax = plt.subplots(5, 2, figsize=(12, 12))\n    for i, img in enumerate(photo_ds.take(5)):\n        prediction = monet_generator(img, training=False)[0].numpy()\n        prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n        img = (img[0] * 127.5 + 127.5).numpy().astype(np.uint8)\n\n        ax[i, 0].imshow(img)\n        ax[i, 1].imshow(prediction)\n        ax[i, 0].set_title(\"Input Photo\")\n        ax[i, 1].set_title(\"Monet-esque\")\n        ax[i, 0].axis(\"off\")\n        ax[i, 1].axis(\"off\")\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:54:34.599326Z","iopub.execute_input":"2024-08-05T16:54:34.599898Z","iopub.status.idle":"2024-08-05T16:54:34.607185Z","shell.execute_reply.started":"2024-08-05T16:54:34.599864Z","shell.execute_reply":"2024-08-05T16:54:34.606230Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class CustomCallback(keras.callbacks.Callback):\n    def on_epoch_end(self, epoch, logs=None):\n        plot_results(self.model.m_gen)\n\nif not USE_CACHED_MODELS:\n    hist1 = cycle_gan_model.fit(\n        tf.data.Dataset.zip((monet_ds, photo_ds)),\n        epochs=5,\n        callbacks=[CustomCallback()]\n    )\n    monet_generator.save(\"original_monet_generator.keras\")\n    monet_generator1 = monet_generator\nelse:\n    monet_generator1 = tf.keras.models.load_model(\"original_monet_generator.keras\")","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:54:34.608494Z","iopub.execute_input":"2024-08-05T16:54:34.608885Z","iopub.status.idle":"2024-08-05T16:54:35.743939Z","shell.execute_reply.started":"2024-08-05T16:54:34.608860Z","shell.execute_reply":"2024-08-05T16:54:35.742895Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_results(monet_generator1)","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:54:35.745278Z","iopub.execute_input":"2024-08-05T16:54:35.745581Z","iopub.status.idle":"2024-08-05T16:54:37.978104Z","shell.execute_reply.started":"2024-08-05T16:54:35.745556Z","shell.execute_reply":"2024-08-05T16:54:37.977181Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Original Work Starts Here\n\nhow can we make this introductory notebook better? It's very possible that with more generations of training, the weird artifacts would go away. But if you do what everybody else does, you'll probably get the results everybody else gets.\n\n## Exploratory Data Analysis\n","metadata":{}},{"cell_type":"code","source":"# the data is lazy loaded\nif not USE_CACHED_MODELS:\n    num_photos=0\n    for x in photo_ds:\n        num_photos += 1\n\n    print(\"num photos: \", num_photos)\n\n    num_monets=0\n    for x in monet_ds:\n        num_monets += 1\n\n    print(\"num monets: \", num_monets)","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:54:37.979480Z","iopub.execute_input":"2024-08-05T16:54:37.979839Z","iopub.status.idle":"2024-08-05T16:54:37.985536Z","shell.execute_reply.started":"2024-08-05T16:54:37.979810Z","shell.execute_reply":"2024-08-05T16:54:37.984587Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"there are 7038 photos and 300 monets total. They are all 256x256x3 dimensions.  there is more exploratory data analysis later, when looking at the generated images.","metadata":{}},{"cell_type":"markdown","source":"## A new discriminator\n\nMy first idea is to create a different Discriminator model. How about using more convolutional layers? In the cancer detection project, I used a triple convolutional layer design that worked pretty well, so I'm reusing it here (with minor modifications)","metadata":{}},{"cell_type":"code","source":"def getAltDiscriminator():\n    gamma_init = keras.initializers.RandomNormal(mean=0.0, stddev=0.02)\n    initializer = tf.random_normal_initializer(0., 0.02)\n    return keras.Sequential(\n        [\n            keras.Input(shape=(256,256,3)),\n            layers.ZeroPadding2D(),\n\n            layers.Conv2D(64, kernel_size=(3, 3), activation=\"leaky_relu\"),\n            layers.Conv2D(64, kernel_size=(3, 3), activation=\"leaky_relu\"),\n            layers.Conv2D(64, kernel_size=(3, 3), activation=\"leaky_relu\"),\n            layers.MaxPooling2D(pool_size=(2, 2)),\n            \n            ## the following is basically the same as the original discriminator\n            ## output needs to be 30x30x1\n            downsample(128,4),\n            downsample(256,4),\n            layers.ZeroPadding2D(),\n            layers.Conv2D(512, 4, strides=1,\n                         kernel_initializer=initializer,\n                         use_bias=False),\n            layers.GroupNormalization(groups=-1, gamma_initializer=gamma_init),\n            layers.LeakyReLU(),\n            layers.ZeroPadding2D(),\n\n            layers.Conv2D(1, 4, strides=1,\n                         kernel_initializer=initializer)\n\n        ]\n    )","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:54:37.989090Z","iopub.execute_input":"2024-08-05T16:54:37.989445Z","iopub.status.idle":"2024-08-05T16:54:37.999550Z","shell.execute_reply.started":"2024-08-05T16:54:37.989394Z","shell.execute_reply":"2024-08-05T16:54:37.998592Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"Discriminator().summary()","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:54:38.000596Z","iopub.execute_input":"2024-08-05T16:54:38.000854Z","iopub.status.idle":"2024-08-05T16:54:38.098966Z","shell.execute_reply.started":"2024-08-05T16:54:38.000832Z","shell.execute_reply":"2024-08-05T16:54:38.097762Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"getAltDiscriminator().summary()","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:54:38.100053Z","iopub.execute_input":"2024-08-05T16:54:38.100377Z","iopub.status.idle":"2024-08-05T16:54:38.205962Z","shell.execute_reply.started":"2024-08-05T16:54:38.100352Z","shell.execute_reply":"2024-08-05T16:54:38.205026Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n    monet_generator2 = Generator() # transforms photos to Monet-esque paintings\n    photo_generator2 = Generator() # transforms Monet paintings to be more like photos\n\n    monet_discriminator2 = getAltDiscriminator() # differentiates real Monet paintings and generated Monet paintings\n    photo_discriminator2 = getAltDiscriminator() # differentiates real photos and generated photos\n    to_monet = monet_generator2(example_photo)\n    \n    monet_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    photo_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n\n    monet_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    photo_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n\n    cycle_gan_model2 = CycleGan(\n        monet_generator2, photo_generator2, \n        monet_discriminator2, photo_discriminator2\n    )\n\n    cycle_gan_model2.compile(\n        m_gen_optimizer = monet_generator_optimizer,\n        p_gen_optimizer = photo_generator_optimizer,\n        m_disc_optimizer = monet_discriminator_optimizer,\n        p_disc_optimizer = photo_discriminator_optimizer,\n        gen_loss_fn = generator_loss,\n        disc_loss_fn = discriminator_loss,\n        cycle_loss_fn = calc_cycle_loss,\n        identity_loss_fn = identity_loss\n    )","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:54:38.207255Z","iopub.execute_input":"2024-08-05T16:54:38.207695Z","iopub.status.idle":"2024-08-05T16:54:38.972602Z","shell.execute_reply.started":"2024-08-05T16:54:38.207661Z","shell.execute_reply":"2024-08-05T16:54:38.971777Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"class CustomCallback(keras.callbacks.Callback):\n    def on_epoch_end(self, epoch, logs=None):\n        plot_results(self.model.m_gen)\n\nif not USE_CACHED_MODELS:\n    hist2 = cycle_gan_model2.fit(\n        tf.data.Dataset.zip((monet_ds, photo_ds)),\n        epochs=5,\n        callbacks=[CustomCallback()]\n    )\n    monet_generator2.save(\"monet_generator2.keras\")\nelse:\n    monet_generator2 = tf.keras.models.load_model(\"monet_generator2.keras\")","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:54:38.973915Z","iopub.execute_input":"2024-08-05T16:54:38.974318Z","iopub.status.idle":"2024-08-05T16:54:40.068983Z","shell.execute_reply.started":"2024-08-05T16:54:38.974283Z","shell.execute_reply":"2024-08-05T16:54:40.068183Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_results(monet_generator2)","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:54:40.070112Z","iopub.execute_input":"2024-08-05T16:54:40.070430Z","iopub.status.idle":"2024-08-05T16:54:42.111063Z","shell.execute_reply.started":"2024-08-05T16:54:40.070405Z","shell.execute_reply":"2024-08-05T16:54:42.110188Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"OK, not bad. We didn't totally ruin things with the new Discriminator. It might even be a little better.","metadata":{}},{"cell_type":"markdown","source":"## Can We Improve On UNet?\nThe structure of the Generator probably matters more than the Discriminator. The default CycleGAN uses a model called UNet for the Generator.\n\nI did some reading on improvements to UNet and came across this paper: https://www.nature.com/articles/s41523-021-00358-x\n\nThey created a model called a connected UNet, which is two UNets connected together (like a \"W\"). They had great success using it for cancer detection, so I figured I'd give it a shot for image generation.\n\nThe following code is heavily based on the paper's code found here: https://github.com/AsmaBaccouche/Connected-Unets-and-more/blob/main/Connected_Unets_segmnetation.py\n\nI had to change the last layers of the model to output a 256x256x3 image.","metadata":{}},{"cell_type":"code","source":"from keras.layers import Input, concatenate, Conv2D, Add, MaxPooling2D, Conv2DTranspose, BatchNormalization\nfrom keras.models import Model\n\ndef get_connected_unet():\n    def aspp_block(x, num_filters, rate_scale=1):\n        x1 = Conv2D(num_filters, (3, 3), dilation_rate=(6 * rate_scale, 6 * rate_scale), padding=\"same\")(x)\n        x1 = BatchNormalization()(x1)\n\n        x2 = Conv2D(num_filters, (3, 3), dilation_rate=(12 * rate_scale, 12 * rate_scale), padding=\"same\")(x)\n        x2 = BatchNormalization()(x2)\n\n        x3 = Conv2D(num_filters, (3, 3), dilation_rate=(18 * rate_scale, 18 * rate_scale), padding=\"same\")(x)\n        x3 = BatchNormalization()(x3)\n\n        x4 = Conv2D(num_filters, (3, 3), padding=\"same\")(x)\n        x4 = BatchNormalization()(x4)\n\n        y = Add()([x1, x2, x3, x4])\n        y = Conv2D(num_filters, (1, 1), padding=\"same\")(y)\n        return y \n\n    def get_wnet():\n        inputs = Input((256, 256, 3))\n        conv1 = Conv2D(32, (3, 3), activation='relu', padding='same')(inputs)\n        conv1 = BatchNormalization()(conv1)\n        conv1 = Conv2D(32, (3, 3), activation='relu', padding='same')(conv1)\n        conv1 = BatchNormalization()(conv1)\n        pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)\n\n        conv2 = Conv2D(64, (3, 3), activation='relu', padding='same')(pool1)\n        conv2 = BatchNormalization()(conv2)\n        conv2 = Conv2D(64, (3, 3), activation='relu', padding='same')(conv2)\n        conv2 = BatchNormalization()(conv2)\n        pool2 = MaxPooling2D(pool_size=(2, 2))(conv2)\n\n        conv3 = Conv2D(128, (3, 3), activation='relu', padding='same')(pool2)\n        conv3 = BatchNormalization()(conv3)\n        conv3 = Conv2D(128, (3, 3), activation='relu', padding='same')(conv3)\n        conv3 = BatchNormalization()(conv3)\n        pool3 = MaxPooling2D(pool_size=(2, 2))(conv3)\n\n        conv4 = Conv2D(256, (3, 3), activation='relu', padding='same')(pool3)\n        conv4 = BatchNormalization()(conv4)\n        conv4 = Conv2D(256, (3, 3), activation='relu', padding='same')(conv4)\n        conv4 = BatchNormalization()(conv4)\n        pool4 = MaxPooling2D(pool_size=(2, 2))(conv4)\n\n        conv5 = aspp_block(pool4, 512)\n\n        up6 = concatenate([Conv2DTranspose(256, (2, 2), strides=(2, 2), padding='same')(conv5), conv4], axis=3)\n        conv6 = Conv2D(256, (3, 3), activation='relu', padding='same')(up6)\n        conv6 = BatchNormalization()(conv6)\n        conv6 = Conv2D(256, (3, 3), activation='relu', padding='same')(conv6)\n        conv6 = BatchNormalization()(conv6)\n\n        up7 = concatenate([Conv2DTranspose(128, (2, 2), strides=(2, 2), padding='same')(conv6), conv3], axis=3)\n        conv7 = Conv2D(128, (3, 3), activation='relu', padding='same')(up7)\n        conv7 = BatchNormalization()(conv7)\n        conv7 = Conv2D(128, (3, 3), activation='relu', padding='same')(conv7)\n        conv7 = BatchNormalization()(conv7)\n\n        up8 = concatenate([Conv2DTranspose(64, (2, 2), strides=(2, 2), padding='same')(conv7), conv2], axis=3)\n        conv8 = Conv2D(64, (3, 3), activation='relu', padding='same')(up8)\n        conv8 = BatchNormalization()(conv8)\n        conv8 = Conv2D(64, (3, 3), activation='relu', padding='same')(conv8)\n        conv8 = BatchNormalization()(conv8)\n\n        up9 = concatenate([Conv2DTranspose(32, (2, 2), strides=(2, 2), padding='same')(conv8), conv1], axis=3)\n        conv9 = Conv2D(32, (3, 3), activation='relu', padding='same')(up9)\n        conv9 = BatchNormalization()(conv9)\n        conv9 = Conv2D(32, (3, 3), activation='relu', padding='same')(conv9)\n        conv9 = BatchNormalization()(conv9)\n\n        down10 = concatenate([Conv2D(32, (3, 3), activation='relu', padding='same')(conv9), conv9], axis=3)\n        conv10 = Conv2D(32, (3, 3), activation='relu', padding='same')(down10)\n        conv10 = BatchNormalization()(conv10)\n        conv10 = Conv2D(32, (3, 3), activation='relu', padding='same')(conv10)\n        conv10 = BatchNormalization()(conv10)    \n        pool10 = MaxPooling2D(pool_size=(2, 2))(conv10)\n\n        down11 = concatenate([Conv2D(64, (3, 3), activation='relu', padding='same')(pool10), conv8], axis=3)\n        conv11 = Conv2D(64, (3, 3), activation='relu', padding='same')(down11)\n        conv11 = BatchNormalization()(conv11)\n        conv11 = Conv2D(64, (3, 3), activation='relu', padding='same')(conv11)\n        conv11 = BatchNormalization()(conv11)   \n        pool11 = MaxPooling2D(pool_size=(2, 2))(conv11)\n\n        down12 = concatenate([Conv2D(128, (3, 3), activation='relu', padding='same')(pool11), conv7], axis=3)\n        conv12 = Conv2D(128, (3, 3), activation='relu', padding='same')(down12)\n        conv12 = BatchNormalization()(conv12)\n        conv12 = Conv2D(128, (3, 3), activation='relu', padding='same')(conv12)    \n        conv12 = BatchNormalization()(conv12)\n        pool12 = MaxPooling2D(pool_size=(2, 2))(conv12)\n\n        down13 = concatenate([Conv2D(256, (3, 3), activation='relu', padding='same')(pool12), conv6], axis=3)\n        conv13 = Conv2D(256, (3, 3), activation='relu', padding='same')(down13)\n        conv13 = BatchNormalization()(conv13)\n        conv13 = Conv2D(256, (3, 3), activation='relu', padding='same')(conv13)  \n        conv13 = BatchNormalization()(conv13)    \n        pool13 = MaxPooling2D(pool_size=(2, 2))(conv13)\n\n        conv14 = aspp_block(pool13, 512)\n\n        up15 = concatenate([Conv2DTranspose(256, (2, 2), strides=(2, 2), padding='same')(conv14), conv13], axis=3)\n        conv15 = Conv2D(256, (3, 3), activation='relu', padding='same')(up15)\n        conv15 = BatchNormalization()(conv15)    \n        conv15 = Conv2D(256, (3, 3), activation='relu', padding='same')(conv15)\n        conv15 = BatchNormalization()(conv15) \n\n        up16 = concatenate([Conv2DTranspose(128, (2, 2), strides=(2, 2), padding='same')(conv15), conv12], axis=3)\n        conv16 = Conv2D(128, (3, 3), activation='relu', padding='same')(up16)\n        conv16 = BatchNormalization()(conv16)     \n        conv16 = Conv2D(128, (3, 3), activation='relu', padding='same')(conv16)\n        conv16 = BatchNormalization()(conv16)      \n\n        up17 = concatenate([Conv2DTranspose(64, (2, 2), strides=(2, 2), padding='same')(conv16), conv11], axis=3)\n        conv17 = Conv2D(64, (3, 3), activation='relu', padding='same')(up17)\n        conv17 = BatchNormalization()(conv17)      \n        conv17 = Conv2D(64, (3, 3), activation='relu', padding='same')(conv17)\n        conv17 = BatchNormalization()(conv17)  \n\n        up18 = concatenate([Conv2DTranspose(32, (2, 2), strides=(2, 2), padding='same')(conv17), conv10], axis=3)\n        conv18 = Conv2D(32, (3, 3), activation='relu', padding='same')(up18)\n        conv18 = BatchNormalization()(conv18)      \n\n        conv18 = Conv2D(32, (3, 3), activation='relu', padding='same')(conv18)\n        conv18 = BatchNormalization()(conv18)    \n        \n        conv18 = aspp_block(conv18, 3)\n\n        # casey: commented out\n        #conv19 = Conv2D(1, (1, 1), activation='sigmoid')(conv18)\n\n        model = Model(inputs=[inputs], outputs=[conv18])\n        return model\n    return get_wnet()","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:54:42.112409Z","iopub.execute_input":"2024-08-05T16:54:42.112732Z","iopub.status.idle":"2024-08-05T16:54:42.159875Z","shell.execute_reply.started":"2024-08-05T16:54:42.112706Z","shell.execute_reply":"2024-08-05T16:54:42.158943Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Connected UNet Summary\n\nHere's a summary of the model we just built. It's enormous.","metadata":{}},{"cell_type":"code","source":"get_connected_unet().summary()","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:54:42.161096Z","iopub.execute_input":"2024-08-05T16:54:42.161455Z","iopub.status.idle":"2024-08-05T16:54:43.103625Z","shell.execute_reply.started":"2024-08-05T16:54:42.161428Z","shell.execute_reply":"2024-08-05T16:54:43.102537Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's see how a model with the new generator and the original discriminator performs.","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    monet_generator3 = get_connected_unet()# transforms photos to Monet-esque paintings\n    photo_generator3 = get_connected_unet() # transforms Monet paintings to be more like photos\n\n    monet_discriminator3 = Discriminator() # differentiates real Monet paintings and generated Monet paintings\n    photo_discriminator3 = Discriminator() # differentiates real photos and generated photos\n    \n    monet_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    photo_generator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n\n    monet_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n    photo_discriminator_optimizer = tf.keras.optimizers.Adam(2e-4, beta_1=0.5)\n\n    cycle_gan_model3 = CycleGan(\n        monet_generator3, photo_generator3, \n        monet_discriminator3, photo_discriminator3\n    )\n\n    cycle_gan_model3.compile(\n        m_gen_optimizer = monet_generator_optimizer,\n        p_gen_optimizer = photo_generator_optimizer,\n        m_disc_optimizer = monet_discriminator_optimizer,\n        p_disc_optimizer = photo_discriminator_optimizer,\n        gen_loss_fn = generator_loss,\n        disc_loss_fn = discriminator_loss,\n        cycle_loss_fn = calc_cycle_loss,\n        identity_loss_fn = identity_loss\n    )\n    \n# model_checkpoint_callback = keras.callbacks.ModelCheckpoint(\n#     filepath=\"/kaggle/working/checkpoint{epoch:02d}.model3.keras\",\n#     save_freq=600)\nclass CustomCallback(keras.callbacks.Callback):\n    def on_epoch_end(self, epoch, logs=None):\n        plot_results(self.model.m_gen)\n\nif not USE_CACHED_MODELS:\n    hist3 = cycle_gan_model3.fit(\n        tf.data.Dataset.zip((monet_ds, photo_ds)),\n        callbacks=[CustomCallback()],\n        epochs=5\n    )\n    ## casey: ensuring it won't clobber the good one here\n    monet_generator3.save(\"monet_generator3rightmodel.keras\")\nelse:\n    monet_generator3 = tf.keras.models.load_model(\"monet_generator3rightmodel.keras\")","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:54:43.105181Z","iopub.execute_input":"2024-08-05T16:54:43.105535Z","iopub.status.idle":"2024-08-05T16:54:46.275744Z","shell.execute_reply.started":"2024-08-05T16:54:43.105487Z","shell.execute_reply":"2024-08-05T16:54:46.274929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_results(monet_generator3)","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:54:46.276974Z","iopub.execute_input":"2024-08-05T16:54:46.277271Z","iopub.status.idle":"2024-08-05T16:54:49.381007Z","shell.execute_reply.started":"2024-08-05T16:54:46.277247Z","shell.execute_reply":"2024-08-05T16:54:49.380163Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"while there's some funny stuff going on, this new Generator has some promise.","metadata":{}},{"cell_type":"markdown","source":"## The Royale With Cheese\nHow about my generator plus my discriminator?","metadata":{}},{"cell_type":"code","source":"with strategy.scope():\n    monet_generator4 = get_connected_unet()# transforms photos to Monet-esque paintings\n    photo_generator4 = get_connected_unet() # transforms Monet paintings to be more like photos\n\n    monet_discriminator4 = getAltDiscriminator() # differentiates real Monet paintings and generated Monet paintings\n    photo_discriminator4 = getAltDiscriminator() # differentiates real photos and generated photos\n    to_monet = monet_generator4(example_photo)\n    \n    monet_generator_optimizer = tf.keras.optimizers.Adam()#2e-4, beta_1=0.5)\n    photo_generator_optimizer = tf.keras.optimizers.Adam()#2e-4, beta_1=0.5)\n\n    monet_discriminator_optimizer = tf.keras.optimizers.Adam()#2e-4, beta_1=0.5)\n    photo_discriminator_optimizer = tf.keras.optimizers.Adam()#2e-4, beta_1=0.5)\n\n    cycle_gan_model4 = CycleGan(\n        monet_generator4, photo_generator4, \n        monet_discriminator4, photo_discriminator4\n    )\n\n    cycle_gan_model4.compile(\n        m_gen_optimizer = monet_generator_optimizer,\n        p_gen_optimizer = photo_generator_optimizer,\n        m_disc_optimizer = monet_discriminator_optimizer,\n        p_disc_optimizer = photo_discriminator_optimizer,\n        gen_loss_fn = generator_loss,\n        disc_loss_fn = discriminator_loss,\n        cycle_loss_fn = calc_cycle_loss,\n        identity_loss_fn = identity_loss\n    )\n    \n# model_checkpoint_callback = keras.callbacks.ModelCheckpoint(\n#     filepath=\"/kaggle/working/checkpoint{epoch:02d}.model3.keras\",\n#     save_freq=600)\n    class CustomCallback(keras.callbacks.Callback):\n        def on_epoch_end(self, epoch, logs=None):\n            self.model.m_gen.save(f\"royale{epoch}.keras\")\n            plot_results(self.model.m_gen)\n    if not USE_CACHED_MODELS:\n        hist4 = cycle_gan_model4.fit(\n            tf.data.Dataset.zip((monet_ds, photo_ds)),\n            callbacks=[CustomCallback()],\n            epochs=25\n        )\n        monet_generator4.save(\"monet_generator4royale.keras\")\n    else:\n        # take the 5th generation so we're comparing head to head\n        monet_generator4 = tf.keras.models.load_model(\"royale5.keras\")","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:54:49.382050Z","iopub.execute_input":"2024-08-05T16:54:49.382367Z","iopub.status.idle":"2024-08-05T16:54:52.774619Z","shell.execute_reply.started":"2024-08-05T16:54:49.382342Z","shell.execute_reply":"2024-08-05T16:54:52.773666Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Head to head comparison of different models","metadata":{}},{"cell_type":"markdown","source":"here are all four models after 5 epochs of training.","metadata":{}},{"cell_type":"code","source":"def head_to_head(monet_models, num_photos=5, skip=0):\n    fig_height = round(2.5 * num_photos)\n    _, ax = plt.subplots(num_photos, len(monet_models)+1, figsize=(12, fig_height))\n    ret = []\n    ax[0, 0].set_title(\"Input Photo\")\n    for k, m in enumerate(monet_models):\n        ax[0, k+1].set_title(f\"model {k+1}\")\n    \n\n    for i, img in enumerate(photo_ds.skip(skip).take(num_photos)):\n            \n        img_to_show = (img[0] * 127.5 + 127.5).numpy().astype(np.uint8)\n        ax[i, 0].imshow(img_to_show)\n        ax[i,0].axis('off')\n        \n        for j, model in enumerate(monet_models):\n            prediction = model(img, training=False)[0].numpy()\n            prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n            ax[i,j+1].imshow(prediction)\n            ax[i,j+1].axis('off')\n            ret.append(prediction)\n    plt.show()\n    return ret\n    \n\nmodels = [monet_generator1, monet_generator2, monet_generator3, monet_generator4]\nignore = head_to_head(models,num_photos=10,skip=40)\n            ","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:54:52.775752Z","iopub.execute_input":"2024-08-05T16:54:52.778445Z","iopub.status.idle":"2024-08-05T16:55:01.103084Z","shell.execute_reply.started":"2024-08-05T16:54:52.778416Z","shell.execute_reply":"2024-08-05T16:55:01.100740Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## another generator function\n\nWell, all of the models have their moments. I would say model 2 (the base generator plus my new discriminator) is a little better than model 1 (base generator and discriminator). models 3 and 4 with the new generator are more interesting, but they have the tendency to go pathological and throw in a bunch of strange colors. here is model 4 over time. some epochs will be normal, some will not.","metadata":{}},{"cell_type":"code","source":"royales = [tf.keras.models.load_model(f\"royale{i}.keras\") for i in [2,4,8,16] ]","metadata":{"execution":{"iopub.status.busy":"2024-08-05T20:47:25.482520Z","iopub.execute_input":"2024-08-05T20:47:25.482952Z","iopub.status.idle":"2024-08-05T20:47:32.347206Z","shell.execute_reply.started":"2024-08-05T20:47:25.482922Z","shell.execute_reply":"2024-08-05T20:47:32.346191Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ignore=head_to_head(royales,skip=120, num_photos=10)","metadata":{"execution":{"iopub.status.busy":"2024-08-05T20:47:35.800133Z","iopub.execute_input":"2024-08-05T20:47:35.801043Z","iopub.status.idle":"2024-08-05T20:47:45.696765Z","shell.execute_reply.started":"2024-08-05T20:47:35.801007Z","shell.execute_reply":"2024-08-05T20:47:45.693293Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## EDA - Generated Photos\nA natural question to ask is: what are those wacky colors? What does that tell us about the model?","metadata":{}},{"cell_type":"code","source":"royale16 = tf.keras.models.load_model(f\"royale16.keras\")","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:55:15.889796Z","iopub.execute_input":"2024-08-05T16:55:15.890082Z","iopub.status.idle":"2024-08-05T16:55:16.998070Z","shell.execute_reply.started":"2024-08-05T16:55:15.890059Z","shell.execute_reply":"2024-08-05T16:55:16.997260Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"generated16 = head_to_head([royale16], skip=120, num_photos=10)","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:55:16.999233Z","iopub.execute_input":"2024-08-05T16:55:16.999532Z","iopub.status.idle":"2024-08-05T16:55:20.542771Z","shell.execute_reply.started":"2024-08-05T16:55:16.999506Z","shell.execute_reply":"2024-08-05T16:55:20.541098Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install colorgram.py","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:55:20.544240Z","iopub.execute_input":"2024-08-05T16:55:20.544578Z","iopub.status.idle":"2024-08-05T16:55:33.238920Z","shell.execute_reply.started":"2024-08-05T16:55:20.544550Z","shell.execute_reply":"2024-08-05T16:55:33.237562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# taken from \n# https://stackoverflow.com/questions/20548590/python-imaging-library-can-i-simply-fill-my-image-with-one-color#49825484\n# and https://pillow.readthedocs.io/en/latest/reference/ImageDraw.html\nfrom PIL import Image, ImageDraw\n\n\nout = Image.new(\"RGB\", (150, 100), (1, 255, 255))\nd = ImageDraw.Draw(out)\ndisplay(out)\n\n\nout = Image.new(\"RGB\", (150, 100), (255, 1, 255))\nd = ImageDraw.Draw(out)\ndisplay(out)\n\nout = Image.new(\"RGB\", (150, 100), (255, 255, 1))\nd = ImageDraw.Draw(out)\ndisplay(out)\n","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:55:33.240651Z","iopub.execute_input":"2024-08-05T16:55:33.241074Z","iopub.status.idle":"2024-08-05T16:55:33.258129Z","shell.execute_reply.started":"2024-08-05T16:55:33.241040Z","shell.execute_reply":"2024-08-05T16:55:33.257095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import PIL\nimport colorgram\nimage16 = PIL.Image.fromarray(generated16[1])\ncolor16 = colorgram.extract(image16, 10)\n\ndisplay(image16)\n\ndef draw_swath(r,g,b):\n    out = Image.new(\"RGB\", (150, 100), (r,g,b))\n    d = ImageDraw.Draw(out)\n    display(out)\n\nfor color in color16:\n    print(repr(color.rgb))\n    draw_swath(color.rgb.r, color.rgb.g, color.rgb.b)\n    \n","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:55:33.263886Z","iopub.execute_input":"2024-08-05T16:55:33.264269Z","iopub.status.idle":"2024-08-05T16:55:33.590020Z","shell.execute_reply.started":"2024-08-05T16:55:33.264244Z","shell.execute_reply":"2024-08-05T16:55:33.589081Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"compare to the original:","metadata":{}},{"cell_type":"code","source":"_, ax = plt.subplots(1)#num_photos, len(monet_models)+1, figsize=(12, fig_height))\n    \n\nfor i, img in enumerate(photo_ds.skip(121).take(1)):\n    img_to_show = (img[0] * 127.5 + 127.5).numpy().astype(np.uint8)\n    ax.imshow(img_to_show)\n    ax.axis('off')\nplt.show()\n\noriginal_colors = colorgram.extract(PIL.Image.fromarray(img_to_show), 10)\n    \nfor color in original_colors:\n    print(repr(color.rgb))\n    draw_swath(color.rgb.r, color.rgb.g, color.rgb.b)","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:55:33.591464Z","iopub.execute_input":"2024-08-05T16:55:33.591937Z","iopub.status.idle":"2024-08-05T16:55:34.534234Z","shell.execute_reply.started":"2024-08-05T16:55:33.591902Z","shell.execute_reply":"2024-08-05T16:55:34.533277Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I'll omit some of playing I did with this function. The color analysis is a little slow, so I can't really do it in bulk. and it would be more useful if it bucketed the colors by say rounding the rgb values to the nearest 10. But what I observe is the \"weird\" colors in the generated images are all ones where one channel has a really high value and the other two have low values.","metadata":{}},{"cell_type":"markdown","source":"## A Final Generator Model Tweak\nSo, I think the extreme colors might be because it's not able to learn more moderate color values. So I switched the activations on the last batch of layers to leaky_relu and added a BatchNormalization at the end. After some experimentation with learning rates, I also switched to the RMSProp algorithm for the generators.\n\nMy changes were a bit arbitrary, but they did improve things.","metadata":{}},{"cell_type":"code","source":"from keras.layers import Input, concatenate, Conv2D, Add, MaxPooling2D, Conv2DTranspose, BatchNormalization, GroupNormalization\nfrom keras.models import Model\n\ndef get_connected_unet2():\n    def aspp_block(x, num_filters, rate_scale=1):\n        x1 = Conv2D(num_filters, (3, 3), dilation_rate=(6 * rate_scale, 6 * rate_scale), padding=\"same\")(x)\n        x1 = BatchNormalization()(x1)\n\n        x2 = Conv2D(num_filters, (3, 3), dilation_rate=(12 * rate_scale, 12 * rate_scale), padding=\"same\")(x)\n        x2 = BatchNormalization()(x2)\n\n        x3 = Conv2D(num_filters, (3, 3), dilation_rate=(18 * rate_scale, 18 * rate_scale), padding=\"same\")(x)\n        x3 = BatchNormalization()(x3)\n\n        x4 = Conv2D(num_filters, (3, 3), padding=\"same\")(x)\n        x4 = BatchNormalization()(x4)\n\n        y = Add()([x1, x2, x3, x4])\n        y = Conv2D(num_filters, (1, 1), padding=\"same\")(y)\n        return y \n\n    def get_wnet():\n        inputs = Input((256, 256, 3))\n        conv1 = Conv2D(32, (3, 3), activation='relu', padding='same')(inputs)\n        conv1 = BatchNormalization()(conv1)\n        conv1 = Conv2D(32, (3, 3), activation='relu', padding='same')(conv1)\n        conv1 = BatchNormalization()(conv1)\n        pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)\n\n        conv2 = Conv2D(64, (3, 3), activation='relu', padding='same')(pool1)\n        conv2 = BatchNormalization()(conv2)\n        conv2 = Conv2D(64, (3, 3), activation='relu', padding='same')(conv2)\n        conv2 = BatchNormalization()(conv2)\n        pool2 = MaxPooling2D(pool_size=(2, 2))(conv2)\n\n        conv3 = Conv2D(128, (3, 3), activation='relu', padding='same')(pool2)\n        conv3 = BatchNormalization()(conv3)\n        conv3 = Conv2D(128, (3, 3), activation='relu', padding='same')(conv3)\n        conv3 = BatchNormalization()(conv3)\n        pool3 = MaxPooling2D(pool_size=(2, 2))(conv3)\n\n        conv4 = Conv2D(256, (3, 3), activation='relu', padding='same')(pool3)\n        conv4 = BatchNormalization()(conv4)\n        conv4 = Conv2D(256, (3, 3), activation='relu', padding='same')(conv4)\n        conv4 = BatchNormalization()(conv4)\n        pool4 = MaxPooling2D(pool_size=(2, 2))(conv4)\n\n        conv5 = aspp_block(pool4, 512)\n\n        up6 = concatenate([Conv2DTranspose(256, (2, 2), strides=(2, 2), padding='same')(conv5), conv4], axis=3)\n        conv6 = Conv2D(256, (3, 3), activation='relu', padding='same')(up6)\n        conv6 = BatchNormalization()(conv6)\n        conv6 = Conv2D(256, (3, 3), activation='relu', padding='same')(conv6)\n        conv6 = BatchNormalization()(conv6)\n\n        up7 = concatenate([Conv2DTranspose(128, (2, 2), strides=(2, 2), padding='same')(conv6), conv3], axis=3)\n        conv7 = Conv2D(128, (3, 3), activation='relu', padding='same')(up7)\n        conv7 = BatchNormalization()(conv7)\n        conv7 = Conv2D(128, (3, 3), activation='relu', padding='same')(conv7)\n        conv7 = BatchNormalization()(conv7)\n\n        up8 = concatenate([Conv2DTranspose(64, (2, 2), strides=(2, 2), padding='same')(conv7), conv2], axis=3)\n        conv8 = Conv2D(64, (3, 3), activation='relu', padding='same')(up8)\n        conv8 = BatchNormalization()(conv8)\n        conv8 = Conv2D(64, (3, 3), activation='relu', padding='same')(conv8)\n        conv8 = BatchNormalization()(conv8)\n\n        up9 = concatenate([Conv2DTranspose(32, (2, 2), strides=(2, 2), padding='same')(conv8), conv1], axis=3)\n        conv9 = Conv2D(32, (3, 3), activation='relu', padding='same')(up9)\n        conv9 = BatchNormalization()(conv9)\n        conv9 = Conv2D(32, (3, 3), activation='relu', padding='same')(conv9)\n        conv9 = BatchNormalization()(conv9)\n\n        down10 = concatenate([Conv2D(32, (3, 3), activation='relu', padding='same')(conv9), conv9], axis=3)\n        conv10 = Conv2D(32, (3, 3), activation='relu', padding='same')(down10)\n        conv10 = BatchNormalization()(conv10)\n        conv10 = Conv2D(32, (3, 3), activation='relu', padding='same')(conv10)\n        conv10 = BatchNormalization()(conv10)    \n        pool10 = MaxPooling2D(pool_size=(2, 2))(conv10)\n\n        down11 = concatenate([Conv2D(64, (3, 3), activation='relu', padding='same')(pool10), conv8], axis=3)\n        conv11 = Conv2D(64, (3, 3), activation='relu', padding='same')(down11)\n        conv11 = BatchNormalization()(conv11)\n        conv11 = Conv2D(64, (3, 3), activation='relu', padding='same')(conv11)\n        conv11 = BatchNormalization()(conv11)   \n        pool11 = MaxPooling2D(pool_size=(2, 2))(conv11)\n\n        down12 = concatenate([Conv2D(128, (3, 3), activation='relu', padding='same')(pool11), conv7], axis=3)\n        conv12 = Conv2D(128, (3, 3), activation='relu', padding='same')(down12)\n        conv12 = BatchNormalization()(conv12)\n        conv12 = Conv2D(128, (3, 3), activation='relu', padding='same')(conv12)    \n        conv12 = BatchNormalization()(conv12)\n        pool12 = MaxPooling2D(pool_size=(2, 2))(conv12)\n\n        down13 = concatenate([Conv2D(256, (3, 3), activation='relu', padding='same')(pool12), conv6], axis=3)\n        conv13 = Conv2D(256, (3, 3), activation='relu', padding='same')(down13)\n        conv13 = BatchNormalization()(conv13)\n        conv13 = Conv2D(256, (3, 3), activation='relu', padding='same')(conv13)  \n        conv13 = BatchNormalization()(conv13)    \n        pool13 = MaxPooling2D(pool_size=(2, 2))(conv13)\n\n        conv14 = aspp_block(pool13, 512)\n\n        up15 = concatenate([Conv2DTranspose(256, (2, 2), strides=(2, 2), padding='same')(conv14), conv13], axis=3)\n        conv15 = Conv2D(256, (3, 3), activation='relu', padding='same')(up15)\n        conv15 = BatchNormalization()(conv15)    \n        conv15 = Conv2D(256, (3, 3), activation='relu', padding='same')(conv15)\n        conv15 = BatchNormalization()(conv15)\n        \n        #casey: switched these to leaky relu, I think the relus are forcing the more \n        #extreme primary color patches.\n        up16 = concatenate([Conv2DTranspose(128, (2, 2), strides=(2, 2), padding='same')(conv15), conv12], axis=3)\n        conv16 = Conv2D(128, (3, 3), activation='leaky_relu', padding='same')(up16)\n        conv16 = BatchNormalization()(conv16)     \n        conv16 = Conv2D(128, (3, 3), activation='leaky_relu', padding='same')(conv16)\n        conv16 = BatchNormalization()(conv16)      \n\n        up17 = concatenate([Conv2DTranspose(64, (2, 2), strides=(2, 2), padding='same')(conv16), conv11], axis=3)\n        conv17 = Conv2D(64, (3, 3), activation='leaky_relu', padding='same')(up17)\n        conv17 = BatchNormalization()(conv17)      \n        conv17 = Conv2D(64, (3, 3), activation='leaky_relu', padding='same')(conv17)\n        conv17 = BatchNormalization()(conv17)  \n\n\n        up18 = concatenate([Conv2DTranspose(32, (2, 2), strides=(2, 2), padding='same')(conv17), conv10], axis=3)\n        conv18 = Conv2D(32, (3, 3), activation='leaky_relu', padding='same')(up18)\n        conv18 = BatchNormalization()(conv18)      \n        conv18 = Conv2D(32, (3, 3), activation='leaky_relu', padding='same')(conv18)\n        conv18 = BatchNormalization()(conv18)\n        \n        conv18 = aspp_block(conv18, 3)\n        conv19 = BatchNormalization()(conv18)\n\n        model = Model(inputs=[inputs], outputs=[conv19])\n        return model\n    return get_wnet()","metadata":{"execution":{"iopub.status.busy":"2024-08-05T16:55:34.535777Z","iopub.execute_input":"2024-08-05T16:55:34.536069Z","iopub.status.idle":"2024-08-05T16:55:34.576182Z","shell.execute_reply.started":"2024-08-05T16:55:34.536043Z","shell.execute_reply":"2024-08-05T16:55:34.575423Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import json\n\nwith strategy.scope():\n    monet_generator5 = get_connected_unet2()# transforms photos to Monet-esque paintings\n    photo_generator5 = get_connected_unet2() # transforms Monet paintings to be more like photos\n\n    monet_discriminator5 = getAltDiscriminator() # differentiates real Monet paintings and generated Monet paintings\n    photo_discriminator5 = getAltDiscriminator() # differentiates real photos and generated photos\n    \n    monet_generator_optimizer = tf.keras.optimizers.RMSprop()#Adam()#2e-4, beta_1=0.5)\n    photo_generator_optimizer = tf.keras.optimizers.RMSprop()#.Adam()#2e-4, beta_1=0.5)\n\n    monet_discriminator_optimizer = tf.keras.optimizers.Adam()#2e-4, beta_1=0.5)\n    photo_discriminator_optimizer = tf.keras.optimizers.Adam()#2e-4, beta_1=0.5)\n\n    cycle_gan_model5 = CycleGan(\n        monet_generator5, photo_generator5, \n        monet_discriminator5, photo_discriminator5\n    )\n\n    cycle_gan_model5.compile(\n        m_gen_optimizer = monet_generator_optimizer,\n        p_gen_optimizer = photo_generator_optimizer,\n        m_disc_optimizer = monet_discriminator_optimizer,\n        p_disc_optimizer = photo_discriminator_optimizer,\n        gen_loss_fn = generator_loss,\n        disc_loss_fn = discriminator_loss,\n        cycle_loss_fn = calc_cycle_loss,\n        identity_loss_fn = identity_loss\n    )\n    \n\n    class CustomCallback(keras.callbacks.Callback):\n        def on_epoch_end(self, epoch, logs=None):\n            self.model.m_gen.save(f\"model5.{epoch}.keras\")\n            #plot_results(self.model.m_gen)\n            \nif True:#fixme\n    hist5 = cycle_gan_model5.fit(\n        tf.data.Dataset.zip((monet_ds, photo_ds)),\n        callbacks=[CustomCallback()],\n        epochs=25\n    )\n    monet_generator5.save(\"monet_generator5b.keras\")\n\nelse:\n    monet_generator5 = tf.keras.models.load_model(\"monet_generator5.keras\")","metadata":{"execution":{"iopub.status.busy":"2024-08-05T19:24:34.450876Z","iopub.execute_input":"2024-08-05T19:24:34.451389Z","iopub.status.idle":"2024-08-05T20:29:11.729396Z","shell.execute_reply.started":"2024-08-05T19:24:34.451356Z","shell.execute_reply":"2024-08-05T20:29:11.728246Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"mogens = [tf.keras.models.load_model(f\"model5.{epoch}.keras\") for epoch in [1,6,12,18,24]]\n","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Visualizing model training progress","metadata":{}},{"cell_type":"code","source":"ignore = head_to_head(mogens,skip=229, num_photos=20)","metadata":{"execution":{"iopub.status.busy":"2024-08-05T20:40:58.598742Z","iopub.execute_input":"2024-08-05T20:40:58.599151Z","iopub.status.idle":"2024-08-05T20:41:21.436060Z","shell.execute_reply.started":"2024-08-05T20:40:58.599099Z","shell.execute_reply":"2024-08-05T20:41:21.434936Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Above are the original images, then the 1st, 6th, 12th, 18th and 24th generations.\n\nAlthough it takes quite a few epochs to get away from the big splotches of bright color, I think the end result is fairly \"painterly\". It looks more like something a human did rather than a hallucinating robot.\n\n### You wanted a graph...\nA big problem is the loss numbers don't really track with how good the images are. Epoch 24 is way better than epoch 12. But you can't tell that from the numbers.\n\n```Epoch 12/25\n300/300 ━━━━━━━━━━━━━━━━━━━━ 154s 494ms/step - cycle_loss: 5.2868 - monet_disc_loss: 0.4312 - monet_gen_loss: 7.9524 - monet_identity_loss: 1.1905 - photo_disc_loss: 0.4790 - photo_gen_loss: 7.9917 - photo_identity_loss: 1.4061\n```\n\n```Epoch 24/25\n300/300 ━━━━━━━━━━━━━━━━━━━━ 154s 493ms/step - cycle_loss: 5.2948 - monet_disc_loss: 0.3971 - monet_gen_loss: 8.1387 - monet_identity_loss: 1.2201 - photo_disc_loss: 0.5448 - photo_gen_loss: 7.9938 - photo_identity_loss: 1.5263\n```\n\nThe cycle_loss, monet_gen_loss and photo_gen_loss are all worse in epoch 24 than epoch 12 even though the output is (subjectively) better. Unlike other deep learning problems we've done in this class, it's not obvious when to stop training. Another 24 epochs and it might be perfect, or it might be useless.","metadata":{}},{"cell_type":"markdown","source":"I ended up training the final model 4 times. 3 times it was looking very good at the end; one time it was still putting blotches of hot pink and red in images. So there is an element of luck to the training.","metadata":{}},{"cell_type":"code","source":"\n\ngen_loss_history = np.mean(hist5.history['monet_gen_loss'],axis=3)\n\nmonet_gen_history = []\nfor x in hist5.history['monet_gen_loss']:\n    monet_gen_history.append(np.mean(x))\n    \nmonet_id_history = []\nfor x in hist5.history['monet_identity_loss']:\n    monet_id_history.append(np.mean(x))\n\nplt.plot(monet_gen_history, label=\"monet gen loss\")\nplt.plot(monet_id_history, label=\"monet ID loss\")\nplt.plot(hist5.history['cycle_loss'], label=\"cycle loss\")\nplt.legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2024-08-05T20:41:59.211621Z","iopub.execute_input":"2024-08-05T20:41:59.212354Z","iopub.status.idle":"2024-08-05T20:41:59.469138Z","shell.execute_reply.started":"2024-08-05T20:41:59.212324Z","shell.execute_reply":"2024-08-05T20:41:59.468159Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The ID and cycle loss numbers being stable makes sense because they're meant to keep the system from going off the rails, not assess how good the transformation is. But the generator loss staying flat is somewhat surprising to me.","metadata":{}},{"cell_type":"markdown","source":"## Data Augmentation to the Rescue?\nI made another notebook for the final submission: https://www.kaggle.com/code/caseydurfee/monet-contest-submission\n\nAfter submitting and getting a respectable score (69.5), I tried using Diff Augment to make the training data more robust, as detailed in this notebook: https://www.kaggle.com/code/unfriendlyai/diffaugment-is-all-you-need\nWith the augmented data, I got a score of 80, which is worse. It's not clear to me why augmented data led to a worse model, though there is an element of chance to the training. another run with diffaugment might get a better result.","metadata":{}},{"cell_type":"markdown","source":"## Conclusions and Next Steps\nWe still have a learning rate issue here, I think. I experimented with 3 different optimizers: Adam with the starting learning rate from the intro notebook, Adam with default settings, and RMSProp. None of them seemed ideal.\n\nI think there's probably a better solution using adaptive learning rates so it doesn't keep cycling between OK and bad models in the later epochs of training. If the model isn't repeatable, I'm not sure we've solved the problem in a truly unsupervised way.\n\nI think better tools for visualizing what it's doing while training would help. I'm still not clear on where the garish colors are coming from. \n\nIt may be possible to initialize the model with more monet-like colors. Perhaps this could be done by having another GAN that learns a mapping of colors in the real photos to colors in the monet paintings. The generator of the 2nd GAN would take an image and generate a plausible palette of colors to use in the transformation. For each photo, generate the more monet like versions of those colors. And for each monet, generate the more photo-like versions of those colors. The learned mapping would then be fed into generator of the first GAN. There would be an additional loss function that would penalize drawing colors that are outside the monet-like spectrum.\n\nI think the combination of Generator and Discriminator architectures matters. With a bunch more time, I could've tested more combinations of architectures. Perhaps there's a perfect combo out there that's better than the sum of its parts, like John Lennon and Paul McCartney.","metadata":{}},{"cell_type":"markdown","source":"# Create submission file","metadata":{}},{"cell_type":"markdown","source":"the actual submission is in this notebook: https://www.kaggle.com/code/caseydurfee/monet-contest-submission","metadata":{}},{"cell_type":"code","source":"import zipfile","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import os\nos.getcwd()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import PIL\nimport io\nfrom tqdm import tqdm\nSELECTED_MONET_GENERATOR = monet_generator5\nif False:\n    with zipfile.ZipFile('/kaggle/working/images.zip', 'w') as zipped_f:\n        for i, img in tqdm(enumerate(photo_ds), total=7100):\n            prediction = SELECTED_MONET_GENERATOR(img, training=False)[0].numpy()\n            prediction = (prediction * 127.5 + 127.5).astype(np.uint8)\n            im = PIL.Image.fromarray(prediction)\n\n            file_object = io.BytesIO()\n            im.save(file_object, \"JPEG\")\n            im.close()\n\n            zipped_f.writestr(str(i) + \".jpg\", file_object.getvalue())\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Sources\n* https://stackoverflow.com/questions/40886234/how-to-directly-add-file-to-zip-in-python\n* https://github.com/eriklindernoren/Keras-GAN\n* [Zhu 2017] https://arxiv.org/abs/1703.10593\nhttps://www.kaggle.com/code/unfriendlyai/diffaugment-is-all-you-need\n* [Baccouche 2021] https://www.nature.com/articles/s41523-021-00358-x\n* https://www.kaggle.com/code/amyjang/monet-cyclegan-tutorial","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}