\n
Part 4.3: Model Tuning 4 - Exploring Learning Rate Scheduling
","metadata":{}},{"cell_type":"markdown","source":"\nOur previous tuning model does seem to improve performance. The Monet-style images look more like Monet paintings, with a bit more Monet-style textures and styles. This is a good sign that the model generalizes better. Clearly, we haven't trained the model long enough. Given the current memory resources, the next model would be our final model.\n- For our final model, we will try our best and train the model as long as necessary.\n\n- With more epochs, in later steps, the 'optimal' learning rate that works well for smaller epochs won't work well for later steps where the model is about to find the ultimate points. It will overshoot. We may either lower the learning rate as a whole or use learning rate scheduling so we have a smaller learning rate for later steps.\n\n- The other thing we will do is to increase the batch size. In our previous models, due to limited memory resources, we haven't done that. From the Monet-style images we've created so far, they don't look too Monet to me. I think increasing the batch size will make the model generalize more to create more Monet-alike images.\n\n","metadata":{}},{"cell_type":"code","source":"# Model 5\nEPOCHS = 20\nBATCH_SIZE=9\nLEARNING_RATE = 3e-4\nLEARNING_RATE_MIN = 1e-5\nNUM_MONET = 300*2 #we got this from part 1 of this notebook. Doulbe it since we will repeat selection. \nNUM_PHOTO = 7038\nDECAY_STEPS = int( NUM_PHOTO / BATCH_SIZE)*5\nDECAY_RATE=0.6\n\nBUFFER_SIZE=50 # controls how oftern to update discriminator. Still working on the coding :)\nBETA_1 = 0.5\nLAMBDA = 3\n\nDROPOUT_RATE_GEN = 0.5\nTRANSFER_LAYERS_GEN = 5\nSTRIDE_NUM_GEN, STRIDE_NUM_DIS = 2,1\nPADDING_GEN ='same'\nFITLER_SIZE_GEN, FITLER_SIZE_DIS = 4, 4\nFILTER_SIZE_E1_GEN =4\nFILTER_CHANEL_1ST_GEN,FILTER_CHANEL_1ST_DIS = 64, 64\nDOWN_LAYER_NUM_DIS=3","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define the learning rate scheduler\nclass CustomSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):\n def __init__(self, initial_learning_rate, learning_rate_min, decay_steps, decay_rate):\n super(CustomSchedule, self).__init__()\n self.initial_learning_rate = initial_learning_rate \n self.min_learning_rate = learning_rate_min\n self.decay_steps = decay_steps\n self.decay_rate = decay_rate\n \n\n def __call__(self, step):\n learning_rate = tf.math.maximum(self.initial_learning_rate * self.decay_rate ** (step / self.decay_steps), self.min_learning_rate)\n return learning_rate\n \n# learning rate scheduler instance\nlearning_rate_scheduler = CustomSchedule(initial_learning_rate=LEARNING_RATE, learning_rate_min=LEARNING_RATE_MIN ,decay_steps=DECAY_STEPS, decay_rate=DECAY_RATE)\n\n# plot\n# Number of steps in training\nnum_steps = int(NUM_PHOTO /BATCH_SIZE * EPOCHS )\n\n# Generate learning rates for each step\nlearning_rates = [learning_rate_scheduler(tf.constant(step, dtype=tf.float32)).numpy() for step in range(num_steps)]\n\n# Plot learning rates\nplt.figure(figsize=(10,5))\nplt.plot(range(num_steps), learning_rates, color='#ebb73d')\nplt.xlabel('Step')\nplt.ylabel('Learning Rate')\nplt.title('Learning Rate Schedule')\nplt.grid(True)\nplt.show()\n","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Monet\nmonet_data_augm = load_dataset(MONET_FILENAMES, labeled=True, ordered=False,repeat=True, shuffle=True,rotation=True, flip=True, transpose=True, crop=False, height=256,width=256).batch(batch_size=BATCH_SIZE)\n# Photo\n# photo dataset size of 7038 won't be enoughfor steps per epoch* number of epochs. repeat() on photo dataset too to let date generate indefinitely\nphoto_data_augm = load_dataset(PHOTO_FILENAMES, labeled=True, ordered=False,repeat=False, shuffle=True,rotation=True, flip=True, transpose=True, crop=False, height=256,width=256).batch(batch_size=BATCH_SIZE)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Model 5\nwith strategy.scope():\n monet_generator5 = Generator(dropout_rate=DROPOUT_RATE_GEN,filter_size_E1=FILTER_SIZE_E1_GEN, filter_size=FITLER_SIZE_GEN, filter_chanel_1st =FILTER_CHANEL_1ST_GEN,transfer_layers=TRANSFER_LAYERS_GEN, stride_num=STRIDE_NUM_GEN, padding_style=PADDING_GEN)\n photo_generator5= Generator(dropout_rate=DROPOUT_RATE_GEN, filter_size_E1=FILTER_SIZE_E1_GEN,filter_size=FITLER_SIZE_GEN, filter_chanel_1st =FILTER_CHANEL_1ST_GEN,transfer_layers=TRANSFER_LAYERS_GEN, stride_num=STRIDE_NUM_GEN, padding_style=PADDING_GEN)\n monet_discriminator5 = Discriminator(filter_size=FITLER_SIZE_DIS , filter_chanel_1st=FILTER_CHANEL_1ST_DIS, down_layer_num=DOWN_LAYER_NUM_DIS, strides=STRIDE_NUM_DIS)\n photo_discriminator5= Discriminator(filter_size=FITLER_SIZE_DIS , filter_chanel_1st=FILTER_CHANEL_1ST_DIS, down_layer_num=DOWN_LAYER_NUM_DIS,strides=STRIDE_NUM_DIS)\n \n monet_generator_optimizer5 = tf.keras.optimizers.Adam(learning_rate=learning_rate_scheduler, beta_1 =BETA_1)\n photo_generator_optimizer5 = tf.keras.optimizers.Adam(learning_rate=learning_rate_scheduler,beta_1 =BETA_1)\n monet_discriminator_optimizer5 = tf.keras.optimizers.Adam(learning_rate=learning_rate_scheduler,beta_1 =BETA_1) \n photo_discriminator_optimizer5 = tf.keras.optimizers.Adam(learning_rate=learning_rate_scheduler,beta_1 =BETA_1)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"with strategy.scope():\n cycle_gan_model5 = CycleGan(\n monet_generator5, photo_generator5, monet_discriminator5, photo_discriminator5, lambda_cycle=LAMBDA , buffer_size= BUFFER_SIZE\n )\n\n cycle_gan_model5.compile(\n m_gen_optimizer = monet_generator_optimizer5,\n p_gen_optimizer = photo_generator_optimizer5,\n m_disc_optimizer = monet_discriminator_optimizer5,\n p_disc_optimizer = photo_discriminator_optimizer5,\n gen_loss_fn = generator_loss,\n disc_loss_fn = discriminator_loss,\n cycle_loss_fn = calc_cycle_loss,\n identity_loss_fn = identity_loss\n )","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Train model 5\nhist_model5= cycle_gan_model5.fit(\n tf.data.Dataset.zip((monet_data_augm ,photo_data_augm)),\n batch_size=BATCH_SIZE,\n epochs= EPOCHS)\n# steps_per_epoch=min(NUM_MONET, NUM_PHOTO) //BATCH_SIZE ) \n","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_losses(hist_model5.history)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"visualize_predictions(photo_data, monet_generator5,photo_generator5, num_images=20)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Observation of model performance:\nGladly, the images look significantly better in our very last model, and the losses are much lower compared to all previous models. For example, the monet generator loss went from 1.49 to 1.39. I think the generated Monet-style images finally look like Monet paintings.😸 Despite not having additional memory for further tuning, here's what I learned through this model tuning:\n\n- Learning rate scheduling significantly improves CycleGAN performance.\n - During the training with more epochs, I experimented with using an initial learning rate of 2e-4 for the entire training process. However, I observed that the loss oscillated with larger fluctuations after some steps, indicating that the learning rate was too high.\n - Conversely, decreasing the learning rate overall would result in excessively slow initial training. It appears that CycleGAN tends to get stuck at saddle points during the initial training iterations if a small learning rate is used.\n- The most critical challenge encountered during the last training session was balancing the generator and the discriminator.\n - The discriminator seemed to quickly plateau or even degrade in performance. This was evident from the generator loss continuing to decrease while the discriminator loss decreased slowly, ceased decreasing altogether, or even began increasing early on in the training process.\n - I attempted to address this issue by adding noise to the discriminator input, but it didn't yield significant improvements. This imbalance likely affects the performance of the generators as well, as when the discriminator stops improving, it fails to provide meaningful feedback to the generator for improvement.\n\n\n","metadata":{}},{"cell_type":"markdown","source":"#
\n
\n
Part 5: Submit Results 🪅
\n\n","metadata":{}},{"cell_type":"code","source":"#submit results\n# Create new directory\n! mkdir ../images\n\n# Generate monet-style photos\ni = 1\nfor photo in photo_data:\n with strategy.scope():\n photo_to_monet= monet_generator5(photo, training=False)[0].numpy()\n photo_to_monet = (photo_to_monet * 127.5 + 127.5).astype(np.uint8)\n im = PIL.Image.fromarray(photo_to_monet)\n im.save(\"../images/\" + str(i) + \".jpg\")\n i += 1\n# save\nshutil.make_archive(\"/kaggle/working/images\", 'zip', \"/kaggle/images\")","metadata":{"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#
\n
\n
Part 6: Conclusions 👒
\n\n","metadata":{}},{"cell_type":"markdown","source":"
\n
\n \n### Modeling Summary:\nIn this notebook, the model training process encountered limitations due to memory constraints and training time restrictions. Consequently, the initial tuning iterations did not utilize the most optimal epoch numbers and batch sizes. The primary objective of these iterations was to gradually identify key modifications that could enhance CycleGAN training:\n- For the baseline model, a slightly simpler architecture was employed, adhering to the necessary CycleGAN structure. This included typical generator and discriminator structures comprising downsampling blocks for encoding and upsampling blocks for decoding, along with implementation of good practices such as utilizing stride instead of max-pooling, padding, instance normalization (as opposed to standard normalization), LeakyReLU (instead of ReLU), and incorporation of skip connections. The generated images bearing little resemblance to either real photos or Monet paintings. It's clear we need more complex structure. \n\n- In the 1st tuning model, we got inspiration from existing models in literature and practical works, subsequent tuning iterations involved the addition of more layers and channels to increase model complexity. Additionally, hyperparameters such as learning rate, filter size, beta for Adam optimizer, and lambda for controlling the importance of cycle consistency were adjusted. These hyperparameters were found to significantly impact CycleGAN performance, alongside model complexity. Following these adjustments, the quality of generated images notably improved, though the monet-style images still closely resembled the original photos. This indicated a potential issue of overfitting.\n\n- In the second tuning model,image augmentation techniques and repeated sampling of Monet images for each epoch in subsequent tuning iterationsis introduced to reduce model overfitting. As a results, overfitting was clearly reduced and loss metrics are also improved compared to the previous model. Additionally, the generated monet-style images began to exhibit characteristics more akin to Monet paintings.\n\n- For the final tuning model, training duration was extended to accommodate as much training as resources allowed. Learning rate scheduling was implemented, starting with a larger learning rate and progressively decaying it to a smaller value during later stages of training. Moreover, increasing the batch size for training helped in generalizing Monet-style changes to photo images more effectively, resulting in further improvements.\n\n### Key Learnings and Take-Aways:\n**Model Structure Sensitivity**: CycleGAN's effectiveness heavily depends on the complexity of its architecture. A relatively intricate structure is necessary to achieve satisfactory results.\n\n**Complex Training Dynamics**: Training CycleGAN is intricate due to the management of various loss functions. Balancing the discriminator and generator is crucial but challenging. The discriminator's improvement often plateaus quickly, possibly due to rapid learning or imbalanced learning rates compared to the generator. This imbalance adversely affects training performance.\n\n**Tuning Lambda Hyperparameter**: Adjusting the hyperparameter lambda can significantly impact the generated images' quality. Reducing lambda can encourage the generator to produce Monet-style images more faithfully, rather than making only minor adjustments to the original photos, leading to a more authentic Monet-like appearance.\n\n**Optimizing Adam's Lambda Parameter**: Reducing the lambda parameter in the Adam optimizer (e.g., from 0.9 to 0.5) has been found to enhance model performance. This adjustment prevents the learning rate from being too influenced by past steps, potentially improving convergence.\n\n**Image augmentation and resampling**: Implementing image augmentation techniques and increasing the frequency of Monet image samples in each epoch effectively combats overfitting. This enables longer training durations, facilitating the generalization of Monet-style characteristics across the generated images.\n\n**Impact of Batch Size**: Increasing the batch size from 1 or 2 to a slightly larger value like 6, contributes to generating Monet-style images more effectively. This adjustment aids in producing a greater variety of Monet-like images, enhancing the overall quality of generated outputs. It seems though if the batch size is too big, it's not good either too.\n\n \n### Future Improvements:\n\n- **Buffer for Discriminator**: Incorporating a buffer for the discriminator could address the issue of rapid convergence and potential overfitting. Similar to the approach in the original CycleGAN paper, maintaining a history of the last 50 generated images to train the discriminator can mitigate oscillations and promote a more generalized discriminator. By training the discriminator against a variety of recent generator outputs, it can better discern real from fake images, thus improving the overall stability and performance of the model.\n\n- **Separate Monitoring of Adversary Loss**: Monitoring the adversary loss separately for real and fake images can provide deeper insights into the training dynamics and aid in fine-tuning the model. By analyzing the individual contributions of real and fake images to the discriminator's loss, adjustments can be made to better balance the training process and mitigate issues such as overfitting.\n\n- **Decoupling Lambda Hyperparameters**: The lambda hyperparameter currently controls both the cycle consistency loss and the identity loss. Identity loss is for regularization, so it's not necessarily linked to cycle consistency loss. Decoupling these hyperparameters would provide more flexibility in tuning the model.\n\n- **Optimizing Loss Functions**: Exploring different weightings for the various loss components could lead to further improvements in model performance. For instance, adjusting the ratio of real to fake images in the discriminator's loss function could prioritize specific aspects of image generation. By fine-tuning the balance between real and fake image contributions, the model can be steered towards generating more desirable outputs, aligning more closely with the desired artistic style.\n \n- **Experience More Image Augmentation and Data Sampling**: I could try more image augementation (e.g.,cropping) and data sampling method. I believe those would help to diversify the training data. This increased variation can enable the model to generalize more effectively to unseen data and improves its capacity to capture diverse artistic styles.","metadata":{}},{"cell_type":"markdown","source":"## 🫶Thanks for reading! If you like my notebook, please upvote😸🐾","metadata":{}},{"cell_type":"markdown","source":"# \n# References\nhttps://www.kaggle.com/code/ryanholbrook/tfrecords-basics
\nhttps://www.kaggle.com/code/ihelon/monet-visualization-and-augmentation/notebook
\nhttps://www.kaggle.com/code/dimitreoliveira/introduction-to-cyclegan-monet-paintings
\nhttps://www.kaggle.com/code/amyjang/monet-cyclegan-tutorial
\nhttps://keras.io/examples/generative/cyclegan/
\nhttps://www.tensorflow.org/tutorials/generative/cyclegan
\nhttps://arxiv.org/abs/1703.10593
\nhttps://medium.com/@jonathan-hui/gan-gan-series-2d279f906e7b
","metadata":{}}]}