{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 1. Project Topic\n\nGitHub Repo: https://github.com/wr0b1n/MSDS-5511-Week5\n\nIn this competition, we will use Generative Adversarial Networks (GANs) to recreate Claude Monet's artistic style in images. GANs consist of a generator and discriminator, where the generator produces Monet-style images, and the discriminator distinguishes real from generated ones. The goal is to generate 7,000 to 10,000 Monet-style images that will be evaluated on the MiFID (Memorization-informed Fréchet Inception Distance).\n\nGenerative Deep Learning models are a subset of deep learning models designed to generate new data samples that resemble a given dataset. These models are used in various applications, such as image generation, text generation, and more. The key idea behind generative models is to learn the underlying distribution of the training data and then sample from this distribution to create new data samples.\n\nGANs in particular are a powerful class of generative models that consist of two neural networks, namely a generator and a discriminator that are trained adversarially. The goal of the generator is to create realistic data samples, while the discriminator tries to decide between real and fake samples. This adversarial training process results in the generator improving its ability to produce high-quality, realistic data.","metadata":{}},{"cell_type":"markdown","source":"# 2. Data\n\nWe are provided with monet paintings both in JPG and TFRecords format as our training data. In case of the JPG data we have about 300 image files. For generating new images we are provided with 7038 photos (again in JPG and TFRecords format). These photos will be manipulated by our model in order get the monet style and make them look like real monet paintings.\n\nSince we are working with a large amount of image data we will probably face extremely long computation time. For this reason we will need to activate the TPU accelerator in the Kaggle notebook.\n\nBoth the monet paintings as well as the photos have a size of 256x256 pixels. The submission format is a zip-file containing 7,000-10,000 images sized 256x256 as well. ","metadata":{}},{"cell_type":"markdown","source":"# 3. Import Python Libraries\n\nLet's import the most important libraries for our project.","metadata":{}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport cv2\nimport os\nimport math\nfrom PIL import Image\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.nn.init as init\nfrom torch.utils.data import Dataset, random_split, DataLoader\n\nimport torchvision.models as models\nimport torchvision.transforms as transforms\n\nfrom tqdm.notebook import tqdm\nimport itertools\nimport time\nimport shutil","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:17:26.495770Z","iopub.execute_input":"2023-10-18T09:17:26.496212Z","iopub.status.idle":"2023-10-18T09:17:30.342510Z","shell.execute_reply.started":"2023-10-18T09:17:26.496188Z","shell.execute_reply":"2023-10-18T09:17:30.341715Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. EDA and Data Preprocessing\n\nFirst, we have to determine the paths were our image data is located and have a look at how many images we are working with.","metadata":{}},{"cell_type":"code","source":"# access folders\npath_monet = \"../input/gan-getting-started/monet_jpg/\"\npath_photo = \"../input/gan-getting-started/photo_jpg/\"\n\nprint(\"Number of monet paintings: {}\".format(len(os.listdir(path_monet))))\nprint(\"Number of photos: {}\".format(len(os.listdir(path_photo))))","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:17:30.346097Z","iopub.execute_input":"2023-10-18T09:17:30.348126Z","iopub.status.idle":"2023-10-18T09:17:30.909867Z","shell.execute_reply.started":"2023-10-18T09:17:30.348099Z","shell.execute_reply":"2023-10-18T09:17:30.909178Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, let's display some example images, both monet and real photos.","metadata":{}},{"cell_type":"code","source":"# function to display sample images in a grid format\ndef display_sample_images(path, num_samples=16):\n    sample_images = os.listdir(path)[:num_samples]\n    \n    w = int(num_samples ** .5)\n    h = math.ceil(num_samples / w)\n    \n    for ind, image_name in enumerate(sample_images):\n        img = cv2.imread(os.path.join(path, image_name))\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) \n        plt.subplot(h, w, ind + 1)\n        plt.imshow(img)\n        plt.axis(\"off\")\n    \n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:17:30.910687Z","iopub.execute_input":"2023-10-18T09:17:30.910921Z","iopub.status.idle":"2023-10-18T09:17:30.915847Z","shell.execute_reply.started":"2023-10-18T09:17:30.910901Z","shell.execute_reply":"2023-10-18T09:17:30.915264Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display_sample_images(path_monet)","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:17:30.917629Z","iopub.execute_input":"2023-10-18T09:17:30.918117Z","iopub.status.idle":"2023-10-18T09:17:31.566208Z","shell.execute_reply.started":"2023-10-18T09:17:30.918095Z","shell.execute_reply":"2023-10-18T09:17:31.565600Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display_sample_images(path_photo)","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:17:31.566863Z","iopub.execute_input":"2023-10-18T09:17:31.567072Z","iopub.status.idle":"2023-10-18T09:17:32.295529Z","shell.execute_reply.started":"2023-10-18T09:17:31.567053Z","shell.execute_reply":"2023-10-18T09:17:32.294957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see monet paintings have that distinctive style that many of us instantly relate to the famous painter called Monet. The photos on the other hand seem to be taken with a normal camera. Both types of images are colored ones, not just black and white.\n\nWe can verify the stated image sizes from the kaggle description by inspecting the width and height of both the monet paintings as well as the real photos.","metadata":{}},{"cell_type":"code","source":"# function to analyze and show the image sizes in height and width \ndef analyze_image_sizes(image_dir):\n    image_sizes = []\n    \n    # determine image height and width\n    for image_filename in os.listdir(image_dir):\n        img = Image.open(os.path.join(image_dir, image_filename))\n        width, height = img.size\n        image_sizes.append((width, height))\n    \n    widths, heights = zip(*image_sizes)\n    \n    # plot the image widths\n    plt.figure(figsize=(10, 4))\n    plt.subplot(1, 2, 1)\n    plt.hist(widths, bins=30, color='blue', alpha=0.7)\n    plt.xlabel('Image Width')\n    plt.ylabel('Frequency')\n    plt.title('Distribution of Image Widths')\n    \n    # plot the image heights\n    plt.subplot(1, 2, 2)\n    plt.hist(heights, bins=30, color='green', alpha=0.7)\n    plt.xlabel('Image Height')\n    plt.ylabel('Frequency')\n    plt.title('Distribution of Image Heights')\n    \n    plt.tight_layout()\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:17:32.296606Z","iopub.execute_input":"2023-10-18T09:17:32.297054Z","iopub.status.idle":"2023-10-18T09:17:32.302114Z","shell.execute_reply.started":"2023-10-18T09:17:32.297031Z","shell.execute_reply":"2023-10-18T09:17:32.301511Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"analyze_image_sizes(path_monet)","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:17:32.303189Z","iopub.execute_input":"2023-10-18T09:17:32.303657Z","iopub.status.idle":"2023-10-18T09:17:34.195193Z","shell.execute_reply.started":"2023-10-18T09:17:32.303634Z","shell.execute_reply":"2023-10-18T09:17:34.194486Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"analyze_image_sizes(path_photo)","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:17:34.196269Z","iopub.execute_input":"2023-10-18T09:17:34.196499Z","iopub.status.idle":"2023-10-18T09:18:27.862067Z","shell.execute_reply.started":"2023-10-18T09:17:34.196480Z","shell.execute_reply":"2023-10-18T09:18:27.861352Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see in the histograms all images are indeed of size 256x256 pixels. Next, we will have a look at how the different color channels are distributed in the images. ","metadata":{}},{"cell_type":"code","source":"# function to display the present color channels in an image alongside the image itself\ndef display_color_channels(path):\n    plt.figure(figsize=(16, 4))\n    \n    img = cv2.imread(path)\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) \n    plt.subplot(1, 4, 1)\n    plt.imshow(img)\n    plt.axis(\"off\")\n    \n    colors = [\"red\", \"green\", \"blue\"]\n    for i in range(len(colors)):\n        plt.subplot(1, 4, i + 2)\n        plt.hist(\n            img[:, :, i].reshape(-1),\n            bins=50,\n            color=colors[i],\n            density=True\n        )\n        plt.xlim(0, 255)\n        plt.xticks([])\n        plt.yticks([])\n     \n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:18:27.863081Z","iopub.execute_input":"2023-10-18T09:18:27.863283Z","iopub.status.idle":"2023-10-18T09:18:27.868013Z","shell.execute_reply.started":"2023-10-18T09:18:27.863264Z","shell.execute_reply":"2023-10-18T09:18:27.867264Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# show monet example color channels\ndisplay_color_channels(path_monet + os.listdir(path_monet)[0])\ndisplay_color_channels(path_monet + os.listdir(path_monet)[1])\ndisplay_color_channels(path_monet + os.listdir(path_monet)[2])","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:18:27.870269Z","iopub.execute_input":"2023-10-18T09:18:27.870472Z","iopub.status.idle":"2023-10-18T09:18:28.784591Z","shell.execute_reply.started":"2023-10-18T09:18:27.870454Z","shell.execute_reply":"2023-10-18T09:18:28.784014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# show photo example color channels\ndisplay_color_channels(path_photo + os.listdir(path_photo)[0])\ndisplay_color_channels(path_photo + os.listdir(path_photo)[1])\ndisplay_color_channels(path_photo + os.listdir(path_photo)[2])","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:18:28.785306Z","iopub.execute_input":"2023-10-18T09:18:28.785657Z","iopub.status.idle":"2023-10-18T09:18:29.902218Z","shell.execute_reply.started":"2023-10-18T09:18:28.785636Z","shell.execute_reply":"2023-10-18T09:18:29.901576Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The above plots show the color channels for three monet paintings respectively photos. We could now ask the question if a certain color channel is more present in the monet paintings than in the photos.","metadata":{}},{"cell_type":"code","source":"# function to calculate the average channel values (Red, Green, Blue)\ndef calculate_average_channel_values(image_dir):\n    \n    # init each channel (R, G, B)\n    channel_sum = np.zeros(3) \n    total_images = 0\n\n    for image_filename in os.listdir(image_dir):\n        img = Image.open(os.path.join(image_dir, image_filename))\n        img_array = np.array(img)\n        \n        # make sure it is a color image\n        if img_array.shape == (256, 256, 3):\n            channel_sum += np.sum(img_array, axis=(0, 1))\n            total_images += 1\n\n    average_channel_values = channel_sum / total_images\n    return average_channel_values\n\n# calculate average channel values\naverage_monet_values = calculate_average_channel_values(path_monet)\naverage_photos_values = calculate_average_channel_values(path_photo)\n\n# ceate bar charts to compare average channel values\nchannels = ['Red', 'Green', 'Blue']\nx = np.arange(len(channels))\nwidth = 0.35\n\nplt.bar(x - width/2, average_monet_values, width, label='Monet Paintings', color='b', alpha=0.7)\nplt.bar(x + width/2, average_photos_values, width, label='Normal Photos', color='g', alpha=0.7)\n\nplt.xlabel('Color Channel')\nplt.ylabel('Average Value')\nplt.title('Average Channel Values: Monet Paintings vs. Normal Photos')\nplt.xticks(x, channels)\nplt.legend()\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:18:29.904629Z","iopub.execute_input":"2023-10-18T09:18:29.905039Z","iopub.status.idle":"2023-10-18T09:18:48.911929Z","shell.execute_reply.started":"2023-10-18T09:18:29.905017Z","shell.execute_reply":"2023-10-18T09:18:48.911359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The bar plot shows us that all color channels (Red, Green, Blue) have higher average values in Monet's paintings compared to normal photos. This makes sense since Monet is known for his impressionist style which often uses intense colors to capture the play of light and atmosphere. They tend to have a richer and more saturated color palette. This characteristic aligns with Monet's artistic style, where he emphasized the use of color to show emotions and capture the essence of a scene rather than adhering to strict color realism. Now that we are fimiliar with the given data we can begin with model building.","metadata":{}},{"cell_type":"markdown","source":"# 5. Model Building\n\nAs we already know Generative Adversarial Network (GAN) is a type of machine learning model consisting of a generator and a discriminator that compete against each other to produce realistic data. In this project we will work with CycleGAN which is a specific GAN variant used for image-to-image translation tasks without paired data. It employs two generators and two discriminators to enable translating images from one domain to another while ensuring that the translations are consistent when reversed. This is commonly used for tasks like style transfer and image transformation.\n\nSince we are working with neural networks and image data we will make use of the Kaggle GPU to reduce running time of the notebook. ","metadata":{}},{"cell_type":"code","source":"device = 'cuda' if torch.cuda.is_available() else 'cpu'\nprint(\"Device used for modeling: \", device)","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:18:48.912940Z","iopub.execute_input":"2023-10-18T09:18:48.913364Z","iopub.status.idle":"2023-10-18T09:18:48.940503Z","shell.execute_reply.started":"2023-10-18T09:18:48.913342Z","shell.execute_reply":"2023-10-18T09:18:48.939752Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.1 Dataset and Dataloader\n\nFirst we create a dataset for training our generative model. It takes the two directories where our monet paintings and photos can be found. The class loads and preprocesses the images, applying resizing and optionally normalization to create a consistent input for our neural network. With the __getitem__ method, we can randomly select a Monet-style image and the corresponding photo, processes them, and returns them as tensors. The __len__ method returns the length of the smaller of the two sets of images, ensuring that the dataset is limited by the number of available pairs.","metadata":{}},{"cell_type":"code","source":"class ImageDataset(Dataset):\n    def __init__(self, path_monet, path_photo, size=(256, 256), normalize=True):\n        super().__init__()\n        self.monet_dir = path_monet\n        self.photo_dir = path_photo\n        self.monet_idx = dict()\n        self.photo_idx = dict()\n        \n        # normalize the images for consistency\n        if normalize:\n            self.transform = transforms.Compose([\n                transforms.Resize(size),\n                transforms.ToTensor(),\n                transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))                                \n            ])\n        else:\n            self.transform = transforms.Compose([\n                transforms.Resize(size),\n                transforms.ToTensor()])\n            \n        for i, fl in enumerate(os.listdir(self.monet_dir)):\n            self.monet_idx[i] = fl\n        for i, fl in enumerate(os.listdir(self.photo_dir)):\n            self.photo_idx[i] = fl\n\n    # randomnly select a pair of monet painting and photo\n    def __getitem__(self, idx):\n        rand_idx = int(np.random.uniform(0, len(self.monet_idx.keys())))\n        photo_path = os.path.join(self.photo_dir, self.photo_idx[rand_idx])\n        monet_path = os.path.join(self.monet_dir, self.monet_idx[idx])\n        photo_img = Image.open(photo_path)\n        photo_img = self.transform(photo_img)\n        monet_img = Image.open(monet_path)\n        monet_img = self.transform(monet_img)\n        return photo_img, monet_img\n\n    # ensure same amount of available photos\n    def __len__(self):\n        return min(len(self.monet_idx.keys()), len(self.photo_idx.keys()))","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:18:48.941385Z","iopub.execute_input":"2023-10-18T09:18:48.941609Z","iopub.status.idle":"2023-10-18T09:18:48.955439Z","shell.execute_reply.started":"2023-10-18T09:18:48.941589Z","shell.execute_reply":"2023-10-18T09:18:48.954911Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"img_ds = ImageDataset(path_monet, path_photo)\nimg_dl = DataLoader(img_ds, batch_size=1, pin_memory=True)","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:18:48.957944Z","iopub.execute_input":"2023-10-18T09:18:48.959359Z","iopub.status.idle":"2023-10-18T09:18:48.977260Z","shell.execute_reply.started":"2023-10-18T09:18:48.959337Z","shell.execute_reply":"2023-10-18T09:18:48.976771Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The following is a helper method that performs the reverse operation of normalizing an image. Since our model only works with normalized versions of the images, this is needed for plotting the original version of the images after modeling them.","metadata":{}},{"cell_type":"code","source":"# reverse the normalization process\ndef reverse_normalize(img, mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]):\n    for image, mu, std in zip(img, mean, std):\n        image.mul_(std).add_(std)\n        \n    return img","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:18:48.979864Z","iopub.execute_input":"2023-10-18T09:18:48.980706Z","iopub.status.idle":"2023-10-18T09:18:48.986878Z","shell.execute_reply.started":"2023-10-18T09:18:48.980678Z","shell.execute_reply":"2023-10-18T09:18:48.986342Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.2 Model building blocks\n\nThe upsampling part is important for increasing the spatial resolution of an image. It is typically used in the generator network to gradually upsample the feature maps, which helps in generating higher-resolution output images from lower-resolution input images.\n\nThe purpose of the following method is to define the logic for upsampling within a neural network. The choice of whether to use dropout or not is controlled by the use_dropout argument, providing more flexibility in designing the generator architecture.","metadata":{}},{"cell_type":"code","source":"# method for upsampling feature maps\ndef Upsample(in_ch, out_ch, use_dropout=True, dropout_ratio=0.5):\n    if use_dropout:\n        return nn.Sequential(\n            nn.ConvTranspose2d(in_ch, out_ch, 3, stride=2, padding=1, output_padding=1),\n            nn.InstanceNorm2d(out_ch),\n            nn.Dropout(dropout_ratio),\n            nn.GELU())\n    else:\n        return nn.Sequential(\n            nn.ConvTranspose2d(in_ch, out_ch, 3, stride=2, padding=1, output_padding=1),\n            nn.InstanceNorm2d(out_ch),\n            nn.GELU())","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:18:48.989301Z","iopub.execute_input":"2023-10-18T09:18:48.989793Z","iopub.status.idle":"2023-10-18T09:18:48.999743Z","shell.execute_reply.started":"2023-10-18T09:18:48.989770Z","shell.execute_reply":"2023-10-18T09:18:48.999211Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Next, we need the convolutional layer which is responsible for extracting features from the input images. These layers apply convolution operations to process the image data and capture important patterns and details. They are used both in the generator and discriminator networks.\n\nThe below method returns a PyTorch Sequential object that can be used to create the architecture of both the generator and discriminator in a CycleGAN model. The flexibility provided by the function's arguments allows for different architectural choices.","metadata":{}},{"cell_type":"code","source":"# define a convolutional layer with options for padding and more\ndef Convlayer(in_ch, out_ch, kernel_size=3, stride=2, use_leaky=True, use_inst_norm=True, use_pad=True):\n    if use_pad:\n        conv = nn.Conv2d(in_ch, out_ch, kernel_size, stride, 1, bias=True)\n    else:\n        conv = nn.Conv2d(in_ch, out_ch, kernel_size, stride, 0, bias=True)\n\n    if use_leaky:\n        actv = nn.LeakyReLU(negative_slope=0.2, inplace=True)\n    else:\n        actv = nn.GELU()\n\n    if use_inst_norm:\n        norm = nn.InstanceNorm2d(out_ch)\n    else:\n        norm = nn.BatchNorm2d(out_ch)\n\n    return nn.Sequential(conv, norm, actv)","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:18:49.002357Z","iopub.execute_input":"2023-10-18T09:18:49.004143Z","iopub.status.idle":"2023-10-18T09:18:49.015238Z","shell.execute_reply.started":"2023-10-18T09:18:49.004121Z","shell.execute_reply":"2023-10-18T09:18:49.014549Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The Residual Block is part of the generator network. Its purpose is to make the network learn residual functions, which help preserve important details and prevent vanishing gradients during training. This is particularly useful for maintaining image quality and consistency during the translation process.","metadata":{}},{"cell_type":"code","source":"# defines the residual block for the generator part\nclass Resblock(nn.Module):\n    def __init__(self, in_features, use_dropout=True, dropout_ratio=0.5):\n        super().__init__()\n        layers = list()\n        layers.append(nn.ReflectionPad2d(1))\n        layers.append(Convlayer(in_features, in_features, 3, 1, False, use_pad=False))\n        layers.append(nn.Dropout(dropout_ratio))\n        layers.append(nn.ReflectionPad2d(1))\n        layers.append(nn.Conv2d(in_features, in_features, 3, 1, padding=0, bias=True))\n        layers.append(nn.InstanceNorm2d(in_features))\n        self.res = nn.Sequential(*layers)\n\n    def forward(self, x):\n        return x + self.res(x)","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:18:49.017603Z","iopub.execute_input":"2023-10-18T09:18:49.017820Z","iopub.status.idle":"2023-10-18T09:18:49.027993Z","shell.execute_reply.started":"2023-10-18T09:18:49.017801Z","shell.execute_reply":"2023-10-18T09:18:49.027322Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The generator is the component responsible for translating images from one domain to another. It takes an input image (e.g., a photo) and transforms it into an image that mimics the style or characteristics of another domain (e.g., a Monet painting). The purpose of the generator is to learn a mapping between these domains and generate high-quality images in the target domain.\n\nThe class below represents this generator and defines the architecture and operations needed to perform the image translation task. It makes use of the Upsampling, Convolutional Layers, and ResBlock which we defined above.","metadata":{}},{"cell_type":"code","source":"# defines the generator class built upon convolutional layers\nclass Generator(nn.Module):\n    def __init__(self, in_ch, out_ch, num_res_blocks=6):\n        super().__init__()\n        model = list()\n        model.append(nn.ReflectionPad2d(3))\n        model.append(Convlayer(in_ch, 64, 7, 1, False, True, False))\n        model.append(Convlayer(64, 128, 3, 2, False))\n        model.append(Convlayer(128, 256, 3, 2, False))\n        for _ in range(num_res_blocks):\n            model.append(Resblock(256))\n        model.append(Upsample(256, 128))\n        model.append(Upsample(128, 64))\n        model.append(nn.ReflectionPad2d(3))\n        model.append(nn.Conv2d(64, out_ch, kernel_size=7, padding=0))\n        model.append(nn.Tanh())\n\n        self.gen = nn.Sequential(*model)\n\n    def forward(self, x):\n        return self.gen(x)","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:18:49.028863Z","iopub.execute_input":"2023-10-18T09:18:49.029261Z","iopub.status.idle":"2023-10-18T09:18:49.041608Z","shell.execute_reply.started":"2023-10-18T09:18:49.029241Z","shell.execute_reply":"2023-10-18T09:18:49.040912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The discriminator is the component responsible for distinguishing between real and generated images. Its purpose is to provide feedback to the generator by assessing the realism of the generated images and helping the generator improve over time. The discriminator is trained to classify input images as either real (belonging to the target domain) or fake (generated by the generator).","metadata":{}},{"cell_type":"code","source":"# defines the discriminator component\nclass Discriminator(nn.Module):\n    def __init__(self, in_ch, num_layers=4):\n        super().__init__()\n        model = list()\n        model.append(nn.Conv2d(in_ch, 64, 4, stride=2, padding=1))\n        model.append(nn.LeakyReLU(0.2, inplace=True))\n        for i in range(1, num_layers):\n            in_chs = 64 * 2**(i-1)\n            out_chs = in_chs * 2\n            if i == num_layers -1:\n                model.append(Convlayer(in_chs, out_chs, 4, 1))\n            else:\n                model.append(Convlayer(in_chs, out_chs, 4, 2))\n        model.append(nn.Conv2d(512, 1, kernel_size=4, stride=1, padding=1))\n        self.disc = nn.Sequential(*model)\n\n    def forward(self, x):\n        return self.disc(x)","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:18:49.042561Z","iopub.execute_input":"2023-10-18T09:18:49.042877Z","iopub.status.idle":"2023-10-18T09:18:49.058210Z","shell.execute_reply.started":"2023-10-18T09:18:49.042857Z","shell.execute_reply":"2023-10-18T09:18:49.057590Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As with deep learning in general we also need a weight initialization in CycleGAN to set the initial values of the model's weights in a way that helps improve the training process and convergence speed. This is really important since a proper weight initialization can prevent issues like vanishing gradients and allow the model to learn more effectively.","metadata":{}},{"cell_type":"code","source":"# initalize with normally distributed weights around mean 0 and standard deviation of 0.02\ndef init_weights(net, init_type='normal', std=0.02):\n    def init_func(m):\n        classname = m.__class__.__name__\n        if hasattr(m, 'weight') and (classname.find('Conv') != -1 or classname.find('Linear') != -1):\n            init.normal_(m.weight.data, 0.0, std)\n            if hasattr(m, 'bias') and m.bias is not None:\n                init.constant_(m.bias.data, 0.0)\n        elif classname.find('BatchNorm2d') != -1:\n            init.normal_(m.weight.data, 1.0, std)\n            init.constant_(m.bias.data, 0.0)\n    net.apply(init_func)","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:18:49.060669Z","iopub.execute_input":"2023-10-18T09:18:49.060971Z","iopub.status.idle":"2023-10-18T09:18:49.071451Z","shell.execute_reply.started":"2023-10-18T09:18:49.060951Z","shell.execute_reply":"2023-10-18T09:18:49.070772Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.3 Cycle GAN\n\nFirst, we will need some additional helper methods that are explained and defined below.\n\nThe purpose of the first function is to provide a convenient way to control whether the parameters of a list of models should be trainable (require gradients) or frozen (not require gradients) during training. This can be useful in scenarios where you want to fine-tune certain parts of a pre-trained model while keeping others fixed, or when you want to selectively train or freeze specific model components during different training phases.","metadata":{}},{"cell_type":"code","source":"# decide whether certain model parameters should be trainable or not\ndef update_req_grad(models, requires_grad=True):\n    for model in models:\n        for param in model.parameters():\n            param.requires_grad = requires_grad","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:18:49.072353Z","iopub.execute_input":"2023-10-18T09:18:49.073003Z","iopub.status.idle":"2023-10-18T09:18:49.088680Z","shell.execute_reply.started":"2023-10-18T09:18:49.072981Z","shell.execute_reply":"2023-10-18T09:18:49.087997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The purpose of the \"sample fake\" mechanism in CycleGAN is to help stabilize training by sampling a subset of generated fake images and using them as inputs to the discriminator. This approach is used to reduce the variability in the inputs to the discriminator, which can help avoid large oscillations in the discriminator's outputs from one training iteration to the next. By doing so, it can make the training process more stable and lead to better convergence.\n\nFor this reason we provide a method seen below to control the number of fake images used for each training iteration and introduce a level of randomness in the selection of fake images to stabilize training in CycleGAN.","metadata":{}},{"cell_type":"code","source":"# class to save 50 generated fake images and sample through them to feed discriminator\nclass sample_fake(object):\n    def __init__(self, max_imgs=50):\n        self.max_imgs = max_imgs\n        self.cur_img = 0\n        self.imgs = list()\n\n    def __call__(self, imgs):\n        ret = list()\n        for img in imgs:\n            if self.cur_img < self.max_imgs:\n                self.imgs.append(img)\n                ret.append(img)\n                self.cur_img += 1\n            else:\n                if np.random.ranf() > 0.5:\n                    idx = np.random.randint(0, self.max_imgs)\n                    ret.append(self.imgs[idx])\n                    self.imgs[idx] = img\n                else:\n                    ret.append(img)\n        return ret","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:18:49.089610Z","iopub.execute_input":"2023-10-18T09:18:49.089837Z","iopub.status.idle":"2023-10-18T09:18:49.101304Z","shell.execute_reply.started":"2023-10-18T09:18:49.089817Z","shell.execute_reply":"2023-10-18T09:18:49.100610Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The purpose of learning rate scheduling is to dynamically adjust the learning rate during training to optimize the convergence of the neural network. Learning rate scheduling typically involves reducing the learning rate over time to help the model converge more smoothly and reach a better minimum of the loss function.\n\nBy using this linear decay strategy, the learning rate gradually decreases as training progresses beyond **self.decay_epochs**. It's a simple yet effective way to improve training stability.","metadata":{}},{"cell_type":"code","source":"# class to schedule the learning rate during training\nclass lr_sched():\n    def __init__(self, decay_epochs=100, total_epochs=200):\n        self.decay_epochs = decay_epochs\n        self.total_epochs = total_epochs\n\n    def step(self, epoch_num):\n        if epoch_num <= self.decay_epochs:\n            return 1.0\n        else:\n            fract = (epoch_num - self.decay_epochs)  / (self.total_epochs - self.decay_epochs)\n            return 1.0 - fract","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:18:49.102399Z","iopub.execute_input":"2023-10-18T09:18:49.102760Z","iopub.status.idle":"2023-10-18T09:18:49.114714Z","shell.execute_reply.started":"2023-10-18T09:18:49.102616Z","shell.execute_reply":"2023-10-18T09:18:49.114031Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, we define a class that is able to collect and track important statistics, like loss values and iteration numbers, over the course of training. These statistics can be used for various purposes, such as monitoring the training progress, visualizing loss curves, or making decisions about model training.","metadata":{}},{"cell_type":"code","source":"# class for saving some training metrics\nclass AvgStats(object):\n    def __init__(self):\n        self.reset()\n        \n    def reset(self):\n        self.losses =[]\n        self.its = []\n        \n    def append(self, loss, it):\n        self.losses.append(loss)\n        self.its.append(it)","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:18:49.115643Z","iopub.execute_input":"2023-10-18T09:18:49.115854Z","iopub.status.idle":"2023-10-18T09:18:49.134189Z","shell.execute_reply.started":"2023-10-18T09:18:49.115836Z","shell.execute_reply":"2023-10-18T09:18:49.133463Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The following class defines the Cycle GAN itself together with the required training loop. It trains the model to translate images between the two domains (photos and Monet paintings). It includes the training logic for both generators and discriminators, learning rate scheduling, and tracking of training statistics. The model's losses are backpropagated, and the parameters are updated accordingly during training.","metadata":{}},{"cell_type":"code","source":"# defines the Cycle GAN and its training loop\nclass CycleGAN(object):\n    def __init__(self, in_ch, out_ch, epochs, device, start_lr=2e-4, lmbda=10, idt_coef=0.5, decay_epoch=0):\n        self.epochs = epochs\n        self.decay_epoch = decay_epoch if decay_epoch > 0 else int(self.epochs/2)\n        self.lmbda = lmbda\n        self.idt_coef = idt_coef\n        self.device = device\n        self.gen_mtp = Generator(in_ch, out_ch)\n        self.gen_ptm = Generator(in_ch, out_ch)\n        self.desc_m = Discriminator(in_ch)\n        self.desc_p = Discriminator(in_ch)\n        self.init_models()\n        self.mse_loss = nn.MSELoss()\n        self.l1_loss = nn.L1Loss()\n        self.adam_gen = torch.optim.Adam(itertools.chain(self.gen_mtp.parameters(), self.gen_ptm.parameters()),\n                                         lr = start_lr, betas=(0.5, 0.999))\n        self.adam_desc = torch.optim.Adam(itertools.chain(self.desc_m.parameters(), self.desc_p.parameters()),\n                                          lr=start_lr, betas=(0.5, 0.999))\n        self.sample_monet = sample_fake()\n        self.sample_photo = sample_fake()\n        gen_lr = lr_sched(self.decay_epoch, self.epochs)\n        desc_lr = lr_sched(self.decay_epoch, self.epochs)\n        self.gen_lr_sched = torch.optim.lr_scheduler.LambdaLR(self.adam_gen, gen_lr.step)\n        self.desc_lr_sched = torch.optim.lr_scheduler.LambdaLR(self.adam_desc, desc_lr.step)\n        self.gen_stats = AvgStats()\n        self.desc_stats = AvgStats()\n        \n    def init_models(self):\n        init_weights(self.gen_mtp)\n        init_weights(self.gen_ptm)\n        init_weights(self.desc_m)\n        init_weights(self.desc_p)\n        self.gen_mtp = self.gen_mtp.to(self.device)\n        self.gen_ptm = self.gen_ptm.to(self.device)\n        self.desc_m = self.desc_m.to(self.device)\n        self.desc_p = self.desc_p.to(self.device)\n        \n    def train(self, photo_dl):\n        for epoch in range(self.epochs):\n            start_time = time.time()\n            avg_gen_loss = 0.0\n            avg_desc_loss = 0.0\n            t = tqdm(photo_dl, leave=False, total=photo_dl.__len__())\n            for i, (photo_real, monet_real) in enumerate(t):\n                photo_img, monet_img = photo_real.to(device), monet_real.to(device)\n                update_req_grad([self.desc_m, self.desc_p], False)\n                self.adam_gen.zero_grad()\n\n                # forward pass through generator\n                fake_photo = self.gen_mtp(monet_img)\n                fake_monet = self.gen_ptm(photo_img)\n\n                cycl_monet = self.gen_ptm(fake_photo)\n                cycl_photo = self.gen_mtp(fake_monet)\n\n                id_monet = self.gen_ptm(monet_img)\n                id_photo = self.gen_mtp(photo_img)\n\n                # generator losses\n                idt_loss_monet = self.l1_loss(id_monet, monet_img) * self.lmbda * self.idt_coef\n                idt_loss_photo = self.l1_loss(id_photo, photo_img) * self.lmbda * self.idt_coef\n\n                cycle_loss_monet = self.l1_loss(cycl_monet, monet_img) * self.lmbda\n                cycle_loss_photo = self.l1_loss(cycl_photo, photo_img) * self.lmbda\n\n                monet_desc = self.desc_m(fake_monet)\n                photo_desc = self.desc_p(fake_photo)\n\n                real = torch.ones(monet_desc.size()).to(self.device)\n\n                adv_loss_monet = self.mse_loss(monet_desc, real)\n                adv_loss_photo = self.mse_loss(photo_desc, real)\n\n                # total generator loss\n                total_gen_loss = cycle_loss_monet + adv_loss_monet\\\n                              + cycle_loss_photo + adv_loss_photo\\\n                              + idt_loss_monet + idt_loss_photo\n                \n                avg_gen_loss += total_gen_loss.item()\n\n                # backward pass\n                total_gen_loss.backward()\n                self.adam_gen.step()\n\n                # forward pass through Descriminator\n                update_req_grad([self.desc_m, self.desc_p], True)\n                self.adam_desc.zero_grad()\n\n                fake_monet = self.sample_monet([fake_monet.cpu().data.numpy()])[0]\n                fake_photo = self.sample_photo([fake_photo.cpu().data.numpy()])[0]\n                fake_monet = torch.tensor(fake_monet).to(self.device)\n                fake_photo = torch.tensor(fake_photo).to(self.device)\n\n                monet_desc_real = self.desc_m(monet_img)\n                monet_desc_fake = self.desc_m(fake_monet)\n                photo_desc_real = self.desc_p(photo_img)\n                photo_desc_fake = self.desc_p(fake_photo)\n\n                real = torch.ones(monet_desc_real.size()).to(self.device)\n                fake = torch.zeros(monet_desc_fake.size()).to(self.device)\n\n                # descriminator losses\n                monet_desc_real_loss = self.mse_loss(monet_desc_real, real)\n                monet_desc_fake_loss = self.mse_loss(monet_desc_fake, fake)\n                photo_desc_real_loss = self.mse_loss(photo_desc_real, real)\n                photo_desc_fake_loss = self.mse_loss(photo_desc_fake, fake)\n\n                monet_desc_loss = (monet_desc_real_loss + monet_desc_fake_loss) / 2\n                photo_desc_loss = (photo_desc_real_loss + photo_desc_fake_loss) / 2\n                total_desc_loss = monet_desc_loss + photo_desc_loss\n                avg_desc_loss += total_desc_loss.item()\n\n                # backward\n                monet_desc_loss.backward()\n                photo_desc_loss.backward()\n                self.adam_desc.step()\n                \n                t.set_postfix(gen_loss=total_gen_loss.item(), desc_loss=total_desc_loss.item())\n            \n            avg_gen_loss /= photo_dl.__len__()\n            avg_desc_loss /= photo_dl.__len__()\n            time_req = time.time() - start_time\n            \n            self.gen_stats.append(avg_gen_loss, time_req)\n            self.desc_stats.append(avg_desc_loss, time_req)\n            \n            print(\"Epoch %d  -  Generator Loss:%f  -  Discriminator Loss:%f\" % (epoch+1, avg_gen_loss, avg_desc_loss))\n      \n            self.gen_lr_sched.step()\n            self.desc_lr_sched.step()","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:18:49.136590Z","iopub.execute_input":"2023-10-18T09:18:49.136823Z","iopub.status.idle":"2023-10-18T09:18:49.154008Z","shell.execute_reply.started":"2023-10-18T09:18:49.136792Z","shell.execute_reply":"2023-10-18T09:18:49.153292Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 5.4 Training process\n\nNow that we have defined all necesarry building blocks we can finally train our model! We create a new object of our Cycle GAN class and start the training process with 50 epochs.","metadata":{}},{"cell_type":"code","source":"gan = CycleGAN(3, 3, 50, device)\ngan.train(img_dl)","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:18:49.158432Z","iopub.execute_input":"2023-10-18T09:18:49.158733Z","iopub.status.idle":"2023-10-18T09:20:53.919971Z","shell.execute_reply.started":"2023-10-18T09:18:49.158703Z","shell.execute_reply":"2023-10-18T09:20:53.918926Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's visualize the results.","metadata":{}},{"cell_type":"code","source":"# plot losses\ndef plot_gan_loss():\n    plt.xlabel(\"Epochs\")\n    plt.ylabel(\"Losses\")\n    plt.plot(gan.gen_stats.losses, 'r', label='Generator Loss')\n    plt.plot(gan.desc_stats.losses, 'b', label='Descriminator Loss')\n    plt.legend()\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:20:53.920625Z","iopub.status.idle":"2023-10-18T09:20:53.921006Z","shell.execute_reply.started":"2023-10-18T09:20:53.920872Z","shell.execute_reply":"2023-10-18T09:20:53.920886Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_gan_loss()","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:20:53.924088Z","iopub.status.idle":"2023-10-18T09:20:53.924581Z","shell.execute_reply.started":"2023-10-18T09:20:53.924423Z","shell.execute_reply":"2023-10-18T09:20:53.924439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that both loss functions gradually decrease over time which is what we would expect from a working model. In order to see the actual results we can plot same random photos and the corresponding Monet-esque Photos created by our model.","metadata":{}},{"cell_type":"code","source":"# plot realistic photo and created monet-style image \n_, ax = plt.subplots(3, 2, figsize=(12, 12))\nfor i in range(3):\n    photo_img, _ = next(iter(img_dl))\n    pred_monet = gan.gen_ptm(photo_img.to(device)).cpu().detach()\n    photo_img = reverse_normalize(photo_img)\n    pred_monet = reverse_normalize(pred_monet)\n    \n    ax[i, 0].imshow(photo_img[0].permute(1, 2, 0))\n    ax[i, 1].imshow(pred_monet[0].permute(1, 2, 0))\n    ax[i, 0].set_title(\"Input Photo\")\n    ax[i, 1].set_title(\"Monet-esque Photo\")\n    ax[i, 0].axis(\"off\")\n    ax[i, 1].axis(\"off\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:20:53.925394Z","iopub.status.idle":"2023-10-18T09:20:53.925884Z","shell.execute_reply.started":"2023-10-18T09:20:53.925741Z","shell.execute_reply":"2023-10-18T09:20:53.925757Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can clearly see the difference and the expected monet-esque style. We talk about this in more detail in the Results and Analysis section. ","metadata":{}},{"cell_type":"markdown","source":"## 5.5 Hyperparameter Tuning\n\nFor this project, I decided to manually tune some hyperparameters. An automated tuning process would result in training times that were too long considering the running time of the whole notebook is already quite long. Since the available usage time for the GPU on Kaggle is limited, I had to make some compromise by tuning only the learning rate and the beta value of the optimizer manually. I tried out different combination during the project, but only left the ones yielding the best results in the code see before. \n\nThe values are the following:\n\n* Starting learning rate: 2e-4\n* Betas of optimizer: 0.5 and 0.999\n\nConsidering the limited resources in this project the results seem to be quite good. Of course, in an industrial scenario the tuning process has to be more sophisticated.","metadata":{}},{"cell_type":"markdown","source":"## 5.6 Alternative loss functions\n\nDuring training I also experimented with an alternative optimizer, namely RMSprop instead of the Adam optimizer. RMSprop (Root Mean Square Propagation) is an adaptive learning rate optimization algorithm. Is uses a moving average of the squared gradients to adjust the learning rate, while Adam also includes a term to correct for bias (momentum). There are two main advantages using RMSprop. First, is can be more stable than Adam in some cases, as it doesn't include the momentum term, which might result in smaller updates and reduce the likelihood of divergence during training. And second, RMSprop can lead to faster convergence in certain situations, especially when dealing with non-stationary or noisy gradients. However, the final results did not differ much from the Adam optimizer, which is why I kept the Adam version in the code.","metadata":{}},{"cell_type":"markdown","source":"# 6. Results and Analysis\n\nNow that we have trained our model we can use it to create the actual monet-like images from the provided photos in order to make a submission to the competition.","metadata":{}},{"cell_type":"code","source":"# class to store and access the photos (similar to the paintings done before)\nclass PhotoDataset(Dataset):\n    def __init__(self, photo_dir, size=(256, 256), normalize=True):\n        super().__init__()\n        self.photo_dir = photo_dir\n        self.photo_idx = dict()\n        \n        # normalize images if needed\n        if normalize:\n            self.transform = transforms.Compose([\n                transforms.Resize(size),\n                transforms.ToTensor(),\n                transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))                                \n            ])\n        else:\n            self.transform = transforms.Compose([\n                transforms.Resize(size),\n                transforms.ToTensor()                               \n            ])\n        for i, fl in enumerate(os.listdir(self.photo_dir)):\n            self.photo_idx[i] = fl\n\n    # retrieve an image\n    def __getitem__(self, idx):\n        photo_path = os.path.join(self.photo_dir, self.photo_idx[idx])\n        photo_img = Image.open(photo_path)\n        photo_img = self.transform(photo_img)\n        return photo_img\n\n    def __len__(self):\n        return len(self.photo_idx.keys())","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:20:53.926722Z","iopub.status.idle":"2023-10-18T09:20:53.927207Z","shell.execute_reply.started":"2023-10-18T09:20:53.927064Z","shell.execute_reply":"2023-10-18T09:20:53.927080Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# make a dataloader and the required directory for storing the images to be created\nph_ds = PhotoDataset(path_photo)\nph_dl = DataLoader(ph_ds, batch_size=1, pin_memory=True)\n!mkdir ../images\ntrans = transforms.ToPILImage()","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:20:53.928019Z","iopub.status.idle":"2023-10-18T09:20:53.928481Z","shell.execute_reply.started":"2023-10-18T09:20:53.928344Z","shell.execute_reply":"2023-10-18T09:20:53.928359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# use model to create a monet style image\nt = tqdm(ph_dl, leave=False, total=ph_dl.__len__())\nfor i, photo in enumerate(t):\n    with torch.no_grad():\n        pred_monet = gan.gen_ptm(photo.to(device)).cpu().detach()\n    \n    # revert the normalization process to obtain the original image style\n    pred_monet = reverse_normalize(pred_monet)\n    img = trans(pred_monet[0]).convert(\"RGB\")\n    \n    # store the image\n    img.save(\"../images/\" + str(i+1) + \".jpg\")","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:20:53.929279Z","iopub.status.idle":"2023-10-18T09:20:53.929767Z","shell.execute_reply.started":"2023-10-18T09:20:53.929619Z","shell.execute_reply":"2023-10-18T09:20:53.929634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# create a zip file from all created images\nshutil.make_archive(\"/kaggle/working/images\", 'zip', \"/kaggle/images\")","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:20:53.930566Z","iopub.status.idle":"2023-10-18T09:20:53.931034Z","shell.execute_reply.started":"2023-10-18T09:20:53.930897Z","shell.execute_reply":"2023-10-18T09:20:53.930912Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The generated zip file will be used by Kaggle to determine our place on the competition leaderboard. We can have a look at the images created for the submission. We can clearly recognize the typical monet style.","metadata":{}},{"cell_type":"code","source":"display_sample_images(\"../images/\")","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:20:53.931819Z","iopub.status.idle":"2023-10-18T09:20:53.932280Z","shell.execute_reply.started":"2023-10-18T09:20:53.932142Z","shell.execute_reply":"2023-10-18T09:20:53.932157Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# display best hyperparameters of model\nlearning_rate = [2e-4]\nbeta_1 = [0.5]\nbeta_2 = [0.999]  \ndf_results = pd.DataFrame({'Learning Rate': learning_rate, 'Beta 1': beta_1, 'Beta 2': beta_2})\nprint(df_results)","metadata":{"execution":{"iopub.status.busy":"2023-10-18T09:20:53.933065Z","iopub.status.idle":"2023-10-18T09:20:53.933523Z","shell.execute_reply.started":"2023-10-18T09:20:53.933386Z","shell.execute_reply":"2023-10-18T09:20:53.933401Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see from the resulting images during training and testing (= create submission images) we can confidently say that our model performed quite well with the shown set of hyperparameters above. If we had we more computing power we could automate this process of hyperparameter tuning even further to optimize the results. But for the context of this project the results are good enough thanks to the underlying CycleGAN model.\n\nSince CycleGAN is effective for a wide range of image-to-image translation tasks, such as turning photos into paintings or converting day scenes to night scenes it also worked out well for our use case. Additionally, the adversarial training process helps generate high-quality, realistic images. Of course, there are other models we could explore. For example U-Net which is excellent for tasks that involve dense pixel-to-pixel correspondence, such as image segmentation. It even has a simpler architecture compared to GANs.","metadata":{}},{"cell_type":"markdown","source":"# 7. Conclusion\n\n## 7.1 Result Summary\n\nAs we have already seen in the previous section, the CycleGAN network works really well with our images and the shown set of hyperparameters.","metadata":{}},{"cell_type":"markdown","source":"## 7.2 Learnings and Takeaways\n\nWorking with neural networks involves a distinct workflow compared to simpler supervised learning methods like regression analysis. Training models in this context can be time-consuming, sometimes taking hours. Therefore, maintaining a clean and precise work process is crucial to avoid excessive training iterations. An often underestimated but highly valuable practice in this regard is the strategic use of print statements within your code. These statements enable effective debugging, inspection of intermediate results, and quicker error identification. Another challenge I encountered during my work was the constrained memory and time resources of the GPU provided by Kaggle.","metadata":{}},{"cell_type":"markdown","source":"## 7.3 What didn't work\n\nOverall, everything worked quite well. However, it would have been great to work with more epochs in order to get more precise results or perform an automated and more sohpisticated hyperparameter tuning. The limited GPU time really makes things a bit hard since processing image data requires so much disk space and computation time. It took quite a few iterations to find good values for the hyperparameters.","metadata":{}},{"cell_type":"markdown","source":"## 7.4 Possible improvements\n\nIf we had enough computing power, we could expand our hyperparameter tuning by including more parameters and a wider value range. We could also experiment with totally different model and compare their results.","metadata":{}}]}