{"cells":[{"metadata":{},"cell_type":"markdown","source":"# CycleGAN Image-to-Image Translation\n![](https://junyanz.github.io/CycleGAN/images/failures.jpg)\n### CycleGAN and Notebook Structure\n\nA CycleGAN is made of two types of networks: **discriminators, and generators**. In this example, the discriminators are responsible for classifying images as real or fake (for both $X$ and $Y$ kinds of images). The generators are responsible for generating convincing, fake images for both kinds of images. \n\nThis notebook will detail the steps you should take to define and train such a CycleGAN. \n\n>1. You'll load in the image data using PyTorch's DataLoader class to efficiently read in images from a specified directory. \n2. Then, you'll be tasked with defining the CycleGAN architecture according to provided specifications. You'll define the discriminator and the generator models.\n3. You'll complete the training cycle by calculating the adversarial and cycle consistency losses for the generator and discriminator network and completing a number of training epochs. *It's suggested that you enable GPU usage for training.*\n4. Finally, you'll evaluate your model by looking at the loss over time and looking at sample, generated images.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Load and Visualize the Data","execution_count":null},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import os\nimport torch\nimport torchvision\nfrom torch.utils.data import DataLoader\nfrom torchvision import transforms\nimport numpy as np\nfrom torchvision.datasets import ImageFolder\nfrom torch.utils.data import Dataset\nimport cv2\nimport functools","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### DataLoaders\n\nThe `get_data_loader` function returns training and test DataLoaders that can load data efficiently and in specified batches. The function has the following parameters:\n* `image_type`: `summer` or `winter`,  the names of the directories where the X and Y images are stored\n* `image_dir`: name of the main image directory, which holds all training and test images\n* `image_size`: resized, square image dimension (all images will be resized to this dim)\n* `batch_size`: number of images in one batch of data\n\nThe test data is strictly for feeding to our generators, later on, so we can visualize some generated samples on fixed, test data.\n\nYou can see that this function is also responsible for making sure our images are of the right, square size (128x128x3) and converted into Tensor image types.\n\n**It's suggested that you use the default values of these parameters.**\n\nNote: If you are trying this code on a different set of data, you may get better results with larger `image_size` and `batch_size` parameters. If you change the `batch_size`, make sure that you create complete batches in the training loop otherwise you may get an error when trying to save sample data. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"data_dir='../input/gan-getting-started'\nclass GANdataGenerator(Dataset):\n    def __init__(self,data_dir ,  data_files , image_size ):\n        super(GANdataGenerator , self).__init__()\n        self.data_dir = data_dir\n        self.data = data_files\n        self.transform = transforms.Compose([\n            transforms.ToPILImage(),\n            transforms.Resize(image_size),\n            transforms.CenterCrop(image_size),\n            transforms.ToTensor()\n        ])\n        \n    def __len__(self):\n        return len(self.data)\n    \n    def __getitem__(self, index):\n        data_file = self.data[index]\n        data_file = os.path.join(self.data_dir , data_file)\n        img_bgr = cv2.imread(data_file)\n        img_rgb = cv2.cvtColor(img_bgr , cv2.COLOR_BGR2RGB)\n        img_tensor = self.transform(img_rgb)\n        \n        return img_tensor","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"monet_jpg = os.listdir('../input/gan-getting-started/monet_jpg')\nphoto_jpg = os.listdir('../input/gan-getting-started/photo_jpg')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#load the Dataloader for X and Y image sets\nX_type='photo_jpg'\nY_type='monet_jpg'\nX_set = GANdataGenerator(data_dir='../input/gan-getting-started/photo_jpg' , data_files=photo_jpg , image_size=128 )\nY_set = GANdataGenerator(data_dir='../input/gan-getting-started/monet_jpg' , data_files=monet_jpg , image_size=128 )\n\nX_dataloader = DataLoader(X_set , batch_size=16 , shuffle=True , num_workers=0)\nY_dataloader = DataLoader(Y_set , batch_size=16 , shuffle=True , num_workers=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x_data = next(iter(X_dataloader))\ny_data = next(iter(Y_dataloader))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualize the data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nprint(\"X Data Visualization\")\ndef imshow(img):\n    npimg = img.numpy()\n    plt.imshow(np.transpose(npimg, (1, 2, 0)))\nfig = plt.figure(figsize=(12, 8))\nimshow(torchvision.utils.make_grid(x_data))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Y data Visualization\")\nfig = plt.figure(figsize=(12, 8))\nimshow(torchvision.utils.make_grid(y_data))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\ndef viz_sample(samples_x , samples_y):\n    #get random sample from the x and y\n    #plot the sample\n    plt.title(\"Real sample\")\n    plt.imshow(samples_x)\n    plt.show()\n    plt.title(\"Fake sample\")\n    plt.imshow(samples_y)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Pre-processing: scaling from -1 to 1\n\nWe need to do a bit of pre-processing; we know that the output of our `tanh` activated generator will contain pixel values in a range from -1 to 1, and so, we need to rescale our training images to a range of -1 to 1. (Right now, they are in a range from 0-1.)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"img = x_data[0]\n\nprint(\"Min : \",img.min())\nprint(\"Max : \",img.max())\n\ndef rescale(x , feature_range=(-1,1)):\n    min , max = feature_range\n    x = x*(max - min) +min\n    \n    return x","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# scaled range\nscaled_img = rescale(img)\n\nprint('Scaled min: ', scaled_img.min())\nprint('Scaled max: ', scaled_img.max())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"---\n## Define the Model\n\nA CycleGAN is made of two discriminator and two generator networks.\n\n## Discriminators\n\nThe discriminators, $D_X$ and $D_Y$, in this CycleGAN are convolutional neural networks that see an image and attempt to classify it as real or fake. In this case, real is indicated by an output close to 1 and fake as close to 0. The discriminators have the following architecture:\n\n![](https://ohinfa.files.wordpress.com/2017/04/discriminator.png?w=676)\n\nThis network sees a 256x256x3 image, and passes it through 5 convolutional layers that downsample the image by a factor of 2. The first four convolutional layers have a BatchNorm and ReLu activation function applied to their output, and the last acts as a classification layer that outputs one value.\n\n### Convolutional Helper Function\n\nTo define the discriminators, you're expected to use the provided `conv` function, which creates a convolutional layer + an optional batch norm layer.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch.nn as nn\nimport torch.nn.functional as F\n\n#define the conv block for discriminator\ndef conv_block(in_channels , out_channels , kernel_size=4 , strides=2 ,padding_type='reflect' , padding=1 , batch_norm=True):\n    layers = []\n    if(padding_type=='reflect'):\n        pad = nn.ReflectionPad2d(padding)\n        layers.append(pad)\n        \n    if(padding_type=='replicate'):\n        pad = nn.ReplicationPad2d(padding)\n        layers.append(pad)\n        \n        \n    conv_layer = nn.Conv2d(in_channels=in_channels , out_channels=out_channels , \n                          kernel_size=kernel_size , stride=strides , padding=0 ,  bias=False)\n    layers.append(conv_layer)\n    if(batch_norm):\n        bn = nn.InstanceNorm2d(out_channels)\n        layers.append(bn)\n        \n    return nn.Sequential(*layers)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Define the Discriminator Architecture\n\nThe task is to fill in the `__init__` function with the specified 5 layer conv net architecture. Both $D_X$ and $D_Y$ have the same architecture, so we only need to define one class, and later instantiate two discriminators. \n> It's recommended that you use a **kernel size of 4x4** and use that to determine the correct stride and padding size for each layer. [This Stanford resource](http://cs231n.github.io/convolutional-networks/#conv) may also help in determining stride and padding sizes.\n\n* Define convolutional layers in `__init__`\n* Then fill in the forward behavior of the network\n\nThe `forward` function defines how an input image moves through the discriminator, and the most important thing is to pass it through your convolutional layers in order, with a **ReLu** activation function applied to all but the last layer.\n\nYou should **not** apply a sigmoid activation function to the output, here, and that is because we are planning on using a squared error loss for training. And you can read more about this loss function, later in the notebook.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class Discriminator(nn.Module):\n    \n    def __init__(self , conv_dim , padding_mode):\n        super(Discriminator , self).__init__()\n        \"\"\"\n        Define the discriminator model to classify the images as real or fake\n        in_channels =3\n        out_channels=1\n        \"\"\"\n        self.input_channels = 3\n        self.output_channels = 1\n        self.padding_mode = padding_mode\n        # 256 - 256 -3 --> 128 - 128 -64\n        self.conv_block1 = conv_block(self.input_channels , conv_dim  , padding_type=padding_mode)\n        # 128 - 128 - 64 --> 64 - 64 - 128\n        self.conv_block2 = conv_block(conv_dim , conv_dim*2 , padding_type=padding_mode)\n        # 64 - 64 - 128 --> 32 - 32 - 256\n        self.conv_block3 = conv_block(conv_dim*2 , conv_dim*4 , padding_type=padding_mode )\n        # 32 - 32 - 256 --> 16 - 16 - 512\n        #self.conv_block4 = conv_block(conv_dim*4 , conv_dim*8 , padding_type=padding_mode)\n        # 16 - 16 - 512 --> 8 - 8 - 1024\n        self.conv_block5 = conv_block(conv_dim*4 , self.output_channels , kernel_size=1 ,\n                                      strides=1 ,padding_type=None ,  padding=0 , batch_norm=False)\n        \n        self.leaky_relu = nn.LeakyReLU(negative_slope=0.2, inplace=True)\n        \n    def forward(self , x):\n        x = self.leaky_relu(self.conv_block1(x))\n        x = self.leaky_relu(self.conv_block2(x))\n        x = self.leaky_relu(self.conv_block3(x))\n        #x = self.leaky_relu(self.conv_block4(x))\n        \n        out = self.conv_block5(x)\n        \n        return out\n        ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Generators\n\nThe generators, `G_XtoY` and `G_YtoX` (sometimes called F), are made of an **encoder**, a conv net that is responsible for turning an image into a smaller feature representation, and a **decoder**, a *transpose_conv* net that is responsible for turning that representation into an transformed image. These generators, one from XtoY and one from YtoX, have the following architecture:\n\n![](https://miro.medium.com/max/2692/1*_KxtJIVtZjVaxxl-Yl1vJg.png)\n\nThis network sees a 256x256x3 image, compresses it into a feature representation as it goes through three convolutional layers and reaches a series of residual blocks. It goes through a few (typically 6 or more) of these residual blocks, then it goes through three transpose convolutional layers (sometimes called *de-conv* layers) which upsample the output of the resnet blocks and create a new image!\n\nNote that most of the convolutional and transpose-convolutional layers have BatchNorm and ReLu functions applied to their outputs with the exception of the final transpose convolutional layer, which has a `tanh` activation function applied to the output. Also, the residual blocks are made of convolutional and batch normalization layers, which we'll go over in more detail, next.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"---\n### Residual Block Class\n\nTo define the generators, you're expected to define a `ResidualBlock` class which will help you connect the encoder and decoder portions of the generators. You might be wondering, what exactly is a Resnet block? It may sound familiar from something like ResNet50 for image classification, pictured below.\n\n![](https://cdn-images-1.medium.com/max/1024/1*S3TlG0XpQZSIpoDIUCQ0RQ.jpeg)\n\nResNet blocks rely on connecting the output of one layer with the input of an earlier layer. The motivation for this structure is as follows: very deep neural networks can be difficult to train. Deeper networks are more likely to have vanishing or exploding gradients and, therefore, have trouble reaching convergence; batch normalization helps with this a bit. However, during training, we often see that deep networks respond with a kind of training degradation. Essentially, the training accuracy stops improving and gets saturated at some point during training. In the worst cases, deep models would see their training accuracy actually worsen over time!\n\nOne solution to this problem is to use **Resnet blocks** that allow us to learn so-called *residual functions* as they are applied to layer inputs. You can read more about this proposed architecture in the paper, [Deep Residual Learning for Image Recognition](https://arxiv.org/pdf/1512.03385.pdf) by Kaiming He et. al, and the below image is from that paper.\n\n![](https://cdn-images-1.medium.com/max/1600/1*5zSgo2L71FJos8XendgCvQ.jpeg)\n\n### Residual Functions\n\nUsually, when we create a deep learning model, the model (several layers with activations applied) is responsible for learning a mapping, `M`, from an input `x` to an output `y`.\n>`M(x) = y` (Equation 1)\n\nInstead of learning a direct mapping from `x` to `y`, we can instead define a **residual function**\n> `F(x) = M(x) - x`\n\nThis looks at the difference between a mapping applied to x and the original input, x. `F(x)` is, typically, two convolutional layers + normalization layer and a ReLu in between. These convolutional layers should have the same number of inputs as outputs. This mapping can then be written as the following; a function of the residual function and the input x. The addition step creates a kind of loop that connects the input x to the output, y:\n>`M(x) = F(x) + x` (Equation 2) or\n\n>`y = F(x) + x` (Equation 3)\n\n#### Optimizing a Residual Function\n\nThe idea is that it is easier to optimize this residual function `F(x)` than it is to optimize the original mapping `M(x)`. Consider an example; what if we want `y = x`?\n\nFrom our first, direct mapping equation, **Equation 1**, we could set `M(x) = x` but it is easier to solve the residual equation `F(x) = 0`, which, when plugged in to **Equation 3**, yields `y = x`.\n\n\n### Defining the `ResidualBlock` Class\n\nTo define the `ResidualBlock` class, we'll define residual functions (a series of layers), apply them to an input x and add them to that same input. This is defined just like any other neural network, with an `__init__` function and the addition step in the `forward` function. \n\nIn our case, you'll want to define the residual block as:\n* Two convolutional layers with the same size input and output\n* Batch normalization applied to the outputs of the convolutional layers\n* A ReLu function on the output of the *first* convolutional layer\n\nThen, in the `forward` function, add the input x to this residual block. Feel free to use the helper `conv` function from above to create this block.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#define the discriminator network with encoder - resblock - decoder\nclass ResidualBlock(nn.Module):\n    \"\"\"\n    apply two conv layers with Batchnormalization and relu activation on first block\n    \"\"\"\n    def __init__(self , conv_dim , padding_mode):\n        super(ResidualBlock , self).__init__()\n        self.conv_dim = conv_dim\n        self.conv_block1 = conv_block(self.conv_dim , self.conv_dim , kernel_size=3 ,\n                                      strides=1 ,padding_type=padding_mode , padding=1 , batch_norm=True)\n        \n        self.conv_block2 = conv_block(self.conv_dim , self.conv_dim , kernel_size=3 ,\n                                      strides=1 ,padding_type=padding_mode, padding=1 , batch_norm=True)\n        \n        self.dropout = nn.Dropout(p=0.3)\n        self.relu = nn.LeakyReLU(negative_slope=0.3 , inplace=True)\n        \n    def forward(self , x):\n        res_1 = self.dropout(self.relu(self.conv_block1(x)))\n        res_2 = self.conv_block2(res_1)\n        out  = self.relu(res_2 + x)\n        \n        return out","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Transpose Convolutional Helper Function\n\n![](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZEAAAB+CAMAAADSmtyGAAABVlBMVEX///9Qn5dGicufrbFSoppTpJwzaJM4dHT4+fn7/Pyjsbb19vfv8fLs7u+ksrdQnpcyaWlMl5E0bGxBg4Dk5+hIjdJHjond4eJDh4MsXmA5dXUvY2TP1NbIztDW2tw0a5nAx8qJl5yys7OYpKg6gcXByMq2vsGAj5UmVFcye3fQ0dG4uLiUlZaRnqMfZJ4rbGqfoaKFhodIXGNDV15xgohWZ20weLocQEZaa3A4Wl5/g4YYO0IhTVA3gn2pqqqPkJB6enpITE52hYslXo0cTnMuQ0pYcoY2WnUiY5ktaZ4yUlcTTU4AKjIAGicAQ0YaY2AmNTwSMjkNKTcSMDxjZWtSWV4TVlYvTl0AEyhvb29gdIBtg5IYT3kePVBMa4YDO1kTOFIABxkAIj4AQWgoMDQUPlk5YoRbdo0AIiUQMUIqSV0kTGtBao0APmkAVo8AK1IAN2lEXW1MHNzqAAAgAElEQVR4nO1961/b1rK2HEtBsi6+yRa+YlvY5mYgtgwuEEJsnITQJk2a5JwAITm7bfZJ05337f//5Z1ZV6WFBNnmy/vz7N8OhhqN1jxzeWbWklGUucxlLnOZy1zmMpe5zGUuc5nLXOYyl7nMZS5zmctc5jKXucxlLnOZy1zmctvSXL9tDWbDu20V7qF+2yoqrdvWQKU52Hq+fasazMMfVleqt6rC3SutDsxbVVEBOx3eqgYqtf5WWtW2Nm5Pg7n9w31Vja2s3Z4Kb6+QV9XFvnV7KsBvk6p6m3aiUumCnlgspq3euyUN+vbT+xpRcdq8JRXuXiGNGtR8z74lFc0+tZO6undLGqisdV8QY92iqsPdAlehndZuQ4O992JRSxINyeTTW4mS5vjHJF/EanAbGqiUu8/V9ApVlNTSpwNj5ioOd+9rxQWNYq7mf2rMXIO19yytlfLUXmpy4cfZR8na+NlCajfJPDd92p+5BiqV/vOYCgoKySTqKWUgDc+2MuqHu6sQH2o2Q+I9W8hqs2YQLuIB1yaQqMliKa/tzpZBGM3XzxaKC8XMLiZGNZ9aSOZfz951OR4kOkppokcDdd0Zxrx5+HSVBcdiSo0tlrKgT9tqz06DUt17k6cqtBQkLsQjGVNnmRuNZhfxAEFItDTYSUU7zTwQgcel0ryAJFdKC2mNRuTMKiPw3a2UyiJdy6+ksqo643JVDZ49W2CLiGmZQoaho76ZWW4EfrWSIoAAJAv7YCeqL92bbXcFeCQ1FbyKWitTlOjEnrqz0KBD/6FiQqT1I5sqpmIcnfuzgcQL9kvF4kqJ3LmWLmYWihoviaXZQAJ5BDLHSgEhKRZLB6mS9OKnM8yNa4MtShUzWUxUqSLER4pUxqSaXlg4mF6VAXgQapIGUgp4lLJYpxg6i5mtGTAId28/VQRLFQsFNQn1PAPdSJYGjBpbzcyikVtDv02CUQqFIuCxC24LXswcILOwUp5eBZFK/1mMA72QTZeKlGVnQBXoyaTV5EFlOg3m9s/POd+FGgV4oAoImDR4bx4UavnXU4473L2fSgQPhGQluUAdCsCGHK8WS4vJ6RtetBOLOa20ktpNcS8WdppNw1t5vRXLMjIKfPdgRaCTySZTRGsyuTLNkMvcHr8p7i7yxWR/KQp0Csl8oUjReTpNuXL3VtL5/QWW3Yv7L0QuyaeA0JFqMmW5qvSlnQCB/d2k9GJiJ6SnM2h4K9B/EDKqkQxVWlCFVlXdPUiyNkt7MzEkZufFG3De4m5WI3isZEWrAAXrlwxDJ5ncnbhc2XunaQ3u/gVS0uLCwUGelStYhJZ5medmXB1MqkGp9IidFlNkEelCJmSn2MFBmhKWpDYtg2h2Gd9FMgo8LgP5l6IDTrtQSmY5b1G1ZxOq2v4R8UDP3YXQWIT+AxZGci90h6WiREdNv5wMEooHMchuhuChkXJFjAXxQbsrsoitCbuGNfBbyjzzJbBTKRMTLRXkK0jDRWGnF9N0V5XuswPOFNV8IbNA0dEWS8CJFlKwTI01ckDr05M0ckbj4PkBoSX4/4OS4LuQe1VSP4AEkcqoQiJOTsIg7PZpQaQoiGrAQ6UJEfw2W1qE72jAAIEoZfOTNLzlwfOSKtpzsFOSMtLFFODxDztNXK6MZv8ZcNwDuho1n8mUJDqlYob9fDFD214oXlv1iCpw3q5pxQMaIgsrByVORmNacSFDCQRFhxTGWGwlahp2H6zkNUF3ksVMgYdcMllaJXgwdEg7Cm7Qj1qu1gbAd3kWBDsthOyUAjtxdFSwE9IJbfVBRA1EjFqf9J3QdeZVoicttUK+KoS0rqZoIlajzdNM0n9g5djFYrtyABHBsyAoLPA0HFMxq6Up1S5E2v7BebvGQI1RgyQzFB2Ij9SKQCeWyqaylEGmu1FyowH9ILF5ErMgzjHATqU0uSi1E4+dfClL6d2EDGLvRzoHIJCkUwshreiuXCssbLW0ovKkcD9C12B0t1jNVgESxENlsxPiWyF00tBaa3xhUXKj+6NIV8VsrJgKowP5StVYw4tqV3h5V9M/RMiNh7wvgJBLo51IS4VdIeCRCXkx5JGQnYLoVP7BW15vi6WXqSRDGrQmafqgWjVcpiq0qtk3/ZtD4v3A7IV0pxjjwVEi/YfKWoWYRtxBoJNceBYBkvU3nAomVw44OtpCVkM8VN7wYv1Iqix2gN6VdiN0DcEqJ2r5/RKzUzJkJ+DvgM7f7XQaeefS2lCOTpGSFlOgh0UbdiMvRJXEYeMCG5+iKtBTWFTz3Zujb/fS9P5Ki+kCy1CQw16y+kHQAbrNYgentckMlMkolbF1ijbSkgtAdwQZ1Vb288xdoVXQgN4lVRY7MeoOpxEg2UNIwDLwexlB2tO/SDsV0rTOMn6nEneIPJ91obvYe0bxAC9KZSlTgD4nJcioesC1JtUVcOMCul0yytzR7OZVNAj+3gphoBr0g4KMQjncXxDoFLRYhsyIIqXhygFUa8BDw9qqMcfJL2YEgygdZEPopGk7qj2LwCDubWnwe2S+S0AFB0ilkpK0a/vCToAOs1MsPY4GSQ2dZPvZboHxuAW47SQYBNosylsgLxbyQmtM213hLXf6aYQR56vVQpal8pU0EJ0Cki1Srkg7msnzRg7S8n4pzd66+vbmubF6UCzQbgRBJYFM0GE8vpAVLVVSS+0WGTraaYTuavt5JsP6AsiCxE6Q0BlpV8FO0ou1gxW+dRnbj0Tqtsm7jwr8tzXIYMwgGpJRsq0gtEKbtSgiVo29iEBWXt0XjAq4CBgkSQOd8F1whyQbBUPeyZc4Otr9CL21Oxa0MF1aLFDHgSyIDSiGJ2vkCKXPSvYdBZJGQZJ2qLuUBTE7AdlLstj5m520H6OQOpqqN2pvaAJR08WVlEQnmy3lOToa0hZcWIahs/pDIwKR2N6jlRHMkzngYQaVMb/AiDxFB3lSMk0H21pyaxBlhmo/Za1TLFuSM7l8Jl8i5Ju2CmT/TaADybMdxVy1AqeFxUJKorPK7aRijolRO6UmspNJe5gHSm2F7N6ieSTdyR6EfCJLGiuuVV39IdqRtCOlvYV0B/muoDsQH/tZGTv5IpvTIiRq+vkg4gDV7GFLFcP+XJB2dbEgJ4GQVjKUtyL7VgGPvYhtYmWF0EJip4y0U0mc3yiucjuR2AE8DiPNa2zKMCFS1g5ipP/grYKGbZbUGgN0BMsuZkFPRFYH0G9vaQQPCirSFmxHUwKdhV1OMGPAZp4Pog/+9f79WDaFBVUlkOB2WDaWLonmrbCbDsVOtPiggnZaIHwXE2Lyn3ZaEadrJrLTGu2MN8Ddj57yzUIko3QOwBs55K2qiB1tdRxVj1LFbN14tsoZVXFVY20WI6PId2XsJJ/3J5pm68FpnoMKaX6Rbr8QBkG3X8QoWMsf7E20+VrdDdmJzZVCdsqy+SyqmMBOyiG9qW0gaBve0xjPH+ldzhpQq0p5HGsVtNWn0fUo6yQBtd6IvdWF3SIbXCM6gEeexw7iMUF8UDH2VkU3crCS5X02sDDWjpLJFuBxOkF8UHGfyjj7yk4aqR9iUD+RnZQNypWRcW1b1jgteFxBVMb0Sol7LrBKbXU3Sp0Ssk3X31yhRAH6waxsFYorGe520Cqoz/tTbMCtv9oiIxTku6m8UFEosHaUtArpZ1HrR1isHrdTiBaq6QM6HmDoTGgngx0eRXute4rewxYXeZxKWwXSZmVLMnZOdyfcHDliX+89w43PFPBdlbUKwHdTRTFs1Fbf9Kbao96wtrfQT5Hv8hqlQj+Yki1V4XRvunNOBjS8xG/BTitJYadCTAzqTw8mtJPOEDkERNbQMfv32cZnLEbPJeC+Em8V1PtPJ96E4eeGH6wd0P6DuhLpP9IanWwhOlvd6TbyjQc6MAjSf8Qo3UmCngW4edoq4FmkaeKDyVd2onio0ovvP534bEWVIYnlxCNXOdri3horpRdpn50krYJ6/4ftic8kePwW7+nuz2L4mk+l6bk2WhkBj8nqeUgshP5wX5LRLD3XxmoU8t2ZnHCC3MgDAuxUomjjBBDtNMWjKg2WsBtVWAv1f14Zgcfti0wPjdz9yDwuLGt8c/6BpdhdXjSSqf28xjeAUtAPTn9awCWrqJ2KKnhwILqRYnZWeCghOyUX9zPi/EYqf393cr9VRL2lTIi27/pDzOmARyoLZJSfkl4dN6Y69XvIiSYgoljdPG2zUvk0P3amxd68msWBmnUKKnQNJENB+8MSIn63OzM8IOpfb5EhNfIrYacY2Gl9qmUcsa/kKg8othsPtjR6ro3uKsSi9+f/lHu8lD5AYBu9RY2cayO7L2isZOT+/Bo5ZJtQ1RV6/g/KCGUQWv7NxHz3KlmvHj7X0G/RTgtZ2hfgvMSaaCuXy1v2tYbldIMWvGaz8SbFiLxaLGoT8uqvxHjAEd3AS62vBc/5IA7JaOx5f8qDeUIe8Fv1norteySjUM9niQdRtH6aWeR2WiV2Iqt8+71f/Ya4nDo1MdYPKSLuhtI6FWVr8c30eCiKLfaetvFi5XWRhvHg+puZ4aEYR+LlhhiYAOqng1k/3XpkkJ1LpkHNvtnn86vGFNEu6m0FEamxK93TlbVndByr3d+dySOugmrR6T8CdG+LjYKfd2f4BGIoZWx7tJGDgnUKfPdodkpQTOJjtVM6dtDuHzREGpgmbYl6S6DhVBiLShUaORXxmPziYZHVjnAJQlG3IQ0D33090ydC12QobNvGazyik35D+o+N2T7UwdLL2kGS9x+HIitOkbaO+AuyEDYHpsq8g/Tf+pxpjqxvi7slTqBv4MUOn8e2ejN+QjdEdFDnq/tpPm9fn+2Dp012Oe9pmvmtVH04+eMjAkwSHiZP9qS/9vpf8+pWbYrJw4YoRQ3ChajDNvq18NkGs+NO/RCXVES9YO8htxOJy2pHn9HnDGzzkHP7h0fkBUsxTtO2J55sCAjY1e6xMDj8ByuxHMVVWhN/pIEuDc8Q4RrEC6VaNi2lMy0fOpIvt4mnronQeGC1LFexox7HvEYehOgOtZsBi7RahqvUWhM/ai7rrUcRYVrcv1UPW6+Q1VmTurArnYbGdoMfXauwvA9448UNc8psH7IFRUSq3qjV0HKmPpOnKcPBfY/G3T3TLJN1mduTrkJmPorIBr/Xrw/l2D73h6o/mSLpqEynKMAWXVlThF98qsGvHfIliojJiU+8ybDR6zN45tgKJyYaL/pDYafypI+nPBDF2iWICIQ2Qm4UryktkXutybJKQ57mpBXREwuCuDTbtlUTb5iqljRD5ZsiQqfb5bpSrfLlGt70z9VVw+0Nmkuv9+sDYaejCYPkSLyiiKzxmT7nwYrpKKDNFevslIeTKApFMbWZzCW1w7ICrlUThdKfvFyx/pO/poxnAy8OP9ZbnALVWvWp2/dWmE5tmGan30kkOuKY51unPRHqspOhlUPWlSPyr8XqB9gRDVbBoqjrE6gKJV2aryj9BbHdV+xVC7/6iIY1uQsfhV7zDWvbrtEfVNENrMDFcjVtlHzV3Wx0unUnjkKeZC+3IU1PxOlCBZzyNdlskhrv1kUCMeNKy6bf6cPIacX6ByKcqrQSZBqBUi7TgFSQo0bVwOUo9JogYtTG8mKv9GqZWcqf8lnaDQmp6f80pHgQSFoWoGLeU/R29HLVlN0UY9BH/Huwm1OBJMVjU2/9FRd3EDlK3NAOJ8uMZCgPiJt1Pqy1/f8SmcCesB01wvUWi1erO3QCAcnG+ZnwXHuqKJF9g9Xpdo7G7QTBIxEP/puSH4gh04gcJofS2ZkXi47RfqVgqjUsclGj9ngn91HYqxJETMMh6HmJanhe1UT9Fi0plj/aWT7jVtLj9YkMthaut+trrSBIJOJOQI3knV+cbAq/sofTfGAcz++2P2g7ibd+HyFx4sG746V3ZGq6XsFH5KKWq3sSQ4bIA4qRrZSPWMi1moqxfrmTu3Mnd5eo0qtG2bSjoR+m51W6lkZTjAAemtC23UUVy4/IG23X8uyJMn24tTU2ukGcuK4zDAylenaynDvObdKA8fSq5U4RJZTOIx6gIfG24/QDisfS3aX3CLX7wFMqUe0UKuyctVOW6tb1mqCRXvzfaCyQ3AjQUWo1/GkiUhoOd1MEkWpwJg/0Hlbqd5epiuVLW9EVH+2q1ydgwQ+kBVr91z2e3J12cA543Mkd38nttOHaro9oeBN2VwqtUZbfH5wh5ImgE3cG/T+Ol+6iLL1v4K5AjdxFpG2G8PSFIYLTuHgF7rZWpj/QXav+hVoLIDnpiOYqyizNDE+nId6rgzM/EednTlx3L9gRKi79BHujNQGlP+IvWt12oj3o+QyQbvAhRxZwApCcd7gzuBPPuDYMsw181wkGGCNHfjzoD95TQBCS7Y5JnduINHMMJ112Ssg7bCos8OmDG7gP2hyxGNm87EgFZefGisJUS3HvBYgH2KlPErnXATvWKSS53Jd2R1op8pDLYNC3+sNEPNEexhESp947izv+O7KG49zOhS/v3JysXIG5OoM6CY92FzTtDXrthDNkkCwdv+v5ogcuRyhX66HHIulJOsP/STjmYUNpsqheu8AysvkIE5mIwvbNNYWhd4MXiQTLJQCJj9fcMJX6JuSU3GgIiJscdXcYdahNeVxl3HfQc9vAScd+fRxQhe8wbf06woG2xyO9NRkihjVm/Qdo6TkBsF8s7O0PUEWWjn8JHGfY4ePHTgREvmpy7iGvHnfiPX6z1iuZyKsfczuP6ZXpR9p6cf7PTWRbWNYNuj7YiyES/5GRIDDk4cmdkzP6vg7RS3b+nWhdA06Bmr3A6ZF6295znPb/EHSIjHeWR/R4hUXKuxlHAwyjdw1u27RZQownEv3/ad+r0wXVPywdfwqIOwRtZBnUTjcuV/fCWfSeWe8j7v6A4GS2Ln9dfiTeUP2XRBpoilnV6aubKeJTsmow8B3/qN0nSavT/zD6lY7Gj+D/9T9CGdGA/9FfieTD2y7JVxB9CEk9qP/X+6V3be4A9Z8CgydQy7U9m86zo1IuSmrsgU/7j3cXd8cBRSThDH4aMAdI1PvraKkoi/gqu9sPAxaHft+DfhD6D+gPHtFEXqvqniSjrZ7DX1v+TXSxwwjlNuCBkDvtHsUDYvzXIaK6YbdMS963EUgiVq7dbDXk914FnO8CGU0MX36+Cyp+DxxqoUEHLss6LrfXFhkgEaFc2RVeq/V+h/Ldu0ujlyQmE8PesN73Ob8LfpH16mZ7MnL8Cry6/1tdDAIGPuk/dka55cc2RkIFrUZzieKdPdk5F4tp3eQJfcIaqsB8KOKDhFPv9n8fkTK4dDHQFXOd9O02i+7a5c6JSIiGf9OsYlguoM3tcTbo/f6ZqgBICB5OPNGvEQpkdkbLf4qu1b1p9lUso1aWkRwM3l1QDce/wMWH2P4kOr0OQafeG+2cixxSu1H2FR2H6XfrzusBnc0A7v3xSY7ykju5j26N1XIyh68SWo9dA/lFvWXdwGDQpXuYrzpj5FhgtU6//27EmeLFwPBdyvRILmz9ezOXy20yr9K9mpu4/tIh0ZEHtmiNSiTa/e77u5859xkM+j79D4PDbagif4LH5UZ0YbrS9G5mMKCFcIM+c0ejpb++YHz3+PMfAW9H/V4HyldvhHb6i0JiKnHLv0F3xQ4jmHXkcc5rZwDRDXh0IRF3EZLcx+Xc5uN2qOdqtZ/QTm750iXfK9hNfFfReiMIEIuE30X/eT2AeO8wSJZG78d9MStQOq1/71CmvUM+1LSOHl2+QWqsMD8vd2n/AXw3+MRiZPSh1xexM3yY+JNR7SW8fw+n5sZNEPF5WsU3m6YHX4JfWYx8eDduy2Fj0Bvd+bud7Bvwxgfk4FR93CF19rWTAEiCcTtBKuMTcKKdi8umIXscwxuIRm75Y7luMqzi38HEa4veGfzH7/9MuhF/DJAs3X3fqzt+3+RbZa3Hy7xV3DkHK1MV+nc4qlFTRHkud4GPnqG7JoaflggeEJ5DBonjDwYXoht9knA42t/rGuyKIpYZr4Ib0+/aCMnS8csBUC4GCaRkaac7H8sdi9kp8T1McHppgomGXYch4vRe8GLiDEbQDx6RN/q0VgTwpc3nKcsXslfUv1lLdCNuBHxY7fg/93zur+PR0ntK6/2nHrIMNzAUk0MCdFsQCNk/XCmG11JknW7+L+s/4v7Dd4BHn4TnsOdgRu6dubS7ooicyR3Hb087jJbnyqF+Q35w0vav2H/UIa84XVylUx/7+tV2Mr9Xc+0NpYMGAf9JEESCTxfHIrpbvU9NGkVoKsVz6Hh8YxPT2fJd4LPyVHOifr2uOPE9vGPEo/th9KH92mFE/sefBZvovzVqNCnoZ8sEj8uGYortSrt+fbnSSR/Jd5urZ0+W/2B0x39b/+9unHtuL+H3h+Rda3QG8eSsqtQEHSp/o+OtktTNF7n+eGdT0tTDn5A3BGhCyDH1XpxEhE/tNILeV0LitL+NydrrPjUIRDmY5yHhccfdBC2M4+HQF7W/06jyanK4mdsZ1emInjYuGM7XnE2TZd8/S0ByInz3w0NCRuODcadfZ81bIv5bU9zt2Y5oRw3i+Z5n6fY1iymLnRcHn0k6/5IDU/xBpzT13wadbpwPCNovhA95H3PLJ2f0O7JfaVR0V6lcA7oj0nYHP2TrEqcLO4x4uHWjBWgQRCBKxnFup8aT3M5daieF2qkM9vp2anT7/GaBGgbvXjIyCuggj0sAux7S49Ne+8mmpDud4zhPEPGyYrH9Zu9KTEJmbAQED1SBTuUPum0ko7RViAd/nPwp3/vXR341o20qlu4QU7WuLFdV8VOjCniwEfK7DtDPbh+6NZ+VSX/wOUTay5/PuZmhodZNr0au4FxVrnSJlGvXHm9SVrCD8VvxcO+u2U103sJigO9K0q74x37YTpACiLbaVYsQ12fNDBik9/LkAyOjo96wT3lcot2Brt47B34lyKjnmp4E2m10+BpaV9Ze8VeEdGf4gTHFpXfHg6BL99yQTBA8gO+OKAx62XArkvpUxVjIvjoQfQ6JGw82RUH9PAS+G3/lEDKB4fl5R5J2qwyVR6BjVeocnfKVtdcUH2Vfi/8RIh6uwY7oNPsdzFfAr3KbbW4nq3q1na5cBBf7DPwHDNJvJzrvPlFzLR2/G/t8iNYZN4YnxO2AjMI1LRPjo8KqC9D6L3JiY135aKpO5t0uDpQbfDL66Xjc4/UjEZwRPEidxbvVyUL4fXvByUhuPzavzFw6gaTiWpZSP2EFdefja4g+f0Bg77cHxzuSjJqkrbU56K1Hm49kuopf1TUYRIXZgkXYjwTxuJTl3e297o64ndAWCqbrsJ0kK7C/2V2Z/TrpPxCX30aEx70L4n7XZ6OZdn+8zHnJzpniMEMZCR1rBNB6viOH/VnjyhYIkoKt1KjRMEqA7/5Wd+hki4Rn/3NOUJ+4oAsuqqq2wR1ydzkkVf9qB2tVFNugMUoL6s5HSBjgtgQRpzN4/USSds/jAw2Sbkk7ynYu8dv4laAbdcswPcLGzEc7dF+iaRjihKfnjQXf3TkXdlLiBrcTrztQi5xvt4rDn9uUKjqvgIwiHthVk9zr1Pu+bv7FfWL5PzXRK3qe0fi4Q32CeAqlLFdvmFSEf1TfYf/RgcoOZIKMGoJBU6kIMrrzhywVTcNl7Whu1CIrQVu5tStV1AV1xYJ6kSBr7pwlXmG+An7V2eRk9MtQzLuNllJ5tEk6OTIswp1TJXRu7Cuxxe+Z5zuAR4XwgSG1CIS1eSnt1BIh9087xYnib3ZXQ9YqOIP4+FPAaz20bfUxIqzoBBLk1aYu59bVl9InhkrNpLCXr3QwSx65c//Vqycc5FpQdxNO8JSkDtofwDJb4VNBnf8jwvNJS68p7DDRldPmivSF1n8Ev3MGb6F9JpYkkORyT86rSoip977IRs6uslmqeXUg+vLXzh6Ltxh4HpOenKSxA3ayzKF03Z92hLsN4eYprFeXKyH1gI6zXo+DvhgEOJDp+WWBjC6PSANm6GSxBtydLXvrL/uyDl9tMH7yWR9W+3Btwn6dzs/ik0/sYwjrx6QO6dQsNrhra8QXs/yr/KRD/WqDcQ+u+FWZFPz/FXy3Acnpydla6G4qUAKHm8Jen+R0tnq1wbgKvxKiGEFDHGMK0E40lAxqJ+BYITs92ZVN6LejpHMG3Xp7/NBPOJSMEh53V/pEMOJbV0bbhv6DjFUMVuJO+muGIa7vXLk3Y5Ajdy10ZLPfaWHWQn51LkLOfvyY7zICwQeiQs7qVQkkuEzLCg3qr7SXiwXVJP7POaztfzyRVXT94oyvqNWC/sMg35G6g+OBQ8UUzmR3r7RTAg3qkXMGLnOL6vDJXUk8hhfiEAIAY5XJXqhO7ZQ7+esrO3172uwE7W7gvO3QXQXE4wIi/IJpNeNGVaJj2w53O4idOyfE7RAo/GKbtas1GLrV0llDNvChZx8iv1r+zNyuWjOaInubrhtnC6t+hDeN6vifPJrOLLN6TadoGm6TtqwKeYvpXyDx4BmzZdsyuGxXbLr4cB87H9cRjTh1YWjjrukUTbPsshkVeYsbQPvDSbtixfVQ1nZtfi7YxByz+Yj4Ch31XG8nIY39uJNAROJOMAA8WEHFX4P2w1U471b0+Ogi5BNPztf4vYaWdKWETqu1fxsyvnuHTEY9paIrwl5ue3QpvfV4xA8/2Ohgev364ylWT9KCGsTHjiSjpocjJX6CWVm7DJHR1pc/+erI2apvPY7RPBMv9Zp3PgoRD9fCGTj3fL1xHLJT+wu3k24iks4NDgxVBgmKSKI+/l0W1ITBxyAEVMrj2K4CRLkXmsjGW985qSufoqm9uuB8987Hqhi6kuztDZeWgYyy94K7yug2O985Mq8b3LdNvycL6pmpOMx3a/iGJvIrTtoNywyfronHv11zLXG81ht+lnaKGyazE6ly5jba6W7ITvIKfs2/2a6MktQAAAaOSURBVE54ueccdeJOe9Axy5KM/i6bvpZF8CA3QH5qobtaTKvRrH735JNDolt3zYogo2AZ+Re4gONU20vLlIwSdy+ju/Iya7a+uxFj0Aiyzaqni0Yu9yWQZLSiNM8370gyqjg1RZZZr2Z/74QYS554WrEqicd76SqQHQ+ZnUaUquDYXmSpmnfTowmK1z/qQP+Bzln5SON95NvyEIVufhI+cdIixZGsiE4CzdoN9CBFNugDVB1WUC9rhjzzZ9T+I1R89JQOferGcujvejfYU9KRC1pk99mkE+Tcl/OqIQf1SiDJ1TmSCHLbZXILntL6dutGF4FRUiaEocrsdOHbrogCw/pDhOeJg30aWQWtjK75bY71tbgvx7yguh+R4PhkP4tdoVJXmiJ2cv+3LvaHDDw/f8MTPOW2uPHWFyiolySBC1JZI2e2mNv9qyLc1fYUq30DY5GL1IW7QkHN0UKns3VZbdu6FGR0c+zJwaMh9iC/J2ZbMGr3GIkHKXT8Zst1pcJj507YTrj9EvX5CFMyDPvxiPM4O0DHpuNdCklu86wcso8f4QOSQzmhdSH4LpYHu0MyNCOjyxd1O6Sid/PDk2EVwy+C7+IokQ49jTNKRqH9kXRX8X65+WmU0K9Zj0VfYLUN6NPIVbxjsgi0U+hu/Ah/94AL47AKlumyUGtarslu13sMek7+QnhcVtQr0c5+ixpntsvSYWxX51Vo+wm6Hel5WR03a9EewhBEplr3ZJmGOOP6+juiHWXMzW1GeljFCMSUvVUWbqzbITvlSP+BetkouDLZAWOd9GG2o+hAq9jPzMbFY+Gu9uXmI8bjXPBp3bai5EVyBdKSl8kRe246t/4f+Zdbm6MRZ7hYR0yzGvVRKFJ+DAdNEGdXMiqP/iP77Pbmxwa9bQOTChC6iCe+TdKU2I4RspPVOJZ2sh5LO5nUThEXIcTH7Q/qWS5ps+JA6/muAnTrYR5X/2b/cZ1Ydehkm9SzyE6QOwRaLyajdlU+Ra1UHfOm9SMsZcdEQoei4xejcrkpSbtStuVnGgAvLk/ywALYyeJ2wi+mH7aT3nTlswB6x5zETlwMT+ys6D7ooTxu+TEZB+jISVzuT94ED9mh2DVhcq8FtJ60WUBGaZHHgYDImK0Jn+5w5Ui95eL+B7Y/nLRbDjgtjzu9M5mxjKoYZ+pxMBW3E9lGIXayxQZyO2IE/l3o4zpEFR405Y2cRx4ZRH2UkFfsqA8PSfHFLbr10TLfYjoXdIfuvZlNY/I/iSihrPbpvB0ZSUOMFejOuVc2JrZWs8ZfmWE7ubRPw6ghrt2cwk5c+KFWJe7KRm75vewVIetYdm2K58VMRql1K+FKMrrzi2SKnm7qV0/2byi8jhqVSoiMfhnIJ1rLWD+m+SQDdrc6VCxJ2nPvpS+gnSL1H9dKFT21SULBoWQUaH0lNF3otKf9G76+hztQ6K762Q6b79ar0l+t9iQPvIVFx+M8NkkYlIzixmKrKv21ObzxWd9rpIptIn3ApsHstHleCfmRP7WduHi6w931EE82bpJjZnxTyBtO9TFb7CJ2nE2mjbMdMm/HDMUnTM7EzESKbVXYWEHxPuNpnsd4bsdix3XM+jQPhTKp6j5voA+/5MQDT8xO1boxi/hgF5ObfpW7lFcr9LAr1o+ZqAjtmNdPRuwTC3RCIGr6bD7fUv5BUOuS9R9IRvGfymw+PMiVdlobbXI7kXOAYKfZ4aGEz3RW2qEnGuKGPQPvpSICWncCGemtslKZpn6ERZfPdXXkNiuebP/O0YObi7RTsy0LiNvS3ZnZiYtORsZ6GTfaZHwHM/u8UYUzKtvDRlfYzprkL91eK3TPqGp6umKKuPverkE0IVkQup4qCz8iwazqR1g8SzFZV83mv5N+sNa1Am2HxY6/sf2Lm+zkRBFsO0ybnPtkn6hldmYVHkxcV9ipSneQq9MyhmvFFx9Ho9uW0TTdWXovEWsoZ/3A45uRnpC/mVSkuxoe9B8zKlFhiUs7WSb2HzO3E5fQAyjAImfCq/8uoQd1mk75NkI9/KBO252q/7hOQnayhpHnfNFEfKRXZ1bF9h/CP0zAnrb/uFbEIeVpPkXt28LtZPi3ZicmOk2PM/rkzyvFo2Vktp/D+7XU8B/zGycmphaD2mnWdfZKqVTK5uwz71eScG8juYfFjLvexB/BejOplMFOt5qvhMRvNy8qODC5jfrxlZRvM0Co+Ldup7nMZS5zmctc5jKXuczl/wP5f5OZhzJRruH1AAAAAElFTkSuQmCC)\n\nTo define the generators, you're expected to use the above `conv` function, `ResidualBlock` class, and the below `deconv` helper function, which creates a transpose convolutional layer + an optional batchnorm layer.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def deconv_block(in_channels , out_channels , kernel_size=4 , strides=2 , padding=1 , batch_norm=True):\n    layers = []        \n    deconv_layer = nn.ConvTranspose2d(in_channels=in_channels , out_channels=out_channels ,\n                                      kernel_size=kernel_size , stride=strides , padding=padding , bias=False)\n    layers.append(deconv_layer)\n    if(batch_norm):\n        bn = nn.InstanceNorm2d(out_channels)\n        layers.append(bn)\n        \n    return nn.Sequential(*layers)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"---\n## Define the Generator Architecture\n\n* Design the `__init__` function with the specified 3 layer **encoder** convolutional net, a series of residual blocks (the number of which is given by `n_res_blocks`), and then a 3 layer **decoder** transpose convolutional net.\n* Then complete the `forward` function to define the forward behavior of the generators. Recall that the last layer has a `tanh` activation function.\n\nBoth $G_{XtoY}$ and $G_{YtoX}$ have the same architecture, so we only need to define one class, and later instantiate two generators.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## UNet Cycle Generator","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class CycleGenerator(nn.Module):\n    \"\"\"Create a Unet-based generator\"\"\"\n\n    def __init__(self, input_nc, output_nc, num_downs, ngf=64, norm_layer=nn.BatchNorm2d, use_dropout=False):\n        \"\"\"Construct a Unet generator\n        Parameters:\n            input_nc (int)  -- the number of channels in input images\n            output_nc (int) -- the number of channels in output images\n            num_downs (int) -- the number of downsamplings in UNet. For example, # if |num_downs| == 7,\n                                image of size 128x128 will become of size 1x1 # at the bottleneck\n            ngf (int)       -- the number of filters in the last conv layer\n            norm_layer      -- normalization layer\n        We construct the U-Net from the innermost layer to the outermost layer.\n        It is a recursive process.\n        \"\"\"\n        super(CycleGenerator, self).__init__()\n        # construct unet structure\n        unet_block = UnetSkipConnectionBlock(ngf * 8, ngf * 8, input_nc=None, submodule=None, norm_layer=norm_layer, innermost=True)  # add the innermost layer\n        for i in range(num_downs - 5):          # add intermediate layers with ngf * 8 filters\n            unet_block = UnetSkipConnectionBlock(ngf * 8, ngf * 8, input_nc=None, submodule=unet_block, norm_layer=norm_layer, use_dropout=use_dropout)\n        # gradually reduce the number of filters from ngf * 8 to ngf\n        unet_block = UnetSkipConnectionBlock(ngf * 4, ngf * 8, input_nc=None, submodule=unet_block, norm_layer=norm_layer)\n        unet_block = UnetSkipConnectionBlock(ngf * 2, ngf * 4, input_nc=None,no_skip=True ,  no_skip_outter=True , submodule=unet_block, norm_layer=norm_layer)\n        unet_block = UnetSkipConnectionBlock(ngf, ngf * 2, input_nc=None, no_skip=True ,  submodule=unet_block, norm_layer=norm_layer)\n        self.model = UnetSkipConnectionBlock(output_nc, ngf, input_nc=input_nc, submodule=unet_block, outermost=True, norm_layer=norm_layer)  # add the outermost layer\n\n    def forward(self, input):\n        \"\"\"Standard forward\"\"\"\n        return self.model(input)\n\n\nclass UnetSkipConnectionBlock(nn.Module):\n    \"\"\"Defines the Unet submodule with skip connection.\n        X -------------------identity----------------------\n        |-- downsampling -- |submodule| -- upsampling --|\n    \"\"\"\n\n    def __init__(self, outer_nc, inner_nc, input_nc=None,\n                 submodule=None, outermost=False, innermost=False, no_skip=False , no_skip_outter=False  , norm_layer=nn.BatchNorm2d, use_dropout=False):\n        \"\"\"Construct a Unet submodule with skip connections.\n        Parameters:\n            outer_nc (int) -- the number of filters in the outer conv layer\n            inner_nc (int) -- the number of filters in the inner conv layer\n            input_nc (int) -- the number of channels in input images/features\n            submodule (UnetSkipConnectionBlock) -- previously defined submodules\n            outermost (bool)    -- if this module is the outermost module\n            innermost (bool)    -- if this module is the innermost module\n            norm_layer          -- normalization layer\n            use_dropout (bool)  -- if use dropout layers.\n        \"\"\"\n        super(UnetSkipConnectionBlock, self).__init__()\n        self.outermost = outermost\n        self.no_skip = no_skip\n        if type(norm_layer) == functools.partial:\n            use_bias = norm_layer.func == nn.InstanceNorm2d\n        else:\n            use_bias = norm_layer == nn.InstanceNorm2d\n        if input_nc is None:\n            input_nc = outer_nc\n        downconv = nn.Conv2d(input_nc, inner_nc, kernel_size=4,\n                             stride=2, padding=1, bias=use_bias)\n        downrelu = nn.LeakyReLU(0.2, True)\n        downnorm = norm_layer(inner_nc)\n        uprelu = nn.ReLU(True)\n        upnorm = norm_layer(outer_nc)\n        \n        if no_skip :\n            \n            if(no_skip_outter):\n                \n                upconv = nn.ConvTranspose2d(inner_nc*2  , outer_nc,\n                                            kernel_size=4, stride=2,\n                                            padding=1, bias=use_bias)\n                \n            else:\n                upconv = nn.ConvTranspose2d(inner_nc , outer_nc,\n                                            kernel_size=4, stride=2,\n                                            padding=1, bias=use_bias)\n            \n            down = [downrelu, downconv, downnorm]\n            \n            up = [uprelu, upconv, upnorm]\n\n            if use_dropout:\n                model = down + [submodule] + up + [nn.Dropout(0.5)]\n            else:\n                model = down + [submodule] + up\n            \n\n        elif outermost:\n            upconv = nn.ConvTranspose2d(inner_nc , outer_nc,\n                                        kernel_size=4, stride=2,\n                                        padding=1)\n            down = [downconv]\n            up = [uprelu, upconv, nn.Tanh()]\n            model = down + [submodule] + up\n        elif innermost:\n            upconv = nn.ConvTranspose2d(inner_nc, outer_nc,\n                                        kernel_size=4, stride=2,\n                                        padding=1, bias=use_bias)\n            down = [downrelu, downconv]\n            up = [uprelu, upconv, upnorm]\n            model = down + up\n        else:\n            upconv = nn.ConvTranspose2d(inner_nc * 2, outer_nc,\n                                        kernel_size=4, stride=2,\n                                        padding=1, bias=use_bias)\n            down = [downrelu, downconv, downnorm]\n            up = [uprelu, upconv, upnorm]\n\n            if use_dropout:\n                model = down + [submodule] + up + [nn.Dropout(0.5)]\n            else:\n                model = down + [submodule] + up\n\n        self.model = nn.Sequential(*model)\n\n    def forward(self, x):\n        if self.no_skip:\n          return self.model(x)  \n    \n        elif self.outermost:\n            return self.model(x)\n        else:   # add skip connections\n            return torch.cat([x, self.model(x)], 1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Residual Cycle Generator","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class CycleGenerator(nn.Module):\n    def __init__(self , conv_dim , n_resblocks , padding_mode):\n        super(CycleGenerator , self).__init__()\n        self.conv_dim = conv_dim\n        self.n_resblocks = n_resblocks\n        self.in_channels = 3\n        self.out_channels = 3\n        self.encoder = nn.Sequential(\n            conv_block(self.in_channels , conv_dim , padding_type=padding_mode) ,\n            nn.ReLU(True),\n            conv_block(conv_dim , conv_dim*2 , padding_type=padding_mode), \n            nn.ReLU(True),\n            conv_block(conv_dim*2 , conv_dim*4 , padding_type=padding_mode),\n            nn.ReLU(True)\n        )\n        layers = []\n        for i_res in range(self.n_resblocks):\n            layers.append(ResidualBlock(conv_dim*4 , padding_mode=padding_mode))\n        self.res_block = nn.Sequential(*layers)\n        \n        self.decoder =nn.Sequential(\n            deconv_block(conv_dim*4 , conv_dim*2 ),\n            nn.ReLU(True),\n            deconv_block(conv_dim*2, conv_dim ),\n            nn.ReLU(True),\n            deconv_block(conv_dim , self.out_channels , batch_norm=False),\n            nn.Tanh()\n        )\n    def forward(self, x):\n        #inference throght the encoder\n        x = self.encoder(x)\n        x = self.res_block(x)\n        x = self.decoder(x)\n        \n        return x","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"---\n## Create the complete network\n\nUsing the classes you defined earlier, you can define the discriminators and generators necessary to create a complete CycleGAN. The given parameters should work for training.\n\nFirst, create two discriminators, one for checking if $X$ sample images are real, and one for checking if $Y$ sample images are real. Then the generators. Instantiate two of them, one for transforming a painting into a realistic photo and one for transforming a photo into  into a painting.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def weights_init_normal(m):\n    classname = m.__class__.__name__\n    if(classname.find('Conv') != -1):\n        torch.nn.init.normal(m.weight.data , 0.0 , 0.02)\n    elif(classname.find('Batchnorm2d')!= -1):\n        torch.nn.init.normal(m.weight.data , 0.0 , 0.02)\n        torch.nn.init.constant(m.bias.data , 0.0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_model(g_conv_dim=64 , d_conv_dim=64 , n_resblocks = 6 , padding_mode='reflect'):\n    G_X2Y = CycleGenerator( input_nc =3 , output_nc=3, num_downs=5, ngf=64,  \n                           norm_layer=nn.InstanceNorm2d, use_dropout=True)\n    \n    G_Y2X = CycleGenerator( input_nc=3, output_nc=3, num_downs=5, ngf=64, \n                           norm_layer=nn.InstanceNorm2d, use_dropout=True)\n    \n    #G_X2Y = CycleGenerator(conv_dim = g_conv_dim , n_resblocks=n_resblocks , padding_mode=padding_mode)\n    #G_Y2X = CycleGenerator(conv_dim = g_conv_dim , n_resblocks=n_resblocks , padding_mode=padding_mode)\n    \n    D_X = Discriminator(d_conv_dim  ,padding_mode)\n    D_Y = Discriminator(d_conv_dim , padding_mode)\n    \n    G_X2Y.apply(weights_init_normal)\n    G_Y2X.apply(weights_init_normal)\n    D_X.apply(weights_init_normal)\n    D_Y.apply(weights_init_normal)\n    \n    device = 'cuda:0' if torch.cuda.is_available() else 'cpu'\n    #move the models to gpu if availabele\n    G_X2Y.to(device)\n    G_Y2X.to(device)\n    D_X.to(device)\n    D_Y.to(device)\n    \n    return G_X2Y , G_Y2X , D_X , D_Y","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"G_X2Y , G_Y2X , D_X , D_Y = create_model(padding_mode='replicate')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"G_X2Y.load_state_dict(torch.load('../input/cyclegan/generator_x2y.pth'))\nG_Y2X.load_state_dict(torch.load('../input/cyclegan/generator_y2x.pth'))\nD_X.load_state_dict(torch.load('../input/cyclegan/descriminator_x.pth'))\nD_Y.load_state_dict(torch.load('../input/cyclegan/descriminator_y.pth'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Discriminator and Generator Losses\n\nComputing the discriminator and the generator losses are key to getting a CycleGAN to train.\n\n![](https://ruotianluo.github.io/assets/triplets/cycle4.png)\n\n**Image from [original paper](https://arxiv.org/abs/1703.10593) by Jun-Yan Zhu et. al.**\n\n* The CycleGAN contains two mapping functions $G: X \\rightarrow Y$ and $F: Y \\rightarrow X$, and associated adversarial discriminators $D_Y$ and $D_X$. **(a)** $D_Y$ encourages $G$ to translate $X$ into outputs indistinguishable from domain $Y$, and vice versa for $D_X$ and $F$.\n\n* To further regularize the mappings, we introduce two cycle consistency losses that capture the intuition that if\nwe translate from one domain to the other and back again we should arrive at where we started. **(b)** Forward cycle-consistency loss and **(c)** backward cycle-consistency loss.\n\n## Least Squares GANs\n\nWe've seen that regular GANs treat the discriminator as a classifier with the sigmoid cross entropy loss function. However, this loss function may lead to the vanishing gradients problem during the learning process. To overcome such a problem, we'll use a least squares loss function for the discriminator. This structure is also referred to as a least squares GAN or LSGAN, and you can [read the original paper on LSGANs, here](https://arxiv.org/pdf/1611.04076.pdf). The authors show that LSGANs are able to generate higher quality images than regular GANs and that this loss type is a bit more stable during training! \n\n### Discriminator Losses\n\nThe discriminator losses will be mean squared errors between the output of the discriminator, given an image, and the target value, 0 or 1, depending on whether it should classify that image as fake or real. For example, for a *real* image, `x`, we can train $D_X$ by looking at how close it is to recognizing and image `x` as real using the mean squared error:\n\n```\nout_x = D_X(x)\nreal_err = torch.mean((out_x-1)**2)\n```\n\n### Generator Losses\n\nCalculating the generator losses will look somewhat similar to calculating the discriminator loss; there will still be steps in which you generate fake images that look like they belong to the set of $X$ images but are based on real images in set $Y$, and vice versa. You'll compute the \"real loss\" on those generated images by looking at the output of the discriminator as it's applied to these _fake_ images; this time, your generator aims to make the discriminator classify these fake images as *real* images. \n\n#### Cycle Consistency Loss\n\nIn addition to the adversarial losses, the generator loss terms will also include the **cycle consistency loss**. This loss is a measure of how good a reconstructed image is, when compared to an original image. \n\nSay you have a fake, generated image, `x_hat`, and a real image, `y`. You can get a reconstructed `y_hat` by applying `G_XtoY(x_hat) = y_hat` and then check to see if this reconstruction `y_hat` and the orginal image `y` match. For this, we recommed calculating the L1 loss, which is an absolute difference, between reconstructed and real images. You may also choose to multiply this loss by some weight value `lambda_weight` to convey its importance.\n\n\nThe total generator loss will be the sum of the generator losses and the forward and backward cycle consistency losses.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"criterion_Idt = nn.L1Loss()\ncriterion_gan = nn.MSELoss()\ndevice='cuda' if torch.cuda.is_available() else 'cpu'\n#define loss functions which are helpful\ndef real_mse_loss(D_out):\n    # how close is the produced output from being \"real\"?\n    target = torch.ones(D_out.shape,dtype=torch.float32).to(device)\n    return criterion_gan(D_out, target)\n\ndef fake_mse_loss(D_out):\n    # how close is the produced output from being \"false\"?\n    target = torch.zeros(D_out.shape, dtype=torch.float32).to(device)\n    return criterion_gan(D_out , target)\n    \n\ndef cycle_consistency_loss(real_im, reconstructed_im, lambda_weight):\n    # calculate reconstruction loss \n    # return weighted loss\n    reconstruct_loss = torch.mean(torch.abs(real_im-reconstructed_im))\n    return reconstruct_loss*lambda_weight\n\ndef identity_loss(lambda_idt , idt_B ,real_B , idt_A , real_A , lambda_weight):\n    \n    if lambda_idt > 0:\n        # G_A should be identity if real_B is fed: ||G_A(B) - B||\n        loss_idt_A = criterion_Idt(idt_B, real_B) * lambda_weight * lambda_idt\n        # G_B should be identity if real_A is fed: ||G_B(A) - A||\n        loss_idt_B = criterion_Idt(idt_A, real_A) * lambda_weight * lambda_idt\n    else:\n        self.loss_idt_A = 0\n        self.loss_idt_B = 0\n        \n    return loss_idt_A + loss_idt_B","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define the optimizers\n\n![](https://static.wixstatic.com/media/884a24_abb5ff293e4f4067bbf4b8618e5190ae~mv2.png)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"#define the optimizers\nimport torch.optim as optim\nfrom torch.optim.lr_scheduler import ReduceLROnPlateau\n\n# hyperparams for Adam optimizers\nlr=0.0001\nbeta1=0.5\nbeta2= 0.999\n\ng_params = list(G_X2Y.parameters()) + list(G_Y2X.parameters())  # Get generator parameters\n\n# Create optimizers for the generators and discriminators\ng_optimizer   = optim.Adam(g_params, lr, [beta1, beta2])\nd_x_optimizer = optim.Adam(D_X.parameters(), lr, [beta1, beta2])\nd_y_optimizer = optim.Adam(D_Y.parameters(), lr, [beta1, beta2])\n\nlambda1 = 3500\nlambda2 = 8500\n\n\n\"\"\"\nReduce learning rate when a metric has stopped improving. Models often benefit from reducing the learning rate by a factor of 2-10 once learning stagnates. This scheduler \nreads a metrics quantity and if no improvement is seen for a ‘patience’ number of epochs, the learning rate is reduced.\n\"\"\"\n\ng_lr_scheduler = ReduceLROnPlateau(g_optimizer, mode='min', factor=0.1, patience=1500, \n                                threshold=0.01, threshold_mode='rel', cooldown=0,\n                                   min_lr=0.000001, eps=1e-08, verbose=False)\n\n#define lr scheduler\n#g_lr_scheduler = MultiStepLR(g_optimizer, milestones=[lambda1,lambda2], gamma=0.1)\n#dx_lr_scheduler =MultiStepLR(d_x_optimizer, milestones=[lambda1,lambda2], gamma=0.1)\n#dy_lr_scheduler = MultiStepLR(d_y_optimizer, milestones=[lambda1,lambda2], gamma=0.1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"---\n\n## Training a CycleGAN\n\nWhen a CycleGAN trains, and sees one batch of real images from set $X$ and $Y$, it trains by performing the following steps:\n\n**Training the Discriminators**\n1. Compute the discriminator $D_X$ loss on real images\n2. Generate fake images that look like domain $X$ based on real images in domain $Y$\n3. Compute the fake loss for $D_X$\n4. Compute the total loss and perform backpropagation and $D_X$ optimization\n5. Repeat steps 1-4 only with $D_Y$ and your domains switched!\n\n\n**Training the Generators**\n1. Generate fake images that look like domain $X$ based on real images in domain $Y$\n2. Compute the generator loss based on how $D_X$ responds to fake $X$\n3. Generate *reconstructed* $\\hat{Y}$ images based on the fake $X$ images generated in step 1\n4. Compute the cycle consistency loss by comparing the reconstructions with real $Y$ images\n5. Repeat steps 1-4 only swapping domains\n6. Add up all the generator and reconstruction losses and perform backpropagation + optimization\n\n![](https://www.researchgate.net/profile/Hritwick_Banerjee/publication/329368817/figure/fig7/AS:717806182076420@1548149738922/A-Training-Architecture-of-CycleGAN-B-Training-Architecture-of-a-CyCADA.png)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# train the network\ndef training_loop(dataloader_X, dataloader_Y, test_dataloader_X , test_dataloader_Y, \n                  n_epochs=10000):\n    \n    print_every=50\n    checkpoint_every=1000\n    # keep track of losses over time\n    losses = []\n\n    test_iter_X = iter(dataloader_X)\n    test_iter_Y = iter(dataloader_Y)\n\n    # Get some fixed data from domains X and Y for sampling. These are images that are held\n    # constant throughout training, that allow us to inspect the model's performance.\n    fixed_X = test_iter_X.next()[0]\n    fixed_Y = test_iter_Y.next()[0]\n    fixed_X = rescale(fixed_X) # make sure to scale to a range -1 to 1\n    fixed_Y = rescale(fixed_Y)\n\n    # batches per epoch\n    iter_X = iter(dataloader_X)\n    iter_Y = iter(dataloader_Y)\n    batches_per_epoch = min(len(iter_X), len(iter_Y))\n\n    for epoch in range(1, n_epochs+1):\n        \n        \n        # Reset iterators for each epoch\n        if epoch % batches_per_epoch == 0:\n            iter_X = iter(dataloader_X)\n            iter_Y = iter(dataloader_Y)\n\n        images_X = iter_X.next()\n        images_X = rescale(images_X) # make sure to scale to a range -1 to 1\n\n        images_Y = iter_Y.next()\n        images_Y = rescale(images_Y)\n        \n        # move images to GPU if available (otherwise stay on CPU)\n        device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n        images_X = images_X.to(device)\n        images_Y = images_Y.to(device)\n\n\n        # ============================================\n        #            TRAIN THE DISCRIMINATORS\n        # ============================================\n\n        ##   First: D_X, real and fake loss components   ##\n\n        # 1. Compute the discriminator losses on real images in X domain\n        Dx_real_out = D_X(images_X)\n        Dx_real_loss = real_mse_loss(Dx_real_out)\n        \n        # 2. Generate fake images that look like domain X based on real images in domain Y\n        Gx_fake = G_Y2X(images_Y)\n        \n        # 3. Compute the fake loss for D_X on fake X\n        Dx_fake_out= D_X(Gx_fake)\n        Dx_fake_loss = fake_mse_loss(Dx_fake_out)\n        \n        # 4. Compute the total loss and perform backprop\n        d_x_loss = Dx_real_loss + Dx_fake_loss\n        #reste the optimizer\n        d_x_optimizer.zero_grad()\n        #backprop the loss\n        d_x_loss.backward()\n        #optimze the descrimintaer x\n        d_x_optimizer.step()\n\n        \n        ##   Second: D_Y, real and fake loss components   ##\n        Dy_real_out = D_Y(images_Y)\n        Dy_real_loss = real_mse_loss(Dy_real_out)\n        \n        #generate fake images looks like the domain y on the real images in domain x\n        Gy_fake = G_X2Y(images_X)\n        \n        #compute the descreminter fake loss\n        Dy_fake_out = D_Y(Gy_fake)\n        Dy_fake_loss = fake_mse_loss(Dy_fake_out)\n        \n        d_y_loss = Dy_fake_loss + Dy_real_loss\n        #reset the optimizer\n        d_y_optimizer.zero_grad()\n        #backprop the loss\n        d_y_loss.backward()\n        #optimize the model\n        d_y_optimizer.step()\n        \n\n\n        # =========================================\n        #            TRAIN THE GENERATORS\n        # =========================================\n\n        ##    First: generate fake X images and reconstructed Y images    ##\n\n        # 1. Generate fake images that look like domain X based on real images in domain Y\n        Gx_fake = G_Y2X(images_Y)\n\n        # 2. Compute the generator loss based on domain X\n        Dx_fake_out = D_X(Gx_fake)\n        Gx_fake_loss = real_mse_loss(Dx_fake_out)\n\n        # 3. Create a reconstructed y\n        Gy_fake_re =  G_X2Y(Gx_fake) \n        \n        # 4. Compute the cycle consistency loss (the reconstruction loss)\n        Gy_reconstruct_loss = cycle_consistency_loss(images_Y , Gy_fake_re , lambda_weight=10)\n\n        ##    Second: generate fake Y images and reconstructed X images    ##\n        Gy_fake = G_X2Y(images_X)\n        \n        #compute the generator fake loss on descriminator\n        Dy_fake_out = D_Y(Gy_fake)\n        Gy_fake_loss = real_mse_loss(Dy_fake_out)\n        \n        #regenerate the x images from fake y domain images\n        Gx_fake_re = G_Y2X(Gy_fake)\n        \n        #compute the cycle consistnace loss on the real x and reconstructed x images\n        Gx_reconstuct_loss = cycle_consistency_loss(images_X , Gx_fake_re , lambda_weight=10)\n        \n        #compute the identity loss\n        idt_B = G_X2Y(images_Y)\n        idt_A = G_Y2X(images_X)\n        id_loss = identity_loss(0.1 , idt_B , images_Y ,\n                                idt_A , images_X , lambda_weight=8)\n\n        # 5. Add up all generator and reconstructed losses and perform backprop\n        g_total_loss = Gx_fake_loss + Gy_fake_loss + Gy_reconstruct_loss + Gx_reconstuct_loss + id_loss\n        \n        #optimize the generator model\n        g_optimizer.zero_grad()\n        #back prop the loss\n        g_total_loss.backward()\n        #optimize the generator model\n        g_optimizer.step()\n\n        \n        # Print the log info\n        if epoch % print_every == 0:\n            # append real and fake discriminator losses and the generator loss\n            losses.append((d_x_loss.item(), d_y_loss.item(), g_total_loss.item()))\n            print('Epoch [{:5d}/{:5d}] | d_X_loss: {:6.4f} | d_Y_loss: {:6.4f} | g_total_loss: {:6.4f}'.format(\n                    epoch, n_epochs, d_x_loss.item(), d_y_loss.item(), g_total_loss.item()))\n\n            \n        sample_every=1000\n        # Save the generated samples\n        if epoch % sample_every == 0:\n            G_Y2X.eval() # set generators to eval mode for sample generation\n            G_X2Y.eval()\n            save_samples(epoch, fixed_Y, fixed_X, G_Y2X, G_X2Y, batch_size=16)\n            G_Y2X.train()\n            G_X2Y.train()\n        \n        g_lr_scheduler.step(g_total_loss)\n        # uncomment these lines, if you want to save your model\n#         checkpoint_every=1000\n#         # Save the model parameters\n        #if epoch % checkpoint_every == 0:\n            #checkpoint(epoch, G_XtoY, G_YtoX, D_X, D_Y)\n\n    return losses\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Helper functions","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# helper functions for saving sample data and models\n\n# import data loading libraries\nimport os\nimport pdb\nimport pickle\nimport argparse\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n\n# import torch\nimport torch\n\n\n# numpy & scipy imports\nimport numpy as np\nimport scipy\nimport scipy.misc\n\n\ndef checkpoint(iteration, G_XtoY, G_YtoX, D_X, D_Y, checkpoint_dir='checkpoints_cyclegan'):\n    \"\"\"Saves the parameters of both generators G_YtoX, G_XtoY and discriminators D_X, D_Y.\n        \"\"\"\n    G_XtoY_path = os.path.join(checkpoint_dir, 'G_XtoY.pkl')\n    G_YtoX_path = os.path.join(checkpoint_dir, 'G_YtoX.pkl')\n    D_X_path = os.path.join(checkpoint_dir, 'D_X.pkl')\n    D_Y_path = os.path.join(checkpoint_dir, 'D_Y.pkl')\n    torch.save(G_XtoY.state_dict(), G_XtoY_path)\n    torch.save(G_YtoX.state_dict(), G_YtoX_path)\n    torch.save(D_X.state_dict(), D_X_path)\n    torch.save(D_Y.state_dict(), D_Y_path)\n\n\ndef merge_images(sources, targets, batch_size=16):\n    \"\"\"Creates a grid consisting of pairs of columns, where the first column in\n        each pair contains images source images and the second column in each pair\n        contains images generated by the CycleGAN from the corresponding images in\n        the first column.\n        \"\"\"\n    _, _, h, w = sources.shape\n    row = int(np.sqrt(batch_size))\n    merged = np.zeros([3, row*h, row*w*2])\n    for idx, (s, t) in enumerate(zip(sources, targets)):\n        i = idx // row\n        j = idx % row\n        merged[:, i*h:(i+1)*h, (j*2)*h:(j*2+1)*h] = s\n        merged[:, i*h:(i+1)*h, (j*2+1)*h:(j*2+2)*h] = t\n    merged = merged.transpose(1, 2, 0)\n    return merged\n    \n\ndef to_data(x):\n    \"\"\"Converts variable to numpy.\"\"\"\n    if torch.cuda.is_available():\n        x = x.cpu()\n    x = x.squeeze(0)\n    x = x.data.numpy()\n    x = np.transpose(x,(1,2,0))\n    x = ((x +1)*255 / (2)).astype(np.uint8) # rescale to 0-255\n    return x\n\ndef save_samples(iteration, fixed_Y, fixed_X, G_YtoX, G_XtoY, batch_size=16, sample_dir='samples_cyclegan'):\n    \"\"\"Saves samples from both generators X->Y and Y->X.\n        \"\"\"\n    # move input data to correct device\n    device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n\n    fake_X = G_YtoX(fixed_Y.unsqueeze(0).to(device))\n    fake_Y = G_XtoY(fixed_X.unsqueeze(0).to(device))\n    \n    X, fake_X = to_data(fixed_X), to_data(fake_X)\n    Y, fake_Y = to_data(fixed_Y), to_data(fake_Y)\n    \n    #merged = merge_images(X, fake_Y, batch_size)\n    #path = os.path.join(sample_dir, 'sample-{:06d}-X-Y.png'.format(iteration))\n    #scipy.misc.imsave(path, merged)\n    #print('Saved {}'.format(path))\n    viz_sample(X,fake_Y)\n    viz_sample(Y,fake_X)\n    #return fake_Y","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Tips on Training and Loss Patterns\n\nA lot of experimentation goes into finding the best hyperparameters such that the generators and discriminators don't overpower each other. It's often a good starting point to look at existing papers to find what has worked in previous experiments, I'd recommend this [DCGAN paper](https://arxiv.org/pdf/1511.06434.pdf) in addition to the original [CycleGAN paper](https://arxiv.org/pdf/1703.10593.pdf) to see what worked for them. Then, you can try your own experiments based off of a good foundation.\n\n#### Discriminator Losses\n\nWhen you display the generator and discriminator losses you should see that there is always some discriminator loss; recall that we are trying to design a model that can generate good \"fake\" images. So, the ideal discriminator will not be able to tell the difference between real and fake images and, as such, will always have some loss. You should also see that $D_X$ and $D_Y$ are roughly at the same loss levels; if they are not, this indicates that your training is favoring one type of discriminator over the other and you may need to look at biases in your models or data.\n\n#### Generator Loss\n\nThe generator's loss should start significantly higher than the discriminator losses because it is accounting for the loss of both generators *and* weighted reconstruction errors. You should see this loss decrease a lot at the start of training because initial, generated images are often far-off from being good fakes. After some time it may level off; this is normal since the generator and discriminator are both improving as they train. If you see that the loss is jumping around a lot, over time, you may want to try decreasing your learning rates or changing your cycle consistency loss to be a little more/less weighted.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"n_epochs = 10000 # keep this small when testing if a model first works, then increase it to >=1000\n\nlosses = training_loop(X_dataloader, Y_dataloader, X_dataloader, Y_dataloader, n_epochs=n_epochs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Conversion Visualization","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"test_iter_X = iter(X_dataloader)\ntest_iter_Y = iter(Y_dataloader)\n\n# Get some fixed data from domains X and Y for sampling. These are images that are held\n# constant throughout training, that allow us to inspect the model's performance.\nfixed_X = test_iter_X.next()\nfixed_Y = test_iter_Y.next()\nfixed_X = rescale(fixed_X) # make sure to scale to a range -1 to 1\nfixed_Y = rescale(fixed_Y)\nfor fix_x , fix_y in zip(fixed_X , fixed_Y):\n    save_samples(0, fix_y, fix_x, G_Y2X, G_X2Y, batch_size=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Submission data generator","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"X_set = GANdataGenerator(data_dir='../input/gan-getting-started/photo_jpg' , data_files=photo_jpg , image_size=128 )\n\n\nX_dataloader = DataLoader(X_set , batch_size=1 , shuffle=True , num_workers=0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Generate Submission File","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"!rm -rf images","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import PIL\n! mkdir ../images\n\ni = 1\nfor fix_x  in X_dataloader:\n    device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n    fake_Y = G_X2Y(fix_x.to(device))\n    \n    prediction = to_data(fake_Y)\n    im = PIL.Image.fromarray(prediction)\n    \n    im.save(\"../images/\" + str(i) + \".jpg\")\n    i += 1\n    \n    \nimport shutil\nshutil.make_archive(\"/kaggle/working/images\", 'zip', \"/kaggle/images\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.save(G_X2Y.state_dict(),\"generator_x2y.pth\")\ntorch.save(G_Y2X.state_dict(),\"generator_y2x.pth\")\ntorch.save(D_X.state_dict(),\"descriminator_x.pth\")\ntorch.save(D_Y.state_dict(),\"descriminator_y.pth\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}