{"cells":[{"metadata":{},"cell_type":"markdown","source":"### Working through chapter 8 of Deep Learning with Pytorch book."},{"metadata":{},"cell_type":"markdown","source":"If you follow through this notebook, it will present you with basic of CNN, as in the nitty gritty detail of shapes of layers, why maxpooling, creating your own model from scratch etc."},{"metadata":{},"cell_type":"markdown","source":"![](https://th.bing.com/th/id/R2dcc41f9079d1abf5883a113c0d1ca31?rik=i3uC2F68Rj8a6w&riu=http%3a%2f%2fpytorch.org%2ftutorials%2f_images%2fcifar10.png&ehk=25wY39EtQMRbUWi0278TX3wGQjI11w6Uxr732%2fd0C%2bA%3d&risl=&pid=ImgRaw)"},{"metadata":{},"cell_type":"markdown","source":"Here we will be using a simple Convolutional Neural Network to work with CIFAR 10 data. To keep things simple we would be using a subset of two classes in cifar dataset. We will be recognising between airoplane and a bird.\n\nWe will be using convolutional neural network here. For creating , we’ll resort to nn.Conv2d. \n\nAt a minimum, the arguments we provide to nn.Conv2d are the number of input features (or channels, since we’re dealingwith multichannel images: that is, more than one value per pixel), the number of output features, and the size of the kernel. For instance, for our first convolutional module,\n\nwe’ll have 3 input features per pixel (the RGB channels) and an arbitrary number of\nchannels in the output—say, 16. Also, because we are randomly initializing them, some of the features we’ll\nget, even after training, will turn out to be useless.\n\nLet’s stick to a kernel size of 3 × 3."},{"metadata":{},"cell_type":"markdown","source":"![](https://cdn-images-1.medium.com/max/1200/1*1okwhewf5KCtIPaFib4XaA.gif)"},{"metadata":{},"cell_type":"markdown","source":"It is very common to have kernel sizes that are the same in all directions, so\nPyTorch has a shortcut for this: whenever kernel_size=3 is specified for a 2D convolution, it means 3 × 3 (provided as a tuple (3, 3) in Python)"},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch.nn as nn","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"conv = nn.Conv2d(3, 16, kernel_size=3)\nconv","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### About the size of the kernel\n\nWhat do we expect to be the shape of the weight tensor? The kernel is of size 3 × 3, so we want the weight to consist of 3 × 3 parts. \n\nFor a single output pixel value, our kernel\nwould consider, say, in_ch = 3 input channels, so the weight component for a single\noutput pixel value (and by translation the invariance for the entire output channel) is\nof shape in_ch × 3 × 3. \n\nFinally, we have as many of those as we have output channels,\nhere out_ch = 16, so the complete weight tensor is out_ch × in_ch × 3 × 3, in our case\n16 × 3 × 3 × 3. \n\nThe bias will have size 16 (we haven’t talked about bias for a while for\nsimplicity, but just as in the linear module case, it’s a constant value we add to each\nchannel of the output image). Let’s verify our assumptions:"},{"metadata":{"trusted":true},"cell_type":"code","source":"conv.weight.shape, conv.bias.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Lets get the cifar triaining and validation set now\n\nRemember our final aim for getting the dataset is to recognise between a plane and a bird."},{"metadata":{"trusted":true},"cell_type":"code","source":"from torchvision import datasets","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from torchvision import transforms","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We will work on a limited dataset, recognising between an aeroplane and a bird. But even for that first we download the entire dataset:"},{"metadata":{"trusted":true},"cell_type":"code","source":"data_path = \"./\"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cifar10 = datasets.CIFAR10(data_path, train=True, download=True, transform=transforms.Compose([\n    transforms.ToTensor(),\n    transforms.Normalize((0.4915, 0.4823, 0.4468),\n                         (0.2470, 0.2435, 0.2616))\n]) )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cifar10_val = datasets.CIFAR10(data_path, train=False, download=True, transform=transforms.Compose([\n    transforms.ToTensor(),\n    transforms.Normalize((0.4915, 0.4823, 0.4468),\n                         (0.2470, 0.2435, 0.2616))\n]) )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we need a subset of the database.if you see the original dataset has labels for aeroplane and bird as 0 and 2."},{"metadata":{"trusted":true},"cell_type":"code","source":"cifar10.classes","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"we need to convert that into 0 and 1"},{"metadata":{"trusted":true},"cell_type":"code","source":"label_map = {0:0, 2:1}\nclass_names = ['airplane', 'bird']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"now lets get the dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"cifar2 = [(img, label_map[label]) for img, label in cifar10 if label in [0,2]]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cifar2_val = [(img, label_map[label]) for img, label in cifar10_val if label in [0,2]]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The data is ready to go. Lets work on our model."},{"metadata":{},"cell_type":"markdown","source":"### why should we be using Convolutional neural networks anyway\n\nWe can see how convolutions are a convenient choice for learning from images.\n\nWe have smaller models looking for local patterns whose weights are optimized across the entire image.\n\nA 2D convolution pass produces a 2D image as output, whose pixels are a weighted sum over neighborhoods of the input image. In our case, both the kernel weights and the bias conv.weight are initialized randomly, so the output image will not be particularly meaningful. As usual, we need to add the zeroth batch dimension with\nunsqueeze if we want to call the conv module with one input image, since nn.Conv2d\nexpects a B × C × H × W shaped tensor as input:\n\nB is batch size\nC is color\nH is Height\nW is Width\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"img, _ = cifar2[0]\n\noutput = conv(img.unsqueeze(0))\nimg.unsqueeze(0).shape, output.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"note the output shape here is different to input shape. Can you guess how?"},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig  = plt.figure()\nfig.add_subplot(1,2,1 )\nplt.imshow(img.permute(1,2,0))\nfig.add_subplot(1,2,2)\nplt.imshow(output[0,0].detach())\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### output shape != input shape\n\nThe fact that our output image is smaller than the input is a side effect of deciding what\nto do at the boundary of the image.\n\nApplying a convolution kernel as a weighted sum\nof pixels in a 3 × 3 neighborhood requires that there are neighbors in all directions.\n\nIf we are at i00, we only have pixels to the right of and below us. By default, PyTorch will\nslide the convolution kernel within the input picture, getting width - kernel_width +1\nhorizontal and vertical positions. For odd-sized kernels, this results in images that are one-half the convolution kernel’s width (in our case, 3//2 = 1) smaller on each side.\n\nThis explains why we’re missing two pixels in each dimension.\n"},{"metadata":{},"cell_type":"markdown","source":"### Padding the boundary\n\nHowever, PyTorch gives us the possibility of padding the image by creating ghost pixels around the border that have value zero as far as the convolution is concerned.\n\n In our case, specifying padding=1 when kernel_size=3 means i00 has an extra set\nof neighbors above it and to its left, so that an output of the convolution can be computed even in the corner of our original image.\n\nThe net result is that the output has\nnow the exact same size as the input:\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"conv = nn.Conv2d(3,1,kernel_size=3, padding=1)\noutput = conv(img.unsqueeze(0))\nimg.unsqueeze(0).shape, output.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Doing padding helps us separate the matters of convolution and changing image sizes, so we have one less thing to\nremember. \n\n![](https://images.deepai.org/glossary-terms/5361a0e82d3941e58903b90f64e71491/download.png)"},{"metadata":{},"cell_type":"markdown","source":"### Detecting features with convolution\n\nWe said earlier that weight and bias are parameters that are learned through backpropagation, exactly as it happens for weight and bias in nn.Linear. \n\nHowever, we can play with convolution by setting weights by hand and see what happens.\n\nLet’s first zero out bias, just to remove any confounding factors, and then set weights to a constant value so that each pixel in the output gets the mean of its neighbors. For each 3 × 3 neighborhood"},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with torch.no_grad():\n    conv.bias.zero_()\n\nwith torch.no_grad():\n    conv.weight.fill_(1.0/ 9.0)\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### what weights and bias\n\nWe could have gone with conv.weight.one_()— that would result in each pixel in the\noutput being the sum of the pixels in the neighborhood. Not a big difference, except that the values in the output image would have been nine times larger.\n\nAnyway, let’s see the effect on our CIFAR image:"},{"metadata":{"trusted":true},"cell_type":"code","source":"try:\n    output = conv(img.unsqueeze(0))\n    plt.imshow(output[0,0])\n    plt.show()\nexcept Exception as e:\n    print(e)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So we need to call detach on the output"},{"metadata":{"trusted":true},"cell_type":"code","source":"output = conv(img.unsqueeze(0))\nplt.imshow(output[0,0].detach())\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It looks like a blob of a skeleton rising from a swamp. "},{"metadata":{"trusted":true},"cell_type":"code","source":"conv = nn.Conv2d(3,1,kernel_size=3, padding=1)\n\nwith torch.no_grad():\n    conv.weight[:] = torch.tensor([[-1.0,0.0,1.0],[-1.0,0.0,1.0],[-1.0,0.0,1.0]])\n    conv.bias.zero_()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"output = conv(img.unsqueeze(0))\nplt.imshow(output[0,0].detach())\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This convolution is supposed to enhance the vertical edges of images."},{"metadata":{},"cell_type":"markdown","source":"### Looking further with depth and pooling\n\n\nThis is all well and good, but conceptually there’s an elephant in the room. We got all excited because by moving from fully connected layers to convolutions, we achieve\nlocality and translation invariance.\n\nThen we recommended the use of small kernels, like 3 × 3, or 5 × 5: that’s peak locality, all right. What about the big picture?\n\nHow do we know that all structures in our images are 3 pixels or 5 pixels wide? Well, we don’t,\nbecause they aren’t. And if they aren’t, how are our networks going to be equipped to\nsee those patterns with larger scope?\n\nThis is something we’ll really need if we want to solve our birds versus airplanes problem effectively, since although CIFAR-10 images\nare small, the objects still have a (wing-)span several pixels across.\n\nOne possibility could be to use large convolution kernels. Well, sure, at the limit we\ncould get a 32 × 32 kernel for a 32 × 32 image, but we would converge to the old fully\nconnected, affine transformation and lose all the nice properties of convolution.\n\nAnother option, which is used in convolutional neural networks, is stacking one convolution after the other and at the same time downsampling the image between successive convolutions."},{"metadata":{},"cell_type":"markdown","source":"### Downsampling\n\nScaling the image by half is equivalent of taking four neighboring pixels and producing one pixel.\n\nwe could:\n\n1. Average the four pixels\n\n2. Take the maximim of th efout pixels\n\n3. Perform a strided convolution where every nth pixel is calculated"},{"metadata":{},"cell_type":"markdown","source":"### Maxpooling\n\nIntuitively, the output images form a convolution layer, especially since they are followed by an activation just like any other linear layer, they tend to have a high magnitude, where certain features corresponding to the estimated kernel are detected, such as vertical lines.\n\nBy taking the highest value we ensure that features are found to survive the downsampling"},{"metadata":{"trusted":true},"cell_type":"code","source":"pool = nn.MaxPool2d(2)\noutput = pool(img.unsqueeze(0))\n\nimg.unsqueeze(0).shape, output.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Combining convolutions and downsampling\n\nLets see how to use it on larger structures. We start by applying a set of 3 × 3 kernels on our 8 × 8\nimage, obtaining a multichannel output image of the same size. \n\nThen we scale down the output image by half, obtaining a 4 × 4 image, and apply another set of 3 × 3 kernels to it. This second set of kernels operates on a 3 × 3 neighborhood of something that has been scaled down by half, so it effectively maps back to 8 × 8 neighborhoods\nof the input. \n\nIn addition, the second set of kernels takes the output of the first set of\nkernels (features like averages, edges, and so on) and extracts additional features on\ntop of those.\n\nSo, on one hand, the first set of kernels operates on small neighborhoods on firstorder, low-level features, while the second set of kernels effectively operates on wider\nneighborhoods, producing features that are compositions of the previous features.\n\nThis is a very powerful mechanism that provides convolutional neural networks with\nthe ability to see into very complex scenes—much more complex than our 32 × 32\nimages from the CIFAR-10 dataset. "},{"metadata":{},"cell_type":"markdown","source":"### Putting it together for our network\n\nWith these building blocks in our hands, we can now proceed to build our convolutional neural network for detecting birds and airplanes. \n\nLet’s take our previous fully\nconnected model as a starting point and introduce nn.Conv2d and nn.MaxPool2d as\ndescribed previously:"},{"metadata":{"trusted":true},"cell_type":"code","source":"model = nn.Sequential(\n            nn.Conv2d(3, 16, kernel_size=3, padding=1),\n            nn.Tanh(),\n            nn.MaxPool2d(2), # converts 32 by 32 to 16 by 16\n            nn.Conv2d(16,8,kernel_size=3, padding=1),\n            nn.Tanh(),\n            nn.MaxPool2d(2),\n    \n            # something missing\n            # to be filled later\n            # we need a reshaping block here\n    \n            nn.Linear(8 * 8 * 8, 32),\n            nn.Tanh(),\n            nn.Linear(32,2)\n    \n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### looking into the convolutions\n\nYou dont need to read the explanation in this para, just scroll down where I define my own model and look at the model output to inspect the intermediate sizes.\n\nThe first convolution takes us from 3 RGB channels to 16, thereby giving the network a chance to generate 16 independent features that operate to (hopefully) discriminate low-level features of birds and airplanes. \n\nThen we apply the Tanh activation function. The resulting 16-channel 32 × 32 image is pooled to a 16-channel 16 × 16 image by the first MaxPool3d.\n\nAt this point, the downsampled image undergoes another convolution that generates an 8-channel 16 × 16 output.\n\nWith any luck, this output will consist of higher-level features. Again, we apply a Tanh activation and then pool to an\n8-channel 8 × 8 output.\n\nWhere does this end? After the input image has been reduced to a set of 8 × 8 features, we expect to be able to output some probabilities from the network that we can\nfeed to our negative log likelihood.\n\nHowever, probabilities are a pair of numbers in a\n1D vector (one for airplane, one for bird), but here we’re still dealing with multichannel 2D features.\n\nThinking back to the beginning of this chapter, we already know what we need to\ndo: turn the 8-channel 8 × 8 image into a 1D vector and complete our network with a\nset of fully connected layers:"},{"metadata":{"trusted":true},"cell_type":"code","source":"numel_list = [p.numel() for p in model.parameters()]\nsum(numel_list) , numel_list","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That’s very reasonable for a limited dataset of such small images. \n\nIn order to increase the capacity of the model, we could increase the number of output channels for the\nconvolution layers (that is, the number of features each convolution layer generates), which would lead the linear layer to increase its size as well.\n\nWe put the “Warning” note in the code for a reason. The model has zero chance of running without complaining"},{"metadata":{"trusted":true},"cell_type":"code","source":"try:\n    model(img.unsqueeze(0))\nexcept Exception as e:\n    print(e)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### The error of our ways\n\nAdmittedly, the error message is a bit obscure, but not too much so. We find references to linear in the traceback: looking back at the model, we see that only module\nthat has to have a 512 × 32 tensor is nn.Linear(512, 32), the first linear module after\nthe last convolution block.\n\nWhat’s missing there is the reshaping step from an 8-channel 8 × 8 image to a 512-\nelement, 1D vector (1D if we ignore the batch dimension, that is). \n\nThis could be achieved by calling view on the output of the last nn.MaxPool2d, but unfortunately, we\ndon’t have any explicit visibility of the output of each module when we use nn.Sequential."},{"metadata":{},"cell_type":"markdown","source":"### Subclassing nn.Module"},{"metadata":{},"cell_type":"markdown","source":"At some point in developing neural networks, we will find ourselves in a situation where\nwe want to compute something that the premade modules do not cover. Here, it is something very simple like reshaping. \n\nWhen we want to build models that do more complex things than just applying\none layer after another, we need to leave nn.Sequential for something that gives us\nadded flexibility. PyTorch allows us to use any computation in our model by subclassing nn.Module.\n\nIn order to subclass nn.Module, at a minimum we need to define a forward function that takes the inputs to the module and returns the output. This is where we define our\nmodule’s computation. \n\nWith PyTorch, if we use standard torch operations, autograd will take care of the backward pass automatically; and indeed, an nn.Module never comes with a backward.\n\nTypically, our computation will use other modules—premade like convolutions or\ncustomized. To include these submodules, we typically define them in the constructor\n__init__ and assign them to self for use in the forward function. They will, at the\nsame time, hold their parameters throughout the lifetime of our module. Note that you\nneed to call super().__init__() before you can do that (or PyTorch will remind you)."},{"metadata":{},"cell_type":"markdown","source":"### Our network as an nn.Module"},{"metadata":{"trusted":true},"cell_type":"code","source":"class Net(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.debug = True\n        self.conv1 = nn.Conv2d(3, 16, kernel_size=3, padding=1)\n        self.act1 = nn.Tanh()\n        self.pool1 = nn.MaxPool2d(2)\n        self.conv2 = nn.Conv2d(16,8, kernel_size=3, padding=1)\n        self.act2 = nn.Tanh()\n        self.pool2 = nn.MaxPool2d(2)\n        self.fc1 = nn.Linear(8*8*8, 32)\n        self.act3 = nn.Tanh()\n        self.fc2 = nn.Linear(32,2)\n    \n    def forward(self, x):\n        if (self.debug):\n            print(\"0 input shape : \", x.shape)\n            out = self.conv1(x)\n            print(\"1 after convolution : \", self.conv1, \"\\n\", out.shape, \"\\n\")\n\n            out = self.act1(out)\n            print(\"2 after activation : \", self.act1, \"\\n\", out.shape, \"\\n\")\n\n            out = self.pool1(out)\n            print(\"3 after Maxpooling : \",self.pool1, \"\\n\", out.shape, \"\\n\")\n\n            out = self.conv2(out)\n            print(\"4 after convolution : \",self.conv2, \"\\n\", out.shape, \"\\n\")\n            \n            out = self.act2(out)\n            print(\"5 after activation : \", self.act2, \"\\n\" ,out.shape, \"\\n\")\n            \n            out = self.pool2(out)\n            print(\"6 after maxpooling : \", self.pool2, \"\\n\", out.shape, \"\\n\")\n            \n            out = out.view(-1, 8 * 8 * 8)\n            print(\"> converting into out.view(-1, 8 * 8 * 8) : \", \"\\n\", out.shape, \"\\n\")\n            \n            out = self.fc1(out)\n            print(\"7 after Linear affine transformation : \", self.fc1, \"\\n\", out.shape, \"\\n\")\n            \n            out = self.act3(out)\n            print(\"8 after activation : \", self.act3, \"\\n\", out.shape, \"\\n\")\n            \n            out = self.fc2(out)\n            print(\"9 after Linear affine transformation : \", self.fc2, \"\\n\", out.shape, \"\\n\")\n            \n            return out","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = Net()\n\noutput = model(img.unsqueeze(0))\nprint(output)\n# plt.imshow(output[0,0].detach())\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Discussion on the model\n\nThe Net class is equivalent to the nn.Sequential model\nwe built earlier in terms of submodules; but by writing\nthe forward function explicitly, we can manipulate the\noutput of self.pool3 directly and call view on it to turn\nit into a B × N vector.\n\nNote that we leave the batch dimension as –1 in the call to view, since in principle we\ndon’t know how many samples will be in the batch.\n\nWe are making some ad hoc choices about what information to present where.\nRecall that the goal of classification networks typically is to compress information in the sense that we\nstart with an image with a sizable number of pixels and compress it into (a vector of probabilities of) classes."},{"metadata":{},"cell_type":"markdown","source":"### How pytorch keeps track of parameters and submodules\n\nInterestingly, assigning an instance of nn.Module to an attribute in an nn.Module, as\nwe did in the earlier constructor, automatically registers the module as a submodule.\n\nThe submodules must be top-level attributes, not buried inside list or\ndict instances! Otherwise the optimizer will not be able to locate the submodules (and, hence, their parameters). For situations where your model\nrequires a list or dict of submodules, PyTorch provides nn.ModuleList and\nnn.ModuleDict\n"},{"metadata":{},"cell_type":"markdown","source":"### can we have arbitrary functions ?\n\nYes, We can call arbitrary methods of an nn.Module subclass. For example, for a model\nwhere training is substantially different than its use, say, for prediction, it may make\nsense to have a predict method. Be aware that calling such methods will be similar to\ncalling forward instead of the module itself—they will be ignorant of hooks, and the\nJIT does not see the module structure when using them because we are missing the\nequivalent of the __call__ bits shown in section"},{"metadata":{"trusted":true},"cell_type":"code","source":"model = Net()\n\nnumel_list = [p.numel() for p in model.parameters()]\nsum(numel_list), numel_list","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### how parameters are found \n\nWhat happens here is that the parameters() call delves into all submodules assigned\nas attributes in the constructor and recursively calls parameters() on them. No matter how nested the submodule, any nn.Module can access the list of all child parameters. By accessing their grad attribute, which has been populated by autograd, the\noptimizer will know how to change parameters to minimize the loss"},{"metadata":{},"cell_type":"markdown","source":"### The functional API\n\nPyTorch has functional counterparts for every nn module.\nBy “functional” here we mean “having no internal state”—in other words, “whose output value is solely and fully determined by the value input arguments.” \n\nIndeed, torch.nn.functional provides many functions that work like the modules we find in nn.\nBut instead of working on the input arguments and stored parameters like the module counterparts, they take inputs and parameters as arguments to the function call.\n\nFor instance, the functional counterpart of nn.Linear is nn.functional.linear,\nwhich is a function that has signature linear(input, weight, bias=None). The\nweight and bias parameters are arguments to the function.\n\nBack to our model, it makes sense to keep using nn modules for nn.Linear and\nnn.Conv2d so that Net will be able to manage their Parameters during training. However, we can safely switch to the functional counterparts of pooling and activation,\nsince they have no parameters:\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch.nn.functional as F\n\nclass Net(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.conv1 = nn.Conv2d(3,16, kernel_size=3, padding=1)\n        self.conv2 = nn.Conv2d(16, 8, kernel_size=3, padding=1)\n        self.fc1 = nn.Linear(8*8*8, 32)\n        self.fc2 = nn.Linear(32,2)\n    \n    def forward(self, x):\n        out = F.max_pool2d(torch.tanh(self.conv1(x)), 2)\n        out = F.max_pool2d(torch.tanh(self.conv2(out)), 2)\n        out = out.view(-1, 8 * 8 * 8)\n        out = torch.tanh(self.fc1(out))\n        out = self.fc2(out)\n        return out","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### To use or not to use functional API\n\nWhether to use the functional or the modular API is a decision based on style and\ntaste. When part of a network is so simple that we want to use nn.Sequential, we’re in\nthe modular realm. When we are writing our own forwards, it may be more natural to\nuse the functional interface for things that do not need state in the form of parameters."},{"metadata":{},"cell_type":"markdown","source":" So now we can make our own nn.Module if we need to, and we also have the functional API for cases when instantiating and then calling an nn.Module is overkill. This\nhas been the last bit missing to understand how the code organization works in just\nabout any neural network implemented in PyTorch."},{"metadata":{"trusted":true},"cell_type":"code","source":"model = Net()\nmodel(img.unsqueeze(0))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We got two numbers! Information flows correctly. We might not realize it right now,\nbut in more complex models, getting the size of the first linear layer right is sometimes a source of frustration. We’ve heard stories of famous practitioners putting in\narbitrary numbers and then relying on error messages from PyTorch to backtrack the\ncorrect sizes for their linear layers. Lame, eh? Nah, it’s all legit!"},{"metadata":{},"cell_type":"markdown","source":"### Training our Convnet"},{"metadata":{},"cell_type":"markdown","source":"The core of our convnet is two nested loops: an outer one over the\nepochs and an inner one of the DataLoader that produces batches from our Dataset.\n\nIn each loop, we then have to\n\n1. Feed the inputs through the model (the forward pass).\n2. Compute the loss (also part of the forward pass).\n3. Zero any old gradients.\n4. Call loss.backward() to compute the gradients of the loss with respect to all\nparameters (the backward pass).\n5. Have the optimizer take a step in toward lower loss.\n\nAlso, we collect and print some information"},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch.optim as optim","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import datetime\n\ndef training_loop(n_epochs, optimizer, model, loss_fn, train_loader):\n    for epoch in range(1, n_epochs + 1):\n        loss_train = 0.0\n        \n        for imgs, labels in train_loader:\n            \n            outputs = model(imgs)\n            loss = loss_fn(outputs, labels) # the loss we wish to minimise\n            optimizer.zero_grad() # getting rid of gradients from last round\n            loss.backward() # performs the backward step, we computer the gradients of all parameters we want the network to learn\n            optimizer.step() # updates the model\n            loss_train += loss.item()\n            \n        if epoch == 1 or epoch %10 == 0:\n            print('{} Epoch {}, training loss {}'.format(datetime.datetime.now(), epoch, loss_train / len(train_loader)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_loader = torch.utils.data.DataLoader(cifar2, batch_size=64, shuffle=True)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = Net()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"optimizer = optim.SGD(model.parameters(), lr = 1e-2)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"loss_fn = nn.CrossEntropyLoss()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"training_loop(\n    n_epochs = 10, # change it to 100 for better values\n    optimizer = optimizer,\n    model = model,\n    loss_fn = loss_fn,\n    train_loader = train_loader\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Measuring accuracy\n\nIn order to have an accuracy more interpretable thatn the loss we can take alookat our accuracies on the training and validation datasets"},{"metadata":{"trusted":true},"cell_type":"code","source":"val_loader = torch.utils.data.DataLoader(cifar2_val, batch_size=64, shuffle=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def validate(model, train_loader, val_loader):\n    for name, loader in [(\"train\", train_loader), (\"val\", val_loader)]:\n        correct = 0\n        total = 0\n        \n        with torch.no_grad(): # since we dont want to update the parameters\n            \n            for imgs, labels in loader:\n                outputs = model(imgs)\n                _, predicted = torch.max(outputs, dim=1) # index of highest value as output\n                total += labels.shape[0] # counts the number of examples so total is increased by the batch size\n                correct += int((predicted == labels).sum())\n                # Comparing the predicted class that had the\n                # maximum probability and the ground-truth\n                # labels, we first get a Boolean array. Taking the\n                # sum gives the number of items in the batch\n                # where the prediction and ground truth agree.\n                \n            print(\"Accuracy {}: {:.2f}\".format(name, correct/total))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"validate(model, train_loader, val_loader)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### how to save your model\n\nIts pretty easy to do, we will store it in birds_vs_airplanes.pt. The thing isonyl weight is stored"},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.save(model.state_dict(), data_path + 'birds_vs_airplanes.pt')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"While loading we need to give the structure of the model."},{"metadata":{"trusted":true},"cell_type":"code","source":"loaded_model = Net()\nloaded_model.load_state_dict(torch.load(data_path + 'birds_vs_airplanes.pt'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Note about training on the GPU\n\nThere is a somewhat subtle difference between Module.to and Tensor.to.\nModule.to is in place: the module instance is modified. But Tensor.to is out of place\n(in some ways computation, just like Tensor.tanh), returning a new tensor. One\nimplication is that it is good practice to create the Optimizer after moving the parameters to the appropriate device.\n\n It is considered good style to move things to the GPU if one is available. A good\npattern is to set the a variable device depending on torch.cuda.is_available:"},{"metadata":{"trusted":true},"cell_type":"code","source":"device = (torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu'))\nprint(\"Training on \", device)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Amending the training loop to cater for GPU\n\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"def training(n_epochs, optimizer, model, loss_fn, train_loader):\n    for epoch in range(1, n_epochs + 1):\n        loss_train = 0.0\n        for imgs, labels in train_loader:\n            imgs = imgs.to(device=device)\n            labels = labels.to(device=device)\n            outputs = model(imgs)\n            loss = loss_fn(outputs, labels)\n            \n            optimizer.zero_grad()\n            loss.backward()\n            optimizer.step()\n            \n            loss_train += loss.item()\n        \n        if epoch == 1 or epoch % 10 == 0:\n            print( '{} epoch {}, training loss {}'.format(\n                datetime.datetime.now(), epoch, loss_train / len(train_loader)\n            ))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_loader = torch.utils.data.DataLoader(cifar2, batch_size=64, shuffle=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = Net().to(device=device)\noptimizer = optim.SGD(model.parameters(), lr=1e-2)\nloss_fn = nn.CrossEntropyLoss()\n\ntry:\n    training_loop(\n        n_epochs = 100,\n        optimizer = optimizer,\n        model = model,\n        loss_fn = loss_fn,\n        train_loader= train_loader\n    )\nexcept Exception as e:\n    print(e)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"https://stackoverflow.com/questions/59013109/runtimeerror-input-type-torch-floattensor-and-weight-type-torch-cuda-floatte"},{"metadata":{},"cell_type":"markdown","source":"### Model design\n\nWe can add the width of the network or the number of neurons per layer. we can increase te number of channels starting from first convolution"},{"metadata":{"trusted":true},"cell_type":"code","source":"class NetWidth(nn.Module):\n    def __init__(self):\n        super().__init__()\n        self.conv1 = nn.Conv2d(3, 32, kernel_size=3, padding=1)\n        self.conv2 = nn.Conv2d(32, 16, kernel_size=3, padding=1)\n        self.fc1 = nn.Linear(16 * 8 * 8, 32)\n        self.fc2 = nn.Linear(32,2)\n        \n    def forward(self, x):\n        out = F.max_pool2d(torch.tanh(self.conv1(x)), 2)\n        out = F.max_pool2d(torch.tanh(Self.conv2(out)), 2)\n        out = out.view(-1, 16 * 8 * 8)\n        out = torch.tanh(self.fc1(out))\n        out = self.fc2(out)\n        return out","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can easily pass a parameter to init and parameterize the width taking care to also paramterize the call to view in the forward function"},{"metadata":{"trusted":true},"cell_type":"code","source":"class Netwidth(nn.Module):\n    def __init__(self, n_chans1=32):\n        self.n_chans1 = n_chans1\n        self.conv1 = nn.Conv2d(3, n_chans1, kernel_size=3, padding=1)\n        self.conv2 = nn.Conv2d(n_chans1, n_chans1 // 2, kernel_size=3, padding=1)\n        self.fc1 = nn.Linear(8*8*n_chans1//2, 32)\n        self.fc2 = nn.Linear(32,2)\n    \n    def forward(self,x):\n        out = F.max_pool2d(torch.tanh(self.conv1(x)), 2)\n        out = F.max_pool2d(torch.tanh(self.conv2(out)), 2)\n        out = out.view(-1, 8*8*self.n_chans1//2)\n        out = torch.tanh(self.fc1(out))\n        out = self.fc2(out)\n        return out","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets look at the number of parameters of the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"sum(p.numel() for p in model.parameters())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"greater the capacity more variability but also might lwad to overfitting."},{"metadata":{},"cell_type":"markdown","source":"### Helping model to converge and regularize\n\n1. keeping the parameters in check\n\nAdding a regularisation termto the loss. This term is crafted so that the weights of the model tend to be small on their own, limiting how much training makes themgrow. In otherwords it is a penalty on larger weight values. \n\nThere are again 2 \n> L1 regularisation which is the sum of absolute values of all weights in the model\n\n> L2 regularisation which is the sum of squared of all the weights in the model. Also called weight decay.\n\nSo it punishes having more weights in the model. regularisation is added by using a term in the loss"},{"metadata":{"trusted":true},"cell_type":"code","source":"def training_loop_l2reg(n_epochs,optimizer, model, loss_fn, train_loader):\n    for epoch in range(1, n_epochs+1):\n        loss_train = 0.0\n        for imgs, labels in train_loader:\n            imgs = imgs.to(device=device)\n            labels = labels.to(device=device)\n            outputs = model(imgs)\n            loss = loss_fn(outputs, labels)\n            \n            l2_lambda = 0.001\n            l2_norm = sum(p.pow(2.0).sum() for p in model.parameters())\n            loss = loss + l2_lambda * l2_norm\n            optimizer.zero_grad()\n            loss.backward()\n            optimizer.step()\n            \n            loss_train += loss.item()\n        \n        if epoch ==1 or epoch %10 == 0:\n            print('{} Epoch {}, Training loss {}'.format(datetime.datetime.now(), epoch, loss_train /len(train_loader)))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"it is worth noting that the SGD optimizer in PyTorch already has a weight_decay parameter that\ncorresponds to 2 * lambda, and it directly performs weight decay during the update\nas described previously. "},{"metadata":{},"cell_type":"markdown","source":"### using Dropout without relying on a single input\n\nZero out a random fraction of outputs from neurons accross network where randomisation happens at each iteration."},{"metadata":{"trusted":true},"cell_type":"code","source":"class NetDropout(nn.Module):\n    def __init__(self, n_chans1=32, debug=False):\n        self.debug = debug\n        super().__init__()\n        self.n_chans1 = n_chans1\n        self.conv1 = nn.Conv2d(3, n_chans1, kernel_size=3, padding=1)\n        self.conv1_dropout = nn.Dropout3d(p=0.4)\n        self.conv2 = nn.Conv2d(n_chans1, n_chans1//2, kernel_size=3, padding=1)\n        self.conv2_dropout = nn.Dropout2d(p=0.4)\n        self.fc1 = nn.Linear(8*8*n_chans1//2, 32)\n        self.fc2 = nn.Linear(32,2)\n    \n    def forward(self, x):\n        if (self.debug == False ):\n            out = F.max_pool2d(torch.tanh(self.conv1(x)), 2)\n            out = self.conv1_dropout(out)\n            out = F.max_pool2d(torch.tanh(self.conv2(out)), 2)\n            out = self.conv2_dropout(out)\n            out = out.view(-1,8*8*self.n_chans1//2)\n            out = torch.tanh(self.fc1(out))\n            out = self.fc2(out)\n            return out\n        \n        else:\n            print(\"X shape: \", x.shape, \"\\n\")\n            out = F.max_pool2d(torch.tanh(self.conv1(x)), 2)\n            print(\"After 1st convolution: \", out.shape)\n            out = self.conv1_dropout(out)\n            print(\"After 1st dropout: \", out.shape)\n            out = F.max_pool2d(torch.tanh(self.conv2(out)), 2)\n            print(\"After 2nd convolution: \", out.shape)\n            out = self.conv2_dropout(out)\n            print(\"After 2nd dropout: \", out.shape)\n            out = out.view(-1,8*8*self.n_chans1//2)\n            print(\"After flattening: \", out.shape)\n            out = torch.tanh(self.fc1(out))\n            print(\"After 1st linear: \", out.shape)\n            out = self.fc2(out)\n            print(\"result shape: \", out.shape)\n            return out\n            ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = NetDropout(debug=True)\n\noutput = model(img.unsqueeze(0))\nprint(output)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Note that dropout is only active during training and it is taken as zero in all subsequent run. This is controlled through the train property of the dropout \n\nthis is achieved through `model.train()` and `model.eval()`\non any of the nn.Model subclass."},{"metadata":{},"cell_type":"markdown","source":"### Batch Normalisation for keeping activations in check\n\n\nMain idea is to rescale the inputs to the activations of the network so that minibatches have a certain desirable distibution. In practice it shifts and scales an intermediate input using the mean and standard deviation collected at that intermediate location over the samples of minibatch.\n\n Batch normalization in PyTorch is provided through the nn.BatchNorm1D,\nnn.BatchNorm2d, and nn.BatchNorm3d modules, depending on the dimensionality of\nthe input. Since the aim for batch normalization is to rescale the inputs of the activations, the natural location is after the linear transformation (convolution, in this case)\nand the activation, as shown here:"},{"metadata":{"trusted":true},"cell_type":"code","source":"class NetBatchNorm(nn.Module):\n    def __init__(self, n_chans1=32):\n        super().__init__()\n        self.n_chans1 = n_chans1\n        self.conv1 = nn.Conv2d(3, n_chans1, kernel_size=3, padding=1)\n        self.conv1_batchnorm = nn.BatchNorm2d(num_features=n_chans1)\n        self.conv2 = nn.Conv2d(n_chans1, n_chans1 // 2, kernel_size=3, padding=1)\n        self.conv2_batchnorm = nn.BatchNorm2d(num_features=n_chans1//2)\n        self.fc1 = nn.Linear(8*8*n_chans1//2, 32)\n        self.fc2 = nn.Linear(32,2)\n    \n    def forward(self, x):\n        print(\"input shape \",x.shape)\n        out = self.conv1(x)\n        print(\"before batch normalisation \" , out.shape)\n        out = self.conv1_batchnorm(out)\n        print(\"after batch normalisation \", out.shape)\n        out = F.max_pool2d(torch.tanh(out), 2)\n        print(\"after max pooling over activations \", out.shape)\n        out = self.conv2_batchnorm(self.conv2(out))\n        print(\"after convolution and batchnorm \", out.shape)\n        out = F.max_pool2d(torch.tanh(out), 2)\n        print(\"after maxpooling \", out.shape)\n        out = out.view(-1, 8*8*self.n_chans1 //2)\n        print(\"flattening \", out.shape)\n        out = torch.tanh(self.fc1(out))\n        print(\"activation over linear 1 \",out.shape)\n        out =self.fc2(out)\n        print(\"activation over linear 2 \",out.shape)\n        return out","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = NetBatchNorm()\n\noutput = model(img.unsqueeze(0))\nprint(output)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"similar to dropout batch normalisation also behaves differnetly "},{"metadata":{},"cell_type":"markdown","source":"## Going deeper and learning more complex structures : Depth\n\nThere’s another way to think about depth: increasing depth is related to increasing\nthe length of the sequence of operations that the network will be able to perform\nwhen processing input\n\nDepth comes with some additional challenges, which prevented deep learning models\nfrom reaching 20 or more layers until late 2015. Adding depth to a model generally\nmakes training harder to converge. \n\nThe derivatives of the loss function with respect to\nthe parameters, especially those in early layers, need to be multiplied by a lot of other\nnumbers originating from the chain of derivative operations between the loss and the\nparameter. Those numbers being multiplied could be small, generating ever-smaller\nnumbers, or large, swallowing smaller numbers due to floating-point approximation.\nThe bottom line is that a long chain of multiplications will tend to make the contribution of the parameter to the gradient vanish, leading to ineffective training of that layer\nsince that parameter and others like it won’t be properly updated.\n\n\n"},{"metadata":{},"cell_type":"markdown","source":"### Skip connections\n\nA skip connection is nothing but\nthe addition of the input to the output\nof a block of layers. This is exactly how\nit is done in PyTorch. Let’s add one\nlayer to our simple convolutional\nmodel, and let’s use ReLU as the activation for a change. The vanilla module with an extra layer looks like this"},{"metadata":{"trusted":true},"cell_type":"code","source":"class NetRes(nn.Module):\n    def __init__(self, n_chans1=32):\n        super().__init__()\n        self.n_chans1 = n_chans1\n        self.conv1 = nn.Conv2d(3, n_chans1, kernel_size=3, padding=1)\n        self.conv2 = nn.Conv2d(n_chans1, n_chans1 // 2, kernel_size=3, padding=1)\n        self.conv3 = nn.Conv2d(n_chans1//2, n_chans1 // 2, kernel_size=3, padding=1)\n        self.fc1 = nn.Linear(4*4*n_chans1//2, 32)\n        self.fc2 = nn.Linear(32, 2)\n    \n    def forward(self, x):\n        out = F.max_pool2d(torch.relu(self.conv1(x)), 2)\n        out = F.max_pool2d(torch.relu(self.conv2(out)), 2)\n        out1 = out\n        out = F.max_pool2d(torch.relu(self.conv3(out)) + out1, 2)\n        out = out.view(-1, 4*4*self.n_chans1//2)\n        out = torch.relu(self.fc1(out))\n        return out\n        ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = NetRes()\n\noutput = model(img.unsqueeze(0))\nprint(output)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In other words, we’re using the output of the first activations as inputs to the last, in\naddition to the standard feed-forward path. This is also referred to as identity mapping.\nSo, how does this alleviate the issues with vanishing gradients we were mentioning\nearlier?\n Thinking about backpropagation, we can appreciate that a skip connection, or a\nsequence of skip connections in a deep network, creates a direct path from the deeper\nparameters to the loss. This makes their contribution to the gradient of the loss more\ndirect, as partial derivatives of the loss with respect to those parameters have a chance\nnot to be multiplied by a long chain of other operations."},{"metadata":{"trusted":true},"cell_type":"code","source":"class ResBlock(nn.Module):\n    def __init__(self, n_chans):\n        super(ResBlock, self).__init__()\n        self.conv = nn.Conv2d(n_chans, n_chans, kernel_size=3, padding=1, bias=False)\n        # bias is false because Batch norm will cancel the effect of bias\n        self.batch_norm = nn.BatchNorm2d(num_features=n_chans)\n        torch.nn.init.kaiming_normal_(self.conv.weight, nonlinearity='relu')\n        torch.nn.init.constant_(self.batch_norm.weight, 0.5)\n        torch.nn.init.zeros_(self.batch_norm.bias)\n        # Uses custom initializations\n        # . kaiming_normal_ initializes with\n        # normal random elements with standard\n        # deviation as computed in the ResNet paper.\n        # The batch norm is initialized to produce output\n        # distributions that initially have 0 mean and 0.5 varianc\n    \n    def forward(self, x):\n        out = self.conv(x)\n        out = self.batch_norm(out)\n        return out + x","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In init we create nn.sequential containing a list of nn.sequential containing a list of RsBlock instances. nn.Sequentil will ensure that the output of one block is used as the input to the next. It will also ensure that allthe parameters in the block are visibele to Net.\n\nthen in forwardwe traverse it 100 times"},{"metadata":{"trusted":true},"cell_type":"code","source":"class NetResDeep(nn.Module):\n    def __init__(self, n_chans1=32, n_blocks=10):\n        super().__init__()\n        self.n_chans1 = n_chans1\n        self.conv1 = nn.Conv2d(3, n_chans1, kernel_size=3,padding=1)\n        self.resblocks = nn.Sequential(\n            *(n_blocks * [ResBlock(n_chans=n_chans1)])\n        )\n        self.fc1 = nn.Linear( 8*8*n_chans1, 32)\n        self.fc2 = nn.Linear(32, 2)\n        \n    def forward(self, x):\n        out = F.max_pool2d(torch.relu(self.conv1(x)), 2)\n        out = self.resblocks(out)\n        out = F.max_pool2d(out, 2)\n        out = out.view(1, 8*8*self.n_chans1)\n        out = torch.relu(self.fc1(out))\n        out = self.fc2(out)\n        return out","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = NetResDeep()\noutput = model(img.unsqueeze(0))\nprint(output)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### slowed down convergence\n\nIn the implementation, we parameterize the actual number of layers, which is important for experimentation and reuse. Also, needless to say, backpropagation will work as\nexpected. Unsurprisingly, the network is quite a bit slower to converge.\n\ne. This is why we used more-detailed initializations and trained\nour NetRes with a learning rate of 3e – 3 instead of the 1e – 2 we used for the other\nnetworks. We trained none of the networks to convergence, but we would not have\ngotten anywhere without these tweaks."},{"metadata":{},"cell_type":"markdown","source":"### Initialisation\n\nInitialisation is a running problem with training neural network in pytorch. We found that our model did not converge and looked\nat what people commonly choose as initialization (a smaller variance in weights; and\nzero mean and unit variance outputs for batch norm), and then we halved the output\nvariance in the batch norm when the network would not converge."},{"metadata":{},"cell_type":"markdown","source":"## Conclusion\n\n In this chapter, we have built reasonable, working models in PyTorch that can\nlearn from images. We did it in a way that helped us build our intuition around convolutional networks. We also explored ways in which we can make our models wider and\ndeeper, while controlling effects like overfitting. Although we still only scratched the\nsurface, we have taken another significant step ahead from the previous chapter. We\nnow have a solid basis for facing the challenges we’ll encounter when working on\ndeep learning projects.`"},{"metadata":{},"cell_type":"markdown","source":"## extra bits : Exercises"},{"metadata":{},"cell_type":"markdown","source":"Change our model to use a 5 × 5 kernel with kernel_size=5 passed to the\nnn.Conv2d constructor."},{"metadata":{"trusted":true},"cell_type":"code","source":"class NetDropoutMod(nn.Module):\n    def __init__(self, n_chans1=64,k_size=5,k_pad=2, debug=False):\n        self.debug = debug\n        super().__init__()\n        self.n_chans1 = n_chans1\n        self.conv1 = nn.Conv2d(3, n_chans1, kernel_size=k_size, padding=k_pad)\n        self.conv1_dropout = nn.Dropout3d(p=0.4)\n        self.conv2 = nn.Conv2d(n_chans1, n_chans1//2, kernel_size=k_size, padding=k_pad)\n        self.conv2_dropout = nn.Dropout2d(p=0.4)\n        self.fc1 = nn.Linear(8*8*n_chans1//2, 32)\n        self.fc2 = nn.Linear(32,2)\n    \n    def forward(self, x):\n        if (self.debug == False ):\n            out = F.max_pool2d(torch.tanh(self.conv1(x)), 2)\n            out = self.conv1_dropout(out)\n            out = F.max_pool2d(torch.tanh(self.conv2(out)), 2)\n            out = self.conv2_dropout(out)\n            out = out.view(-1,8*8*self.n_chans1//2)\n            out = torch.tanh(self.fc1(out))\n            out = self.fc2(out)\n            return out\n        \n        else:\n            print(\"X shape: \", x.shape, \"\\n\")\n            out = F.max_pool2d(torch.tanh(self.conv1(x)), 2)\n            print(\"After 1st convolution: \", self.conv1, \" : \", out.shape)\n            out = self.conv1_dropout(out)\n            print(\"After 1st dropout: \", out.shape)\n            out = F.max_pool2d(torch.tanh(self.conv2(out)), 2)\n            print(\"After 2nd convolution: \", self.conv2, \" : \", out.shape)\n            out = self.conv2_dropout(out)\n            print(\"After 2nd dropout: \", out.shape)\n            out = out.view(-1,8*8*self.n_chans1//2)\n            print(\"After flattening: \", out.shape)\n            out = torch.tanh(self.fc1(out))\n            print(\"After 1st linear: \", out.shape)\n            out = self.fc2(out)\n            print(\"result shape: \", out.shape)\n            return out\n            ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = NetDropoutMod(debug=True)\n\noutput = model(img.unsqueeze(0))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"a What impact does this change have on the number of parameters in the\nmodel?"},{"metadata":{"trusted":true},"cell_type":"code","source":"sum([p.numel() for p in model.parameters()])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model2 = NetDropoutMod(k_size=3, k_pad=1)\noutput = model2(img.unsqueeze(0))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sum([p.numel() for p in model2.parameters()])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Parameters increase with changing the kernel_size of convolutions"},{"metadata":{"trusted":true},"cell_type":"code","source":"model = NetDropoutMod()\n\noptimizer = optim.SGD(model.parameters(), lr=1e-2)\nloss_fn = nn.CrossEntropyLoss()\n\ntraining_loop(\n    n_epochs=10,\n    optimizer = optimizer,\n    model = model,\n    loss_fn = loss_fn,\n    train_loader = train_loader\n)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"validate(model, train_loader, val_loader)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"b Does the change improve or degrade overfitting?\n\nMaybe. It is matching closely with val accuracy."},{"metadata":{},"cell_type":"markdown","source":"d Can you describe what kernel_size=(1,3) will do?"},{"metadata":{"trusted":true},"cell_type":"code","source":"class NetDropoutKernelMod(nn.Module):\n    def __init__(self, n_chans1=64,k_size=5,k_pad=2, debug=False):\n        self.debug = debug\n        super().__init__()\n        self.n_chans1 = n_chans1\n        self.conv1 = nn.Conv2d(3, n_chans1, kernel_size=(1,3), padding=1)\n        self.conv1_dropout = nn.Dropout3d(p=0.4)\n        self.conv2 = nn.Conv2d(n_chans1, n_chans1//2, kernel_size=(1,3), padding=1)\n        self.conv2_dropout = nn.Dropout2d(p=0.4)\n        self.fc1 = nn.Linear(8*8*n_chans1//2, 32)\n        self.fc2 = nn.Linear(32,2)\n    \n    def forward(self, x):\n        if (self.debug == False ):\n            out = F.max_pool2d(torch.tanh(self.conv1(x)), 2)\n            out = self.conv1_dropout(out)\n            out = F.max_pool2d(torch.tanh(self.conv2(out)), 2)\n            out = self.conv2_dropout(out)\n            out = out.view(-1,8*8*self.n_chans1//2)\n            out = torch.tanh(self.fc1(out))\n            out = self.fc2(out)\n            return out\n        \n        else:\n            print(\"X shape: \", x.shape, \"\\n\")\n            out = F.max_pool2d(torch.tanh(self.conv1(x)), 2)\n            print(\"After 1st convolution: \", self.conv1, \" : \", out.shape)\n            out = self.conv1_dropout(out)\n            print(\"After 1st dropout: \", out.shape)\n            out = F.max_pool2d(torch.tanh(self.conv2(out)), 2)\n            print(\"After 2nd convolution: \", self.conv2, \" : \", out.shape)\n            out = self.conv2_dropout(out)\n            print(\"After 2nd dropout: \", out.shape)\n            out = out.view(-1,2304)\n            print(\"After flattening: \", out.shape)\n            out = torch.tanh(self.fc1(out))\n            print(\"After 1st linear: \", out.shape)\n            out = self.fc2(out)\n            print(\"result shape: \", out.shape)\n            return out\n            ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = NetDropoutKernelMod(debug=True)\n\ntry:\n    output = model(img.unsqueeze(0))\nexcept Exception as e:\n    print(e)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"model does not work with this kernel."},{"metadata":{},"cell_type":"markdown","source":"2. Finding an image and manually feeding` it into the model.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"img, _ = cifar2[1001]\n\noutput = conv(img.unsqueeze(0))\nimg.unsqueeze(0).shape, output.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig  = plt.figure()\nfig.add_subplot(1,2,1 )\nplt.imshow(img.permute(1,2,0))\nfig.add_subplot(1,2,2)\nplt.imshow(output[0,0].detach())\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"output = model(img.unsqueeze(0))\nprint(output)\n\n# aeroplane","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img, _ = cifar2[3001]\n\noutput = conv(img.unsqueeze(0))\nimg.unsqueeze(0).shape, output.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"fig  = plt.figure()\nfig.add_subplot(1,2,1 )\nplt.imshow(img.permute(1,2,0))\nfig.add_subplot(1,2,2)\nplt.imshow(output[0,0].detach())\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"output = model(img.unsqueeze(0))\nprint(output)\n\n# bird","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}