{"cells":[{"metadata":{"_uuid":"1a991e7f539673da033393d34767c7335318ab9a"},"cell_type":"markdown","source":"**Agenda:**\n
\nFor this tutorial in Deep Learning(DL) with Pytorch, we are going to explore Multi Layered Perceptron architecture and learn Pytorch by implementing algorithms under a certain usecase.We will cover the following:\n1. Deep Learning basics with Pytorch\n2. Multilayered Perceptron (MLP) implemention on MNIST\n
\n\n[Kaggle Kernel to run this notebook](https://www.kaggle.com/u6yuvi/dl-with-pytorch-mnist-classification?scriptVersionId=9612691)\n\nLets get started !!\n"},{"metadata":{"_uuid":"8f02da80b362a4233b75cb0f9e9656525e37befa"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"6145a827010b47e713d5bcdb6f89d8042040d75f"},"cell_type":"markdown","source":"# **1. Deep Learning basics with Pytorch**\n
\nIn this part we will cover the following:\n1. Learn to play with tensors on numpy and pytorch \n2. Learn to build a simple feed forward network from scratch with random data \n3. Learn to build an end to end MLP for MNIST dataset"},{"metadata":{"_uuid":"22b42f1fa9df82528f39a3e2c6e0f965eaa69804"},"cell_type":"markdown","source":"## Imports"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(\"List of files\",os.listdir(\"../input\"))\nimport torch\nimport numpy as np\nprint(\"Torch Version:\",torch.__version__)\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1e8017c3ed94f083df4e2bf071f7f5f422c40ca1"},"cell_type":"markdown","source":"## Helper Functions"},{"metadata":{"_uuid":"8f08b918b1e2513ad7c1f382cfba39b0205aba6e","trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport numpy as np\nfrom torch import nn, optim\nfrom torch.autograd import Variable\n\n\ndef test_network(net, trainloader):\n\n criterion = nn.MSELoss()\n optimizer = optim.Adam(net.parameters(), lr=0.001)\n\n dataiter = iter(trainloader)\n images, labels = dataiter.next()\n\n # Create Variables for the inputs and targets\n inputs = Variable(images)\n targets = Variable(images)\n\n # Clear the gradients from all Variables\n optimizer.zero_grad()\n\n # Forward pass, then backward pass, then update weights\n output = net.forward(inputs)\n loss = criterion(output, targets)\n loss.backward()\n optimizer.step()\n\n return True\n\n\ndef imshow(image, ax=None, title=None, normalize=True):\n \"\"\"Imshow for Tensor.\"\"\"\n if ax is None:\n fig, ax = plt.subplots()\n image = image.numpy().transpose((1, 2, 0))\n\n if normalize:\n mean = np.array([0.485, 0.456, 0.406])\n std = np.array([0.229, 0.224, 0.225])\n image = std * image + mean\n image = np.clip(image, 0, 1)\n\n ax.imshow(image)\n ax.spines['top'].set_visible(False)\n ax.spines['right'].set_visible(False)\n ax.spines['left'].set_visible(False)\n ax.spines['bottom'].set_visible(False)\n ax.tick_params(axis='both', length=0)\n ax.set_xticklabels('')\n ax.set_yticklabels('')\n\n return ax\n\n\ndef view_recon(img, recon):\n ''' Function for displaying an image (as a PyTorch Tensor) and its\n reconstruction also a PyTorch Tensor\n '''\n\n fig, axes = plt.subplots(ncols=2, sharex=True, sharey=True)\n axes[0].imshow(img.numpy().squeeze())\n axes[1].imshow(recon.data.numpy().squeeze())\n for ax in axes:\n ax.axis('off')\n ax.set_adjustable('box-forced')\n\ndef view_classify(img, ps, version=\"MNIST\"):\n ''' Function for viewing an image and it's predicted classes.\n '''\n ps = ps.data.numpy().squeeze()\n\n fig, (ax1, ax2) = plt.subplots(figsize=(6,9), ncols=2)\n ax1.imshow(img.resize_(1, 28, 28).numpy().squeeze())\n ax1.axis('off')\n ax2.barh(np.arange(10), ps)\n ax2.set_aspect(0.1)\n ax2.set_yticks(np.arange(10))\n if version == \"MNIST\":\n ax2.set_yticklabels(np.arange(10))\n elif version == \"Fashion\":\n ax2.set_yticklabels(['T-shirt/top',\n 'Trouser',\n 'Pullover',\n 'Dress',\n 'Coat',\n 'Sandal',\n 'Shirt',\n 'Sneaker',\n 'Bag',\n 'Ankle Boot'], size='small');\n ax2.set_title('Class Probability')\n ax2.set_xlim(0, 1.1)\n\n plt.tight_layout()\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cc2337505994f4530913183436b9f4bf8a118599"},"cell_type":"markdown","source":"## Tensors\nIt turns out neural network computations are just a bunch of linear algebra operations on *tensors*, a generalization of matrices. A vector is a 1-dimensional tensor, a matrix is a 2-dimensional tensor, an array with three indices is a 3-dimensional tensor (RGB color images for example). The fundamental data structure for neural networks are tensors and PyTorch (as well as pretty much every other deep learning framework) is built around tensors.\n\n![](\"images/tensor_examples.svg\")
\n"},{"metadata":{"_uuid":"1afe587702c9d925968dc34d4e7f515b700d9089"},"cell_type":"markdown","source":" ## Numpy to Torch and back\n\nPyTorch has a great feature for converting between Numpy arrays and Torch tensors. Let us see how easy it is to switch between the two\n\n### Ceate a tensor using numpy array"},{"metadata":{"_uuid":"35d8b903e1564ee833e2d0ed2ea00d40b61aa16e","trusted":true},"cell_type":"code","source":"np_array=np.random.randn(5,3)\nprint(f' Numpy array:\\n {np_array}')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1d82db999b723fa737510352cf47d96a62b8d63b"},"cell_type":"markdown","source":"### Convert to torch tensor"},{"metadata":{"_uuid":"1cbd6fd48cf54645f924d354fc9fb5856bf1ddc1","trusted":true},"cell_type":"code","source":"torch_tensor=torch.from_numpy(np_array)\nprint(f'Torch tensor:\\n {torch_tensor}')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"31fe8979751ba061cc81055bbc5079bed1cb0124"},"cell_type":"markdown","source":"### Convert back to numpy array"},{"metadata":{"_uuid":"e72bac0d8c4870f97b1d1724c0bce3fd84b50450","trusted":true},"cell_type":"code","source":"torch_tensor.numpy()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"532f4e612d823389740276fdb2f1d0c9d69cb78d"},"cell_type":"markdown","source":"***An important thing to note here is memory is shared between the Numpy array and Torch tensor, so if you change the values in-place of one object, the other will change as well.* \nLet see what does it mean**"},{"metadata":{"_uuid":"7a08bc968bbb69b18d7d376748f7a5333f931efe","trusted":true},"cell_type":"code","source":"# Add 2 to PyTorch Tensor, in place\ntorch_tensor.add_(2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"15c233bd4f3539d1cf8c07878d01802607a6ab89"},"cell_type":"markdown","source":"### Numpy array matches new values from Tensor"},{"metadata":{"_uuid":"3d4047f95de1ca8055e276de3246da2c6a01ccee","trusted":true},"cell_type":"code","source":"np_array","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b183396c1d82a600e1c3b1848e94e3ce5a6fbe28"},"cell_type":"markdown","source":" ## Simple Neural Network using Pytorch \n Let us see how we can use PyTorch to build a simple neural network.\n\n\nMathematically this looks like: \n\n$$\n\\begin{align}\ny &= f(w_1 x_1 + w_2 x_2 + b) \\\\\ny &= f\\left(\\sum_i w_i x_i +b \\right)\n\\end{align}\n$$\n\nWith vectors this is the dot/inner product of two vectors:\n\n$$\nh = \\begin{bmatrix}\nx_1 \\, x_2 \\cdots x_n\n\\end{bmatrix}\n\\cdot \n\\begin{bmatrix}\n w_1 \\\\\n w_2 \\\\\n \\vdots \\\\\n w_n\n\\end{bmatrix}\n$$"},{"metadata":{"_uuid":"496c485f3e7f43a27bdce042df78fa11eb296631"},"cell_type":"markdown","source":"With the basics covered, it's time to explore how we can use PyTorch to build a simple neural network."},{"metadata":{"_uuid":"93562b119ff2797b3660bb0968bc6c31e890b7e5"},"cell_type":"markdown","source":"### Generate some random data \n We will create a tensor with shape (1, 5), one row and five columns, that contains values randomly distributed according to the normal distribution with a mean of zero and standard deviation of one."},{"metadata":{"_uuid":"fd1ee538877f19c8a2cfac8ea123ecbdb2f4a94d","trusted":true},"cell_type":"code","source":"# Set the random seed so things are predictable\ntorch.manual_seed(7) \nfeatures=torch.randn(1,3)\nprint(f'Number of Inout features:{features.shape[1]}')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b0673d28508dd5eed49da6731c4cc1408f1efd5c"},"cell_type":"markdown","source":"### Initialize Weights and Biases "},{"metadata":{"_uuid":"7f39730f8dbaff0834b223744d0bab6bfc516e2c"},"cell_type":"markdown","source":"Weights = torch.randn_like(features) creates another tensor with the same shape as features, again containing values from a normal distribution.\n\nFinally, bias = torch.randn((1, 1)) creates a single value from a normal distribution."},{"metadata":{"_uuid":"3d2b5d527e0650f3d3c1942a5ee1da56bbbac7de","trusted":true},"cell_type":"code","source":"n_input=features.shape[1]\nn_hidden=2\nn_output=1\n#Weights for input to hidden layer\nW1=torch.randn(n_input,n_hidden)\nW2=torch.randn(n_hidden,n_output)\n#Bias term for hidden and output layer\nB1=torch.randn(n_hidden)\nB2=torch.randn(n_output)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cd62c805fb7363693b56ec9dbcd00a8c404adcba","trusted":true},"cell_type":"code","source":"#Using a Sigmoid Activation Function\ndef activation(x):\n return(1/1+torch.exp(-x))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fbfeec62cfa4a35cb829897b71a4450b70ec8392"},"cell_type":"markdown","source":"### Calculate Weight and Biases\nWe will calculate the output for this multi-layer network using the weights `W1` & `W2`, and the biases, `B1` & `B2`."},{"metadata":{"_uuid":"bd98acc8aa8bb1d2fe06cb77ffd4a5054c07a40b","trusted":true},"cell_type":"code","source":"h1=activation(torch.matmul(features,W1)+B1)\nprint(f'Hidden Layer activations:{h1}')\nout=activation(torch.matmul(h1,W2)+B2)\nprint(f'Output of the network:{out}')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"21b8946d94159fc86b191a82bbbb4e34d2289f53"},"cell_type":"markdown","source":"## Building our Network\nNow we're going to build a larger network that can solve a (formerly) difficult problem, identifying text in an image using MNIST data\nFor now our goal will be to build a neural network that can take one of these images and predict the digit in the image.First, let's try to build this network for this dataset using weight matrices and matrix multiplications. Then, we'll see how to do it using PyTorch's `nn` module which provides a much more convenient and powerful method for defining network architectures."},{"metadata":{"_uuid":"c4c2229e3f3534db73e8d22ba73485022765f3af"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"d2974386321060bd66905127f3a48b9f1b177310","trusted":true},"cell_type":"code","source":"# Import necessary packages\n\n%matplotlib inline\n%config InlineBackend.figure_format = 'retina'\n\nimport numpy as np\nimport torch\nimport helper\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"237092ef839e8201ecc710f53ed7154e0f865b2e"},"cell_type":"markdown","source":"### Load Dataset \nFirst up, we need to get our dataset.Right now we will be using MNIST dataset which is already in`torchvision` package. The code below will download the MNIST dataset, then create training and test datasets for us. "},{"metadata":{"_uuid":"45cde9b49e2c0e1802ff640ebab0d766233e6abe","trusted":true},"cell_type":"code","source":"from torchvision import datasets, transforms\n\n# Define a transform to normalize the data\ntransform = transforms.Compose([transforms.ToTensor(),\n transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),\n ])\n\n# Download and load the training data\ntrainset = datasets.MNIST('MNIST_data/', download=True, train=True, transform=transform)\ntrainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"00ab24a0af33038559ec91ea83458558e4d574bf"},"cell_type":"markdown","source":"We have the training data loaded into trainloader \n\nWith dataloaded we make an iterator with iter(trainloader). Later, we'll use this to loop through the dataset for training, like below:"},{"metadata":{"_uuid":"9808e8dab8f56248ae40759f20b1e59dad3ede7b","trusted":true},"cell_type":"code","source":"dataiter = iter(trainloader)\nimages, labels = dataiter.next()\nprint(type(images))\nprint(images.shape)\nprint(labels.shape)\n#Printing the size of one image\nprint(images[1].numpy().squeeze().shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f544046855f3efd0eb1d88fb11f0e4e27318d5eb","trusted":true},"cell_type":"code","source":"#Look at the image\nplt.imshow(images[1].numpy().squeeze(), cmap='Greys_r');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b11c0c543d20cfb96cbb0635eea5f93e37c58cb0","trusted":true},"cell_type":"code","source":"#Sigmoid Activation Function\ndef activation(x):\n return (1/(1+torch.exp(-x)))\n\n#Input 64x784\ninputs=images.view(images.shape[0],-1)\n#Number of input features-784\nn_input=inputs.shape[1]\n#Number of neurons in hidden layer-256\nn_hidden=256\n#Number of output neuron-10\nn_out=10\n#Weight at hidden neuron-784x256\nW1=torch.randn(n_input,n_hidden)\n#Bias at hidden neuron-256\nB1=torch.randn(n_hidden)\n#Weight at output neuron-256x10\nW2=torch.randn(n_hidden,n_out)\n#Bias at output neuron-10\nB2=torch.randn(n_out)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e494abcbcf8bf4e1210b1acdc0790ca77f826e21","trusted":true},"cell_type":"code","source":"print(\"Shape of a batch of an image:\",images.shape)\nprint(\"Shape of the input to the network:\",inputs.shape)\nprint(\"Shape of the input features:\",n_input)\nprint(\"Shape of the Weight matrix of neurons in the hidden layer\",W1.shape)\nprint(\"Shape of the Bias vector of neurons in the hidden layer\",B1.shape)\nprint(\"Shape of the Weight matrix of neurons in the output layer\",W2.shape)\nprint(\"Shape of the Bias vector of neurons in the output layer\",W2.shape)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e751026ecf95253bf33db0462ce8bc250c082bb7","trusted":true},"cell_type":"code","source":"#Hidden layer activations\nh1=activation(torch.mm(inputs,W1)+B1)\n#Output layer activations\nout=activation(torch.mm(h1,W2)+B2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2cb2a0f4a6d9852da5d4811e89734c8906bbb878","trusted":true},"cell_type":"code","source":"print(f'Shape of the Hidden activation of the network{h1.shape}')\nprint(f'Shape of the Output of the network{out.shape}')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"66671445031ee39d4142f84c832b2ab097f6d14a","trusted":true},"cell_type":"code","source":"#Let us see the network output to one of the feeded input image\nout[1]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8c7ab4cad2d8a2d619e0b376780e0d5b2dfa7be3"},"cell_type":"markdown","source":"Now we have 10 outputs for our network. This raw output is usually called **logits or scores**.\n
\nHowever,We want to pass in an image to our network and get out a probability distribution over the classes that tells us the likely class(es) the image belongs to."},{"metadata":{"_uuid":"1903f5520e57c8ca69b4252ed7a7c4d5e5beb51a"},"cell_type":"markdown","source":"\n### Probability Distribution using Softmax\nTo calculate this probability distribution, we often use the [softmax function](https://en.wikipedia.org/wiki/Softmax_function)\n$$\n\\Large \\sigma(x_i) = \\cfrac{e^{x_i}}{\\sum_k^K{e^{x_k}}}\n$$\nWhat this does is squish each input $x_i$ between 0 and 1 and normalizes the values to give you a proper probability distribution where the probabilites sum up to one."},{"metadata":{"_uuid":"f4d7ba5cc02acf1610fa113490099e51e84978aa","trusted":true},"cell_type":"code","source":"def softmax(x):\n return(torch.exp(x)/torch.sum(torch.exp(x),dim=1).view(-1,1))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bdf3ab4229b26432555148c0d1b67f6e857055e6"},"cell_type":"markdown","source":"Let us understand what we are doing above by an example\n
\nStep 1:Calculating the numerator of the softmax function"},{"metadata":{"_uuid":"d06ddfc05e73456929293906a0ee9672cf2f45a6","trusted":true},"cell_type":"code","source":"torch.exp(out[1:3])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0fd149046af107a72d73208528b3a5cf1a6eb13e"},"cell_type":"markdown","source":"Step 2:For every predicted image output, calculate the sum over the predicted values over all classes"},{"metadata":{"_uuid":"2277f07ce4566f91bf8ad70e3969f2c4790e676d","trusted":true},"cell_type":"code","source":"#print(torch.sum(torch.exp(out[1:3])))\n#Dim=1 says, we want to take the sum across all columns\ntorch.sum(torch.exp(out[1:3]),dim=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e247c285cf5dfdd02702bc79ddc8e49383bc17c1"},"cell_type":"markdown","source":"Step3:Rearrange the sums in an order for broadcasting to work"},{"metadata":{"_uuid":"8c226fe7a4a2e9f31f63ddca18ed71959d9a139d","trusted":true},"cell_type":"code","source":"torch.sum(torch.exp(out[1:3]),dim=1).view(-1,1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ecbcd104b966f3ae051d09048e7d708f992f1a41"},"cell_type":"markdown","source":"Step 3:For every predicted image output, divide the predictions of each class with the sum over all classes."},{"metadata":{"_uuid":"fce3f0e89d812e35f44a46ee5eab0fc85bf05406","trusted":true},"cell_type":"code","source":"#print(torch.exp(out[1:3])/torch.sum(torch.exp(out[1:3]),dim=1))\ntemp=torch.exp(out[1:3])/torch.sum(torch.exp(out[1:3]),dim=1).view(-1,1)\nprint(temp)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0973f2a30eacf6d699df25a24ce00f9487f35559"},"cell_type":"markdown","source":"Voila!! We got the softmax output .One last thing to do is check whether the sum across all classes sum to 1 for understanding the predicted class"},{"metadata":{"_uuid":"5f899daaaf970fbb7b821a9cf4f70af1b00a9773","trusted":true},"cell_type":"code","source":"temp.sum(dim=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"da2f00d033b1ef41cf3941998ef34e24167a08cc","trusted":true},"cell_type":"code","source":"probabilities = softmax(out)\n# Does it have the right shape? Should be (64, 10)\nprint(probabilities.shape)\n# Does it sum to 1?\n#print(probabilities.sum(dim=1))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f9cd86feb1156d1a7ba69037197aabf876735be6"},"cell_type":"markdown","source":"## Building our Network with Pytorch\n\n"},{"metadata":{"_uuid":"6b0737a96ecaa7d4bcd0f6bc4592961e5ffc8e42"},"cell_type":"markdown","source":"PyTorch provides a module `nn` that makes building networks much simpler. Here I'll show you how to build the same one as above with 784 inputs, 256 hidden units, 10 output units and a softmax output."},{"metadata":{"_uuid":"e82cac77674746147dc9dd5d9367b21c1e02d40c","trusted":true},"cell_type":"code","source":"from torch import nn","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"72a3786079aa4a387146ff066176d5af56b11b4b","trusted":true},"cell_type":"code","source":"class Network(nn.Module):\n def __init__(self):\n super().__init__()\n self.hidden=nn.Linear(784,256)\n self.output=nn.Linear(256,10)\n self.sigmoid=nn.Sigmoid()\n self.softmax=nn.Softmax(dim=1)\n \n def forward(self,x):\n x=self.hidden(x)\n x=self.sigmoid(x)\n x=self.output(x)\n x=self.softmax(x)\n return x","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b1546fd3ac7e6ab35a81dfee3e704817627a5618"},"cell_type":"markdown","source":"Let's go through this bit by bit.\n\n```python\nclass Network(nn.Module):\n```\n\nHere we're inheriting from `nn.Module`. Combined with `super().__init__()` this creates a class that tracks the architecture and provides a lot of useful methods and attributes. It is mandatory to inherit from `nn.Module` when you're creating a class for your network. The name of the class itself can be anything.\n\n```python\nself.hidden = nn.Linear(784, 256)\n```\n\nThis line creates a module for a linear transformation, $x\\mathbf{W} + b$, with 784 inputs and 256 outputs and assigns it to `self.hidden`. The module automatically creates the weight and bias tensors which we'll use in the `forward` method. You can access the weight and bias tensors once the network once it's create at `net.hidden.weight` and `net.hidden.bias`.\n\n```python\nself.output = nn.Linear(256, 10)\n```\n\nSimilarly, this creates another linear transformation with 256 inputs and 10 outputs.\n\n```python\nself.sigmoid = nn.Sigmoid()\nself.softmax = nn.Softmax(dim=1)\n```\n\nHere I defined operations for the sigmoid activation and softmax output. Setting `dim=1` in `nn.Softmax(dim=1)` calculates softmax across the columns.\n\n```python\ndef forward(self, x):\n```\n\nPyTorch networks created with `nn.Module` must have a `forward` method defined. It takes in a tensor `x` and passes it through the operations you defined in the `__init__` method.\n\n```python\nx = self.hidden(x)\nx = self.sigmoid(x)\nx = self.output(x)\nx = self.softmax(x)\n```\n\nHere the input tensor `x` is passed through each operation a reassigned to `x`. We can see that the input tensor goes through the hidden layer, then a sigmoid function, then the output layer, and finally the softmax function. It doesn't matter what you name the variables here, as long as the inputs and outputs of the operations match the network architecture you want to build. The order in which you define things in the `__init__` method doesn't matter, but you'll need to sequence the operations correctly in the `forward` method.\n\nNow we can create a `Network` object."},{"metadata":{"_uuid":"36bc9a286ffafd0a0b1ac7890906092965f5fee0","trusted":true},"cell_type":"code","source":"model=Network()\nmodel","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aae8f6f5e7c42a3df7d79b6e3a20293f3a673b0a"},"cell_type":"markdown","source":"We can define the network somewhat more concisely and clearly using the `torch.nn.functional` module. This is the most common way you'll see networks defined as many operations are simple element-wise functions. We normally import this module as `F`, `import torch.nn.functional as F`.\n"},{"metadata":{"_uuid":"1ebea291b4d96d74d435885e60bc2a792c4abd12","trusted":true},"cell_type":"code","source":"import torch.nn.functional as F","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"41e688fd7486e6ac4ea420cdc23e464a4e2c3432","trusted":true},"cell_type":"code","source":"import torch.nn.functional as F\n\nclass Network(nn.Module):\n def __init__(self):\n super().__init__()\n # Inputs to hidden layer linear transformation\n self.hidden = nn.Linear(784, 128)\n # Output layer, 10 units - one for each digit\n self.output = nn.Linear(128, 10)\n \n def forward(self, x):\n # Hidden layer with sigmoid activation\n x = F.sigmoid(self.hidden(x))\n # Output layer with softmax activation\n x = F.softmax(self.output(x), dim=1)\n \n return x","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"30f4a6c129eb490f265d2570e16f86dc96534952","trusted":true},"cell_type":"code","source":"model=Network()\nmodel","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8c3820bb60313a368c0a3d8322b21b7d8adeafeb"},"cell_type":"markdown","source":"### Initializing weights and biases\n\nThe weights and bias are automatically initialized for you, but it's possible to customize how they are initialized. The weights and biases are tensors attached to the layer you defined, you can get them with `model.fc1.weight` for instance."},{"metadata":{"_uuid":"d148d7e2446d76d5ca6a8948d23ec513f103cdba","trusted":true},"cell_type":"code","source":"print(model.hidden.weight,model.hidden.weight.shape)\nprint(model.hidden.bias,model.hidden.bias.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1a1989c40df55dd4324cad2736e72481c148299d"},"cell_type":"markdown","source":"For custom initialization, we can these tensors in place."},{"metadata":{"_uuid":"dc98f7292fe8d6f40b1ea58b681acf522f500bb2","trusted":true},"cell_type":"code","source":"# Set biases to all zeros\nmodel.hidden.bias.data.fill_(0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"354f3f9fbcf8b0c6808394ea97e794b9f43ffe60","trusted":true},"cell_type":"code","source":"# sample from random normal with standard dev = 0.01\nmodel.hidden.weight.data.normal_(std=0.01)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e3b948402fbcd6e8529fe5ebcd00347f1db63a1c","trusted":true},"cell_type":"code","source":"netowrk=Network()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1eb03d730d47d46308e593ff4debbd6899280bbe"},"cell_type":"markdown","source":"### Forward pass\n\nNow that we have a network, let's see what happens when we pass in an image."},{"metadata":{"_uuid":"59df5bda3337698adad159777311294b7f8cfce4","trusted":true},"cell_type":"code","source":"# Grab some data \ndataiter = iter(trainloader)\nimages, labels = dataiter.next()\n\n# Resize images into a 1D vector, new shape is (batch size, color channels, image pixels) \nimages.resize_(images.shape[0], 1, 784)\n\n# Forward pass through the network\nimg_idx = 0\nps = model.forward(images[img_idx,:])\n\nimg = images[img_idx]\nview_classify(img.view(1, 28, 28), ps)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2103e632ea834fcbd4741de8b9ee7edace23ba6b"},"cell_type":"markdown","source":"As you can see above, our network has basically no idea what this digit is. It's because we haven't trained it yet, , all the weights are random!"},{"metadata":{"_uuid":"6d611ac89023080d6ee0e42b9369376ed261a777"},"cell_type":"markdown","source":"## Add-on-People from the keras would love this!!!\nPyTorch provides a convenient way to build networks like this where a tensor is passed sequentially through operations, `nn.Sequential`.\nLets try to build the above network using this method:"},{"metadata":{"_uuid":"e6f5405fe980380825786437c0aa8e94cb8cb92e","trusted":true},"cell_type":"code","source":"# Hyperparameters for our network\ninput_size = 784\nhidden_sizes = [128]\noutput_size = 10\n\nmodel=nn.Sequential(nn.Linear(input_size,hidden_sizes[0]),\n nn.ReLU(),\n nn.Linear(hidden_sizes[0],output_size),\n nn.ReLU(),\n nn.Softmax(dim=1))\nprint(model)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d461b698ae5973918405c9120db6e94194347a27","trusted":true},"cell_type":"code","source":"# Forward pass through the network and display output\nimages, labels = next(iter(trainloader))\nimages.resize_(images.shape[0], 1, 784)\nps = model.forward(images[0,:])\nview_classify(images[0].view(1, 28, 28), ps)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d137ebac2a3367c41e6c001f2edf2a37b0b22211"},"cell_type":"markdown","source":"### Access Layers of the network\nWe can access layers by integer "},{"metadata":{"_uuid":"0cf54881f8e022928a962d1d83fd5614a97a039e","trusted":true},"cell_type":"code","source":"print(model[0])\nmodel[0].weight","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0499ff7a787fc1885f5998f81f2395fc6bcc4108"},"cell_type":"markdown","source":"### Ordered Dict- Better way to create a network\nWe can also pass in an `OrderedDict` to name the individual layers and operations, instead of using incremental integers. Note that dictionary keys must be unique, so _each operation must have a different name_."},{"metadata":{"_uuid":"ca7567562b0610e6ec4ec7d98f03032bbe32c061","trusted":true},"cell_type":"code","source":"from collections import OrderedDict\nmodel = nn.Sequential(OrderedDict([\n ('hidden', nn.Linear(input_size, hidden_sizes[0])),\n ('relu1', nn.ReLU()),\n ('output', nn.Linear(hidden_sizes[0], output_size)),\n ('softmax', nn.Softmax(dim=1))]))\nmodel","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c67ffda9a2ace0093371017d9abf4bc02a8656dc"},"cell_type":"markdown","source":"### Access Layers using integer or name \nNow we can access layers either by integer or name"},{"metadata":{"_uuid":"6da7947d84c734eea2bf19acc3127bdf87ccce1c","trusted":true},"cell_type":"code","source":"print(model[0])\nprint(model.hidden)\nprint(model.hidden.weight)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"df59c9d06c7e7d2ec2b6582c6bb0934e2d49c58f"},"cell_type":"markdown","source":"### Recollect everything \nBefore we go ahead and train a neural network to accuractly predict the numbers appearing in the MNIST images,let us recollect the important modules that is necessary for any model training exercise"},{"metadata":{"_uuid":"f0df7ddfb84bf00cee50f9b916a9c61f1a65aaa9"},"cell_type":"markdown","source":"#### Imports"},{"metadata":{"_uuid":"ff53d2bd4296f89b134eb961832043cce0e43e4d","trusted":true},"cell_type":"code","source":"import torch\nfrom torch import nn\nimport torch.nn.functional as F\nfrom torchvision import datasets,transforms","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7a1f47e2928ea88a7c9b45edb08ea75016b6fca6"},"cell_type":"markdown","source":"#### Load Data"},{"metadata":{"_uuid":"c2c9d59a82c6bce28a611e1f409ced8b4f8fdc39","trusted":true},"cell_type":"code","source":"transform=transforms.Compose([transforms.ToTensor(),\n transforms.Normalize([0.5,0.5,0.5],[0.5,0.5,0.5])])\ntrainset=datasets.MNIST('~/.pytorch/MNIST_data/',train=True,transform=transform,download=True)\ntestset=datasets.MNIST('~/.pytorch/MNIST_data/',train=False,transform=transform,download=True)\n\ntrainloader=torch.utils.data.DataLoader(trainset,batch_size=64,shuffle=True,num_workers=0)\n#will explain later\ntestloader=torch.utils.data.DataLoader(testset,batch_size=64,shuffle=True,num_workers=0)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c7161f89857d013b0f837c620c0ed24657bed1b2"},"cell_type":"markdown","source":"#### Build a feedforward Network"},{"metadata":{"_uuid":"0e73fc284a626e4d7bbca629f817585aeb06fdd6","trusted":true},"cell_type":"code","source":"# TODO: Build a feed-forward network in one of the three ways mentioned above:\nmodel = nn.Sequential(nn.Linear(784, 128),\n nn.ReLU(),\n nn.Linear(128, 64),\n nn.ReLU(),\n nn.Linear(64, 10),\n nn.LogSoftmax(dim=1))\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dfdb486a01347095c7c8d75de1d5b74443bbfd4f"},"cell_type":"markdown","source":"#### Lets run one image through the network to check our work"},{"metadata":{"_uuid":"0864af9a75655db17435ef4940945a7d9532c671","trusted":true},"cell_type":"code","source":"# Get our data\nimages, labels = next(iter(trainloader))\n# Flatten images\nimages = images.view(images.shape[0], -1)\n\n# Forward pass, get our logits\nlogits = model(images)\nprint(logits.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0d9976b19b6ca3910a3a4855754bdc1e3de1f52f"},"cell_type":"markdown","source":"#### Define a loss function"},{"metadata":{"_uuid":"22466cb3a58aad481e903428de2642ff67e9bc1f","trusted":true},"cell_type":"code","source":"criterion = nn.NLLLoss()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2e3bda9d30daf829a8e19537715860ff7c678329","trusted":true},"cell_type":"code","source":"# Calculate the loss with the logits and the labels\nloss=criterion(logits,labels)\nprint(loss)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8a7dd6c45e94460f48dd048b35229f74a399a988"},"cell_type":"markdown","source":"## Autograd\n\nNow that we know how to calculate a loss, how do we use it to perform backpropagation? Torch provides a module, `autograd`, for automatically calculating the gradients of tensors. We can use it to calculate the gradients of all our parameters with respect to the loss. Autograd works by keeping track of operations performed on tensors, then going backwards through those operations, calculating gradients along the way.\n\nPyTorch keeps track of operations on a tensor and calculates the gradients, you need to set `requires_grad = True` on a tensor. You can do this at creation with the `requires_grad` keyword, or at any time with `x.requires_grad_(True)`.\n"},{"metadata":{"_uuid":"d3361f46b9d9dbf06bc4a6b257f472129c2b01e1"},"cell_type":"markdown","source":"Let's see an example to understand it better.Then again we will head back to our modelling task"},{"metadata":{"_uuid":"5ab21186791f7e319778106ca1ffb327d0847e16","trusted":true},"cell_type":"code","source":"x = torch.randn(2,2, requires_grad=True)\nprint(x)\ny = x**2\nprint(y)\n## grad_fn shows the function that generated this variable\nprint(y.grad_fn)\nz = y.mean()\nprint(z)\nprint(x.grad)\nz.backward()\nprint(x.grad)\nprint(x/2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"94d46a1835ad4987ff28365b28ee304d366336dc"},"cell_type":"markdown","source":"## Loss and Autograd together"},{"metadata":{"_uuid":"b0c5e229539704b93a5991c27849bd7b41abae80","trusted":true},"cell_type":"code","source":"# Build a feed-forward network\nmodel = nn.Sequential(nn.Linear(784, 128),\n nn.ReLU(),\n nn.Linear(128, 64),\n nn.ReLU(),\n nn.Linear(64, 10),\n nn.LogSoftmax(dim=1))\n\ncriterion = nn.NLLLoss()\nimages, labels = next(iter(trainloader))\nimages = images.view(images.shape[0], -1)\n\nlogits = model(images)\nloss = criterion(logits, labels)\n\n\nprint('Before backward pass: \\n', model[0].weight.grad)\n\nloss.backward()\n\nprint('After backward pass: \\n', model[0].weight.grad)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d184b346f9ed4194de74a29675da09076e52e0c4"},"cell_type":"markdown","source":"## Defining the optimizer"},{"metadata":{"_uuid":"efadf9e312ec51f9f36d25f043d011c27f4f8fe2","trusted":true},"cell_type":"code","source":"from torch import optim\noptimizer=optim.Adam(model.parameters(),lr=0.1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"91c0777ba686873bd972d96e384c090c9baae3d8"},"cell_type":"markdown","source":"## Training for real"},{"metadata":{"_uuid":"6b6ac5071aad8b3232596ed1d4807d73aab09df3","trusted":true},"cell_type":"code","source":"epochs = 5\nfor e in range(epochs):\n running_loss = 0\n for images, labels in trainloader:\n # Flatten MNIST images into a 784 long vector\n images = images.view(images.shape[0], -1)\n optimizer.zero_grad()\n output=model.forward(images)\n # TODO: Training pass\n \n loss = criterion(output,labels)\n loss.backward()\n optimizer.step()\n \n running_loss += loss.item()\n else:\n print(f\"Epoch:{e} Training loss: {running_loss/len(trainloader)}\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6d837662f725dc7196ba3d24130aa23090dc1616","trusted":true},"cell_type":"code","source":"%matplotlib inline\nimport helper\n\nimages, labels = next(iter(trainloader))\n\nimg = images[0].view(1, 784)\n# Turn off gradients to speed up this part\nwith torch.no_grad():\n logits = model.forward(img)\n\n# Output of the network are logits, need to take softmax for probabilities\nps = F.softmax(logits, dim=1)\n#helper.view_classify(img.view(1, 28, 28), ps)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"37a44994683bfe1fee884125d2e58ed08b62720b"},"cell_type":"markdown","source":"## Inference and Validation\n\nThe goal of validation is to measure the model's performance on data that isn't part of the training set. Typically this is just accuracy, the percentage of classes the network predicted correctly. Other options are precision and recall and top-5 error rate. We'll focus on accuracy here. First I'll do a forward pass with one batch from the test set."},{"metadata":{"_uuid":"e94326ca7cdc742c2c575294d260c257a97cab87"},"cell_type":"markdown","source":"### Inference on a batch of images\nLet us try to do this for a batch of images.Before that we will make some changes in our architecture"},{"metadata":{"_uuid":"e4ec78de08b2f24f5032ba181ec7a7d79428261a","trusted":true},"cell_type":"code","source":"images, labels = next(iter(testloader))\nimages.shape,labels.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3f2244405b587c87c9347abd1ed4ab150f8d80aa","trusted":true},"cell_type":"code","source":"images, labels = next(iter(testloader))\nimg = images.view(images.shape[0], 784)\n# Get the class probabilities\nps = torch.exp(model(img))\n# Make sure the shape is appropriate, we should get 10 class probabilities for 64 examples\nprint(ps.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"72a2ebdd77e3380b972e24e1565bc7744beedd27","trusted":true},"cell_type":"code","source":"top_prob,top_class=ps.topk(1,dim=1)\ntop_prob.shape,top_class.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9f6368720bf311e1ee9977f0f2562a02b1afc12f","trusted":true},"cell_type":"code","source":"top_class.view(64)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a9a05d7571a12641d0da644ae00b1608346faf6a","trusted":true},"cell_type":"code","source":"import pandas as pd\npd.DataFrame({\"Predicted\":top_class.view(top_class.shape[0]),\"Actual\":labels})","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3f958e5da0aed9b404c743c8798b64e0d4341fdd","trusted":true},"cell_type":"code","source":"equals=top_class == labels.view(*top_class.shape)\naccuracy=torch.mean(equals.type(torch.FloatTensor))\naccuracy.item()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"352673a9d93a8f6b52165952364a7793ff43b8f0","trusted":true},"cell_type":"code","source":"class Network(nn.Module):\n def __init__(self):\n super().__init__()\n self.fc1 = nn.Linear(784, 256)\n self.fc2 = nn.Linear(256, 128)\n self.fc3 = nn.Linear(128, 64)\n self.fc4 = nn.Linear(64, 10)\n\n # Dropout module with 0.2 drop probability\n self.dropout = nn.Dropout(p=0.2)\n\n def forward(self, x):\n # make sure input tensor is flattened\n x = x.view(x.shape[0], -1)\n\n # Now with dropout\n x = self.dropout(F.relu(self.fc1(x)))\n x = self.dropout(F.relu(self.fc2(x)))\n x = self.dropout(F.relu(self.fc3(x)))\n\n # output so no dropout here\n x = F.log_softmax(self.fc4(x), dim=1)\n\n return x\n \nmodel=Network()\noptimizer=optim.Adam(model.parameters(),lr=0.01)\ncriterion=nn.NLLLoss()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d7ef3898265ce167202ee321100129a95e3b748e","trusted":true},"cell_type":"code","source":"epochs=5\ntrain_losses,test_losses=[],[]\nfor e in range(epochs):\n running_loss=0\n for images,labels in trainloader:\n optimizer.zero_grad()\n #images=images.view(images.shape[0],-1)\n log_ps=model(images)\n loss=criterion(log_ps,labels)\n loss.backward()\n optimizer.step()\n running_loss+=loss.item()\n \n else:\n test_loss=0\n accuracy=0\n \n with torch.no_grad():\n model.eval()\n for images,labels in testloader:\n log_ps=model(images)\n test_loss+=criterion(log_ps,labels)\n ps=torch.exp(log_ps)\n top_p,top_class=ps.topk(1,dim=1)\n equals=top_class==labels.view(*top_class.shape)\n accuracy+=torch.mean(equals.type(torch.FloatTensor))\n model.train()\n train_losses.append(running_loss/len(trainloader))\n test_losses.append(test_loss/len(testloader))\n\n print(\"Epoch: {}/{}.. \".format(e+1, epochs),\n \"Training Loss: {:.3f}.. \".format(running_loss/len(trainloader)),\n \"Test Loss: {:.3f}.. \".format(test_loss/len(testloader)),\n \"Test Accuracy: {:.3f}\".format(accuracy/len(testloader))) ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"086d29056a5322e8c76820fcdc87ddc5424539ff","trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\n%matplotlib inline\nplt.plot(train_losses, label='Training loss')\nplt.plot(test_losses, label='Validation loss')\nplt.legend(frameon=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5612d928a89de850888b2caa3251617ff1ebc9fc"},"cell_type":"markdown","source":"## Inference time"},{"metadata":{"_uuid":"e908e92d92cf3f2634fa665acbca4012bacdb3bf","trusted":true},"cell_type":"code","source":"model.eval()\n\ndataiter = iter(testloader)\nimages, labels = dataiter.next()\nimg = images[0]\n# Convert 2D image to 1D vector\nimg = img.view(1, 784)\n\n# Calculate the class probabilities (softmax) for img\nwith torch.no_grad():\n output = model.forward(img)\n\nps = torch.exp(output)\ntop_prob,top_class=ps.topk(1,dim=1)\ntop_class.item(),labels[0]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"086a197d3e597422ee8b592832f811444577b876"},"cell_type":"markdown","source":"The parameters for PyTorch networks are stored in a model's state_dict\n Optimizer objects (torch.optim) also have a state_dict, which contains information about the optimizer’s state, as well as the hyperparameters used.\n\nBecause state_dict objects are Python dictionaries, they can be easily saved, updated, altered, and restored, adding a great deal of modularity to PyTorch models and optimizers."},{"metadata":{"_uuid":"2dbde41ca47a505766ec29ad2c89443f9a4d8d7a","trusted":true},"cell_type":"code","source":"print(\"Model's state_dict:\")\nfor param_tensor in model.state_dict():\n print(param_tensor, \"\\t\", model.state_dict()[param_tensor].size())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"67e383a0385379d68f1c4f0d38143504b098a8d5","trusted":true},"cell_type":"code","source":"# Print optimizer's state_dict\nprint(\"Optimizer's state_dict:\")\nfor var_name in optimizer.state_dict():\n print(var_name, \"\\t\", optimizer.state_dict()[var_name])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"98bee7eaf09d225730006f2b24ca5662d28cfb65"},"cell_type":"markdown","source":"# Kaggle- Multilayered Perceptron (MLP) implemention on MNIST dataset\nUntill now we were using the MNIST dataset that is available in torchvision.dataset.Let us now load the dataset from Kaggle repo and train our model"},{"metadata":{"_uuid":"4e83a9e422c10eb07cbfb779be01803d2b8a5334","trusted":true},"cell_type":"code","source":"import os\nfrom pathlib import Path\nimport torch\nfrom torch.utils.data import TensorDataset ,DataLoader\nfrom torch import nn,optim\nimport torch.nn.functional as F\nimport numpy as np\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\nPATH=Path(\"../input/digit-recognizer\")\nprint(os.listdir(\"../input/digit-recognizer\"))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"661e23266c2c882d34cdae4c9074d26c0d2ac040"},"cell_type":"markdown","source":"## Load Data "},{"metadata":{"_uuid":"72b61aac15c29fc295d77130b07d1110c9cb1825","trusted":true},"cell_type":"code","source":"train=pd.read_csv(PATH/'train.csv')\ntest=pd.read_csv(PATH/'test.csv')\ntrain.shape,test.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6d5eb02dbfc5e385ffd113560494e0b2275e5f66"},"cell_type":"markdown","source":"## Extracting Input and Target Variable"},{"metadata":{"_uuid":"858a074c0e1dea92562d0f1bd93ff5302f861643","trusted":true},"cell_type":"code","source":"x=train.drop(\"label\",axis=1)\ny=np.array(train['label'])\nx.shape,y.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"387109d77b8c6f76d4de868158340faf1a38b2dc"},"cell_type":"markdown","source":"## Normalization "},{"metadata":{"_uuid":"b8ff683f50227c31ee39ec05a7bb8b4e2a8c5f94","trusted":true},"cell_type":"code","source":"#x_train=x/255","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"614e6245bdf2e3372dca00d5111b3fbe8e993a64"},"cell_type":"markdown","source":"## Train Test Split"},{"metadata":{"trusted":true,"_uuid":"65e189e8b8c6232146d0dde1fceb773353ad8389"},"cell_type":"code","source":"#x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.15)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f37069db924850a07cd3e98378ae9c61bab1f8f"},"cell_type":"markdown","source":"## Train Test in Pytorch"},{"metadata":{"trusted":true,"_uuid":"709aaea373c664fad6fd42d8779ae091145b8308"},"cell_type":"code","source":"'''\n# create feature and targets tensor for train set.\ntorch_X_train = torch.from_numpy(x_train.values).type(torch.FloatTensor)\ntorch_y_train = torch.from_numpy(y_train).type(torch.LongTensor)\n\n# create feature and targets tensor for test set.\ntorch_X_test = torch.from_numpy(x_test.values).type(torch.FloatTensor)\ntorch_y_test = torch.from_numpy(y_test).type(torch.LongTensor)\n\n# Pytorch train and test sets\ntrain = torch.utils.data.TensorDataset(torch_X_train,torch_y_train)\ntest = torch.utils.data.TensorDataset(torch_X_test,torch_y_test)\n'''","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"80525db377fe319e7a3fc538ef8b76639c28c995"},"cell_type":"code","source":"'''\nBATCH_SIZE=64\n# data loader\ntrain_loader = torch.utils.data.DataLoader(train, batch_size = BATCH_SIZE, shuffle = False)\ntest_loader = torch.utils.data.DataLoader(test, batch_size = BATCH_SIZE, shuffle = False)\n'''","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"454f4fb8e0a416e60aed471a49850c01a64a77e6"},"cell_type":"markdown","source":"## Train -Test Split -Pytorch"},{"metadata":{"trusted":true,"_uuid":"d616496f4e453322efe8aecfa71b17fcdddaa8dd"},"cell_type":"code","source":"torch_X_train = torch.from_numpy(x.values).type(torch.FloatTensor)/255\ntorch_y_train = torch.from_numpy(y).type(torch.LongTensor)\nmyDataset = torch.utils.data.TensorDataset(torch_X_train,torch_y_train)\nvalid_no = int(0.2 * len(myDataset))\n# so divide the data into trainset and testset\ntrainSet,testSet = torch.utils.data.random_split(myDataset,(len(myDataset)-valid_no,valid_no))\nprint(f\"len of trainSet {len(trainSet)} , len of testSet {len(testSet)}\")\nbatch_size=64\ntrain_loader = DataLoader(trainSet , batch_size=batch_size ,shuffle=True) \ntest_loader = DataLoader(testSet , batch_size=batch_size ,shuffle=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a06249ad57104d74d0258fd5972961f720bdc3cf","trusted":true},"cell_type":"markdown","source":"trainData = torch.from_numpy(x_train.values)\ntrainLabel=torch.from_numpy(y_train)\ntestData = torch.from_numpy(x_test.values)\ntestLabel = torch.from_numpy(y_test)\ntrainData, testData = trainData.type(torch.FloatTensor), testData.type(torch.LongTensor)\ntrainLabel, testLabel = trainLabel.type(torch.FloatTensor), testLabel.type(torch.LongTensor)\ntrainData.shape,testData.shape\ntrainData = trainData.unsqueeze_(dim=1)\ntestData = testData.unsqueeze_(dim=1)\ntrainData.shape,testData.shape\n#transforms =transforms.Compose(transforms.ToTensor())\ntrain_dataset = TensorDataset(trainData,trainLabel)\ntrain_loader = DataLoader(train_dataset,batch_size=64,shuffle=True)\n\ntest_dataset = TensorDataset(testData,testLabel)\ntest_loader = DataLoader(test_dataset,batch_size=64,shuffle=True)"},{"metadata":{"_uuid":"97b2925a24f59b10a16864a76f00e02a4c92b36f"},"cell_type":"markdown","source":"## Network"},{"metadata":{"trusted":true,"_uuid":"8cbe1c508bacadbb875014318a35fed17ab6a3a1"},"cell_type":"code","source":"class Network(nn.Module):\n def __init__(self):\n super().__init__()\n self.fc1 = nn.Linear(784, 80)\n self.fc2 = nn.Linear(80, 10)\n #self.fc3 = nn.Linear(88, 30)\n #self.fc4 = nn.Linear(30, 10)\n\n # Dropout module with 0.2 drop probability\n self.dropout = nn.Dropout(p=0.2)\n\n def forward(self, x):\n # make sure input tensor is flattened\n x = x.view(x.shape[0], -1)\n\n # Now with dropout\n x = self.dropout(F.relu(self.fc1(x)))\n #x = self.dropout(F.relu(self.fc2(x)))\n #x = self.dropout(F.relu(self.fc3(x)))\n\n # output so no dropout here\n x = F.log_softmax(self.fc2(x), dim=1)\n\n return x\n \nmodel=Network()\noptimizer=optim.Adam(model.parameters(),lr=0.003)\npytorch_total_params = sum(p.numel() for p in model.parameters())\nprint(pytorch_total_params)\ncriterion=nn.NLLLoss()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# ****How many neurons in hidden layer?****"},{"metadata":{},"cell_type":"markdown","source":"Number of neurons in hidden layer can be found approximately using **Geometric Rule proposed by Masters(1993)**. If there are m inputs and n outputs in a 1 hidden layer network, \n\nIn a *1 hidden layer network*, \n\n* Number of neurons in hidden layer = sqrt(m * n) = sqrt(784*10) = 88.54\n\nIn a *2 hidden layer network*, \n\n* r = (m/n)^(1/3) = 4.28; \n* 1st layer neurons = n * (r)^2 = 183;\n* 2nd layer neurons = n * (r)^1 = 42.8\n\nIn a *3 hidden layer network*, \n\n* r = (m/n)^(1/4) = 2.97; \n* 1st layer neurons = n * (r)^3 = 261.98;\n* 2nd layer neurons = n * (r)^2 = 88.20\n* 3rd layer neurons = n * (r)^1 = 29.7\n \nAddition of a 2nd layer and a 3rd layer with number of neurons found using Master's rule did not have any improvement over the 1 hidden layer network. All 3 had test accuracies of 0.97.\n\nWhen accuracies are similar, we choose the model with the *least number of parameters*, the 1 hidden layer network with 88 neurons.\n\nTexts for above rule - https://books.google.co.in/books?id=Bn2jBQAAQBAJ&pg=PA135&lr=&source=gbs_toc_r&cad=4#v=onepage&q&f=false page 177;\n\nhttps://link.springer.com/article/10.1007/s40096-014-0117-6\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Different model validation results #\n\n self.fc1 = nn.Linear(784, 256)\n self.fc2 = nn.Linear(256, 128)\n self.fc3 = nn.Linear(128, 64)\n self.fc4 = nn.Linear(64, 10)\n \n - 0.94\n \n self.fc1 = nn.Linear(784, 128)\n self.fc2 = nn.Linear(128, 128)\n self.fc3 = nn.Linear(128, 64)\n self.fc4 = nn.Linear(64, 10)\n \n - 0.953\n \n self.fc1 = nn.Linear(784, 128)\n self.fc2 = nn.Linear(128, 64)\n self.fc3 = nn.Linear(64, 32)\n self.fc4 = nn.Linear(32, 10)\n \n - 0.9512\n\n self.fc1 = nn.Linear(784, 128)\n self.fc2 = nn.Linear(128, 64)\n self.fc3 = nn.Linear(64, 10)\n \n - 0.94\n \n self.fc1 = nn.Linear(784, 16)\n self.fc2 = nn.Linear(16, 10)\n \n - 0.93\n \n self.fc1 = nn.Linear(784, 24)\n self.fc2 = nn.Linear(24, 10)\n \n - 0.942\n \nUsing masters rule to find out number of neurons in hidden layer\n\n self.fc1 = nn.Linear(784, 88)\n self.fc2 = nn.Linear(88, 10)\n \n - 0.962\n \n Above model with learning rate as 0.003\n\n - 0.970\n\n Using masters rule with 2 and 3 hidden layers\n \n self.fc1 = nn.Linear(784, 183)\n self.fc2 = nn.Linear(183, 43)\n self.fc3 = nn.Linear(43, 10)\n\n - 0.970\n\n self.fc1 = nn.Linear(784, 262)\n self.fc2 = nn.Linear(262, 88)\n self.fc3 = nn.Linear(88, 30)\n self.fc3 = nn.Linear(30, 10)\n\n - 0.970"},{"metadata":{},"cell_type":"markdown","source":"### Lets check if accuracy drops if we gradually reduce the number of neurons from 88\n\n* 80 - 0.993; (investigate)\n* 75 - 0.970\n* 70 - 0.972\n* 65 - 0.969\n\n**80 seems to be the ideal(too good to be true?) configuration for our problem.**"},{"metadata":{},"cell_type":"markdown","source":"# **Choice of learning rate**\n\nInitially training loss kept fluctuating when learning rate was 0.01. Reducing it to 0.003 made it more stable."},{"metadata":{"_uuid":"2bea512cd5bd9f4f41bd77044f471e644505a5fa"},"cell_type":"markdown","source":"## Train "},{"metadata":{"_uuid":"2df4882ed86f9b17b4bba52d56adfde46d1f718d","trusted":true},"cell_type":"code","source":"epochs=10\ntrain_losses,test_losses=[],[]\nfor e in range(epochs):\n running_loss=0\n for images,labels in train_loader:\n optimizer.zero_grad()\n log_ps=model(images)\n loss=criterion(log_ps,labels)\n loss.backward()\n optimizer.step()\n running_loss+=loss.item()\n \n else:\n test_loss=0\n accuracy=0\n \n with torch.no_grad():\n model.eval()\n for images,labels in test_loader:\n log_ps=model(images)\n test_loss+=criterion(log_ps,labels)\n ps=torch.exp(log_ps)\n top_p,top_class=ps.topk(1,dim=1)\n equals=top_class==labels.view(*top_class.shape)\n accuracy+=torch.mean(equals.type(torch.FloatTensor))\n model.train()\n train_losses.append(running_loss/len(train_loader))\n test_losses.append(test_loss/len(test_loader))\n\n print(\"Epoch: {}/{}.. \".format(e+1, epochs),\n \"Training Loss: {:.3f}.. \".format(running_loss/len(train_loader)),\n \"Test Loss: {:.3f}.. \".format(test_loss/len(test_loader)),\n \"Test Accuracy: {:.3f}\".format(accuracy/len(test_loader)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**I am suspicious of the extremely high accuracy. I suspect it's related to train test splits. But for submissions purposes, I will be using this as it's my best accuracy.**\n\nChecked with kaggle score. While this was still my best kaggle score, it definitely isn't 0.993 validation accuracy good."},{"metadata":{"_uuid":"9373aea13cc5891684fb8b801cbcd0ac17eff458"},"cell_type":"markdown","source":"## Save our model"},{"metadata":{"_uuid":"bfcc3f17cadca0bf48cec130586b907965905ed6","trusted":true},"cell_type":"code","source":"print(\"Our model: \\n\\n\", model, '\\n')\nprint(\"The state dict keys: \\n\\n\", model.state_dict().keys())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f83335753469344d38ac362053a74fad30b0ca3e","trusted":true},"cell_type":"code","source":"torch.save(model.state_dict(), 'checkpoint.pth')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"70b1a508fd748a8823a9e1af57538e4cacb0621d"},"cell_type":"markdown","source":"## Load our model"},{"metadata":{"_uuid":"abd854b1c8bfed532cb4e64578c40fd29dea9a36","trusted":true},"cell_type":"code","source":"state_dict = torch.load('checkpoint.pth')\nprint(state_dict.keys())","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7a889737952161eb1834f521476f0f1c8448570a","trusted":true},"cell_type":"code","source":"model.load_state_dict(state_dict)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a08eff1adeb3a8f8f7a31356828ee732a557c3d5","trusted":true},"cell_type":"code","source":"checkpoint = {'input_size': 784,\n 'output_size': 10,\n 'hidden_layers': [256,128,64],\n 'state_dict': model.state_dict()}\n\ntorch.save(checkpoint, 'checkpoint.pth')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b3356c9580d6c01685a52a11f906ee1c9dbe7ef1"},"cell_type":"markdown","source":"## Load Test Data"},{"metadata":{"trusted":true,"_uuid":"34abb15aa48ad4f133ee15a2f9a5268b45692c36"},"cell_type":"code","source":"test_images = pd.read_csv(\"../input/digit-recognizer/test.csv\")\ntest_image = test_images.loc[:,test_images.columns != \"label\"].values\ntest_dataset = torch.from_numpy(test_image).type(torch.FloatTensor)/255\nprint(test_dataset.shape)\n#test_dataset = torch.utils.data.TensorDataset(test_dataset)\nnew_test_loader = torch.utils.data.DataLoader(test_dataset, batch_size = 100, shuffle = False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"790dd7f94c59a6de5441827b999b6cff6f686154"},"cell_type":"code","source":"results = []\nwith torch.no_grad():\n model.eval()\n for images in new_test_loader:\n output = model(images)\n ps = torch.exp(output)\n top_p, top_class = ps.topk(1, dim = 1)\n results += top_class.numpy().tolist()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0f9b334b1c25c4bb1a48fda6b64634f1f90b7565"},"cell_type":"markdown","source":"## Check the results"},{"metadata":{"trusted":true,"_uuid":"56ffd3051274596232af3ca4d1a9ee7d6322881a"},"cell_type":"code","source":"predictions = np.array(results).flatten()\nprint(predictions[:5])\nprint(predictions.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f946013a3eab6274d1becd93dfe99aa9f7491cf9"},"cell_type":"markdown","source":"## Submit for Scoring"},{"metadata":{"trusted":true,"_uuid":"655160a6e2d490651c0fe70b8ba7480ed8ba1fcc"},"cell_type":"code","source":"submissions=pd.DataFrame({\"ImageId\": list(range(1,len(predictions)+1)),\n \"Label\": predictions})\nsubmissions.to_csv(\"my_submissions.csv\", index=False, header=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"177c548a264bbaaa76e216eeaf1d747db88b1030"},"cell_type":"markdown","source":"# Reference \n\n[Introduction to Pytorch-Udacity](https://github.com/udacity/deep-learning-v2-pytorch/tree/master/intro-to-pytorch)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}