{"cells":[{"metadata":{},"cell_type":"markdown","source":"
\n

MNIST with Pytorch Logistic Regression

\n"},{"metadata":{},"cell_type":"markdown","source":"`We are using Mnist kaggle training set as our training dataset. It consists of 28px by 28px grayscale images of handwritten digits (0 to 9), along with labels for each image indicating which digit it represents. Here are some sample images from the dataset:`\n\n![mnist-sample](https://i.imgur.com/CAYnuo1.jpg)"},{"metadata":{},"cell_type":"markdown","source":"# Importing Libraries"},{"metadata":{"trusted":true},"cell_type":"code","source":"import warnings \nwarnings.filterwarnings('ignore')\nimport torch\nimport torchvision\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nsns.set_context(\"paper\", font_scale = 1, rc={\"grid.linewidth\": 3})\npd.set_option('display.max_rows', 100, 'display.max_columns', 400)\nfrom torch.utils.data import DataLoader,Dataset,ConcatDataset\nfrom torchvision import transforms\nimport torch.optim as optim\nfrom torchvision.datasets import MNIST\nfrom torch.utils.data import random_split\nimport torch.nn as nn\nimport torch.nn.functional as F","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset = MNIST(root='data/', download=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(dataset)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The dataset has 60,000 images which can be used to train the model. There is also an additonal test set of 10,000 images which can be created by passing `train=False` to the `MNIST` class. We will use it as test dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"test_dataset = MNIST(root='data/', train=False,transform=transforms.ToTensor())\nlen(test_dataset)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`Let's look at a sample element from the training dataset.`"},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset[0]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It's a pair, consisting of a 28x28 image and a label. The image is an object of the class `PIL.Image.Image`, which is a part of the Python imaging library [Pillow](https://pillow.readthedocs.io/en/stable/). We can view the image within Jupyter using [`matplotlib`](https://matplotlib.org/), the de-facto plotting and graphing library for data science in Python."},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(20,8))\nfor i in range(10,18):\n image,label=dataset[i]\n plt.subplot(231+(i))\n plt.imshow(image,cmap='gray')\n plt.title('Label:'+str(label),fontweight='bold',size=20)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"PyTorch datasets allow us to specify one or more transformation functions which are applied to the images as they are loaded. `torchvision.transforms` contains many such predefined functions, and we'll use the `ToTensor` transform to convert images into PyTorch tensors."},{"metadata":{"trusted":true},"cell_type":"code","source":"# MNIST dataset (images and labels)\ndataset = MNIST(root='data/', \n train=True,\n transform=transforms.ToTensor())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img_tensor, label=dataset[0]\nprint(img_tensor.shape,label)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The image is now converted to `1x28x28 tensor`. The first dimension is used to keep track of the color channels. Since images in the MNIST dataset are grayscale, there's just one channel."},{"metadata":{"trusted":true},"cell_type":"code","source":"print(img_tensor[:,5:10,5:10])\nprint(torch.max(img_tensor), torch.min(img_tensor))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The values range from 0 to 1, with 0 representing black, 1 white and the values in between different shades of grey. We can also plot the tensor as an image using `plt.imshow`."},{"metadata":{},"cell_type":"markdown","source":"Let's visualise the colours for above 5x5 matrix."},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(6,4))\nplt.imshow(img_tensor[0,5:10,5:10],cmap='gray')\nplt.title('0:Black,1:White',fontweight='bold',size=12)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the MNIST dataset, there are `60,000` training images, and `10,000` test images. The test set is standardized so that different researchers can report the results of their models against the same set of images. \n\nSince there's no predefined validation set, we must manually split the `60,000` images into training and validation datasets. Let's set aside `10,000` randomly chosen images for validation."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df,val_df= random_split(dataset,[50000,10000])\nprint(len(train_df))\nprint(len(val_df))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can now created data loaders to help us load the data in batches. We'll use a batch size of 128."},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size=128\ntrain_loader=DataLoader(train_df,batch_size,shuffle=True)\nval_loader=DataLoader(val_df,batch_size)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We set `shuffle=True` for the training dataloader, so that the batches generated in each epoch are different, and this randomization helps generalize & speed up the training process. On the other hand, since the validation dataloader is used only for evaluating the model, there is no need to shuffle the images. \nTotal no of Batches formed for `train_df= 50000/128 ~ 391` and for `val_df= 10000/128 ~ 79`."},{"metadata":{},"cell_type":"markdown","source":"* A **logistic regression** model is almost identical to a linear regression model i.e. there are weights and bias matrices, and the output is obtained using simple matrix operations (`pred = x @ w.t() + b`). \n\n* Just as we did with linear regression, we can use `nn.Linear` to create the model instead of defining and initializing the matrices manually.\n\n* Since `nn.Linear` expects the each training example to be a vector, each `1x28x28` image tensor needs to be flattened out into a vector of size 784 (`28*28`), before being passed into the model. \n\n* The output for each image is vector of size 10, with each element of the vector signifying the probability a particular target label (i.e. 0 to 9). The predicted label for an image is simply the one with the highest probability."},{"metadata":{"trusted":true},"cell_type":"code","source":"input_size=28*28 #784\nnum_classes=10 #0-9\nlog_model = nn.Linear(in_features=input_size,out_features=num_classes) # Logistic regression model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#model weights \nprint(log_model.weight)\nprint('\\n')\nprint(log_model.weight.shape) #7840\nprint('\\n')\nprint(log_model.bias) #shape=10 ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Total Number of paramters: `7840 + 10= 7850`"},{"metadata":{"trusted":true},"cell_type":"code","source":"class MnistModel(nn.Module):\n def __init__(self):\n super().__init__()\n self.linear=nn.Linear(input_size,num_classes)\n \n def forward(self,x):\n # Calls to layer to make predictions\n # performs the actual computation, that is, it outputs a prediction, given the input x.\n x=x.reshape(-1,784)\n out=self.linear(x)\n return out","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model=MnistModel()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Inside the `__init__` constructor method, we instantiate the weights and biases using `nn.Linear`. And inside the `forward` method, which is invoked when we pass a batch of inputs to the model, we flatten out the input tensor, and then pass it into `self.linear`.\n\n`x.reshape(-1, 28*28)` indicates to PyTorch that we want a *view* of the `x` tensor with two dimensions, where the length along the 2nd dimension is 28\\*28 (i.e. 784). One argument to `.reshape` can be set to `-1` (in this case the first dimension), to let PyTorch figure it out automatically based on the shape of the original tensor.\n\nNote that the model no longer has `.weight` and `.bias` attributes (as they are now inside the `.linear` attribute), but it does have a `.parameters` method which returns a list containing the weights and bias, and can be used by a PyTorch optimizer."},{"metadata":{"trusted":true},"cell_type":"code","source":"#model weights \nprint(model.linear.weight)\nprint('\\n')\nprint(model.linear.weight.shape) #7840\nprint('\\n')\nprint(model.linear.bias.shape) #shape=10 \nprint('\\n')\nprint(model.parameters) #shape=10 ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for images,label in train_loader:\n outputs=model(images)\n break\nprint('outputs.shape : ', outputs.shape)\nprint('Sample outputs :\\n', outputs[:2].data)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For each of the 100 input images, we get 10 outputs, one for each class. As discussed earlier, we'd like these outputs to represent probabilities, but for that the elements of each output row must lie between 0 to 1 and add up to 1, which is clearly not the case here. \n\nTo convert the output rows into probabilities, we use the softmax function, which has the following formula:\n\n![softmax](https://i.imgur.com/EAh9jLN.png)\n\nFirst we replace each element `yi` in an output row by `e^yi`, which makes all the elements positive, and then we divide each element by the sum of all elements to ensure that they add up to 1. \n\nWhile it's easy to implement the softmax function (you should try it!), we'll use the implementation that's provided within PyTorch, because it works well with multidimensional tensors (a list of output rows in our case)."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Apply softmax for each output row\nprob = F.softmax(outputs, dim=1)\n# Look at sample probabilities\nprint(\"Sample probabilities:\\n\", prob[:2].data)\nprint('\\n')\n# Add up the probabilities of an output row\nprint(\"Sum: \", torch.sum(prob[0]).item())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, we can determine the predicted label for each image by simply choosing the index of the element with the highest probability in each output row. This is done using `torch.max`, which returns the largest element and the index of the largest element along a particular dimension of a tensor."},{"metadata":{"trusted":true},"cell_type":"code","source":"max_prob, predict = torch.max(prob, dim=1)\nprint(predict)\nprint(max_prob)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Evaluation Metric and Loss Function"},{"metadata":{"trusted":true},"cell_type":"code","source":"def accuracy(outputs,label):\n _, pred = torch.max(outputs, dim=1)\n return torch.tensor(torch.sum(pred==label).item()/len(pred))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"accuracy(outputs,label)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"A commonly used loss function for classification problems is the **cross entropy**, which has the following formula:\n\n![cross-entropy](https://i.imgur.com/VDRDl1D.png)\n\nWhile it looks complicated, it's actually quite simple:\n\n* For each output row, pick the predicted probability for the correct label. E.g. if the predicted probabilities for an image are `[0.1, 0.3, 0.2, ...]` and the correct label is `1`, we pick the corresponding element `0.3` and ignore the rest.\n\n* Then, take the [logarithm](https://en.wikipedia.org/wiki/Logarithm) of the picked probability. If the probability is high i.e. close to 1, then its logarithm is a very small negative value, close to 0. And if the probability is low (close to 0), then the logarithm is a very large negative value. We also multiply the result by -1, which results is a large postive value of the loss for poor predictions.\n\n* Finally, take the average of the cross entropy across all the output rows to get the overall loss for a batch of data.\n\nUnlike accuracy, cross-entropy is a continuous and differentiable function that also provides good feedback for incremental improvements in the model (a slightly higher probability for the correct label leads to a lower loss). This makes it a good choice for the loss function. \n\nAs you might expect, PyTorch provides an efficient and tensor-friendly implementation of cross entropy as part of the `torch.nn.functional` package. Moreover, it also performs softmax internally, so we can directly pass in the outputs of the model without converting them into probabilities."},{"metadata":{"trusted":true},"cell_type":"code","source":"loss_fn=F.cross_entropy","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Loss for first batch of data\nloss = loss_fn(outputs, label)\nprint(loss)\nprint(loss.item())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training the model\n\nNow that we have defined the data loaders, model, loss function and optimizer, we are ready to train the model. The training process is identical to linear regression, with the addition of a \"validation phase\" to evaluate the model in each epoch. Here's what it looks like in pseudocode:\n\n```\nfor epoch in range(num_epochs):\n # Training phase\n for batch in train_loader:\n # Generate predictions\n # Calculate loss\n # Compute gradients\n # Update weights\n # Reset gradients\n \n # Validation phase\n for batch in val_loader:\n # Generate predictions\n # Calculate loss\n # Calculate metrics (accuracy etc.)\n # Calculate average validation loss & metrics\n \n # Log epoch, loss & metrics for inspection\n```\n\nSome parts of the training loop are specific the specific problem we're solving (e.g. loss function, metrics etc.) whereas others are generic and can be applied to any deep learning problem. Let's impelment the problem-specific parts within our `MnistModel` class:"},{"metadata":{"trusted":true},"cell_type":"code","source":"class MnistModel(nn.Module):\n def __init__(self):\n super().__init__()\n self.linear=nn.Linear(input_size,num_classes)\n \n def forward(self,x):\n x=x.reshape(-1,784)\n out=self.linear(x)\n return out\n \n def training_step(self,batch):\n images,labels=batch\n out=self(images) #Generate predictions\n loss=F.cross_entropy(out,labels) # calculating loss for cost/loss optimisation\n return loss\n \n def validation_step(self,batch):\n images,labels=batch\n out=self(images)\n loss=F.cross_entropy(out,labels) #loss\n acc=accuracy(out,labels) # accuracy\n return {'val_loss': loss, 'val_acc': acc} \n \n def validation_epoch_end(self,outputs):\n batch_losses=[x['val_loss'] for x in outputs]\n epoch_loss= torch.stack(batch_losses).mean() \n batch_accs=[x['val_acc'] for x in outputs]\n epoch_acc = torch.stack(batch_accs).mean()\n return {'val_loss': epoch_loss.item(), 'val_acc': epoch_acc.item()}\n \n def epoch_end(self, epoch, result):\n print(\"Epoch [{}], val_loss: {:.4f}, val_acc: {:.4f}\".format(epoch, result['val_loss'], result['val_acc']))\n\nmodel=MnistModel()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we'll define an `evaluate` function, which will perform the validation phase, and a `fit` function which will peform the entire training process."},{"metadata":{"trusted":true},"cell_type":"code","source":"def evaluate(model,val_loader):\n outputs=[model.validation_step(batch) for batch in val_loader]\n return model.validation_epoch_end(outputs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def fit(epochs,lr,model,train_loader,val_loader,opt_func=torch.optim.Adam):\n history=[]\n optimizer=opt_func(model.parameters(),lr)\n for epoch in range(epochs):\n # Training Phase \n for batch in train_loader:\n loss=model.training_step(batch)\n loss.backward()\n optimizer.step()\n optimizer.zero_grad()\n #Validation Phase\n result=evaluate(model,val_loader)\n model.epoch_end(epoch,result)\n history.append(result)\n return history","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"result0=evaluate(model,val_loader)\nresult0","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The initial accuracy for one epoch or one iteration is around 10%, which is what one might expect from a randomly intialized model (since it has a 1 in 10 chance of getting a label right by guessing randomly). \n\nWe are now ready to train the model. Let's train for 5 epochs and look at the results."},{"metadata":{"trusted":true},"cell_type":"code","source":"history1 = fit(5, 0.001, model, train_loader, val_loader)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history2 = fit(5, 0.001, model, train_loader, val_loader)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history3 = fit(5, 0.001, model, train_loader, val_loader)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"history4 = fit(5, 0.001, model, train_loader, val_loader)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"While the accuracy does continue to increase as we train for more epochs, the improvements get smaller with every epoch. This is easier to see using a line graph."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Replace these values with your results\nhistory = [result0] + history1 + history2 + history3 + history4\naccuracies = [result['val_acc'] for result in history]\nplt.plot(accuracies, '-x')\nplt.xlabel('epoch')\nplt.ylabel('accuracy')\nplt.title('Accuracy vs. No. of epochs');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It's quite clear from the above picture that the model probably won't cross the accuracy threshold of 93% even after training for a very long time. One possible reason for this is that the learning rate might be too high. It's possible that the model's paramaters are \"bouncing\" around the optimal set of parameters that have the lowest loss. You can try reducing the learning rate and training for a few more epochs to see if it helps.\n\nThe more likely reason that **the model just isn't powerful enough**. If you remember our initial hypothesis, we have assumed that the output (in this case the class probabilities) is a **linear function** of the input (pixel intensities), obtained by perfoming a matrix multiplication with the weights matrix and adding the bias. This is a fairly weak assumption, as there may not actually exist a linear relationship between the pixel intensities in an image and the digit it represents. While it works reasonably well for a simple dataset like MNIST (getting us to 85% accuracy), we need more sophisticated models that can capture non-linear relationships between image pixels and labels for complex tasks like recognizing everyday objects, animals etc. \n"},{"metadata":{},"cell_type":"markdown","source":"## Testing with individual images"},{"metadata":{},"cell_type":"markdown","source":"While we have been tracking the overall accuracy of a model so far, it's also a good idea to look at model's results on some sample images. Let's test out our model with some images from the predefined test dataset of 10000 images. We begin by recreating the test dataset with the `ToTensor` transform."},{"metadata":{"trusted":true},"cell_type":"code","source":"img, label = test_dataset[0]\nplt.imshow(img[0], cmap='gray')\nprint('Shape:', img.shape)\nprint('Label:', label)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img.unsqueeze(0).shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's define a helper function `predict_image`, which returns the predicted label for a single image tensor."},{"metadata":{"trusted":true},"cell_type":"code","source":"def predict_image(img,model):\n x=img.unsqueeze(0)\n y=model(x)\n _, preds = torch.max(y, dim=1)\n return preds[0].item()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`img.unsqueeze` simply adds another dimension at the begining of the 1x28x28 tensor, making it a 1x1x28x28 tensor, which the model views as a batch containing a single image.\n\nLet's try it out with a few images."},{"metadata":{"trusted":true},"cell_type":"code","source":"img, label = test_dataset[0]\nplt.imshow(img[0], cmap='gray')\nprint('Label:', label, ', Predicted:', predict_image(img, model))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img, label = test_dataset[10]\nplt.imshow(img[0], cmap='gray')\nprint('Label:', label, ', Predicted:', predict_image(img, model))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img, label = test_dataset[200]\nplt.imshow(img[0], cmap='gray')\nprint('Label:', label, ', Predicted:', predict_image(img, model))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"img, label = test_dataset[1839]\nplt.imshow(img[0], cmap='gray')\nprint('Label:', label, ', Predicted:', predict_image(img, model))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Identifying where our model performs poorly can help us improve the model, by collecting more training data, increasing/decreasing the complexity of the model, and changing the hypeparameters.\n\nAs a final step, let's also look at the overall loss and accuracy of the model on the test set."},{"metadata":{"trusted":true},"cell_type":"code","source":"test_loader = DataLoader(test_dataset, batch_size=256)\nresult = evaluate(model, test_loader)\nresult","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Saving and loading the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"torch.save(model.state_dict(), 'mnist-logistic.pth')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The `.state_dict` method returns an `OrderedDict` containing all the weights and bias matrices mapped to the right attributes of the model."},{"metadata":{"trusted":true},"cell_type":"code","source":"model.state_dict()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To load the model weights, we can instante a new object of the class `MnistModel`, and use the `.load_state_dict` method."},{"metadata":{"trusted":true},"cell_type":"code","source":"model2 = MnistModel()\nmodel2.load_state_dict(torch.load('mnist-logistic.pth'))\nmodel2.state_dict()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Just as a sanity check, let's verify that this model has the same loss and accuracy on the test set as before."},{"metadata":{"trusted":true},"cell_type":"code","source":"test_loader = DataLoader(test_dataset, batch_size=256)\nresult = evaluate(model2, test_loader)\nresult","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"
\n

If you found this notebook helpful , some upvotes would be very much appreciated - That will keep me motivated :)

\n"},{"metadata":{},"cell_type":"markdown","source":"
\n

Thank You:)

\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}