{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Introduction\nEvery once in a while, a python library is developed that has the potential of changing the landscape in the field of Deep Learning. PyTorch is one such library. In the last few weeks, I have been dabbling a bit in PyTorch. I have been blown away by how easy it is to grasp. Among the various deep learning libraries I have used till date – PyTorch has been the most flexible and effortless of them all.\n\nNow we're going to build a larger network that can solve a (formerly) difficult problem, identifying text in an image. Here we'll use the MNIST dataset which consists of greyscale handwritten digits. Each image is 28x28 pixels, you can see a sample below\n\n![MNIST](https://github.com/abhinand5/intro-to-pytorch/raw/8255d46031d127f57a7727a5e8a65ed9d3676e3a/intro-to-pytorch/assets/mnist.png)\n\nOur goal is to build a neural network that can take one of these images and predict the digit in the image. Let's get straight into it\n\n### **If you like this kernel or wish to fork it, please UPVOTE to show your support.**"},{"metadata":{},"cell_type":"markdown","source":"## Import Libraries"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt # for plotting beautiful graphs\n\n# train test split from sklearn\nfrom sklearn.model_selection import train_test_split\n\n# Import Torch \nimport torch\nimport torch.nn as nn\nfrom torchvision import transforms, models\n# from torch.utils.data import SubsetRandomSampler\nfrom torch.autograd import Variable\nfrom torch import nn, optim\nimport torch.nn.functional as F\n\n# What's in the current directory?\nimport os\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Import the Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"../input/train.csv\", dtype=np.float32)\nfinal_test = pd.read_csv(\"../input/test.csv\", dtype=np.float32)\nsample_sub = pd.read_csv(\"../input/sample_submission.csv\")\ntrain.label.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Preparing Dataset\n* What we are doing here is taking the raw dataset and splitting into targets and features. Dividing by 255 makes each pixel value to scale between 0 and 1 instead of 0 and 255, which helps in training our model. This step in Machine Learning is generally known as Normalization. Then we split into train and test sets using sklearn's train_test_split function.\n\n* Converting the numpy arrays into PyTorch Tensors using from_numpy function. Don’t let the word “tensor” scare you. It is nothing more than a simple mathematical concept. Tensors are mathematical objects that generalize scalars, vectors and matrices to higher dimensions.\n\n* Batch size is set. The batch size is usually set between 64 and 256. The batch size does have an effect on the final test accuracy. One way to think about it is that smaller batches means that the number of parameter updates per epoch is greater. \n\n* To pass our data into our PyTorch models we need to convert it to a PyTorch Dataset. A Tensor Dataset in this case. \n\n* We have the training data loaded into trainloader and we can make an iterator with iter(trainloader) that can help us grab data. Later, we'll use this to loop through the dataset for training. Each time we can pull out data of the size of the batch that is defined."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Seperate the features and labels\ntargets_np = train.label.values\nfeatures_np = train.loc[:, train.columns != 'label'].values/255\n\n# Split into training and test set\nfeatures_train, features_test, target_train, target_test = train_test_split(features_np, targets_np, test_size=0.2, random_state=42)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# create feature and targets tensor for train set. As you remember we need variable to accumulate gradients. Therefore first we create tensor, then we will create variable\nfeaturesTrain = torch.from_numpy(features_train)\ntargetsTrain = torch.from_numpy(target_train).type(torch.LongTensor) # data type is long\n\n# create feature and targets tensor for test set.\nfeaturesTest = torch.from_numpy(features_test)\ntargetsTest = torch.from_numpy(target_test).type(torch.LongTensor) # data type is long","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Set batch size\nbatch_size = 256\n\n# Pytorch train and test sets\ntrain = torch.utils.data.TensorDataset(featuresTrain,targetsTrain)\ntest = torch.utils.data.TensorDataset(featuresTest,targetsTest)\n\n# data loader\ntrain_loader = torch.utils.data.DataLoader(train, batch_size = batch_size, shuffle = True)\ntest_loader = torch.utils.data.DataLoader(test, batch_size = batch_size, shuffle = True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# visualize one of the images in data set\ndef visualize_image(data, index, pred=False, val=0):\n    '''This funtion can be used to visualize the images'''\n    plt.imshow(data[index].reshape(28,28))\n    plt.axis(\"off\")\n    plt.title(\"Handwritten Digit Image\")\n    plt.show()\nvisualize_image(features_np, 12)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"featuresTrain.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Define Network Architecture\nPyTorch provides a module nn that makes building networks much simpler. Here I'll show you how to build the same with 784 inputs, hidden units with 512, 256, 128, 64 neurons in each hidden layer, 10 output units as we have 10 classes to classify and a softmax output for multi-class classification. It is quite similar to the one example image below.\n\n![Example](https://assets.digitalocean.com/articles/handwriting_tensorflow_python3/cnwitLM.png)"},{"metadata":{"trusted":true},"cell_type":"code","source":"class Classifier(nn.Module):\n    def __init__(self):\n        super().__init__()\n        # 5 Hidden Layer Network\n        self.fc1 = nn.Linear(28*28, 512)\n        self.fc2 = nn.Linear(512, 256)\n        self.fc3 = nn.Linear(256, 128)\n        self.fc4 = nn.Linear(128, 64)\n        self.fc5 = nn.Linear(64, 10)\n        \n        # Dropout module with 0.2 probbability\n        self.dropout = nn.Dropout(p=0.2)\n        # Add softmax on output layer\n        self.log_softmax = F.log_softmax\n        \n    def forward(self, x):\n        x = self.dropout(F.relu(self.fc1(x)))\n        x = self.dropout(F.relu(self.fc2(x)))\n        x = self.dropout(F.relu(self.fc3(x)))\n        x = self.dropout(F.relu(self.fc4(x)))\n        \n        x = self.log_softmax(self.fc5(x), dim=1)\n        \n        return x","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Well, that was a bit too much isn't it.\n\nLet's go through this bit by bit.\n\n`class Network(nn.Module):`\n\nHere we're inheriting from nn.Module. Combined with super().__init__() this creates a class that tracks the architecture and provides a lot of useful methods and attributes. It is mandatory to inherit from nn.Module when you're creating a class for your network. The name of the class itself can be anything.\n\n`self.hidden = nn.Linear(784, 512)`\n\nThis line creates a module for a linear transformation, $x\\mathbf{W} + b$, with 784 inputs and 512 outputs and assigns it to self.hidden. The module automatically creates the weight and bias tensors which we'll use in the forward method. You can access the weight and bias tensors once the network (net) is created with net.hidden.weight and net.hidden.bias.\n\n`self.output = nn.Linear(64, 10)`\n\nSimilarly, this creates another linear transformation with 64 inputs and 10 outputs.\n\n`\nself.log_softmax = F.log_softmax`\n\nHere I defined operations for the log softmax activation output. Setting dim=1 in F.log_softmax(dim=1) calculates softmax across the columns, you'll see this down below.\n\n`self.dropout = nn.Dropout(p=0.2)`\n\nThe most common method to reduce overfitting (outside of early-stopping) is dropout, where we randomly drop input units. This forces the network to share information between weights, increasing it's ability to generalize to new data. Adding dropout in PyTorch is straightforward using the nn.Dropout module.\n\n`def forward(self, x):`\n\nPyTorch networks created with nn.Module must have a forward method defined. It takes in a tensor x and passes it through the operations you defined in the __init__ method.\n\n`x = self.dropout(F.relu(self.fc1(x)))\nx = self.dropout(F.relu(self.fc2(x)))\nx = self.dropout(F.relu(self.fc3(x)))\nx = self.dropout(F.relu(self.fc4(x)))`\n\nHere the input tensor x is passed through each operation and reassigned to x. We can see that the input tensor goes through the hidden layer, then a ReLU function, then the output layer, and finally the softmax function. "},{"metadata":{},"cell_type":"markdown","source":"## Training and Validation "},{"metadata":{},"cell_type":"markdown","source":"Now we should create our own network and train it. First we'll want to define the criterion (something like nn.CrossEntropyLoss or nn.NLLLoss) and the optimizer (typically optim.SGD or optim.Adam).\n* Make a forward pass through the network\n* Use the network output to calculate the loss\n* Perform a backward pass through the network with loss.backward() to calculate the gradients\n* Take a step with the optimizer to update the weights"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Instantiate our model\nmodel = Classifier()\n# Define our loss function\ncriterion = nn.NLLLoss()\n# Define the optimier\noptimizer = optim.Adam(model.parameters(), lr=0.0015)\n\nepochs = 25\nsteps = 0\nprint_every = 50\ntrain_losses, test_losses = [], []\n\nfor e in range(epochs):\n    running_loss = 0\n    for images, labels in train_loader:\n        steps += 1\n        # Prevent accumulation of gradients\n        optimizer.zero_grad()\n        # Make predictions\n        log_ps = model(images)\n        loss = criterion(log_ps, labels)\n        #backprop\n        loss.backward()\n        optimizer.step()\n        \n        running_loss += loss.item()\n        if steps % print_every == 0:\n            test_loss = 0\n            accuracy = 0\n\n            # Turn off gradients for validation\n            with torch.no_grad():\n                model.eval()\n                for images, labels in test_loader:\n                    log_ps = model(images)\n                    test_loss += criterion(log_ps, labels)\n\n                    ps = torch.exp(log_ps)\n                    # Get our top predictions\n                    top_p, top_class = ps.topk(1, dim=1)\n                    equals = top_class == labels.view(*top_class.shape)\n                    accuracy += torch.mean(equals.type(torch.FloatTensor))\n\n            model.train()\n\n            train_losses.append(running_loss/len(train_loader))\n            test_losses.append(test_loss/len(test_loader))\n\n            print(\"Epoch: {}/{}.. \".format(e+1, epochs),\n                  \"Training Loss: {:.3f}.. \".format(train_losses[-1]),\n                  \"Test Loss: {:.3f}.. \".format(test_losses[-1]),\n                  \"Test Accuracy: {:.3f}\".format(accuracy/len(test_loader)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Oh what just happended there? I'll explain not to worry.\n\n#### Training: \n* I'm looping over the train loader, pulling out the images and labels.\n* Note that I have a line of code optimizer.zero_grad(). When you do multiple backwards passes with the same parameters, the gradients are accumulated. This means that you need to zero the gradients on each training pass or you'll retain gradients from previous training batches.\n* I have named the next variable log_ps because our model gives us back logs of class probabilities, you can take exponent to convert it to normal probabilities which I've done down below for validation. \n* We calculate the loss. Then backpropagate through the network. We then make one optimizer step. Which brings us closer and closer to the global optimum.\n\n#### Validation\n* We turn off the gradients for validation as it is not needed and saves a lot of memory and computation. Note that we should turn it back on after each step of validation.\n* We loop over the test_loader and essentially repeat some steps we have done above. Since it's validation we don't need to backpropagate. \n* The next step - With the probabilities, we can get the most likely class using the ps.topk method. This returns the $k$ highest values. Since we just want the most likely class, we can use ps.topk(1). This returns a tuple of the top-$k$ values and the top-$k$ indices. If the highest value is the fifth element, we'll get back 4 as the index.\n* Then we check if the predicted value is equal to the actual value. \n* We then calculate the percentage of correct predictions, which indeed is using the mean of our top predictions. But you cannot just use torch.mean because topk returns a byte tensor but we need a float tensor to perform torch.mean we do that in the next step.\n\nThe same process is repeated over and over again. The results are printed on each step. With this simple model we're able to get about 98% accuracy on validation which is awesome, isn't it?\n\nHope that made sense. "},{"metadata":{},"cell_type":"markdown","source":"## Visualizing Model Performance"},{"metadata":{"trusted":true},"cell_type":"code","source":"%matplotlib inline\n%config InlineBackend.figure_format = 'retina'\n\nplt.plot(train_losses, label='Training loss')\nplt.plot(test_losses, label='Validation loss')\nplt.legend(frameon=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This graph looks decent to me. We're doing fairly well for our first model in PyTorch."},{"metadata":{},"cell_type":"markdown","source":"## Inference\nYou can see what out model is predicting here on the test data. You can try playing around with this function for different images. \n\nAs expected our model seems to do well, infact really well."},{"metadata":{"trusted":true},"cell_type":"code","source":"def view_classify(img, ps):\n    ''' Function for viewing an image and it's predicted classes.\n    '''\n    ps = ps.data.numpy().squeeze()\n\n    fig, (ax1, ax2) = plt.subplots(figsize=(6,9), ncols=2)\n    ax1.imshow(img.resize_(1, 28, 28).numpy().squeeze())\n    ax1.axis('off')\n    ax2.barh(np.arange(10), ps)\n    ax2.set_aspect(0.1)\n    ax2.set_yticks(np.arange(10))\n    ax2.set_yticklabels(np.arange(10))\n    ax2.set_title('Class Probability')\n    ax2.set_xlim(0, 1.1)\n\n    plt.tight_layout()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%matplotlib inline\ndef make_prediction(data):\n    images, labels = next(iter(data))\n\n    img = images[42].view(1, 784)\n    # Turn off gradients to speed up this part\n    with torch.no_grad():\n        logps = model(img)\n\n    # Output of the network are log-probabilities, need to take exponential for probabilities\n    ps = torch.exp(logps)\n    view_classify(img.view(1, 28, 28), ps)\nmake_prediction(test_loader)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Preparing Test Data for Prediction"},{"metadata":{"trusted":true},"cell_type":"code","source":"final_test_np = final_test.values/255\ntest_tn = torch.from_numpy(final_test_np)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Creating fake labels for convenience of passing into DataLoader\n## CAUTION: There are other ways of doing this, I just did it this way\nfake_labels = np.zeros(final_test_np.shape)\nfake_labels = torch.from_numpy(fake_labels)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_tn_data = torch.utils.data.TensorDataset(test_tn, fake_labels)\n\nsubmission_loader = torch.utils.data.DataLoader(submission_tn_data, batch_size = batch_size, shuffle = False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Seeing what our model does on test data\nmake_prediction(submission_loader)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Making Predictions on Test data"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Making it submission ready\nsubmission = [['ImageId', 'Label']]\n\n# Turn off gradients for validation\nwith torch.no_grad():\n    model.eval()\n    image_id = 1\n    for images, _ in submission_loader:\n        log_ps = model(images)\n        ps = torch.exp(log_ps)\n        top_p, top_class = ps.topk(1, dim=1)\n        \n        for prediction in top_class:\n            submission.append([image_id, prediction.item()])\n            image_id += 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df = pd.DataFrame(submission)\nsubmission_df.columns = submission_df.iloc[0]\nsubmission_df = submission_df.drop(0, axis=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission_df.to_csv(\"submission.csv\", index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### **If you like this kernel or wish to fork it, please UPVOTE to show your support.**\n\n**Authored by:**\n[Abhinand](https://www.kaggle.com/abhinand05)"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}