{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# <center>PyTorch Implementation of a Convolutional Neural Network for MNIST Digit Recognizer</center>","metadata":{}},{"cell_type":"markdown","source":"# Introduction\nIn this project, I will develop a convolutional neural network for handwritten digit classification using PyTorch. This tutorial is divided into the following parts:\n* Data preparation\n* Define a model\n* Train the model\n* Make some predictions","metadata":{}},{"cell_type":"markdown","source":"# Import libraries","metadata":{}},{"cell_type":"code","source":"import os\n\n# PyTorch Modules we need for this kernel\n\nfrom torch.utils.data import Dataset, DataLoader, TensorDataset\nfrom torchvision import transforms\nimport torch \nimport torch.nn as nn\nimport torchvision.transforms as transforms\nimport torchvision.datasets as dsets\ntorch.manual_seed(0)\n\n# Non-PyTorch Modules\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Other non-PyTorch Modules\nfrom sklearn.model_selection import train_test_split\n\nfrom matplotlib.pyplot import imshow\nimport matplotlib.pylab as plt\n\nfrom PIL import Image\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data prepration\nThe train and test sets are stored in csv files.","metadata":{}},{"cell_type":"code","source":"# Loading the data\ntrain_df = pd.read_csv(\"../input/digit-recognizer/train.csv\")\ntest_df = pd.read_csv('../input/digit-recognizer/test.csv')\ntrain_df.head(5)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print('Train data shape:', train_df.shape)\nprint('Test data shape:', test_df.shape)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We first need to split datasets into labels and images (pixels). <br>\nWe will then split the training set into a train and a validation dataset. This allows us to estimate the performance of a model at a training run by making predictions on the validation dataset. The following lines of code are from [frankle muchahary](https://www.kaggle.com/franklemuchahary/mnist-digit-recognition-using-pytorch).\n","metadata":{}},{"cell_type":"code","source":"train_labels = train_df['label'].values\ntrain_images = (train_df.iloc[:,1:].values).astype('float32')\ntest_images = (test_df.iloc[:,:].values).astype('float32')\n\n#Training and Validation Split\ntrain_images, val_images, train_labels, val_labels = train_test_split(train_images, train_labels,\n                                                                     stratify=train_labels, random_state=123,\n                                                                     test_size=0.20)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we will reshape the image arrays to a single colour channel.","metadata":{}},{"cell_type":"code","source":"train_images = train_images.reshape(train_images.shape[0], 1, 28, 28)\nval_images = val_images.reshape(val_images.shape[0], 1, 28, 28)\ntest_images = test_images.reshape(test_images.shape[0], 1, 28, 28)\nprint('Now all images are squared with {dim1} x {dim2} pixels'\\\n      .format(dim1=train_images.shape[2], dim2=train_images.shape[2]))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Plot some image sampples","metadata":{}},{"cell_type":"code","source":"#train samples\nfor i in range(6, 9):\n    plt.subplot(330 + (i+1))\n    plt.imshow(train_images[i].squeeze(), cmap=plt.get_cmap('gray'))\n    plt.title(train_labels[i])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In order to use the data in PyTorch, we should convert them to tensors. We'll also normalise images.","metadata":{}},{"cell_type":"code","source":"#train\ntrain_images_tensor = torch.tensor(train_images)/255.0\ntrain_labels_tensor = torch.tensor(train_labels)\ntrain_tensor = TensorDataset(train_images_tensor, train_labels_tensor)\n\n#validation dataset\nval_images_tensor = torch.tensor(val_images)/255.0\nval_labels_tensor = torch.tensor(val_labels)\nvalidation_tensor = TensorDataset(val_images_tensor, val_labels_tensor)\n\n#test dataset\ntest_tensor = torch.tensor(test_images)/255.0\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we can define a [DataLoader](https://pytorch.org/tutorials/beginner/data_loading_tutorial.html). <br>\nWe will choose batch_size = 10 to load the data.","metadata":{}},{"cell_type":"code","source":"train_loader = torch.utils.data.DataLoader(dataset=train_tensor, batch_size=10 )\nvalidation_loader = torch.utils.data.DataLoader(dataset=validation_tensor, batch_size=10 )\n\n#test\ntest_loader = torch.utils.data.DataLoader(dataset=test_tensor, batch_size=10, shuffle=False)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Define a model\nIn order to make our custom model, we should make a class using nn.Module package. The class has two main parts:\n* A constructor, in which we will define a neural network with two layers.  \n* The Forward method, which takes in a tensor x and passes it through the operations we have defined in the constructor.\n\nWe are going to have 16 input channels and 32 output channels. \nWe will have 10 neurons, as we have 10 classes.","metadata":{}},{"cell_type":"code","source":"class CNN_batch(nn.Module):\n    \n    # Contructor\n    def __init__(self, out_1=16, out_2=32,number_of_classes=10):\n        # The Super instructor allows us to creates object from\n        # the package  nn.module inside the object without initialising it  explicitly.\n        super(CNN_batch, self).__init__()\n        self.cnn1 = nn.Conv2d(in_channels=1, out_channels=out_1, kernel_size=2, stride=1, padding=0)\n        self.conv1_bn = nn.BatchNorm2d(out_1)\n        self.maxpool1=nn.MaxPool2d(kernel_size=2, stride=1)\n        \n        self.cnn2 = nn.Conv2d(in_channels=out_1, out_channels=out_2, kernel_size=2, stride=1, padding=0)\n        self.conv2_bn = nn.BatchNorm2d(out_2)\n        self.maxpool2=nn.MaxPool2d(kernel_size=2, stride=1)\n        #We'll calculate the dimension of  the linear array using (M -k)/stride +1 at each layer, where M is the image dimension and\n        # K is the kernel size at the layer.\n        self.fc1 = nn.Linear(out_2 * 24 * 24, number_of_classes)\n        self.bn_fc1 = nn.BatchNorm1d(10)\n    \n    # Prediction\n    def forward(self, x):\n        x = self.cnn1(x)\n        x=self.conv1_bn(x)\n        x = torch.relu(x)\n        x = self.maxpool1(x)\n        x = self.cnn2(x)\n        x=self.conv2_bn(x)\n        x = torch.relu(x)\n        x = self.maxpool2(x)\n        x = x.view(x.size(0), -1)\n        x = self.fc1(x)\n        x=self.bn_fc1(x)\n        return x","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model = CNN_batch(out_1=16, out_2=32,number_of_classes=10)\n# To check the model's parameters\nmodel.parameters","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train the model\nWe will create a function to train our model iteratively.\n* We'll use nn.CrossEntropyLoss() to calculate the loss. \n* We'll use Stochastic Gradient Descent (SGD) optimiser for updating the model parameters.\n* We'll specify the learning rate of 0.1. \n* We'll run the training loop for 5 iterations.\n* At the end of each run, we'll make a prediction for the validation dataset and save the accuracy, so that we can evaluate and visualise the performance later.","metadata":{}},{"cell_type":"code","source":"import time\nstart_time = time.time()\n\ncost_list=[]\naccuracy_list=[]\nN_val=len(validation_tensor)\nlearning_rate =0.1\noptimizer = torch.optim.SGD(model.parameters(), lr = learning_rate)\ncriterion = nn.CrossEntropyLoss()\nn_epochs=5\nprint('epoch   accuracy   cost')\nfor epoch in range(n_epochs):\n    cost=0\n    \n    for x, y in train_loader:\n        model.train()\n        optimizer.zero_grad()\n        z = model(x)\n        loss = criterion(z, y)\n        #loss.backward() calculates dloss/dx for every parameter x with requires_grad=True, \n        #which is accumulated into x.grad for every parameter x. \n        loss.backward()\n        #optimizer.step updates  parameters based on the current gradient ( .grad attribute)\n        optimizer.step()\n        cost+=loss.item()\n    correct=0\n    \n    #perform a prediction on the validation  data \n    for x_test, y_test in validation_loader:\n        model.eval()\n        z = model(x_test)\n        _, yhat = torch.max(z.data, 1)\n        correct += (yhat == y_test).sum().item()\n    accuracy = correct / N_val\n    accuracy_list.append(accuracy)\n    cost_list.append(cost)\n    print ('{}   {:.3f}   {:.3f}'.format(epoch, accuracy, cost))\n    \nend_time = time.time()\nprint ('Execution time : %.3f (sec)' % (end_time - start_time))   ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Present results\nOnce the model has been evaluated, we can visualise the performance. Here the plot shows the accuracy vs cost at each epoch.","metadata":{}},{"cell_type":"code","source":"fig, ax1 = plt.subplots()\ncolor = 'tab:red'\nax1.plot(cost_list, color=color)\nax1.set_xlabel('epoch', color=color)\nax1.set_ylabel('Cost', color=color)\nax1.tick_params(axis='y', color=color)\n\n    \nax2 = ax1.twinx()  \ncolor = 'tab:blue'\nax2.set_ylabel('accuracy', color=color) \nax2.set_xlabel('epoch', color=color)\nax2.plot( accuracy_list, color=color)\nax2.tick_params(axis='y', color=color)\nfig.tight_layout()\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Make some predictions\nNow we can make some predictions for images in the test dataset.","metadata":{}},{"cell_type":"code","source":"predictions = []\nmodel.eval()\nfor x_test in test_loader:\n        z = model(x_test)\n        _, yhat = torch.max(z.data, 1)\n        yhat = yhat.numpy()\n        predictions.append(yhat)\n\n# reshape the prediction list, as it has been stored in batches of 10\npredictions = np.reshape(predictions, (1, len(test_df))).T        ","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission","metadata":{}},{"cell_type":"code","source":"SubmissionDf = pd.DataFrame({'ImageId': np.arange(1, len(predictions)+1), 'Label': predictions[:,0]})\n#SubmissionDf.shape\nSubmissionDf.to_csv(\"Submission.csv\", index=False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}