{"cells":[{"metadata":{},"cell_type":"markdown","source":"We have been given datasets containing images of digits. These digits may be written in multiple styles. However, these images while being different in style will all be representing positive whole single digit numbers only.\n<br><br>\nOur objective is to correctly recognise the number represented/written within any image in the test dataset."},{"metadata":{},"cell_type":"markdown","source":"**__This notebook is based mostly off of the [notebook by JuiyangChang](https://www.kaggle.com/juiyangchang/cnn-with-pytorch-0-995-accuracy/notebook)__**<br><br>\n\n***It is divided into the following parts:***<br><br>\n**1. Importing packages and the datasets<br><br>2. Data Visualisation<br><br>3. Pre-processing and Data Loading <br><br>4. Defining Training Model and Details<br><br>5. Prediction**"},{"metadata":{},"cell_type":"markdown","source":"# **1. Importing Packages and the Datasets**"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport random\nimport math\nimport seaborn as sns\nfrom PIL import Image, ImageOps, ImageEnhance\nimport numbers\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport matplotlib.image as mpimg\n\n\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.optim import lr_scheduler\nfrom torch.autograd import Variable\nfrom torch.utils.data import DataLoader, Dataset\nfrom torchvision import transforms\nfrom torchvision.utils import make_grid\n\n\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Import all the csv data into data frame variables"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"/kaggle/input/digit-recognizer/train.csv\")\ntest = pd.read_csv(\"/kaggle/input/digit-recognizer/test.csv\")\nsample_sub = pd.read_csv(\"/kaggle/input/digit-recognizer/sample_submission.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **2. Data Visualisation**"},{"metadata":{},"cell_type":"markdown","source":"Few Key Metrics to be observed before using the Training Dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"n_train = len(train)\nn_pixels = len(train.columns) - 1\nn_class = len(set(train['label']))\n\nprint('Number of training samples: {0}'.format(n_train))\nprint('Number of training pixels: {0}'.format(n_pixels))\nprint('Number of classes: {0}'.format(n_class))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"n_test = len(test)\nn_pixels = len(test.columns)\n\nprint('Number of train samples: {0}'.format(n_test))\nprint('Number of test pixels: {0}'.format(n_pixels))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Observe the number of samples associated with each label"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.rcParams['figure.figsize'] = (8, 5)\nplt.bar(train['label'].value_counts().index, train['label'].value_counts())\nplt.xticks(np.arange(n_class))\nplt.xlabel('Class', fontsize=16)\nplt.ylabel('Count', fontsize=16)\nplt.grid('on', axis='y')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Display some images"},{"metadata":{"trusted":true},"cell_type":"code","source":"random_sel = np.random.randint(n_train, size=8)\n\ngrid = make_grid(torch.Tensor((train.iloc[random_sel, 1:].values/255.).reshape((-1, 28, 28))).unsqueeze(1), nrow=8)\nplt.rcParams['figure.figsize'] = (16, 2)\nplt.imshow(grid.numpy().transpose((1,2,0)))\nplt.axis('off')\nprint(*list(train.iloc[random_sel, 0].values), sep = ', ')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **3. Pre-processing and Data Loading**"},{"metadata":{},"cell_type":"markdown","source":"Our formal **MNIST Dataset Loader.**\n\nWe define the following class to:<br><br>*a. Load the training and testing dataset.<br>b. Index each value of the dataset<br>c. Define the point of largest index(size of the set). Step c. happens before Step b. in our class.*\n\nHave a look at **[this  Stanford blog article](https://stanford.edu/~shervine/blog/pytorch-how-to-generate-data-parallel)** and this **[PyTorch tutorial](https://pytorch.org/tutorials/beginner/data_loading_tutorial.html#dataset-class)** that explains the inclusion of our dataloader class.\n\nWe have added in the __init__ function to perform some transforms on every sample, that make the image easier to process.<br><br>\nThis is a part of the pre-processing."},{"metadata":{"trusted":true},"cell_type":"code","source":"class MNIST_dataset(Dataset):\n    \n    def __init__(self, file_path, \n                 transform = transforms.Compose([transforms.ToPILImage(), transforms.ToTensor(), \n                     transforms.Normalize(mean=(0.5,), std=(0.5,))])\n                ):\n        \n        df = pd.read_csv(file_path)\n        \n        if len(df.columns) == n_pixels:\n            # test data\n            self.X = df.values.reshape((-1,28,28)).astype(np.uint8)[:,:,:,None]\n            self.y = None\n        else:\n            # training data\n            self.X = df.iloc[:,1:].values.reshape((-1,28,28)).astype(np.uint8)[:,:,:,None]\n            self.y = torch.from_numpy(df.iloc[:,0].values)\n            \n        self.transform = transform\n    \n    def __len__(self):\n        return len(self.X)\n\n    def __getitem__(self, idx):\n        if self.y is not None:\n            return self.transform(self.X[idx]), self.y[idx]\n        else:\n            return self.transform(self.X[idx])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# ***Data Augmentation Techniques***"},{"metadata":{},"cell_type":"markdown","source":"This modifies every image of the image dataset by performing certain transformation techniques to increase the size and variety of the image dataset while using the available data.<br><br>The following classes describe and perform the augmentation.<br><br>This __[PyTorch repository](https://github.com/pytorch/vision/blob/master/torchvision/transforms/transforms.py)__ is the source of the data augmentation codes"},{"metadata":{},"cell_type":"markdown","source":"### Random Rotation Transformation\nRandomly rotate the image.\n<br><br>For the training set,we may apply a random rotation within the range of (-45, 45) degrees"},{"metadata":{"trusted":true},"cell_type":"code","source":"class RandomRotation(object):\n    \n    def __init__(self, degrees, resample=False, expand=False, center=None):\n        if isinstance(degrees, numbers.Number):\n            if degrees < 0:\n                raise ValueError(\"If degrees is a single number, it must be positive.\")\n            self.degrees = (-degrees, degrees)\n        else:\n            if len(degrees) != 2:\n                raise ValueError(\"If degrees is a sequence, it must be of len 2.\")\n            self.degrees = degrees\n\n        self.resample = resample\n        self.expand = expand\n        self.center = center\n\n    @staticmethod\n    def get_params(degrees):\n       \n        angle = np.random.uniform(degrees[0], degrees[1])\n\n        return angle\n\n    def __call__(self, img):\n       \n        def rotate(img, angle, resample=False, expand=False, center=None):\n                          \n            return img.rotate(angle, resample, expand, center)\n\n        angle = self.get_params(self.degrees)\n\n        return rotate(img, angle, self.resample, self.expand, self.center)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Random Vertical and Horizontal Shift\nFor training set we may apply shift by (-3, 3) pixels"},{"metadata":{"trusted":true},"cell_type":"code","source":"class RandomShift(object):\n    def __init__(self, shift):\n        self.shift = shift\n        \n    @staticmethod\n    def get_params(shift):\n       \n        hshift, vshift = np.random.uniform(-shift, shift, size=2)\n\n        return hshift, vshift \n    def __call__(self, img):\n        hshift, vshift = self.get_params(self.shift)\n        \n        return img.transform(img.size, Image.AFFINE, (1,0,hshift,0,1,vshift), resample=Image.BICUBIC, fill=1)    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Mini Batch Gradient Descent** is a best of two worlds kind of methodology to train the neural network.<br><br> This [article](https://datascience.stackexchange.com/a/16818) explains and justifies the technique. It also recommends smaller mini batch sizes, however, that may be computationally heavy, and may require the [use of a GPU](https://towardsdatascience.com/what-is-a-gpu-and-do-you-need-one-in-deep-learning-718b9597aa0d)"},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 256","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"\"pp\" used to denote pre-processed. The training dataset is trasnformed and augmented and is ready to be used for iteration through the NN after this step."},{"metadata":{},"cell_type":"markdown","source":" For training set normalize pixel values to [-1, 1].  For the test set, we only apply nomalization."},{"metadata":{"trusted":true},"cell_type":"code","source":"pp_train = MNIST_dataset('/kaggle/input/digit-recognizer/train.csv', transform= transforms.Compose(\n                            [transforms.ToPILImage(), RandomRotation(degrees=20), RandomShift(3),\n                             transforms.ToTensor(), transforms.Normalize(mean=(0.5,), std=(0.5,))]))\npp_test = MNIST_dataset('/kaggle/input/digit-recognizer/test.csv')\n                            \ntrainset_loader = torch.utils.data.DataLoader(dataset=pp_train, batch_size=batch_size, shuffle=True)\ntestset_loader = torch.utils.data.DataLoader(dataset=pp_test, batch_size=batch_size, shuffle=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Visualising the transforms on a sample image"},{"metadata":{"trusted":true},"cell_type":"code","source":"rotate = RandomRotation(20)\nshift = RandomShift(3)\ncomposed = transforms.Compose([RandomRotation(20),\n                               RandomShift(3)])\n\n# Apply each of the above transforms on sample.\nfig = plt.figure()\nsample = transforms.ToPILImage()(train.iloc[65,1:].values.reshape((28,28)).astype(np.uint8)[:,:,None])\nfor i, tsfrm in enumerate([rotate, shift, composed]):\n    transformed_sample = tsfrm(sample)\n\n    ax = plt.subplot(1, 3, i + 1)\n    plt.tight_layout()\n    ax.set_title(type(tsfrm).__name__)\n    ax.imshow(np.reshape(np.array(list(transformed_sample.getdata())), (-1,28)), cmap='gray')    \n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **4. Defining Training Model and Details**"},{"metadata":{},"cell_type":"markdown","source":"**Defining the Neural Network**<br><br>CNNs are used in computer vision techniques because they preserve the features of the image over the network.<br>\nAnswers listed on [this page](https://www.quora.com/Why-are-CNNs-used-more-for-computer-vision-tasks-than-other-tasks) should help explain the exact reason for this. "},{"metadata":{},"cell_type":"markdown","source":"Class CNN is used to define the neural network framework"},{"metadata":{"trusted":true},"cell_type":"code","source":"class CNN(nn.Module):\n    def __init__(self):\n        super(CNN, self).__init__()\n        \n        self.feature_extract = nn.Sequential(\n        nn.Conv2d(1, 32, 5), nn.ReLU(inplace=True), nn.MaxPool2d(2,2),  \n        nn.Conv2d(32, 64, 5), nn.ReLU(inplace=True), nn.MaxPool2d(2,2)) \n        #Layer1: 28-5+1=24*24*32 12*12*6\n        #Layer2: 12-5+1=8*8*64 4*4*64\n            \n        \n        self.classification = nn.Sequential(\n        nn.Dropout(p=0.5),\n        nn.Linear(4*4*64, 1024), nn.ReLU(inplace=True), nn.Dropout(p=0.5),\n        nn.Linear(1024, 120), nn.ReLU(inplace=True), nn.Dropout(p=0.5),\n        nn.Linear(120, 10))\n        \n        \n    def forward(self, x):\n            x = self.feature_extract(x)\n            x = x.view(x.size(0), -1)\n            x = self.classification(x)\n            \n            return x","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Defining the model variable, the optimization function, the loss function and learning rate scheduler for our training process. These parameters including the batch size defined earlier are ***hyperparamters***"},{"metadata":{"trusted":true},"cell_type":"code","source":"model = CNN()\noptimizer = optim.Adam(model.parameters(), lr = 0.01)\nloss_fn = nn.CrossEntropyLoss()\nexp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)\n\nif torch.cuda.is_available(): #Determines if the system supports CUDA\n    model = model.cuda()\n    loss_fn = loss_fn.cuda()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"CUDA is an API that helps in fatser training of our CNN by roping in the processing power of GPUs(if available)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def train(epoch):\n    model.train()\n    \n    for batch_idx, (data, target) in enumerate(trainset_loader):\n        data, target = Variable(data), Variable(target)\n        \n        if torch.cuda.is_available():\n            data = data.cuda()\n            target = target.cuda()\n        \n        optimizer.zero_grad()\n        output = model(data)\n        loss = loss_fn(output, target)\n        \n        loss.backward()\n        optimizer.step()\n        \n        if (batch_idx + 1)% 100 == 0:\n            print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n                epoch, (batch_idx + 1) * len(data), len(trainset_loader.dataset),\n                100. * (batch_idx + 1) / len(trainset_loader), loss.data))\n    \n    exp_lr_scheduler.step()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def evaluate(data_loader):\n    model.eval()\n    loss = 0\n    correct = 0\n    \n    for data, target in data_loader:\n        data, target = Variable(data, volatile=True), Variable(target)\n        if torch.cuda.is_available():\n            data = data.cuda()\n            target = target.cuda()\n        \n        output = model(data)\n        \n        loss += F.cross_entropy(output, target, size_average=False).data\n\n        pred = output.data.max(1, keepdim=True)[1]\n        correct += pred.eq(target.data.view_as(pred)).cpu().sum()\n        \n    loss /= len(data_loader.dataset)\n        \n    print('\\nAverage loss: {:.4f}, Accuracy: {}/{} ({:.3f}%)\\n'.format(\n        loss, correct, len(data_loader.dataset),\n        100. * correct / len(data_loader.dataset)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Epochs are the number of times a network passes over the The source notebook suggests that we take as many as 50 epochs to achieve a 99.5% accuracy. However, I have used a different network and 50 epochs can take a lot of computing time, even if you use the GPU allocated by Kaggle."},{"metadata":{"trusted":true},"cell_type":"code","source":"n_epochs = 50\n\nfor epoch in range(n_epochs):\n    train(epoch)\n    evaluate(trainset_loader)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def prediciton(data_loader):\n    model.eval()\n    test_pred = torch.LongTensor()\n    \n    for i, data in enumerate(data_loader):\n        data = Variable(data, volatile=True)\n        if torch.cuda.is_available():\n            data = data.cuda()\n            \n        output = model(data)\n        \n        pred = output.cpu().data.max(1, keepdim=True)[1]\n        test_pred = torch.cat((test_pred, pred), dim=0)\n        \n    return test_pred","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_pred = prediciton(testset_loader)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"out_df = pd.DataFrame(np.c_[np.arange(1, len(pp_test)+1)[:,None], test_pred.numpy()], \n                      columns=['ImageId', 'Label'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"out_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"out_df.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}