{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Introduction\nHi! 😉\n\nThere are a lot of notesbooks and tutorials for this digital recognizer competition. This notebook tries to summarize the various techniques out there and demonstrates the process that I learn to write my first neural network. Let's go!"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"## MNIST Data\nBefore writing the code, we need to understand the dataset. I recommend to read the data description from the contest page: [https://www.kaggle.com/c/digit-recognizer/data](http://) \n\nSome key points:\n- For each image: height 28 pixels * width 28 pixels = 784 pixels\n- Pixel value: from 0 (lightest) to 255 (darkest)\n- For the whole training dataset: 785 columns\n- The first column is label, the rest corresponds to one of the 784 pixels, so each row is a complete image\n- To convert each image's pixel coordinate (i, j) to pixel number x (from 0 to 783), do x = i * 28 + j"},{"metadata":{},"cell_type":"markdown","source":"Let's take a closer look to the data. If you have never tried data preprocessing, check out this course: [https://www.kaggle.com/learn/data-cleaning](http://)"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\n\n# load data\ntrain = pd.read_csv(\"../input/digit-recognizer/train.csv\")\ntest = pd.read_csv(\"../input/digit-recognizer/test.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# take a look of first several rows of the training dataset\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# check for missing data\ntrain.isnull().sum().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see most of the pixels are white (0 in value). Thus, this dataset consists white background with black numbers (I thought it is black background). There is no missing data, so we can continue."},{"metadata":{"trusted":true},"cell_type":"code","source":"# separate label and pixels for the training set\n# the testing set does not contain lables\nlabels = train[\"label\"]\npureimg_train = train.drop(labels = [\"label\"], axis = 1) # drop the label column\ndel train # no longer needed","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, we need to normalize training and testing data to make pixel value ranging from \\[0, 1\\], because CNN will converge faster on \\[0, 1\\]."},{"metadata":{"trusted":true},"cell_type":"code","source":"# normalize train and test\nnorm_train = pureimg_train/255\nnorm_test = test/255","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We randomly separate part of the training set as validation set. Here I choose 10% of the data to be validation set. You can choose whatever fraction you want. The reason that we can choose randomly is that the digits are evenly spread between 0 to 9. If we have an uneven amount of each digit, we cannot randomly split.\n\nValidation set is used for hyperparameter tuning."},{"metadata":{"trusted":true},"cell_type":"code","source":"# split the training data into training and validation set\nfrom sklearn.model_selection import train_test_split\n\nfeature_train, feature_validate, target_train, target_validate = train_test_split(norm_train, labels, test_size = 0.1, random_state = 0)\n\n# feature: non-label part of the image\n# target: what we want to get, so it is the label of the image\n# we should not reshape just yet because this function defaultly separates data row wise by axis 0\n# so the shape of 2D training data set array should be one image per row\n# otherwise it will be separated incorrectly","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# change data frame to numpy, and then to tensor form\n# time to reshape\nimport numpy as np\nimport torch\n\nTest = torch.from_numpy(norm_test.values.reshape((-1,1,28,28)))\nfeaturesTrain = torch.from_numpy(feature_train.values.reshape((-1,1,28,28)))\ntargetsTrain = torch.from_numpy(target_train.values)\nfeaturesValidation = torch.from_numpy(feature_validate.values.reshape((-1,1,28,28)))\ntargetsValidation = torch.from_numpy(target_validate.values)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data Augmentation\nIn order to have higher accuracy, we need more data. We can do data augmentation, rotation and filp of the existing images, to increase data set."},{"metadata":{"trusted":true},"cell_type":"code","source":"import torchvision\nimport torchvision.transforms as transforms\nfrom torch.utils.data import DataLoader, Dataset\n\n# define batch size\nbatch_size = 88","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# define own dataset\nclass MNISTDataset(Dataset):\n    \"\"\"MNIST dataset\"\"\"\n    \n    def __init__(self, feature, target=None, transform=None):\n        \n        self.X = feature\n        self.y = target\n            \n        self.transform = transform\n    \n    def __len__(self):\n        return len(self.X)\n\n    def __getitem__(self, idx):\n        # training\n        if self.transform is not None:\n            return self.transform(self.X[idx]), self.y[idx]\n        # testing\n        elif self.y is None:\n            return [self.X[idx]]\n        # validation\n        return self.X[idx], self.y[idx]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# define transform operation\ndata_transform = transforms.Compose([\n    transforms.ToPILImage(),\n    transforms.RandomAffine(degrees=45, translate=(0.1, 0.1), scale=(0.8, 1.2)),\n    transforms.ToTensor()])\n\n# create dataset\ntrain_set = MNISTDataset(featuresTrain.float(), targetsTrain, transform=data_transform)\nvalidate_set = MNISTDataset(featuresValidation.float(), targetsValidation)\ntest_set = MNISTDataset(Test.float())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# if choose not to do data augmentation\n# create dataset like this, move this cell to the end of the section before data loading\ntrain_set = torch.utils.data.TensorDataset(featuresTrain.float(), targetsTrain)\nvalidate_set = torch.utils.data.TensorDataset(featuresValidation.float(), targetsValidation)\ntest_set = torch.utils.data.TensorDataset(Test.float())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# load the data\ntrain_loader = torch.utils.data.DataLoader(train_set, batch_size = batch_size, shuffle = True)\nvalidate_loader = torch.utils.data.DataLoader(validate_set, batch_size = batch_size, shuffle = False)\ntest_loader = torch.utils.data.DataLoader(test_set, batch_size = batch_size, shuffle = False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## CNN\nWe are done with data. Let's build the neural network using torch.\n\nNetwork structure:\n1. Convolutional(Conv2D) layer: filter=32, activation=relu\n2. Convolutional(Conv2D) layer: filter=32, activation=relu\n3. Max pooling layer: kernel size=2, stride=1\n4. Dropout: rate=0.25\n5. Convolutional(Conv2D) layer: filter=64, activation=relu\n6. Convolutional(Conv2D) layer: filter=64, activation=relu\n7. Max pooling layer: kernel size=2, stride=2\n8. Dropout: rate=0.25\n9. Flatten layer\n10. Fully connected(Dense) layer\n11. Dropout: rate=0.5\n12. Fully connected(Dense) layer\n\n🤔There must be a ton of questions regarding this structure.\n\n*Q: What is a filter?* \n\nA: A filter performs dot product with each portion of the input image, so the portion of the most matched between input and a filter will have the highest value for the output. It means certain portion of the input contains the feature represented by the filter, so we call the output a feature map. The number of filters is the number of feature maps and the number of out channels.\n\n\n*Q: What is activation?* \n\nA: Activation function introduces nonlinearity to the neural network. It is a function that, if the output value meets the threshold, the nueron switches on and the value is passed through, otherwise the value is not passed through.\n\n*Q: What is an inplace relu?*\n\nA: \"Inplace\" means that it will modify the input directly, without allocating any additional output. It can sometimes slightly decrease the memory usage. For this network specifically, adding inplace makes the training accuracy increases from 92 to 99.9. Weird...\n\n*Q: What is max pooling?* \n\nA: Max pooling is a way of down sampling to reduce computational cost and make the feature map more robust. This operation calculates the maximum value for each patch, for example, 2\\*2 here, of the feature map.\n\n*Q: What is stride?* \n\nA: Stride is the number of pixels shifts over the input matrix. When the stride is 1 then we move the filters to 1 pixel at a time. When the stride is 2 then we move the filters to 2 pixels at a time and so on. \n\n*Q: What is dropout?* \n\nA: During training, some number of layer outputs are randomly ignored with certain probability, for example, 0.25 and 0.5 here. Dropout regularization is for reducing overfitting and improving the generalization of deep neural networks.\n\n*Q: What is a flattern layer?* \n\nA: This layer simply flattens the array into a 1-dimensional array for inputting it to the next layer.\n\n*Q: What is a dense layer?* \n\nA: A dense layer is a fully connected layer. The layer performs a linear operation on the layer’s input vector.\n\n*Q: How to calculate the dimension that goes into the first dense layer?* \n\nA: It is the last conv layer's out channels number times the size of 1 feature map. For this network specifically,it is 64\\*3\\*3.\n\n*Q: Why do you use this structure?* \n\nA: This is the simpliest structure with the highest accuracy that I can find among Kaggle notebooks."},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch.nn as nn\n\nclass CNNModel(nn.Module):\n    def __init__(self):\n        super(CNNModel, self).__init__()\n        \n        self.cnn = nn.Sequential(nn.Conv2d(in_channels=1, out_channels=32, kernel_size=5),\n                                     nn.ReLU(inplace=True),\n                                     nn.Conv2d(in_channels=32, out_channels=32, kernel_size=5),\n                                     nn.ReLU(inplace=True),\n                                     nn.MaxPool2d(kernel_size=2),\n                                     nn.Dropout(0.25),\n                                     nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3),\n                                     nn.ReLU(inplace=True),\n                                     nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3),\n                                     nn.ReLU(inplace=True),\n                                     nn.MaxPool2d(kernel_size=2, stride=2),\n                                     nn.Dropout(0.25))\n        \n        self.classifier = nn.Sequential(nn.Linear(576, 256),\n                                       nn.Dropout(0.5),\n                                       nn.Linear(256, 10))\n\n        \n    def forward(self, x):\n        x = self.cnn(x)\n        x = x.view(x.size(0), -1) # flatten layer\n        x = self.classifier(x)\n        \n        return x","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch.optim as optim\n\n# defining the model\nmodel = CNNModel()\n\n# set optimizer, loss function, and learning rate reduction\n# all these parameter comes from the first notebook in citation, the author explains the reason of choosing well\noptimizer = optim.RMSprop(model.parameters(), lr=0.001, alpha=0.9)\n\ncriterion = nn.CrossEntropyLoss()\n\nlr_reduction = optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=0.5, patience=3, threshold=0.0001, threshold_mode='rel', cooldown=0, min_lr=0.00001)\n\nif torch.cuda.is_available():\n    model = model.cuda()\n    criterion = criterion.cuda()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training and Evaluation"},{"metadata":{"trusted":true},"cell_type":"code","source":"# for visualization\ncount = 0\nloss_list = []\niteration_list = []\naverage_training_accuracy = []\naverage_validation_accuracy = []\naverage_training_loss = []\naverage_validation_loss = []","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from torch.autograd import Variable\n\ndef train(epoch):\n    # print('Epoch ', epoch) # uncomment me to show verbose\n    global count\n    model.train()\n\n    for batch_idx, (data, target) in enumerate(train_loader):\n        data, target = Variable(data), Variable(target)\n        \n        if torch.cuda.is_available():\n            data = data.cuda()\n            target = target.cuda()\n        \n        optimizer.zero_grad()\n        output = model(data)\n        loss = criterion(output, target)\n        \n        loss.backward()\n        optimizer.step()\n        \n        if (batch_idx + 1)% 100 == 0:\n            # store loss and iteration\n            loss_list.append(loss.item())\n            iteration_list.append(count)\n            count += 1\n            # print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(epoch, (batch_idx + 1) * len(data), len(train_loader.dataset), 100. * (batch_idx + 1) / len(train_loader), loss.item())) # uncomment me to show verbose","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch.nn.functional as F\n\ndef evaluate(data_loader, validate=False):\n    model.eval()\n    loss = 0\n    correct = 0\n    \n    for data, target in data_loader:\n        data, target = Variable(data), Variable(target)\n        if torch.cuda.is_available():\n            data = data.cuda()\n            target = target.cuda()\n        \n        output = model(data)\n        \n        loss += F.cross_entropy(output, target, size_average=False).item()\n\n        pred = output.data.max(1, keepdim=True)[1]\n        correct += pred.eq(target.data.view_as(pred)).cpu().sum()\n        \n    loss /= len(data_loader.dataset)\n    \n    accuracy = 100. * correct / len(data_loader.dataset)\n    \n    if not validate:\n        lr_reduction.step(loss)\n        average_training_accuracy.append(accuracy)\n        average_training_loss.append(loss)\n        # print('Average training loss: {:.4f}, Accuracy: {}/{} ({:.3f}%)'.format(loss, correct, len(data_loader.dataset), accuracy)) # uncomment me to show verbose\n    else:\n        average_validation_accuracy.append(accuracy)\n        average_validation_loss.append(loss)\n        # print('Average validation loss: {:.4f}, Accuracy: {}/{} ({:.3f}%)\\n'.format(loss, correct, len(data_loader.dataset), accuracy)) # uncomment me to show verbose","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import matplotlib.pyplot as plt\n\nn_epochs = 50\n# without data augmentation reaches 99.3, with augmentation reaches 99.2\n\nfor epoch in range(n_epochs):\n    train(epoch)\n    evaluate(train_loader)\n    evaluate(validate_loader, True)\n    \nplt.plot(iteration_list,loss_list)\nplt.xlabel(\"Number of iteration\")\nplt.ylabel(\"Loss\")\nplt.title(\"Training Loss vs Number of iteration\")\nplt.show()\n\nepoch_list = [i for i in range(n_epochs)]\n\nplt.plot(epoch_list, average_training_loss)\nplt.plot(epoch_list, average_validation_loss)\nplt.xlabel(\"Epoch\")\nplt.ylabel(\"Loss\")\nplt.title(\"Loss vs Epoch\")\nplt.show()\n\nplt.plot(epoch_list, average_training_accuracy)\nplt.plot(epoch_list, average_validation_accuracy)\nplt.xlabel(\"Epoch\")\nplt.ylabel(\"Accuracy\")\nplt.title(\"Accuracy vs Epoch\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prediction"},{"metadata":{"trusted":true},"cell_type":"code","source":"def prediciton(data_loader):\n    model.eval()\n    test_pred = torch.LongTensor()\n    \n    for i, data in enumerate(data_loader):\n        \n        data = Variable(data[0])\n        if torch.cuda.is_available():\n            data = data.cuda()\n            \n        output = model(data)\n        \n        pred = output.cpu().data.max(1, keepdim=True)[1]\n        test_pred = torch.cat((test_pred, pred), dim=0)\n        \n    return test_pred\n\ntest_pred = prediciton(test_loader)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"out_df = pd.DataFrame(np.c_[np.arange(1, len(test_set)+1)[:,None], test_pred.numpy()], \n                      columns=['ImageId', 'Label'])\n\n# check prediction makes sense\nout_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"out_df.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If this notebook helps you, please upvote and tell me!❤️\nIf you have any question or there is a bug, please tell me!🤔\nI will continue to make notebooks and learn with you😉"},{"metadata":{},"cell_type":"markdown","source":"## Citation\n- Notebook CNN with PyTorch (0.995 Accuracy) [https://www.kaggle.com/juiyangchang/cnn-with-pytorch-0-995-accuracy](http://)\n- Notebook from Yassine Ghouzam, PhD, Introduction to CNN Keras - 0.997 (top 6%) [https://www.kaggle.com/yassineghouzam/introduction-to-cnn-keras-0-997-top-6](http://)\n- Notebook Pytorch Tutorial for Deep Learning Lovers [https://www.kaggle.com/kanncaa1/pytorch-tutorial-for-deep-learning-lovers](http://)\n- How Do Convolutional Layers Work in Deep Learning Neural Networks? [https://machinelearningmastery.com/convolutional-layers-for-deep-learning-neural-networks/#:~:text=Convolutional%20layers%20are%20the%20major,that%20results%20in%20an%20activation.&text=The%20result%20is%20highly%20specific,detected%20anywhere%20on%20input%20images.](http://)\n- A Gentle Introduction to Pooling Layers for Convolutional Neural Networks [https://machinelearningmastery.com/pooling-layers-for-convolutional-neural-networks/](http://)\n- A Gentle Introduction to Dropout for Regularizing Deep Neural Networks [https://machinelearningmastery.com/dropout-for-regularizing-deep-neural-networks/](http://)\n- Determining size of FC layer after Conv layer in PyTorch [https://datascience.stackexchange.com/questions/40906/determining-size-of-fc-layer-after-conv-layer-in-pytorch](http://)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}