{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Introduction\nIn this competition, The goal is to correctly identify digits from a dataset of tens of thousands of handwritten images. \n\n## Dataset\nFor this competition, we will be using the popular MNIST database. It is a collection of 70.000 handwritten digits split into training and test set of 42.000 and 28.000 images respectively."},{"metadata":{},"cell_type":"markdown","source":"## Setting up the Environment\nWe will be using PyTorch to train a convolutional neural network to recognize MNIST's handwritten digits."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\n\nfrom PIL import Image\nfrom sklearn.model_selection import train_test_split\nimport math\nimport copy\nimport time\n\nimport torch\nimport torchvision\nfrom torch.utils.data import DataLoader, Dataset\nfrom torchvision import transforms\nfrom torch.utils.data import DataLoader, TensorDataset\nfrom torchvision.utils import make_grid\n\n#neural net imports\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nfrom torch.autograd import Variable\n\n%matplotlib inline\nprint(torch.__version__)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Preparing the Dataset\n\nwe'll define the hyperparameters we'll be using for the experiment. Here the number of epochs defines how many times we'll loop over the complete training dataset, while learning_rate is the hyperparameter for the optimizer we'll be using later on.\n\nWe read both training and test set into dataframe then we split our training dataset into train and valid dataset in order to train and test our model. Train/Valid ratio is 15%."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"random_seed = 42\ntorch.backends.cudnn.enabled = False\ntorch.manual_seed(random_seed)\n\n# Load the data\ntrain_df = pd.read_csv(\"../input/digit-recognizer/train.csv\")\ntest_df = pd.read_csv(\"../input/digit-recognizer/test.csv\")\n\ny = train_df[\"label\"]\nx = train_df.drop(\"label\", axis = 1)\n\n#Split training data into Train and validation set\nX_train, X_valid, y_train, y_valid = train_test_split(x, y, test_size=0.15, shuffle=True)\n\nnum_epoch = 25\nbatch_size_train = 32\nbatch_size_test = 32\nlearning_rate = 0.002\nmomentum = 0.9\nlog_interval = 100","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The training dataset has 28x28 = 784 dimensions features and a column of label. The test dataset has 784-D data. Next we'll define Custom Dataset in order to do convert to tensor, do some transitions abd process data in mini-batches. Dataset takes two dataframe data. "},{"metadata":{"trusted":true},"cell_type":"code","source":"#CustomDatasetFromDF\nclass MNISTDataset(Dataset):\n    def __init__(self,  data, target, train=True, transform=None):\n        \"\"\"\n        Args:\n            csv_path (string): path to csv file\n            transform: pytorch transforms for transforms and tensor conversion\n        \"\"\"\n        self.train = train\n        if self.train :\n            self.data = data\n            self.labels = np.asarray(target.iloc[:])\n        else:\n            self.data = data\n            self.labels = None\n        self.height = 28 # Height of image\n        self.width = 28 # Width of image\n        self.transform = transform\n\n    def __getitem__(self, index):\n        # Read each 784 pixels and reshape the 1D array ([784]) to 2D array ([28,28])\n        img_as_np = np.asarray(self.data.iloc[index][0:]).reshape(self.height, self.width).astype('uint8')\n        # Convert image from numpy array to PIL image, mode 'L' is for grayscale\n        img_as_img = Image.fromarray(img_as_np)\n        img_as_img = img_as_img.convert('L')\n        img_as_tensor = img_as_img\n        \n        if self.train:\n            single_image_label = self.labels[index]\n        else:\n            single_image_label = None\n            \n        # Transform image to tensor\n        if self.transform is not None:\n            img_as_tensor = self.transform(img_as_img)\n        \n        if self.train:\n        # Return image and the label                \n            return (img_as_tensor, single_image_label)\n        else:\n            return img_as_tensor\n    \n    def __len__(self):\n        return len(self.data.index)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"PyTorch DataLoaders are objects that act as Python generators. They supply data in chunks or batches while training and validation.  \nOur loader will behave like an iterator, so we can loop over it and fetch a different mini-batch every time. For each \n\n### Data Augmentation\nData augmentation is a common deep learning technique where we modify images on the fly while training the neural network to see additional images flipped or rotated at different axes and angles. This usually results in better training performance since the network sees multiple views of the same image and has a better chance of identifying its class when minimizing the loss function.\nWe use following augmentation:\n* RandomRotation at a specific degree (15 in our case below) that rotates some of them randomly at an angle of 15 degree again with a probability of p which defaults to 0.5.\n\n### Data Normalization\nIn data normalization, we statistically normalize the pixel values in our images. This mostly results in better training performance and faster convergence. A common way to perform normalization is to subtract the mean of pixel values of the whole dataset from each pixel, and then divide by the standard deviation of the pixels of the whole dataset.\n\nThe values 0.1310 and 0.3085 used for the Normalize() transformation below are the global mean and standard deviation of the MNIST dataset. Following code is used to calculate mean/std dev of dataset. It take dataloader as input.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"def calculate_img_stats_full(dataset):\n    imgs_ = torch.stack([img for img,_ in dataset],dim=1)\n    imgs_ = imgs_.view(1,-1)\n    imgs_mean = imgs_.mean(dim=1)\n    imgs_std = imgs_.std(dim=1)\n    return imgs_mean,imgs_std\n\n#transformations_org = transforms.Compose([transforms.ToTensor()])\n#train_org = MNISTDataset(x, y, True, transformations_org)\n\n#calculate_img_stats_full(train_org)\n# (tensor([0.1310]), tensor([0.3085]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now we can create our datasets again from scratch with all the transformations, augmentations, and normalization applied—splitting them into train and validation, and obtaining the final DataLoaders."},{"metadata":{"trusted":true},"cell_type":"code","source":"transformations_train = transforms.Compose([transforms.RandomRotation(15),                                       \n                                            transforms.RandomAffine(0, shear=10, scale=(0.8,1.2)),\n                                            transforms.ToTensor(),\n                                            transforms.Normalize(mean=[0.1310], std=[0.3085])\n                                           ])\n\n\ntransformations_valid = transforms.Compose([transforms.ToTensor(),\n                                            transforms.Normalize(mean=[0.1310], std=[0.3085])\n                                           ])\n\ntrain = MNISTDataset(X_train, y_train, True, transformations_train)\nvalid = MNISTDataset(X_valid, y_valid, True, transformations_valid)\ntest  = MNISTDataset(data=test_df, target=None, train=False, transform=transformations_valid)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_loader = DataLoader(train, batch_size=batch_size_train,num_workers=2, shuffle=True)\nvalid_loader = DataLoader(valid, batch_size=batch_size_test, num_workers=2, shuffle=True)\ntest_loader  = DataLoader(test,  batch_size=batch_size_test, shuffle=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Building the Network"},{"metadata":{"trusted":true},"cell_type":"code","source":"class Net(nn.Module):\n    def __init__(self):\n        super(Net, self).__init__()\n        self.conv_block = nn.Sequential(\n            nn.Conv2d(1, 32, kernel_size=3, stride=1, padding=1),\n            nn.BatchNorm2d(32),\n            nn.ReLU(inplace=True),\n            nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1),\n            nn.BatchNorm2d(64),\n            nn.ReLU(inplace=True),\n            nn.MaxPool2d(kernel_size=2, stride=2),\n            nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1),\n            nn.BatchNorm2d(128),\n            nn.ReLU(inplace=True),\n            nn.MaxPool2d(kernel_size=2, stride=2) \n        )\n        \n        self.linear_block = nn.Sequential(\n            nn.Dropout(p=0.5),\n            nn.Linear(128*7*7, 128),\n            nn.BatchNorm1d(128),\n            nn.ReLU(inplace=True),\n            nn.Dropout(0.5),\n            nn.Linear(128, 64),\n            nn.BatchNorm1d(64),\n            nn.ReLU(inplace=True),\n            nn.Dropout(0.5),\n            nn.Linear(64, 10)\n        )\n\n    def forward(self, x):\n        x = self.conv_block(x)\n        x = x.view(x.size(0), -1)\n        x = self.linear_block(x)\n        return x","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Training the Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"cnn_model = Net()    \ncriterion = nn.CrossEntropyLoss()\n\nif torch.cuda.is_available():\n    cnn_model.cuda()\n    criterion.cuda()                       \n\noptimizer = optim.Adam(params=cnn_model.parameters(), lr=learning_rate)    \n\nexp_lr_scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min')\n\ntrain_losses = []\ntrain_counter = []\ntest_losses = []\ntest_counter = [i*len(train_loader.dataset) for i in range(1, num_epoch + 1)]    \n\nbest_model_wts = copy.deepcopy(cnn_model.state_dict())\nbest_acc = 0.0\n\nsince = time.time()\n\nfor epoch in range(1, num_epoch + 1):\n    cnn_model.train()    \n    for i, (images, labels) in enumerate(train_loader):\n        images = Variable(images).cuda()\n        labels = Variable(labels).cuda()\n        # Clear gradients\n        optimizer.zero_grad()\n        # Forward pass\n        outputs = cnn_model(images)\n        # Calculate loss\n        loss = criterion(outputs, labels)\n        # Backward pass\n        loss.backward()\n        # Update weights\n        optimizer.step()\n        if (i + 1)% log_interval == 0:\n            print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n                epoch, (i + 1) * len(images), len(train_loader.dataset),\n                100. * (i + 1) / len(train_loader), loss.data))\n            train_losses.append(loss.item())\n            train_counter.append((i*64) + ((epoch-1)*len(train_loader.dataset)))\n    cnn_model.eval()    \n    loss = 0    \n    running_corrects = 0\n    with torch.no_grad():       \n        for i, (data, target) in enumerate(valid_loader):\n            data = Variable(data).cuda()\n            target = Variable(target).cuda()\n            output = cnn_model(data)\n            loss += F.cross_entropy(output, target, reduction='sum').item()            \n            _, preds = torch.max(output, 1)            \n            running_corrects += torch.sum(preds == target.data)\n    loss /= len(valid_loader.dataset)\n    test_losses.append(loss)\n    epoch_acc = 100. * running_corrects.double() / len(valid_loader.dataset)\n    print('\\nAverage Val Loss: {:.4f}, Val Accuracy: {}/{} ({:.3f}%)\\n'.format(\n        loss, running_corrects, len(valid_loader.dataset), epoch_acc))\n    if epoch_acc > best_acc:\n        best_acc = epoch_acc\n        best_model_wts = copy.deepcopy(cnn_model.state_dict())\n    exp_lr_scheduler.step(loss)\n             \ntime_elapsed = time.time() - since\nprint('Training complete in {:.0f}m {:.0f}s'.format(time_elapsed // 60, time_elapsed % 60))\nprint('Best val Acc: {:4f}'.format(best_acc))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Evaluating the Model's Performance"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig = plt.figure()\nplt.plot(train_counter, train_losses, color='blue')\nplt.scatter(test_counter, test_losses, color='red')\nplt.legend(['Train Loss', 'Test Loss'], loc='upper right')\nplt.xlabel('number of training examples seen')\nplt.ylabel('negative log likelihood loss')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Test the Model"},{"metadata":{"trusted":true},"cell_type":"code","source":"cnn_model.eval()\ntest_preds = None\ntest_preds = torch.LongTensor()\n    \nfor i, data in enumerate(test_loader):\n    data = Variable(data).cuda()   \n    output = cnn_model(data)\n    preds = output.cpu().data.max(1, keepdim=True)[1]\n    test_preds = torch.cat((test_preds, preds), dim=0)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"out_df = pd.DataFrame({'ImageId':np.arange(1, len(test_loader.dataset)+1), 'Label':test_preds.numpy().squeeze()})\nout_df.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}