{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"collapsed":true},"cell_type":"markdown","source":"# Introduction"},{"metadata":{"_uuid":"47ba058e3ee90950e875f0ce81ebb5921280222c"},"cell_type":"markdown","source":"In this competition, we are given a dataset of handwritten images . The handwritten images are of digits from 0 to 9. We have to then submit the labels file for the new test images.\nWe will be creating a convolutional neural network and then train it for 20 epochs. I will also do some data augmentation, and also apply learning rate annealing. "},{"metadata":{"_uuid":"0b34ef519f68344de0e8853f95f0a56b6647052a"},"cell_type":"markdown","source":"# Loading the required libraries. "},{"metadata":{"trusted":true,"_uuid":"35448c06ccce16761d84fee1990b14b550682534","collapsed":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport os\n\nimport warnings\nwarnings.filterwarnings('ignore')\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"781d9c68224046c3d052198295370bdbfaade488"},"cell_type":"code","source":"from torch.utils.data.dataset import Dataset\nfrom torchvision import transforms\nimport torch\nimport torch.nn as nn\nfrom PIL import Image\nfrom torch.autograd import Variable\nimport torch.nn.functional as F\nfrom sklearn.model_selection import train_test_split\nfrom torch.utils.data.sampler import SubsetRandomSampler","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"404e65b07714497dc2e47608a322cddd63886492"},"cell_type":"markdown","source":"# Reading and Viewing the input data files. "},{"metadata":{"trusted":true,"_uuid":"834620b1823702eb3ea02972a403d487f3d5dd03","collapsed":true},"cell_type":"code","source":"df_train = pd.read_csv('../input/train.csv')\ndf_test = pd.read_csv('../input/test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e00bf9ea561faf48d14933da9a14d043e36da874"},"cell_type":"code","source":"print(\"Number of rows in the traning data are : \", df_train.shape[0])\nprint(\"Number of rows in the test data are : \", df_test.shape[0])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4d20ff198d8a73049b7a3b249601b41f73224e24"},"cell_type":"markdown","source":"Viewing some rows of the training data and test data"},{"metadata":{"trusted":true,"_uuid":"574d0af1e6702c162ce4d369eb063c6215645b55"},"cell_type":"code","source":"df_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1dbb2766b49747dfc857eeaf0e96127d91ea1e88"},"cell_type":"code","source":"df_test.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"caa8edf3f800841e9d9e652c3041d53da5461cf9"},"cell_type":"markdown","source":"So, we can see that image pixels are given in the form of a 1-D array for each image of dimension 784. Also train dataset has label column which gives us the label for the image.  "},{"metadata":{"trusted":true,"_uuid":"ca8ff24ecb2eab0a9a49e28ba02e69e6313c5a5c"},"cell_type":"code","source":"y_train = df_train['label']\nx_train = df_train.drop('label', axis=1)\nx_train = x_train.as_matrix()\nprint(\"Shape of the training data is -:\", x_train.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"18434acd17dadf793088ed7408998a5db4b79555"},"cell_type":"code","source":"fig = plt.figure(figsize=(8, 8))\ncolumns = 5\nrows = 2\nfor i in range(1, columns*rows +1):\n    img = np.array(x_train[i].reshape(28, 28))\n    fig.add_subplot(rows, columns, i)\n    plt.imshow(img, cmap='gray')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3431914ab42e14c99708f05624910634ccc867cd"},"cell_type":"markdown","source":"so, we can see that the images are of number and are grayscale."},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"4ad880964b2433170acd4979d4b2ebe694783ec1"},"cell_type":"markdown","source":"# Creating custom Data Loaders for the MNIST dataset "},{"metadata":{"_uuid":"9c8e67129d3259df2f9f94780ff55bc4aa9b5c67"},"cell_type":"markdown","source":"We need to create a custom data loader in pytorch for loading the dataset. Our custom dataset should inherit Dataset and override the methods : \n __len__ so that len(dataset) returns the size of the dataset.\n __getitem__ to support the indexing such that dataset[i] can be used to get i\nWe will read the csv in __init__ method\n\nMore about this can be read from-:\n1) https://pytorch.org/tutorials/beginner/data_loading_tutorial.html\n2) https://github.com/utkuozbulak/pytorch-custom-dataset-examples"},{"metadata":{"trusted":true,"_uuid":"27b427a196c29ab6d9381fc9b4de53df4b366ae7","collapsed":true},"cell_type":"code","source":"class CustomDatasetFromCSV_new(Dataset):\n    \n    # In this we read the dataset csv file and then converting the training data file into train and label file. \n    def __init__(self, csv_path, transforms=None, is_test=False):\n        self.data = pd.read_csv(csv_path)\n        self.is_test = is_test\n        if not self.is_test:\n            self.labels = np.asarray(self.data.iloc[:, 0])\n            self.data = self.data.iloc[:, 1:]\n        # Declaring torchvision transforms. more about these can be read here https://pytorch.org/docs/0.2.0/torchvision/transforms.html\n        self.transforms = transforms\n        \n    \n    # This function returns an image and label from the dataset.\n    def __getitem__(self, index):\n        \n        # Reshaping the image to size 28*28\n        img_as_np = np.array(np.reshape(self.data.iloc[index], (28, 28))).astype(np.uint8)\n        img_as_img = Image.fromarray(img_as_np)        \n        #Converting the image to black and white form. \n        img_as_img = img_as_img.convert('L')\n        # Applying torchvision transforms.\n        if self.transforms is not None:\n            img_as_tensor = self.transforms(img_as_img)\n        \n        if not self.is_test:\n            single_image_label = self.labels[index]\n            # Return image and the label\n            return (img_as_tensor, single_image_label)\n        return img_as_tensor\n        \n    # This returns the length of the dataset\n    def __len__(self):\n        return len(self.data.index)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9d7b061925521f661ec9a19340a0a0f6892fefac"},"cell_type":"markdown","source":"# Convolutional Network class."},{"metadata":{"trusted":true,"_uuid":"c5c10d432338071f5f91c148daafcb871bca47c2","collapsed":true},"cell_type":"code","source":"class ConvNet(nn.Module):\n    # Defining the layers of the convolutional network.\n    def __init__(self,layers, c):\n        super().__init__()\n\n        # If we give a list of layers as input, we can also write it in the below way. \n        # self.layers = nn.ModuleList([ConvLayer(layers[i], layers[i+1]) for i in range(len(layers) - 1)])\n        self.layer1 =  nn.Conv2d(1, 10, kernel_size=5, stride=2, padding=2)\n        self.layer2 =  nn.Conv2d(10, 20, kernel_size=5, stride=2, padding=2)\n        self.layer3 =  nn.Conv2d(20, 40, kernel_size=3, stride=2, padding=1)\n        self.layer4 =  nn.Conv2d(40, 80, kernel_size=3, stride=2, padding=1)\n        \n#         self.out = nn.Linear(layers[-1], c)\n        # c stands for the number of classes\n        self.out = nn.Linear(80, c)\n            \n    # Defining the forward function.\n    def forward(self, x):\n        x = F.relu(self.layer1(x))\n        x = F.relu(self.layer2(x))\n        x = F.relu(self.layer3(x))\n        x = F.relu(self.layer4(x))\n\n        x = F.adaptive_max_pool2d(x, 1)\n        x = x.view(x.size(0), -1)\n        return self.out(x)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"818d07abae3ca32f0e1ebfa48d319325101e2272","collapsed":true},"cell_type":"code","source":"# Number of neurons in the input layer.\nn_inputs = 784\n# Number of output layer neurons.\nn_outputs = 10\n# The Learning rate. We will also use learning rate decay in which after every 10 epochs, we decrease the learning rateby half\nlearning_rate = 0.001\n# The number of epochs,\nn_epochs = 30\n\nmodel = ConvNet([1, 10, 20, 40, 80], 10).cuda()\n\n# Defining the loss function\ncriterion = nn.CrossEntropyLoss()\n# Defining the optimiser.\noptimizer = torch.optim.RMSprop(model.parameters(), lr=learning_rate, weight_decay=1e-6)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"fafba399c269a70ae792cd25c229299cddfae1cd"},"cell_type":"code","source":"# This function decreses the learning rate by half after every 10 epochs.\ndef adjust_learning_rate(optimizer, epoch):\n    \"\"\"Sets the learning rate to the initial LR decayed by 10 every 30 epochs\"\"\"\n    learning_rate = 0.001\n    learning_rate = learning_rate * (0.5 ** (epoch // 10))\n    for param_group in optimizer.param_groups:\n        param_group['lr'] = learning_rate\n#     print(\"Epoch, \",epoch,  learning_rate)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2aca16930e09b24f5c721c6430b8d8ee11e57e26","collapsed":true},"cell_type":"code","source":"if __name__ == \"__main__\":\n    \n    df_train = pd.read_csv('../input/train.csv')\n    imageRotAngle = 20\n    \n    imageRotate = lambda mI: mI.rotate((2 * imageRotAngle * np.random.rand(1)) - imageRotAngle)\n    # Creating augented data transformations\n    train_aug_transformations = transforms.Compose([transforms.RandomRotation(10), transforms.Lambda(imageRotate), transforms.ToTensor()])\n    test_transformations = transforms.Compose([transforms.ToTensor()])\n    train_transformations = transforms.Compose([transforms.ToTensor()])\n    \n    custom_mnist_from_csv = CustomDatasetFromCSV_new('../input/train.csv', train_transformations)\n    custom_mnist_aug_from_csv = CustomDatasetFromCSV_new('../input/train.csv', train_aug_transformations)\n    custom_test_mnist_from_csv = CustomDatasetFromCSV_new('../input/test.csv', test_transformations, is_test=True)\n    \n    # Creating data loader with a batch size of 32 for training data + augmented training data.\n    my_train_data_loader = torch.utils.data.DataLoader(dataset=torch.utils.data.ConcatDataset([custom_mnist_from_csv, custom_mnist_aug_from_csv]), batch_size=32, shuffle=False, num_workers=10)\n    # Creating data loader with a batch size of 32 for testing data.\n    #my_test_data_loader = torch.utils.data.DataLoader(dataset=custom_test_mnist_from_csv, batch_size=32, shuffle=False, num_workers=10)\n    \n    cnt_1 = 0\n    for epoch in range(n_epochs):\n        # adjusting learning rate in the beginning of the epoch\n        adjust_learning_rate(optimizer, epoch)\n        for i, (images, labels) in enumerate(my_train_data_loader):\n            cnt_1 = cnt_1 + 1\n            images = Variable(images.view(-1, 1, 28, 28)).cuda()\n            labels = Variable(labels).cuda()\n            \n            output = model(images)\n            train_loss = criterion(output, labels)\n            \n            optimizer.zero_grad()\n            train_loss.backward()\n            optimizer.step()\n            \n#         outputs = model(images)\n#         _, predicted = torch.max(outputs.data, 1)\n#         total += labels.size(0)\n#         predicted = predicted.cpu().numpy()\n#         correct += (predicted == labels).sum()\n#         print(\"Epoch : %d, Train Loss : %.4f, Accuracy = %0.3f\" % (epoch, train_loss.abs(), correct/total))\n        \n    #\n    ans_arr = []\n    for i, images in enumerate(custom_test_mnist_from_csv):\n        images = Variable(images.view(-1, 1, 28, 28)).cuda()\n        outputs = model(images)\n        _, predicted = torch.max(outputs.data, 1)\n        ans_arr.append(predicted)    \n    print(len(ans_arr))\n    print(cnt_1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6fc44ab8a9944f6c5a287e0e721a5197ded4cc6f"},"cell_type":"markdown","source":"# Creating the submission file"},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"1295b7009be02465a67b29387594d84c96cfaeb6"},"cell_type":"code","source":"\ndf_test = pd.read_csv(\"../input/test.csv\")\nans_arr = [int(x) for x in ans_arr]\ndf_preds = pd.DataFrame()\ndf_preds['ImageId'] = pd.Series([i+1 for i in range(28000)])\ndf_preds['Label'] = pd.Series(ans_arr)\ndf_preds.to_csv(\"base.csv\", index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}