{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n

MNIST Pytorch: Convoluton Neural Networks

\n","metadata":{}},{"cell_type":"markdown","source":"`We are using Mnist kaggle training set as our training dataset. It consists of 28px by 28px grayscale images of handwritten digits (0 to 9), along with labels for each image indicating which digit it represents. Here are some sample images from the dataset:`\n\n![mnist-sample](https://i.imgur.com/CAYnuo1.jpg)\n\n`Our goal is to correctly identify digits from a dataset of tens of thousands of handwritten images.`","metadata":{}},{"cell_type":"markdown","source":"
\n

Table of Contents

\n\n \n- [Import Libaries](#2)\n- [Data Augmentation](#3) \n- [Pytorch Dataset Classes](#4)\n- [Feature Engineering](#5)\n- [CNN(LeNet5)](#6)\n- [Training and Validation losses](#7)\n- [Prediction & Submission](#8)\n","metadata":{}},{"cell_type":"markdown","source":"\n# Import Libraries","metadata":{}},{"cell_type":"code","source":"import warnings \nwarnings.filterwarnings('ignore')\nimport torch\nimport torchvision\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nsns.set_context(\"paper\", font_scale = 1, rc={\"grid.linewidth\": 3})\npd.set_option('display.max_rows', 100, 'display.max_columns', 400)\nfrom torch.utils.data import DataLoader,Dataset,ConcatDataset\nfrom torchvision import transforms\nimport torch.optim as optim\nfrom sklearn.model_selection import train_test_split\nimport torch.nn as nn","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train_data=pd.read_csv('../input/digit-recognizer/train.csv')\ntest_data=pd.read_csv('../input/digit-recognizer/test.csv')\nsample_data = pd.read_csv('../input/digit-recognizer/sample_submission.csv')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(train_data.info())\nprint('\\n')\nprint(test_data.info())\nprint('\\n')\nprint(sample_data.info())","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The dataset has 42,000 images for train data which can be used to train the model. 28,000 images for test set.","metadata":{}},{"cell_type":"code","source":"train_df = train_data.iloc[:, 1:].values\ny_train = train_data.iloc[:, 0].values\ntest_df = test_data.values","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"image_1 = train_df.reshape(train_df.shape[0], 28, 28) #rehsaping it to plot image\nplt.figure(figsize=(20,8))\nfor i in range(10,18):\n plt.subplot(231 + (i))\n plt.imshow(image_1[i], cmap=\"gray\")\n plt.title('Label:'+str(y_train[i]),fontweight='bold',size=20)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It's evident that these images are quite small in size, and recognizing the digits can sometimes be hard even for the human eye. While it's useful to look at these images, there's just one problem here: PyTorch doesn't know how to work with images. We need to convert the images into tensors. We can do this by specifying a transform while creating our dataset.","metadata":{}},{"cell_type":"markdown","source":"PyTorch datasets allow us to specify one or more transformation functions which are applied to the images as they are loaded. `torchvision.transforms` contains many such predefined functions, and we'll use the `ToTensor` transform to convert images into PyTorch tensors.","metadata":{}},{"cell_type":"markdown","source":"\n# Data Augmentation ","metadata":{}},{"cell_type":"code","source":"img_tform_1 = transforms.Compose([\n transforms.ToPILImage(),transforms.ToTensor(),transforms.Normalize((0.5),(0.5))])\n\nimg_tform_2 = transforms.Compose([\n transforms.ToPILImage(),transforms.RandomRotation(10),transforms.ToTensor(),transforms.Normalize((0.5),(0.5))])\n\nimg_tform_3 = transforms.Compose([\n transforms.ToPILImage(),transforms.RandomRotation(20),transforms.ToTensor(),transforms.Normalize((0.5),(0.5))])\n\nimg_tform_4 = transforms.Compose([\n transforms.ToPILImage(),transforms.RandomAffine(degrees=15, translate=(0.1,0.1), scale=(0.85,0.85)),\\\n transforms.ToTensor(),transforms.Normalize((0.5),(0.5))])\n\nimg_tform_5 = transforms.Compose([\n transforms.ToPILImage(),transforms.RandomAffine(0,shear=30,scale=[1.15,1.15]),\\\n transforms.ToTensor(),transforms.Normalize((0.5),(0.5))])\n\nimg_tform_6 = transforms.Compose([\n transforms.ToPILImage(),transforms.RandomAffine(0,shear=20,scale=[0.8,0.8]),\\\n transforms.ToTensor(),transforms.Normalize((0.5),(0.5))])\n\nimg_tform_7 = transforms.Compose([\n transforms.ToPILImage(),transforms.RandomAffine(degrees=30, scale=(1.2,1.2)),\\\n transforms.ToTensor(),transforms.Normalize((0.5),(0.5))])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# Pytorch Dataset Classes","metadata":{}},{"cell_type":"code","source":"class MnistDataset(Dataset):\n #it takes whatever arguments needed to build a list of tuples — it may be the name of a CSV file that will be loaded and processed; it may be two tensors, one for features, another one for labels; or anything else, depending on the task at hand.\n def __init__(self, features,transform=img_tform_1): \n self.features = features.iloc[:,1:].values.reshape((-1,28,28)).astype(np.uint8)\n self.targets = torch.from_numpy(features.label.values)\n self.transform=transform\n \n #it should simply return the size of the whole dataset so, whenever it is sampled, its indexing is limited to the actual size.\n def __len__(self):\n return (self.features.shape[0])\n #There is no need to load the whole dataset in the constructor method (__init__). If your dataset is big (tens of thousands of image files, for instance), loading it at once would not be memory efficient. It is recommended to load them on demand (whenever __get_item__ is called).\n # it allows the dataset to be indexed, so it can work like a list (dataset[i]) — it must return a tuple (features, label) corresponding to the requested data point. \n def __getitem__(self, idx):\n return self.transform(self.features[idx]),self.targets[idx]\n\n\n# Checking this class \nclass TestDataset(Dataset):\n def __init__(self, features,transform=img_tform_1):\n self.features = features.values.reshape((-1,28,28)).astype(np.uint8)\n self.targets = None\n self.transform=transform\n \n def __len__(self):\n return (self.features.shape[0])\n \n def __getitem__(self, idx):\n return self.transform(self.features[idx])","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# Feature Engineering","metadata":{}},{"cell_type":"code","source":"def create_dataloaders(seed, test_size=0.1, df=train_data, batch_size=32):\n # Create training set and validation set\n train_df, val_df = train_test_split(df,test_size=test_size,random_state=seed)\n \n # Create Datasets\n train_data_1 = MnistDataset(train_df)\n train_data_2 = MnistDataset(train_df, img_tform_2)\n train_data_3 = MnistDataset(train_df, img_tform_3)\n train_data_4 = MnistDataset(train_df, img_tform_4)\n train_data_5 = MnistDataset(train_df, img_tform_5)\n train_data_6 = MnistDataset(train_df, img_tform_6)\n train_data_7 = MnistDataset(train_df, img_tform_7)\n train_final = ConcatDataset([train_data_1, train_data_2, train_data_3, train_data_4, train_data_5,\\\n train_data_6,train_data_7])\n\n val_data = MnistDataset(val_df)\n \n # Create Dataloaders\n train_loader = torch.utils.data.DataLoader(train_final, batch_size=batch_size, shuffle=True)\n valid_loader = torch.utils.data.DataLoader(val_data, batch_size=batch_size, shuffle=False)\n\n return train_loader, valid_loader","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# CNN (Custom LeNet5) Model","metadata":{}},{"cell_type":"code","source":"class Model(nn.Module):\n def __init__(self):\n super().__init__()\n \n self.conv1 = nn.Sequential(\n nn.Conv2d(1, 32, kernel_size=3), #26x26x32\n nn.BatchNorm2d(32),\n nn.LeakyReLU(inplace=True), # inplace=True helps to save some memory\n \n nn.Conv2d(32, 32, kernel_size=3), # 24x24x32\n nn.BatchNorm2d(32),\n nn.LeakyReLU(inplace=True),\n \n nn.Conv2d(32, 32, kernel_size=5, stride=2, padding=14), # 24x24x32 (same padding)\n nn.BatchNorm2d(32),\n nn.LeakyReLU(inplace=True),\n \n nn.MaxPool2d(2, 2), #12x12x32\n nn.Dropout2d(0.25),\n \n nn.Conv2d(32, 64, kernel_size=3), #10x10x64\n nn.BatchNorm2d(64),\n nn.LeakyReLU(inplace=True), \n \n nn.Conv2d(64, 64, kernel_size=3), # 8x8x64\n nn.BatchNorm2d(64),\n nn.LeakyReLU(inplace=True), \n \n nn.Conv2d(64, 64, kernel_size=5, stride=2, padding=6), # 8x8x64(same padding)\n nn.BatchNorm2d(64),\n nn.LeakyReLU(inplace=True),\n \n nn.MaxPool2d(2, 2),# 4x4x64 (half)\n nn.Dropout2d(0.25),\n \n nn.Conv2d(64, 128, kernel_size=4), # 1x128\n nn.BatchNorm2d(128),\n nn.LeakyReLU(inplace=True),\n nn.Dropout2d(0.25)\n )\n \n self.fc = nn.Sequential(\n nn.Linear(128*1*1, 10)\n )\n \n def forward(self, x):\n x = self.conv1(x)\n x = x.view(-1, 128*1*1)\n x = self.fc(x)\n \n return x","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# Training and Validation losses","metadata":{}},{"cell_type":"code","source":"def train_fn(model, optimizer, scheduler, loss_fn, dataloader, device):\n model.train() # set the model to training mode\n final_loss = 0 # Initialise final loss to zero\n train_acc=0\n total=0\n train_preds=[]\n \n for features,labels in dataloader:\n optimizer.zero_grad() #every time we use the gradients to update the parameters, we need to zero the gradients afterwards\n inputs, targets = features.to(device), labels.to(device) #Sending data to GPU(cuda) if gpu is available otherwise CPU\n outputs = model(inputs) #output \n loss = loss_fn(outputs, targets) #loss function\n loss.backward() #compute gradients(work its way BACKWARDS from the specified loss)\n optimizer.step() #gradient optimisation\n scheduler.step() #scheduler optimisation\n total+=len(targets)\n final_loss += loss.item() #Final loss\n train_preds.append(outputs.sigmoid().detach().cpu().numpy()) # get CPU tensor as numpy array # cannot get GPU tensor as numpy array directly\n _, predicted = torch.max(outputs, 1)\n train_acc+=((predicted == targets).sum().item())\n final_loss /= len(dataloader) #average loss\n train_preds = np.concatenate(train_preds)#concatenating predictions under train_pred\n train_acc=(train_acc/total)*100\n \n return final_loss,train_acc\n\n\ndef valid_fn(model, loss_fn, dataloader, device):\n model.eval() # set the model to evaluation/validation mode\n final_loss = 0 # Initialise validation final loss to zero\n valid_preds = [] #Empty list for appending prediction\n val_acc=0\n total=0\n for features,labels in dataloader:\n inputs, targets = features.to(device), labels.to(device) #Sending data to GPU(cuda) if gpu is available otherwise CPU\n outputs = model(inputs) #output\n loss = loss_fn(outputs, targets) #loss calculation\n total+=len(targets)\n final_loss += loss.item() #final validation loss\n valid_preds.append(outputs.sigmoid().detach().cpu().numpy()) # get CPU tensor as numpy array # cannot get GPU tensor as numpy array directly\n _, predicted = torch.max(outputs, 1)\n val_acc+=((predicted == targets).sum().item())\n \n final_loss /= len(dataloader)\n valid_preds = np.concatenate(valid_preds) #concatenating predictions under valid_preds\n val_acc=(val_acc/total)*100\n \n return final_loss, valid_preds,val_acc\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# HyperParameters\n\nDEVICE = ('cuda' if torch.cuda.is_available() else 'cpu')\nEPOCHS = 12\nBATCH_SIZE = 128\nLEARNING_RATE = 1e-3\nWEIGHT_DECAY = 1e-8\nseed=42\n#EARLY_STOPPING_STEPS = 10\n#EARLY_STOP = False\n#Dropout_model_val=0.2619422201258426","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def run_training(seed):\n # train and data val dataloaders\n train_loader, valid_loader= create_dataloaders(seed=seed)\n model=Model()\n model.to(DEVICE)\n #using adam optimizer for optimization\n optimizer = torch.optim.Adam(model.parameters(), lr=LEARNING_RATE,weight_decay=WEIGHT_DECAY)\n scheduler = optim.lr_scheduler.OneCycleLR(optimizer=optimizer, pct_start=0.1, div_factor=1e2, \n max_lr=1e-2, epochs=EPOCHS, steps_per_epoch=len(train_loader))\n loss_fn = nn.CrossEntropyLoss()\n for epoch in range(EPOCHS):\n train_loss,train_acc = train_fn(model, optimizer,scheduler, loss_fn, train_loader, DEVICE) #training loss and accuracy\n print(f\"EPOCH: {epoch}, train_loss: {train_loss},, train_accuracy:{train_acc}\")\n val_loss, val_preds, val_acc = valid_fn(model, loss_fn, valid_loader, DEVICE) #validation loss and accuracy\n print(f\"EPOCH: {epoch}, valid_loss: {val_loss}, val_accuracy:{val_acc}\")\n \n test_pred = torch.LongTensor() \n testdataset = TestDataset(test_data)\n testloader = torch.utils.data.DataLoader(testdataset, batch_size=BATCH_SIZE, shuffle=False)\n for features in testloader:\n features=features.to(DEVICE)\n outputs=model(features)\n _, predicted = torch.max(outputs, 1)\n test_pred = torch.cat((test_pred.to(DEVICE), predicted.to(DEVICE)), dim=0)\n pred_df['predict'] = test_pred.cpu().numpy()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pred_df = sample_data.copy()\nrun_training(seed)\n","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n# Prediction & Submission","metadata":{"trusted":true}},{"cell_type":"code","source":"#Prediction\nfinal_pred = pred_df['predict']\nsample_data.Label = final_pred.astype(int)\nsample_data.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_data.to_csv('./submission.csv', index=False) # submission file","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n

If you found this notebook helpful , some upvotes would be very much appreciated - That will keep me motivated 😊

\n","metadata":{}},{"cell_type":"markdown","source":"
\n

Thank You 😊🙏

\n","metadata":{}}]}