{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
Pytorch: Introduction to CNN
\n
on MNIST digit dataset
\n
\n
","metadata":{}},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"markdown","source":"The objective of this notebook is to create a model running on pytorch that allows to correctly classify a handwritten digit.","metadata":{}},{"cell_type":"markdown","source":"#
Summary
\n\n**1. Load libraries and check TPU settings**\n**
2. EDA and preprocessing**\n**
3. CNN**\n**
4. Evaluation**","metadata":{}},{"cell_type":"markdown","source":"#
1. Load libraries
","metadata":{}},{"cell_type":"code","source":"# Remove warning messages\nimport warnings\nwarnings.filterwarnings('ignore')\n\nimport random\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nimport plotly\nimport plotly.graph_objects as go\n%matplotlib inline\n\nimport os\n\nfrom sklearn.calibration import calibration_curve\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import confusion_matrix\nimport itertools\n\nimport torch\nimport torchvision\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torch.autograd import Variable\nfrom torch.optim import lr_scheduler","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Set seed\nnp.random.seed(42)","metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"PATH_TO_DATA = '../input/digit-recognizer/'","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load train and test \ntrain = pd.read_csv(PATH_TO_DATA + 'train.csv', dtype = np.float32)\ntest = pd.read_csv(PATH_TO_DATA + 'test.csv', dtype = np.float32)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# First rows of train\ntrain.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"--------\n\n**Back to summary**","metadata":{}},{"cell_type":"markdown","source":"#
2. EDA and preprocessing
","metadata":{}},{"cell_type":"markdown","source":"## 2.1 Class distribution","metadata":{}},{"cell_type":"code","source":"def plot_distribution_classes(x_values, y_values):\n\n fig = go.Figure(data=[go.Bar(\n x=x_values, \n y=y_values,\n text=y_values\n )])\n\n fig.update_layout(height=600, width=1200, title_text=\"Distribution of classes\")\n fig.update_xaxes(type=\"category\")\n\n fig.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"x = np.sort(train.label.unique())\ny = train.label.value_counts().sort_index()\n\nplot_distribution_classes(x, y)","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.2 Preprocessing","metadata":{}},{"cell_type":"code","source":"def preprocessing(train, test, split_train_size = 0.2):\n \n # Split data into features(pixels) and labels(numbers from 0 to 9)\n targets = train.label.values\n features = train.drop([\"label\"], axis = 1).values\n \n # Normalization\n features = features/255.\n X_test = test.values/255.\n \n # Train test split. Size of train data is (1-split_train_size)*100% and size of test data is split_train_size%. \n X_train, X_val, y_train, y_val = train_test_split(features,\n targets,\n test_size = split_train_size,\n random_state = 42) \n \n # Create feature and targets tensor for train set. I need variable to accumulate gradients. Therefore first I create tensor, then I will create variable\n X_train = torch.from_numpy(X_train)\n y_train = torch.from_numpy(y_train).type(torch.LongTensor) # data type is long\n\n # Create feature and targets tensor for test set.\n X_val = torch.from_numpy(X_val)\n y_val = torch.from_numpy(y_val).type(torch.LongTensor) # data type is long\n \n # Create feature tensor for train set.\n X_test = torch.from_numpy(X_test)\n \n return X_train, y_train, X_val, y_val, X_test","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_train, y_train, X_val, y_val, X_test = preprocessing(train, test)\n\nprint(f'Shape of training data: {X_train.shape}')\nprint(f'Shape training labels: {y_train.shape}')\nprint(f'Shape of validation data: {X_val.shape}')\nprint(f'Shape of valiation labels: {y_val.shape}')\nprint(f'Shape of testing data: {X_test.shape}')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# batch_size, epoch and iteration\nBATCH_SIZE = 100\nN_ITER = 2500\nEPOCHS = 5\n# I will be trainin the model on another 10 epochs to show flexibility of pytorch\nEXTRA_EPOCHS = 10\n\n# Pytorch train and test sets\ntrain_tensor = torch.utils.data.TensorDataset(X_train, y_train)\nval_tensor = torch.utils.data.TensorDataset(X_val, y_val)\ntest_tensor = torch.utils.data.TensorDataset(X_test)\n\n# data loader\ntrain_loader = torch.utils.data.DataLoader(train_tensor, \n batch_size = BATCH_SIZE,\n shuffle = True)\nval_loader = torch.utils.data.DataLoader(val_tensor, \n batch_size = BATCH_SIZE, \n shuffle = False)\ntest_loader = torch.utils.data.DataLoader(test_tensor, \n batch_size = BATCH_SIZE,\n shuffle = False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.3 Display some examples","metadata":{}},{"cell_type":"code","source":"def display_images(graph_indexes = np.arange(9)):\n \n plt.figure(figsize=(12,12))\n for graph_index in graph_indexes:\n \n # Draw randomly an index\n index = random.randint(1, X_train.shape[0])\n \n # Get corresponding label (.numpy to get value of a tensor)\n label = y_train[index].numpy()\n \n # define subplot\n plt.subplot(330 + 1 + graph_index)\n plt.title('Label: %s \\n'%label,\n fontsize=18)\n # plot raw pixel data (1d tensor that needs to be resized)\n plt.imshow(X_train[index].resize(28,28), cmap=plt.get_cmap('gray'))\n \n # the bottom of the subplots of the figure\n plt.subplots_adjust(bottom = 0.001)\n plt.subplots_adjust(top = 0.99)\n \n # show the figure\n plt.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display_images()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Back to summary**\n\n-------","metadata":{}},{"cell_type":"markdown","source":"#
3. CNN
","metadata":{}},{"cell_type":"markdown","source":"## 3.1 What is a CNN ?\n\nA CNN is quite similar to Classic Neural Networks (RegularNets) where there are neurons with weights and biases. Just like in RegularNets, we use a loss function and an optimizer in CNNs. Additionally though, in CNNs, there are Convolutional Layers, Pooling Layers, and Flatten Layers. CNNs are mainly used for image classification.\n\n### CNN layers\n* **Convolutional layer** \n\nThe very first layer where we extract features from the images in our datasets. Due to the fact that pixels are only related with the adjacent and close pixels, convolution allows us to preserve the relationship between different parts of an image. Convolution is basically filtering the image with a smaller pixel filter to decrease the size of the image without loosing the relationship between pixels. When we apply convolution to 5x5 image by using a 3x3 filter with 1x1 stride (1 pixel shift at each step). We will end up having a 3x3 output (64% decrease in complexity).\n\n\n* **Pooling layer**\n\nWhen constructing CNNs, it is common to insert pooling layers after each convolution layer to reduce the spatial size of the representation to reduce the parameter counts which reduces the computational complexity. In addition, pooling layers also **helps with the overfitting problem**. Basically we select a pooling size to reduce the amount of the parameters by selecting the maximum, average, or sum values inside these pixels.\n\n\n* **Flatten layer**\n\nFlattens the input. Does not affect the batch size.","metadata":{}},{"cell_type":"markdown","source":"## 3.2 Network Structure","metadata":{}},{"cell_type":"markdown","source":"To build a model with pytorch, a class should be created. This class will contain an __init__ with the different layers that will be used to define the architecture of the neural network. Then, the forward method will consist in building the network.","metadata":{}},{"cell_type":"code","source":"class CNNModel(nn.Module):\n def __init__(self):\n super(CNNModel, self).__init__()\n \n # convolution 1\n self.c1 = nn.Conv2d(in_channels=1, out_channels=16, kernel_size=(5,5), stride=1, padding=0)\n self.relu1 = nn.ReLU()\n \n # maxpool 1\n self.maxpool1 = nn.MaxPool2d(kernel_size=(2,2))\n \n # dropout 1\n self.dropout1 = nn.Dropout(0.25)\n \n # convolution 2\n self.c2 = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=(3,3), stride=1, padding=0)\n self.relu2 = nn.ReLU()\n \n # maxpool 2\n self.maxpool2 = nn.MaxPool2d(kernel_size=(2,2))\n\n # dropout 2\n self.dropout2 = nn.Dropout(0.25)\n \n # linear 1\n self.fc1 = nn.Linear(32*5*5, 256)\n \n # dropout 3\n self.dropout3 = nn.Dropout(0.25)\n \n # linear 2\n self.fc2 = nn.Linear(256, 10)\n \n def forward(self, x):\n \n out = self.c1(x) # [BATCH_SIZE, 16, 24, 24]\n out = self.relu1(out) \n out = self.maxpool1(out) # [BATCH_SIZE, 16, 12, 12]\n out = self.dropout1(out) \n \n out = self.c2(out) # [BATCH_SIZE, 32, 10, 10]\n out = self.relu2(out) \n out = self.maxpool2(out) # [BATCH_SIZE, 32, 5, 5]\n out = self.dropout2(out) \n \n out = out.view(out.size(0), -1) # [BATCH_SIZE, 32*5*5=800]\n out = self.fc1(out) # [BATCH_SIZE, 256]\n out = self.dropout3(out)\n out = self.fc2(out) # [BATCH_SIZE, 10]\n \n return out","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Create CNN\nmodel = CNNModel()\n\n# Optimizer\noptimizer = torch.optim.Adam(model.parameters(), lr=0.003)\n\n# Cross Entropy Loss \ncriterion = nn.CrossEntropyLoss()\n\n# LR scheduler\nexp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=3, gamma=0.5)\n\n# On GPU if possible\nif torch.cuda.is_available():\n print(\"Model will be training on GPU\")\n model = model.cuda()\n criterion = criterion.cuda()\nelse:\n print(\"Model will be training on CPU\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.3 Training and evaluation","metadata":{}},{"cell_type":"code","source":"def fit(epoch):\n \n print(\"Training...\")\n # Set model on training mode\n model.train()\n \n # Update lr parameter\n exp_lr_scheduler.step()\n \n # Initialize train loss and train accuracy\n train_running_loss = 0.0\n train_running_correct = 0\n train_running_lr = optimizer.param_groups[0]['lr']\n \n for batch_idx, (data, target) in enumerate(train_loader):\n data, target = Variable(data.view(BATCH_SIZE,1,28,28)), Variable(target)\n \n if torch.cuda.is_available():\n data = data.cuda()\n target = target.cuda()\n \n optimizer.zero_grad()\n output = model(data)\n loss = criterion(output, target)\n \n train_running_loss += loss.item()\n _, preds = torch.max(output.data, 1)\n train_running_correct += (preds == target).sum().item()\n \n loss.backward()\n optimizer.step()\n \n if (batch_idx + 1)% 50 == 0:\n print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n epoch+1, \n (batch_idx + 1) * len(data), \n len(train_loader.dataset),\n BATCH_SIZE * (batch_idx + 1) / len(train_loader), \n loss.cpu().detach().numpy())\n )\n \n train_loss = train_running_loss/len(train_loader.dataset)\n train_accuracy = 100. * train_running_correct/len(train_loader.dataset) \n \n return train_loss, train_accuracy, train_running_lr","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def validate(data_loader):\n \n print(\"Validating...\")\n # Set model on validating mode\n model.eval()\n val_preds = torch.LongTensor().cuda()\n val_proba = torch.LongTensor().cuda()\n \n # Initialize validation loss and validation accuracy\n val_running_loss = 0.0\n val_running_correct = 0\n \n for data, target in data_loader:\n # Regarding volatile argument, check the note below\n data, target = Variable(data.view(BATCH_SIZE,1,28,28), volatile=True), Variable(target)\n \n if torch.cuda.is_available():\n data = data.cuda()\n target = target.cuda()\n \n output = model(data)\n loss = criterion(output, target)\n \n val_running_loss += loss.item()\n pred = output.data.max(1, keepdim=True)[1]\n proba = torch.nn.functional.softmax(output.data)\n\n val_running_correct += pred.eq(target.data.view_as(pred)).cpu().sum() \n \n # Store val_predictions with probas for confusion matrix calculations & best errors made\n val_preds = torch.cat((val_preds, pred), dim=0)\n val_proba = torch.cat((val_proba, proba))\n\n val_loss = val_running_loss/len(data_loader.dataset)\n val_accuracy = 100. * val_running_correct/len(data_loader.dataset) \n \n return val_loss, val_accuracy, val_preds, val_proba","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Volatile is recommended for purely inference mode, when you’re sure you won’t be even calling .backward(). It’s more efficient than any other autograd setting - it will use the absolute minimal amount of memory to evaluate the model. volatile also determines that requires_grad is False","metadata":{}},{"cell_type":"code","source":"train_loss, train_accuracy = [], []\nval_loss, val_accuracy = [], []\nval_preds, val_proba = [], []\ntrain_lr = []\n\nfor epoch in range(EPOCHS):\n \n print(f\"Epoch {epoch+1} of {EPOCHS}\\n\")\n \n train_epoch_loss, train_epoch_accuracy, train_epoch_lr = fit(epoch)\n val_epoch_loss, val_epoch_accuracy, val_epoch_preds, val_epoch_proba = validate(val_loader)\n \n train_loss.append(train_epoch_loss)\n train_accuracy.append(train_epoch_accuracy)\n train_lr.append(train_epoch_lr)\n \n val_loss.append(val_epoch_loss)\n val_accuracy.append(val_epoch_accuracy)\n val_preds.append(val_epoch_preds)\n val_proba.append(val_epoch_proba)\n \n print(f\"Train Loss: {train_epoch_loss:.4f}, Train Acc: {train_epoch_accuracy:.2f}\")\n print(f'Val Loss: {val_epoch_loss:.4f}, Val Acc: {val_epoch_accuracy:.2f}\\n')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After the training of 5 epochs, the model can be saved here with its latest weights and any information that qualify the way this CNN was trained (optimizer, loss...).","metadata":{}},{"cell_type":"code","source":"# save model checkpoint\ntorch.save({'epoch': EPOCHS,\n 'model_state_dict': model.state_dict(),\n 'optimizer_state_dict': optimizer.state_dict(),\n 'loss': criterion,},\n './model.pth')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now the model will be loaded from its latest status. This technic is convenient if you cannot train all of your data through all the epochs in once.","metadata":{}},{"cell_type":"code","source":"# load the model checkpoint\ncheckpoint = torch.load('./model.pth')\n\n# load model weights state_dict\nmodel.load_state_dict(checkpoint['model_state_dict'])\nprint('Previously trained model weights state_dict loaded...')\n\n# load trained optimizer state_dict\noptimizer.load_state_dict(checkpoint['optimizer_state_dict'])\nprint('Previously trained optimizer state_dict loaded...')\nEPOCHS = checkpoint['epoch']\n\n# load the criterion\ncriterion = checkpoint['loss']\nprint('Trained model loss function loaded...')\nprint(f\"Previously trained for {EPOCHS} number of epochs...\")\n\n# train for more epochs\nprint(f\"Train for {EXTRA_EPOCHS} more epochs...\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for epoch in range(EXTRA_EPOCHS):\n \n print(f\"Epoch {epoch+1} of {EXTRA_EPOCHS}\\n\")\n \n train_epoch_loss, train_epoch_accuracy, train_epoch_lr = fit(epoch)\n val_epoch_loss, val_epoch_accuracy, val_epoch_preds, val_epoch_proba = validate(val_loader)\n \n train_loss.append(train_epoch_loss)\n train_accuracy.append(train_epoch_accuracy)\n train_lr.append(train_epoch_lr)\n \n val_loss.append(val_epoch_loss)\n val_accuracy.append(val_epoch_accuracy)\n val_preds.append(val_epoch_preds)\n val_proba.append(val_epoch_proba)\n \n print(f\"Train Loss: {train_epoch_loss:.4f}, Train Acc: {train_epoch_accuracy:.2f}\")\n print(f'Val Loss: {val_epoch_loss:.4f}, Val Acc: {val_epoch_accuracy:.2f}\\n')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# save model checkpoint\ntorch.save({'epoch': EXTRA_EPOCHS,\n 'model_state_dict': model.state_dict(),\n 'optimizer_state_dict': optimizer.state_dict(),\n 'loss': criterion}, \n './model.pth')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This part could have been done much quicker by chosing to train directly on 15 epochs. However I wanted to show how interesting it was to split the training in two parts and how to save and load a pytorch model.","metadata":{}},{"cell_type":"markdown","source":"## 3.4 History of CNN","metadata":{}},{"cell_type":"code","source":"def plot_history():\n\n plt.figure(figsize = (20,15))\n \n plt.subplot(221)\n \n # summarize history for accuracy\n plt.plot(train_accuracy)\n plt.plot(val_accuracy)\n plt.title('model accuracy')\n plt.ylabel('accuracy')\n plt.xlabel('epoch')\n plt.legend(['train', 'test'], loc='upper left')\n plt.grid()\n \n \n plt.subplot(222)\n # summarize history for loss\n plt.plot(train_loss)\n plt.plot(val_loss)\n plt.title('model loss')\n plt.ylabel('loss')\n plt.xlabel('epoch')\n plt.legend(['train', 'test'], loc='upper left')\n plt.grid()\n \n plt.subplot(223)\n # summarize history for lr\n plt.plot(train_lr)\n plt.title('learning rate')\n plt.ylabel('lr')\n plt.xlabel('epoch')\n plt.grid()\n \n plt.show()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot_history()","metadata":{"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Back to summary**\n\n-------","metadata":{}},{"cell_type":"markdown","source":"#
4 Evaluation
","metadata":{}},{"cell_type":"markdown","source":"## 4.1 Confusion Matrix","metadata":{}},{"cell_type":"code","source":"def plot_confusion_matrix(confusion_matrix, \n cmap=plt.cm.Reds):\n \n classes = range(10)\n \n plt.figure(figsize=(8,8))\n plt.imshow(confusion_matrix, \n interpolation='nearest', \n cmap=cmap)\n plt.title('Confusion matrix')\n plt.colorbar()\n tick_marks = np.arange(len(classes))\n plt.xticks(tick_marks, classes, rotation=45)\n plt.yticks(tick_marks, classes)\n\n thresh = confusion_matrix.max() / 2.\n for i, j in itertools.product(range(confusion_matrix.shape[0]), range(confusion_matrix.shape[1])):\n plt.text(j, i, confusion_matrix[i, j],\n horizontalalignment=\"center\",\n color=\"white\" if confusion_matrix[i, j] > thresh else \"black\")\n\n plt.tight_layout()\n plt.ylabel('True label')\n plt.xlabel('Predicted label')","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Get predictions of validation set from last epoch\n# val_preds is a list of (EPOCHS + EXTRA_EPOCHS) tensors\ny_pred_classes = val_preds[EPOCHS + EXTRA_EPOCHS - 1].cpu().numpy().ravel()\n\n# compute the confusion matrix\ncm = confusion_matrix(y_val, y_pred_classes) \n\n# plot the confusion matrix\nplot_confusion_matrix(cm)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.2 Some examples of predicted images","metadata":{}},{"cell_type":"code","source":"def display_predicted_images(graph_indexes = np.arange(9)):\n \n # plot first few images\n plt.figure(figsize=(12,12))\n \n for graph_index in graph_indexes:\n \n index = random.randint(1, X_val.shape[0])\n \n # Get corresponding label\n predicted_label = y_pred_classes[index]\n true_label = y_val[index]\n \n # define subplot\n plt.subplot(330 + 1 + graph_index)\n plt.title('Predicted label: %s \\n'%predicted_label+\\\n 'True label %s \\n'%true_label.item(),\n fontsize=18)\n # plot raw pixel data\n plt.imshow(X_val[index].view(28,28), cmap=plt.get_cmap('gray'))\n \n plt.subplots_adjust(bottom = 0.001) # the bottom of the subplots of the figure\n plt.subplots_adjust(top = 0.99)\n \n # show the figure\n plt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"display_predicted_images()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4.3 \"Best\" errors","metadata":{}},{"cell_type":"code","source":"# Retrieve validation proba predictions of last epoch\ny_pred = val_proba[EPOCHS + EXTRA_EPOCHS - 1].cpu().numpy()\n\n# Display errors \nerrors = (y_pred_classes - y_val.cpu().numpy() != 0)\n\ny_pred_classes_errors = y_pred_classes[errors]\ny_pred_errors = y_pred[errors]\ny_true_errors = y_val[errors]\n\nX_val_errors = X_val[errors]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def display_top9_wrongly_predicted_images(list_of_indexes, graph_indexes = np.arange(9)):\n \n # plot first few images\n plt.figure(figsize=(12,12))\n \n for graph_index in graph_indexes:\n \n index = list_of_indexes[graph_index]\n \n # Get corresponding label\n predicted_label = y_pred_classes_errors[index]\n true_label = y_true_errors[index]\n \n \n # define subplot\n plt.subplot(330 + 1 + graph_index)\n plt.title('Predicted label: %s \\n'%predicted_label+\\\n 'True label %s \\n'%true_label.item(),\n fontsize=18)\n # plot raw pixel data\n plt.imshow(X_val_errors[index].view(28,28), cmap=plt.get_cmap('gray'))\n \n plt.subplots_adjust(bottom = 0.001) # the bottom of the subplots of the figure\n plt.subplots_adjust(top = 0.99)\n \n # show the figure\n plt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Probabilities of the wrong predicted numbers\ny_pred_errors_prob = np.max(y_pred_errors,axis = 1)\n\n# Predicted probabilities of the true values in the error set\ntrue_prob_errors = np.diagonal(np.take(y_pred_errors, y_true_errors, axis=1))\n\n# Difference between the probability of the predicted label and the true label\ndelta_pred_true_errors = y_pred_errors_prob - true_prob_errors\n\n# Sorted list of the delta prob errors\nsorted_dela_errors = np.argsort(delta_pred_true_errors)\n\n# Top 9 errors \nmost_important_errors = sorted_dela_errors[-9:]\n\n# Show the top 9 errors\ndisplay_top9_wrongly_predicted_images(list_of_indexes = most_important_errors)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Submission","metadata":{}},{"cell_type":"code","source":"def prediction(data_loader):\n \n print(\"Infering predictions...\")\n # Set model on validating mode\n model.eval()\n test_pred = torch.LongTensor()\n \n for batch_idx, data in enumerate(data_loader):\n data = Variable(data[0].view(BATCH_SIZE,1,28,28), volatile=True)\n \n if torch.cuda.is_available():\n data = data.cuda()\n \n output = model(data)\n \n pred = output.cpu().data.max(1, keepdim=True)[1]\n test_pred = torch.cat((test_pred, pred), dim=0)\n \n print(\"Completed\") \n return test_pred","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# predict results\ny_test_pred = prediction(test_loader)\n\n# Associate max probability obs with label class\ny_test_pred = y_test_pred.numpy().ravel()\ny_test_pred = pd.Series(y_test_pred, name=\"Label\")\n\nsubmission = pd.concat([pd.Series(range(1,28001), name = \"ImageId\"), y_test_pred], axis = 1)\n\nsubmission.to_csv(\"CNN_model_TPU_submission.csv\", index = False)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Back to summary**\n\n-------","metadata":{}},{"cell_type":"markdown","source":"# References\n\n* https://debuggercafe.com/effective-model-saving-and-resuming-training-in-pytorch/\n* https://pytorch.org/tutorials/","metadata":{}},{"cell_type":"markdown","source":"
\n
There is also CNN implementation using Tensorflow/keras.
\n
Keras: CNN for MNIST digit recognition with TPUs
\n
\n
Thank you for taking the time to read this notebook. I hope that I was able to answer your questions or your curiosity and that it was quite understandable. any constructive comments are welcome. They help me progress and motivate me to share better quality content. I am above all a passionate person who tries to advance my knowledge but also that of others. If you liked it, feel free to upvote and share my work.
\n
\n
Thank you and may passion guide you.
","metadata":{}}]}