{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\nAuthor: Aaron Chen <br>\nI used Google Colab Notebook to run CNN with CUDA initially(free for basic computation power). <br>\nYou can download this notebook to run it on Google Colab. ","metadata":{}},{"cell_type":"code","source":"### Author: Aaron Chen ###\nimport matplotlib.pyplot as plt\nimport pandas as pd\nimport numpy as np\nimport torch\nfrom torch.autograd import Variable\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torchvision import datasets\nimport torchvision.transforms as transforms\nfrom torch.utils.data import Dataset, DataLoader\n# this step mount your personal Google Drive to the Google Colab directory\n# from google.colab import drive\n# drive.mount('/content/drive')","metadata":{"id":"DkXIpAjOOmLR","outputId":"28078dc7-fbf7-4e1c-c0c4-5829351b6b67","execution":{"iopub.status.busy":"2021-06-06T14:22:53.234810Z","iopub.execute_input":"2021-06-06T14:22:53.235173Z","iopub.status.idle":"2021-06-06T14:22:54.443363Z","shell.execute_reply.started":"2021-06-06T14:22:53.235094Z","shell.execute_reply":"2021-06-06T14:22:54.442596Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 1. Create Pytorch custom dataset\nThe first step is to make a Pytorch custom dataset. Pytorch website has a nice tutorial here: https://pytorch.org/tutorials/beginner/data_loading_tutorial.html <br>\nThe basic idea and purpose of the torch.Dataset is to read some data (i.e. Pandas Dataframe from csv file), make a `__getitem__()` class function so that the Pytorch Dataloader can load your original data in a batch fashion. <ba>\nThe MNIST dataset are 28x28, one channel images. These pixels are formatted in the csv file so that each row represents one image, and pixels of this image are flattened in one dimension (1x784). Therefore, we need to reshape each row to 1x28x28. In many cases where we load real images directly (instead of formatted csv file), images pixels are Width x Height x Channels. However, Pytorch requires the input tensors formatted as Channels x Height x Width. This is just a small detail to be noticed.","metadata":{}},{"cell_type":"code","source":"# df = pd.read_csv('/content/drive/MyDrive/Colab Notebooks/Digit Recognizer/train.csv')\n# df_test = pd.read_csv('/content/drive/MyDrive/Colab Notebooks/Digit Recognizer/test.csv')\ndf = pd.read_csv('../input/digit-recognizer/train.csv')\ndf_test = pd.read_csv('../input/digit-recognizer/test.csv')\n\nclass MyMNIST(Dataset):\n    # reference from https://pytorch.org/tutorials/beginner/data_loading_tutorial.html\n    \n    def __init__(self, df, transform = None, test = False):\n        \"\"\"\n        Args:\n            df (Dataframe): Dataframe of specific test, validation or test.\n            transform (callable, optional): Optional transform to be applied\n                on a sample.\n            test (Boolean): is this dataframe a training set or test set\n        \"\"\"\n        self.df = df\n        self.transform = transform\n        self.test = test\n\n    def __len__(self):\n        return len(self.df)\n\n    def __getitem__(self, idx):\n        if torch.is_tensor(idx):\n            idx = idx.tolist()\n\n        if not self.test:\n            # here x is the pixel (without the label) so [idx, 1:].\n            # we reshape it to 1x28x28 and divided by 255 so that all the pixel values are between [0, 1]\n            # I specifies the dtype = np.float32 because otherwise Pytorch throws an error later I think\n            x = torch.tensor(self.df.iloc[idx,1:].to_numpy(dtype=np.float32).reshape(1,28,28)/255)\n            # for the training/validation set, we need to preserve the label information as well\n            y = torch.tensor(self.df.iloc[idx,0])\n        else:\n            # for the test set, we don't have the label information so return pixels directly\n            x = torch.tensor(self.df.iloc[idx].to_numpy(dtype=np.float32).reshape(1,28,28)/255)\n            if self.transform:\n                x = self.transform(x)\n            return x\n            \n        # Here you can specify what transforms to use on each data set. \n        if self.transform:\n            x = self.transform(x)\n\n        return (x, y)","metadata":{"id":"Q3ShJV5Tqq9D","execution":{"iopub.status.busy":"2021-06-06T14:22:54.444697Z","iopub.execute_input":"2021-06-06T14:22:54.445031Z","iopub.status.idle":"2021-06-06T14:22:59.250358Z","shell.execute_reply.started":"2021-06-06T14:22:54.444995Z","shell.execute_reply":"2021-06-06T14:22:59.249514Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Data prepossessing and data augmentation \nHere, I want to train the model on the first 90% of the data and use the last 10% for the validation. <br>\nIn the custom datset, I devide all the pixel values by 255 to make them between 0 and 1. During the transform, I set the mean and std of the normalization to be 0.5 and 0.5 so that the normalized pixel values will be between -1 and 1. This normalization is applied to all training, validation and test dataset. For the training set, I applied the data augmentation of random horizontal/vertical flip and random rotation with a probability of 0.5. <br>\n<br>\nThen I load the training and validation set using `DataLoader` with a batch size of 100. The batch size is a hyperparameter you can choose but larger the batch size, larger the memory requirement; if it's too small, the optimization of weight and bias might not be stable.<br>\n<br>\nI plotted an image from the training set for visualization purpose. Because of the random transform, the ouput image might be flipped or rotated.","metadata":{}},{"cell_type":"code","source":"split = len(df)//10 * 9\ndf_train = df.iloc[:split]\ndf_val = df.iloc[split:]\n# mean, std = 0.5, 0.5 to normalize the pixels in the range of [-1, 1]\nmean, std = 0.5, 0.5\n# 1. normalization and 2. simple data augmentation with random flips and rotation\ntrain_transform = transforms.Compose([transforms.RandomHorizontalFlip(0.5),\n                                    transforms.RandomVerticalFlip(0.5),\n                                    transforms.RandomApply([transforms.RandomRotation((-25,25))], p=0.5),\n                                    transforms.Normalize((mean,),(std,))])\n# for validation and testing, we only applies normalization\nval_transform = transforms.Normalize((mean,),(std,))\ntest_transform = transforms.Normalize((mean,),(std,))\n# create train, val, test MyMNIST dataset\ntrain = MyMNIST(df_train, train_transform, test  =False)\nval = MyMNIST(df_val, val_transform, test = False)\ntest = MyMNIST(df_test, test_transform, test = True)\n# Load train, val dataset in a batch size of 100\n# when you loop throught them, the dimension would be \n# [100, 1, 28, 28] == [batch_size, num_of_channel, H, W]\ntrain_loader = DataLoader(train, batch_size = 100, shuffle = True)\nval_loader = DataLoader(val, batch_size = 100, shuffle = True)\n# Load test datset, since we just need to make prediction, we can load them one by one\n# you can definitely load all of them, but it would simply requires a lot more memory usage\ntest_loader = DataLoader(test, batch_size = 1, shuffle = False)\n# plot one sample from the training set along with its label\nsample_x, sample_y = next(iter(train_loader))\nprint(sample_x.shape, sample_x.view(-1,28,28).shape)\nplt.imshow(sample_x[0][0].cpu().data.numpy(), cmap='gray')\nplt.show()\nprint('label:', sample_y[0])\n# show pixel distribution if you want to\nshow_pixel_distribution = True\nif show_pixel_distribution:\n    pixel = []  \n    for x, y in train_loader:\n        pixel.append(x.numpy())\n    plt.hist(np.array(x).ravel(), bins=30, density=True)\n    plt.xlabel(\"pixel values\")\n    plt.ylabel(\"relative frequency\")\n    plt.title(\"distribution of pixels\")         ","metadata":{"id":"q8VAyEbsTENe","outputId":"2c406178-1325-4e32-b110-3ef1b6bc2f29","execution":{"iopub.status.busy":"2021-06-06T14:22:59.252114Z","iopub.execute_input":"2021-06-06T14:22:59.252428Z","iopub.status.idle":"2021-06-06T14:23:26.843500Z","shell.execute_reply.started":"2021-06-06T14:22:59.252401Z","shell.execute_reply":"2021-06-06T14:23:26.842514Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Build the model and calculate the dimension in the CNN layers\nI just want to build a relatively simple CNN with downsampling of 32 and 64 CNN layers. Some people went aggresive with multiple 128, 256 layers or even more advanced CNN backbone (such as VGG or ResNet) designed for ImageNet classification. In my humble opinion, I don't think it's necessary (although yes, this code will only achieve ~96% accuracy). MNIST is a small and simple dataset and we are not even training on the whole dataset. With a extremely large CNN, the model might overfit; sometimes the CNN has such high capacity that it \"remembers\" rather than learns the training set. <br>\n<br>\nTo implement CNN, we do need to have a basic understanding of how it affects the dimension of the input. Stanford CS231n is an awesome class to look at (http://cs231n.stanford.edu/). I will try to use their materials here to at least show you how to calculate the dimension (I hope).<br>\n## **Conv2D layer** \nInput image/data/etc: n_channels (D1), height (H1), width (W1) <br>\nConv2D layer parameters: in_channels (D1), out_channels (aka number of CNN filters K), kernel_size (F), stride (S), padding (P) etc (skip dilation here). <br>\nOutput image/data/etc after this Conv2D layer:<br>\nD2 = K<br>\nH2 = (H1−F+2P)/S+1<br>\nW2 = (W1−F+2P)/S+1<br>\nFor example, the input batch is 100x1x28x28 where D1 = 1, H1 = 28, W1 = 28. After self.conv1 (defined below in the code):\nD2 = 32<br>\nH2 = (28-3+2*1)/1+1 = 28<br>\nW2 = (28-3+2*1)/1+1 = 28<br>\nso that the output batch is 100x32x28x28<br>\nYou can certainly play with the Conv2D parameters and there are tons of research what hyperparameters to use. Here (also common setting) is F = 3, S = 1, P = 1. The padding is nice because we don't lose the boundary information and it keeps the height and width the same after the Conv2D.<br>\n## **MaxPool2d**\nMax Pooling layer reduces the spatial size and keep the pixel info with the highest activation value. For example, a 2x2 MaxPool2d takes the highest value from adjacent 2x2 block. To calculate the output dimension:<br>\nInput image/data/etc: n_channels (D1), height (H1), width (W1) <br>\nMaxPool2d layer parameters: kernel_size (F), stride (S). <br>\nOutput image/data/etc after this MaxPool2d layer: <br>\nD2 = D1<br>\nH2 = (H1-F)/S+1<br>\nW2 = (W1-F)/S+1<br>\nFor example, the input batch is 100x32x28x28 where D1 = 32, H1 = 28, W1 = 28. After self.pool (defined below in the code):\nD2 = 32<br>\nH2 = (28-2)/2+1 = 14<br>\nW2 = (28-2)/2+1 = 14<br>\nso that the output batch is 100x32x14x14<br>\n## **ReLu** \nReLu acts as the non-linear layer in the series linear transformation in the CNN. <br>\n## **Drop out layer**\nDrop out layer acts as a regularization method to \"drop\" some node in the neural network to curb overfitting issue. I referenced this website of the choice of the drop out layer position and probability. https://stats.stackexchange.com/questions/240305/where-should-i-place-dropout-layers-in-a-neural-network<br>\n## **BatchNorm2d**\nBatch normalization makes the model more robust (some sort of regularization) and allow faster training (higher learning rate). I am not gonna pretend that I know a lot about the batch normalization but it has became a common practice nowadays. Note: Batch normalization is placed right after the Conv2d/Linear layer and before the activation function. You may read the original paper if you are interested: https://arxiv.org/pdf/1502.03167.pdf <br>\n## **Linear layer aka fully connected layer** \nHere the linear layer works as \"fully connected\" layer to represent and transforms the high dimensional pixel information to our final prediction (i.e. which class or which number the input image belongs to) <br>","metadata":{}},{"cell_type":"code","source":"class CNN(torch.nn.Module):\n\n    def __init__(self, num_classes):\n        super(CNN, self).__init__()\n\n        ##forward layers:\n        self.conv1 = nn.Conv2d(1, 32, kernel_size=3, padding=1) \n        self.conv2 = nn.Conv2d(32, 64, kernel_size=3, padding=1)\n        self.ff1 = nn.Linear(64*7*7, 128) \n        self.out = nn.Linear(128, num_classes)    \n\n        ##activations:\n        self.relu = nn.ReLU()\n\n        #other activations:\n        self.tanh = nn.Tanh()\n        self.sigmoid = nn.Sigmoid()\n        # pooling layer:\n        self.pool = nn.MaxPool2d(kernel_size=2, stride=2)\n        \n        #dropout:\n        self.do1 = nn.Dropout(0.1)\n        self.do2 = nn.Dropout(0.25)\n\n        #batch-normalization:\n        self.bn1 = nn.BatchNorm2d(32)\n        self.bn2 = nn.BatchNorm2d(64)\n\n    def forward(self, x):\n        # conv 1\n        x = self.conv1(x) # 28*28*1 >(32 3*3 pad 1)> 28*28*32\n        x = self.bn1(x)\n        x = self.relu(x)\n        x = self.do1(x)\n        x = self.pool(x)\n        # conv 2\n        x = self.conv2(x) # 14*14*32 >(64 3*3 pad 1)> 14*14*64\n        x = self.bn2(x)\n        x = self.relu(x)\n        x = self.do1(x)\n        x = self.pool(x) # 7*7*64 \n        # fc and output\n        x = x.view(-1,7*7*64)\n        x = self.ff1(x)\n        x = self.do2(x)\n        out = self.out(x)\n\n        return out #returns class probabilities for each image","metadata":{"id":"bWnrJBT6Jhyt","execution":{"iopub.status.busy":"2021-06-06T14:23:26.845296Z","iopub.execute_input":"2021-06-06T14:23:26.845675Z","iopub.status.idle":"2021-06-06T14:23:26.856392Z","shell.execute_reply.started":"2021-06-06T14:23:26.845636Z","shell.execute_reply":"2021-06-06T14:23:26.855488Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Initalize the model/loss function and use GPU if you can\nHere I printed the dimesion of the parameters for each layer. For example, the first Conv2d layer has 32*1*3*3 (weight) + 32 (bias) parameters which is 320","metadata":{}},{"cell_type":"code","source":"model = CNN(num_classes=10)\nfor p in model.parameters():\n    print(p.size())\ngpu_boole = torch.cuda.is_available()\ndevice = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\nprint(gpu_boole, device)\nmodel = model.to(device)","metadata":{"id":"k-5wi81LWQD7","outputId":"1f9471c9-a06d-4e35-edf7-a59def9af6c0","execution":{"iopub.status.busy":"2021-06-06T14:23:26.857860Z","iopub.execute_input":"2021-06-06T14:23:26.858209Z","iopub.status.idle":"2021-06-06T14:23:30.960037Z","shell.execute_reply.started":"2021-06-06T14:23:26.858157Z","shell.execute_reply":"2021-06-06T14:23:30.958900Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I Use CrossEntropyLoss() for the Loss function. Use Adam optimizer for the model's parameter optimization. (I used the default setting for the Adam) <br>\nI also made train_eval, val_eval and test_eval (if necessary, not actually used here) for online evaluation of the model each epoch. I don't want to print or record the loss or accuracy of each batch but each epoch. <br>\nA few things to pay attention: <br>\n1. If you use GPU, your data and label need to be on GPU as well. data.cuda() does the trick.\n2. model.train() tells Pytorch it should run model in the training mode. Sometimes we use random dropout during training but we don't want to have these drop out during validation or prediction. model.eval() tells Pytorch it should run the model in the evaluation mode.","metadata":{}},{"cell_type":"code","source":"criterion = nn.CrossEntropyLoss()\noptimizer = torch.optim.Adam(model.parameters(), lr = 0.001)\n\ndef train_eval():\n    \n    model.train()\n\n    correct = 0\n    total = 0\n    loss_sum = 0\n    # forward pass\n    with torch.no_grad():\n        for data, label in train_loader:\n            if gpu_boole:\n                data, label = data.cuda(), label.cuda()\n            output = model(data)\n            # calculate loss and error rate\n            loss_sum += criterion(output, label)\n            _, pred_label = torch.max(output.data, 1)\n            total += label.size(0)\n            correct += (pred_label.float() == label.float()).sum()\n\n    # return average loss and average accuracy\n    # after looping through the whole dataset\n    return loss_sum.cpu().data.numpy().item()/total, 100.*correct/total \n\ndef val_eval():\n\n    model.eval()\n\n    correct = 0\n    total = 0\n    loss_sum = 0\n    # forward pass\n    with torch.no_grad():\n        for data, label in val_loader:\n            if gpu_boole:\n                data, label = data.cuda(), label.cuda()\n            output = model(data)\n            # calculate loss and error rate\n            loss_sum += criterion(output, label)\n            _, pred_label = torch.max(output.data, 1)\n            total += label.size(0)\n            correct += (pred_label.float() == label.float()).sum()\n\n    # return average loss and average accuracy\n    # after looping through the whole dataset\n    return loss_sum.cpu().data.numpy().item()/total, 100.*correct/total \n\ndef test_eval():\n\n    model.eval()\n\n    correct = 0\n    total = 0\n    loss_sum = 0\n    # forward pass\n    with torch.no_grad():\n        for data, label in test_loader:\n            if gpu_boole:\n                data, label = data.cuda(), label.cuda()\n            output = model(data)\n            # calculate loss and error rate\n            loss_sum += criterion(output, label)\n            _, pred_label = torch.max(output.data, 1)\n            total += label.size(0)\n            correct += (pred_label.float() == label.float()).sum()\n\n    # return average loss and average accuracy\n    # after looping through the whole dataset\n    return loss_sum.cpu().data.numpy().item()/total, 100.*correct/total ","metadata":{"id":"qk44NqVYYqqD","execution":{"iopub.status.busy":"2021-06-06T14:23:30.964693Z","iopub.execute_input":"2021-06-06T14:23:30.965054Z","iopub.status.idle":"2021-06-06T14:23:30.991423Z","shell.execute_reply.started":"2021-06-06T14:23:30.965020Z","shell.execute_reply":"2021-06-06T14:23:30.990460Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 5. Run the model\nYou can initialize the weights in your model. Alternatively, you can make this a class function inside your model. You can also use many other initialization but I don't think it would make a huge difference for this project.<br>\nSimilarly, tells Pytorch you are in training model by calling model.train() and make sure your model and data are on the GPU. ","metadata":{}},{"cell_type":"code","source":"def init_weights(model):\n    # reference: https://stackoverflow.com/a/49433937\n    if type(model) == nn.Linear or type(model) == nn.Conv2d:\n        torch.nn.init.xavier_normal_(model.weight)\nmodel.apply(init_weights)\n\nresults = []\nloss_per_epoch = []\nn_epochs = 50\nn_epochs_per_eval = 5\nbest_train_acc = 0.\nbest_val_acc = 0.\nfor epoch in range(n_epochs):\n    model.train()\n    for i,(data,label) in enumerate(train_loader):\n        if gpu_boole:\n            data, label = data.cuda(), label.cuda()\n        # calculate loss and gradient\n        if i > 0 or epoch > 0:\n            optimizer.zero_grad()\n        output = model.forward(data)\n        loss = criterion(output, label)\n        loss.backward()\n\n        # call the optimizer\n        optimizer.step()\n    \n    if epoch % n_epochs_per_eval == 0:\n        train_loss, train_acc = train_eval()\n        val_loss, val_acc = val_eval()\n        if train_acc > best_train_acc:\n            best_train_acc = train_acc\n        if val_acc > best_val_acc:\n            best_val_acc = val_acc\n            print('Step {:5d}: We have a best validation accuracy of {:.3f}%.'.format(epoch, val_acc))\n        results.append([epoch,train_loss,train_acc,val_loss, val_acc])","metadata":{"id":"R1VtkuLMel3Q","outputId":"15a6e093-1ddb-471f-acda-334c66e22085","execution":{"iopub.status.busy":"2021-06-06T14:23:30.992833Z","iopub.execute_input":"2021-06-06T14:23:30.993158Z","iopub.status.idle":"2021-06-06T14:51:31.510502Z","shell.execute_reply.started":"2021-06-06T14:23:30.993125Z","shell.execute_reply":"2021-06-06T14:51:31.509633Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, ax = plt.subplots(2,1,sharex=True)\nepoch, train_loss, train_acc, val_loss, val_acc = zip(*results)\nax[0].plot(epoch, train_loss, epoch, val_loss)\nax[0].legend(['Train loss', 'Validation Loss'])\nax[1].plot(epoch, train_acc, epoch, val_acc)\nax[1].legend(['Train accuracy', 'Validation accuracy'])\nprint ('best train accuracy: ', best_train_acc,'best validation accuracy: ',best_val_acc)","metadata":{"id":"HVZrGILu498r","outputId":"d4094d3a-e489-410d-b53a-e4ae19a87682","execution":{"iopub.status.busy":"2021-06-06T14:51:31.512544Z","iopub.execute_input":"2021-06-06T14:51:31.512891Z","iopub.status.idle":"2021-06-06T14:51:31.758248Z","shell.execute_reply.started":"2021-06-06T14:51:31.512847Z","shell.execute_reply":"2021-06-06T14:51:31.757296Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Compared with my Version 1, you don't really see overfitting happening, which means that adding drop out layer and more data augmentation helps. However, the loss curve goes kind of flat and perhaphs we can tune the Adam optimizer for a better optimization result.","metadata":{}},{"cell_type":"code","source":"### make a prediction on the test set\nmodel.eval()\nid = []\nlabel = []\nwith torch.no_grad():\n    for i, data in enumerate(test_loader):\n        if gpu_boole:\n            data = data.cuda()\n        output = model(data)\n        _, pred_label = torch.max(output.data, 1)\n        id.append(i+1)\n        label.append(pred_label.cpu().item())\ndf_pred = pd.DataFrame({'ImageID':id, 'Label': label})\n# df_pred.to_csv('/content/drive/MyDrive/Colab Notebooks/Digit Recognizer/prediction.csv', index = False)\ndf_pred.to_csv('prediction.csv', index = False)","metadata":{"id":"KI-m73Wnq__-","execution":{"iopub.status.busy":"2021-06-06T14:51:31.759724Z","iopub.execute_input":"2021-06-06T14:51:31.760067Z","iopub.status.idle":"2021-06-06T14:52:11.770222Z","shell.execute_reply.started":"2021-06-06T14:51:31.760030Z","shell.execute_reply":"2021-06-06T14:52:11.769408Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"What if I use all the data from train.csv for the model training? Spoiler alert, it doesn't improve too much. Honestly I think it's nice to use some data for the validation purpose so that you can monitor your model performance better and identify overfitting early.","metadata":{}},{"cell_type":"code","source":"train = MyMNIST(df, train_transform, test = False)\ntrain_loader = DataLoader(train, batch_size = 100, shuffle = True)\n\nmodel.apply(init_weights)\n\nresults = []\nloss_per_epoch = []\nn_epochs = 50\nn_epochs_per_eval = 5\nbest_train_acc = 0.\nfor epoch in range(n_epochs):\n    model.train()\n    for i,(data,label) in enumerate(train_loader):\n        if gpu_boole:\n            data, label = data.cuda(), label.cuda()\n        # calculate loss and gradient\n        if i > 0 or epoch > 0:\n            optimizer.zero_grad()\n        output = model.forward(data)\n        loss = criterion(output, label)\n        loss.backward()\n\n        # call the optimizer\n        optimizer.step()\n\n    if epoch % n_epochs_per_eval == 0:\n        train_loss, train_acc = train_eval()\n        if train_acc > best_train_acc:\n            best_train_acc = train_acc\n            print('Step {:5d}: We have a best training accuracy of {:.3f}%.'.format(epoch, train_acc))\n            \n        results.append([epoch,train_loss,train_acc])\nfig, ax = plt.subplots(2,1,sharex=True)\nepoch, train_loss, train_acc = zip(*results)\nax[0].plot(epoch, train_loss)\nax[0].legend(['Train loss'])\nax[1].plot(epoch, train_acc)\nax[1].legend(['Train accuracy'])\nprint ('best train accuracy: ', best_train_acc)\n\nmodel.eval()\nid = []\nlabel = []\nwith torch.no_grad():\n    for i, data in enumerate(test_loader):\n        if gpu_boole:\n            data = data.cuda()\n        output = model(data)\n        _, pred_label = torch.max(output.data, 1)\n        id.append(i+1)\n        label.append(pred_label.cpu().item())\ndf_pred = pd.DataFrame({'ImageID':id, 'Label': label})\n# df_pred.to_csv('/content/drive/MyDrive/Colab Notebooks/Digit Recognizer/prediction_v2.csv', index = False)\ndf_pred.to_csv('prediction_with_all.csv', index = False)","metadata":{"id":"wIWZlOxSzx95","outputId":"554f1cec-6a09-47d2-e41b-f49f560d8f75","execution":{"iopub.status.busy":"2021-06-06T14:52:11.771489Z","iopub.execute_input":"2021-06-06T14:52:11.771811Z","iopub.status.idle":"2021-06-06T15:23:12.228968Z","shell.execute_reply.started":"2021-06-06T14:52:11.771779Z","shell.execute_reply":"2021-06-06T15:23:12.228202Z"},"trusted":true},"execution_count":null,"outputs":[]}]}