{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\nfrom sklearn.model_selection import train_test_split\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n\nimport torch\nfrom torchvision import datasets\nimport torchvision.transforms as transforms\nimport torch.nn as nn\nimport torch.nn.functional as F\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = pd.read_csv('/kaggle/input/digit-recognizer/train.csv')\ntest = pd.read_csv('/kaggle/input/digit-recognizer/test.csv')\nsubmission = pd.read_csv('/kaggle/input/digit-recognizer/sample_submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Steps to create a Neural Networks with Pytorch\n\n1. Define a Neural Network (forward propagation)\n2. Define dataset, data loader and transformations\n3. Define optimizer and criterion\n4. Train the model (back propagation)\n5. Evaluate the model\n6. Make prediction","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Step 1: Define a Neural Networks\nThe neural network architectures in Pytorch can be defined in a class which inherits the properties from the base class from **nn** package called Module. This inheritance from the nn.Module class allows us to implement, access, and call a number of methods easily. We can define all the layers inside the constructor of the class, and the forward propagation steps inside the forward function.\n\nWe will define a simple Multilayer Perceptron with the following architecture:\n\n* Input layer\n```Python\nnn.Linear(28 * 28, 512)\n```\n    * Layer type: nn.Linear(), which refers to a fully connection layer\n    * Input size: 28*28, corresponding to the size of input data.\n    * Output size: 512, the number of \"neurons\".\n    \n* Hidden layer\n```\nnn.Linear(512, 256)\n```\n\n    * Layer type: nn.Linear()\n    * Input size: 512, output size of the previous layer(input layer).\n    * Output size: 256, the number of \"neurons\" in this layer.\n    \n* Output layer\n```\nnn.Linear(256, 10)\n```\n\n    * Layer type: nn.Linear()\n    * Input size: 256, output size of the previous layer(hidden layer).\n    * Output size: 10, the number of classes we need to predict.\n\n* Activation functions\nEach linear layer's output needs to go through an activation function to \"activate\" it. We will get started with **F.sigmoid()** but can try F.relu() or others later.\n\nThe best practice is to name each layer and initialize them in the **__init__()** function as named building blocks and put the building blocks together in the **forward()** function which defines how the data actually flows in the network. In our case, each layer simply takes the output of the previous layer and perform transformations the generate outputs in sequence.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class Net(nn.Module):\n## The following two lines are the reinforced format for contructing a Pytorch network class\n    def __init__(self):\n        super(Net, self).__init__()\n# Input layer\n        self.input = nn.Linear(28 * 28, 512)\n# Hidden layer\n        self.hidden = nn.Linear(512, 256)\n# Output layer\n        self.output = nn.Linear(256, 10)\n    \n    def forward(self, x):\n        x = x.view(-1, 28 * 28)\n        x = F.sigmoid(self.input(x))\n        x = F.sigmoid(self.hidden(x))\n        x = self.output(x)\n        return x\n    \nmodel = Net()\nprint(model)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 2: Define dataset, data loader and transformation\n\nPyTorch gives use the freedom to pretty much do anything with the Dataset class so long as you override two of the subclass functions:\n* the __len__ function which returns the size of the dataset, and\n* the __getitem__ function which returns a sample from the dataset given an index.\n\nHowever, in our case we can simply construct a TensorDataset with two items: the feature data and the target where the feature data is the matrix of pixel 1 - pixel 784 and the target is the digit of the image.\n\n\nWhile the Dataset class is a nice way of containing data systematically, it seems that in a training loop, we will need to index or slice the dataset's samples list. This is no better than what we would do for a typical list or NumPy matrix. Rather than going down that route, PyTorch supplies another utility function called the DataLoader which acts as a data feeder for a Dataset object.\n\n\nIn order to construct the data loader we will need to provide two parameters: **batch_size** which indicates how many samples we want to use to train  the model in a batch, and **shuflle**, suggesting if we want to shuffle the data before sending it to the network. \n\nTypically we would want to set batch_size as 2^N e.g. 128, 256, 512, and set shuffle as True for traning data and False for validatio and test data( you can take a moment to think why?)\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"batch_size = 128\ntransform = transforms.ToTensor()\n\n\nx_train, x_val, y_train, y_val = train_test_split(\n    train.values[:,1:], train.values[:,0], test_size=0.2)\n\n\ntrain_dataset = torch.utils.data.TensorDataset(torch.from_numpy(x_train.astype(np.float32)/255),\n                                               torch.from_numpy(y_train))\n\nval_dataset = torch.utils.data.TensorDataset(torch.from_numpy(x_val.astype(np.float32)/255),\n                                               torch.from_numpy(y_val))\n\ntest_dataset = torch.utils.data.TensorDataset(torch.from_numpy(test.values[:,:].astype(np.float32)/255))\n\n# data loader\ntrain_loader = torch.utils.data.DataLoader(train_dataset, batch_size = batch_size, shuffle = True)\nval_loader = torch.utils.data.DataLoader(val_dataset, batch_size = batch_size, shuffle = False)\ntest_loader = torch.utils.data.DataLoader(test_dataset, batch_size = batch_size, shuffle = False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 3: Define optimizer and criterion\n\nOptimizer is used to perform the gradient descent process. Here we will use SGD( Stochastic Gradient Descent). The tricky part is how to set the right size of learning rate which could have a huge impact on the final result. For now let's simply use 0.1 as the starting point but later we will revisit the options and strategies of learning rate selection.\n\nCriterion will be used to calculate the cost (or loss) so we can use the cost to do back propagation and update the weights we want to train. In our case, we will use nn.CrossEntropyLoss() since we are working on a multiclassfication problem.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"criterion = nn.CrossEntropyLoss()\noptimizer = torch.optim.SGD(model.parameters(), lr=0.1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 4 train the model (back propagation)\n\nTraining the model is an iterative process which contains many epoches. For each epoch we will repeatly load batches of data, perform forward propagation, calculate cost, perform back propagation using the optimizer.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"n_epochs = 30\nfor epoch in range(n_epochs):\n    train_loss = 0.0\n    for data, target in train_loader:\n        optimizer.zero_grad()\n        # Forward propagation\n        output = model(data)\n        # Calculate the loss\n        loss = criterion(output, target)\n        # Back propagation\n        loss.backward()\n        # Update weights using the optimizer\n        optimizer.step()\n        # Calculate the cumulated loss\n        train_loss += loss.item()*data.size(0)\n    \n    train_loss = train_loss/len(train_loader.dataset)\n    \n    print(f\"Epoch: {epoch}, train loss: {train_loss}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 5: Evaluate the model\n\nWe will use the trained model to make predictions on the validation dataset and compare the predictions against the actual targets. Dataloader will be used to iterate the validation dataset as well.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"val_loss = 0.0\nclass_correct = list(0. for i in range(10))\nclass_total = list(0. for i in range(10))\n\nmodel.eval() # prep model for evaluation\n\nfor data, target in val_loader:\n    # forward pass: compute predicted outputs by passing inputs to the model\n    output = model(data)\n    # calculate the loss\n    loss = criterion(output, target)\n    # update val loss \n    val_loss += loss.item()*data.size(0)\n    # convert output probabilities to predicted class\n    _, pred = torch.max(output, 1)\n    # compare predictions to true label\n    correct = np.squeeze(pred.eq(target.data.view_as(pred)))\n    # calculate val accuracy for each object class\n    for i in range(len(target)):\n        label = target.data[i]\n        class_correct[label] += correct[i].item()\n        class_total[label] += 1\n\n# calculate and print avg val loss\nval_loss = val_loss/len(val_loader.sampler)\nprint('val Loss: {:.6f}\\n'.format(val_loss))\n\nfor i in range(10):\n    if class_total[i] > 0:\n        print('val Accuracy of %5s: %2d%% (%2d/%2d)' % (\n            str(i), 100 * class_correct[i] / class_total[i],\n            np.sum(class_correct[i]), np.sum(class_total[i])))\n    else:\n        print('val Accuracy of %5s: N/A (no training examples)' % (classes[i]))\n\nprint('\\nval Accuracy (Overall): %2d%% (%2d/%2d)' % (\n    100. * np.sum(class_correct) / np.sum(class_total),\n    np.sum(class_correct), np.sum(class_total)))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Step 6: make prediction\n\nThis step is similar to the validation step except that we are not comparing the predictions as there's no ground truth of target to compare with.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"model.eval() # prep model for evaluation\n\npreds = []\n\nfor data in test_loader:\n    # forward pass: compute predicted outputs by passing inputs to the model\n    output = model(data[0])\n    # calculate the loss\n    _, pred = torch.max(output, 1)\n    preds.extend(pred.tolist())\n    # compare predictions to true label\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"submission['Label'] = preds\nsubmission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Assignments:\n\n1. Can you implment the following functions for model training, evaluation and prediction so we can reuse them when we need to test different things afterwards without having to replicate the codes every time.\n\n```python\n\ndef train_model(nn_model, train_loader, optimizer, criterion, n_epoch):\n    # YOUR IMPLEMENTATION\n    return nn_model\n\ndef eval_model(nn_model, val_loader):\n    # YOUR IMPLEMENTATION\n    return calculated_accuracy\n\n\ndef predict(nn_model, test_loader):\n    # YOUR IMPLEMENTATION\n    return predictions\n\n```\n\n\n2. F.sigmoid() was used as the activation function for the MLP model we implemented. Can you try other activation functions such as F.relu() and F.tanh()? You may want to refer to the [PyTorch Functional](https://pytorch.org/docs/stable/nn.functional.html) for more details. Use the train_model(), eval_model() functions you implmented so you don't have to repeat the same codes.\n\n3. Can you add a dropout layer between input and the hidden layer, and another one between the hidden layer and the output layer?\n\n4. Try to add/ remove hiden layers, as well as different number of neurons and report your validation results.\n\n5. Try different learning rate (0.0001, 0.001, 0.01, 0.1). What is your observation?\n\n6. Try different values for batch_size(64, 128, 256, 512).\n\n7. Try different values for n_epochs.\n\n8. What can you think of if we want to improve the current model?\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}