{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Digit Recognizer","metadata":{}},{"cell_type":"markdown","source":"Hello everyone,\n\nI'm going to implement a **neural network** using **PyTorch** to predict hand written digits.\n\nIn this notebook, I will **NOT** implement **Convolutional NN**. For now, my plan is using a simple model with an input layer, 2 hidden layers and an output layer. \n\nAfter finishing this project, i am also planning to implement CNN in another notebook.","metadata":{}},{"cell_type":"markdown","source":"# 1. Preparing Data","metadata":{}},{"cell_type":"markdown","source":"In this section, I will\n* import packages\n* import csv files\n* check missing values\n* plot some images\n* create tensors and dataloaders","metadata":{}},{"cell_type":"code","source":"# importing packages\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom sklearn.model_selection import train_test_split\n%matplotlib inline\n\nimport torch\nfrom torch import nn\nfrom torch import optim\nfrom torch.utils.data import DataLoader, TensorDataset","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pd.read_csv(\"../input/digit-recognizer/train.csv\")\ntest = pd.read_csv(\"../input/digit-recognizer/test.csv\")\ntrain.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Checking if there are any missing values:\")\nprint(\"Train: {}\".format(train.isnull().sum().sum()))\nprint(\"Test: {}\".format(test.isnull().sum().sum()))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Splitting train data set into train and validation sets\n# validation size is selected as 0.2\ntrain_X, val_X, train_y, val_y = train_test_split(train.drop(\"label\", axis=1), train[\"label\"], test_size=0.20)\n\nprint(\"Shape of training set: {}\".format(train_X.shape))\nprint(\"Shape of validation set: {}\".format(val_X.shape))\nprint(\"Shape of test set: {}\".format(test.shape))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Lets plot some random images from training data and see the labels\nindices = [42, 314, 2022, 33333]\n\nf,ax = plt.subplots(1, len(indices))\nfor i in range(len(indices)):\n    title = \"Label: {}\".format(train_y.iloc[indices[i]])\n    ax[i].imshow( train_X.iloc[indices[i]].values.reshape(28,28) )\n    ax[i].set_title(title)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Converting train, validation and test image data into tensors\n# dividing by 255 is for normalization\ntrain_X_tensor = torch.tensor(train_X.values)/255.0\nval_X_tensor = torch.tensor(val_X.values)/255.0\ntest_tensor = torch.tensor(test.values)/255.0\n\n# Converting train and validation labels into tensors\ntrain_y_tensor = torch.tensor(train_y.values)\nval_y_tensor = torch.tensor(val_y.values)\n\n# Creating train and validation tensors\ntrain_tensor = TensorDataset(train_X_tensor, train_y_tensor)\nval_tensor = TensorDataset(val_X_tensor, val_y_tensor)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Defining the dataloaders\ndataloaders = dict()\ndataloaders[\"train\"] = DataLoader(train_tensor, batch_size=32, shuffle=True)\ndataloaders[\"val\"] = DataLoader(val_tensor, batch_size=16, shuffle=True)\ndataloaders[\"test\"] = DataLoader(test_tensor, batch_size=16)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2. Model Initiation","metadata":{}},{"cell_type":"markdown","source":"In this section, I will define\n* nn model\n* validation function\n* training function\n* criterion and optimizer","metadata":{}},{"cell_type":"code","source":"# Creating neural network model:\n#   Layers: one input, one output and 2 hidden\ninput_size = 784\noutput_size = 10\nhidden_layers = [512,128]\np_drop = 0.25\nmodel_recognizer = nn.Sequential(nn.Linear(input_size, hidden_layers[0]),\n                                 nn.ReLU(),\n                                 nn.Dropout(p=p_drop),\n                                 nn.Linear(hidden_layers[0], hidden_layers[1]),\n                                 nn.ReLU(),\n                                 nn.Dropout(p=p_drop),\n                                 nn.Linear(hidden_layers[1], output_size),\n                                 nn.LogSoftmax(dim=1))\nprint(model_recognizer)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"### VALIDATION FUNCTION\ndef validation(model, loader, criterion, device=\"cpu\"):\n    model.eval()\n    loss = 0\n    acc = 0\n    \n    with torch.no_grad():\n        for images, labels in loader:\n            images, labels = images.to(device), labels.to(device)\n            \n            output = model.forward(images)\n            loss += criterion(output,labels).item()\n            \n            probs = torch.exp(output)\n            equality = (labels.data == probs.max(dim=1)[1])\n            acc += equality.type(torch.FloatTensor).mean()\n    res_loss = loss/len(loader)\n    res_acc = (acc.item())/len(loader)\n    return res_loss, res_acc","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"### TRAINING FUNCTION\ndef train_model(model, trainloader, validloader, criterion, optimizer, epochs=3, print_every=40, device=\"cpu\"):\n    model.to(device)\n    \n    train_loss_per_epoch = []\n    valid_loss_per_epoch = []\n    \n    for e in range(epochs):\n        running_loss = 0\n        steps = 0\n        model.train()\n        \n        for images, labels in trainloader:\n            steps += 1\n            images, labels = images.to(device), labels.to(device)\n            \n            # Training \n            optimizer.zero_grad()\n            output = model.forward(images)\n            loss = criterion(output, labels)\n            loss.backward()\n            optimizer.step()\n            \n            running_loss += loss.item()\n            \n            # Printing current loss and accuracy\n            if steps % print_every == 0:\n                model.eval()\n                valid_loss, valid_accuracy = validation(model, validloader, criterion, device)\n                \n                to_print = \"Epoch: \"+str(e+1)+\" of \"+str(epochs)\n                to_print += \"... Train Loss: {:.3f}\".format(running_loss/print_every)\n                to_print += \"... Valid Loss: {:.3f}\".format(valid_loss)\n                to_print += \"... Valid Accuracy: {:.3f}\".format(valid_accuracy)\n                print(to_print)\n                \n                running_loss = 0\n                model.train()\n        # at the end of each epoch calculate loss:\n        train_loss, train_accuracy = validation(model, trainloader, criterion, device)\n        valid_loss, valid_accuracy = validation(model, validloader, criterion, device)\n        train_loss_per_epoch.append(train_loss)\n        valid_loss_per_epoch.append(valid_loss)\n    return train_loss_per_epoch, valid_loss_per_epoch","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define criterion and optimizer:\ncriterion = nn.NLLLoss()\noptimizer = optim.Adam(model_recognizer.parameters(), lr=0.001)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 3. Training the Model","metadata":{}},{"cell_type":"markdown","source":"In this section, I will\n* TRAIN THE MODEL !!!","metadata":{}},{"cell_type":"code","source":"# \"epoch\" is the number of times that model will train the entire train dataset\n# we have soooo many images, i don't want to print something on the screen for each batch\n# so in every \"print_every\" steps, i will print accuracy and loss\n# i selected device as cpu since the model is not too complicated\n# gpu is an option for faster training (device=\"cuda\")\ntrain_loss, valid_loss = train_model(model = model_recognizer,\n                                     trainloader = dataloaders[\"train\"],\n                                     validloader = dataloaders[\"val\"],\n                                     criterion = criterion,\n                                     optimizer = optimizer,\n                                     epochs = 8,\n                                     print_every = 300,\n                                     device =\"cpu\"\n                                     )","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"Training loss in each epoch:\")\nprint(train_loss)\nprint()\nprint(\"Validation loss in each epoch:\")\nprint(valid_loss)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# I have trained the model for 16 epochs\n# for future plots, I want to save these values in the variables below:\ntrain_loss_16_epochs = [0.13787669561581598, 0.07582466917600306, 0.05383466555792395, 0.034825331529801974, 0.028129039008843913, 0.020988315712311304, 0.02332191020201814, 0.017009653160293098, 0.016772397063941753, 0.021548685492307415, 0.009186462120274947, 0.011247794017871716, 0.010653088813236254, 0.008269465251884347, 0.009493477538097289, 0.005730943549323689]\nvalid_loss_16_epochs = [0.17423813392363843, 0.12283760847795444, 0.11133580147956743, 0.09436281601324611, 0.10040175634666368, 0.09744789005078063, 0.1031095781708843, 0.09291011415699922, 0.10281138533085445, 0.12173442720597276, 0.10909194191648783, 0.10568566672965449, 0.1245951993337975, 0.11474877005043205, 0.12414950424937972, 0.11632858605553643]","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Lets plot training and validation loss\n# I will use results of 16 epochs\ntrain_loss = train_loss_16_epochs\nvalid_loss = valid_loss_16_epochs\nx = np.arange(len(train_loss)) + 1 # epoch array\n\n# finding minimum validation loss for annotation\nvalid_min = min(valid_loss)\nvalid_ind = valid_loss.index(valid_min)\nx_min = x[valid_ind]\nannot_text = \"Min Valid Loss\\n\"\nannot_text += \"Loss: {:.4f}\\n\".format(valid_min)\nannot_text += \"Epoch: {}\".format(x_min)\n\n# Plot\nplt.subplots(figsize=(10, 6))\nplt.plot(x, train_loss, color=\"blue\", lw=2, ls='-', marker='h', label = \"Train Loss\")\nplt.plot(x, valid_loss, color=\"red\",  lw=2, ls='-', marker='d', label = \"Valid Loss\")\nplt.annotate(annot_text, xy=(x_min, valid_min), xytext=(x_min-2, valid_min+0.05),\n             arrowprops=dict(arrowstyle=\"fancy\"))\nplt.legend()\nplt.show()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In the training section, first I have trained my model for 16 epochs. By plotting the loss values of each epoch, I see that:\n* Training loss keeps decreasing. It is expected since the model learns from train dataset.\n* But validation loss stops decreasing and at some point starts increasing.\n* That's because overfitting. The model learns too much about the training data.\n* The minimum value for validation loss is after 8 epochs.\n* As a result, i will train my model for 8 epochs and make a submission.","metadata":{}},{"cell_type":"markdown","source":"# 4. Prediction","metadata":{}},{"cell_type":"markdown","source":"In this section, I will\n* define function for prediction\n* predict the test dataset\n* submit my results","metadata":{}},{"cell_type":"code","source":"### PREDICTION FUNCTION\ndef prediction(model, loader, device=\"cpu\"):\n    model.to(device)\n    model.eval()\n    preds_all = torch.LongTensor()\n    \n    with torch.no_grad():\n        for images in loader:\n            images = images.to(device)\n            \n            output = model.forward(images)            \n            probs = torch.exp(output)\n            pred = probs.to('cpu').max(dim=1)[1]\n            preds_all = torch.cat((preds_all, pred), dim=0)\n    return preds_all","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y_pred = prediction(model_recognizer, dataloaders[\"test\"])\ny_pred","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Creating a dataframe for results\nresult = pd.DataFrame({'ImageId': test.index, 'Label': y_pred})\nresult[\"ImageId\"] += 1\nresult.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"result.to_csv('submission.csv', index=False)\nprint(\"Resuls are saved to submission.csv\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]}]}