{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Early Stopping - Vanilla RNN - PyTorch\n\n**RNN(Recurrent neural networks) the definition:**\n The neurone is a building block of the human brain. It analyses complex signals within microseconds and sends signals to the nervous system to perform tasks. \n \n The architecture of the neurone is the same for every neurone which means structural layers do not change from neurone to neurone. Make these layers successive(recurrent) and it could very easily replicate our brain. These successive “layers\" helps us with our daily activities, complex decision making, language processing.\n \n But, how to generalise our problem across these layers? What kind of modelling would be required to generalise?\nThe answer came to researchers in the form of parameter sharing. It helps extend and apply the model to different forms of data. This is done by sharing members of the output as a function of previous members of the output. The members of the output are produced by the same update rule. An easier way to comprehend this structure of computations would be to use ‘Unfolding computational graphs’. Unfolding of graph results in sharing the deep network of parameters in the structure.\n \n The unfolding process has some major advantages and leads to factors that make it possible to make the model f ubiquitous which further allows generalisation.\n\n* Despite the length of the input sequence, the model has the same input size.\n* It is possible to use the same transition function f with the same parameters at every time step because it is specified from one state to another. The unfolded graph illustrates the idea of explicit description and information flow both forward and backward in time by showing the path along  this information flows.\n\nThese ideas were important to building the recurrent neural network as RNN produced output at each time step and had connections between hidden units could produce an output by reading an entire sequence and then produce a single output. This leads to a conclusion that any function that is computable by a Turing machine can be computed by a recurrent neural network of a finite size. It is this nature of using past outputs, hidden layers connections that have led RNN to accomplish its laurels today."},{"metadata":{},"cell_type":"markdown","source":"The computation in RNN can be decomposed to three blocks of parameters:\n\n 1. From input to hidden state\n 2. From previous hidden state to present hidden state\n 3. From hidden state to the output\n\n Each of these blocks is associated with a separate weight matrix. When the network is unfolded each of these blocks correspond to shallow transformation(transformation that affects a single layer).\n"},{"metadata":{},"cell_type":"markdown","source":"\n![RNN](https://i.imgur.com/vEFxN0k.png)"},{"metadata":{},"cell_type":"markdown","source":"## 1. Import data and Directories"},{"metadata":{"trusted":true},"cell_type":"code","source":"import torch\nimport torch.nn as nn\nimport torchvision.transforms as transforms\nimport torchvision.datasets as dsets\nimport argparse\nimport numpy as np\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\nfrom torch.autograd import Variable\n\n\n\n\ntrain_dataset = pd.read_csv('../input/digit-recognizer/train.csv',dtype = np.float32)\n\ntest_dataset = pd.read_csv('../input/digit-recognizer/test.csv',dtype = np.float32)\n\ntargets_numpy = train_dataset.label.values\nfeatures_numpy = train_dataset.loc[:,train_dataset.columns != \"label\"].values/255 # normalization\n\n# Using SKLEARN we train test split. Size of train data is 80% and size of test data is 20%. \nfeatures_train, features_test, targets_train, targets_test = train_test_split(features_numpy,\n                                                                             targets_numpy,\n                                                                             test_size = 0.2,\n                                                                             random_state = 42) \n\n# create feature and targets tensor for train set. As you remember we need variable to accumulate gradients. Therefore first we create tensor, then we will create variable\nfeaturesTrain = torch.from_numpy(features_train)\ntargetsTrain = torch.from_numpy(targets_train).type(torch.LongTensor) # data type is long\n\n# create feature and targets tensor for test set.\nfeaturesTest = torch.from_numpy(features_test)\ntargetsTest = torch.from_numpy(targets_test).type(torch.LongTensor) # data type is long\n\n# batch_size, epoch and iteration\nbatch_size = 100\nn_iters = 10000\nnum_epochs = n_iters / (len(features_train) / batch_size)\nnum_epochs = int(num_epochs)\n\n# Pytorch train and test sets\ntrain = torch.utils.data.TensorDataset(featuresTrain,targetsTrain)\ntest = torch.utils.data.TensorDataset(featuresTest,targetsTest)\n\n# data loader\ntrain_loader = torch.utils.data.DataLoader(train, batch_size = batch_size, shuffle = False)\ntest_loader = torch.utils.data.DataLoader(test, batch_size = batch_size, shuffle = False)\n\n\n#print(train_dataset.train_data.size())\n\n#print(train_dataset.train_labels.size())\n#Here we would have 10k testing images of the same size, 28 x 28 pixels.\n\n\n#print(test_dataset.test_data.size())\n\n#print(test_dataset.test_labels.size())\n\n\n\ntrain_loader = torch.utils.data.DataLoader(dataset=train_dataset, \n                                           batch_size=batch_size, \n                                           shuffle=True)\n\ntest_loader = torch.utils.data.DataLoader(dataset=test_dataset, \n                                          batch_size=batch_size, \n                                          shuffle=False)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2. Create RNN\nThe above figure Computation Graph to compute training loss of recurrent neural network.The sequence of output values o is compared to the training targets y, this leads to the computation of the loss function. We assume o is the unnormalised log probabilities. The loss function L internally computes y^ = softmax(o) and compares this to target y.The RNN has input to hidden connections parameterised by a weight matrix U, parameterised by a weight matrix W, and hidden to output connection parameterised by a weight matrix V."},{"metadata":{"trusted":true},"cell_type":"code","source":"\n# Create RNN Model\n\nclass RNNModel(nn.Module):\n    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim):\n        super(RNNModel, self).__init__()\n        # Hidden dimensions\n        self.hidden_dim = hidden_dim\n\n        # Number of hidden layers\n        self.layer_dim = layer_dim\n\n        # Building your RNN\n        # batch_first=True causes input/output tensors to be of shape\n        # (batch_dim, seq_dim, input_dim)\n        # batch_dim = number of samples per batch\n        self.rnn = nn.RNN(input_dim, hidden_dim, layer_dim, batch_first=True, nonlinearity='tanh')\n\n        # Readout layer\n        self.fc = nn.Linear(hidden_dim, output_dim)\n\n    def forward(self, x):\n        # Initialize hidden state with zeros\n        # (layer_dim, batch_size, hidden_dim)\n        h0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).requires_grad_()\n\n        # We need to detach the hidden state to prevent exploding/vanishing gradients\n        # This is part of truncated backpropagation through time (BPTT)\n        out, hn = self.rnn(x, h0.detach())\n\n        # Index hidden state of last time step\n        # out.size() --> 100, 28, 10\n        # out[:, -1, :] --> 100, 10 --> just want last time step hidden states! \n        out = self.fc(out[:, -1, :]) \n        # out.size() --> 100, 10\n        return out\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3. *About the model*\n\nOur model has 3 hidden layers, 100 hidden neurons (per layer) and takes in an input data of 28 dimensions while letting out a 10-dimensional data. The activation function we assume is the hyperbolic ‘tanh’ . The stochastic gradient descent is used to find the gradient of the cost function of a single example at each iteration instead of the sum of the gradient of the cost function of all the examples."},{"metadata":{"trusted":true},"cell_type":"code","source":"# batch_size, epoch and iteration\nbatch_size = 100\nn_iters = 3000\nnum_epochs = n_iters / (len(features_train) / batch_size)\nnum_epochs = int(num_epochs)\n\n# Pytorch train and test sets\ntrain = torch.utils.data.TensorDataset(featuresTrain,targetsTrain)\ntest = torch.utils.data.TensorDataset(featuresTest,targetsTest)\n\n# data loader\ntrain_loader = torch.utils.data.DataLoader(train, batch_size = batch_size, shuffle = False)\ntest_loader = torch.utils.data.DataLoader(test, batch_size = batch_size, shuffle = False)\n    \n# Create RNN\ninput_dim = 28    # input dimension\nhidden_dim = 100  # hidden layer dimension\nlayer_dim = 3     # number of hidden layers\noutput_dim = 10   # output dimension\n\nmodel = RNNModel(input_dim, hidden_dim, layer_dim, output_dim)\n\n# Cross Entropy Loss \nerror = nn.CrossEntropyLoss()\n\n# SGD Optimizer\nlearning_rate = 0.05\noptimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)\n\nseq_dim = 28  \nloss_list = []\niteration_list = []\naccuracy_list = []\ncount = 0\nmin_val_loss = np.Inf\nval_array = []\ncorrect = 0\niter = 0\ncount = 0\niter_array = []\nloss_array = []\ntotal = 0\naccuracy_array = []\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4. *Early Stopping*\n\nEarly Stopping is an optimisation technique done by calculating the Validation loss. If the validation loss does not decrease over a specified number of iterations the model halts its training. For our model I have not used early stopping by importing the algorithm from some other directories. Instead, I have made my own validation loss that inturn reduced computation.\n\nDropout is meant to block information from certain neurons completely to make sure the neurons do not co-adapt. So, the batch normalization has to be after dropout otherwise you are passing information through normalization statistics.\n\n The only bottleneck I faced was with nested statements, I used nested break statements to introduce **Early Stopping**."},{"metadata":{"trusted":true},"cell_type":"code","source":"n_epochs_stop = 6\nepochs_no_improve = 0\nearly_stop = False","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for epoch in range(num_epochs):\n    val_loss = 0\n    for i, (images, labels) in enumerate(train_loader):\n\n        train  = Variable(images.view(-1, seq_dim, input_dim))\n        labels = Variable(labels )\n            \n        # Clear gradients\n        optimizer.zero_grad()\n        \n        # Forward propagation\n        outputs = model(train)\n        \n        # Calculate softmax and ross entropy loss\n        loss = error(outputs, labels)\n        \n        # Calculating gradients\n        loss.backward()\n        \n        # Update parameters\n        optimizer.step()\n        val_loss += loss\n        val_loss = val_loss / len(train_loader)\n        # If the validation loss is at a minimum\n        if val_loss < min_val_loss:\n  # Save the model\n             #torch.save(model)\n             epochs_no_improve = 0\n             min_val_loss = val_loss\n  \n        else:\n            epochs_no_improve += 1\n        iter += 1\n        if epoch > 5 and epochs_no_improve == n_epochs_stop:\n            print('Early stopping!' )\n            early_stop = True\n            break\n        else:\n            continue\n        break\n        if iter % 336 == 0:\n            # Calculate Accuracy         \n            correct = 0\n            total = 0\n            #print(iter)\n            # Iterate through test dataset\n  # Check early stopping condition\n        \n    if early_stop:\n        print(\"Stopped\")\n        break\n        \n        \n        \n    for images, labels in test_loader:\n       \n                # Resize images\n        images = images.view(-1, seq_dim, input_dim)\n\n                # Forward pass only to get logits/output\n        outputs = model(images)\n\n                # Get predictions from the maximum value\n        _, predicted = torch.max(outputs.data, 1)\n\n                # Total number of labels\n        total += labels.size(0)\n\n                # Total correct predictions\n        correct += (predicted == labels).sum()\n\n        accuracy = 100 * correct / total\n        \n        #Print Loss\n        count = count +1\n        if iter % 336 == 0 and count % 100 == 0  : \n            iter_array.append(iter)\n            loss_array.append(loss.item())\n            accuracy_array.append(accuracy.item())\n            print('Epoch: {}. Iteration: {}. Loss: {}. Accuracy: {}, Count: {}'.format(epoch,iter, loss.item(),accuracy.item(),count))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"examples = enumerate(test_loader)\nbatch_idx, (images, labels) = next(examples)\nimages = images.numpy()\nlabels = labels.numpy()\n\nimport matplotlib.pyplot as plt\n\nfig = plt.figure()\nfor i in range(6):\n  plt.subplot(2,3,i+1)\n  plt.tight_layout()\n  plt.imshow(images[i].reshape(28,28), cmap='gray', interpolation='none')\n  plt.title(\"Number: {}\".format(labels[i]))\n  plt.xticks([])\n  plt.yticks([])\nprint(fig)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.DataFrame({'Iterations': iter_array, 'Loss': loss_array, 'Accuracy': accuracy_array})\ndf['Index'] = range(1, len(iter_array) + 1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from bokeh.plotting import figure, output_file, show\nfrom bokeh.io import output_notebook\nfrom bokeh.models import CustomJS, ColumnDataSource, Select,HoverTool,LinearInterpolator,Column\nfrom bokeh.layouts import column\nfrom bokeh.models.widgets import Div\noutput_notebook()\nsource_CDS = ColumnDataSource(df)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hover = HoverTool(tooltips = '@Loss= Loss')\nLoss_line = figure(plot_width=700, plot_height=300,tools = [hover])\n\nLoss_line.line('Iterations','Loss',source = source_CDS, line_width=2)\nLoss_line.background_fill_color = '#fffce6'\n\ntitle_div = Div(text=\"<b> Loss vs Iterations </b>\", style={'font-size': '400%', 'color': '#FF6347'})\np2 = column(title_div,Loss_line)\n\nshow(p2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"hover = HoverTool(tooltips = ' Accuracy: @Accuracy%')\nAccuracy_line = figure(plot_width=700, plot_height=300,tools = [hover])\nAccuracy_line.line('Iterations','Accuracy',source = source_CDS, line_width=2)\ntitle_div2 = Div(text=\"<b> Accuracy vs Iterations </b>\", style={'font-size': '400%', 'color': '#008080'})\nAccuracy_line.background_fill_color = '#fffce6'\np2 = column(title_div2,Accuracy_line)\nshow(p2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_dataset = pd.read_csv('../input/digit-recognizer/test.csv',dtype = np.float32)\ntest_dataset.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_dataset = torch.from_numpy(test_dataset.values)\ntest_loader = torch.utils.data.DataLoader(dataset=test_dataset, \n                                          shuffle=False)\nmodel.eval()\ntest_pred = torch.LongTensor()\nfor i, data in enumerate(test_loader):\n    \n    \n    predict = data.view(-1,seq_dim,input_dim)\n    predict = Variable(predict)\n    output = model(predict)\n    pred = output.data.max(1, keepdim=True)[1]\n    test_pred =  torch.cat((test_pred,pred),dim =0)\n        ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_pred.size()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"Submission_df = pd.DataFrame(np.c_[np.arange(1, len(test_pred.numpy())+1)[:,None], test_pred.numpy()], \n                      columns=['ImageId', 'Label'])\nprint(Submission_df.head())\n\n\n\nSubmission_df.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_dataset2 = pd.read_csv('../input/digit-recognizer/test.csv',dtype = np.float32)\ntest_dataset2 = test_dataset2.values\n\nplt.imshow(test_dataset2[4].reshape(28,28))\nprint(test_pred[4])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_pred","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from collections import Counter\nlist = Counter(Submission_df['Label'].values)\nlist","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_dataset2 =  pd.read_csv('../input/digit-recognizer/train.csv',dtype = np.float32)\n\ntrain_dataset2 = train_dataset2[\"label\"].values # normalization\n\n\n\nlist = Counter(train_dataset2)\nlist","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## New to Pytorch?? Checkout these kernels\n\n[Pytorch Tutorial](https://www.kaggle.com/kanncaa1/pytorch-tutorial-for-deep-learning-lovers) by The DATA AI Team - @kanncaa1\n\n\n[In Depth Keras Vs Pytorch](https://www.kaggle.com/littleraj30/in-depth-keras-v-s-pytorch-approach-comparison) by Rishi - @littleraj30\n\n\n[Pytorch Tutorial data preparation stage](https://www.kaggle.com/leighplt/pytorch-tutorial-dataset-data-preparetion-stage) by Leigh - @leighplt\n\n\n[MNIST: Introduction to ComputerVision with PyTorch](https://www.kaggle.com/abhinand05/mnist-introduction-to-computervision-with-pytorch) by @abhinand05\n\n\n"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}