\nIlustrated Guide to Recurrent Neural Networks: Understanding the Intuition
\nFrom Michael Phi
\n
\n\n# 1. Introduction π\n\nThis notebook is just me being frustrated on **deep learning** and trying to understand in \"baby steps\" what is going on here. For somebody that starts in this area with no background whatsoever it can be very confusing, especially because I seem to be unable to find code with many explanations and comments.\n\nSo, if you are frustrated just like I was when I started this stuff I hope the following guidelines will help you. I am by no means a teacher, but in this notebook I will:\n1. Share articles/videos I watched that TRULY helped\n2. Explain code along the way to the best of my ability\n\n
\n\n# 2. Before we start π\n\n> This is my third notebook in the \"series\": **How I taught myself Deep Learning**.\n1. **[How I taught myself Deep Learning: Vanilla NNs](https://www.kaggle.com/andradaolteanu/how-i-taught-myself-deep-learning-1-pytorch-fnn)**\n * PyTorch and Tensors\n * Neural Network Basics, Perceptrons and a Plain Vanilla Neural Net model\n * MNIST Classification using FNN\n * Activation Functions\n * Forward Pass\n * Backpropagation (Loss and Optimizer Functions)\n * Batching, Iterations and Epochs\n * Computing Classification Accuracy\n * Overfitting: Data Augmentation, Weight Decay, Learning Rate, Dropout() and Layer Optimization \n2. **[Convolutional Neural Nets (CNNs) Explained](https://www.kaggle.com/andradaolteanu/convolutional-neural-nets-cnns-explained)**\n * Why ConvNets\n * Convolutions Explained\n * Computing Activation Maps\n * Kernels, Padding, Stride\n * AlexNet\n * MNIST Classification using Convolutions","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Imports\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\nimport os\nimport numpy as np\n\nimport torchvision\nimport torchvision.transforms as transforms\n\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport sklearn.metrics\nimport seaborn as sns\nimport random\n\n# To display youtube videos\nfrom IPython.display import YouTubeVideo\n\n\ndef set_seed(seed = 1234):\n '''Sets the seed of the entire notebook so results are the same every time we run.\n This is for REPRODUCIBILITY.'''\n np.random.seed(seed)\n random.seed(seed)\n torch.manual_seed(seed)\n torch.cuda.manual_seed(seed)\n # When running on the CuDNN backend, two further options must be set\n torch.backends.cudnn.deterministic = True\n # Set a fixed value for the hash seed\n os.environ['PYTHONHASHSEED'] = str(seed)\n \nset_seed()\n\ndevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\nprint('Device available now:', device)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 3. RNN with 1 Layer π\nRecurrent Neural Networks are very different from [FNNs](https://www.kaggle.com/andradaolteanu/how-i-taught-myself-deep-learning-vanilla-nns) or [CNNs](https://www.kaggle.com/andradaolteanu/how-i-taught-myself-deep-learning-convnet-cnns). \n\nRNNs model **sequential data**, meaning they have **sequential memory**. An RNN takes in different kind of inputs (text, words, letters, parts of an image, sounds, etc.) and returns different kinds of outputs (the next word/letter in the sequence, paired with an FNN it can return a classification etc.).\n\n![](\"https://i.imgur.com/RW41Wqj.png\")
\n\n**How RNN works**:\n1. It uses previous information to affect later ones\n2. There are 3 layers: *Input*, *Output* and *Hidden* (where the information is stored)\n3. The loop: passes the input forward sequentialy, while *retaining information* about it\n4. This info is stored in the *hidden state*\n5. There are only 3 matrixes (U, V, W) that contain weights as parameters. These *DON'T change* with the input, they stay the same through the entire sequence.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## 3.1 Youtube Videos to Save you Time π₯\nI *highly recommend* watching the following to better understand RNNs.\n\n","execution_count":null},{"metadata":{"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"YouTubeVideo(\"LHXXI4-IEns\", width=500, height=200)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.2 RNN with 1 Layer and 1 Neuron (π)\nYou can always increase the number of neurons in an RNN. For the moment we'll stick with 1. We'll have 2 timesteps, 0 and 1. The Architecture of our `class` will look like the figure below:\n\n
\n\n* `torch.mm()` - matrix multiplication","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# ==== STATICS ====\nn_inputs = 4\nn_neurons = 1\n# =================\n\n# RNN inputs\ninput0 = torch.tensor([[0, 1, 2, 0], [3, 4, 5, 0], [6, 7, 8, 0], [9, 0, 1, 0]], dtype = torch.float)\nprint('input time_0 shape:', input0.shape)\n\ninput1 = torch.tensor([[9, 8, 7, 0], [3, 4, 5, 0], [6, 7, 8, 0], [9, 0, 1, 0]], dtype = torch.float)\nprint('input time_1 shape:', input1.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# The Neural Network\nclass RNNVanilla(nn.Module):\n # __init__: the function where we create the architecture\n def __init__(self, n_inputs, n_neurons):\n super(RNNVanilla, self).__init__()\n \n # Weights are random at first\n # U contains connection weights for the inputs of the current time step\n self.U = torch.randn(n_inputs, n_neurons) # for 1 neuron: size = 4 rows and 1 column\n \n # W contains connection weights for the outputs of the previous time step\n self.W = torch.randn(n_neurons, n_neurons) # for 1 neuron: size = 1 row and 1 column\n \n # The bias\n self.b = torch.zeros(1, n_neurons) # for 1 neuron: size = 1 row and 1 column\n \n # forward: function where we apply the architecture to the input\n def forward(self, input0, input1):\n # Computes two outputs, one for each time step (two overall).\n self.output0 = torch.tanh(torch.mm(input0, self.U) + self.b)\n \n self.output1 = torch.tanh(torch.mm(self.output0, self.W) + torch.mm(input1, self.U) + self.b)\n \n return self.output0, self.output1","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Creating the model\nrnn_1_neuron = RNNVanilla(n_inputs, n_neurons)\n\n# Checking the output\noutput0, output1 = rnn_1_neuron(input0, input1)\nprint('output0:', output0, '\\n')\nprint('output1:', output1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Parameters\nprint('U:', rnn_1_neuron.U)\nprint('W:', rnn_1_neuron.W)\nprint('bias:', rnn_1_neuron.b)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.3 RNN with 1 Layer and Multiple Neurons (πππ)\n\n**Difference vs RNN 1 neuron 1 layer:**\n* size of output changes (because size of `n_neurons` changes)\n* size of the bias changes (it's the size of `n_neurons`) and `W` matrix\n \n
","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# ==== STATICS ====\nn_inputs = 3\nn_neurons = 5\n# =================\n\n# RNN inputs\ninput0 = torch.tensor([[0, 1, 2], [3, 4, 5], [6, 7, 8], [9, 0, 1]], dtype = torch.float)\nprint('input time_0 shape:', input0.shape)\n\ninput1 = torch.tensor([[9, 8, 7], [0, 0, 0], [6, 5, 4], [3, 2, 1]], dtype = torch.float)\nprint('input time_1 shape:', input1.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Creating the model\nrnn_n_neurons = RNNVanilla(n_inputs, n_neurons)\n\n# Checking the output\noutput0, output1 = rnn_n_neurons(input0, input1)\nprint('output0:', output0, '\\n')\nprint('output1:', output1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Parameters\nprint('U:', rnn_n_neurons.U)\nprint('W:', rnn_n_neurons.W)\nprint('bias:', rnn_n_neurons.b)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 3.4 Vanilla RNN for MNIST Classification π’\n\nFrom now on we'll use the build in `nn.RNN()` from `PyTorch`. As you see, the previous examples can't support large inputs and outputs, as we would have to input the information at every timestep and output the results.\n\nNote: When using RNN in image classification, it is hard to find the logic of \"why\" exactly are we doing this. It is not like CNNs, when we know we put many \"filters\" on the image to extract the essence. I observed that when using RNNs is just another mathematical method in which the computer learns numbers and can therefore identify patterns.\n\nThis is why RNNs might be weird in the approach for image classification, but nevertheless very effective.\n\nPro Tip: Use
","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# ==== STATICS ====\nbatch_size = 64 # how many images to be trained in one iteration\ninput_size = 28 # image 28 by 28\nhidden_size = 150 # can be changed to any number: neurons\noutput_size = 10 # 10 different digits\n# =================","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create a train_loader to select a batch from it\ntrain_loader = torch.utils.data.DataLoader(mnist_train, batch_size=64)\n\n# Select one full batch from the data\nimages_example, labels_example = next(iter(train_loader))\nprint('original images shape:', images_example.shape)\n\n# Reshape\nimages_example = images_example.view(-1, 28, 28)\nprint('changed images shape:', images_example.shape)\nprint('labels shape:', labels_example.shape, '\\n')\n\n# Creating the model\nmodel_example = VanillaRNN_MNIST(batch_size, input_size, hidden_size, output_size)\n\n\nout = model_example(images_example, prints=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> If we unfold the RNN:\n
","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Understand Model Parameters\nprint('Len parameters:', len(list(model_example.parameters())), '\\n' +\n 'Parameters 0 - U:', list(model_example.parameters())[0].shape, '\\n' +\n 'Parameters 1 - W:', list(model_example.parameters())[1].shape, '\\n' +\n 'Parameters 2 - Bias:', list(model_example.parameters())[2].shape, '\\n' +\n 'Parameters 3 - Bias:', list(model_example.parameters())[3].shape, '\\n' +\n 'Parameters 4 - FNN weights:', list(model_example.parameters())[4].shape, '\\n' +\n 'Parameters 5 - Predictions:', list(model_example.parameters())[5].shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 3.4.3 Training... π\n\n> We'll use `get_accuracy()` and `train_network()` functions from my [previous notebook](https://www.kaggle.com/andradaolteanu/how-i-taught-myself-deep-learning-vanilla-nns), but with some changes (suited to the RNN's needs).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_accuracy(out, actual_labels, batchSize):\n '''Saves the Accuracy of the batch.\n Takes in the log probabilities, actual label and the batchSize (to average the score).'''\n predictions = out.max(dim=1)[1]\n correct = (predictions == actual_labels).sum().item()\n accuracy = correct/batch_size\n \n return accuracy","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def train_network(model, train_data, test_data, batchSize=64, num_epochs=1, learning_rate=0.001):\n \n '''Trains the model and computes the average accuracy for train and test data.'''\n \n print('Get data ready...')\n # Create dataloader for training dataset - so we can train on multiple batches\n # Shuffle after every epoch\n train_loader = torch.utils.data.DataLoader(dataset=train_data, batch_size=batchSize, shuffle=True, drop_last=True)\n test_loader = torch.utils.data.DataLoader(dataset=test_data, batch_size=batchSize, shuffle=True, drop_last=True)\n \n # Create criterion and optimizer\n criterion = nn.CrossEntropyLoss()\n optimizer = optim.Adam(model.parameters(), lr=learning_rate)\n \n \n print('Training started...')\n # Train the data multiple times\n for epoch in range(num_epochs):\n \n # Save Train and Test Loss\n train_loss = 0\n train_acc = 0\n \n # Set model in training mode:\n model.train()\n \n for k, (images, labels) in enumerate(train_loader):\n \n # Get rid of the channel\n images = images.view(-1, 28, 28)\n \n # Create log probabilities\n out = model(images)\n # Clears the gradients from previous iteration\n optimizer.zero_grad()\n # Computes loss: how far is the prediction from the actual?\n loss = criterion(out, labels)\n # Computes gradients for neurons\n loss.backward()\n # Updates the weights\n optimizer.step()\n \n # Save Loss & Accuracy after each iteration\n train_loss += loss.item()\n train_acc += get_accuracy(out, labels, batchSize)\n \n \n # Print Average Train Loss & Accuracy after each epoch\n print('TRAIN | Epoch: {}/{} | Loss: {:.2f} | Accuracy: {:.2f}'.format(epoch+1, num_epochs, train_loss/k, train_acc/k))\n \n \n print('Testing Started...')\n # Save Test Accuracy\n test_acc = 0\n # Evaluation mode\n model.eval()\n \n for k, (images, labels) in enumerate(test_loader):\n # Get rid of the channel\n images = images.view(-1, 28, 28)\n \n # Create logit predictions\n out = model(images)\n # Add Accuracy of this batch\n test_acc += get_accuracy(out, labels, batchSize)\n \n # Print Final Test Accuracy\n print('TEST | Average Accuracy per {} Loaders: {:.5f}'.format(k, test_acc/k) )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# ==== STATICS ====\nbatch_size=64\ninput_size=28\nhidden_size=150\noutput_size=10\n\n# Instantiate the model\nvanilla_rnn = VanillaRNN_MNIST(batch_size, input_size, hidden_size, output_size)\n\n# ==== TRAIN ====\ntrain_network(vanilla_rnn, mnist_train, mnist_test, num_epochs=10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"learning_rate to 0.01 and see what happens. Also try changing the batch_size to 20 instead of 64. Try adding weight_decay to the optimizer functions. The accuracy of the model now is impressive, but by altering some of these hyperparameters can change this \"sweet spot\" we found instantly.\n
\n\n**Activation functions: ReLU vs Tanh**\n> use them to try to erase the vanishing gradients problem (we'll come back to these in the next chapter LSTM)\n
\n\n## 4.2 Multilayer RNN for MNIST Classification π’","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class MultilayerRNN_MNIST(nn.Module):\n def __init__(self, input_size, hidden_size, layer_size, output_size, relu=True):\n super(MultilayerRNN_MNIST, self).__init__()\n self.input_size, self.hidden_size, self.layer_size, self.output_size = input_size, hidden_size, layer_size, output_size\n \n # Create RNN\n if relu:\n self.rnn = nn.RNN(input_size, hidden_size, layer_size, batch_first=True, nonlinearity='relu')\n else:\n self.rnn = nn.RNN(input_size, hidden_size, layer_size, batch_first=True, nonlinearity='tanh')\n \n # Create FNN\n self.fnn = nn.Linear(hidden_size, output_size)\n \n def forward(self, images, prints=False):\n if prints: print('images shape:', images.shape)\n \n # Instantiate hidden_state at timestamp 0\n hidden_state = torch.zeros(self.layer_size, images.size(0), self.hidden_size)\n hidden_state = hidden_state.requires_grad_()\n if prints: print('Hidden State shape:', hidden_state.shape)\n \n # Compute RNN\n # .detach() is required to prevent vanishing gradient problem\n output, last_hidden_state = self.rnn(images, hidden_state.detach())\n if prints: print('RNN Output shape:', output.shape, '\\n' +\n 'RNN last_hidden_state shape', last_hidden_state.shape)\n \n # Compute FNN\n # We get rid of the second size\n output = self.fnn(output[:, -1, :])\n if prints: print('FNN Output shape:', output.shape)\n \n return output","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Understanding the Model:\n\n> Here is what's happening in the batch below:\n
","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# ===== STATICS =====\nbatch_size = 64\ninput_size = 28\nhidden_size = 100 # neurons\nlayer_size = 2 # layers\noutput_size = 10\n# ===================","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_loader_example = torch.utils.data.DataLoader(mnist_train, batch_size=64)\n\n# Taking a single batch of the images\nimages, labels = next(iter(train_loader_example))\nprint('original images shape:', images.shape)\n\n# Remove channel from shape\nimages = images.reshape(-1, 28, 28)\nprint('reshaped images shape:', images.shape, '\\n')\n\n# Create model instance\nmultilayer_rnn_example = MultilayerRNN_MNIST(input_size, hidden_size, layer_size, output_size, relu=False)\nprint(multilayer_rnn_example)\n\n\n# Making log predictions:\nout = multilayer_rnn_example(images, prints=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> If we unfold the Multilayer RNN Example:\n
","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Training... π\nLet's see how the model performs by adding 1 more layer.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# ==== STATICS ====\nbatch_size = 64\ninput_size = 28\nhidden_size = 100 \nlayer_size = 2 \noutput_size = 10\n\n# Instantiate the model\n# We'll use TANH as our activation function\nmultilayer_rnn = MultilayerRNN_MNIST(input_size, hidden_size, layer_size, output_size, relu=False)\n\n# ==== TRAIN ====\ntrain_network(multilayer_rnn, mnist_train, mnist_test, num_epochs=10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Accuracy improves *faster* compared to the Vanilla RNN, while final TEST Accuracy is slightly bigger.\n\n# 5. LSTM (Long Short Term Memory RNNs) πΎ\n\n## 5.1 Material to Save you Time π₯\n\nI *highly recommend* going through the references below before continuing. You will understand how LSTMs are different from RNNs, how they works and what is Vanishing Gradient Problem.\n\n\nIllustrated Guide to LSTM's and GRU's: A step by step explanation
\nAlso this amazing blog post explains the Vanishing Gradient Problem and LSTMs
\n
\n\n## 5.4 How does LSTM work?\nAn LSTM is more complex than an simple RNN:\n* it is composed by cell states and gates\n* it has the purpose to LEARN what to remember and forget reduntant information\n* it uses SIGMOID functions instead of TANH\n* Composition of the cell in LSTM:\n * the cell has 2 outputs: the cell state and the hidden state\n \n 1. Forget Gate (`Xt + ht-1`): desides what information to FORGET; the closer to 0 is forget, the closer to 1 is remain\n 2. Input Gate (`Xt + ht-1`): creates a candidate with what information to remain\n 3. Current Cell State: `ft*Ct-1 + it*Ct`\n 4. Output Gate (`(Xt + ht-1) * ct`): desides what the next hidden state should be (which contains info about previous inputs)\n \n> Note: [Check THIS blog post for more detailed explanation.](https://towardsdatascience.com/illustrated-guide-to-lstms-and-gru-s-a-step-by-step-explanation-44e9eb85bf21)\n \n
\n \n> The **Stacked LSTM** is like the Multilayer RNN: it has multiple hidden LSTM layers which contain multiple memory cells\n\n## 5.5 LSTM for MNIST Classification π’\n\n> **Bidirectional LSTMs** : are an extension of traditional LSTMs that can improve model performance on sequence classification problems. They train the model *forward* and *backward* on the same input (so for 1 layer LSTM we get 2 hidden and cell states)\n\n
","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"class LSTM_MNIST(nn.Module):\n def __init__(self, input_size, hidden_size, layer_size, output_size, bidirectional=True):\n super(LSTM_MNIST, self).__init__()\n \n self.input_size, self.hidden_size, self.layer_size, self.output_size = input_size, hidden_size, layer_size, output_size\n self.bidirectional = bidirectional\n \n # Step1: the LSTM model\n self.lstm = nn.LSTM(input_size, hidden_size, layer_size, batch_first=True, bidirectional=bidirectional)\n \n # Step2: the FNN\n if bidirectional: # we'll have 2 more layers\n self.layer = nn.Linear(hidden_size*2, output_size)\n else:\n self.layer = nn.Linear(hidden_size, output_size)\n \n \n def forward(self, images, prints=False):\n if prints: print('images shape:', images.shape)\n \n # Set initial states\n if self.bidirectional:\n # Hidden state:\n hidden_state = torch.zeros(self.layer_size*2, images.size(0), self.hidden_size)\n # Cell state:\n cell_state = torch.zeros(self.layer_size*2, images.size(0), self.hidden_size)\n else:\n # Hidden state:\n hidden_state = torch.zeros(self.layer_size, images.size(0), self.hidden_size)\n # Cell state:\n cell_state = torch.zeros(self.layer_size, images.size(0), self.hidden_size)\n if prints: print('hidden_state t0 shape:', hidden_state.shape, '\\n' +\n 'cell_state t0 shape:', cell_state.shape)\n \n # LSTM:\n output, (last_hidden_state, last_cell_state) = self.lstm(images, (hidden_state, cell_state))\n if prints: print('LSTM: output shape:', output.shape, '\\n' +\n 'LSTM: last_hidden_state shape:', last_hidden_state.shape, '\\n' +\n 'LSTM: last_cell_state shape:', last_cell_state.shape)\n # Reshape\n output = output[:, -1, :]\n if prints: print('output reshape:', output.shape)\n \n # FNN:\n output = self.layer(output)\n if prints: print('FNN: Final output shape:', output.shape)\n \n return output","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### How the Model Works:\n> Below is a schema of how the example code works\n
","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# ====== STATICS ======\nbatch_size = 64\ninput_size = 28 # width of image\nhidden_size = 128 # number of hidden neurons\nlayer_size = 2 # number of layers\noutput_size = 10 # possible choices\n# =====================","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Taking a single batch of the images\nimages, labels = next(iter(train_loader_example))\nprint('original images shape:', images.shape)\n# Remove channel from shape\nimages = images.reshape(-1, 28, 28)\nprint('reshaped images shape:', images.shape, '\\n')\n\n# Creating the Model\nlstm_example = LSTM_MNIST(input_size, hidden_size, layer_size, output_size)\nprint('lstm_example:', lstm_example, '\\n')\n\n# Making log predictions:\nout = lstm_example(images, prints=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Training on ALL IMAGES π","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# ==== STATICS ====\nbatch_size = 64\ninput_size = 28\nhidden_size = 100 \nlayer_size = 2 \noutput_size = 10\n\n# Instantiate the model\n# We'll use TANH as our activation function\nlstm_rnn = LSTM_MNIST(input_size, hidden_size, layer_size, output_size)\n\n# ==== TRAIN ====\ntrain_network(lstm_rnn, mnist_train, mnist_test, num_epochs=10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Now we get even *HIGHER* accuracies than the ones before. \n> If we make a recap, FNNs from my previous notebook had an accuracy of ~ 80%, CNNs had and accuracy of almost 90%, while RNN reached 97%. Lastly, **LSTMs** were the best performing ones (99% accuracy).\n\n# 6. Bonuses β\n## 6.1 Confusion Matrix π€\n\nA good way to visualize better how the model is performing is through a confusion matrix. So, you can see how well each label is predicted and what labels the model confuses with other labels (for example a 7 can be sometimes confused with 1).","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_confusion_matrix(model, test_data):\n # First we make sure we disable Gradient Computing\n torch.no_grad()\n \n # Model in Evaluation Mode\n model.eval()\n \n preds, actuals = [], []\n\n for image, label in mnist_test:\n image = image.view(-1, 28, 28)\n out = model(image)\n\n prediction = torch.max(out, dim=1)[1].item()\n preds.append(prediction)\n actuals.append(label)\n \n return sklearn.metrics.confusion_matrix(preds, actuals)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(16, 5))\nsns.heatmap(get_confusion_matrix(lstm_rnn, mnist_test), cmap='icefire', annot=True, linewidths=0.1,\n fmt = ',')\nplt.title('Confusion Matrix: LSTM', fontsize=15);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 6.2 Why shouldn't you use Transfer Learning?π§ \n\nTransfer learning is a genious way to use the weights of a pretrained model on another set of images. This technique is often used in deep learning classification problems that use CNN (like EffNets, ResNets etc).\n\nHowever, this is not a regular technique used for RNN networks. This is mainly because *recurrent* data cannot really be generalized like static data (images) can be.\n\n[Read more about this here.](https://www.quora.com/In-recurrent-neural-networks-like-LSTMs-is-it-possible-to-do-transfer-learning-Has-there-been-any-research-in-this-area)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Other How I taught myself Deep Learning Notebooksπ\n* [How I taught myself Deep Learning: Vanilla NNs](https://www.kaggle.com/andradaolteanu/how-i-taught-myself-deep-learning-1-pytorch-fnn)\n* [Convolutional Neural Nets (CNNs) Explained](https://www.kaggle.com/andradaolteanu/convolutional-neural-nets-cnns-explained)\n\nIf you have any questions, please do not hesitate to ask. This notebook is made to bring more clear understanding of concepts and coding, so this would also help me add, modify and improve it. \n\nIf you liked this, upvote!
\nCheers!
\n